Can We Increase the Value and Decrease the Cost of Clinical Skills Assessment?

Cancel culture: exploring the unintended consequences of cancelling the Canadian national licensing clinical examination

Assessment drives learning. However, when it comes to high-stakes examinations (e.g., for licensure or certification), these assessments of learning may be seen as unnecessary hurdles by some. Licensing clinical skills assessment in particular have come under fire over the years. Recently, assessments such as the Medical Council of Canada Qualifying Examination Part II, a clinical skills objective structured clinical examination, have been permanently cancelled. The authors explore potential consequences of this cancellation including those that are inadvertent and undesirable. Future next steps for clinical skills assessment are explored.

Licensing exams in Canada: a closer look at the validity of the MCCQE Part II

The Medical Council of Canada Qualifying Exam (MCCQE) Part II aims to protect societal interests through examining recently graduated physicians using clinical scenarios with standardized patients. This position paper debates the role of the MCCQE Part II in the national licensing of physicians in Canada by focusing on the consequential validity evidence of this exam and considering future directions through discussing contemporary developments in high stakes examinations. Specifically, this paper compares both MCCQE Part I and Part II in their ability to predict future practice patterns of physicians and generalizability across specialties. In weighing up the evidence this paper considers commonly used counterarguments as well as the financial implications of this exam for both the candidates and the MCC. Finally, it concludes by providing recommendations for future licensing of physicians in Canada. The available consequential validity evidence for MCCQE Part II is limited. Though still limited, MCCQE Part I has more robust evidence that it is a better predictor of future practice patterns compared to with Part II. Combined with a lack of evidence that national licensing examinations lead to graduation of substandard doctors or an improvement of care, and the shift away from assessment of learning towards assessment for learning, the maximum impact of the MCC on safeguarding public's interests will lie in working closely with residency programs and specialty colleges to facilitate a robust assessment program of essential competencies and clinical skills during residency training and specialty certification.

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate

BACKGROUND

Most medical schools in the United States report having a 5- to 10-station objective structured clinical examination (OSCE) at the end of the core clerkship phase of the curriculum to assess clinical skills. We set out to investigate an alternative OSCE structure in which each clerkship has a 2-station OSCE. This study looked to determine the reliability of clerkship OSCEs in isolation to inform composite clerkship grading, as well as the reliability in aggregate, as a potential alternative to an end-of-third-year examination.

DESIGN

Clerkship OSCE data from the 2017-2018 academic year were analyzed: the generalizability coefficient (ρ²) and index of dependability (φ) were calculated for clerkships in isolation and in aggregate using variance components analysis.

RESULTS

In all, 93 students completed all examinations. The average generalizability coefficient for the individual clerkships was .47. Most often, the largest variance component was the interaction between the student and the station, indicating inconsistency in the performance of students between the 2 stations. Aggregate clerkship OSCE analysis demonstrated good reliability for consistency (ρ² = .80). About one-third (33.8%) of the variance can be attributed to students, 8.2% can be attributed to the student by clerkship interaction, and 42.6% can be attributed to the student by block interaction, indicating that students' relative performances varied by block.

CONCLUSIONS

Two-station clerkship OSCEs have poor to fair reliability, and this should inform the weighting of the composite clerkship grade. Aggregating data results in good reliability. The largest source of variance in the aggregate was student by block, suggesting testing over several blocks may have advantages compared with a single day examination.

What we measure … and what we should measure in medical education

CONTEXT

As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society.

OBJECTIVES

We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role.

METHODS

We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies.

DISCUSSION

As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.