The impact of large scale licensing examinations in highly developed countries: a systematic review

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Ethiopian Anesthetist Licensing Examination: A Qualitative Study of Concerns and Unintended Consequences

Background

Ethiopia increased its anesthesia workforce drastically by expanding the training of associate clinician anesthetists. Following this expansion, the Ministry of Health established an entry-level anesthesia licensing examination to ensure patient safety. However, there is limited empirical evidence on the impacts of licensing exams in low- and middle-income countries. This study aimed to explore the concerns and undesirable consequences of the anesthetist licensing examination in Ethiopia.

Methods

A qualitative design using a grounded theory approach was employed by collecting data from 10 anesthesia teaching institutions. We conducted 15 in-depth interviews with instructors and six focus groups with students and graduates who took the exam recently. Interviews and focus groups were audio-recorded, transcribed verbatim, and analyzed using Atlas.ti 23. We also extracted secondary data from the academic committee meeting minutes, curricula, faculty appraisal reports, and program quality self-review reports.

Results

Qualitative analysis revealed three central categories of concerns and untoward consequences of the anesthetist NLE: exam management, educational management, and student behavior. Exam management concerns were related to exam validity, fairness, and consistent enforcement of pass/fail decisions. The unintended consequences of the exam on education management were perceived as promoting teaching and learning for the exam, increasing faculty workload, and resulting in superficial and patchy educational reforms. Study participants also reported adverse psychosocial effects and increased cheating behaviors among students as undesirable consequences of the exam on student behavior.

Conclusion

Our study identified some concerns and unintended consequences of the Ethiopian anesthetist licensing examination. These lessons learned may contribute to improving the quality of licensing examinations in Ethiopia and beyond.

Design, delivery and effectiveness of health practitioner regulation systems: an integrative review

BACKGROUND

Health practitioner regulation (HPR) systems are increasingly recognized as playing an important role in supporting health workforce availability, accessibility, quality, and sustainability, while promoting patient safety. This review aimed to identify evidence on the design, delivery and effectiveness of HPR to inform policy decisions.

METHODS

We conducted an integrative analysis of literature published between 2010 and 2021. Fourteen databases were systematically searched, with data extracted and synthesized based on a modified Donabedian framework.

FINDINGS

This large-scale review synthesized evidence from a range of academic (n = 410) and grey literature (n = 426) relevant to HPR. We identified key themes and findings for a series of HPR topics organized according to our structures-processes-outcomes conceptual framework. Governance reforms in HPR are shifting towards multi-profession regulators, enhanced accountability, and risk-based approaches; however, comparisons between HPR models were complicated by a lack of a standardized HPR typology. HPR can support government workforce strategies, despite persisting challenges in cross-border recognition of qualifications and portability of registration. Scope of practice reform adapted to modern health systems can improve access and quality. Alternatives to statutory registration for lower-risk health occupations can improve services and protect the public, while standardized evaluation frameworks can aid regulatory strengthening. Knowledge gaps remain around the outcomes and effectiveness of HPR processes, including continuing professional development models, national licensing examinations, accreditation of health practitioner education programs, mandatory reporting obligations, remediation programs, and statutory registration of traditional and complementary medicine practitioners.

CONCLUSION

We identified key themes, issues, and evidence gaps valuable for governments, regulators, and health system leaders. We also identified evidence base limitations that warrant caution when interpreting and generalizing the results across jurisdictions and professions. Themes and findings reflect interests and concerns in high-income Anglophone countries where most literature originated. Most studies were descriptive, resulting in a low certainty of evidence. To inform regulatory design and reform, research funders and governments should prioritize evidence on regulatory outcomes, including innovative approaches we identified in our review. Additionally, a systematic approach is needed to track and evaluate the impact of regulatory interventions and innovations on achieving health workforce and health systems goals.

The Ethiopian Anesthetist Licensing Examination and Associated Improvement in in-School Student Performance: A Retrospective Study

Background

Ethiopia introduced a national licensing examination (NLE) in response to growing concerns about the competence of graduates and the quality of education. This study aimed to assess the associated in-school student performance changes in anesthetist training programs following NLE implementation.

Methods

Academic records of 1493 graduate anesthetists were retrospectively obtained from eight universities before (n=932) and after (n=561) NLE implementation. Four universities were first-generation (oldest), three were second-generation, and one was third-generation (newest). We compared the yearly (Y1 to Y4) and cumulative grade point averages (GPA) to assess if there were in-school student performance differences between the two periods. The Kruskal-Wallis and Mann-Whitney U-tests were used to compare groups. Results are presented as a median, interquartile range, a 95% confidence interval (CI) for median differences, and Cohen's r effect size.

Results

Overall, there was a small to moderate improvement in student academic performance following NLE implementation. However, the statistically significant differences were limited to first-generation university students and those entering directly from high school. We found considerable positive differences in all five performance measures in first-generation university students, with Year-1 GPA and cumulative GPA measurements exhibiting large effect sizes (Cohen's r = 0.96 and 0.79, respectively, p <0.005). Those entering from high school demonstrated significant differences in four of five performance measures, with the largest positive gains on the year-1 GPA (median before [n=765] and after [n=480]: 3.11-3.30, 95% CI (0.09, 0.22), r=0.46, p <0.005)). Second- and third-generation university students showed no significant differences, while nurse entrants exhibited a significant difference in their Y2GPA scores only with an actual drop in performance.

Conclusion

The Ethiopian anesthetist NLE is associated with an overall modest in-school academic performance improvement, supporting its use. The stagnant or declining performance among nurse entrants and the newest (second and third) generation university students deserve further scrutiny.

Impact of anesthetist licensing examination on quality of education in Ethiopia: a qualitative study of faculty and student perceptions

BACKGROUND

Ethiopia drastically increased the anesthesia workforce density by training 'associate clinician anesthetists' as a task-shifting and sharing strategy. However, there were growing concerns about educational quality and patient safety. Accordingly, the Ministry of Health introduced the anesthetist national licensing examination (NLE) to assure the quality of education. However, empirical evidence is scarce to support or refute the overall impact of NLEs, which are relatively costly for low- and middle-income settings. Therefore, this study aimed to explore the impact of introducing NLE on anesthetists' education in Ethiopia.

METHODS

We conducted a qualitative study using a constructivist grounded theory approach. Data were prospectively collected from ten anesthetist teaching institutions. Fifteen in-depth interviews were conducted with instructors and academic leaders, and six focus groups were held with students and recently tested anesthetists. Additional data were gathered by analyzing relevant documents, including versions of curricula, academic committee minutes, program quality review reports, and faculty appraisal reports. Interviews and group discussions were audiotaped, transcribed verbatim and analyzed using Atlas.ti 9 software.

RESULTS

Both faculty and students demonstrated positive attitudes toward the NLE. Student motivation, faculty performance, and curriculum strengthening were the three primary changes that emerged, resulting in three subsequent spin-offs on assessment, learning, and quality management practices. Academic leaders' dedication to evaluating examination data and turning these into action led to changes that improved education quality. Increased accountability, engagement, and collaboration were the predominant factors facilitating change.

CONCLUSION

Our study indicates that the Ethiopian NLE has prompted anesthesia teaching institutions to improve their teaching, learning, and assessment practices. However, more work is required to improve exam acceptability among stakeholders and drive broader changes.

Factors associated with medical students’ scores on the National Licensing Exam in Peru: a systematic review

PURPOSE

This study aimed to identify factors that have been studied for their associations with National Licensing Examination (ENAM) scores in Peru.

METHODS

A search was conducted of literature databases and registers, including EMBASE, SciELO, Web of Science, MEDLINE, Peru’s National Register of Research Work, and Google Scholar. The following key terms were used: “ENAM” and “associated factors.” Studies in English and Spanish were included. The quality of the included studies was evaluated using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

In total, 38,500 participants were enrolled in 12 studies. Most (11/12) studies were cross-sectional, except for one case-control study. Three studies were published in peer-reviewed journals. The mean MERSQI was 10.33. A better performance on the ENAM was associated with a higher-grade point average (GPA) (n=8), internship setting in EsSalud (n=4), and regular academic status (n=3). Other factors showed associations in various studies, such as medical school, internship setting, age, gender, socioeconomic status, simulations test, study resources, preparation time, learning styles, study techniques, test-anxiety, and self-regulated learning strategies.

CONCLUSION

The ENAM is a multifactorial phenomenon; our model gives students a locus of control on what they can do to improve their score (i.e., implement self-regulated learning strategies) and faculty, health policymakers, and managers a framework to improve the ENAM score (i.e., design remediation programs to improve GPA and integrate anxiety-management courses into the curriculum).

Associations between Drinking Behaviors and Meaning in Life: Evidence from Primary Care Professionals in China

(1) Background: Although the associations between drinking behaviors and emotional problems have been supported in several previous studies, the associations between drinking behaviors and meaning in life have not been explored until now. We aimed to test the associations between drinking behaviors and meaning in life among primary care professionals, after controlling for depression. (2) Methods: In the current study, we collected 1453 valid questionnaires based on a cross-sectional design. Meaning in life, drinking behaviors, physical diseases, depression, work-related variables, and some other social-demographic variables were evaluated. (3) Results: The results support that after controlling for depression, regular milk drinking (β = 1.387, p = 0.026), and regular juice drinking (β = 2.316, p = 0.030) were associated with higher meaning in life, while regular water drinking (β = −1.448, p = 0.019) was negatively associated with meaning in life. In addition to this, the results showed that the older age (β = 0.098, p = 0.001), preventive medicine majors (β = 4.281, p = 0.013), working fewer days per week (β = −0.942, p = 0.004), licensed (assistant) technician qualification (β = 2.921, p = 0.036), and no depression (β = −0.203, p < 0.001) were positively associated with meaning in life. (4) Conclusion: This study supported the association between drinking behaviors and meaning in life, even after controlling depression. These findings imply that we can further explore this association and its mechanisms in future studies.

Essential steps in the development, implementation, evaluation and quality assurance of the written part of the Swiss federal licensing examination for human medicine

PURPOSE

This report describes the essential steps in the development, implementation, evaluation and quality assurance of the written part of the Swiss Federal Licensing Examination for Human Medicine (FLE) and the insights gained since its introduction in 2011.

METHODS

Based on existing scientific evidence, international expertise, and experience gained from previous examinations, the FLE is developed by experts from all five medical faculties in Switzerland with the support of the Institute for Medical Education and is held simultaneously at five locations. The exam organisers document and review every examination held and continuously optimise the processes; they have summarised the results in this report.

RESULTS

The essential steps comprise the development, revision and translation of questions; construction of the exam and production of materials; candidate preparation; implementation and analysis. The quality assurance measures consist of guideline coherence in the development of the questions and implementation of the exam, revision processes, construction of the exam based on the national blueprint, multiphase review of the translations and exam material, and statistical analysis of the exam and the comments from candidates. The intensive collaboration, especially on the part of representatives from all the participating faculties and a central coordination unit, which provides methodological support throughout and oversees the analysis of the exam, has proven successful. Successfully completed examinations and reliable results in the eleven examinations so far implemented represent the outcomes of the quality assurance measures. Significant insights in recent years are the importance of appreciating the work of those involved and the central organisation of exam development, thus ensuring the long-term success of the process.

CONCLUSION

Common guidelines and workshops, quality assurance measures accompanied by the continuous improvement of all processes, and appreciation of everyone involved, are essential to carrying out such an examination at a high-quality level in the long term.

Medical Council of Canada Qualifying Examinations and performance in future practice

The purpose of medical licensing examinations is to protect the public from practitioners who do not have adequate knowledge, skills, and abilities to provide acceptable patient care, and therefore evaluating the validity of these examinations is a matter of accountability. Our objective was to discuss the Medical Council of Canada's Qualifying Examinations (MCCQEs) Part I (QE1) and Part II (QE2) in terms of how well they reflect future performance in practice. We examined the supposition that satisfactory performance on the MCCQEs are important determinants of practice performance and, ultimately, patient outcomes. We examined the literature before the implementation of the QE2 (pre-1992), post QE2 but prior to the implementation of the new Blueprint (1992-2018), and post Blueprint (2018-present). The literature suggests that MCCQE performance is predictive of future physician behaviours, that the relationship between examination performance and outcomes did not attenuate with practice experience, and that associations between examination performance and outcomes made sense clinically. While the evidence suggests the MCC qualifying examinations measure the intended constructs and are predictive of future performance, the validity argument is never complete. As new competency requirements emerge, we will need to develop valid and reliable mechanisms for determining practice readiness in these areas.

Licensing exams in Canada: a closer look at the validity of the MCCQE Part II

The Medical Council of Canada Qualifying Exam (MCCQE) Part II aims to protect societal interests through examining recently graduated physicians using clinical scenarios with standardized patients. This position paper debates the role of the MCCQE Part II in the national licensing of physicians in Canada by focusing on the consequential validity evidence of this exam and considering future directions through discussing contemporary developments in high stakes examinations. Specifically, this paper compares both MCCQE Part I and Part II in their ability to predict future practice patterns of physicians and generalizability across specialties. In weighing up the evidence this paper considers commonly used counterarguments as well as the financial implications of this exam for both the candidates and the MCC. Finally, it concludes by providing recommendations for future licensing of physicians in Canada. The available consequential validity evidence for MCCQE Part II is limited. Though still limited, MCCQE Part I has more robust evidence that it is a better predictor of future practice patterns compared to with Part II. Combined with a lack of evidence that national licensing examinations lead to graduation of substandard doctors or an improvement of care, and the shift away from assessment of learning towards assessment for learning, the maximum impact of the MCC on safeguarding public's interests will lie in working closely with residency programs and specialty colleges to facilitate a robust assessment program of essential competencies and clinical skills during residency training and specialty certification.

Dental foundation training in the COVID-19 era - the national recruitment lottery

National recruitment has radically transformed the selection and allocation of dental graduates to dental foundation training (DFT) schemes across England, Wales and Northern Ireland, with recruitment in Scotland via a separate, independent process. It has been reported as a mostly positive change to the previous deanery-led model, in which nepotism allegedly featured too widely. A candidate's ranking is typically based on performance across two face-to-face assessments and a situational judgement test (SJT). The COVID-19 pandemic, however, has created a recruitment 'lottery' of sorts, in which ranking for 2021 is now solely based on the SJT. Subject matter experts assert that neither preparation nor revision for the SJT is required; yet, following the announcement of the adaptation to DFT recruitment for September 2021, a rapid rise of exorbitantly priced SJT question banks, mock papers, workshops and courses has ensued - a shameless monetisation of the collective angst and increased pressure faced by dental students. Preparation courses present a conceivable risk of SJT 'coaching' and 'faking'. Where medicine leads, dentistry usually follows and future selection to DFT needs to strongly consider the introduction of academic performance measures. SJTs remain one of the most well-accepted, reliable and cost-effective means of selection into healthcare roles, however, only when part of a wider selection process.

Association between graduates' factors and success rate on the Saudi Pharmacist Licensure Examination: A single-Institution cross-sectional study

Introduction

The Saudi Pharmacists Licensure Examination’s (SPLE) initial results were revealed in February 2020, which showed enormous inconsistency with the passing rates. Hence, we aimed to investigate the predictors of pharmacy graduates’ success rate on the SPLE of a single-college in Saudi Arabia.

Methods

In a cross-sectional study of 2019-graduates, data were collected via-phone interviews and retrieved from the college’s database. Graduates’ data were retrieved from the college’s database, which includes high school grade point average (GPA), General Aptitude Test, Scholastic Achievement Admission Test, preparatory year GPA, pharmacy GPA, and all courses' grades. The data collected via-phone interview includes information that was not recorded in the college’s database, such as date of birth, SPLE score, and the number of SPLE attempts. We only included students who agreed to be interviewed and had attempted the SPLE at least once. We developed a prediction model using correlational analyses and stepwise multiple linear regression to examine graduates’ factors in predicting success on the SPLE.

Results

Out of the 149 graduates, 105 fulfilled the inclusion criteria. Around 84% of them passed the SPLE. Correlation analyses showed a significant direct correlation between preparatory year GPA and pharmacy GPA with SPLE scores (r = 0.228, P = 0.0193 and r = 0.533, P < 0.0001, respectively). In addition, the results showed that all of the four SPLE content areas were positively correlated with SPLE scores (P < 0.0001). The stepwise multivariable regression showed that being a male graduate and scoring high-grade points in pharmacology and therapeutics courses were significantly associated with high SPLE scores (P = 0.0053, P = 0.0256, and P = 0.0001, respectively).

Conclusion

This single cross-sectional study found that being a male, pharmacology GPA, and therapeutics GPA were significantly associated with a higher SPLE score. Further studies should focus on the GPA cut off below which we should give remediation to improve SPLE passing rate.

The introduction of a standardised national licensing exam as a driver of change in medical education: A qualitative study from Switzerland

INTRODUCTION

Only a few studies have described the impacts, strengths and needs for further development of national licensing exams (NLE). To gain such insights regarding the Swiss NLE, which includes a multiple-choice and a standardised clinical skills exam, we explored the perceptions of involved experts and stakeholders.

METHODS

We explored participants' perceptions in four focus group discussions. The interviews were recorded, transcribed verbatim and qualitatively analysed using a thematic analysis approach.

RESULTS

The analysis resulted in five perceived impacts, two strengths and two needs for further developments of the NLE. Perceived impacts were (1) steering students' learning behaviour, (2) supporting teachers and assessors to align teaching to competencies, (3) elevating the importance of the Swiss Catalogue of Learning Objectives, (4) setting incentives for the further development of curricula, and (5) fostering the collaboration between the faculties of medicine. Perceived strengths were the blend of assessment formats, including their competency-based orientation, and the collaborative development approach. Perceived needs lay in the NLE's further development to sustain its fit for purpose and in incentives for people involved.

CONCLUSION

According to our study, this NLE had, and has, notable impacts on medical education in Switzerland. Our insights can be useful for others planning a similar undertaking.

Differential attainment in the MRCPsych according to ethnicity and place of qualification between 2013 and 2018: a UK cohort study

PURPOSE OF THE STUDY

To explore if differential pass rates exist in the clinical component of the UK postgraduate clinical psychiatry exam, the Clinical Assessment of Skills and Competencies (CASC), according to ethnicity and place of qualification (UK vs EEA vs overseas graduates).

STUDY DESIGN

Observational study using data from the UK Medical Education Database for 2140 doctors sitting the CASC for the first time between 2013 and 2018.

RESULTS

After controlling for age, sex, time of sitting and performance in the written components of the MRCPsych, differences in CASC pass rates persisted between UK graduates self-identifying as Black and Minority Ethnicity (BME) and non-BME (OR for passing 0.36, 95% CI 0.23 to 0.56, p<0.001). Both EEA (OR 0.25, 0.15 to 0.40, p<0.001) and overseas graduates (OR 0.07, 0.05 to 0.11, p<0.001) were less likely to pass the CASC at first attempt, even after controlling for the influence of educational and background variables. These groups, on average, had lower scores on written exams with substantial content relating to procedural skills (eg, critical appraisal) rather than pure recall of factual knowledge.

CONCLUSIONS

Substantial differences exist in clinical examination performance between UK BME and non-BME candidates, as well as between UK and non-UK graduates. These differences are not explained by differing levels of clinical knowledge. In the interests of equality, this situation requires further investigation and remediation. Future research should focus on understanding how potential bias may be acting within different stages of recruitment, training and assessment within psychiatry.

An international stakeholder survey of the role of chiropractic qualifying examinations: A qualitative analysis

OBJECTIVE

Many countries require examinations as a gateway to chiropractic licensure; however, the relevance of these exams to the profession has not been explored. The purposes of this study were to analyze perceptions of international stakeholders about chiropractic qualifying examinations (CQEs), observe if their beliefs were in alignment with those that society expects of professions, and suggest how this information may be used when making future decisions about CQEs.

METHODS

We designed an electronic survey that included open-ended questions related to CQEs. In August 2019, the survey was distributed to 234 international stakeholders representing academic institutions, qualifying boards, students, practitioners, association officers, and others. Written comments were extracted, and concepts were categorized and collapsed into 4 categories (benefits, myths, concerns, solutions). Qualitative analysis was used to identify themes.

RESULTS

The response rate was 56.4% representing 43 countries and yielding 775 comments. Perceived benefits included that CQEs certify a minimum standard of knowledge and competency and are part of the professionalization of chiropractic. Myths included that CQEs are able to screen for future quality of care or ethical practices. Concerns included a lack of standardization between jurisdictions and uncertainty about the cost/value of CQEs and what they measure. Solutions included suggestions to standardize exams across jurisdictions and focus on competencies.

CONCLUSION

International stakeholders identified concepts about CQEs that may facilitate or hinder collaboration and efforts toward portability. Stakeholder beliefs were aligned with those expected of learned professions. This qualitative analysis identified 9 major themes that may be used when making future decisions about CQEs.

Performance in candidates declaring versus those not declaring dyslexia in a licensing clinical examination

CONTEXT

High-stakes medical examinations seek to be fair to all candidates, including an increasing proportion of trainee doctors with specific learning differences. We aimed to investigate the performance of doctors declaring dyslexia in the clinical skills assessment (CSA), an objective structured clinical examination for licensing UK general practitioners.

METHODS

We employed a cross-sectional design using performance and attribute data from candidates taking the CSA between 2010 and 2017. We compared candidates who declared dyslexia ('early' before their first attempt or 'late' after failing at least once) with those who did not, using multivariable negative binomial regression investigating the effect of declaring dyslexia on passing the CSA, accounting for relevant factors previously associated with performance, including number of attempts, initial score, sex, place of primary medical qualification and ethnicity.

RESULTS

Of 20 879 CSA candidates, 598 (2.9%) declared that they had dyslexia. Candidates declaring dyslexia were more likely to be male (47.3% versus 37.8%; p < 0.001) and to have a non-UK primary medical qualification (26.9% versus 22.4%; p < 0.01), but were no different in ethnicity compared with those who never declared dyslexia. Candidates who declared dyslexia late were significantly more likely to fail compared with those candidates who declared dyslexia early (40.6% versus 9.2%; p < 0.001) and were more likely to have a non-UK medical qualification (79.3% versus 15.6%; p < 0.001) or come from a minority ethnic group (84.9% versus 39.2%; p < 0.001). The chance of passing was lower for candidates declaring dyslexia compared to those who never declared dyslexia and lower in those declaring late (incident rate ratio [IRR], 0.82; 95% confidence interval [CI], 0.70-0.96) compared with those declaring early (IRR, 0.95; 95% CI, 0.93-0.97).

CONCLUSIONS

A small proportion of candidates declaring dyslexia were less likely to pass the CSA, particularly if dyslexia was declared late. Further investigation of potential causes and solutions is needed.

Does performance on United States national board of medical examiners reflect student clinical experiences in United Arab Emirates?

This article was migrated. The article was marked as recommended. Background: A number of medical schools around the world use the United States National Board of Medical Examiners Subject Examinations as a clerkship assessment of student performance, yet these exams were blueprinted against the United States national core clerkship curriculum which might not be the same as the local curricula to which they are applied in other parts of the world. In this study, we investigated the correlations between the internal medicine clinical experiences at United Arab Emirates University with student performance on the National Board of Medical Examiners subject of internal medicine (NBME). Methods: One hundred and seven junior clerkship students out of 145 (74%) who finished their Internal Medicine clerkship during academic years 2014-2015 and 2015-2016 participated in this study. The students' clinical experiences were measured by the clinical learning evaluation questionnaire (CLEQ) and by the logged number of meaningful patient contacts during their internal medicine clerkship. Results: Linear regression analysis showed no significant association between performance on the subject test and student clinical experiences measured by the CLEQ or the number of logged patients. NBME scores were weakly correlated with OSCEs scores (ɸ 0.20). Conclusions: The study findings raised uncertainties about the suitability of using NBME in the clerkship assessment program in the United Arab Emirates.

What we measure … and what we should measure in medical education

CONTEXT

As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society.

OBJECTIVES

We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role.

METHODS

We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies.

DISCUSSION

As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.

The International Landscape of Medical Licensing Examinations: A Typology Derived From a Systematic Review

BACKGROUND

National licensing examinations (NLEs) are large-scale examinations usually taken by medical doctors close to the point of graduation from medical school. Where NLEs are used, success is usually required to obtain a license for full practice. Approaches to national licensing, and the evidence that supports their use, varies significantly across the globe. This paper aims to develop a typology of NLEs, based on candidacy, to explore the implications of different examination types for workforce planning.

METHODS

A systematic review of the published literature and medical licensing body websites, an electronic survey of all medical licensing bodies in highly developed nations, and a survey of medical regulators.

RESULTS

The evidence gleaned through this systematic review highlights four approaches to NLEs: where graduating medical students wishing to practice in their national jurisdiction must pass a national licensing exam before they are granted a license to practice; where all prospective doctors, whether from the national jurisdiction or international medical graduates, are required to pass a national licensing exam in order to practice within that jurisdiction; where international medical graduates are required to pass a licensing exam if their qualifications are not acknowledged to be comparable with those students from the national jurisdiction; and where there are no NLEs in operation. This typology facilitates comparison across systems and highlights the implications of different licensing systems for workforce planning.

CONCLUSION

The issue of national licensing cannot be viewed in isolation from workforce planning; future research on the efficacy of national licensing systems to drive up standards should be integrated with research on the implications of such systems for the mobility of doctors to cross borders.

UK nationals who received their medical degrees abroad: selection into, and subsequent performance in postgraduate training: a national data linkage study

OBJECTIVES

To compare the likelihood of success at selection into specialty training for doctors who were UK nationals but obtained their primary medical qualification (PMQ) from outside the UK ('UK overseas graduates') with other graduate groups based on their nationality and where they gained their PMQ. We also compared subsequent educational performance during postgraduate training between the graduate groups.

DESIGN

Observational study linking UK medical specialty recruitment data with postgraduate educational performance (Annual Review of Competence Progression (ARCP) ratings).

SETTING

Doctors recruited into national programmes of postgraduate specialist training in the UK from 2012 to 2016.

PARTICIPANTS

34 755 UK-based trainee doctors recruited into national specialty training programmes with at least one subsequent ARCP outcome reported during the study period, including 1108 UK overseas graduates.

MAIN OUTCOME MEASURES

Odds of being deemed appointable at specialty selection and subsequent odds of obtaining a less versus more satisfactory category of ARCP outcome.

RESULTS

UK overseas graduates were more likely to be deemed appointable compared with non-EU medical graduates who were not UK citizens (OR 1.29, 95% CI 1.16 to 1.42), although less so than UK (OR 0.25, 95% CI 0.23 to 0.27) or European graduates (OR 0.66, 95% CI 0.58 to 0.75). However, UK overseas graduates were subsequently more likely to receive a less satisfactory outcome at ARCP than other graduate groups. Adjusting for age, sex, experience and the economic disparity between country of nationality and place of qualification reduced intergroup differences.

CONCLUSIONS

The failure of recruitment patterns to mirror the ARCP data raises issues regarding consistency in selection and the deaneries' subsequent annual reviews. Excessive weight is possibly given to interview performance at specialty recruitment. Regulators and selectors should continue to develop robust processes for selection and assessment of doctors in training. Further support could be considered for UK overseas graduates returning to practice in the UK.

Can We Increase the Value and Decrease the Cost of Clinical Skills Assessment?

Although the authors do not agree with medical students' bid to end the United States Medical Licensing Examination Step 2 Clinical Skills or Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation tests, they concur with Ecker and colleagues that conducting further research to support the validity argument, providing greater feedback on performance, and exploring options to reduce costs are important for addressing students' concerns. Evidence to support the validity of clinical skills exam scores and associated inferences already exists. What is lacking, and would help further justify the use of these examinations, is more evidence to support the "extrapolation" argument-that is, is performance on these examinations related to actual patient care? Enhanced feedback on exam performance should also be considered. While performance data from licensing examinations should be used judiciously given the primary purpose of these tests, additional data would be helpful to learners and their institutions. Centralized testing remains the least costly design, but efficiencies of standardized patient training, case development, and scoring can be reviewed. Scoring modifications made in the past several years at substantial cost should be evaluated to determine whether they have achieved desired goals. Testing organizations can and should embrace these essential elements of transparency and accountability to address concerns about the value of clinical skills examinations.

Progress test utopia

This paper discusses the advantages of progress testing. A utopia is described where medical schools would work together to develop and administer progress testing. This would lead to a significant reduction of cost, an increase in the quality of measurement and phenomenal feedback to learner and school. Progress testing would also provide more freedom and resources for more creative in-school assessment. It would be an educationally attractive alternative for the creation of cognitive licensing exams. A utopia is always far away in the future, but by formulating a vision for that future we may engage in discussions on how to get there.

Public health matters: Innovative approaches for engaging medical students

BACKGROUND

Public health faces the paradox of being increasingly emphasized by the key health and social care regulators and stakeholders, while remaining a largely under-represented discipline in the context of medical curricula. Enhancing medical student engagement in public health teaching is one way to address this concern.

METHODS

We discuss four key solutions to the challenges faced by public health educators in medical schools, and present five case studies which demonstrate innovative approaches to engaging medical students in our discipline.

RESULTS

Four different approaches have been piloted by members of the Public Health Educators in Medical Schools (PHEMS) network: (i) ensuring social accountability, (ii) demonstrating clinical relevance, (iii) mapping the core curriculum, and (iv) using technology enhanced learning. Preliminary student feedback suggests that these approaches can be used to position public health as an enabler of modern medical practice, and promote a more holistic understanding of medicine by linking patient-centred care to the population level.

CONCLUSIONS

The zeitgeist in both academia and the healthcare system supports the teaching of public health within the medical curriculum; there is also consensus at the political and pedagogical level. The challenge of ensuring engagement now needs to be met at the student-teacher interface.