Prediction of success at UK Specialty Board Examinations using the mandatory postgraduate UK surgical examination

Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.SMITHSEx are viewed as the "gold standard" assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees' competence while also improving trainee welfare.Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals' well-being and success.

Preparing for success in final summative medical specialist examinations: The case for RACE

BACKGROUND

Failure rates on medical specialist final summative examinations in Australia are high, regardless of speciality. Examination failure can have detrimental psycho-social, financial and job security effects on the trainee, while delays in completion of training adversely impacts workforce growth and health outcomes for the community. The study aimed to explore the preparation factors that contribute to ophthalmology trainee success in their final summative examination.

METHODS

Semi-structured in-depth interviews were conducted with 29 participants via telephone or Zoom with ophthalmology trainees and Fellows. To be eligible, interviewees had to have sat the Royal Australian and New Zealand College of Ophthalmologists Advanced Clinical Examination (RACE) within the past five years or were providing supervision to trainees preparing for RACE. Interviews were audio-recorded, transcribed and thematically analysed.

RESULTS

Examination success was underpinned by six themes relating to preparation: (i) 'Those who fail to plan, plan to fail', which related to development and adherence to a study plan; (ii) 'It takes a village' encompassed trainees establishing and activating personal and professional supports; (iii) 'Get to know your opponent', which encompassed developing an understanding of the examination construct, format and requirements; (iv) 'There is no substitute for hard work', which related to intensive study over a period of 12-18 months; (v) 'Keep pace with the herd', which referred to benchmarking preparation efforts and progress against peers; and (vi) 'Don't jump the gun', which related to ensuring readiness to sit.

CONCLUSIONS

Maximising medical specialist examination pass rates is in the best interest of trainees, training Colleges, health care systems and communities. Recognising and facilitating preparation approaches that foster success in final summative examinations are the collective responsibility of trainees, specialist training Colleges, training networks and health systems. Trainees need to plan for examination success, be self-determined to commit to intensive study over an extended time period and be realistic about their readiness to sit.

Establishing the predictive validity of the intercollegiate membership of the Royal Colleges of surgeons written examination: MRCS Part A

Successful completion of the Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examination is mandatory for surgical trainees entering higher specialist training in the United Kingdom. Despite its international reputation, and the value placed on the examination in surgical training, there has been little evidence of its predictive validity until recently. In this review, we present a summary of findings of four recent Intercollegiate studies assessing the predictive validity of the MRCS Part A (written) examination. Data from all four studies showed statistically significant positive correlations between the MRCS Part A and other written examinations taken by surgical trainees over the course of their education. The studies summarised in this review provide compelling evidence for the predictive validity of this gatekeeping examination. This review will be of interest to trainees, training institutions and the Royal Colleges given the value placed on the examination by surgical training programmes.

Establishing the predictive validity of the intercollegiate membership of the Royal Colleges of surgeons written examination: MRCS part B

The Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) is a high-stakes postgraduate examination taken by thousands of surgical trainees worldwide every year. The MRCS is a challenging assessment, highly regarded by surgical training programmes and valued as a gatekeeper to the surgical profession. The examination is taken at considerable personal, social and financial cost to surgical trainees, and failure has significant implications for career progression. Given the value placed on MRCS, it must be a reliable and valid assessment of the knowledge and skills of early-career surgeons. Our first article 'Establishing the Predictive Validity of the Intercollegiate Membership of the Royal Colleges of Surgeons Written Examination: MRCS Part A' discussed the principles of assessment reliability and validity and outlined the mounting evidence supporting the predictive validity of the MRCS Part A (the multiple-choice questionnaire component of the examination). This, the second article in the series discusses six recently published studies investigating the predictive validity of the MRCS Part B (the clinical component of the examination). All national longitudinal cohort studies reviewed have demonstrated significant correlations between MRCS Part B and other assessments taken during the UK surgical training pathway, supporting the predictive validity of MRCS Part B. This review will be of interest to trainees, trainers and Royal Colleges given the value placed on the examination by surgical training programmes.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

The role of Gender in Operative Autonomy in orthopaedic Surgical Trainees (GOAST)

Aims

Global literature suggests that female surgical trainees have lower rates of independent operating (operative autonomy) than their male counterparts. The objective of this study was to identify any association between gender and lead/independent operating in speciality orthopaedic trainees within the UK national training programme.

Methods

This was a retrospective case-control study using electronic surgical logbook data from 2009 to 2021 for 274 UK orthopaedic trainees. Total operative numbers and level of supervision were compared between male and female trainees, with correction for less than full-time training (LTFT), prior experience, and time out during training (OOP). The primary outcome was the percentage of cases undertaken as lead surgeon (supervised and unsupervised) by UK orthopaedic trainees by gender.

Results

All participants gave permission for their data to be used. In total, 274 UK orthopaedic trainees submitted data (65% men (n = 177) and 33% women (n = 91)), with a total of 285,915 surgical procedures logged over 1,364 trainee-years. Males were lead surgeon (under supervision) on 3% more cases than females (61% (115,948/189,378) to 58% (50,285/86,375), respectively; p < 0.001), and independent operator (unsupervised) on 1% more cases. A similar trend of higher operative numbers in male trainees was seen for senior (ST6 to 8) trainees (+5% and +1%; p < 0.001), those with no time OOP (+6% and +8%; p < 0.001), and those with orthopaedic experience prior to orthopaedic specialty training (+7% and +3% for lead surgeon and independent operator, respectively; p < 0.001). The gender difference was less marked for those on LTFT training, those who took time OOP, and those with no prior orthopaedic experience.

Conclusion

This study showed that males perform 3% more cases as the lead surgeon than females during UK orthopaedic training (p < 0.001). This may be due to differences in how cases are recorded, but must engender further research to ensure that all surgeons are treated equitably during their training.

A cross-sectional study examining MRCS performance by core surgical training location

BACKGROUND

In the UK, core surgical training (CST) is the first specialty experience that early-career surgeons receive but training differs significantly across CST deaneries. To identify the impact these differences have on trainee performance, we assessed whether success at the Membership of the Royal College of Surgeons (MRCS) examinations is associated with CST deanery.

METHODS

A retrospective cohort study of UK trainees in CST who attempted MRCS between 2014 and 2020 (n = 1104). Chi-squared tests examined associations between locality and first-attempt MRCS performance. Multivariate logistic regression models identified the likelihood of MRCS success depending on CST deanery.

RESULTS

MRCS Part A and Part B pass rates were associated with CST deanery (p < 0.001 and p = 0.013, respectively). Candidates that trained in Thames Valley (Odds Ratio [OR] 2.52 (95% Confidence Interval [CI] 1.00-6.42), North Central and East London (OR 2.37 [95% CI 1.04-5.40]) or South London (OR 2.36 [95% CI 1.09-5.10]) were each more than twice as likely to pass MRCS Part A at first attempt. Trainees from North Central and East London were more than ten times more likely to pass MRCS Part B at first attempt (OR 10.59 [95% CI 1.23-51.00]). However, 68% of candidates attempted Part A prior to CST and 48% attempted Part B before or during the first year of CST.

CONCLUSION

MRCS performance is associated with CST deanery; however, many candidates passed the exam with little or any CST experience suggesting that some deaneries attract high academic performers. MRCS performance is therefore not a suitable marker of CST training quality.

Differential attainment at MRCS according to gender, ethnicity, age and socioeconomic factors: a retrospective cohort study

Objective

A recent independent review on diversity and inclusivity highlighted concerns that barriers to surgical career progression exist for some groups of individuals and not others. Group-level differences in performance at the Intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examinations have been identified but are yet to be investigated. We aimed to characterise the relationship between sociodemographic differences and performance at MRCS.

Design

Retrospective cohort study.

Setting

Secondary care.

Participants

All UK MRCS candidates attempting Part A (n = 5780) and Part B (n = 2600) between 2013 and 2019 with linked sociodemographic data in the UK Medical Education Database (https://www.ukmed.ac.uk).

Main outcome measures

Chi-square tests established univariate associations with MRCS performance. Multiple logistic regression identified independent predictors of success, adjusted for medical school performance.

Results

Statistically significant differences in MRCS pass rates were found according to gender, ethnicity, age, graduate status, educational background and socioeconomic status (all p < 0.05). After adjusting for prior academic attainment, being male (odds ratio [OR] 2.34, 95% confidence interval [CI] 1.87–2.92) or a non-graduate (OR 1.98, 95% CI 1.44–2.74) were independent predictors of MRCS Part A success and being a non-graduate (OR 1.77, 95% CI 1.15–2.71) and having attended a fee-paying school (OR 1.51, 95% CI 1.08–2.10) were independent predictors of Part B success. Black and minority ethnic groups were significantly less likely to pass MRCS Part B at their first attempt (OR 0.41, 95% CI 0.18–0.92 for Black candidates and OR 0.49, 95% CI 0.35–0.69 for Asian candidates) compared to White candidates.

Conclusions

There is significant group-level differential attainment at MRCS, likely to represent the accumulation of privilege and disadvantage experienced by individuals throughout their education and training. Those leading surgical education now have a responsibility to identify and address the causes of these attainment differences.

Does performance at the intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examination vary according to UK medical school and course type? A retrospective cohort study

OBJECTIVES

The knowledge, skills and behaviours required of new UK medical graduates are the same but how these are achieved differs given medical schools vary in their mission, curricula and pedagogy. Medical school differences seem to influence performance on postgraduate assessments. To date, the relationship between medical schools, course types and performance at the Membership of the Royal Colleges of Surgeons examination (MRCS) has not been investigated. Understanding this relationship is vital to achieving alignment across undergraduate and postgraduate training, learning and assessment values.

DESIGN AND PARTICIPANTS

A retrospective longitudinal cohort study of UK medical graduates who attempted MRCS Part A (n=9730) and MRCS Part B (n=4645) between 2007 and 2017, using individual-level linked sociodemographic and prior academic attainment data from the UK Medical Education Database.

METHODS

We studied MRCS performance across all UK medical schools and examined relationships between potential predictors and MRCS performance using χ2 analysis. Multivariate logistic regression models identified independent predictors of MRCS success at first attempt.

RESULTS

MRCS pass rates differed significantly between individual medical schools (p<0.001) but not after adjusting for prior A-Level performance. Candidates from courses other than those described as problem-based learning (PBL) were 53% more likely to pass MRCS Part A (OR 1.53 (95% CI 1.25 to 1.87) and 54% more likely to pass Part B (OR 1.54 (1.05 to 2.25)) at first attempt after adjusting for prior academic performance. Attending a Standard-Entry 5-year medicine programme, having no prior degree and attending a Russell Group university were independent predictors of MRCS success in regression models (p<0.05).

CONCLUSIONS

There are significant differences in MRCS performance between medical schools. However, this variation is largely due to individual factors such as academic ability, rather than medical school factors. This study also highlights group level attainment differences that warrant further investigation to ensure equity within medical training.

Does performance at medical school predict success at the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination? A retrospective cohort study

BACKGROUND

Identifying predictors of success in postgraduate examinations can help guide the career choices of medical students and may aid early identification of trainees requiring extra support to progress in specialty training. We assessed whether performance on the educational performance measurement (EPM) and situational judgement test (SJT) used for selection into foundation training predicted success at the Membership of the Royal College of Surgeons (MRCS) examination.

METHODS

This was a longitudinal, cohort study using data from the UK Medical Education Database (https://www.ukmed.ac.uk). UK medical graduates who had attempted Part A (n=2585) and Part B (n=755) of the MRCS between 2014 and 2017 were included. χ² and independent t-tests were used to examine the relationship between medical school performance and sociodemographic factors with first-attempt success at MRCS Part A and B. Multivariate logistic regression was employed to identify independent predictors of MRCS performance.

RESULTS

The odds of passing MRCS increased by 55% for Part A (OR 1.55 (95% CI 1.48 to 1.61)) and 23% for Part B (1.23 (1.14 to 1.32)) for every additional EPM decile point gained. For every point awarded for additional degrees in the EPM, candidates were 20% more likely to pass MRCS Part A (1.20 (1.13 to 1.29)) and 17% more likely to pass Part B (1.17 (1.04 to 1.33)). For every point awarded for publications in the EPM, candidates were 14% more likely to pass MRCS Part A (1.14 (1.01 to 1.28)). SJT score was not a statistically significant independent predictor of MRCS success.

CONCLUSION

This study has demonstrated the EPM's independent predictive power and found that medical school performance deciles are the most significant measure of predicting later success in the MRCS. These findings can be used by medical schools, training boards and workforce planners to inform evidence-based and contemporary selection and assessment strategies.

The impact of disability on performance in a high-stakes postgraduate surgical examination: a retrospective cohort study

OBJECTIVE

Despite rising numbers of doctors in the workforce with disabilities, little is known about the impact of disabilities on postgraduate performance. To ensure all groups are treated fairly in surgical training, it is essential to know whether any attainment differences exist in markers of surgical performance. To address this gap, we assessed the impact of disabilities on performance on the Intercollegiate Membership of the Royal College of Surgeons examination (MRCS).

DESIGN

Retrospective cohort study.

SETTING

Secondary care.

PARTICIPANTS

All UK MRCS candidates attempting Part A (n = 9600) and Part B (n = 4560) between 2007 and 2017 with linked disability data in the UK Medical Education Database (https://www.ukmed.ac.uk) were included.

MAIN OUTCOME MEASURES

Chi-square tests and correlation coefficients established univariate associations with MRCS performance, while multiple logistic regressions identified independent predictors of success.

RESULTS

Though MRCS Part B pass rates were similar (p = 0.339), candidates with registered disabilities had significantly lower first-attempt Part A pass rates (46.3% vs. 59.8%, p < 0.001). Candidates with disabilities also performed less well in examinations taken throughout school and medical school, and after adjusting for prior academic performance and sociodemographic predictors of success, logistic regression found that candidates with disabilities were no less likely to pass MRCS than their peers (odds ratio 1.04, 95% confidence interval 0.66 to 1.62). No significant variation was found in MRCS performance between type of disability or degree of limitations caused by disability (p > 0.05).

CONCLUSION

Although candidates with registered disabilities performed less well in formal, written examinations, our data indicate that they are as likely to pass MRCS at first attempt as their peers who achieved similar grades at high school and medical school. In order to enable equity in career progression, further work is needed to investigate the causes of attainment differences in early career assessments.

A cross-sectional study examining the association between MRCS performance and surgeons receiving sanctions against their medical registration

BACKGROUND

Fitness to practice (FtP) investigations by the General Medical Council (GMC) safeguard patients and maintain the integrity of the medical profession. The likelihood of FtP sanctions is influenced by specialty and socio-demographic factors and can be predicted by performance at postgraduate examinations. This is the first study to characterise the prevalence of FtP sanctions in early-career surgeons and to examine the association with performance at the Membership of the Royal College of Surgeons (MRCS) examination.

METHODS

All UK graduates who attempted MRCS between September 2007-January 2020 were matched to the GMC list of registered medical practitioners. Clinicians who had active FtP sanctions between 28th August 2018 and 28th August 2020 were identified. Data were anonymised by RCS England prior to analysis.

RESULTS

Of 11,660 candidates who attempted MRCS within the study period, only 31 (0.3%) had FtP sanctions between 2018 and 2020. Of these, 12 had active conditions on registration, seven had undertakings and 14 had warnings. There was no statistically significant difference in MRCS performance in either Parts A or B of the examination for those with and those free from FtP sanctions (P > 0.05).

CONCLUSIONS

In this, the largest study of MRCS candidates to date, the prevalence of active FtP sanctions in early-career surgeons was 0.3%, significantly lower than the prevalence of sanctions across more experienced UK surgeons (0.9%). These data highlight early-career surgeons as a low-risk group for disciplinary action and should reassure patients and medical professionals of the rarity of FtP sanctions.

Performance at medical school selection correlates with success in Part A of the intercollegiate Membership of the Royal College of Surgeons (MRCS) examination

Medical schools in the UK typically use prior academic attainment and an admissions test (University Clinical Aptitude Test (UCAT), Biomedical Admissions Test (BMAT) or the Graduate Medical School Admissions Test (GAMSAT)) to help select applicants for interview. To justify their use, more information is needed about the predictive validity of these tests. Thus, we investigated the relationship between performance in admissions tests and the Membership of the Royal College of Surgeons (MRCS) examination.The UKMED database (https://www.ukmed.ac.uk) was used to access medical school selection data for all UK graduates who attempted MRCS Part A (n=11 570) and Part B (n=5690) between 2007 and 2019. Univariate and multivariate logistic regression models identified independent predictors of MRCS success. Pearson correlation coefficients examined the linear relationship between test scores and MRCS performance.Successful MRCS Part A candidates scored higher in A-Levels, UCAT, BMAT and GAMSAT (p<0.05). No significant differences were observed for MRCS Part B. All admissions tests were found to independently predict MRCS Part A performance after adjusting for prior academic attainment (A-Level performance) (p<0.05). Admission test scores demonstrated statistically significant correlations with MRCS Part A performance (p<0.001).The utility of admissions tests is clear with respect to helping medical schools select from large numbers of applicants for a limited number of places. Additionally, these tests appear to offer incremental value above A-Level performance alone. We expect this data to guide medical schools' use of admissions test scores in their selection process.

Predictors of performance on the pediatric board certification examination

BACKGROUND

Examining the predictors of summative assessment performance is important for improving educational programs and structuring appropriate learning environments for trainees. However, predictors of certification examination performance in pediatric postgraduate education have not been comprehensively investigated in Japan.

METHODS

The Pediatric Board Examination database in Japan, which includes 1578 postgraduate trainees from 2015 to 2016, was analyzed. The examinations included multiple-choice questions (MCQs), case summary reports, and an interview, and the predictors for each of these components were investigated by multiple regression analysis.

RESULTS

The number of examination attempts and the training duration were significant negative predictors of the scores for the MCQ, case summary, and interview. Employment at a community hospital or private university hospital were negative predictors of the MCQ and case summary score, respectively. Female sex and the number of academic presentations positively predicted the case summary and interview scores. The number of research publications was a positive predictor of the MCQ score, and employment at a community hospital was a positive predictor of the case summary score.

CONCLUSION

This study found that delayed and repeated examination taking were negative predictors, while the scholarly activity of trainees was a positive predictor, of pediatric board certification examination performance.

Should we recruit trainees into the Oral and Maxillofacial Surgery pathway at the beginning of the second degree to ensure the long term viability of the specialty?

There has been a sustained decline in the number of trainees applying for entry into the Oral and Maxillofacial Surgery (OMFS) training program, which has sparked further debate on the issue of OMFS training and the future of the specialty in the United Kingdom. Here I discuss the option of recruiting the trainees into the program at the beginning of their second degree to streamline the training, and to improve the recruitment drive to ensure long term viability of the specialty.