Differential attainment, socioeconomic factors and surgical training

INTRODUCTION

Differential attainment (DA) is the gap in levels of achievement between different groups; socioeconomic factors are thought to play a significant role in DA. The aim of this study was to review and assess the evidence for DA in early surgical training and to examine the potential influence of socioeconomic status.

METHODS

Data were obtained from the General Medical Council GMC for those taking Membership of the Royal College of Surgeons (MRCS) examinations between 2016 and 2019 and core surgical training annual review of competency progression (ARCP) outcomes between 2017 and 2019. The index of multiple deprivation (IMD) was used as a measure of socioeconomic background. Trainees were then divided into deprivation quintiles (DQ1=most deprived, DQ5=least deprived). MRCS and ARCP outcomes were compared between DQ groups using 95% confidence intervals and chi-square tests.

RESULTS

Those from lower socioeconomic backgrounds had significantly lower overall MRCS pass rates (DQ1=45.5%, DQ2=48.9% vs DQ4=59.6%, DQ5=61.5%, p<0.05) and 1st time pass rates (DQ1&2=46.6% vs DQ4&5=63.5%, p<0.001). Additionally, they had a significantly higher number of attempts required to pass MRCS (DQ 1&2=1.86 vs DQ 4&5=1.54, p<0.01). Those from lower socioeconomic backgrounds had a significantly greater proportion of unsatisfactory ARCP outcomes (DQ1&2=24.4% vs DQ 4&5=14.2%, p<0.05).

CONCLUSIONS

There is clear evidence of the influence of socioeconomic background on DA in early surgical training. However, the reasons for this are likely complex and more work is required to investigate this relationship.

A GP fellowship project looking at whether the multi-source feedback and the educational supervisor's report can be used to identify trainees at risk of difficulty

As a Wessex General Practice (GP) fellow working within the annual review of competency progression (ARCP) team, we wanted to identify trainees at an early stage of GP training who might be at risk of difficulty during their training scheme. Early identification would allow more time for these trainees to access additional training and wider deanery support. This project aimed to identify retrospectively whether the multi-source feedback (MSF) and the educational supervisor's report (ESR) completed in the first year of speciality training (ST1) could be used to identify trainees at risk of needing additional time in training beyond the three-year training programme. For the purposes of this project, a trainee at risk of difficulty is one who received a developmental outcome 3 (OC 3) at their final ARCP in their last year of GP speciality training (ST3) where the desired outcome is an outcome 6 (OC 6). This fellowship project demonstrated it is possible to use the first MSF in GP training alongside the ESR before their ARCP in ST1, to identify a large proportion (88%) of trainees who may be at risk of difficulty further in their training.

Perceptions of the impact of annual review of competence progression (ARCP): a mixed methods case study

The annual review of competence progression (ARCP) is a high-stakes assessment which all UK postgraduate trainees undertake to ensure competence progression. Previous evaluations of the effectiveness of the ARCP as an assessment have reported deficiencies in both validity and reliability, however, there has been little focus on the educational impact of the ARCP.We conducted a mixed methods case study involving questionnaire, interviews and a focus group examining the impact of the ARCP on a respiratory higher specialist training programme. Participants included both trainers and trainees.Perceptions of impact were mixed. The ARCP was reported to promote broad curriculum coverage, enable educational planning, provide educational governance and facilitate relationships with supervisors. However, participants reported that activities promoted by the ARCP may detract from learning and that issues of reliability and validity undermined the process. In some cases, this was reported to lead to disillusionment and stress for trainees. Concerns were raised that the process promoted a reductionist approach to education.This research has resulted in several changes to local training, however, it has potential implications for the ARCP as a wider process. Trainers should be cognisant of the shortcomings of assessments and their impact on trainees, training and the future of the profession.

Virtual ARCP assessment and trainee feedback meetings: facilitating the best experience and practice

As a result of COVID-19, there has been an exponential increase in the use of remote technology for many local, regional and national meetings that would previously have been held on a face-to-face basis. Remote meetings have ensured that essential clinical, educational and strategic work can continue but it is not ‘business as usual’, although colleagues accept this form of communication as the new norm.

In medical education and assessment, the Annual Review of Competence Progression (ARCP) meetings and other formative educational meetings are being conducted remotely. This form of communication has some advantages but may also present possible barriers for feedback and development particularly for ‘trainees in difficulty’ when there are concerns about progression, and when an unsatisfactory outcome has been awarded. It is also worth remembering that there may be generational differences with the ease of use of virtual meeting platforms.

We present some of the important factors for optimising the panel of virtual ARCPs and discuss methods to improve feedback given remotely for trainees.

Assessing professional competence: a critical review of the Annual Review of Competence Progression

The Annual Review of Competence Progression is used to determine whether trainee doctors in the United Kingdom are safe and competent to progress to the next training stage. In this article we provide evidence to inform recommendations to enhance the validity of the summative and formative elements of the Annual Review of Competency Progression. The work was commissioned as part of a Health Education England review. We systematic searched the peer reviewed and grey literature, synthesising findings with information from national, local and specialty-specific Annual Review of Competence Progression guidance, critically evaluating the findings in the context of literature on assessing competence in medical education. National guidance lacked detail resulting in variability across locations and specialties, threatening validity and reliability. Trainees and trainers were concerned that the Annual Review of Competence Progression only reliably identifies the most poorly performing trainees. Feedback is not routinely provided, which can leave those with performance difficulties unsupported and high performers demotivated. Variability in the provision and quality of feedback can negatively affect learning. The Annual Review of Competence Progression functions as a high-stakes assessment, likely to have a significant impact on patient care. It should be subject to the same rigorous evaluation as other high-stakes assessments; there should be consistency in procedures across locations, specialties and grades; and all trainees should receive high-quality feedback.

Perceptions of the Annual Review of Competence Progression (ARCP) in surgical training in the UK and Ireland: A prospective cross sectional questionnaire study

BACKGROUND

Surgical trainees in the UK and Ireland undergo rigorous formative and summative assessments throughout each placement, and appraisal at an Annual Review of Competence Progression (ARCP). The ARCP evaluates performance during each training year and determines progression to the next year of training. It is critical that the ARCP is a robust and fair process. The Association of Surgeons in Training (ASiT) sought to evaluate surgical trainees' experiences of the ARCP process in order to identify areas for improvement.

METHODS

An electronic survey was developed and distributed electronically to the trainee membership of ASiT in the UK and Republic of Ireland. A 57 point survey examined the specifics of one ARCP cycle as well as attitudes to the process in general. Quantitative analysis was performed, along with thematic analysis on the free-text comments.

RESULTS

600 trainees from all deaneries, grades and specialities participated. The survey demonstrated difficulties in preparing for ARCP: insufficient notice (24%), inadequate communication (22%) and lack of engagement of seniors (30-39%). 47% considered the process and standards inconsistent. 82% of trainees considered a face-to-face ARCP advantageous. Such a meeting provided a means of raising concerns regarding training posts (29%), bullying (18%) and patient safety (17%) that would not otherwise have been reported in writing. During qualitative analysis, the following themes emerged: The conflict between potential value and real experience; concerns regarding the quality of assessment and the need for improvement (in process, individual performance and surgical training.) CONCLUSION: This survey demonstrates that trainees appreciate the potential educational value of the ARCP process. However, there is a gap between this potential and trainees experience. Particular concerns include inconsistency, the timing of decision-making and the need to retain a face-to-face meeting. This feedback from trainees can be used to improve the assessment process in relation to procedural developments at the national level and engagement of supervisors and trainers locally. These changes will ensure that the ARCP becomes a higher quality assessment and more constructive for training in future.

Impact of accelerated, graduate-entry medicine courses: a comparison of profile, success, and specialty destination between graduate entrants to accelerated or standard medicine courses in UK

BACKGROUND

Little research has compared the profile, success, or specialty destinations of graduates entering UK medical schools via accelerated, 4-yr, standard 5-yr and 6-yr programmes. Four research questions directed this investigation:- What are the success rates for graduates entering graduate-entry vs. undergraduate medicine courses? How does the sociodemographic and educational profile differ between these two groups? Is success - in medical school and foundation training - dependent on prior degree, demographic factors, or aptitude test performance at selection? What specialty do graduate entry medicine students subsequently enter?

METHODS

The data from two cohorts of graduates entering medical school in 2007 and 2008 (n = 2761) in the UKMED (UK Medical Education Database) database were studied: 1445 taking 4-yr and 1150 taking 5-yr medicine courses, with smaller numbers following other programmes.

RESULTS

Completion rates for degree programmes were high at 95%, with no significant difference between programme types. 4-yr entrants were older, less likely to be from Asian communities, had lower HESA (Higher Education Statistics Agency) tariff scores, but higher UKCAT (UK Clinical Aptitude Test) and GAMSAT (Graduate Medical School Admissions Test) scores, than 5-yr entrants. Higher GAMSAT scores, black or minority ethnicity (BME), and younger age were independent predictors of successful completion of medical school. Foundation Programme (FPAS) selection measures (EPM - educational performance measure; SJT - situational judgment test) were positively associated with female sex, but negatively with black or minority ethnicity. Higher aptitude test scores were associated with EPM and SJT, GAMSAT with EPM, UKCAT with SJT. Prior degree subject, class of degree, HESA tariff, and type of medicine programme were not related to success.

CONCLUSIONS

The type of medicine programme has little effect on graduate entrant completion, or EPM or SJT scores, despite differences in student profile. Aptitude test score has some predictive validity, as do sex, age and BME, but not prior degree subject or class. Further research is needed to disentangle the influences of BME.

Does the Intercollegiate Membership of the Royal College of Surgeons (MRCS) examination predict 'on-the-job' performance during UK higher specialty surgical training?

INTRODUCTION

The Intercollegiate Membership of the Royal College of Surgeons (MRCS) is a mandatory examination to enter higher surgical specialty training in the UK. It is designed to help to ensure that successful candidates are competent to practice as higher surgical trainees. The annual review of competence progression (ARCP) assesses trainees' competence to progress to the next level of training and can be interpreted as a measure of 'on-the-job' performance. We investigated the relationship between MRCS performance and ARCP outcomes.

MATERIALS AND METHODS

All UK medical graduates who passed MRCS (Parts A and B) from 2007 to 2016 were included. MRCS scores, attempts and sociodemographics for each candidate were crosslinked with ARCP outcomes (satisfactory, unsatisfactory and insufficient evidence). Multinomial logistic regression was used to identify potential independent predictors of ARCP outcomes.

RESULTS

A total of 2570 trainees underwent 11,064 ARCPs; 1589 (61.8%) had only satisfactory outcomes recorded throughout training; 510 (19.9%) had at least one unsatisfactory outcome; and 471 (18.3%) supplied insufficient evidence. After adjusting for age, gender, first language and Part A performance, ethnicity (non-white vs white, OR 1.36, 95% CI 1.08 to 1.71), Part B passing score (OR 0.98, 95% CI 0.98 to 1.00) and number of attempts at Part B (two or more attempts vs one attempt, OR 1.50, 95% CI 1.16 to 1.94) were found to be independent predictors of an unsatisfactory ARCP outcome.

CONCLUSIONS

This is the first study to identify predictors of ARCP outcomes during higher surgical specialty training in the UK and provides further evidence of the predictive validity of the MRCS examination.

The validity of the Annual Review of Competence Progression: a qualitative interview study of the perceptions of junior doctors and their trainers

Objective To investigate trainee doctors' and trainers' perceptions of the validity of the Annual Review of Competence Progression (ARCP) using Messick's conceptualisation of construct validity. Design Qualitative semi-structured focus groups and interviews with trainees and trainers. Setting Postgraduate medical training in London, Kent Surrey and Sussex, Yorkshire and Humber, and Wales in November/December 2015. Part of a larger study about the fairness of postgraduate medical training. Participants Ninety-six trainees and 41 trainers, comprising UK and international medical graduates from Foundation, General Practice, Medicine, Obstetrics and Gynaecology, Psychiatry, Radiology, and Surgery, at all levels of training. Main outcome measures Trainee and trainer perceptions of the validity of the ARCP as an assessment tool. Results Participants recognised the need for assessment, but were generally dissatisfied with ARCPs, especially UK graduate trainees. Participants criticised the perceived tick-box nature of ARCPs as measuring clerical rather than clinical ability, and which they found detrimental to learning. Trainees described being able to populate their e-portfolios with just positive feedback; they also experienced difficulty getting assessments signed off by supervisors. ARCPs were perceived as poor at identifying struggling trainees and/or as discouraging excellence by focussing on minimal competency. Positive experiences of ARCPs arose when trainees could discuss their progress with interested supervisors. Conclusions Trainee and trainer criticisms of ARCPs can be conceptualised as evidence that ARCPs lack validity as an assessment tool. Ongoing reforms to workplace-based assessments could address negative perceptions of the 'tick-box' elements, encourage constructive input from seniors and allow trainees to demonstrate excellence as well as minimal competency, while keeping patients safe.

Development of training-related health care software by a team of clinical educators: their experience, from conception to piloting

The difficulties of producing useful, bespoke, and affordable information technology systems for large health care organizations are well publicized, following several high-profile endeavors in the UK. This article describes the experience of a small group of clinical radiologists and their collaborators in producing an information technology system - from conception to piloting. This system, called Trainee Tracker, enables automated target date recalculation of trainee milestones, depending on their work patterns and other individual circumstances. It utilizes an automated email alert system to notify the educational supervisors and trainees of approaching and elapsed target dates, in order to identify trainees in difficulty early and address their training needs accordingly. The challenges and advantages, both common to and contrasting with larger-scale projects, are also considered. The benefits of the development team's "agile" approach to software development and the lessons learned will be of interest to medical educators, particularly those with expertise in e-portfolios and other training-related software.