Do Professionalism Lapses in Medical School Predict Problems in Residency and Clinical Practice?

Factors leading to lapses in professional behaviour of Gynae residents in Pakistan: a study reflecting through the lenses of patients and family, consultants and residents

INTRODUCTION

Professional behaviour is the first manifestation of professionalism. In teaching hospitals, the residents can be considered vulnerable to lapses in professional behaviour when they fail to meet the set standards of professionalism. Residents of some specialties are more at risk of lapses in professional behaviour due to the demanding nature of work. Research focusing on the behaviour of residents in the field of Gynae and the underlying factors contributing to such behaviour is notably lacking in the literature. Additionally, there is a gap in understanding the perspectives of patients from Pakistan on this matter, as it remains unexplored thus far, which constitutes the central focus of this study. An increase in complaints lodged against Gynae resident's professional behaviour in Pakistan Citizen Portal (PCP) was observed. Therefore, an exploratory qualitative study was conducted to investigate the factors and rationales contributing to the lapses in resident's professional behaviour. The study collected the viewpoints of three stakeholder groups: patients and their families, consultants and residents. The study was conducted in three phases. First, the document analysis of written complaints was conducted, followed by face-to-face interviews (11 per group) conducted by trained researchers from an independent 3rd party. Finally, the interview data was transcribed, coded and analysed. In total 15 themes were identified from the interviews with 3 stakeholders, which were then categorized and resulted in 6 overlapping themes. The most prevalent lapse reported by all 3 stakeholders was poor verbal behaviour of residents.

CONCLUSION

The highly ranked factors contributing to triggering the situation were associated with workplace challenges, well-being of residents, limited resources, patients and family characteristics, patients' expectations, lack of administrative and paramedic support, cultural factors and challenges specific to Gynae specialty. Another intriguing and emerging theme was related to the characteristics of patients and attendants which helped in understanding the causes and implications of conflicting environments. The value of competency also emphasized that can be accomplished by training and mentoring systems. The thorough examination of these factors by key stakeholders aided in accurately analysing the issue, its causes, and possible solutions. The study's findings will assist higher authorities in implementing corrective actions and offering evidence-based guidance to policymakers to improve healthcare system.

Unprofessional behaviour of GP residents leading to a dismissal dispute: characteristics and outcomes of those who appeal

BACKGROUND

Recognition of poor performance in General Practice trainees is important because underperformance compromises patients' health and safety. However, in General Practice, research on persistent underperformance while in training and its ultimate consequences is almost completely lacking. We aim to explore the unprofessional behaviours of residents in General Practice who were dismissed from training and who litigated against dismissal.

METHODS

We performed a structured analysis using open-source data from all General Practice cases before the Conciliation Board of the Royal Dutch Medical Association between 2011 and 2020. Anonymised law cases about residents from all Dutch GP training programmes were analysed in terms of the quantitative and qualitative aspects related to performance.

RESULTS

Between 2011 and 2020, 24 residents who were dismissed from training challenged their programme director's decision. Dismissed residents performed poorly in several competencies, including communication, medical expertise and most prominently, professionalism. Over 90% of dismissed residents failed on professionalism. Most lacked self-awareness and/or failed to profit from feedback. Approximately 80% failed on communication, and about 60% on medical expertise as well. A large majority (more than 80%) of dismissed residents had previously participated in some form of remediation.

CONCLUSIONS

Deficiencies in both professionalism and communication were the most prevalent findings among the dismissed General Practice residents. These two deficiencies overlapped considerably. Dismissed residents who challenged their programme director's decision were considered to lack self-awareness, which requires introspection and the appreciation of feedback from others.

Moving Beyond the Dichotomous Assessment of Professionalism in the Internal Medicine Clerkship: Results of a National Survey of Clerkship Directors

PURPOSE

This study examines how internal medicine clerkship faculty and leadership conceptualize professionalism and professional behaviors and attitudes, identifies whether and how faculty use metrics to assess professionalism and factor it into clerkship grades, and describes barriers that prevent faculty from feeling prepared to support the development of professional behaviors in students.

METHOD

The Clerkship Directors in Internal Medicine opened a call for thematic survey section proposals to its physician-faculty members, blind-reviewed all submissions, and selected 4 based on internal medicine clinical clerkship training experience relevance. The survey launched on October 5 and closed on December 7, 2021. Data were analyzed using descriptive statistics.

RESULTS

Of 137 core clerkship directors (CDs) at Liaison Committee on Medical Education-accredited medical schools, 103 (75.2%) responded to the survey. Of 102 respondents (1 nonrespondent), 84 (82.4%) identified professional behavior lapses in involvement and 60 (58.8%) identified introspection lapses. Of 103 respondents, 97 (94.2%) reported that their clerkships ask clinical faculty and residents to formally evaluate professionalism, and 64 (62.1%) reported that they factor professionalism assessments into final clerkship grades. CDs reported multiple barriers to addressing professionalism directly with students, including logistical barriers, professionalism assessment subjectivity concerns, and the possible adverse effect of an unprofessional label for students.

CONCLUSIONS

Professionalism assessment and remediation in medical education currently center on a deficit model that seeks to identify and remediate professionalism lapses, rather than a developmental model that seeks to nurture growth. This dichotomous characterization of behaviors as professional or unprofessional limits assessment and can adversely affect the learning environment. The authors propose a shift to a developmental model that considers professionalism as a continuous process parallel to the acquisition of clinical skills and medical knowledge.

Using Reflective Practice to Enhance Student Professionalism

Reflective practice, critically reflecting on one's actions or attitudes to engage in the process of continuous adaptation and learning, has proven to be an effective strategy for improved patient care. Additionally, literature supports applying reflective practice for professional growth in medical providers. When contemplating professionalism training in physician assistant (PA) education, it is important to consider how to obtain buy-in from students. One way to do this is to link professionalism to the students' future career as a PA. The School of Physician Assistant Studies at Pacific University (the Program) created an Online Orientation Reflective Practice Assignment that was implemented during online orientation. Students were prompted to use reflective practice to create detailed "plans for success." The goal of this assignment was to reduce common professionalism missteps students often experienced. Data regarding the number of professionalism encounters per cohort, broken down into occurrences by curricular phase (didactic vs clinical), was tracked and collated using Microsoft® Excel®. The data was analyzed to determine trends. Implementation of the Online Reflective Practice Assignment for all PA students at the beginning of their education has increased student awareness of the Program's, and the profession's, expectations regarding professionalism and accountability. Thus far, the resulting number of professionalism missteps have not decreased year-over-year. We hypothesize that this is due to the difficulty meeting increased administrative expectations and burdens put on students during the pandemic. Students' use of reflective practice to review what went wrong and to create plans to avoid missteps in the future allows them to focus on productive next steps in building their professional identity. Implementation of the Online Orientation Reflective Practice Assignment has better prepared students for the expectations of the PA profession and provided them with a solid foundation to build their professional identity throughout their education and into their careers.

"Should I Say Something?": A Simulation Curriculum on Addressing Lapses in Professionalism to Improve Patient Safety

Introduction

Medical students may witness lapses in professionalism but lack tools to effectively address such episodes. Current professionalism curricula lack opportunities to practice communication skills in addressing professionalism lapses.

Methods

We designed a simulation curriculum to introduce professionalism expectations, provide communication tools using elements of the Agency for Healthcare Research and Quality TeamSTEPPS program, and address observed professionalism lapses involving patient safety in hierarchical patient care teams. Students were surveyed on knowledge, skills, and attitude regarding professionalism before, immediately after, and 6 months after participation.

Results

Of 253 students, 70 (28%) completed baseline and immediate postsurveys, and 39 (15%) completed all surveys. In immediate postsurveys, knowledge of communication tools (82% to 94%, p = .003) and empowerment to address residents (19% to 44%, p = .001) and attendings (15% to 39%, p < .001) increased. At 6 months, 96% of students reported witnessing a professionalism lapse.

Discussion

The curriculum was successful in reported gains in knowledge of communication tools and empowerment to address professionalism lapses, but few students reported using the techniques to address witnessed lapses in real life.

Professionalism in Physician Assistant Education as a Predictor of Future Licensing Board Disciplinary Actions

PURPOSE

The purpose of this study was to evaluate associations between postgraduate disciplinary actions (PGDA) by state licensing boards and physician assistant (PA) school documented professionalism violations (DPV) and academic probation.

METHODS

This was a retrospective cohort study comprising PA graduates from 2001 to 2011 at 3 institutions (n = 1364) who were evaluated for the main outcome of PGDA and independent variable of DPV and academic probation. Random-effects multiple logistic regression and accelerated failure time parametric survival analysis were used to investigate the association of PGDA with DPV and academic probation.

RESULTS

Postgraduate disciplinary action was statistically significant and positively associated with DPV when unadjusted (odds ratio [OR] = 5.15; 95% CI: 1.62-16.31; P = .01) and when adjusting for age, sex, overall PA program GPA (GPA), and Physician Assistant National Certifying Exam Score (OR = 5.39; 95% CI: 1.54-18.85; P = .01) (fully adjusted). Academic probation increased odds to 8.43 times (95% CI: 2.85-24.92; P < .001) and 9.52 times (95% CI: 2.38-38.01; P < .001) when fully adjusted.

CONCLUSION

Students with professionalism violation or academic probation while in the PA school had significant higher odds of receiving licensing board disciplinary action compared with those who did not. Academic probation had a greater magnitude of effect and could represent an intersection of professionalism and academic performance.

Inpatient Satisfaction With Surgical Resident Care After Elective General and Oncologic Surgery

OBJECTIVE

To investigate inpatient satisfaction with surgical resident care.

BACKGROUND

Surgical trainees are often the primary providers of care to surgical inpatients, yet patient satisfaction with surgical resident care is not well characterized or routinely assessed.

METHODS

English-speaking, general surgery inpatients recovering from elective gastrointestinal and oncologic surgery were invited to complete a survey addressing their satisfaction with surgical resident care. Patients positively identified photos of surgical senior residents and interns before completing a modified version of the Consumer Assessment of Healthcare Providers and Systems Surgical Care Survey (S-CAHPS). Adapted S-CAHPS items were scored using the "top-box" method.

RESULTS

Ninety percent of recruited patients agreed to participate (324/359, mean age=62.2, 50.3% male). Patients were able to correctly identify their seniors and interns 85% and 83% of the time, respectively ( P =0.14). On a 10-point scale, seniors had a mean rating of 9.23±1.27 and interns had a mean rating of 9.01±1.49 ( P =0.14). Ninety-nine percent of patients agreed it was important to help in the education of future surgeons.

CONCLUSIONS

Surgical inpatients were able to recognize their resident physicians with high frequency and rated resident care highly overall, suggesting that they may serve as a willing source of feedback regarding residents' development of core competencies such as interpersonal skills, communication, professionalism, and patient care. Future work should investigate how to best incorporate patient evaluation of surgical resident care routinely into trainee assessment to support resident development.

Challenges with international medical graduate selection: finding positive attributes predictive of success in family medicine residency

Background

Criteria to select residents most likely to succeed, other than proficiency of their medical knowledge, is a challenge facing preceptors. International Medical Graduates (IMGs) play an integral role in mitigating the high demand for family medicine physicians across Canada. Thus, selecting IMG candidates that have a high probability of succeeding in Canadian educational settings is important. The purpose of this study is to elucidate qualitative attributes that positively correspond to success in residency, to ultimately assist in the selection of IMG residents most likely to achieve family medicine residency.

Methods

Interviews of 13 family medicine preceptors from some of the largest IMG training sites in Canada were performed to collect original data. The data was coded in tandem sequences using standardized coding techniques to increase robustness of results.

Results

The identified positive predictors of an IMG residents’ success are: presence of a positive attitude, proficient communication skills, high level of clinical knowledge, trainability.

Conclusions

The results provide adequate guidelines to assist in selection of IMG residents. Canada is a unique sociocultural setting where standardized selection methods of IMGs have not been employed. By selecting IMG residents who possess these attributes upon inception of residency, benefits of instruction will be maximized and result in residents developing increased aptitudes for patient care.

Surgeons and Professionalism: Where We Are and Where We Need to Go

Surgeons are famously decisive, assertive, and confident. Unfortunately, we also often have a reputation for being blunt, harsh, and unprofessional. Unprofessional behaviors are unacceptable, and we believe that they are a symptom of surgeon distress and burnout. Unprofessional behaviors should be prevented, but equally importantly, so should the stressors that drive them. This is critically important for the future of our profession.

Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training

Background

Remediation of struggling learners in pulmonary and critical care fellowship programs is a challenge, even for experienced medical educators.

Objective

This evidence-based narrative review provides a framework program leaders may use to address fellows having difficulty achieving competency during fellowship training.

Methods

The relevant evidence for approaches on the basis of each learner’s needs is reviewed and interpreted in the context of fellowship training in pulmonary medicine and critical care. Issues addressed include bias in fellow assessments and remediation, the impacts of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the specific challenges of pulmonary and critical care fellowship programs, a brief review of relevant legal issues, guidance on building and leveraging program resources, and a discussion of learner outcomes.

Results

This results in a concise, evidence-based toolkit for program leaders based around four pillars: early identification, fellow assessment, collaborative intervention, and reassessment. Important concepts also include the need for documentation, clear and written communication, and fellow-directed approaches to the creation of achievable goals.

Conclusion

Evidence-based remediation helps struggling learners in pulmonary and critical care fellowship to improve their ability to meet Accreditation Council for Graduate Medical Education (ACGME) milestones.

Professional identity formation of medical students: A mixed-methods study in a hierarchical and collectivist culture

BACKGROUND

Professional identity formation (PIF) has been recognized as an integral part of professional development in medical education. PIF is dynamic: it occurs longitudinally and requires immersion in the socialization process. Consequently, in the medical education context, it is vital to foster a nurturing learning environment that facilitates PIF.

AIM

This study assesses PIF among medical students in various stages of study and explores their perceptions of PIF, with its contributing and inhibiting factors.

METHOD

This mixed-methods study uses a sequential explanatory approach with undergraduate (years 2, 4, and 6) and postgraduate medical students in Indonesia. We examine the subjects by administering an adapted questionnaire on PIF. We completed a series of FGDs following questionnaire administration. Quantitative and thematic analyses were conducted sequentially.

RESULTS & DISCUSSION

A total of 433 respondents completed the questionnaire. There were statistically significant differences among subjects on the subscales "Recognition and internalization of professional roles" and "Self-control in professional behavior"; the more senior students had higher scores. We conducted 6 FGDs in total. The results characterize PIF as a complex, dynamic, and longitudinal journey to becoming a medical doctor that is closely related to a student's motivation. The FGDs also highlight the importance of both internal factors (students' values, attributes, and personal circumstances) and external factors (curriculum, the learning environment, workplace-based learning, and external expectations) for PIF in medical education.

CONCLUSION

Higher-level students show higher scores in some aspects of PIF, which further validates the potential use of the questionnaire to monitor PIF, a dynamic process influenced by internal and external factors. Generating awareness among medical students and encouraging reflection on their PIF stage may be crucial for PIF processes.

Associations between admissions factors and the need for remediation

This study examines the way in which student characteristics and pre-admissions measures are statistically associated with the likelihood a student will require remediation for academic and professionalism offenses. We anchor our inquiry within Irby and Hamstra's (2016) conceptual framework of constructs of professionalism. Data from five graduating cohorts (2014-2018) from McMaster University (Hamilton, Canada) (N = 1,021) were retroactively collected and analyzed using traditional and multinominal logistic regression analyses. The relationship among student characteristics, pre-admissions variables, and referral for potential remediation both by occurrence (yes/no) as well as type (academic/professional/no referral) were examined separately. Findings indicate that gender (OR = 0.519, 95% CI 0.326-0.827, p < 0.01) and undergraduate grade point average (GPA) (OR = 0.245, 95% CI 0.070-0.855, p < 0.05) were significantly associated with instances of referral for potential professionalism and academic remediation, respectively. Women were less likely than men to require remediation for professionalism (OR = 0.332, 95% CI 0.174-0.602, p < 0.001). Undergraduate GPAs (OR = 0.826, 95% CI 0.021-0.539, p < 0.01) were significantly associated with remediation for academic reasons. Lower undergraduate GPAs were associated with a higher likelihood of remediation. These findings point to the admissions variables that are associated with instances that prompt referral for potential remediation. Where associations are not significant, we consider the application of different conceptualizations of professionalism across periods of admissions and training. We encourage those involved in applicant selection and student remediation to emphasize the importance of the interactions that occur between personal and contextual factors to influence learner behaviour and professional identity formation.

Unprofessional behaviour of GP residents and its remediation: a qualitative study among supervisors and faculty

BACKGROUND

Lapses in professionalism have profound negative effects on patients, health professionals, and society. The connection between unprofessional behaviour during training and later practice requires timely identification and remediation. However, appropriate language to describe unprofessional behaviour and its remediation during residency is lacking. Therefore, this exploratory study aims to investigate which behaviours of GP residents are considered unprofessional according to supervisors and faculty, and how remediation is applied.

METHODS

We conducted eight semi-structured focus group interviews with 55 broadly selected supervisors from four Dutch GP training institutes. In addition, we conducted individual semi-structured interviews with eight designated professionalism faculty members. Interview recordings were transcribed verbatim. Data were coded in two consecutive steps: preliminary inductive coding was followed by secondary deductive coding using the descriptors from the recently developed 'Four I's' model for describing unprofessional behaviours as sensitising concepts.

RESULTS

Despite the differences in participants' professional positions, we identified a shared conceptualisation in pinpointing and assessing unprofessional behaviour. Both groups described multiple unprofessional behaviours, which could be successfully mapped to the descriptors and categories of the Four I's model. Behaviours in the categories 'Involvement' and 'Interaction' were assessed as mild and received informal, pedagogical feedback. Behaviours in the categories 'Introspection' and 'Integrity', were seen as very alarming and received strict remediation. We identified two new groups of behaviours; 'Nervous exhaustion complaints' and 'Nine-to-five mentality', needing to be added to the Four I's model. The diagnostic phase of unprofessional behaviour usually started with the supervisor getting a 'sense of alarm', which was described as either a 'gut feeling', 'a loss of enthusiasm for teaching' or 'fuss surrounding the resident'. This sense of alarm triggered the remediation phase. However, the diagnostic and remediation phases did not appear consecutive or distinct, but rather intertwined.

CONCLUSIONS

The processes of identification and remediation of unprofessional behaviour in residents appeared to be intertwined. Identification of behaviours related to lack of introspection or integrity were perceived as the most important to remediate. The results of this research provide supervisors and faculty with an appropriate language to describe unprofessional behaviours among residents, which can facilitate timely identification and remediation.

Tutor Uncertainty in Dealing with Unprofessional Behaviours of Medical Students and Residents: a Mixed Methods Study

BACKGROUND

Despite acknowledgement of medical students' expected professional behaviours and attitudes, there remains widespread reluctance to report students that behave inappropriately. Existing literature focuses on why faculty fail to fail, overlooking the tutors who deal with students day to day. We investigated how tutors address inappropriate behaviours and attitudes in students and residents.

METHODS

A mixed methods study was carried out consisting of a survey and two focus groups with tutors. Seventeen tutors from the University of Limerick School of Medicine, Ireland, took part in the survey (n = 22%) and eight tutors participated in two focus groups during the 2018-2019 academic year.

RESULTS

Findings suggested that 59% of tutors would take a different approach to addressing unprofessional behaviours witnessed in medical students and residents. A total of 88% of tutors said they intervened on a professionalism issue with 52% saying 'once in a while'. In contrast to the survey, tutors in the focus groups expressed a lack of confidence in addressing some behaviours due to a lack of time, not seeing the outcome of process/remediation etc. Tutors indicated a strong preference for case-based training on assessing professional identity formation (PIF).

CONCLUSIONS

We found tutors typically work closely with students on a day-to-day basis managing unprofessionalism issues. Tutors valued regular communication about policies and procedures about appropriate conduct as well as support, advice, and/or oversight from independent members of university staff. This research highlights the need for training designed for busy tutors as a distinct type of medical teacher.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s40670-021-01429-1.

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones

PURPOSE

Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones.

METHOD

A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017-2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression.

RESULTS

The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: β = -0.67, P = .003; ICS02: β = -0.70, P = .001; ICS03: β = -0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (β = -0.57, P = .007).

CONCLUSIONS

Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship.

Systematic review of the factors and the key indicators that identify doctors at risk of complaints, malpractice claims or impaired performance

OBJECTIVE

To identify the risk factors associated with complaints, malpractice claims and impaired performance in medical practitioners.

DESIGN

Systematic review.

DATA SOURCES

Ovid-Medline, Ovid Embase, Scopus and Cochrane Central Register of Controlled Trials were searched from 2011 until March 2020. Reference lists and Google were also handsearched.

RESULTS

Sixty-seven peer-reviewed papers and three grey literature publications from 2011 to March 2020 were reviewed by pairs of independent reviewers. Twenty-three key factors identified, which were categorised as demographic or workplace related. Gender, age, years spent in practice and greater number of patient lists were associated with higher risk of malpractice claim or complaint. Risk factors associated with physician impaired performance included substance abuse and burn-out.

CONCLUSIONS

It is likely that risk factors are interdependent with no single factor as a strong predictor of a doctor's risk to the public. Risk factors for malpractice claim or complaint are likely to be country specific due to differences in governance structures, processes and funding. Risk factors for impaired performance are likely to be specialty specific due to differences in work culture and access to substances. New ways of supporting doctors might be developed, using risk factor data to reduce adverse events and patient harm.

PROSPERO REGISTRATION NUMBER

PROSPERO registration number: CRD42020182045.

Perception of social media behaviour among medical students, residents and medical specialists

INTRODUCTION

Behaviour is visible in real-life events, but also on social media. While some national medical organizations have published social media guidelines, the number of studies on professional social media use in medical education is limited. This study aims to explore social media use among medical students, residents and medical specialists.

METHODS

An anonymous, online survey was sent to 3844 medical students at two Dutch medical schools, 828 residents and 426 medical specialists. Quantitative, descriptive data analysis regarding demographic data, yes/no questions and Likert scale questions were performed using SPSS. Qualitative data analysis was performed iteratively, independently by two researchers applying the principles of constant comparison, open and axial coding until consensus was reached.

RESULTS

Overall response rate was 24.8%. Facebook was most popular among medical students and residents; LinkedIn was most popular among medical specialists. Personal pictures and/or information about themselves on social media that were perceived as unprofessional were reported by 31.3% of students, 19.7% of residents and 4.1% of medical specialists. Information and pictures related to alcohol abuse, partying, clinical work or of a sexually suggestive character were considered inappropriate. Addressing colleagues about their unprofessional posts was perceived to be mainly dependent on the nature and hierarchy of the interprofessional relation.

DISCUSSION

There is a widespread perception that the presence of unprofessional information on social media among the participants and their colleagues is a common occurrence. Medical educators should create awareness of the risks of unprofessional (online) behaviour among healthcare professionals, as well as the necessity and ways of addressing colleagues in case of such lapses.

Early Detection and Remediation of Problem Learners

There are myriad types of problem learners in surgical residency and most have difficulty in more than 1 competency. Programs that use a standard curriculum of study and assessment are most successful in identifying struggling learners early. Many problem learners lack appropriate systems for study; a multidisciplinary educational team that is separate from the team that evaluates the success of remediation is critical. Struggling residents who require formal remediation benefit from performance improvement plans that clearly outline the issues of concern, describe the steps required for remediation, define success of remediation, and outline consequences for failure to remediate appropriately.

Evaluation of a High Stakes Physician Competency Assessment: Lessons for Assessor Training, Program Accountability, and Continuous Improvement

INTRODUCTION

There is a dearth of evidence evaluating postlicensure high-stakes physician competency assessment programs. Our purpose was to contribute to this evidence by evaluating a high-stakes assessment for assessor inter-rater reliability and the relationship between performance on individual assessment components and overall performance. We did so to determine if the assessment tools identify specific competency needs of the assessed physicians and contribute to our understanding of physician dyscompetence more broadly.

METHOD

Four assessors independently reviewed 102 video-recorded assessments and scored physicians on seven assessment components and overall performance. Inter-rater reliability was measured using intraclass correlation coefficients using a multiple rater, consistency, two-way random effect model. Analysis of variance with least-significant difference post-hoc analyses examined if the mean component scores differed significantly by quartile ranges of overall performance. Linear regression analysis determined the extent to which each component score was associated with overall performance.

RESULTS

Intraclass correlation coefficients ranged between 0.756 and 0.876 for all components scored and was highest for overall performance. Regression indicated that individual component scores were positively associated with overall performance. Levels of variation in component scores were significantly different across quartile ranges with higher variability in poorer performers.

DISCUSSION

High-stake assessments can be conducted reliably and identify performance gaps of potentially dyscompetent physicians. Physicians who performed well tended to do so in all aspects evaluated, whereas those who performed poorly demonstrated areas of strength and weakness. Understanding that dyscompetence rarely means a complete or catastrophic lapse competence is vital to understanding how educational needs change through a physician's career.