Programmatic Assessment of Clinical Reasoning: New Opportunities to Meet an Ongoing Challenge

Issue: Clinical reasoning is essential to physicians' competence, yet assessment of clinical reasoning remains a significant challenge. Clinical reasoning is a complex, evolving, non-linear, context-driven, and content-specific construct which arguably cannot be assessed at one point in time or with a single method. This has posed challenges for educators for many decades, despite significant development of individual assessment methods. Evidence: Programmatic assessment is a systematic assessment approach that is gaining momentum across health professions education. Programmatic assessment, and in particular assessment for learning, is well-suited to address the challenges with clinical reasoning assessment. Several key principles of programmatic assessment are particularly well-aligned with developing a system to assess clinical reasoning: longitudinality, triangulation, use of a mix of assessment methods, proportionality, implementation of intermediate evaluations/reviews with faculty coaches, use of assessment for feedback, and increase in learners' agency. Repeated exposure and measurement are critical to develop a clinical reasoning assessment narrative, thus the assessment approach should optimally be longitudinal, providing multiple opportunities for growth and development. Triangulation provides a lens to assess the multidimensionality and contextuality of clinical reasoning and that of its different, yet related components, using a mix of different assessment methods. Proportionality ensures the richness of information on which to draw conclusions is commensurate with the stakes of the decision. Coaching facilitates the development of a feedback culture and allows to assess growth over time, while enhancing learners' agency. Implications: A programmatic assessment model of clinical reasoning that is developmentally oriented, optimizes learning though feedback and coaching, uses multiple assessment methods, and provides opportunity for meaningful triangulation of data can help address some of the challenges of clinical reasoning assessment.

Learner Assessment and Program Evaluation: Supporting Precision Education

ABSTRACT

Precision education (PE) systematically leverages data and advanced analytics to inform educational interventions that, in turn, promote meaningful learner outcomes. PE does this by incorporating analytic results back into the education continuum through continuous feedback cycles. These data-informed sequences of planning, learning, assessing, and adjusting foster competence and adaptive expertise. PE cycles occur at individual (micro), program (meso), or system (macro) levels. This article focuses on program- and system-level PE.Data for PE come from a multitude of sources, including learner assessment and program evaluation. The authors describe the link between these data and the vital role evaluation plays in providing evidence of educational effectiveness. By including prior program evaluation research supporting this claim, the authors illustrate the link between training programs and patient outcomes. They also describe existing national reports providing feedback to programs and institutions, as well as 2 emerging, multiorganization program- and system-level PE efforts. The challenges encountered by those implementing PE and the continuing need to advance this work illuminate the necessity for increased cross-disciplinary collaborations and a national cross-organizational data-sharing effort.Finally, the authors propose practical approaches for funding a national initiative in PE as well as potential models for advancing the field of PE. Lessons learned from successes by others illustrate the promise of these recommendations.

How a Team Effectiveness Approach to Health Systems Science Can Illuminate Undergraduate Medical Education Outcomes

ABSTRACT

Health care delivery requires physicians to operate in teams to successfully navigate complexity in caring for patients and communities. The importance of training physicians early in core concepts of working in teams (i.e., "teaming") has long been established. Over the past decade, however, little evidence of team effectiveness training for medical students has been available. The recent introduction of health systems science as a third pillar of medical education provides an opportunity to teach and prepare students to work in teams and achieve related core competencies across the medical education continuum and health care delivery settings. Although educators and health care system leaders have emphasized the teaching and learning of team-based care, conceptual models and evidence that inform effective teaming within all aspects of undergraduate medical education (including classroom, clinical, and community settings) are needed to advance the science regarding learning and working in teams. Anchoring teaming through the core foundational theory of team effectiveness and its operational components could catalyze the empirical study of medical student teams, uncover modifiable factors that lead to the evidence for improved student learning, and improve the link among competency-based assessments between undergraduate medical education and graduate medical education. In this article, authors articulate several implications for medical schools through 5 conceptual areas: admissions, the design and teaching of team effectiveness in health systems science curricula, the related competency-based assessments, and course and program evaluations. The authors then discuss the relevance of the measurable components and intended outcomes to team effectiveness in undergraduate medical education as critical to successfully prepare students for teaming in clerkships and eventually residency and clinical practice.

Consensus-derived Recommended Skills for Transition to Residency Courses

PURPOSE

Transition to residency (TTR) courses facilitate the medical student-residency transition and are an integral part of senior medical student training. The authors established a common set of skills for TTR courses and an expected level of entrustment students should demonstrate in each skill on TTR course completion.

METHOD

A modified Delphi approach was used with 3 survey iterations between 2020 and 2022 to establish skills to be included in a TTR course. Nine TTR experts suggested general candidate skills and conducted a literature search to ensure no vital skills were missed. A stakeholder panel was solicited from email lists of TTR educators, residency program directors, and residents at the panelists' institutions. Consensus was defined as more than 75% of participants selecting a positive inclusion response. An entrustment questionnaire asked panelists to assign a level of expected entrustment to each skill, with 1 indicating observation only and 6 indicating perform independently.

RESULTS

The stakeholder panel initially consisted of 118 respondents with representation across educational contexts and clinical specialties. Response rates were 54% in iteration 2, 42% in iteration 3, and 33% on the entrustment questionnaire. After 3 iterations, 54 skills met consensus and were consolidated into 37 final skills categorized into 18 clinical skills (e.g., assessment and management of inpatient concerns), 14 communication skills (e.g., delivering serious news or having difficult conversations), 4 personal and professional skills (e.g., prioritization of clinical tasks), and 1 procedural skill (mask ventilation). Median entrustment levels were reported for all skills, with 19 skills having a level of expected entrustment of 4 (perform independently and have all findings double-checked).

CONCLUSIONS

These consensus skills can serve as the foundation of a standardized national TTR curriculum framework. Entrustment guidance may help educational leaders optimize training and allocation of resources for TTR curriculum development and implementation.

The Association of Subspecialty and Sex with Industry Payments to Internal Medicine Physicians Who Recently Completed Training

BACKGROUND

Financial relationships with drug and medical device companies may impact quality of care and academic research. However, little is known when and how these financial relationships develop among newly independent physicians who recently completed from residency or fellowship programs in internal medicine (IM).

OBJECTIVE

To compare patterns of industry payments among IM graduates.

DESIGN

Retrospective, observational cohort study.

SUBJECTS

IM graduates from residency or fellowship programs between January 2015 and December 2019.

MAIN MEASURES

We analyzed Open Payments reports made between July 2015 and June 2021 to recent graduates of U.S. Accreditation Council for Graduate Medical Education (ACGME)-accredited residency and fellowship programs in IM. The primary outcome was general payments accepted by these physicians, stratified by procedural (i.e., critical care medicine/pulmonary medicine, cardiac/cardiovascular disease, and gastroenterology) and non-procedural (i.e., infectious disease, general internal medicine, and other specialties) subspecialties. The secondary outcomes included general payments stratified by sex and age at residency or fellowship training completion.

KEY RESULTS

There were 41,669 IM physicians with a median age of 33.0 years. In the first 3 years after completion, the proportion of physicians accepting any general payments was 72.6%, 91.9%, and 86.8% in Critical Care Medicine/Pulmonary Medicine, Cardiac/Cardiovascular Disease, and Gastroenterology, compared to 56.1%, 52.6%, and 52.3% in Infectious Disease, General Internal Medicine, and Other Specialties (p<0.0001). After adjusting for confounding variables, the procedural group showed an increased hazard ratio (HR) for accepting any general payments and at least $5000 of general payments compared to the non-procedural group. The HRs of accepting any general payments in the procedural subspecialty were 2.26 (95% CI, 2.11-2.42) and 2.83 (95% CI, 2.70-2.97) in female and male physicians, respectively (p-value < 0.0001).

CONCLUSION

Industry financial relationships among newly independent physicians in IM exist immediately after completion of training and are influenced by subspecialty, sex, and age.

Competency-based medical education: The spark to ignite healthcare's escape fire

High-value care is what patients deserve and what healthcare professionals should deliver. However, it is not what happens much of the time. Quality improvement master Dr. Don Berwick argued more than two decades ago that American healthcare needs an escape fire, which is a new way of seeing and acting in a crisis situation. While coined in the U.S. context, the analogy applies in other Western healthcare contexts as well. Therefore, in this paper, the authors revisit Berwick's analogy, arguing that medical education can, and should, provide the spark for such an escape fire across the globe. They assert that medical education can achieve this by fully embracing competency-based medical education (CBME) as a way to place medicine's focus on the patient. CBME targets training outcomes that prepare graduates to optimize patient care. The authors use the escape fire analogy to argue that medical educators must drop long-held approaches and tools; treat CBME implementation as an adaptive challenge rather than a technical fix; demand genuine, rich discussions and engagement about the path forward; and, above all, center the patient in all they do.

The Association of ACGME Milestones With Performance on American Board of Surgery Assessments: A National Investigation of Surgical Trainees

OBJECTIVE

To determine the relationship between, and predictive utility of, milestone ratings and subsequent American Board of Surgery (ABS) vascular surgery in-training examination (VSITE), vascular qualifying examination (VQE), and vascular certifying examination (VCE) performance in a national cohort of vascular surgery trainees.

BACKGROUND

Specialty board certification is an important indicator of physician competence. However, predicting future board certification examination performance during training continues to be challenging.

METHODS

This is a national longitudinal cohort study examining relational and predictive associations between Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings and performance on VSITE, VQE, and VCE for all vascular surgery trainees from 2015 to 2021. Predictive associations between milestone ratings and VSITE were conducted using cross-classified random-effects regression. Cross-classified random-effects logistic regression was used to identify predictive associations between milestone ratings and VQE and VCE.

RESULTS

Milestone ratings were obtained for all residents and fellows(n=1,118) from 164 programs during the study period (from July 2015 to June 2021), including 145,959 total trainee assessments. Medical knowledge (MK) and patient care (PC) milestone ratings were strongly predictive of VSITE performance across all postgraduate years (PGYs) of training, with MK ratings demonstrating a slightly stronger predictive association overall (MK coefficient 17.26 to 35.76, β = 0.15 to 0.23). All core competency ratings were predictive of VSITE performance in PGYs 4 and 5. PGY 5 MK was highly predictive of VQE performance [OR 4.73, (95% CI, 3.87-5.78), P <0.001]. PC subcompetencies were also highly predictive of VQE performance in the final year of training [OR 4.14, (95% CI, 3.17-5.41), P <0.001]. All other competencies were also significantly predictive of first-attempt VQE pass with ORs of 1.53 and higher. PGY 4 ICS ratings [OR 4.0, (95% CI, 3.06-5.21), P <0.001] emerged as the strongest predictor of VCE first-attempt pass. Again, all subcompetency ratings remained significant predictors of first-attempt pass on CE with ORs of 1.48 and higher.

CONCLUSIONS

ACGME Milestone ratings are highly predictive of future VSITE performance, and first-attempt pass achievement on VQE and VCE in a national cohort of surgical trainees.

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

PURPOSE

The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM).

METHOD

Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings.

RESULTS

During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS).

CONCLUSIONS

The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Industry Payments Received by Residents During Training

Importance

Those responsible for medical education-specialties, sponsoring institutions, and program directors (PD)-are independently associated with the professional identity formation of the trainees with respect to potential conflicts of interest.

Objective

To identify the relative degree to which factors in the training environment are associated with resident acceptance of payments from pharmaceutical and medical device companies.

Design, Setting, and Participants

Cross-sectional, retrospective study of residents enrolled in the 3 largest primary-care specialties (internal medicine [IM], family medicine [FM], obstetrics and gynecology [OBGYN]) and 3 largest surgical disciplines (general surgery [GS], orthopedic surgery, and urology) during academic year 2020 to 2021. All analyses were conducted January through August 2023.

Exposures

Specialty, sponsoring institutions' ownership (nonprofit, for-profit, federal government, local government, or state government), and the number of payments PDs accepted.

Main outcomes and measures

Modified Poisson regression assessed the relative risk of ownership, specialty, and PD behavior on residents' acceptance of industry payments as recorded in the Open Payments Program (OPP) database.

Results

In total, there were 124 715 residents in all training programs during 2020 to 2021, 12% of whom received payments totaling $6.4 million. There were 65 992 residents in training during 2020 to 2021 in the 6 specialties evaluated in this study, with 4438 in orthopedics, 1779 in urology, 9177 in GS, 5819 in OBGYN, 14 493 in FM, and 30 286 in IM. OPP records $3.9 million in payments to the 8750 residents (13.4%) who received at least 1 industry payment. The record of all payments to residents in OPP totals $6.4 million. Compared with residents in federal sponsoring institutions, those affiliated with for-profit institutions were 3.50 (95% CI, 2.32-5.28) times more likely to accept industry payments, while those affiliated with nonprofit organizations were 2.00 (95% CI, 1.36-2.93) times more likely to accept payments. Compared with IM, residents in each of the following specialties have an elevated risk of accepting payments: orthopedics, 3.21 (95% CI, 2.73-3.77) times; urology, 2.95 (95% CI, 2.44-3.56) times; GS, 1.21 (95% CI, 1.00-1.45) times; OBGYN, 1.30 (95% CI, 1.05-1.62) times. The difference in the risk of accepting a payment between FM and IM residents was not statistically significant. The number of payments PDs accepted slightly elevated the risk of residents to accept a payment by 1.01 (95% CI, 1.01-1.01).

Conclusions and relevance

In this cross-sectional, retrospective study, receipt of industry payments by residents was associated with specialty, institutional control, and PD behavior.

Gender Bias and Ophthalmology Accreditation Council for Graduate Medical Education Milestones Evaluations

Importance

Women remain underrepresented in ophthalmology and gender-based disparities exist in salary, grant receipt, publication rates, and surgical volume throughout training and in practice. Although studies in emergency medicine and general surgery showed mixed findings regarding gender differences in Accreditation Council for Graduate Medical Education (ACGME) Milestones ratings, limited data exist examining such differences within ophthalmology.

Objective

To examine gender differences in ophthalmology ACGME Milestones.

Design, Setting, and Participants

This was a retrospective cross-sectional study of postgraduate year 4 (PGY-4) residents from 120 ophthalmology programs graduating in 2019.

Main Outcomes and Measures

PGY-4 midyear and year-end medical knowledge (MK) and patient care (PC) ratings and Written Qualifying Examination (WQE) scaled scores for residents graduating in 2019 were included. Differential prediction techniques using Generalized Estimating Equations models were performed to identify differences by gender.

Results

Of 452 residents (median [IQR] age, 30.0 [29.0-32.0] years), 275 (61%) identified as men and 177 (39%) as women. There were no differences in PC domain average between women and men for both midyear (-0.07; 95% CI, -0.11 to 0; P =.06) and year-end (-0.04; 95% CI, -0.07 to 0.03; P =.51) assessment periods. For the MK domain average in the midyear assessment period, women (mean [SD], 3.76 [0.50]) were rated lower than men (mean [SD], 3.88 [0.47]; P = .006) with a difference in mean of -0.12 (95% CI, -0.18 to -0.03). For the year-end assessment, however, the average MK ratings were not different for women (mean [SD], 4.10 [0.47]) compared with men (mean [SD], 4.18 [0.47]; P = .20) with a difference in mean of -0.08 (95% CI, -0.13 to 0.03).

Conclusions and Relevance

Results suggest that ACGME ophthalmology Milestones in 2 general competencies did not demonstrate major gender bias on a national level at the time of graduation. There were, however, differences in MK ratings at the midyear mark, and as low ratings on evaluations and examinations may adversely affect career opportunities for trainees, it is important to continue further work examining other competencies or performance measures for potential biases.

Intersectional Disparities in Emergency Medicine Residents' Performance Assessments by Race, Ethnicity, and Sex

Importance

Previous studies have demonstrated sex-specific disparities in performance assessments among emergency medicine (EM) residents. However, less work has focused on intersectional disparities by ethnoracial identity and sex in resident performance assessments.

Objective

To estimate intersectional sex-specific ethnoracial disparities in standardized EM resident assessments.

Design, Setting, and Participants

This retrospective cohort study used data from the Association of American Medical Colleges and the Accreditation Council for Graduate Medical Education Milestones (Milestones) assessments to evaluate ratings for EM residents at 128 EM training programs in the US. Statistical analyses were conducted in June 2020 to January 2023.

Exposure

Training and assessment environments in EM residency programs across comparison groups defined by ethnoracial identity (Asian, White, or groups underrepresented in medicine [URM], ie, African American/Black, American Indian/Alaska Native, Hispanic/Latine, and Native Hawaiian/Other Pacific Islander) and sex (female/male).

Main Outcomes and Measures

Mean Milestone scores (scale, 0-9) across 6 core competency domains: interpersonal and communications skills, medical knowledge, patient care, practice-based learning and improvement, professionalism, and system-based practice. Overall assessment scores were calculated as the mean of the 6 competency scores.

Results

The study sample comprised 128 ACGME-accredited programs and 16 634 assessments for 2708 EM residents of which 1913 (70.6%) were in 3-year and 795 (29.4%) in 4-year programs. Most of the residents were White (n = 2012; 74.3%), followed by Asian (n = 477; 17.6%), Hispanic or Latine (n = 213; 7.9%), African American or Black (n = 160; 5.9%), American Indian or Alaska Native (n = 24; 0.9%), and Native Hawaiian or Other Pacific Islander (n = 4; 0.1%). Approximately 14.3% (n = 386) and 34.6% (n = 936) were of URM groups and female, respectively. Compared with White male residents, URM female residents in 3-year programs were rated increasingly lower in the medical knowledge (URM female score, -0.47; 95% CI, -0.77 to -0.17), patient care (-0.18; 95% CI, -0.35 to -0.01), and practice-based learning and improvement (-0.37; 95% CI, -0.65 to -0.09) domains by postgraduate year 3 year-end assessment; URM female residents in 4-year programs were also rated lower in all 6 competencies over the assessment period.

Conclusions and Relevance

This retrospective cohort study found that URM female residents were consistently rated lower than White male residents on Milestone assessments, findings that may reflect intersectional discrimination in physician competency evaluation. Eliminating sex-specific ethnoracial disparities in resident assessments may contribute to equitable health care by removing barriers to retention and promotion of underrepresented and minoritized trainees and facilitating diversity and representation among the emergency physician workforce.

A consensus statement on the meaning, value and utility of training programme outcomes, with specific reference to anaesthesiology: A consensus statement on training programme outcomes

It is timely and necessary to consider what Postgraduate Medical Training Programme outcomes are, how they are defined and revised over time, and how they can be used to align health professional performance with the healthcare needs of society. This article which addresses those issues, with specific reference to training in anaesthesiology, was prepared using a modified nominal group (or expert panel) approach.

Trainee Physician Milestone Ratings and Patient Complaints in Early Posttraining Practice

Importance

Evaluation of trainees in graduate medical education training programs using Milestones has been in place since 2013. It is not known whether trainees who have lower ratings during the last year of training go on to have concerns related to interactions with patients in posttraining practice.

Objective

To investigate the association between resident Milestone ratings and posttraining patient complaints.

Design, Setting, and Participants

This retrospective cohort study included physicians who completed Accreditation Council for Graduate Medical Education (ACGME)-accredited programs between July 1, 2015, and June 30, 2019, and worked at a site that participated in the national Patient Advocacy Reporting System (PARS) program for at least 1 year. Milestone ratings from ACGME training programs and patient complaint data from PARS were collected. Data analysis was conducted from March 2022 to February 2023.

Exposures

Lowest professionalism (P) and interpersonal and communication skills (ICS) Milestones ratings 6 months prior to the end of training.

Main Outcomes and Measures

PARS year 1 index scores, based on recency and severity of complaints.

Results

The cohort included 9340 physicians with median (IQR) age of 33 (31-35) years; 4516 (48.4%) were women physicians. Overall, 7001 (75.0%) had a PARS year 1 index score of 0, 2023 (21.7%) had a score of 1 to 20 (moderate), and 316 (3.4%) had a score of 21 or greater (high). Among physicians in the lowest Milestones group, 34 of 716 (4.7%) had high PARS year 1 index scores, while 105 of 3617 (2.9%) with Milestone ratings of 4.0 (proficient), had high PARS year 1 index scores. In a multivariable ordinal regression model, physicians in the 2 lowest Milestones rating groups (0-2.5 and 3.0-3.5) were statistically significantly more likely to have higher PARS year 1 index scores than the reference group with Milestones ratings of 4.0 (0-2.5 group: odds ratio, 1.2 [95% CI, 1.0-1.5]; 3.0-3.5 group: odds ratio, 1.2 [95% CI, 1.1-1.3]).

Conclusions and Relevance

In this study, trainees with low Milestone ratings in P and ICS near the end of residency were at increased risk for patient complaints in their early posttraining independent physician practice. Trainees with lower Milestone ratings in P and ICS may need more support during graduate medical education training or in the early part of their posttraining practice career.

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data

OBJECTIVE

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets.

SUMMARY BACKGROUND DATA

ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation.

METHODS

National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings.

RESULTS

A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs.

CONCLUSION

The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

OBJECTIVE

Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories.

DESIGN

This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings.

SETTING

All ACGME-accredited vascular surgery training programs within the United States.

PARTICIPANTS

All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.

RESULTS

The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%).

CONCLUSIONS

As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

Systems and Complexity Thinking to Master Leadership Challenges in Interprofessional Health Professionals Education

The purpose of this article is to propose that knowledge, understanding, and application of systems and complexity thinking can improve assessment, implementation, and evaluation of interprofessional education (IPE). Using a case story, the authors describe and explain a meta-model of systems and complexity thinking to support leaders in implementing and evaluating IPE initiatives. The meta-model incorporates the use of several important, interrelated frameworks that tackle issues of sense making, systems, and complexity thinking as well as polarity management at different levels of scale in an organization. Combined, these theories and frameworks support recognition and management of cross-scale interactions and help leaders make sense of distinctions among simple, complicated, complex, and chaotic situations among IPE issues associated with healthcare disciplines within institutions. The application and use of Liberating Structures and polarity management practices enable leaders to engage people and gain insight into the complexities involved in successful implementation of IPE programs.

Trends in Industry Payments to Physicians in the First 6 Years After Graduate Medical Training

IMPORTANCE

Financial incentives and conflicts of interest may influence physician decision-making. It is important to understand financial interactions between the pharmaceutical and medical device industries and newly independent physicians who have recently completed their graduate medical education using a national transparency program.

OBJECTIVE

To identify trends in industry payments to recent graduates of Accreditation Council for Graduate Medical Education-accredited residency or fellowship programs in orthopedic surgery, neurosurgery, and internal medicine.

DESIGN, SETTING, AND PARTICIPANTS

This retrospective cohort study analyzed Open Payments reports of industry payments made between July 1, 2015, and June 30, 2021, to newly independent physicians from residency or fellowship programs in neurosurgery, orthopedic surgery, and internal medicine who graduated between January 1, 2015, and December 31, 2019.

EXPOSURES

Specialties (neurosurgery and orthopedic surgery, with internal medicine as a comparison group).

MAIN OUTCOMES AND MEASURES

Industry payments to newly independent physicians, including any general payments (noninvestment or nonresearch) and at least $5000 of general payments in aggregate value per year, which are considered significant financial conflicts of interest. The percentage of newly independent physicians accepting general payments during the first 6 years after graduation was analyzed by specialty and sex using cumulative incidence curves and hazard ratios (HRs) in univariable and multivariable analyses.

RESULTS

There were 45 745 recent graduates (28 137 men [62%]; median age at graduation, 33.0 [IQR, 31.0-35.0 years]) in neurosurgery (n = 595), orthopedic surgery (n = 3481), and internal medicine (n = 41 669). In the first 2 years of independent practice, 95% (n = 3297), 92% (n = 546), and 59% (n = 24 522) of newly independent physicians in orthopedic surgery, neurosurgery, and internal medicine, respectively, accepted any general payments. A higher percentage of the newly independent physicians in orthopedic surgery and neurosurgery accepted any general payments (orthopedic surgery vs internal medicine: HR, 5.36 [95% CI, 4.42-6.51] for women and 7.01 [95% CI, 6.35-7.73] for men; neurosurgery vs internal medicine: HR, 3.25 [95% CI, 2.24-4.72] for women and 4.08 [95% CI, 3.37-4.94] for men; P = .03). A higher percentage of male physicians compared with female physicians accepted any general payments (orthopedic surgery, 2884 of 3026 [95%] vs 413 of 455 [91%]; P < .001; neurosurgery, 466 of 502 [93%] vs 80 of 93 [86%]; P = .01; and internal medicine, 15 462 of 24 609 [63%] vs 9043 of 17 034 [53%]; P < .001) and at least $5000 of general payments (orthopedic surgery, 763 of 3026 [25%] vs 71 of 455 [16%]; P < .001; neurosurgery, 87 of 502 [17%] vs 5 of 93 [5%%]; P < .001; and internal medicine, 882 of 24 609 [4%] vs 210 of 17 034 [1%]; P < .001).

CONCLUSIONS AND RELEVANCE

In this cohort study of newly independent physicians in orthopedic surgery, neurosurgery, and internal medicine, the financial relationship with potential conflicts of interest between newly independent physicians and industry began to develop soon after training programs and continued to expand in the early years of newly independent physician practice. Newly independent physicians in surgical specialties and male physicians accepted significantly higher industry payments. Further studies are needed to evaluate whether modifiable factors are associated with the future outcome of newly independent physicians accepting general payments.

Advancing the science of health professions education through a shared understanding of terminology: a content analysis of terms for "faculty"

INTRODUCTION

Health professions educators risk misunderstandings where terms and concepts are not clearly defined, hampering the field's progress. This risk is especially pronounced with ambiguity in describing roles. This study explores the variety of terms used by researchers and educators to describe "faculty", with the aim to facilitate definitional clarity, and create a shared terminology and approach to describing this term.

METHODS

The authors analyzed journal article abstracts to identify the specific words and phrases used to describe individuals or groups of people referred to as faculty. To identify abstracts, PubMed articles indexed with the Medical Subject Heading "faculty" published between 2007 and 2017 were retrieved. Authors iteratively extracted data and used content analysis to identify patterns and themes.

RESULTS

A total of 5,436 citations were retrieved, of which 3,354 were deemed eligible. Based on a sample of 594 abstracts (17.7%), we found 279 unique terms. The most commonly used terms accounted for approximately one-third of the sample and included faculty or faculty member/s (n = 252; 26.4%); teacher/s (n = 59; 6.2%) and medical educator/s (n = 26; 2.7%) were also well represented. Content analysis highlighted that the different descriptors authors used referred to four role types: healthcare (e.g., doctor, physician), education (e.g., educator, teacher), academia (e.g., professor), and/or relationship to the learner (e.g., mentor).

DISCUSSION

Faculty are described using a wide variety of terms, which can be linked to four role descriptions. The authors propose a template for researchers and educators who want to refer to faculty in their papers. This is important to advance the field and increase readers' assessment of transferability.

A call to action: next steps to advance diagnosis education in the health professions

OBJECTIVES

Improving diagnosis-related education in the health professions has great potential to improve the quality and safety of diagnosis in practice. Twelve key diagnostic competencies have been delineated through a previous initiative. The objective of this project was to identify the next steps necessary for these to be incorporated broadly in education and training across the health professions.

METHODS

We focused on medicine, nursing, and pharmacy as examples. A literature review was conducted to survey the state of diagnosis education in these fields, and a consensus group was convened to specify next steps, using formal approaches to rank suggestions.

RESULTS

The literature review confirmed initial but insufficient progress towards addressing diagnosis-related education. By consensus, we identified the next steps necessary to advance diagnosis education, and five required elements relevant to every profession: 1) Developing a shared, common language for diagnosis, 2) developing the necessary content, 3) developing assessment tools, 4) promoting faculty development, and 5) spreading awareness of the need to improve education in regard to diagnosis.

CONCLUSIONS

The primary stakeholders, representing education, certification, accreditation, and licensure, in each profession must now take action in their own areas to encourage, promote, and enable improved diagnosis, and move these recommendations forward.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

IMPORTANCE

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.

OBJECTIVE

To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.

DESIGN, SETTING, AND PARTICIPANTS

This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.

INTERVENTIONS

Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.

MAIN OUTCOMES AND MEASURES

To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.

RESULTS

This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.

CONCLUSIONS AND RELEVANCE

These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Longitudinal Milestone Assessment Extending Through Subspecialty Training: The Relationship Between ACGME Internal Medicine Residency Milestones and Subsequent Pulmonary and Critical Care Fellowship Milestones

PURPOSE

Accreditation Council for Graduate Medical Education (ACGME) milestones were implemented across medical subspecialties in 2015. Although milestones were proposed as a longitudinal assessment tool potentially providing opportunities for early implementation of individualized fellowship learning plans, the association of subspecialty fellowship ratings with prior residency ratings remains unclear. This study aimed to assess the relationship between internal medicine (IM) residency milestones and pulmonary and critical care medicine (PCCM) fellowship milestones.

METHOD

A multicenter retrospective cohort analysis was conducted for all PCCM trainees in ACGME-accredited PCCM fellowship programs, 2017-2018, who had complete prior IM milestone ratings from 2014 to 2017. Only professionalism and interpersonal and communication skills (ICS) were included based on shared anchors between IM and PCCM milestones. Using a generalized estimating equations model, the association of PCCM milestones ≤ 2.5 during the first fellowship year with corresponding IM subcompetencies was assessed at each time point, nested by program. Statistical significance was determined using logistic regression.

RESULTS

The study included 354 unique PCCM fellows. For ICS and professionalism subcompetencies, fellows with higher IM ratings were less likely to obtain PCCM ratings ≤ 2.5 during the first fellowship year. Each ICS subcompetency was significantly associated with future lapses in fellowship (ICS01: β = -0.67, P = .003; ICS02: β = -0.70, P = .001; ICS03: β = -0.60, P = .004) at various residency time points. Similar associations were noted for PROF03 (β = -0.57, P = .007).

CONCLUSIONS

Findings demonstrated an association between IM milestone ratings and low milestone ratings during PCCM fellowship. IM trainees with low ratings in several professionalism and ICS subcompetencies were more likely to be rated ≤ 2.5 during the first PCCM fellowship year. This highlights a potential use of longitudinal milestones to target educational gaps at the beginning of PCCM fellowship.

Trainee Sex and Accreditation Council for Graduate Medical Education Milestone Assessments During General Surgery Residency

Importance

In evaluating the effectiveness of general surgery (GS) training, an unbiased assessment of the progression of residents with attention to individual learner factors is imperative.

Objective

To evaluate the role of trainee sex in milestone achievement over the course of GS residency using national data from the Accreditation Council for Graduate Medical Education (ACGME).

Design, Setting, and Participants

This cross-sectional study evaluated female and male GS residents enrolled in ACGME-accredited programs in the US from 2014 to 2018 with reported variation in milestones performance across years in training and representation. Data were analyzed from November 2019 to June 2021.

Main Outcomes and Measures

Mean reported milestone score at initial and final assessment, and predicted time-to-attainment of equivalent performance by sex.

Results

Among 4476 GS residents from 250 programs who had milestone assessments at any point in their clinical training, 1735 were female (38.8%). Initially, female and male residents received similar mean (SD) milestone scores (1.95 [0.50] vs 1.94 [0.50]; P = .69). At the final assessment, female trainees received overall lower mean milestone scores than male trainees (4.25 vs 4.31; P < .001). Significantly lower mean milestone scores were reported for female residents at the final assessment for several subcompetencies in both univariate and multivariate analyses, with only medical knowledge 1 (pathophysiology, diagnosis, and initial management) common to both. Multilevel mixed-effects linear modeling demonstrated that female trainees had significantly lower rates of monthly milestone attainment in the subcompetency of medical knowledge 1, which was associated with a significant difference in training time of approximately 1.8 months.

Conclusions and Relevance

Both female and male GS trainees achieved the competency scores necessary to transition to independence after residency as measured by the milestones assessment system. Initially, there were no sex differences in milestone score. By graduation, there were differences in the measured assessment of female and male trainees across several subcompetencies. Careful monitoring for sex bias in the evaluation of trainees and scrutiny of the training process is needed to ensure that surgical residency programs support the educational needs of both female and male trainees.

Simulation-Based Assessments and Graduating Neurology Residents' Milestones: Status Epilepticus Milestones

BACKGROUND

The American Board of Psychiatry and Neurology and the Accreditation Council for Graduate Medical Education (ACGME) developed Milestones that provide a framework for residents' assessment. However, Milestones do not provide a description for how programs should perform assessments.

OBJECTIVES

We evaluated graduating residents' status epilepticus (SE) identification and management skills and how they correlate with ACGME Milestones reported for epilepsy and management/treatment by their program's clinical competency committee (CCC).

METHODS

We performed a cohort study of graduating neurology residents from 3 academic medical centers in Chicago in 2018. We evaluated residents' skills identifying and managing SE using a simulation-based assessment (26-item checklist). Simulation-based assessment scores were compared to experience (number of SE cases each resident reported identifying and managing during residency), self-confidence in identifying and managing these cases, and their end of residency Milestones assigned by a CCC based on end-of-rotation evaluations.

RESULTS

Sixteen of 21 (76%) eligible residents participated in the study. Average SE checklist score was 15.6 of 26 checklist items correct (60%, SD 12.2%). There were no significant correlations between resident checklist performance and experience or self-confidence. The average participant's level of Milestone for epilepsy and management/treatment was high at 4.3 of 5 (SD 0.4) and 4.4 of 5 (SD 0.4), respectively. There were no significant associations between checklist skills performance and level of Milestone assigned.

CONCLUSIONS

Simulated SE skills performance of graduating neurology residents was poor. Our study suggests that end-of-rotation evaluations alone are inadequate for assigning Milestones for high-stakes clinical skills such as identification and management of SE.

Do Milestone Ratings Predict Physical Medicine and Rehabilitation Board Certification Examination Scores?

ABSTRACT

The Accreditation Council of Graduate Medical Education developed the Milestones to assist training programs in assessing resident physicians in the context of their participation in Accreditation Council of Graduate Medical Education-accredited training programs. Biannual assessments are done over a resident's entire training period to define the trajectory in achieving specialty-specific competencies. As part of its process of initial certification, the American Board of Physical Medicine and Rehabilitation requires successful completion of two examinations administered approximately 9 mos apart. The Part I Examination measures a single dimensional construct, physical medicine and rehabilitation medical knowledge, whereas Part II assesses the application of medical and physiatric knowledge to multiple domains, including data acquisition, problem solving, patient management, systems-based practice, and interpersonal and communication skills through specific patient case scenarios. This study aimed to investigate the validity of the Milestones by demonstrating its association with performance in the American Board of Physical Medicine and Rehabilitation certifying examinations. A cohort of 233 physical medicine and rehabilitation trainees in 3-yr residency programs (postgraduate year 2 entry) in the United States from academic years 2014-2016, who also took the American Board of Physical Medicine and Rehabilitation Parts I and II certifying examinations between 2016 and 2018, were included in the study. Milestones ratings in four distinct observation periods were correlated with scores in the American Board of Physical Medicine and Rehabilitation Parts I and II Examinations. Milestones ratings of medical knowledge (but not patient care, professionalism, problem-based learning, interpersonal and communication skills, and systems-based practice) predicted performance in subsequent Part I American Board of Physical Medicine and Rehabilitation Examination, but none of the Milestone ratings correlated with Part II Examination scaled scores.

Use of Resident-Sensitive Quality Measure Data in Entrustment Decision Making: A Qualitative Study of Clinical Competency Committee Members at One Pediatric Residency

PURPOSE

Resident-sensitive quality measures (RSQMs) are quality measures that are likely performed by an individual resident and are important to care quality for a given illness of interest. This study sought to explore how individual clinical competency committee (CCC) members interpret, use, and prioritize RSQMs alongside traditional assessment data when making a summative entrustment decision.

METHOD

In this constructivist grounded theory study, 19 members of the pediatric residency CCC at Cincinnati Children's Hospital Medical Center were purposively and theoretically sampled between February and July 2019. Participants were provided a deidentified resident assessment portfolio with traditional assessment data (milestone and/or entrustable professional activity ratings as well as narrative comments from 5 rotations) and RSQM performance data for 3 acute, common diagnoses in the pediatric emergency department (asthma, bronchiolitis, and closed head injury) from the emergency medicine rotation. Data collection consisted of 2 phases: (1) observation and think out loud while participants reviewed the portfolio and (2) semistructured interviews to probe participants' reviews. Analysis moved from close readings to coding and theme development, followed by the creation of a model illustrating theme interaction. Data collection and analysis were iterative.

RESULTS

Five dimensions for how participants interpret, use, and prioritize RSQMs were identified: (1) ability to orient to RSQMs: confusing to self-explanatory, (2) propensity to use RSQMs: reluctant to enthusiastic, (3) RSQM interpretation: requires contextualization to self-evident, (4) RSQMs for assessment decisions: not sticky to sticky, and (5) expectations for residents: potentially unfair to fair to use RSQMs. The interactions among these dimensions generated 3 RSQM data user profiles: eager incorporation, willing incorporation, and disinclined incorporation.

CONCLUSIONS

Participants used RSQMs to varying extents in their review of resident data and found such data helpful to varying degrees, supporting the inclusion of RSQMs as resident assessment data for CCC review.

Why Residencies Should Fly: Towards a Logical Approach to Duty Hour Reform

Since 2011, aviation has revolutionized their approach to safety. The aviation industry has adopted a multi-faceted approach to improve safety through decreasing duty hour limits and implementing processes to mitigate fatigue-related errors as well as creating cultural shifts in responsibility for safety. These changes have been guided by data generated by quality-improvement methodology. In contrast, duty hour limits in graduate medical education have not yet seen dramatic data-driven reform. Key advancements in aviation fatigue mitigation and implications for residency education are explored in this article. Scientifically based processes to optimize duty hours, quality-improvement strategies to iteratively monitor and reform duty limits, systematic change focusing on a just culture, and financial disincentives and incentives as a catalyst for change are discussed.

Health Systems Science in Medical Education: Unifying the Components to Catalyze Transformation

Medical education exists in the service of patients and communities and must continually calibrate its focus to ensure the achievement of these goals. To close gaps in U.S. health outcomes, medical education is steadily evolving to better prepare providers with the knowledge and skills to lead patient- and systems-level improvements. Systems-related competencies, including high-value care, quality improvement, population health, informatics, and systems thinking, are needed to achieve this but are often curricular islands in medical education, dependent on local context, and have lacked a unifying framework. The third pillar of medical education-health systems science (HSS)-complements the basic and clinical sciences and integrates the full range of systems-related competencies. Despite the movement toward HSS, there remains uncertainty and significant inconsistency in the application of HSS concepts and nomenclature within health care and medical education. In this Article, the authors (1) explore the historical context of several key systems-related competency areas; (2) describe HSS and highlight a schema crosswalk between HSS and systems-related national competency recommendations, accreditation standards, national and local curricula, educator recommendations, and textbooks; and (3) articulate 6 rationales for the use and integration of a broad HSS framework within medical education. These rationales include: (1) ensuring core competencies are not marginalized, (2) accounting for related and integrated competencies in curricular design, (3) providing the foundation for comprehensive assessments and evaluations, (4) providing a clear learning pathway for the undergraduate-graduate-workforce continuum, (5) facilitating a shift toward a national standard, and (6) catalyzing a new professional identity as systems citizens. Continued movement toward a cohesive framework will better align the clinical and educational missions by cultivating the next generation of systems-minded health care professionals.

Widening the lens on teaching and assessing clinical reasoning: from "in the head" to "out in the world"

Traditional teaching and assessment of clinical reasoning has focused on the individual clinician because of the preeminence of the information processing (IP) theory perspective. The clinician's mind has been viewed as the main source of effective or ineffective reasoning, and other participants, the environment and their interactions have been largely ignored. A social cognitive theoretical lens could enhance our understanding of how reasoning and error and the environment are linked. Therefore, a new approach in which the clinical reasoning process is situated and examined within the context may be required. The theories of embodied cognition, ecological psychology, situated cognition (SitCog) and distributed cognition (DCog) offer new insights to help the teacher and assessor enhance the quality of clinical reasoning instruction and assessment. We describe the teaching and assessment implications of clinical reasoning and error through the lens of this family of theories. Direct observation in different contexts focused on individual and team performance, simulation (with or without enhancement of technology), stimulated recall, think-aloud, and modeling are examples of teaching and assessment strategies grounded in this family of social cognitive theories. Educators may consider the instructional design of learning environments and educational tools that promote a situated educational approach to the teaching and assessment of clinical reasoning.

Resident-Sensitive Quality Measures in the Pediatric Emergency Department: Exploring Relationships With Supervisor Entrustment and Patient Acuity and Complexity

PURPOSE

This study explores the associations between resident-sensitive quality measures (RSQMs) and supervisor entrustment as well as between RSQMs and patient acuity and complexity for encounters in the pediatric emergency department (PED) in which residents are caring for patients.

METHOD

Pediatric residents rotating through Cincinnati Children's Hospital Medical Center PED as well as supervising pediatric emergency medicine faculty and fellows were recruited during the 2017-2018 academic year for the purpose of collecting the following data from the residents' patient encounters for 3 illnesses (acute asthma exacerbation, bronchiolitis, and closed head injury [CHI]): supervisor entrustment decision rating, RSQMs relevant to the care provided, and supervisor patient acuity and complexity ratings. To measure the association of RSQM composite scores with the other variables of interest, mixed models were used.

RESULTS

A total of 83 residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with CHI. Entrustment decision ratings were positively associated with asthma RSQM composite scores (beta coefficient = 0.03; P < .001). There was no significant association between RSQM composite scores and entrustment decision ratings for bronchiolitis or CHI. RSQM composite scores were significantly higher when acuity was also higher and significantly lower when acuity was also lower for both asthma (P < .001) and bronchiolitis (P = .01). However, RSQM composite scores were almost identical between levels of acuity for CHI (P = .94). There were no significant differences in RSQM composite scores when complexity varied.

CONCLUSION

This study found limited associations between RSQM composite scores and entrustment decision ratings but offers insight into how RSQMs could be used for the purposes of resident assessment and feedback.

Initial Implementation of Resident-Sensitive Quality Measures in the Pediatric Emergency Department: A Wide Range of Performance

PURPOSE

A lack of quality measures aligned with residents' work led to the development of resident-sensitive quality measures (RSQMs). This study sought to describe how often residents complete RSQMs, both individually and collectively, when they are implemented in the clinical environment.

METHOD

During academic year 2017-2018, categorical pediatric residents in the Cincinnati Children's Hospital Medical Center pediatric emergency department were assessed using RSQMs for acute asthma exacerbation (21 RSQMs), bronchiolitis (23 RSQMs), and closed head injury (19 RSQMs). Following eligible patient encounters, all individual RSQMs for the illnesses of interest were extracted from the health record. Frequencies of 3 performance classifications (opportunity and met, opportunity and not met, or no opportunity) were detailed for each RSQM. A composite score for each encounter was calculated by determining the proportion of individual RSQMs performed out of the total possible RSQMs that could have been performed.

RESULTS

Eighty-three residents cared for 110 patients with asthma, 112 with bronchiolitis, and 77 with closed head injury during the study period. Residents had the opportunity to meet the RSQMs in most encounters, but exceptions existed. There was a wide range in the frequency of residents meeting RSQMs in encounters in which the opportunity existed. One closed head injury measure was met in all encounters in which the opportunity existed. Across illnesses, some RSQMs were met in almost all encounters, while others were met in far fewer encounters. RSQM composite scores demonstrated significant range and variation as well-asthma: mean = 0.81 (standard deviation [SD] = 0.11) and range = 0.47-1.00, bronchiolitis: mean = 0.62 (SD = 0.12) and range = 0.35-0.91, and closed head injury: mean = 0.63 (SD = 0.10) and range = 0.44-0.89.

CONCLUSIONS

Individually and collectively, RSQMs can distinguish variations in the tasks residents perform across patient encounters.

The Power of Contribution and Attribution in Assessing Educational Outcomes for Individuals, Teams, and Programs

Recent discussions have brought attention to the utility of contribution analysis for evaluating the effectiveness and outcomes of medical education programs, especially for complex initiatives such as competency-based medical education. Contribution analysis focuses on the extent to which different entities contribute to an outcome. Given that health care is provided by teams, contribution analysis is well suited to evaluating the outcomes of care delivery. Furthermore, contribution analysis plays an important role in analyzing program- and system-level outcomes that inform program evaluation and program-level improvements for the future. Equally important in health care, however, is the role of the individual. In the overall contribution of a team to an outcome, some aspects of this outcome can be attributed to individual team members. For example, a recently discharged patient with an unplanned return to the emergency department to seek care may not have understood the discharge instructions given by the nurse or may not have received any discharge guidance from the resident physician. In this example, if it is the nurse's responsibility to provide discharge instructions, that activity is attributed to him or her. This and other activities attributed to different individuals (e.g., nurse, resident) combine to contribute to the outcome for the patient. Determining how to tease out such attributions is important for several reasons. First, it is physicians, not teams, that graduate and are granted certification and credentials for medical practice. Second, incentive-based payment models focus on the quality of care provided by an individual. Third, an individual can use data about his or her performance on the team to help drive personal improvement. In this article, the authors explored how attribution and contribution analyses can be used in a complimentary fashion to discern which outcomes can and should be attributed to individuals, which to teams, and which to programs.

Coproducing Health Professions Education: A Prerequisite to Coproducing Health Care Services?

In 2016, Batalden et al proposed a coproduction model for health care services. Starting from the argument that health care services should demonstrate service-dominant rather than goods-dominant logic, they argued that health care outcomes are the result of the intricate interaction of the provider and patient in concert with the system, community, and, ultimately, society. The key notion is that the patient is as much an expert in determining outcomes as the provider, but with different expertise. Patients come to the table with expertise in their lived experiences and the context of their lives.The authors posit that education, like health care services, should follow a service-dominant logic. Like the relationship between patients and providers, the relationship between learner and teacher requires the integrated expertise of each nested in the context of their system, community, and society to optimize outcomes. The authors then argue that health professions learners cannot be educated in a traditional, paternalistic model of education and then expected to practice in a manner that prioritizes coproductive partnerships with colleagues, patients, and families. They stress the necessity of adapting the health care services coproduction model to health professions education. Instead of asking whether the coproduction model is possible in the current system, they argue that the current system is not sustainable and not producing the desired kind of clinicians.A current example from a longitudinal integrated clerkship highlights some possibilities with coproduced education. Finally, the authors offer some practical ways to begin changing from the traditional model. They thus provide a conceptual framework and ideas for practical implementation to move the educational model closer to the coproduction health care services model that many strive for and, through that alignment, to set the stage for improved health outcomes for all.

Entrustment decisions: Implications for curriculum development and assessment

With increased interest in the use of entrustable professional activities (EPAs) in undergraduate and postgraduate medical education, comes questions about their implications for curriculum development and assessment. This paper addresses some of those questions, discussed at a symposium of the 2017 conference of AMEE, by presenting the components of an EPA, describing their importance and application, identifying their implications for assessment, and pinpointing some of challenges they pose in undergraduate and postgraduate settings. It defines entrustment, describes the three levels of trust, and presents trainee and supervisor factors that influence it as well as perceived benefits, and risks. Two aspects of EPAs have implications for assessment: units of professional practice and decisions based on entrustment, which impact an assessment's blueprint, test methods, scores, and standards. In an undergraduate setting EPAs have great appeal, but work is needed to identify and develop a robust assessment system for core EPAs. At the postgraduate level, there is tension between the granularity of the competencies and the integrated nature of the EPAs. Even though work remains, EPAs offer an important step in the evolution of competency-based education.

Situativity: a family of social cognitive theories for understanding clinical reasoning and diagnostic error

The diagnostic error crisis suggests a shift in how we view clinical reasoning and may be vital for transforming how we view clinical encounters. Building upon the literature, we propose clinical reasoning and error are context-specific and proceed to advance a family of theories that represent a model outlining the complex interplay of physician, patient, and environmental factors driving clinical reasoning and error. These contemporary social cognitive theories (i.e. embedded cognition, ecological psychology, situated cognition, and distributed cognition) can emphasize the dynamic interactions occurring amongst participants in particular settings. The situational determinants that contribute to diagnostic error are also explored.

Mapping clinical reasoning literature across the health professions: a scoping review

BACKGROUND

Clinical reasoning is at the core of health professionals' practice. A mapping of what constitutes clinical reasoning could support the teaching, development, and assessment of clinical reasoning across the health professions.

METHODS

We conducted a scoping study to map the literature on clinical reasoning across health professions literature in the context of a larger Best Evidence Medical Education (BEME) review on clinical reasoning assessment. Seven databases were searched using subheadings and terms relating to clinical reasoning, assessment, and Health Professions. Data analysis focused on a comprehensive analysis of bibliometric characteristics and the use of varied terminology to refer to clinical reasoning.

RESULTS

Literature identified: 625 papers spanning 47 years (1968-2014), in 155 journals, from 544 first authors, across eighteen Health Professions. Thirty-seven percent of papers used the term clinical reasoning; and 110 other terms referring to the concept of clinical reasoning were identified. Consensus on the categorization of terms was reached for 65 terms across six different categories: reasoning skills, reasoning performance, reasoning process, outcome of reasoning, context of reasoning, and purpose/goal of reasoning. Categories of terminology used differed across Health Professions and publication types.

DISCUSSION

Many diverse terms were present and were used differently across literature contexts. These terms likely reflect different operationalisations, or conceptualizations, of clinical reasoning as well as the complex, multi-dimensional nature of this concept. We advise authors to make the intended meaning of 'clinical reasoning' and associated terms in their work explicit in order to facilitate teaching, assessment, and research communication.

The terminology of clinical reasoning in health professions education: Implications and considerations

Introduction: Clinical reasoning is considered to be at the core of health practice. Here, we report on the diversity and inferred meanings of the terms used to refer to clinical reasoning and consider implications for teaching and assessment. Methods: In the context of a Best Evidence Medical Education (BEME) review of 625 papers drawn from 18 health professions, we identified 110 terms for clinical reasoning. We focus on iterative categorization of these terms across three phases of coding and considerations for how terminology influences educational practices. Results: Following iterative coding with 5 team members, consensus was possible for 74, majority coding was possible for 16, and full team disagreement existed for 20 terms. Categories of terms included: purpose/goal of reasoning, outcome of reasoning, reasoning performance, reasoning processes, reasoning skills, and context of reasoning. Discussion: Findings suggest that terms used in reference to clinical reasoning are non-synonymous, not uniformly understood, and the level of agreement differed across terms. If the language we use to describe, to teach, or to assess clinical reasoning is not similarly understood across clinical teachers, program directors, and learners, this could lead to confusion regarding what the educational or assessment targets are for "clinical reasoning."

A Feasibility Study to Attribute Patients to Primary Interns on Inpatient Ward Teams Using Electronic Health Record Data

PURPOSE

To inform graduate medical education (GME) outcomes at the individual resident level, this study sought a method for attributing care for individual patients to individual interns based on "footprints" in the electronic health record (EHR).

METHOD

Primary interns caring for patients on an internal medicine inpatient service were recorded daily by five attending physicians of record at University of Cincinnati Medical Center in August 2017 and January 2018. These records were considered gold standard identification of primary interns. The following EHR variables were explored to determine representation of primary intern involvement in care: postgraduate year, progress note author, discharge summary author, physician order placement, and logging clicks in the patient record. These variables were turned into quantitative attributes (e.g., progress note author: yes/no), and informative attributes were selected and modeled using a decision tree algorithm.

RESULTS

A total of 1,511 access records were generated; 116 were marked as having a primary intern assigned. All variables except discharge summary author displayed at least some level of importance in the models. The best model achieved 78.95% sensitivity, 97.61% specificity, and an area under the receiver-operator curve of approximately 91%.

CONCLUSIONS

This study successfully predicted primary interns caring for patients on inpatient teams using EHR data with excellent model performance. This provides a foundation for attributing patients to primary interns for the purposes of determining patient diagnoses and complexity the interns see as well as supporting continuous quality improvement efforts in GME.

Learning environment, stress and coping in psychiatry residents within a national training program: a longitudinal study

INTRODUCTION

Perception of one's learning environment is associated with academic performance and well-being of medical students. Yet, few studies have explored learners' perceptions and their correlates within a postgraduate environment. This study examined longitudinal changes in learning environment perception, perceived stress and coping for psychiatry residents in junior and senior years of training. Based on extant social learning theories, we hypothesized that learning environment perceptions will improve with seniority, and be accompanied by lower stress levels and better coping.

METHODS

Eighty-two out of 101 psychiatry residents (81.2%) from our National Psychiatry Residency Program rated their perception of learning environment, perceived stress, and coping strategies from June 2016 to June 2018. Correlations between these variables, and changes across five timepoints were examined.

RESULTS

Senior-year residents reported better perception of learning environment over time, together with increased use of problem-focused coping and lowered perceived stress levels. Junior-year residents reported no changes in learning environment perception and coping strategies, despite rating greater perceived stress levels over time. Perception of learning environment negatively correlated with perceived stress levels and specific coping strategies.

DISCUSSION

Based on these findings, we suggest specific strategies with the emphasis on context, participation, and social interaction within a community of practice to better support residents in training, which are applicable to other similar training programs.

Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance

PURPOSE

An evidence-based approach to assessment is critical for ensuring the development of clinical reasoning (CR) competence. The wide array of CR assessment methods creates challenges for selecting assessments fit for the purpose; thus, a synthesis of the current evidence is needed to guide practice. A scoping review was performed to explore the existing menu of CR assessments.

METHOD

Multiple databases were searched from their inception to 2016 following PRISMA guidelines. Articles of all study design types were included if they studied a CR assessment method. The articles were sorted by assessment methods and reviewed by pairs of authors. Extracted data were used to construct descriptive appendixes, summarizing each method, including common stimuli, response formats, scoring, typical uses, validity considerations, feasibility issues, advantages, and disadvantages.

RESULTS

A total of 377 articles were included in the final synthesis. The articles broadly fell into three categories: non-workplace-based assessments (e.g., multiple-choice questions, extended matching questions, key feature examinations, script concordance tests); assessments in simulated clinical environments (objective structured clinical examinations and technology-enhanced simulation); and workplace-based assessments (e.g., direct observations, global assessments, oral case presentations, written notes). Validity considerations, feasibility issues, advantages, and disadvantages differed by method.

CONCLUSIONS

There are numerous assessment methods that align with different components of the complex construct of CR. Ensuring competency requires the development of programs of assessment that address all components of CR. Such programs are ideally constructed of complementary assessment methods to account for each method's validity and feasibility issues, advantages, and disadvantages.

Group Trust in the Setting of Competency Committee Decisions: A Qualitative Observational Study

This article was migrated. The article was not marked as recommended. Introduction: Trust is an important foundational component of a competency committee. Little work has been done in health professions education to look at trust and group decisions. A recent scoping review defined "group trust," examined factors that may influence group trust, and proposed a model to illustrate the relationship between trust at the individual and group levels. The purpose of this study was to test this model in the context of competency committee decisions and assignment of the ACGME Milestones. Methods: The authors conducted structured observations of competency committee meetings in internal medicine and pediatrics residency and subspecialty programs within the National Capital Consortium. Data were analyzed using a constant comparison method. Results: The authors observed six meetings from May to June 2018 (15 hours) reaching thematic saturation by the final meeting. The proposed model served as an organizing framework for reporting findings into the following categories: individual and group level factors, interpersonal interactions, and environmental factors. Results suggest that diversity of opinion promotes a more complete understanding of trainee competence. Varied faculty experiences, interactions, and actions by committee chairs helped to build a shared group mental model, which was identified as one of the most important aspects of making collective decisions and assigning Milestones. Meetings generally had similar structure based upon the size of the training program, and groupthink was more of a risk for larger training programs, especially when assigning individual Milestones. An important environmental factor identified was decision making fatigue, which was most prevalent towards the end of committee meetings. Conclusions: As training programs develop policies, procedures, and determine the membership of competency committees, utilizing these findings may help improve the design and execution of these groups, enhancing the translation of evaluation data and individual opinions into Milestone and competency decisions.