Diagnostic radiology milestones

Diagnostic Radiology Residency Assessment Tools: A Scoping Review

PURPOSE

The multifaceted nature of learning in diagnostic radiology residency requires a variety of assessment methods. However, the scope and quality of assessment tools has not been formally examined. A scoping review was performed to identify assessment tools available for radiology resident training and to evaluate the validity of these tools.

METHODS

A literature search was conducted through multiple databases and on-line resources. Inclusion criteria were defined as any tool used in assessment of radiology resident competence. Data regarding residents, evaluators and specifics of each tool was extracted. Each tool was subjected through a validation process with a customized rating scale using the 5 categories of validity: content, response process, internal structure, relations to other variables, and consequences.

RESULTS

The initial search returned 447 articles; 35 were included. The most evaluated competency being overall knowledge (31%), most common published journal was Academic Radiology (24%); evaluations were most commonly set in the United States (57%). In terms of validation, we found low adherence to modern integrated validity, with 34% of studies including a definition of validity. When specifically examining the 5 domains of validation evidence presented, most were either absent or of low rigor (70%). Only one study presented a modern definition of validation (3%, 1/35).

CONCLUSION

We identified 35 evaluation tools covering a variety of competency areas. However, few of these tools have been validated. Development of new validated assessment tools or validation of existing tools is essential for the ongoing transition to a competency-based curriculum.

Full Resolution Simulation for Evaluation of Critical Care Imaging Interpretation; Part 1: Fixed Effects Identify Influences of Exam, Specialty, Fatigue, and Training on Resident Performance

RATIONALE AND OBJECTIVES

To describe our full-resolution simulation of critical care imaging coupled with posthoc grading of resident's interpretations and present results from the fixed effects terms in a comprehensive mixed regression model of the resulting scores.

MATERIALS AND METHODS

The system delivered full resolution DICOM studies via clinical-grade viewing software integrated with a custom built web-based workflow and reporting system. The interpretations submitted by participating residents from 47 different programs were graded (scores of 0-10) on a case by case basis by a cadre of faculty members from our department. The data from 5 yearly (2014-2018) cycles consisting of 992 separate 65 case, 8 hour simulation sessions were collated from the transaction records. We used a mixed (hierarchical) statistical model with nine fixed and four random independent variables. In this paper, we present the results from the nine fixed effects.

RESULTS

There were 19,916/63,839 (27.0%, CI 26.7%-27.4%) scores in the 0-2 range (i.e., clinically significant miss). Neurological cases were more difficult with adjusted scores 2.3 (CI 1.9-3.2) lower than body/musculoskeletal cases. There was a small (0.3, CI 0.20-0.38 points) but highly significant (p<0.0001) decrease in score for the final 13/65 cases (fifth quintile) as evidence of fatigue during the last hour of an 8 hour shift. By comparing adjusted scores from mid-R1 (quarter 3) to late-R3 (quarter 12) we estimate the training effect as an increase of 2.2 (CI 1.90-2.50) points.

CONCLUSION

Full resolution simulation based evaluation of critical care radiology interpretation is being conducted remotely and efficiently at large scale. Analysis of the resulting scores yields multiple insights into the interpretative process.

Validation of an informatics tool to assess resident's progress in developing reporting skills

Background

Diagnostic radiology residency programs pursuits as main objectives of the development of diagnostic capabilities and written communication skills to answer clinicians’ questions of referring clinicians.

There has been also an increasing focus on competencies, rather than just education inputs. Then, to show ongoing professional development is necessary for a system to assess and document resident’s competence in these areas.

Therefore, we propose the implementation of an informatics tool to objectively assess resident’s progress in developing diagnostics and reporting skills. We expect to found decreased preliminary report-final report variability within the course of each year of the residency program.

Results

We analyzed 12,162 evaluations from 32 residents (8 residents per year in a 4-year residency program) in a 7-month period. 73.96% of these evaluations belong to 2nd-year residents.

We chose two indicators to study the evolution of evaluations: the total of discrepancies over the total of preliminary reports (excluding score 0) and the total of likely to be clinically significant discrepancies (scores 2b, 3b, and 4b) over the total of preliminary reports (excluding score 0).

With the analysis of these two indicators over the evaluations of 2nd-year residents, we found a slight decrease in the value of the first indicator and relative stable behavior of the second one.

Conclusions

This tool is useful for objective assessment of reporting skill of radiology residents. It can provide an opportunity for continuing medical education with case-based learning from those cases with clinically significant discrepancies between the preliminary and the final report.

Electronic supplementary material

The online version of this article (10.1186/s13244-019-0772-0) contains supplementary material, which is available to authorized users.

Supporting Radiology Residents' Professional Development Through a Competitive Intramural Grant

Research and other scholarly activities are an important and required component of diagnostic radiology training. Several strategies, both at the departmental and the larger organizational levels, have been implemented to encourage radiology trainees to participate in these activities. In this article, we review and discuss our institution's 10-year experience in supporting the development and realization of scholarly projects through a competitive intramural grant for residents.

Resident Case Volume Correlates with Clinical Performance: Finding the Sweet Spot

RATIONALE AND OBJECTIVES

To determine whether the total number of studies interpreted during radiology residency correlates with clinical performance as measured by objective criteria.

MATERIALS AND METHODS

We performed a retrospective cohort study of three graduating classes of radiology residents from a single residency program between the years 2015-2017. The total number of studies interpreted by each resident during residency was tracked. Clinical performance was determined by tracking an individual resident's major discordance rate. A major discordance was recorded when there was a difference between the preliminary resident interpretation and final attending interpretation that could immediately impact patient care. Accreditation council for graduate medical education milestones at the completion of residency, Diagnostic radiology in-training scores in the third year, and score from the American board of radiology core exam were also tabulated. Pearson correlation coefficients and polynomial regression analysis were used to identify correlations between the total number of interpreted films and clinical, test, and milestone performance.

RESULTS

Thirty-seven residents interpreted a mean of 12,709 studies (range 8898-19,818; standard deviation [SD] 2351.9) in residency with a mean major discordance rate of 1.1% (range 0.34%-2.54%; stand dev 0.49%). There was a nonlinear correlation between total number of interpreted films and performance. As the number of interpreted films increased to approximately 16,000, clinical performance (p = 0.004) and test performance (p = 0.01) improved, but volumes over 16,000 correlated with worse performance.

CONCLUSION

The total number of studies interpreted during radiology training correlates with performance. Residencies should endeavor to find the "sweet spot": the amount of work that maximizes clinical exposure and knowledge without overburdening trainees.

Radiology Education in Medical School and Residency: The Views and Needs of Program Directors

RATIONALE AND OBJECTIVES

The authors of this study used the perspectives of residency program directors (PDs) nationally to explore whether trainees are adequately prepared to utilize and interpret medical imaging as interns, to identify the types of imaging skills most important for residency, and to begin to address current shortcomings in radiology education.

MATERIALS AND METHODS

The authors created a survey using a modified version of Accreditation Council for Graduate Medical Education radiology milestones and sent it to 100 randomly selected PDs each in pediatrics, internal medicine, obstetrics and gynecology, and general surgery. The survey asked PDs to assess the actual and desired imaging skills of their incoming interns, the incoming interns' variability of skill level upon matriculation, and which imaging skills were most important from the PDs' perspective.

RESULTS

PDs from all specialties identified a significant shortcoming relative to their expectations for both image interpretation and utilization skills. Additionally, PDs identified a significant variability in imaging skills, and described that variability as a hindrance to their programs. All of the potential imaging skills were rated as highly important with little clinically relevant difference between them.

DISCUSSION

This multidisciplinary national survey found a deficiency in imaging education among interns across specialties and substantiates calls for formalized and improved radiology education in undergraduate medical education. Additionally, PDs had difficulty distinguishing which skills were most important, suggesting an unclear understanding of imaging ability needs for interns in respective specialties. More specific needs assessments are warranted on a national level.

Cultivating Future Radiology Educators: Development and Implementation of a Clinician-Educator Track for Residents

Effective and dedicated educators are critical to the preservation and advancement of the practice of radiology. The need for innovative and adaptable educators is increasingly being recognized, with several institutions granting academic promotions through clinician-educator tracks. The implementation of resident "clinician-educator tracks" or "teaching tracks" should better prepare residents aspiring to become academic radiologists focused on teaching. In this work, we describe our experience in the development and implementation of a clinician-educator track for diagnostic radiology residents at our institution.

Radiology Resident Assessment and Feedback Dashboard

Assessment of residents is optimally performed through processes and platforms that provide daily feedback, which can be immediately acted on. Given the documentation required by the Accreditation Council for Graduate Medical Education (ACGME), effective data management, integration, and presentation are crucial to ease the burden of manual documentation and increase the timeliness of actionable information. To this end, the authors modeled the learning activities of residents using the Experience Application Programming Interface (xAPI) framework, which is a standard framework for the learning community. On the basis of the xAPI framework and using open-source software to extend their existing infrastructure, the authors developed a Web-based dashboard that provides residents with a more holistic view of their educational experience. The dashboard was designed around the ACGME radiology milestones and provides real-time feedback to residents using various assessment metrics derived from multiple data sources. The purpose of this article is to describe the dashboard's architecture and components, the design and technical considerations, and the lessons learned in implementing the dashboard. ^©RSNA, 2018.

Residency Mini-fellowships in the PGY-5 Year: Is There Added Value?

RATIONALE AND OBJECTIVES

With the restructuring of radiology board certification, many residencies created PGY-5 "mini-fellowships," during which residents spend focused time pursuing advanced subspecialty training or developing nonclinical skills in leadership, health policy and health-care economics, education, quality improvement, informatics, research, or global health. We surveyed graduates of an academic diagnostic radiology residency to assess the relative value and impact of PGY-5 mini-fellowships on career satisfaction and success.

METHODS

From 2012 to 2016, 39 radiology residents at our institution were offered the opportunity to pursue a 3- to 6-month mini-fellowship during the PGY-5 year. Thirty of 39 radiology residents (77%) participated, whereas 9 of 39 (23%) opted out. Of 39 residents, 13 completed two clinical mini-fellowships, 3 completed research mini-fellowships only, and 14 completed one nonclinical and one clinical mini-fellowship. Through SurveyMonkey, 23 of 39 residents (59%) responded to a questionnaire that collected basic demographic information and asked respondents about the value of this experience as it relates to fellowship choice and career using a five-point Likert scale.

RESULTS

Of 23 respondents (14 male, 8 female,1 not specified), 78.3% practice in an academic university-based setting, with 8.7% in a community-based hospital practice, 4.3% in the veterans system, and 4.3% in a private practice setting. Of 23 respondents, the most popular clinical mini-fellowships were magnetic resonance imaging (31.6%), neuroradiology (21.1%), and interventional radiology (15.8%). For nonclinical mini-fellowships, the most popular were research (10.5%), education (10.5%), global health (5.3%), and healthcare economics (5.3%). Of 23 respondents who did mini-fellowships, 95% felt that the mini-fellowship prepared them well for their career, 85% felt it gave them the necessary skills to succeed, 85% cited that it gave them additional skills beyond their peers, and 40% felt it helped them create a life-long connection to a mentor. Ninety-five percent of respondents would choose to do the mini-fellowship again. Respondents suggested increasing the duration to 6-9 months and to develop a more structured curriculum and mentorship component. Only one respondent felt that the nonclinical mini-fellowship took away time from furthering clinical skills.

CONCLUSIONS

Graduates of a university-affiliated academic radiology residency who participated in clinical and nonclinical mini-fellowships during the PGY-5 year of residency greatly value this experience and uniformly recommend that this type of program continue to be offered to trainees given its ability to develop skills perceived to be vital to ultimate career satisfaction and success.

Teaching and Assessing Professionalism in Radiology: Resources and Scholarly Opportunities to Contribute to Required Expectations

Teaching and assessing trainees' professionalism now represents an explicit expectation for Accreditation Council Graduate Medical Education-accredited radiology programs. Challenges to meeting this expectation include variability in defining the construct of professionalism; limits of traditional teaching and assessment methods, used for competencies historically more prominent in medical education, for professionalism; and emerging expectations for credible and feasible professionalism teaching and assessment practices in the current context of health-care training and practice. This article identifies promising teaching resources and methods that can be used strategically to augment traditional teaching of the cognitive basis for professionalism, including role modeling, case-based scenarios, debriefing, simulations, narrative medicine (storytelling), guided discussions, peer-assisted learning, and reflective practice. This article also summarizes assessment practices intended to promote learning, as well as to inform how and when to assess trainees as their professional identities develop over time, settings, and autonomous practice, particularly in terms of measurable behaviors. This includes assessment tools (including mini observations, critical incident reports, and appreciative inquiry) for authentic assessment in the workplace; engaging multiple sources (self-, peer, other health professionals, and patients) in assessment; and intentional practices for trainees to take responsibility for seeking our actionable feedback and reflection. This article examines the emerging evidence of the feasibility and value added of assessment of medical competency milestones, including professionalism, coordinated by the Accreditation Council Graduate Medical Education in radiology and other medical specialties. Radiology has a strategic opportunity to contribute to scholarship and inform policies in professionalism teaching and assessment practices.

Designing a Curriculum for Professionalism and Ethics Within Radiology: Identifying Challenges and Expectations

Although professionalism and ethics represent required competencies, they are more challenging than other competencies to design a curriculum for and teach. Reasons include variability in agreed definitions of professionalism within medicine and radiology. This competency is also framed differently whether as roles, duties, actions, skills, behavior, beliefs, and attitudes. Standardizing a curriculum in professionalism is difficult because each learner's (medical student/resident) professional experiences and interactions will be unique. Professionalism is intertwined throughout all (sub) specialties and areas and its teaching cannot occur in isolation as a standalone curriculum. In the past, professionalism was not emphasized enough or at all, with global (or no) assessments, with the potential effect of trainees not valuing it. Although we can teach it formally in the classroom and informally in small groups, much of professionalism is witnessed and learned as "hidden curricula". The formal, informal, and hidden curricula often contradict each other creating confusion, disillusion, and cynicism in trainees. The corporatization of medicine pressurizes us to increase efficiency (throughput) with less focus on aspects of professionalism that add value, creating a disjoint between what we do in practice and preach to trainees. Progressively, expectations for our curriculum include providing evidence for the impacts of our efforts on patient outcomes. Generational differences in the perception of professionalism and the increasingly diverse and multicultural society in which we live affects our interpretation of professionalism, which can add to confusion and misunderstanding. The objectives of this article are to outline challenges facing curriculum design in professionalism and to make suggestions to help educators avoid or overcome them.

Self-Study: Practical Tips for a Successful and Rewarding Experience

The Accreditation Council for Graduate Medical Education (ACGME) self-study is a new process for ACGME accredited radiology programs. This article serves to provide the reader with the evolution of ACGME accreditation leading to the conception of the self-study process, detail the self-study method, and offer practical advice to programs embarking upon their inaugural self-study.

Development of a Standardized Kalamazoo Communication Skills Assessment Tool for Radiologists: Validation, Multisource Reliability, and Lessons Learned

OBJECTIVE

The purpose of this study was to develop and test a standardized communication skills assessment instrument for radiology.

MATERIALS AND METHODS

The Delphi method was used to validate the Kalamazoo Communication Skills Assessment instrument for radiology by revising and achieving consensus on the 43 items of the preexisting instrument among an interdisciplinary team of experts consisting of five radiologists and four nonradiologists (two men, seven women). Reviewers assessed the applicability of the instrument to evaluation of conversations between radiology trainees and trained actors portraying concerned parents in enactments about bad news, radiation risks, and diagnostic errors that were video recorded during a communication workshop. Interrater reliability was assessed by use of the revised instrument to rate a series of enactments between trainees and actors video recorded in a hospital-based simulator center. Eight raters evaluated each of seven different video-recorded interactions between physicians and parent-actors.

RESULTS

The final instrument contained 43 items. After three review rounds, 42 of 43 (98%) items had an average rating of relevant or very relevant for bad news conversations. All items were rated as relevant or very relevant for conversations about error disclosure and radiation risk. Reliability and rater agreement measures were moderate. The intraclass correlation coefficient range was 0.07-0.58; mean, 0.30; SD, 0.13; and median, 0.30. The range of weighted kappa values was 0.03-0.47; mean, 0.23; SD, 0.12; and median, 0.22. Ratings varied significantly among conversations (χ²₆ = 1186; p < 0.0001) and varied significantly by viewing order, rater type, and rater sex.

CONCLUSION

The adapted communication skills assessment instrument is highly relevant for radiology, having moderate interrater reliability. These findings have important implications for assessing the relational competencies of radiology trainees.

Voices emerging from the shadows: Radiologic practitioners' experiences of challenging conversations

OBJECTIVE

Traditionally, radiologists have practiced their profession behind the scenes. Today, radiologic practitioners face mounting expectations to communicate more directly with patients. However, their experiences with patient communication are not well understood. The aim of this study was to describe the challenges of radiologic practitioners when communicating with patients.

METHODS

Twelve day-long interprofessional communication skills workshops for radiologic clinicians were held at Boston Children's Hospital. Prior to each workshop, participants were asked to write narratives describing experiences with difficult radiologic conversations that they found particularly challenging or satisfying. The narratives were transcribed and analyzed through thematic content analysis by two researchers.

RESULTS

Radiologists, radiology trainees, technologists, nurses, and medical interpreters completed 92 narratives. The most challenging aspects of healthcare conversations included: Conveying Serious News (n=44/92; 48%); Expanded Scope of Radiologic Practice (n=37/92; 40%); Inexperience and Gaps in Education (n=15/92; 16%); Clinical Uncertainty (n=14/92; 15%); and Interprofessional Teamwork (n=9/92; 10%).

CONCLUSION

Radiologic clinicians face substantial communicative challenges focused on conveying serious, unexpected and uncertain diagnoses amid practical challenges and limited educational opportunities.

PRACTICE IMPLICATIONS

Innovative educational curricula that address these challenges may enhance radiologic practitioners' success in adopting patient-centered communication.

Genetic Counseling Milestones: A Framework for Student Competency Evaluation

Graduate medical education has recently increased focus on the development of medical specialty competency milestones to provide a targeted tool for medical resident evaluation. Milestones provide developmental assessment of the attainment of competencies over the course of an educational program. An educational framework is described to explore the development of Genetic Counseling Milestones for the evaluation of the development of genetic counseling competencies by genetic counseling students. The development of Genetic Counseling Milestones may provide a valuable tool to assess genetic counseling students across all program activities. Historical educational context, current practices, and potential benefits and challenges in the development of Genetic Counseling Milestones are discussed.

Milestones on a Shoestring: A Cost-Effective, Semi-automated Implementation of the New ACGME Requirements for Radiology

RATIONALE AND OBJECTIVES

The advent of the ACGME's Next Accreditation System represents a significant new challenge for residencies and fellowships, owing to its requirements for more complex and detailed information.

MATERIAL AND METHODS

We developed a system of online assessment tools to provide comprehensive coverage of the twelve ACGME Milestones and digitized them using freely available cloud-based productivity tools. These tools include a combination of point-of-care procedural assessments, electronic quizzes, online modules, and other data entry forms. Using free statistical analytic tools, we also developed an automated system for management, processing, and data reporting.

RESULTS

After one year of use, our Milestones project has resulted in the submission of over 20,000 individual data points. The use of automated statistical methods to generate resident-specific profiles has allowed for dynamic reports of individual residents' progress. These profiles both summarize data and also allow program directors access to more granular information as needed.

CONCLUSION

Informatics-driven strategies for data assessment and processing represent feasible solutions to Milestones assessment and analysis, reducing the potential administrative burden for program directors, residents, and staff.

Continuous Certification Within Residency: An Educational Model

Given that maintaining compliance with Maintenance of Certification is necessary for maintaining licensure to practice as a radiologist and provide quality patient care, it is important for radiology residents to practice fulfilling each part of the program during their training not only to prepare for success after graduation but also to adequately learn best practices from the beginning of their professional careers. This article discusses ways to implement continuous certification (called Continuous Residency Certification) as an educational model within the residency training program.

The use of ACR Appropriateness Criteria: a survey of radiology residents and program directors

PURPOSE

Assess the utilization of American College of Radiology Appropriateness Criteria (ACR-AC) among radiology residency program directors (PDs) and residents.

METHODS

Radiology PD and resident survey.

RESULTS

Seventy-four percent (46/62) of PDs promote ACR-AC in education (P<.05), and 84% (317/376) of residents have read at least a few (P<.05). Seventy-four percent (74/100) of first-year residents compared to 56.8% (157/276) of second- to fourth-year residents report at least occasional faculty reference of ACR-AC (P<.05). ACR-AC are well regarded (P<.05), but 40% believe that they are perplexing.

CONCLUSION

There is widespread resident awareness of ACR-AC and integration into resident training. However, faculty are only beginning to teach with them, and radiologists are not citing them with clinicians.