Developing Teaching Strategies in the EHR Era: A Survey of GME Experts

Development and Validation of a Machine Learning Model for Automated Assessment of Resident Clinical Reasoning Documentation

BACKGROUND

Residents receive infrequent feedback on their clinical reasoning (CR) documentation. While machine learning (ML) and natural language processing (NLP) have been used to assess CR documentation in standardized cases, no studies have described similar use in the clinical environment.

OBJECTIVE

The authors developed and validated using Kane's framework a ML model for automated assessment of CR documentation quality in residents' admission notes.

DESIGN, PARTICIPANTS, MAIN MEASURES

Internal medicine residents' and subspecialty fellows' admission notes at one medical center from July 2014 to March 2020 were extracted from the electronic health record. Using a validated CR documentation rubric, the authors rated 414 notes for the ML development dataset. Notes were truncated to isolate the relevant portion; an NLP software (cTAKES) extracted disease/disorder named entities and human review generated CR terms. The final model had three input variables and classified notes as demonstrating low- or high-quality CR documentation. The ML model was applied to a retrospective dataset (9591 notes) for human validation and data analysis. Reliability between human and ML ratings was assessed on 205 of these notes with Cohen's kappa. CR documentation quality by post-graduate year (PGY) was evaluated by the Mantel-Haenszel test of trend.

KEY RESULTS

The top-performing logistic regression model had an area under the receiver operating characteristic curve of 0.88, a positive predictive value of 0.68, and an accuracy of 0.79. Cohen's kappa was 0.67. Of the 9591 notes, 31.1% demonstrated high-quality CR documentation; quality increased from 27.0% (PGY1) to 31.0% (PGY2) to 39.0% (PGY3) (p < .001 for trend). Validity evidence was collected in each domain of Kane's framework (scoring, generalization, extrapolation, and implications).

CONCLUSIONS

The authors developed and validated a high-performing ML model that classifies CR documentation quality in resident admission notes in the clinical environment-a novel application of ML and NLP with many potential use cases.

Development of a Clinical Reasoning Documentation Assessment Tool for Resident and Fellow Admission Notes: a Shared Mental Model for Feedback

BACKGROUND

Residents and fellows receive little feedback on their clinical reasoning documentation. Barriers include lack of a shared mental model and variability in the reliability and validity of existing assessment tools. Of the existing tools, the IDEA assessment tool includes a robust assessment of clinical reasoning documentation focusing on four elements (interpretive summary, differential diagnosis, explanation of reasoning for lead and alternative diagnoses) but lacks descriptive anchors threatening its reliability.

OBJECTIVE

Our goal was to develop a valid and reliable assessment tool for clinical reasoning documentation building off the IDEA assessment tool.

DESIGN, PARTICIPANTS, AND MAIN MEASURES

The Revised-IDEA assessment tool was developed by four clinician educators through iterative review of admission notes written by medicine residents and fellows and subsequently piloted with additional faculty to ensure response process validity. A random sample of 252 notes from July 2014 to June 2017 written by 30 trainees across several chief complaints was rated. Three raters rated 20% of the notes to demonstrate internal structure validity. A quality cut-off score was determined using Hofstee standard setting.

KEY RESULTS

The Revised-IDEA assessment tool includes the same four domains as the IDEA assessment tool with more detailed descriptive prompts, new Likert scale anchors, and a score range of 0-10. Intraclass correlation was high for the notes rated by three raters, 0.84 (95% CI 0.74-0.90). Scores ≥6 were determined to demonstrate high-quality clinical reasoning documentation. Only 53% of notes (134/252) were high-quality.

CONCLUSIONS

The Revised-IDEA assessment tool is reliable and easy to use for feedback on clinical reasoning documentation in resident and fellow admission notes with descriptive anchors that facilitate a shared mental model for feedback.

First Characterization of Resident Clinical Experience at American Urological Training Programs

OBJECTIVE

To provide the first nationwide characterization of the clinical learning environment in American urological training programs.

METHODS

A survey was sent to program directors (PD) at AUA-accredited urological training programs after requesting their email address from each program coordinator (PC). The 21-question survey was designed to ascertain key components of each training environment: demographics, training model, clinic structure, and resident perception.

RESULTS

The PC of 131 AUA-accredited training programs received an email for participation, yielding the PD email for 113 programs. 60/113 (53%) PDs responded to the survey. Residents participated in clinic at the following types of hospitals: Children's 51 (85%), County/Indigent 23 (38%), Private 29 (48%), University 56 (93%), Veterans Administration 38 (63%). Prevalence of clinical training models is presented in table 1. On average, PDs estimated their residents spend 2.6 half days in clinic each week (1-6). 13 (22%) programs reported a "clinic only" rotation, varying from 1-6 months total. PDs reported time constraint and schedule to be the biggest barrier to teaching in clinic and 40% felt residents see clinic as a valuable part of their training while 30% felt residents see clinic as a necessary exercise but with limitations to learning opportunities.

CONCLUSIONS

We present the first characterization of resident participation in the clinical learning environment. Structure is highly variable and directed effort is necessary to move towards improved assessment and monitoring of resident competency in clinic.

Navigating Through Electronic Health Records: Survey Study on Medical Students' Perspectives in General and With Regard to a Specific Training

Background

An electronic health record (EHR) is the state-of-the-art method for ensuring all data concerning a given patient are up to date for use by multidisciplinary hospital teams. Therefore, medical students need to be trained to use health information technologies within this environment from the early stages of their education.

Objective

As little is known about the effects of specific training within the medical curriculum, this study aimed to develop a course module and evaluate it to offer best practice teaching for today’s students. Moreover, we looked at the acceptance of new technologies such as EHRs.

Methods

Fifth-year medical students (N=104) at the University of Tübingen took part in a standardized two-day training procedure about the advantages and risks of EHR use. After the training, students performed their own EHR entries on hypothetical patient cases in a safe practice environment. In addition, questionnaires—standardized and with open-ended questions—were administered to assess students’ experiences with a new teaching module, a newly developed EHR simulator, the acceptance of the health technology, and their attitudes toward it before and after training.

Results

After the teaching, students rated the benefit of EHR training for medical knowledge significantly higher than before the session (mean 3.74, SD 1.05). However, they also had doubts about the long-term benefit of EHRs for multidisciplinary coworking after training (mean 1.96, SD 0.65). The special training with simulation software was rated as helpful for preparing students (88/102, 86.2%), but they still did not feel safe in all aspects of EHR.

Conclusions

A specific simulated training on using EHRs helped students improve their knowledge and become more aware of the risks and challenges of such a system. Overall, students welcomed the new training module and supported the integration of EHR teaching into the medical curriculum. Further studies are needed to optimize training modules and make use of long-term feedback opportunities a simulated system offers.

Documenting or Operating: Where Is Time Spent in General Surgery Residency?

OBJECTIVE

The utilization of electronic health records (EHR) has become essential in the daily activities of physicians for documentation and as an information source. However, the amount of time spent by residents utilizing the EHR has not been thoroughly evaluated, particularly within surgical specialties. This study aims to analyze EHR usage by general surgery residents and to assess the association between this use and case volume at a single academic institution.

DESIGN

For general surgery residents in clinical years (CY) 1-5, de-identified login and logout time data between September 2016 and June 2017 were retrospectively extracted from the Epic EHR (Verona, WI). A binary time series was created for each resident to indicate and track over time whether he or she was utilizing the EHR system. Comparisons between categorical variables were performed with Fisher's exact test. Continuous variables were compared using Wilcoxon rank sum test. Longitudinal linear mixed-effects models were used to assess the EHR usage among the surgery residents. The association between EHR time and the number of operative cases logged was evaluated with Pearson's correlation coefficient.

SETTING

This study was performed by the Department of Surgery in conjunction with the Office of Graduate Medical Education at Duke University Health System.

PARTICIPANTS

All active general surgery residents during the 2016-2017 academic year.

RESULTS

Thirty-six general surgery residents (28 males, 8 females) spent a median of 2.4 hours per day and 23.7 hours per week using the EHR. CY2 had the highest median usage per week (28.9 hours), while CY3 had the lowest (16.7 hours) but no significant difference based on EHR usage was found among the analyzed CYs (p = 0.164). Residents spent significantly more time logged into the EHR during the week compared to weekends and during the day compared to nights (all p < 0.001). For the residency program as a whole, a median of 151.5 total work hours per day was dedicated to documentation. On average, interns on dedicated night rotations spent 7% of their login time outside regularly scheduled duty hours while interns on dedicated day rotations spent 27%. There was no overall correlation between monthly case logs and EHR usage (r = 0.06, p = 0.30); however, CY2 had a significant negative correlation (r = -0.2, p = 0.038).

CONCLUSIONS

In the era of a maximum 80-hour work week, general surgery residents spend a substantial portion, at least 30%, of their time utilizing the EHR. One third of EHR usage by interns occurred outside the scheduled 12-hour shift, demonstrating the difficulties of completing paperwork as part of the scheduled work day. Additionally, the lack of correlation to case logs is likely due to an underestimation of the documentation burden associated with operating, which includes preparatory effort and operative notes. Ultimately, these quantitative EHR usage results will be correlated to burnout prior to implementing programs to improve efficiency and decrease the burden of charting.

Open Notes in Teaching Clinics: A Multisite Survey of Residents to Identify Anticipated Attitudes and Guidance for Programs

BACKGROUND

Clinicians are increasingly sharing outpatient visit notes with patients through electronic portals. These open notes may bring about new educational opportunities as well as concerns to physicians-in-training and residency programs.

OBJECTIVE

We assessed anticipatory attitudes about open notes and explored factors influencing residents' propensity toward note transparency.

METHODS

Residents in primary care clinics at 4 teaching hospitals were surveyed prior to implementation of open notes. Main measures included resident attitudes toward open notes and the anticipated effect on patients, resident workload, and education. Data were stratified by site.

RESULTS

A total of 176 of 418 (42%) residents responded. Most residents indicated open notes would improve patient engagement, trust, and education but worried about overwhelming patients, residents being less candid, and workload. More than half of residents thought open notes were a good idea, and 32% (56 of 176) indicated they would encourage patients to read these notes. More than half wanted note-writing education and more feedback, and 72% (126 of 175) indicated patient feedback on residents' notes could improve communication skills. Attitudes about effects of open notes on safety, quality, trust, and medical education varied by site.

CONCLUSIONS

Residents reported mixed feelings about the anticipated effects of sharing clinical notes with patients. They advocate for patient feedback on notes, yet worry about workload, supervision, and errors. Training site was correlated with many attitudes, suggesting local culture drives resident support for open notes. Strategies that address resident concerns and promote teaching and feedback related to notes may be helpful.

Collaborating for Competency-A Model for Single Electronic Health Record Onboarding for Medical Students Rotating among Separate Health Systems

BACKGROUND

Use of the electronic health record (EHR) is widespread in academic medical centers, and hands-on EHR experience in medical school is essential for new residents to be able to meaningfully contribute to patient care. As system-specific EHR training is not portable across institutions-even when the same EHR platform is used-students rotating across health systems are often required to spend time away from clinical training to complete each system's, often duplicative, EHR training regardless of their competency within the EHR.

METHODS

We aimed to create a single competency-based Epic onboarding process that would be portable across all the institutions in which our medical students complete clinical rotations. In collaboration with six health systems, we created online EHR training modules using a systematic approach to curriculum development and created an assessment within the Epic practice environment.

RESULTS

All six collaborating health systems accepted successful completion of the developed assessment in lieu of standard site-specific medical student EHR training. In the pilot year, 443 students (94%) completed the modules and assessment prior to their clinical training and successfully entered clinical rotations without time consuming, often repetitive onsite training, decreasing the cumulative time as student might be expected to engage in Epic onboarding as much as 20-fold.

CONCLUSION

Medical schools with multisystem training sites with a single type of EHR can adopt this approach to minimize training burden for their learners and to allow them more time in the clinical setting with optimized access to the EHR.

A New EHR Training Curriculum and Assessment for Pediatric Residents

Background Conventional classroom Electronic Health Record (EHR) training is often insufficient for new EHR users. Studies suggest that enhanced training with a hands-on approach and closely supported clinical use is beneficial. Objectives Our goals were to develop an enhanced EHR learning curriculum for Post Graduate Year 1 (PGY1) residents and measure changes in EHR skill proficiency, efficiency, and self-efficacy. Methods A novel three-phase, multimodal enhanced EHR curriculum was designed for a cohort of PGY1 residents. After basic training, residents began phase 1 of enhanced training, including demonstrations, live practice, and order set review. Phase 2 involved skills-oriented assignments, role playing, and medication entry. Phase 3 included shadowing, scribing histories, and supervised order entry. Residents' EHR skills and attitudes were measured and compared before and after the enhanced curriculum via proficiency test and a survey of efficiency and self-efficacy. Results Nineteen of 26 PGY1 residents participated in the study (73%). There was significant improvement in mean proficiency scores and two of the five individual proficiency scores. There were significant improvements in most efficiency survey responses from pre- to postintervention. For the self-efficacy presurvey, many PGY1s reported to be “very” or “somewhat confident” performing each of the five tasks, and perceptions did not improve or worsened on most postsurvey responses. The greatest resource was the time required to design and deliver the enhanced training. Conclusion An enhanced training curriculum along with a proficiency assessment was developed and described here. An enhanced training curriculum significantly improved PGY1 EHR efficiency and some measures of proficiency but not self-efficacy. This intervention may support improved EHR-related clinic workflows, which ultimately could enable residents and preceptors to prioritize patient care and time for clinical education.