Residency Training The neurology resident case log: A national survey of neurology residents

Measuring Training Disruptions Using an Informatics Based Tool

OBJECTIVE

Training disruptions, such as planned curricular adjustments or unplanned global pandemics, impact residency training in ways that are difficult to quantify. Informatics-based medical education tools can help measure these impacts. We tested the ability of a software platform driven by electronic health record data to quantify anticipated changes in trainee clinical experiences during the COVID-19 pandemic.

METHODS

We previously developed and validated the Trainee Individualized Learning System (TRAILS) to identify pediatric resident clinical experiences (i.e. shifts, resident provider-patient interactions (rPPIs), and diagnoses). We used TRAILS to perform a year-over-year analysis comparing pediatrics residents at a large academic children's hospital during March 15-June 15 in 2018 (Control #1), 2019 (Control #2), and 2020 (Exposure).

RESULTS

Residents in the exposure cohort had fewer shifts than those in both control cohorts (P < .05). rPPIs decreased an average of 43% across all PGY levels, with interns experiencing a 78% decrease in Continuity Clinic. Patient continuity decreased from 23% to 11%. rPPIs with common clinic and emergency department diagnoses decreased substantially during the exposure period.

CONCLUSIONS

Informatics tools like TRAILS may help program directors understand the impact of training disruptions on resident clinical experiences and target interventions to learners' needs and development.

Implementation and Evaluation of Integrating an Electronic Health Record With the ACGME Case Log System

BACKGROUND

It is essential to log resident-performed procedures to assess training programs and fulfill specialty requirements, but resident case numbers are often underreported. Current systems require inefficient data entry steps, and residents and fellows report that user interfaces and administrative burden contribute to logging inaccuracy.

OBJECTIVE

To determine the accuracy, feasibility, and acceptability of a single logging approach for resident case logging.

METHODS

In 2018, we implemented a case logging system integrated with the institutional electronic health record (EHR) and the Accreditation Council for Graduate Medical Education (ACGME) case log system to record procedures performed by ophthalmology residents. We compared the proportion of resident-performed cataract extractions in the EHR that were reported to ACGME for 3 periods: before the deployment of the new system (6 months), during the transition (6 months), and after the change (2 years). Resident satisfaction with the new system was evaluated using surveys.

RESULTS

An analysis of resident cataract surgeries showed that the percentage of resident cases logged increased from 85% prior to implementation to 91% after implementation. The integrated system became the preferred case logging method, with 100% of all logged cases being entered using the new platform. Surveys showed that the percentage of trainees who were moderately or very satisfied with the case log process increased from 55% before implementation to 100% after implementation.

CONCLUSIONS

A resident case log system integrated with an EHR more accurately reflects resident operative volume and increases trainee satisfaction with the logging process.

The ACGME Case Log System May Not Accurately Represent Operative Experience Among General Surgery Interns

OBJECTIVE

To assess if the Accreditation Council for Graduate Medical Education (ACGME) case log system accurately captures operative experience of our postgraduate year 1 (PGY-1) residents.

DESIGN

ACGME case log information was retrospectively obtained for 5 cohorts of PGY-1 residents (2011-2015) and compared to the number of operative cases captured by an institutional automated operative case report system, Surgical Access Utility System (SAUS). SAUS automatically captures all surgical team members who are listed in the operative dictation for a given case, including interns. A paired t-test analysis was used to compare number of cases coded between the 2 systems.

SETTING

Academic, tertiary care referral center with a large general surgery training program.

PARTICIPANTS

PGY-1 general surgery trainees (interns) from the years 2011-2015.

RESULTS

Forty-nine PGY-1 general surgery residents were identified over a 5-year period. Mean operative case volume per intern, per year, captured by the automated SAUS was 176.5 ± 28.1 (SD) compared to 126.3 ± 58.0 ACGME cases logged (mean difference = 50.2 cases, p < 0.001).

CONCLUSIONS

ACGME case log data may not accurately reflect the actual operative experience of our PGY-1 residents. If such data holds true for other general surgery training programs, the true impact of duty hour regulations on operative volume may be unclear when using the ACGME case log data. This current standard approach for using ACGME case logs as a representation of operative experience requires further scrutiny and potential revision to more accurately determine operative experience for accreditation purposes.

Assessing the accuracy of neurological surgery resident case logs at a single institution

BACKGROUND

Despite the importance of case logs in evaluating residents, no studies assess their accuracy in neurological surgery. Studies from other specialties reveal variations in reporting. This study assesses the accuracy of neurological surgery resident case logs at a single institution.

METHODS

Data was collected from three databases: billing data and two separate resident-managed case logs [department log and Accreditation Council for Graduate Medical Education (ACGME) case logs], containing records of procedures performed by 14 neurological surgery residents at a single institution over a 1-year period. The billing data was used as a proxy for a census of procedures performed during the study period. The difference between the number of procedures logged by residents and the number of procedures billed was calculated to determine the accuracy of the resident case logs.

RESULTS

Over the study period, 2150 procedures were billed at the institution, whereas 1749 procedures were logged in the ACGME case log and 1873 in the department log, representing an error rate of -18.65% and -12.88%, respectively. The error rate varied significantly (-1150% to +50.23%) between ACGME procedure categories. In 13 of the 22 ACGME procedure categories, the procedures were under-logged by residents in both resident-managed case logs. No category demonstrated over-logging in both case log systems.

CONCLUSION

Resident managed case logs are an incomplete representation of clinical work. The cause for inaccuracy is multifactorial. The authors suggested that further research is necessary to validate their results and to identify means by which the accuracy of case logs can be increased.

Case-Logging Practices in Otolaryngology Residency Training: National Survey of Residents and Program Directors

Objective (1) Evaluate the consistency and manner in which otolaryngology residents log surgical cases. (2) Assess the extent of instruction and guidance provided by program directors on case-logging practices. Study Design Cross-sectional national survey. Setting Accreditation Council for Graduate Medical Education otolaryngology residency programs in the United States. Subjects and Methods US otolaryngology residents, postgraduate year 2 through graduating chiefs as of July 2016, were recruited to respond to an anonymous questionnaire designed to characterize surgical case-logging practices. Program directors of US otolaryngology residency programs were recruited to respond to an anonymous questionnaire to elucidate how residents are instructed to log cases. Results A total of 272 residents and 53 program directors completed the survey, yielding response rates of 40.6% and 49.5%, respectively. Perceived accuracy of case logs is low among residents and program directors. Nearly 40% of residents purposely choose not to log certain cases, and 65.1% of residents underreport cases performed. More than 80% of program directors advise residents to log procedures performed outside the operating room, yet only 16% of residents consistently log such cases. Conclusion Variability in surgical case-logging behaviors and differences in provided instruction highlight the need for methods to improve consistency of logging practices. It is imperative to standardize practices across otolaryngology residency programs for case logs to serve as an accurate measure of surgical competency. This study provides a foundation for reform efforts within residency programs and for the Resident Case Log System.

Evaluating Coding Accuracy in General Surgery Residents' Accreditation Council for Graduate Medical Education Procedural Case Logs

INTRODUCTION

The Accreditation Council for Graduate Medical Education (ACGME) case log captures resident operative experience based on Current Procedural Terminology (CPT) codes and is used to track operative experience during residency. With increasing emphasis on resident operative experiences, coding is more important than ever. It has been shown in other surgical specialties at similar institutions that the residents' ACGME case log may not accurately reflect their operative experience. What barriers may influence this remains unclear. As the only objective measure of resident operative experience, an accurate case log is paramount in representing one's operative experience. This study aims to determine the accuracy of procedural coding by general surgical residents at a single institution.

METHODS

Data were collected from 2 consecutive graduating classes of surgical residents' ACGME case logs from 2008 to 2014. A total of 5799 entries from 7 residents were collected. The CPT codes entered by residents were compared to departmental billing records submitted by the attending surgeon for each procedure. Assigned CPT codes by institutional American Academy of Professional Coders certified abstract coders were considered the "gold standard." A total of 4356 (75.12%) of 5799 entries were identified in billing records. Excel 2010 and SAS 9.3 were used for analysis. In the event of multiple codes for the same patient, any match between resident codes and billing record codes was considered a "correct" entry. A 4-question survey was distributed to all current general surgical residents at our institution for feedback on coding habits, limitations to accurate coding, and opinions on ACGME case log representation of their operative experience.

RESULTS

All 7 residents had a low percentage of correctly entered CPT codes. The overall accuracy proportion for all residents was 52.82% (range: 43.32%-60.07%). Only 1 resident showed significant improvement in accuracy during his/her training (p = 0.0043). The survey response rate was 100%. Survey results indicated that inability to find the precise code within the ACGME search interface and unfamiliarity with available CPT codes were by far the most common perceived barriers to accuracy. Survey results also indicated that most residents (74%) believe that they code accurately most of the time and agree that their case log would accurately represent their operative experience (66.6%).

CONCLUSION

This is the first study to evaluate correctness of residents' ACGME case logs in general surgery. The degree of inaccuracy found here necessitates further investigation into the etiology of these discrepancies. Instruction on coding practices should also benefit the residents after graduation. Optimizing communication among attendings and residents, improving ACGME coding search interface, and implementing consistent coding practices could improve accuracy giving a more realistic view of residents' operative experience.

General Surgery Resident Case Logs: Do They Accurately Reflect Resident Experience?

INTRODUCTION

Operative experience during residency lays the foundation for independent practice and additional specialty training following general surgery residency. The aim of this study was to examine operative experience of general surgery residents and detail the results of an intervention aimed at improving resident record keeping in the Accreditation Council for Graduate Medical Education (ACGME) case log system to better reflect their experience.

METHODS

Residents were asked to characterize variances in recorded operative experience identified through an audit of operative logs. Based on the results of the audit, an intervention was designed to prompt timely record keeping by residents. The intervention included education and discussion of survey audit results, weekly presentation of graphs detailing operative experience, and possible missed cases in the ACGME logs and addition of a first assistant column in morbidity and mortality (M&M) logs.

RESULTS

The audit of case logs identified discrepancies in 24.2% of the 636 cases examined. Chief residents were significantly more accurate (95.9%) in recording operative experience in ACGME case logs, whereas 50.3% of junior resident case logs contained variances. Residents characterized discrepancies as "forgot to log" (9.6%), "staff did the case" (5.2%), "another resident did more of the case" (3.6%), "other" (3.6%), a "more advanced resident was present for the case" (1.6%), "not present for case" (0.6%), and "left for consult" (0.3%). Over the 4-week intervention period, residents logged between 72.7% and 94.0% of cases. A month following the intervention period, we observed a 13.3% increase in recorded cases compared with the intervention period. Review of first assistant case logging following inclusion of a "first assistant" column in M&M logs demonstrated a 70.5% increase in first assistant cases logged into the ACGME system compared with the same time period a year ago.

CONCLUSIONS

Based on our results, we found that weekly displays of cases improved resident record keeping in the ACGME case log system, especially by junior residents. We believe that the addition of first assistant column on M&M lists, periodic audits reviewed at conferences, and semiannual evaluations will help junior residents more accurately report their experience during training.

Variability in Accreditation Council for Graduate Medical Education Resident Case Log System practices among orthopaedic surgery residents

BACKGROUND

The Accreditation Council for Graduate Medical Education (ACGME) Resident Case Log System is designed to be a reflection of residents' operative volume and an objective measure of their surgical experience. All operative procedures and manipulations in the operating room, Emergency Department, and outpatient clinic are to be logged into the Resident Case Log System. Discrepancies in the log volumes between residents and residency programs often prompt scrutiny. However, it remains unclear if such disparities truly represent differences in operative experiences or if they are reflections of inconsistent logging practices. The purpose of this study was to investigate individual recording practices among orthopaedic surgery residents prior to August 1, 2011.

METHODS

Orthopaedic surgery residents received a questionnaire on case log practices that was distributed through the Council of Orthopaedic Residency Directors list server. Respondents were asked to respond anonymously about recording practices in different clinical settings as well as types of cases routinely logged. Hypothetical scenarios of common orthopaedic procedures were presented to investigate the differences in the Current Procedural Terminology codes utilized.

RESULTS

Two hundred and ninety-eight orthopaedic surgery residents completed the questionnaire; 37% were fifth-year residents, 22% were fourth-year residents, 18% were third-year residents, 15% were second-year residents, and 8% were first-year residents. Fifty-six percent of respondents reported routinely logging procedures performed in the Emergency Department or urgent care setting. Twenty-two percent of participants routinely logged procedures in the clinic or outpatient setting, 20% logged joint injections, and only 13% logged casts or splints applied in the office setting. There was substantial variability in the Current Procedural Terminology codes selected for the seven clinical scenarios.

CONCLUSIONS

There has been a lack of standardization in case-logging practices among orthopaedic surgery residents prior to August 1, 2011. ACGME case log data prior to this date may not be a reliable measure of residents' procedural experience.

Education research: neurology training reassessed. The 2011 American Academy of Neurology Resident Survey results

OBJECTIVE

To assess the strengths and weaknesses of neurology resident education using survey methodology.

METHODS

A 27-question survey was sent to all neurology residents completing residency training in the United States in 2011.

RESULTS

Of eligible respondents, 49.8% of residents returned the survey. Most residents believed previously instituted duty hour restrictions had a positive impact on resident quality of life without impacting patient care. Most residents rated their faculty and clinical didactics favorably. However, many residents reported suboptimal preparation in basic neuroscience and practice management issues. Most residents (71%) noted that the Residency In-service Training Examination (RITE) assisted in self-study. A minority of residents (14%) reported that the RITE scores were used for reasons other than self-study. The vast majority (86%) of residents will enter fellowship training following residency and were satisfied with the fellowship offers they received.

CONCLUSIONS

Graduating residents had largely favorable neurology training experiences. Several common deficiencies include education in basic neuroscience and clinical practice management. Importantly, prior changes to duty hours did not negatively affect the resident perception of neurology residency training.

The more things change the more they stay the same: a case report of neurology residency experiences

This study compared the neurology residency training experience for a single neurology resident at the University of Pennsylvania from the years 2002-2005. The prevalence of encounters seen during this residency was compared to the prevalence of neurological disorders typically observed by ambulatory neurologists in the United States (US). A total of 1,333 patients were evaluated during this residency. Ischemic stroke/transient ischemic accident, epilepsy, metabolic encephalopathy, peripheral neuropathy, and multiple sclerosis were the most common neurological disorders observed. The four most common reasons for an outpatient visit to a neurologist (i.e., headache/migraine, epilepsy, cerebrovascular disease, and peripheral neuropathy) typically account for approximately 49-55% of all appointments, but only contributed to approximately 40% of patient encounters during this neurology residency. While these results reflect the encounters of a single neurology resident, both the total number and distribution of neurological diagnoses were similar to previous experiences over two decades ago at US academic medical centers despite significant changes in health care delivery and policy. This case report demonstrates that neurology residency programs continue to overemphasize acute management of inpatient neurological disorders compared to outpatient care of more prevalent neurological complaints. Additional measures could be instituted to ensure a broader range of experiences during residency (i.e., online resident log). These methods could allow residency coordinators to identify certain areas of deficiency with regards to exposure to patients for a resident and ensure greater competency during residency.

Automated data mining: an innovative and efficient web-based approach to maintaining resident case logs

BACKGROUND

Use of resident case logs has been considered by the Residency Review Committee for Neurology of the Accreditation Council for Graduate Medical Education (ACGME).

OBJECTIVE

This study explores the effectiveness of a data-mining program for creating resident logs and compares the results to a manual data-entry system. Other potential applications of data mining to enhancing resident education are also explored.

DESIGN/METHODS

Patient notes dictated by residents were extracted from the Hospital Information System and analyzed using an unstructured mining program. History, examination and ICD codes were obtained and compared to the existing manual log. The automated data History, examination, and ICD codes were gathered for a 30-day period and compared to manual case logs.

RESULTS

The automated method extracted all resident dictations with the dates of encounter and transcription. The automated data-miner processed information from all 19 residents, while only 4 residents logged manually. The manual method identified only broad categories of diseases; the major categories were stroke or vascular disorder 53 (27.6%), epilepsy 28 (14.7%), and pain syndromes 26 (13.5%). In the automated method, epilepsy 114 (21.1%), cerebral atherosclerosis 114 (21.1%), and headache 105 (19.4%) were the most frequent primary diagnoses, and headache 89 (16.5%), seizures 94 (17.4%), and low back pain 47 (9%) were the most common chief complaints. More detailed patient information such as tobacco use 227 (42%), alcohol use 205 (38%), and drug use 38 (7%) were extracted by the data-mining method.

CONCLUSIONS

Manual case logs are time-consuming, provide limited information, and may be unpopular with residents. Data mining is a time-effective tool that may aid in the assessment of resident experience or the ACGME core competencies or in resident clinical research. More study of this method in larger numbers of residency programs is needed.