Analysis of Resident Case Logs in an Anesthesiology Residency Program

Competency-based anesthesiology teaching: comparison of programs in Brazil, Canada and the United States

In 2017, the Brazilian Society of Anesthesiology (SBA) and the National Medical Residency Committee (CNRM) presented a joint competence matrix to train and evaluate physicians specializing in Anesthesiology, which was enforced in 2019. The competency-based curriculum aims to train residents in relation to certain results, in that residents are considered capable when they are able to act in an appropriate and effective manner within certain standards of performance. Canada and the United States (US) also use competency-based curriculum to train their professionals. In Canada, the format is the basis for using an evaluation method known as Entrustable Professional Activities (EPA), in which the mentor assesses residents’ capacity to perform certain tasks, classified in 5 levels. The US, in turn, uses Milestones as evaluation, in which competencies and sub-competencies are assessed according to residents’ progress during training. The present article aims to describe and compare the different competency-based curriculum and the evaluation methods used in the three countries, and proposes a reflection on future paths for medical education in Anesthesiology in Brazil.

Assessment Scores of a Mock Objective Structured Clinical Examination Administered to 99 Anesthesiology Residents at 8 Institutions

BACKGROUND

Objective Structured Clinical Examinations (OSCEs) are used in a variety of high-stakes examinations. The primary goal of this study was to examine factors influencing the variability of assessment scores for mock OSCEs administered to senior anesthesiology residents.

METHODS

Using the American Board of Anesthesiology (ABA) OSCE Content Outline as a blueprint, scenarios were developed for 4 of the ABA skill types: (1) informed consent, (2) treatment options, (3) interpretation of echocardiograms, and (4) application of ultrasonography. Eight residency programs administered these 4 OSCEs to CA3 residents during a 1-day formative session. A global score and checklist items were used for scoring by faculty raters. We used a statistical framework called generalizability theory, or G-theory, to estimate the sources of variation (or facets), and to estimate the reliability (ie, reproducibility) of the OSCE performance scores. Reliability provides a metric on the consistency or reproducibility of learner performance as measured through the assessment.

RESULTS

Of the 115 total eligible senior residents, 99 participated in the OSCE because the other residents were unavailable. Overall, residents correctly performed 84% (standard deviation [SD] 16%, range 38%-100%) of the 36 total checklist items for the 4 OSCEs. On global scoring, the pass rate for the informed consent station was 71%, for treatment options was 97%, for interpretation of echocardiograms was 66%, and for application of ultrasound was 72%. The estimate of reliability expressing the reproducibility of examinee rankings equaled 0.56 (95% confidence interval [CI], 0.49-0.63), which is reasonable for normative assessments that aim to compare a resident's performance relative to other residents because over half of the observed variation in total scores is due to variation in examinee ability. Phi coefficient reliability of 0.42 (95% CI, 0.35-0.50) indicates that criterion-based judgments (eg, pass-fail status) cannot be made. Phi expresses the absolute consistency of a score and reflects how closely the assessment is likely to reproduce an examinee's final score. Overall, the greatest (14.6%) variance was due to the person by item by station interaction (3-way interaction) indicating that specific residents did well on some items but poorly on other items. The variance (11.2%) due to residency programs across case items was high suggesting moderate variability in performance from residents during the OSCEs among residency programs.

CONCLUSIONS

Since many residency programs aim to develop their own mock OSCEs, this study provides evidence that it is possible for programs to create a meaningful mock OSCE experience that is statistically reliable for separating resident performance.

Logging in: a comparative analysis of electronic health records versus anesthesia resident-driven logbooks

PURPOSE

Resident logbooks (RLBs) documenting clinical case exposure are widespread in medical education despite evidence of poor accuracy. Electronic health records (e.g., anesthesia information management systems [AIMS]) may provide advantages for auditing longitudinal case exposure. We evaluated the agreement between AIMS and RLBs for tracking case exposure during anesthesiology residency.

METHODS

We performed a historical cohort study with anesthesiology residents (2011-2018, all of whom used a RLB contemporaneously with AIMS) working in a multisite academic health sciences network. The primary outcome was total case-load logging; secondary outcomes were volumes for seven surgical specialties (general, gynecology, neuro, orthopedic, thoracic, urology, and vascular surgery). Correlation of case numbers tracked by AIMS vs RLB was assessed using Pearson correlation; agreement was determined using Bland-Altman plots and intraclass correlation coefficients (ICC).

RESULTS

Data from 27 anesthesiology residents were collected. Overall, mean (standard deviation) case numbers were generally greater with AIMS vs RLB [649 (103) vs 583 (191); P = 0.049). Total case volumes between systems had moderate correlation (r = 0.50) and agreement (intraclass correlation coefficient [ICC], 0.42; 95% CI, 0.34 to 0.59). Bland-Altman plots showed variable agreement between AIMS and RLB data [mean (SD) bias = 66 (166) cases]. For general, gynecology, neuro, orthopedic, thoracic, urology, and vascular surgery, there was a range of poor to moderate agreement (ICC, 0.23-0.57) between AIMS and RLB.

CONCLUSION

For anesthesiology resident case-logging, the number of cases logged in an AIMS was higher with lower variance compared with RLBs. Anesthesia information management systems vs RLB data showed low-moderate correlation and agreement. Given the additional time and resources required for RLBs, AIMS may be a superior method for tracking cases where available.

Comparing Anesthesiology Residency Training Structure and Requirements in Seven Different Countries on Three Continents

Little has been published comparing the graduate medical education training structure and requirements across multiple countries. The goal of this study was to summarize and compare the characteristics of anesthesiology training programs in the USA, UK, Canada, Japan, Brazil, Denmark, and Switzerland as a way to better understand efforts to train anesthesiologists in different countries.

Two physicians trained in each of the seven countries (convenience sample) were interviewed using a semi-structured approach. The interview was facilitated by use of a predetermined questionnaire that included, for example, the duration of post-medical school training and national requirements for certain rotations, a number of cases, faculty supervision, national in-training written exams, and duty hour limits. These data were augmented by review of each country’s publicly available residency training documents as available on the internet.

Post-medical school anesthesia residency duration varied: three years (Brazil), four years (USA), five years (Canada and Switzerland), six years (Japan and Denmark) to nine years (UK), as did the number of explicitly required clinical rotations of a defined duration: zero (Denmark), one (Switzerland and UK), four (Brazil), six (Canada), and 12 (USA). Minimum case requirements exist in the USA, Japan, and Brazil, but not in the other countries. National written exams taken during training exist for all countries studied except Japan and Denmark. The countries studied increasingly aim to have competency-based education with milestone assessments. Training duty hour limits also varied including for example 37 hours/week averaged over a one month with limitations on night duties (Denmark), a weekly average of 48 hours taken over a 17 week period (UK), 50 hours/week maximum (Switzerland), 60 hours/week maximum (Brazil), and 80 hours/week averaged over four weeks (USA).

Some countries have highly structured training programs with multiple national requirements with training principally carried out at a home institution. Other countries have a more decentralized and unregulated approach with fewer (if any) specific case or rotation requirements, where the trainee creates his/her own customized training to meet broad objectives and goals. The countries studied have different national training requirements, unique duty hour rules and are at varying stages in transitioning to an outcome based model of residency.