Pediatric fractures - an educational needs assessment of Canadian pediatric emergency medicine residents

A Target Population Derived Method for Developing a Competency Standard in Radiograph Interpretation

CONSTRUCT

For assessing the skill of visual diagnosis such as radiograph interpretation, competency standards are often developed in an ad hoc method, with a poorly delineated connection to the target clinical population.

BACKGROUND

Commonly used methods to assess for competency in radiograph interpretation are subjective and potentially biased due to a small sample size of cases, subjective evaluations, or include an expert-generated case-mix versus a representative sample from the clinical field. Further, while digital platforms are available to assess radiograph interpretation skill against an objective standard, they have not adopted a data-driven competency standard which informs educators and the public that a physician has achieved adequate mastery to enter practice where they will be making high-stakes clinical decisions.

APPROACH

Operating on a purposeful sample of radiographs drawn from the clinical domain, we adapted the Ebel Method, an established standard setting method, to ascertain a defensible, clinically relevant mastery learning competency standard for the skill of radiograph interpretation as a model for deriving competency thresholds in visual diagnosis. Using a previously established digital platform, emergency physicians interpreted pediatric musculoskeletal extremity radiographs. Using one-parameter item response theory, these data were used to categorize radiographs by interpretation difficulty terciles (i.e. easy, intermediate, hard). A panel of emergency physicians, orthopedic surgeons, and plastic surgeons rated each radiograph with respect to clinical significance (low, medium, high). These data were then used to create a three-by-three matrix where radiographic diagnoses were categorized by interpretation difficulty and significance. Subsequently, a multidisciplinary panel that included medical and parent stakeholders determined acceptable accuracy for each of the nine cells. An overall competency standard was derived from the weighted sum. Finally, to examine consequences of implementing this standard, we reported on the types of diagnostic errors that may occur by adhering to the derived competency standard.

FINDINGS

To determine radiograph interpretation difficulty scores, 244 emergency physicians interpreted 1,835 pediatric musculoskeletal extremity radiographs. Analyses of these data demonstrated that the median interpretation difficulty rating of the radiographs was -1.8 logits (IQR -4.1, 3.2), with a significant difference of difficulty across body regions (p < 0.0001). Physician review classified the radiographs as 1,055 (57.8%) as low, 424 (23.1%) medium or 356 (19.1%) high clinical significance. The multidisciplinary panel suggested a range of acceptable scores between cells in the three-by-three table of 76% to 95% and the sum of equal-weighted scores resulted in an overall performance-based competency score of 85.5% accuracy. Of the 14.5% diagnostic interpretation errors that may occur at the bedside if this competency standard were implemented, 9.8% would be in radiographs of low-clinical significance, while 2.5% and 2.3% would be in radiographs of medium or high clinical significance, respectively.

CONCLUSION(S)

This study's novel integration of radiograph selection and a standard setting method could be used to empirically drive evidence-based competency standard for radiograph interpretation and can serve as a model for deriving competency thresholds for clinical tasks emphasizing visual diagnosis.

A think-aloud study to inform the design of radiograph interpretation practice

Models for diagnostic reasoning in radiology have been based on the observed behaviors of experienced radiologists but have not directly focused on the thought processes of novices as they improve their accuracy of image interpretation. By collecting think-aloud verbal reports, the current study was designed to investigate differences in specific thought processes between medical students (novices) as they learn and radiologists (experts), so that we can better design future instructional environments. Seven medical students and four physicians with radiology training were asked to interpret and diagnose pediatric elbow radiographs where fracture is suspected. After reporting their diagnosis of a case, they were given immediate feedback. Participants were asked to verbalize their thoughts while completing the diagnosis and while they reflected on the provided feedback. The protocol analysis of their verbalizations showed that participants used some combination of four processes to interpret the case: gestalt interpretation, purposeful search, rule application, and reasoning from a prior case. All types of processes except reasoning from a prior case were applied significantly more frequently by experts. Further, gestalt interpretation was used with higher frequency in abnormal cases while purposeful search was used more often for normal cases. Our assessment of processes could help guide the design of instructional environments with well-curated image banks and analytics to facilitate the novice's journey to expertise in image interpretation.

Adverse Events from Emergency Physician Pediatric Extremity Radiograph Interpretations: A Prospective Cohort Study

OBJECTIVES

We determined how often emergency physician pediatric musculoskeletal (MSK) radiograph interpretations were discordant to that of a radiologist and led to an adverse event (AE). We also established the variables independently associated with this outcome.

METHODS

This prospective cohort study was conducted in an urban, tertiary care children's emergency department (ED). We enrolled children who presented to an ED with an extremity injury and received radiographs. ED physicians documented their radiograph interpretation, which was compared to a radiology reference standard. Patients received telephone follow-up and had institutional medical records reviewed in 3 weeks. An AE occurred if there were clinical sequelae and/or repeat health care visits due to a delay in correct radiograph interpretation.

RESULTS

We enrolled 2,302 children (mean [±SD] age = 9.0 [4.4] years; 1,288 (56.0%) male]. Of these, 180 (7.8%; 95% confidence interval = 6.8 to 9.0) ED physician discordant interpretations resulted in an AE. Specifically, there were no negative clinical outcomes; however, relative to cases diagnosed correctly at the index ED, patients whose fracture was not initially identified encountered 77.2% more subsequent ED visits, while those falsely diagnosed with a fracture experienced 41.5% additional orthopedic clinic visits. Odds of an ED discrepant interpretation was significantly higher if a physician's pretest probability of a fracture was ≤ 20% versus> 20% (adjusted odds ratio [aOR] = 1.6), patient's pain score was ≤ 2 versus> 2 (aOR = 1.6), and injury was located in a joint versus other location (aOR = 1.7).

CONCLUSIONS

Emergency physician discordant pediatric MSK radiograph interpretations that resulted in an AE occurred with regular frequency in a pediatric ED setting. AEs were primarily an increase in subsequent health care visits. Importantly, a low clinical suspicion for a fracture or injury located in the joint were risk factors for ED physician discordant interpretations.

Building Emergency Medicine Trainee Competency in Pediatric Musculoskeletal Radiograph Interpretation: A Multicenter Prospective Cohort Study

OBJECTIVES

As residency programs transition from time- to performance-based competency standards, validated tools are needed to measure performance-based learning outcomes and studies are required to characterize the learning experience for residents. Since pediatric musculoskeletal (MSK) radiograph interpretation can be challenging for emergency medicine trainees, we introduced Web-based pediatric MSK radiograph learning system with performance endpoints into pediatric emergency medicine (PEM) fellowships and determined the feasibility and effectiveness of implementing this intervention.

METHODS

This was a multicenter prospective cohort study conducted over 12 months. The course offered 2,100 pediatric MSK radiographs organized into seven body regions. PEM fellows diagnosed each case and received feedback after each interpretation. Participants completed cases until they achieved a performance benchmark of at least 80% accuracy, sensitivity, and specificity. The main outcome measure was the median number of cases completed by participants to achieve the performance benchmark.

RESULTS

Fifty PEM fellows from nine programs in the US and Canada participated. There were 301 of 350 (86%) modules started and 250 of 350 (71%) completed to the predefined performance benchmark during the study period. The median (interquartile range [IQR]) number of cases to performance benchmark per participant was 78 (60-104; min = 56, max = 1,333). Between modules, the median number of cases to achieve the performance benchmark was different for the ankle versus other modules (ankle 366 vs. other 76; difference = 290, 95% confidence interval [CI] = 245 to 335). The performance benchmark was achieved for 90.7% of participants in all modules except the ankle/foot, where 34.9% achieved this goal (difference = 55.8%, 95% CI = 45.3 to 66.3). The mean (95% CI) change in accuracy, sensitivity, and specificity from baseline to performance benchmark was +14.6% (13.4 to 15.8), +16.5% (14.8 to 18.1), and +12.6% (10.7 to 14.5), respectively. Median (IQR) time on each case was 31.0 (21.0-45.3) seconds.

CONCLUSIONS

Most participants completed the modules to the performance benchmark within 1 hour and demonstrated significant skill improvement. Further, there was a large variation in the number of cases completed to achieve the performance endpoint in any given module, and this impacted the feasibility of completing specific modules.