Some Thoughts on the Stability of Decision Rules

Structured Clinical Decision Aids Are Seldom Compared With Subjective Physician Judgment, and Are Seldom Superior

STUDY OBJECTIVE

We determine how often studies that evaluate the performance of an aid for decisionmaking, be it a simple laboratory or imaging test or a complex multielement decision instrument, compare the aid's performance to independent, unaided physician judgment.

METHODS

This was a cross-sectional survey of all Original Research and Brief Research Report articles in Annals of Emergency Medicine from 1998 to 2015. We included all articles that evaluated the performance of an aid for decisionmaking in assisting a physician with a decision about testing, treatment, diagnosis, or disposition. Two authors independently characterized the intent and purpose of each aid for decisionmaking, determined whether each study had a comparison to unaided physician judgment within the article or in a separate article, and recorded the result of that comparison.

RESULTS

One hundred seventy-one (8.3%) of 2,060 research articles studied the performance characteristics of an aid for decisionmaking, 48 of which were formal clinical decision instruments. Forty of the 171 studies retrospectively analyzed existing databases and therefore could not assess physician judgment. Investigators compared the aid for decisionmaking to physician judgment in 11% (15/131) of the prospective studies, including 15% (6/41) of studies that evaluated a formal clinical decision instrument. For 9 articles that had no comparison to physician judgment, we found 6 unique external publications that compared that aid to physician clinical judgment. The decision aid was superior to clinical judgment in 2 of the 21 studies that contained a comparison.

CONCLUSION

Physician judgment is infrequently assessed when the performance of an aid for decisionmaking is evaluated, and, when reported, the decision aid seldom outperformed physician judgment.

Interobserver agreement in assessment of clinical variables in children with blunt head trauma

OBJECTIVES

To be useful in development of clinical decision rules, clinical variables must demonstrate acceptable agreement when assessed by different observers. The objective was to determine the interobserver agreement in the assessment of historical and physical examination findings of children undergoing emergency department (ED) evaluation for blunt head trauma.

METHODS

This was a prospective cohort study of children younger than 18 years evaluated for blunt head trauma at one of 25 EDs in the Pediatric Emergency Care Applied Research Network (PECARN). Patients were excluded if injury occurred more than 24 hours prior to evaluation, if neuroimaging was obtained at another hospital prior to evaluation, or if the patient had a clinically trivial mechanism of injury. Two clinicians independently completed a standardized clinical assessment on a templated data form. Assessments were performed within 60 minutes of each other and prior to clinician review of any neuroimaging (if obtained). Agreement between the two observers beyond that expected by chance was calculated for each clinical variable, using the kappa (kappa) statistic for categorical variables and weighted kappa for ordinal variables. Variables with a lower 95% confidence limit (LCL) of kappa > 0.4 were considered to have acceptable agreement,

RESULTS

Fifteen-hundred pairs of observations were obtained. Acceptable agreement was achieved in 27 of the 32 variables studied (84%). Mechanism of injury (low, medium, or high risk) had kappa = 0.83. For subjective symptoms, kappa ranged from 0.47 (dizziness) to 0.93 (frequency of vomiting); all had 95% LCL > 0.4. Of the physical examination findings, kappa ranged from 0.22 (agitated) to 0.89 (Glasgow Coma Scale [GCS] score). The 95% LCL for kappa was < 0.4 for four individual signs of altered mental status and for quality (i.e., boggy or firm) of scalp hematoma if present.

CONCLUSIONS

Both subjective and objective clinical variables in children with blunt head trauma can be assessed by different observers with acceptable agreement, making these variables suitable candidates for clinical decision rules.