Teaching medical students to estimate probability of coronary artery disease

The covariance decomposition of the probability score and its use in evaluating prognostic estimates. SUPPORT Investigators

The probability score (PS) or Brier score has been used in a large number of studies in which physician judgment performance was assessed. However, the covariance decomposition of the PS has not previously been used to evaluate medical judgment. The authors introduce the technique and demonstrate it by analyzing prognostic estimates of three groups: physicians, their patients, and the patients' decision-making surrogates. The major components of the covariance decomposition--bias, slope, and scatter--are displayed in covariance graphs for each of the three groups. The decomposition reveals that whereas the physicians have the best overall estimation performance, their bias and their scatter are not always superior to those of the other two groups. This is primarily due to two factors. First, the physicians' prognostic estimates are pessimistic. Second, the patients place the large majority of their estimates in the most optimistic category, thereby achieving low scatter. The authors suggest that the calculational simplicity of this decomposition, its informativeness, and the intuitive nature of its components make it a useful tool with which to analyze medical judgment.

Assessing predictive accuracy: how to compare Brier scores

Several investigators have used the Brier index to measure the predictive accuracy of a set of medical judgments; the Brier scores of different raters who have evaluated the same patients provides a measure of relative accuracy. However, such comparisons may be difficult to interpret because of the lack of a statistical test for differentiating between two Brier scores. To demonstrate a method for addressing this issue we analyzed the judgments of five medical students, each of whom independently evaluated the same 25 patients with recurrent chest pain. Using the method we determined that two of the students gave judgments that were incompatible with the actual observed outcomes (p less than 0.05); of the three remaining students we detected a significant difference between two (p less than 0.05). These results differed from receiver operating characteristic curve area analysis, another technique used to evaluate predictive accuracy. We suggest that the proposed method can provide a useful tool for investigators using the Brier index to compare how well clinicians express uncertainty using probability judgments.

Evaluating physicians' probabilistic judgments

Physicians increasingly are challenged to make probabilistic judgments quantitatively. Their ability to make such judgments may be directly linked to the quality of care they provide. Many methods are available to evaluate these judgments. Graphic means of assessment include the calibration curve, covariance graph, and receiver operating characteristic (ROC) curve. Statistical tools can measure the significance of departures from ideal calibration, and measure the area under ROC curve. Modeling the calibration curve using linear or logistic regression provides another method to assess probabilistic judgments, although these may be limited by failure of the data to meet the model's assumptions. Scoring rules provide indices of overall judgmental performance, although their reliability is difficult to gauge for small sample sizes. Decompositions of scoring rules separate judgmental performance into functional components. The authors provide preliminary guidelines for choosing methods for specific research in this area.

Use of linear models to analyze physicians' decisions

Linear models of judgment are powerful tools for studying medical decision making. The recent increase in applications of these models to medicine reflects more available computing resources and the parallel development of clinical prediction rules derived from multivariate analysis of patient data. Psychological research into expert and novice decision making shows that linear models derived from judges' decisions usually predict future decisions more accurately than either the judge or a mechanical application of the judge's stated policies. Studies of medical decision making have shown similar results, as well as marked variation among experts in how they appear to use clinical information. Cognitive feedback, which is feedback to the learner of the judgment model derived from previous decisions, is highly effective for teaching complex judgment tasks. Many technical problems remain to be mastered in constructing linear models of medical judgment. These include how to select the correct variables, how to provide a selection of variables broad enough to accommodate individual variations in strategy, how to model intercorrelated variables, and how to characterize and aggregate individual strategies. Despite the methodologic challenges, linear models remain a powerful method for studying how physicians combine multiple items of imperfect information to make a judgment. These techniques may provide important insights into variation in physician judgments. In addition, they hold promise in teaching the appropriate integration of complex data in the day-to-day practice of medicine.