On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

Assessment of medical imaging and computer-assist systems: lessons from recent experience

In the last 2 decades major advances have been made in the field of assessment methods for medical imaging and computer-assist systems through the use of the paradigm of the receiver operating characteristic (ROC) curve. In the most recent decade this methodology was extended to embrace the complication of reader variability through advances in the multiple-reader, multiple-case (MRMC) ROC measurement and analysis paradigm. Although this approach has been widely adopted by the imaging research community, some investigators appear averse to it, possibly from concern that it could place a greater burden on the scarce resources of patient cases and readers compared to the requirements of alternative methods. The present communication argues, however, that the MRMC ROC approach to assessment in the context of reader variability may be the most resource-efficient approach available. Moreover, alternative approaches may also be statistically uninterpretable with regard to estimated summary measures of performance and their uncertainties. The authors propose that the MRMC ROC approach be considered even more widely by the larger community with responsibilities for the introduction and dissemination of medical imaging technologies to society. General principles of study design are reviewed, and important contemporary clinical trials are used as examples.

Predictive model of axillary lymph node involvement in women with small invasive breast carcinoma: axillary metastases in breast carcinoma

BACKGROUND

Axillary lymph node involvement (ALNI) remains the most accurate predictive factor for recurrence risk and survival in patients with invasive breast carcinoma (IBC) and is an essential element in therapeutic decisions. However, axillary dissection (AD) is responsible for several side effects and is now discussed in small IBC. The objective of this study was to define a predictive model of ALNI by using clinical and histologic variables available before surgery.

METHODS

The authors studied 795 cases of IBC (T0, T1, T2 < or = 4 cm; N0; M0) treated between 1980 and 1997 by conservative surgery and radiation therapy. All cases had axillary dissection with at least 10 lymph nodes removed. A stepwise logistic regression analysis was performed to build a predictive model of ALNI. The authors then used the jackknife resampling technique to produce unbiased estimates of the probabilities of ALNI along with their confidence intervals.

RESULTS

The global ALNI rate was 25.7%. The final predictive model included clinical tumor size, location, and histologic subtype and grade as variables independently associated with ALNI. The estimated probability of ALNI varied from 6% to 45%, according to case characteristics for these variables.

CONCLUSIONS

These results show that the omission of AD in surgical procedures for these tumors is debatable. Even when ALNI rates were low, the superior bounds of the confidence intervals could be high. Consequently, we do not recommend to omit AD in women whose estimated risks are higher than 25%. Women with a risk of ALNI lower than 25% could benefit from the sentinel lymph node procedure with, likewise, a limited risk of false-negative.

Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design

RATIONALE AND OBJECTIVES

The authors conducted a series of null-case Monte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) method for comparing modalities with multireader receiver operating characteristic (ROC) discrete rating data.

MATERIALS AND METHODS

Monte Carlo simulations were performed by using discrete ratings on fully crossed factorial designs with two modalities and three, five, and 10 hypothetical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, and normal/abnormal case sample ratios used in another study were used in these simulations.

RESULTS

For equal allocation ratios and small (Az = 0.702) and moderate (Az = 0.855) ROC areas, the empirical type I error rate closely matched the nominal alpha level. For very large ROC areas (Az = 0.961), however, the empirical type I error rate was somewhat smaller than the nominal alpha level. This conservatism increased with decreasing case sample size and asymmetric normal/abnormal case allocation ratio. The empirical type I error rate was sometimes slightly larger than the nominal alpha level with many cases and few readers, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction.

CONCLUSION

The results suggest that the DBM method provides trustworthy alpha levels with discrete ratings when the ROC area is not too large and case and reader sample sizes are not too small. In other situations, the test tends to be somewhat conservative or slightly liberal.

The "proper" binormal model: parametric receiver operating characteristic curve estimation with degenerate data

RATIONALE AND OBJECTIVES

The authors assessed the use of a "proper" binormal model and a new algorithm for maximum-likelihood estimation of receiver operating characteristic (ROC) curves from degenerate data.

METHODS

The proper binormal ROC model uses as its decision variable a monotonic transformation of the likelihood ratio that is associated with a pair of normal distributions, thereby ensuring fitted ROC curves with monotonic slope but maintaining a relationship with the conventional binormal model. A computer program entitled PROPROC was used to fit proper ROC curves to data obtained from computer-simulated and real observer studies.

RESULTS

ROC indexes such as total area were estimated with PROPROC and compared with the corresponding values obtained from the conventional procedures.

CONCLUSION

The proper binormal ROC model overcomes the problem of degeneracy in ROC curve fitting. PROPROC is highly robust and yields ROC estimates with less bias and greater precision than those obtained with the conventional binormal model.

Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation

RATIONALE AND OBJECTIVES

The authors examined the relationship between the critical P value (alpha) and the empirical type I error rate when using the Dorfman-Berbaum-Metz (DMB) method for analysis of variance in multireader, multimodality receiver operating characteristic (ROC) data.

METHODS

The authors developed a linear mixed-effect model to generate continuous, normally distributed random decision variables containing multiple sources (components) of variation. A range of magnitudes for these variance components was used to stimulate experiments in which multiple readers (three or five) read imaged obtained with two modalities from the same set of cases with no re-reading. Three binormal population ROC curves, with areas of 0.962, 0.855, and 0.702, were included. Case-sample sizes ranged from 50 to 400, and either 50% or 10% of cases were actually positive. For each experiment, 2,000 data sets were analyzed by the computer program, and the proportion of 2,000 modality differences that was found to be statistically significant at an alpha level of .05 was tubulated.

RESULTS

The test for modality difference performed well for the low and intermediate ROC curves, even with small case samples. For the high ROC curve, the small-sample results were conservative. No relationship between observed type I error rate and the magnitude of data correlation was evident.

CONCLUSION

For typical ROC curves, the DBM method is robust in testing for modality effects in the null case, given a sufficient sample size. Instructions for obtaining a free copy of the software are given.

Proper receiver operating characteristic analysis: the bigamma model

RATIONALE AND OBJECTIVES

The standard binormal model is the most commonly used model for fitting receiver operating characteristic rating data; however, it sometimes produces inappropriate fits that cross the chance line with degenerate data sets. The authors proposed and evaluated a proper constant-shape bigamma model to handle binormal degeneracy.

METHODS

Monte Carlo samples were generated from both a standard binormal population model and a proper constant-shape bigamma model in a series of Monte Carlo studies.

RESULTS

The results confirm that the standard binormal model is robust in large samples with no degenerate data sets and that the standard binormal model is not robust in small samples because of degenerate data sets.

CONCLUSION

A proper constant-shape bigamma model seems to solve the problem of degeneracy without inappropriate chance line crossings. The bigamma fitting model outperformed the standard binormal fitting model in small samples and gave similar results in large samples.

Degeneracy and discrete receiver operating characteristic rating data

RATIONALE AND OBJECTIVES

Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic.

METHODS

Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies.

RESULTS

RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased.

CONCLUSION

RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.

Diagnostic performance of CT, MPR and 3DCT imaging in maxillofacial trauma

CT imaging of complex maxillofacial fractures is common practice now, but the relative diagnostic value of spiral computed tomography (CT), multiplanar reformations (MPR), and three-dimensional (3D) reconstructions in evaluating maxillofacial fractures is not established with independent validation of correct diagnosis. We studied these modalities and measured their diagnostic value in a carefully controlled observer based rated response experiment. Multiple fractures were created by blunt experimental trauma in nine adult cadaver heads (five males, four females). Spiral CT scans were performed on all specimens before (control) and after trauma. Axial slices (CT), sagittal and coronal multiplanar reconstructions (MPR), and 3D volumetric reconstructions views were generated. Truth was determined by defleshing the specimens and direct inspection of the traumatized skull. Three expert readers separately interpreted CT, MPR and 3D film hard copy images presented in random order blinded to patient identification or experimental conditions. We measured the time to diagnose each case as recorded by a monitor who was present while evaluations were performed. Twenty-eight facial regions were evaluated using rated response and free response illustrative formats. Each region was considered separately. Sensitivity and specificity were calculated to measure observer performance. We found that 3D and CT had a similar performance in fracture detection and both were markedly better than MPR. For free response illustrative data, CT correctly identified 10% more orbital fractures than 3D, and approximately 10% fewer zygomatic fractures. Fracture localization was best with 3D. Reader confidence was highest with CT, but assessment time was faster with 3D. We conclude that CT and 3D are comparable in detecting midfacial fractures and both are superior to MPR. 3D reconstructions are superior for localization of complex fractures involving multiple planes.