Effect of two rating formats in multi-disease ROC study of chest images

A scaling transformation for classifier output based on likelihood ratio: applications to a CAD workstation for diagnosis of breast cancer

PURPOSE

The authors developed scaling methods that monotonically transform the output of one classifier to the "scale" of another. Such transformations affect the distribution of classifier output while leaving the ROC curve unchanged. In particular, they investigated transformations between radiologists and computer classifiers, with the goal of addressing the problem of comparing and interpreting case-specific values of output from two classifiers.

METHODS

Using both simulated and radiologists' rating data of breast imaging cases, the authors investigated a likelihood-ratio-scaling transformation, based on "matching" classifier likelihood ratios. For comparison, three other scaling transformations were investigated that were based on matching classifier true positive fraction, false positive fraction, or cumulative distribution function, respectively. The authors explored modifying the computer output to reflect the scale of the radiologist, as well as modifying the radiologist's ratings to reflect the scale of the computer. They also evaluated how dataset size affects the transformations.

RESULTS

When ROC curves of two classifiers differed substantially, the four transformations were found to be quite different. The likelihood-ratio scaling transformation was found to vary widely from radiologist to radiologist. Similar results were found for the other transformations. Our simulations explored the effect of database sizes on the accuracy of the estimation of our scaling transformations.

CONCLUSIONS

The likelihood-ratio-scaling transformation that the authors have developed and evaluated was shown to be capable of transforming computer and radiologist outputs to a common scale reliably, thereby allowing the comparison of the computer and radiologist outputs on the basis of a clinically relevant statistic.

Reader performance in radiographic diagnosis of signs of mitral regurgitation in cavalier King Charles spaniels

OBJECTIVES

To measure accuracy and variability of diagnosis by radiography of heart enlargement (HE) and heart failure (HF) in mitral regurgitation (MR).

METHODS

Sixteen readers representing four levels of experience evaluated 50 sets of radiographs with varying severity of MR for presence or absence of HE, left atrial enlargement (LAE) and HF. The performance of the readers was compared with a reference standard, using area under the curve (AUC) of receiver operating characteristic (ROC) curves. The interreader agreement value kappa (K) was calculated. A subset of difficult cases of HF was analysed before and after removing an outlying reader from each group.

RESULTS

AUC for HE was 0.89, for LAE it was 0.93 and for HF it was 0.92. Experience increased certainty of diagnosis but not accuracy. K ranges were HE, 0.53 to 0.67; LAE, 0.61 to 0.69 and HF, 0.49 to 0.58. When only difficult cases of HF were read, accuracy decreased and experienced readers performed better than inexperienced. When outlying readers were excluded, the differences between experienced and inexperienced readers increased.

CLINICAL SIGNIFICANCE

LAE, not HE, should be used to evaluate the heart size and indirectly the severity of MR on radiographs. For HF, agreement among individual readers was only moderate. Studies of reader accuracy should consider the effects of interreader variability.

Receiver operating characteristic (ROC) curve: practical review for radiologists

The receiver operating characteristic (ROC) curve, which is defined as a plot of test sensitivity as the y coordinate versus its 1-specificity or false positive rate (FPR) as the x coordinate, is an effective method of evaluating the performance of diagnostic tests. The purpose of this article is to provide a nonmathematical introduction to ROC analysis. Important concepts involved in the correct use and interpretation of this analysis, such as smooth and empirical ROC curves, parametric and nonparametric methods, the area under the ROC curve and its 95% confidence interval, the sensitivity at a particular FPR, and the use of a partial area under the ROC curve are discussed. Various considerations concerning the collection of data in radiological ROC studies are briefly discussed. An introduction to the software frequently used for performing ROC analyses is also presented.

Observer performance studies: detection of single versus multiple abnormalities of the chest

OBJECTIVE

We used receiver operating characteristic (ROC) analysis to compare two methods of evaluating observer performance in detecting an abnormality on chest radiographs. In the first method, the abnormality in question, rib fracture, was one of five investigated, and it was the only one of interest in the second.

MATERIALS AND METHODS

Eight experienced observers viewed 117 posteroanterior chest radiographs in two interpretation modes. Fifty-four of these images depicted rib fractures that had been rated as subtle for detection. The likelihood of the presence of a rib fracture was rated as one of five abnormalities in question in one mode and the sole abnormality of interest in the other mode.

RESULTS

Six of the observers performed better during the single-abnormality mode, one performed equally well in both modes, and one performed better during the multiple-abnormality mode. The average area under the ROC curves (A(z)) was 0.73 +/- 0.07 for the multiple-abnormality mode and 0.80 +/- 0.04 for the single-abnormality mode. The results were significantly different (p < 0.05).

CONCLUSION

Study methodology can significantly affect the results in ROC studies, particularly for abnormalities that may not be perceived as primary or important. The order in which abnormalities appear on a checklist report form may be important.

On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

RATIONALE AND OBJECTIVES

A new methodology that analyzes receiver operating characteristic (ROC) data sets based on jackknifing and that considers both case and reader variability has been proposed. The purpose of this investigation was to compare results using this method to those using commonly reported methodology.

METHODS

ROC data sets using discrete and continuous rating scales were analyzed using the proposed jackknifing method, and results were compared to analysis of the same data sets using the paired t test.

RESULTS

The two methodologies did not result in the same significance levels, and in some cases, the difference was sufficient to affect conclusions regarding comparisons of diagnostic modalities. The probability value for the jackknifing procedure is based on large sample distribution theory, and its appropriateness is unknown for sample sizes used in practice. Also, the jackknifing technique was found to be sensitive to outliers resulting when data from the computer programs used to estimate area under the ROC curve failed to converge.

CONCLUSION

Although the proposed methodology yields reasonable results, several fundamental and practical issues must be addressed before it can be used widely as the analytic method of choice in ROC studies comparing different imaging techniques or reading environments.

An index for diagnostic accuracy in the multiple disease setting

RATIONALE AND OBJECTIVES

Evaluation of diagnostic accuracy in the clinical environment should entail some assessment of performance in patients with multiple abnormalities. Although receiver operating characteristic (ROC) curves often are used to assess the diagnostic accuracy of imaging systems, the concept is not easily generalizable to patients with multiple abnormalities. I propose a measure of diagnostic accuracy that is a generalization of the area under the ROC curve for a single disease.

METHODS

The proposed measure of diagnostic accuracy is a weighted average of the area under individual ROC curves for the single disease setting and of components representing areas under ROC curves constructed for patients with multiple diseases. Several options are discussed for scoring the presence of abnormality for patients who have two or more abnormalities.

RESULTS

Methods of estimating diagnostic accuracy are demonstrated on a set of data in which more than one third of the abnormal cases included multiple abnormalities of chest disease.

CONCLUSION

An easy-to-use method is given to estimate diagnostic accuracy in the multiple abnormality setting. This should make it easier to incorporate cases with multiple abnormalities when assessing the diagnostic accuracy of imaging systems.