Practical issues of experimental ROC analysis. Selection of controls

A Methodology for Anatomic Ultrasound Image Diagnostic Quality Assessment

This paper discusses the methods for the assessment of ultrasound image quality based on our experiences with evaluating new methods for anatomic imaging. It presents a methodology to ensure a fair assessment between competing imaging methods using clinically relevant evaluations. The methodology is valuable in the continuing process of method optimization and guided development of new imaging methods. It includes a three phased study plan covering from initial prototype development to clinical assessment. Recommendations to the clinical assessment protocol, software, and statistical analysis are presented. Earlier uses of the methodology has shown that it ensures validity of the assessment, as it separates the influences between developer, investigator, and assessor once a research protocol has been established. This separation reduces confounding influences on the result from the developer to properly reveal the clinical value. This paper exemplifies the methodology using recent studies of synthetic aperture sequential beamforming tissue harmonic imaging.

Full receiver operating characteristic curve estimation using two alternative forced choice studies

Task-based medical image quality is typically measured by the degree to which a human observer can perform a diagnostic task in a psychophysical human observer study. During a typical study, an observer is asked to provide a numerical score quantifying his confidence as to whether an image contains a diagnostic marker or not. Such scores are then used to measure the observers' diagnostic accuracy, summarized by the receiver operating characteristic (ROC) curve and the area under ROC curve. These types of human studies are difficult to arrange, costly, and time consuming. In addition, human observers involved in this type of study should be experts on the image genre to avoid inconsistent scoring through the lengthy study. In two-alternative forced choice (2AFC) studies, known to be faster, two images are compared simultaneously and a single indicator is given. Unfortunately, the 2AFC approach cannot lead to a full ROC curve or a set of image scores. The aim of this work is to propose a methodology in which multiple rounds of the 2AFC studies are used to re-estimate an image confidence score (a.k.a. rating, ranking) and generate the full ROC curve. In the proposed approach, we treat image confidence score as an unknown rating that needs to be estimated and 2AFC as a two-player match game. To achieve this, we use the ELO rating system, which is used for calculating the relative skill levels of players in competitor-versus-competitor games such as chess. The proposed methodology is not limited to ELO, and other rating methods such as TrueSkill™, Chessmetrics, or Glicko can be also used. The presented results, using simulated data, indicate that a full ROC curve can be recovered using several rounds of 2AFC studies and that the best pairing strategy starts with the first round of pairing abnormal versus normal images (as in the classical 2AFC approach) followed by a number of rounds using random pairing. In addition, the proposed method was tested in a pilot human observer study. These pilot results indicate that three to five rounds of 2AFC studies require less human observer time than a full scoring study and that the re-estimated ROC curves and associated area under ROC curve values have high statistical agreement with the full scoring study.

Experimental system for measurement of radiologists' performance by visual search task

PURPOSE

Detective performance of radiologists for "obvious" targets should be evaluated by visual search task instead of ROC analysis, but visual task have not been applied to radiology studies. The aim of this study was to set up an environment that allows visual search task in radiology, to evaluate its feasibility, and to preliminarily investigate the effect of career on the performance.

MATERIALS AND METHODS

In a darkroom, ten radiologists were asked to answer the type of lesion by pressing buttons, when images without lesions, with bulla, ground-glass nodule, and solid nodule were randomly presented on a display. Differences in accuracy and reaction times depending on board certification were investigated.

RESULTS

The visual search task was successfully and feasibly performed. Radiologists were found to have high sensitivity, specificity, positive predictive values and negative predictive values in non-board and board groups. Reaction time was under 1 second for all target types in both groups. Board radiologists were significantly faster in answering for bulla, but there were no significant differences for other targets and values.

CONCLUSION

We developed an experimental system that allows visual search experiment in radiology. Reaction time for detection of bulla was shortened with experience.

"Binary" and "non-binary" detection tasks: are current performance measures optimal?

We have observed that a very large fraction of responses for several detection tasks during the performance of observer studies are in the extreme ranges of lower than 11% or higher than 89% regardless of the actual presence or absence of the abnormality in question or its subjectively rated "subtleness." This observation raises questions regarding the validity and appropriateness of using multicategory rating scales for such detection tasks. Monte Carlo simulation of binary and multicategory ratings for these tasks demonstrate that the use of the former (binary) often results in a less biased and more precise summary index and hence may lead to a higher statistical power for determining differences between modalities.

Receiver operating characteristic analysis for the detection of simulated microcalcifications on mammograms using hardcopy images

The aim of this study was to compare mammography systems based on three different detectors--a conventional screen-film (SF) combination, an a-Si/CsI flat-panel (FP)-based detector, and a charge-coupled device (CCD)-based x-ray phosphor-based detector--for their performance in detecting simulated microcalcifications (MCs). 112-150 microm calcium carbonate grains were used to simulate MCs and were overlapped with a slab phantom of simulated 50% adipose/50% glandular breast tissue-equivalent material referred to as the uniform background. For the tissue structure background, 200-250 microm calcium carbonate grains were used and overlapped with an anthropomorphic breast phantom. All MC phantom images were acquired with and without magnification (1.8 X). The hardcopy images were reviewed by five mammographers. A five-point confidence level rating was used to score each detection task. Receiver operating characteristic (ROC) analysis was performed, and the areas under the ROC curves (A(z)s) were used to compare the performances of the three mammography systems under various conditions. The results showed that, with a uniform background and contact images, the FP-based system performed significantly better than the SF and the CCD-based systems. For magnified images with a uniform background, the SF and the FP-based systems performed equally well and significantly better than the CCD-based system. With tissue structure background and contact images, the SF system performed significantly better than the FP and the CCD-based systems. With magnified images and a tissue structure background, the SF and the CCD-based systems performed equally well and significantly better than the FP-based system. In the detection of MCs in the fibroglandular and the heterogeneously dense regions, no significant differences were found except that the SF system performed significantly better than the CCD-based system in the fibroglandular regions for the contact images.

History of Research in Medical Image Perception

Human observers engage in 2 interrelated processes when interpreting medical images: perception and analysis. Perception is the unified awareness of the content of a displayed image that is present while the stimulus is on. Analysis is determining the meaning of the perception in the context of the medical problem that initiated the acquisition of the image. Radiologists have, correctly, regarded image analysis as their primary field of research. They have naively assumed that what they perceive in images is a faithful representation of the images' information content and have not been concerned with perception unless it fails. Failures have stimulated research on quantifying observer performance, defining image quality, and understanding perceptual error. This article traces the historical development of the use of receiver operating characteristic analysis for describing performance, the development of signal-to-noise ratio psychophysical models for defining task-dependent image quality, studies of error in small lesion detection, and the beginnings of studies of the nature of expertise in image interpretation. The history is traced through published articles.

Introduction to cost-effectiveness analysis in radiology: principles and practical application

The objectives of this paper are to provide the framework for understanding the principles of cost-effectiveness analysis and to practically apply these principles when reading a cost-effectiveness article. The article focuses on the application of these concepts to radiology procedures. To illustrate these concepts, we will use a hypothetical case of a physician deciding on the most cost-effective method for initially evaluating a 60-year-old menopausal woman who complains of new vaginal bleeding.

Intracranial aneurysms: detection with three-dimensional CT angiography with volume rendering--comparison with conventional angiographic and surgical findings

PURPOSE

To assess the diagnostic accuracy of three-dimensional (3D) computed tomographic (CT) angiography for intracranial aneurysms.

MATERIALS AND METHODS

The 3D CT angiograms obtained in 49 patients with or without intracranial aneurysms were evaluated by four blinded observers. Results were compared with findings at conventional angiography or surgery. A volume-rendering method was used, and 13 images obtained in different directions were reviewed in each study. The diameter of aneurysms was divided into four sizes: large, greater than 13 mm; medium, 5-12 mm; small, 3-4 mm; and very small, less than 3 mm. Results were also evaluated by means of receiver operating characteristic analysis.

RESULTS

At conventional angiography, 47 aneurysms, including 14 less than 3 mm, were depicted in 35 patients. The mean sensitivity of CT angiography for very small aneurysms was 64%; small, 83%; medium, 95%; and large, 100%. Some very small aneurysms that were not depicted at conventional angiography were depicted at CT angiography, and one was proved at surgery.

CONCLUSION

CT angiography has good sensitivity for depiction of intracranial aneurysms 3 mm or larger and relatively good sensitivity for aneurysms less than 3 mm. CT angiography may be a noninvasive technique for detection of asymptomatic unruptured or ruptured aneurysms.

The etiology of fetal acidosis as determined by umbilical cord acid-base studies

OBJECTIVE

Our purpose was to determine whether umbilical cord acid-base values might be used to differentiate abruptio placentae from cord prolapse.

STUDY DESIGN

By use of a retrospective study design, umbilical cord blood acid-base values of 32 cases of severely acidotic acute abruptio placentae were statistically compared with those of 19 cases of severely acidotic acute cord prolapse with Fisher's exact test, Mann-Whitney U tests, and receiver-operator curve characteristic analyses.

RESULTS

No significant differences in electronic fetal heart rate record interpretations were detected. Highly significant differences in umbilical arterial and umbilical venous blood gas values were noted between the two groups. Most notable were the differences between umbilical venous and arterial blood gas values in the cord prolapse group. Receiver-operator characteristic curve analysis demonstrated that an umbilical venous-arterial pH difference of 0.15 was an effective cutoff value in differentiating cord prolapse from abruptio placentae (accuracy 92%).

CONCLUSION

Umbilical arterial blood gas values combined with umbilical venous blood gas values can assist in determining the pathogenesis of marked fetal acidosis.

Adequacy testing of training set sample sizes in the development of a computer-assisted diagnosis scheme

RATIONALE AND OBJECTIVES

The authors assessed the performance changes of a computer-assisted diagnosis (CAD) scheme as a function of the number of regions used for training (rule-setting).

MATERIALS AND METHODS

One hundred twenty regions depicting actual masses and 400 suspicious but actually negative regions were selected as a testing data set from a database of 2,146 regions identified as suspicious on 618 mammograms. An artificial neural network using 24 and 16 region-based features as input neurons was applied to classify the regions as positive or negative for the presence of a mass. CAD scheme performance was evaluated on the testing data set as the number of regions used for training increased from 60 to 496.

RESULTS

As the number of regions in the training sets increased, the results decreased and plateaued beyond a sample size of approximately 200 regions. Performance with the testing data set continued to improve as the training data set increased in size.

CONCLUSION

A trend in a system's performance as a function of training set size can be used to assess adequacy of the training data set in the development of a CAD scheme.

Technology assessment--an American view

Technology assessment of an imaging method such as magnetic resonance (MR) is a complicated concept that includes aspects of epidemiology, biostatistics, clinical efficacy determination, outcomes assessment, and knowledge of the technical and medical bases of the imaging method under study. To enhance understanding of the interrelations of the different aspects of technology assessment, a hierarchical model is proposed. This extends from the basic imaging physics domain through clinical applications in diagnosis and treatment decisions to patient outcomes and ultimately societal considerations. This overview paper presents the conceptual continuum of the hierarchical model, and then describes the interrelationships among efficacy, cost effectiveness, and outcomes research as components embedded within the context of technology assessment. It also points out how scientific quality of research in MR imaging assessment can be enhanced through improved research design that takes into account basic concepts of the model.

The efficacy of diagnostic imaging

The authors discuss the assessment of the contribution of diagnostic imaging to the patient management process. A hierarchical model of efficacy is presented as an organizing structure for appraisal of the literature on efficacy of imaging. Demonstration of efficacy at each lower level in this hierarchy is logically necessary, but not sufficient, to assure efficacy at higher levels. Level 1 concerns technical quality of the images; Level 2 addresses diagnostic accuracy, sensitivity, and specificity associated with interpretation of the images. Next, Level 3 focuses on whether the information produces change in the referring physician's diagnostic thinking. Such a change is a logical prerequisite for Level 4 efficacy, which concerns effect on the patient management plan. Level 5 efficacy studies measure (or compute) effect of the information on patient outcomes. Finally, at Level 6, analyses examine societal costs and benefits of a diagnostic imaging technology. The pioneering contributions of Dr. Lee B. Lusted in the study of diagnostic imaging efficacy are highlighted.