Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data

Prediction Variability to Identify Reduced AI Performance in Cancer Diagnosis at MRI and CT

Background A priori identification of patients at risk of artificial intelligence (AI) failure in diagnosing cancer would contribute to the safer clinical integration of diagnostic algorithms. Purpose To evaluate AI prediction variability as an uncertainty quantification (UQ) metric for identifying cases at risk of AI failure in diagnosing cancer at MRI and CT across different cancer types, data sets, and algorithms. Materials and Methods Multicenter data sets and publicly available AI algorithms from three previous studies that evaluated detection of pancreatic cancer on contrast-enhanced CT images, detection of prostate cancer on MRI scans, and prediction of pulmonary nodule malignancy on low-dose CT images were analyzed retrospectively. Each task's algorithm was extended to generate an uncertainty score based on ensemble prediction variability. AI accuracy percentage and partial area under the receiver operating characteristic curve (pAUC) were compared between certain and uncertain patient groups in a range of percentile thresholds (10%-90%) for the uncertainty score using permutation tests for statistical significance. The pulmonary nodule malignancy prediction algorithm was compared with 11 clinical readers for the certain group (CG) and uncertain group (UG). Results In total, 18 022 images were used for training and 838 images were used for testing. AI diagnostic accuracy was higher for the cases in the CG across all tasks (P < .001). At an 80% threshold of certain predictions, accuracy in the CG was 21%-29% higher than in the UG and 4%-6% higher than in the overall test data sets. The lesion-level pAUC in the CG was 0.25-0.39 higher than in the UG and 0.05-0.08 higher than in the overall test data sets (P < .001). For pulmonary nodule malignancy prediction, accuracy of AI was on par with clinicians for cases in the CG (AI results vs clinician results, 80% [95% CI: 76, 85] vs 78% [95% CI: 70, 87]; P = .07) but worse for cases in the UG (AI results vs clinician results, 50% [95% CI: 37, 64] vs 68% [95% CI: 60, 76]; P < .001). Conclusion An AI-prediction UQ metric consistently identified reduced performance of AI in cancer diagnosis. © RSNA, 2023 Supplemental material is available for this article. See also the editorial by Babyn in this issue.

Fast cancer metastasis location based on dual magnification hard example mining network in whole-slide images

Breast cancer has become the most common form of cancer among women. In recent years, deep learning has shown great potential in aiding the diagnosis of pathological images, particularly through the use of convolutional neural networks for locating lymph node metastasis under gigapixel whole slide images (WSIs). However, the massive size of these images at the highest magnification introduces redundant computation during the inference process. Additionally, the diversity of biological textures and structures within WSIs can cause confusion for classifiers, particularly in identifying hard examples. As a result, the trade-off between accuracy and efficiency remains a critical issue for whole-slide image metastasis localization. In this paper, we propose a novel two-stream network that takes a pair of low- and high-magnification image patches as input for identifying hard examples during the training phase. Specifically, our framework focuses on samples where the outputs of the two magnification networks are dissimilar. We adopt a dual magnification hard mining loss to re-weight the ambiguous samples. To more efficiently locate tumor metastasis cells in whole slide images, the two stream networks are decomposed into a cascaded network during the inference phase. The low magnification WSIs scanned by the low-mag network generate a coarse probability map, and the suspicious areas in the map are refined by the high-mag network. Finally, we evaluate our fast location dual magnification hard example mining network on the Camelyon16 breast cancer whole-slide image dataset. Experiments demonstrate that our proposed method achieves a 0.871 FROC score with a faster inference time, and our high magnification network also achieves a 0.88 FROC score.

End-to-end metastasis detection of breast cancer from histopathology whole slide images

Worldwide breast cancer is one of the most frequent and mortal diseases across women. Early, accurate metastasis cancer detection is a significant factor in raising the survival rate among patients. Diverse Computer-Aided Diagnostic (CAD) systems applying medical imaging modalities, have been designed for breast cancer detection. The impact of deep learning in improving CAD systems' performance is undeniable. Among all of the medical image modalities, histopathology (HP) images consist of richer phenotypic details and help keep track of cancer metastasis. Nonetheless, metastasis detection in whole slide images (WSIs) is still problematic because of the enormous size of these images and the massive cost of labelling them. In this paper, we develop a reliable, fast and accurate CAD system for metastasis detection in breast cancer while applying only a small amount of annotated data with lower resolution. This saves considerable time and cost. Unlike other works which apply patch classification for tumor detection, we employ the benefits of attention modules adding to regression and classification, to extract tumor parts simultaneously. Then, we use dense prediction for mask generation and identify individual metastases in WSIs. Experimental outcomes demonstrate the efficiency of our method. It provides more accurate results than other methods that apply the total dataset. The proposed method is about seven times faster than an expert pathologist, while producing even more accurate results than an expert pathologist in tumor detection.

Spatiality Sensitive Learning for Cancer Metastasis Detection in Whole-Slide Images

Metastasis detection in lymph nodes via microscopic examination of histopathological images is one of the most crucial diagnostic procedures for breast cancer staging. The manual analysis is extremely labor-intensive and time-consuming because of complexities and diversities of histopathology images. Deep learning has been utilized in automatic cancer metastasis detection in recent years. Due to the huge size of whole-slide images, most existing approaches split each image into smaller patches and simply treat these patches independently, which ignores the spatial correlations among them. To solve this problem, this paper proposes an effective spatially sensitive learning framework for cancer metastasis detection in whole-slide images. Moreover, a novel spatial loss function is designed to ensure the consistency of prediction over neighboring patches. Specifically, through incorporating long short-term memory and spatial loss constraint on top of a convolutional neural network feature extractor, the proposed method can effectively learn both the appearance of each patch and spatial relationships between adjacent image patches. With the standard back-propagation algorithm, the whole framework can be trained in an end-to-end way. Finally, the regions with high tumor probability in the resulting probability map are the metastasis locations. Extensive experiments on the benchmark Camelyon 2016 Grand Challenge dataset show the effectiveness of the proposed approach with respect to state-of-the-art competitors. The obtained precision, recall, and balanced accuracy are 0.9565, 0.9167, and 0.9458, respectively. It is also demonstrated that the proposed approach can provide more accurate detection results and is helpful for early diagnosis of cancer metastasis.

Improving Cancer Metastasis Detection via Effective Contrastive Learning

The metastasis detection in lymph nodes via microscopic examination of H&E stained histopathological images is one of the most crucial diagnostic procedures for breast cancer staging. The manual analysis is extremely labor-intensive and time-consuming because of complexities and diversities of histopathological images. Deep learning has been utilized in automatic cancer metastasis detection in recent years. The success of supervised deep learning is credited to a large labeled dataset, which is hard to obtain in medical image analysis. Contrastive learning, a branch of self-supervised learning, can help in this aspect through introducing an advanced strategy to learn discriminative feature representations from unlabeled images. In this paper, we propose to improve breast cancer metastasis detection through self-supervised contrastive learning, which is used as an accessional task in the detection pipeline, allowing a feature extractor to learn more valuable representations, even if there are fewer annotation images. Furthermore, we extend the proposed approach to exploit unlabeled images in a semi-supervised manner, as self-supervision does not need labeled data at all. Extensive experiments on the benchmark Camelyon2016 Grand Challenge dataset demonstrate that self-supervision can improve cancer metastasis detection performance leading to state-of-the-art results.

Fully Automatic Deep Learning Framework for Pancreatic Ductal Adenocarcinoma Detection on Computed Tomography

Simple Summary

Early image-based diagnosis is crucial to improve outcomes in pancreatic ductal adenocarcinoma (PDAC) patients, but is challenging even for experienced radiologists. Artificial intelligence has the potential to assist in early diagnosis by leveraging high amounts of data to automatically detect small (<2 cm) lesions. In this study, the state-of-the-art, self-configuring framework for medical segmentation nnUnet was used to develop a fully automatic pipeline for the detection and localization of PDAC lesions on contrast-enhanced computed tomography scans, with a focus on small lesions. Furthermore, the impact of integrating the surrounding anatomy (which is known to be relevant to clinical diagnosis) into deep learning models was assessed. The developed automatic framework was tested in an external, publicly available test set, and the results showed that state-of-the-art deep learning can detect small PDAC lesions and benefits from anatomy information.

Abstract

Early detection improves prognosis in pancreatic ductal adenocarcinoma (PDAC), but is challenging as lesions are often small and poorly defined on contrast-enhanced computed tomography scans (CE-CT). Deep learning can facilitate PDAC diagnosis; however, current models still fail to identify small (<2 cm) lesions. In this study, state-of-the-art deep learning models were used to develop an automatic framework for PDAC detection, focusing on small lesions. Additionally, the impact of integrating the surrounding anatomy was investigated. CE-CT scans from a cohort of 119 pathology-proven PDAC patients and a cohort of 123 patients without PDAC were used to train a nnUnet for automatic lesion detection and segmentation (nnUnet_T). Two additional nnUnets were trained to investigate the impact of anatomy integration: (1) segmenting the pancreas and tumor (nnUnet_TP), and (2) segmenting the pancreas, tumor, and multiple surrounding anatomical structures (nnUnet_MS). An external, publicly available test set was used to compare the performance of the three networks. The nnUnet_MS achieved the best performance, with an area under the receiver operating characteristic curve of 0.91 for the whole test set and 0.88 for tumors <2 cm, showing that state-of-the-art deep learning can detect small PDAC and benefits from anatomy information.

A radiomics approach for lung nodule detection in thoracic CT images based on the dynamic patterns of morphological variation

OBJECTIVES

To propose and evaluate a set of radiomic features, called morphological dynamics features, for pulmonary nodule detection, which were rooted in the dynamic patterns of morphological variation and needless precise lesion segmentation.

MATERIALS AND METHODS

Two datasets were involved, namely, university hospital (UH) and LIDC datasets, comprising 72 CT scans (360 nodules) and 888 CT scans (2230 nodules), respectively. Each nodule was annotated by multiple radiologists. Denoted the category of nodules identified by at least k radiologists as ALk. A nodule detection algorithm, called CAD-MD algorithm, was proposed based on the morphological dynamics radiomic features, characterizing a lesion by ten sets of the same features with different values extracted from ten different thresholding results. Each nodule candidate was classified by a two-level classifier, including ten decision trees and a random forest, respectively. The CAD-MD algorithm was compared with a deep learning approach, the N-Net, using the UH dataset.

RESULTS

On the AL1 and AL2 of the UH dataset, the AUC of the AFROC curves were 0.777 and 0.851 for the CAD-MD algorithm and 0.478 and 0.472 for the N-Net, respectively. The CAD-MD algorithm achieved the sensitivities of 84.4% and 91.4% with 2.98 and 3.69 FPs/scan and the N-Net 74.4% and 80.7% with 3.90 and 4.49 FPs/scan, respectively. On the LIDC dataset, the CAD-MD algorithm attained the sensitivities of 87.6%, 89.2%, 92.2%, and 95.0% with 4 FPs/scan for AL1-AL4, respectively.

CONCLUSION

The morphological dynamics radiomic features might serve as an effective set of radiomic features for lung nodule detection.

KEY POINTS

• Texture features varied with such CT system settings as reconstruction kernels of CT images, CT scanner models, and parameter settings, and so on. • Shape and first-order statistics were shown to be the most robust features against variation in CT imaging parameters. • The morphological dynamics radiomic features, which mainly characterized the dynamic patterns of morphological variation, were shown to be effective for lung nodule detection.

A Magnified Adaptive Feature Pyramid Network for automatic microaneurysms detection

Diabetic retinopathy (DR), as an important complication of diabetes, is the primary cause of blindness in adults. Automatic DR detection poses a challenge which is crucial for early DR screening. Currently, the vast majority of DR is diagnosed through fundus images, where the microaneurysm (MA) has been widely used as the most distinguishable marker. Research works on automatic DR detection have traditionally utilized manually designed operators, while a few recent researchers have explored deep learning techniques for this topic. But due to issues such as the extremely small size of microaneurysms, low resolution of fundus pictures, and insufficient imaging depth, the DR detection problem is quite challenging and remains unsolved. To address these issues, this research proposes a new deep learning model (Magnified Adaptive Feature Pyramid Network, MAFP-Net) for DR detection, which conducts super-resolution on low quality fundus images and integrates an improved feature pyramid structure while utilizing a standard two-stage detection network as the backbone. Our proposed detection model needs no pre-segmented patches to train the CNN network. When tested on the E-ophtha-MA dataset, the sensitivity value of our method reached as high as 83.5% at false positives per image (FPI) of 8 and the F1 value achieved 0.676, exceeding all those of the state-of-the-art algorithms as well as the human performance of experienced physicians. Similar results were achieved on another public dataset of IDRiD.

Rapid measurement of the low contrast detectability of CT scanners

PURPOSE

Low contrast detectability (LCD) is a metric of fundamental importance in computed tomography (CT) imaging. In spite of this, its measurement is challenging in the context of nonlinear data processing. We introduce a new framework for objectively characterizing LCD with a single scan of a special-purpose phantom and automated analysis software. The output of the analysis software is a "machine LCD" metric which is more representative of LCD than contrast-noise ratio (CNR). It is not intended to replace human observer or model observer studies.

METHODS

Following preliminary simulations, we fabricated a phantom containing hundreds of low-contrast beads. These beads are acrylic spheres (1.6 mm, net contrast ~10 HU) suspended and randomly dispersed in a background matrix of nylon pellets and isoattenuating saline. The task was to search for and localize the beads. A modified matched filter was used to automatically scan the reconstruction and select candidate bead localizations of varying confidence. These were compared to bead locations as determined from a high-dose reference scan to produce free-response ROC curves. We compared iterative reconstruction (IR) and filtered backpropagation (FBP) at multiple dose levels between 40 and 240 mAs. The scans at 60, 120, and 180 mAs were performed three times each to estimate uncertainty.

RESULTS

Experimental scans demonstrated the feasibility of our technique. Our metric for machine LCD was the area under the exponential transform of the FROC curve (AUC). AUC increased monotonically from 0.21 at 40 mAs to 0.84 at 240 mAs. The sample standard deviation of AUC was approximately 0.02. This measurement uncertainty in AUC corresponded to a change in tube current of 4% to 8%. Surprisingly, we found that AUCs for IR were slightly worse than AUCs for FBP. While the phantom was sufficient for these experiments, it contained small air bubbles and alternative fabrication methods will be necessary for widespread utilization.

CONCLUSIONS

It is feasible to measure machine LCD using a search task on a phantom with hundreds of beads and to obtain tight error bars using only a single scan. Our method could facilitate routine quality assurance or possibly enable comparisons between different protocols and scanners.

Receiver Operating Characteristic (ROC) Analysis of Image Search-and-Localize Tasks

RATIONALE AND OBJECTIVES

Receiver operating characteristic (ROC) analysis for the common image search-and-localize task, in which readers search an image for lesion or lesions not knowing a priori any exists, has been studied for over four decades. However, a satisfactory solution seems elusive.

MATERIALS AND METHODS

We show that the ROC curve predictive of clinical outcomes where readers are penalized appropriately for not correctly localizing known lesions cannot be obtained because it is a missing data problem. Further, this ROC curve is between the case-based ROC curve where readers are not penalized and the lesion-based ROC curve where penalty applies. Moreover, the lesion-based ROC curve is the LROC curve proposed by Starr et al. We show maximum-likelihood (ML) estimation of the LROC curve, validation of this procedure with Monte Carlo simulations, and its application to reader ROC datasets.

RESULTS

Monte Carlo simulations validated ML estimation of area under the LROC curve (AUC) and its variance. Example applications showed that ML estimate of LROC curve fits experimental datasets.

CONCLUSION

The ROC curve predictive of clinical performance cannot be estimated from reader ROC data alone because it is a missing data problem, and is between the case-based ROC curve where readers are not penalized for not correctly identifying known lesions and the lesion-based ROC curve where penalty applies. The lesion-based ROC curve is the LROC curve proposed by Starr et al. and can be estimated via ML estimation.

Computer-aided tumor detection in automated breast ultrasound using a 3-D convolutional neural network

BACKGROUND AND OBJECTIVES

Automated breast ultrasound (ABUS) is a widely used screening modality for breast cancer detection and diagnosis. In this study, an effective and fast computer-aided detection (CADe) system based on a 3-D convolutional neural network (CNN) is proposed as the second reader for the physician in order to decrease the reviewing time and misdetection rate.

METHODS

Our CADe system uses the sliding window method, a CNN-based determining model, and a candidate aggregation algorithm. First, the sliding window method is performed to split the ABUS volume into volumes of interest (VOIs). Afterward, VOIs are selected as tumor candidates by our determining model. To achieve higher performance, focal loss and ensemble learning are used to solve data imbalance and reduce false positive (FP) and false negative (FN) rates. Because several selected candidates may be part of the same tumor and they may overlap each other, a candidate aggregation method is applied to merge the overlapping candidates into the final detection result.

RESULTS

In the experiments, 165 and 81 cases are utilized for training the system and evaluating system performance, respectively. On evaluation with the 81 cases, our system achieves sensitivities of 100% (81/81), 95.3% (77/81), and 90.9% (74/81) with FPs per pass (per case) of 21.6 (126.2), 6.0 (34.8), and 4.6 (27.1) respectively. According to the results, the number of FPs per pass (per case) can be diminished by 56.8% (57.1%) at a sensitivity of 95.3% based on our tumor detection model.

CONCLUSIONS

In conclusion, our CADe system using 3-D CNN with the focal loss and ensemble learning may have the capability of being a tumor detection system in ABUS image.

Fast ScanNet: Fast and Dense Analysis of Multi-Gigapixel Whole-Slide Images for Cancer Metastasis Detection

Lymph node metastasis is one of the most important indicators in breast cancer diagnosis, that is traditionally observed under the microscope by pathologists. In recent years, with the dramatic advance of high-throughput scanning and deep learning technology, automatic analysis of histology from whole-slide images has received a wealth of interest in the field of medical image computing, which aims to alleviate pathologists' workload and simultaneously reduce misdiagnosis rate. However, the automatic detection of lymph node metastases from whole-slide images remains a key challenge because such images are typically very large, where they can often be multiple gigabytes in size. Also, the presence of hard mimics may result in a large number of false positives. In this paper, we propose a novel method with anchor layers for model conversion, which not only leverages the efficiency of fully convolutional architectures to meet the speed requirement in clinical practice but also densely scans the whole-slide image to achieve accurate predictions on both micro- and macro-metastases. Incorporating the strategies of asynchronous sample prefetching and hard negative mining, the network can be effectively trained. The efficacy of our method is corroborated on the benchmark dataset of 2016 Camelyon Grand Challenge. Our method achieved significant improvements in comparison with the state-of-the-art methods on tumor localization accuracy with a much faster speed and even surpassed human performance on both challenge tasks.

X-Ray Enhancement Based on Component Attenuation, Contrast Adjustment, and Image Fusion

Inspecting X-ray images is an essential aspect of medical diagnosis. However, due to an X-ray's low contrast and low dynamic range, important aspects such as organs, bones, and nodules become difficult to identify. Hence, contrast adjustment is critical, especially because of its ability to enhance the details in both bright and dark regions. For X-ray image enhancement, we therefore propose a new concept based on component attenuation. Notably, we assumed an X-ray image could be decomposed into tissue components and important details. Since tissues may not be the major primary focus of an X-ray, we proposed enhancing the visual contrast by adaptive tissue attenuation and dynamic range stretching. Via component decomposition and tissue attenuation, a parametric adjustment model was deduced to generate many enhanced images at once. Finally, an ensemble framework was proposed for fusing these enhanced images and producing a high-contrast output in both bright and dark regions. We have used measurement metrics to evaluate our system and achieved promising scores in each. An online testing system was also built for subjective evaluation. Moreover, we applied our system to an X-ray data set provided by the Japanese Society of Radiological Technology to help with nodule detection. The experimental results of which demonstrated the effectiveness of our method.

Tumor Detection in Automated Breast Ultrasound Using 3-D CNN and Prioritized Candidate Aggregation

Automated whole breast ultrasound (ABUS) has been widely used as a screening modality for examination of breast abnormalities. Reviewing hundreds of slices produced by ABUS, however, is time consuming. Therefore, in this paper, a fast and effective computer-aided detection system based on 3-D convolutional neural networks (CNNs) and prioritized candidate aggregation is proposed to accelerate this reviewing. First, an efficient sliding window method is used to extract volumes of interest (VOIs). Then, each VOI is estimated the tumor probability with a 3-D CNN, and VOIs with higher estimated probability are selected as tumor candidates. Since the candidates may overlap each other, a novel scheme is designed to aggregate the overlapped candidates. During the aggregation, candidates are prioritized based on estimated tumor probability to alleviate over-aggregation issue. The relationship between the sizes of VOI and target tumor is optimally exploited to effectively perform each stage of our detection algorithm. On evaluation with a test set of 171 tumors, our method achieved sensitivities of 95% (162/171), 90% (154/171), 85% (145/171), and 80% (137/171) with 14.03, 6.92, 4.91, and 3.62 false positives per patient (with six passes), respectively. In summary, our method is more general and much faster than preliminary works and demonstrates promising results.

Evaluation of diagnostic accuracy in free-response detection-localization tasks using ROC tools

Diagnostic systems designed to detect possibly multiple lesions per patient (e.g. multiple polyps during CT colonoscopy) are often evaluated in “free-response” studies that allow for diagnostic responses unconstrained in their number and locations. Analysis of free-response studies requires extensions of the traditional receiver operating characteristic (ROC) analysis, which are termed free-response ROC (FROC) methodology. Despite substantial developments in this area, FROC tools and approaches are much more cumbersome than traditional ROC methods. Alternative approaches that use well-known ROC tools (e.g. ROI-ROC) require defining and physically delineating regions of interest (ROI) and combine FROC data within ROIs. We propose an approach that allows analyzing FROC data using conventional ROC tools without delineating the actual ROIs or reducing data. The design parameters of FROC study are used to make FROC data analyzable using ROC tools and to calibrate the corresponding FROC and ROC curves on both conceptual and numerical levels. Differences in the performance indices of the nonparametric FROC and the new approach are shown to be asymptotically negligible and typically rather small in practice. Data from a large multi-reader study of colon cancer detection are used to illustrate the new approach.

Detection accuracy of maxillary sinus floor septa in panoramic radiographs using CBCT as gold standard: a multi-observer receiver operating characteristic (ROC) study

OBJECTIVES

To investigate diagnostic accuracy of panoramic radiography in detecting maxillary sinus floor septa by means of a multi-observer receiver operating characteristic (ROC) analysis and a standardized protocol for reporting (STARD protocol; Clin Chem 49(1):1-6, 2003).

MATERIAL AND METHODS

From our database, 25 cone beam computed tomographies (CBCTs) were selected with one maxillary sinus floor septum (height ≥ 2.5 mm). For the same patient, a recent panoramic radiograph (PAN) had to be available in the database. As controls, 28 CBCTs plus corresponding PANs without evidence of a sinus septum were selected. Using the CBCTs as ground truth, 17 observers from our dental school on a five-point confidence scale rated both sinuses in all 53 PANs with respect to presence/absence of a sinus septum. Areas beneath ROC curves (Az-values), sensitivity/specificity (SNT/SPF), positive/negative predictive values (PPV, NPV), and positive/negative likelihood ratios (LR+, LR-) were computed for each observer and pooled over all observers. Inter-rater reproducibility was assessed by means of the intraclass coefficient (ICC) using a two-way random effects model.

RESULTS

A pooled Az-value of 0.839 was observed (SNT 84.6%, SPF 73.5%). PPV ranged between 0.492 and 0.824 (median 0.627) and NPV between 0.838 and 0.976 (median 0.917). A median LR+ of 3.567 was computed (LR- median 0.193). Inter-rater reliability revealed an ICC of 0.55 (95% confidence interval 0.48 to 0.62).

CONCLUSIONS

Our results indicate that PAN is a moderately accurate method for sinus elevation planning for the purpose of septum detection. Ruling out a septum by PAN seems to work more accurately than ruling in.

CLINICAL RELEVANCE

For the purpose of maxillary sinus floor septa detection, panoramic radiography can be relatively safely advocated, particularly for judgment of a septum-free sinus.

The Reproducibility of Changes in Diagnostic Figures of Merit Across Laboratory and Clinical Imaging Reader Studies

RATIONALE AND OBJECTIVES

In this paper we examine which comparisons of reading performance between diagnostic imaging systems made in controlled retrospective laboratory studies may be representative of what we observe in later clinical studies. The change in a meaningful diagnostic figure of merit between two diagnostic modalities should be qualitatively or quantitatively comparable across all kinds of studies.

MATERIALS AND METHODS

In this meta-study we examine the reproducibility of relative measures of sensitivity, false positive fraction (FPF), area under the receiver operating characteristic (ROC) curve, and expected utility across laboratory and observational clinical studies for several different breast imaging modalities, including screen film mammography, digital mammography, breast tomosynthesis, and ultrasound.

RESULTS

Across studies of all types, the changes in the FPFs yielded very small probabilities of having a common mean value. The probabilities of relative sensitivity being the same across ultrasound and tomosynthesis studies were low. No evidence was found for different mean values of relative area under the ROC curve or relative expected utility within any of the study sets.

CONCLUSION

The comparison demonstrates that the ratios of areas under the ROC curve and expected utilities are reproducible across laboratory and clinical studies, whereas sensitivity and FPF are not.

Technical Note: Display window setting: An important factor for detecting subtle but clinically relevant artifacts in daily CT quality control

PURPOSE

This study aimed to investigate the influence of display window setting on technologist performance detecting subtle but clinically relevant artifacts in daily computed tomography (CT) quality control (dQC) images.

METHODS

Fifty three sets of dQC images were retrospectively selected, including 30 sets without artifacts, and 23 with subtle but clinically relevant artifacts. They were randomized and shown to six CT technologists (two new and four experienced). Each technologist reviewed all images in each of two sessions, one with a display window width (WW) of 100 HU, which is currently recommended by the American College of Radiology, and the other with a narrow WW of 40 HU, both at a window level of 0 HU. For each case, technologists rated the presence of image artifacts based on a five point scale. The area under the receiver operating characteristic curve (AUC) was used to evaluate the artifact detection performance.

RESULTS

At a WW of 100 HU, the AUC (95% confidence interval) was 0.658 (0.576, 0.740), 0.532 (0.429, 0.635), and 0.616 (0.543, 0.619) for the experienced, new, and all technologists, respectively. At a WW of 40 HU, the AUC was 0.768 (0.687, 0.850), 0.546 (0.433, 0.658), and 0.694 (0.619, 0.769), respectively. The performance significantly improved at WW of 40 HU for experienced technologists (p = 0.009) and for all technologists (p = 0.040).

CONCLUSIONS

Use of a narrow display WW significantly improved technologists' performance in dQC for detecting subtle but clinically relevant artifacts as compared to that using a 100 HU display WW.

On the meaning of the weighted alternative free-response operating characteristic figure of merit

PURPOSE

The free-response receiver operating characteristic (FROC) method is being increasingly used to evaluate observer performance in search tasks. Data analysis requires definition of a figure of merit (FOM) quantifying performance. While a number of FOMs have been proposed, the recommended one, namely, the weighted alternative FROC (wAFROC) FOM, is not well understood. The aim of this work is to clarify the meaning of this FOM by relating it to the empirical area under a proposed wAFROC curve.

METHODS

The weighted wAFROC FOM is defined in terms of a quasi-Wilcoxon statistic that involves weights, coding the clinical importance, assigned to each lesion. A new wAFROC curve is proposed, the y-axis of which incorporates the weights, giving more credit for marking clinically important lesions, while the x-axis is identical to that of the AFROC curve. An expression is derived relating the area under the empirical wAFROC curve to the wAFROC FOM. Examples are presented with small numbers of cases showing how AFROC and wAFROC curves are affected by correct and incorrect decisions and how the corresponding FOMs credit or penalize these decisions. The wAFROC, AFROC, and inferred ROC FOMs were applied to three clinical data sets involving multiple reader FROC interpretations in different modalities.

RESULTS

It is shown analytically that the area under the empirical wAFROC curve equals the wAFROC FOM. This theorem is the FROC analog of a well-known theorem developed in 1975 for ROC analysis, which gave meaning to a Wilcoxon statistic based ROC FOM. A similar equivalence applies between the area under the empirical AFROC curve and the AFROC FOM. The examples show explicitly that the wAFROC FOM gives equal importance to all diseased cases, regardless of the number of lesions, a desirable statistical property not shared by the AFROC FOM. Applications to the clinical data sets show that the wAFROC FOM yields results comparable to that using the AFROC FOM.

CONCLUSIONS

The equivalence theorem gives meaning to the weighted AFROC FOM, namely, it is identical to the empirical area under weighted AFROC curve.

Quantification of hazard prediction ability at hazard prediction training (Kiken-Yochi Training: KYT) by free-response receiver-operating characteristic (FROC) analysis

The ability to predict hazards in possible situations in a general X-ray examination room created for Kiken-Yochi training (KYT) is quantified by use of free-response receiver-operating characteristics (FROC) analysis for determining whether the total number of years of clinical experience, involvement in general X-ray examinations, occupation, and training each have an impact on the hazard prediction ability. Twenty-three radiological technologists (RTs) (years of experience: 2–28), four nurses (years of experience: 15–19), and six RT students observed 53 scenes of KYT: 26 scenes with hazardous points (hazardous points are those that might cause injury to patients) and 27 scenes without points. Based on the results of these observations, we calculated the alternative free-response receiver-operating characteristic (AFROC) curve and the figure of merit (FOM) to quantify the hazard prediction ability. The results showed that the total number of years of clinical experience did not have any impact on hazard prediction ability, whereas recent experience with general X-ray examinations greatly influenced this ability. In addition, the hazard prediction ability varied depending on the occupations of the observers while they were observing the same scenes in KYT. The hazard prediction ability of the radiologic technology students was improved after they had undergone patient safety training. This proposed method with FROC observer study enabled the quantification and evaluation of the hazard prediction capability, and the application of this approach to clinical practice may help to ensure the safety of examinations and treatment in the radiology department.

Observer performance for adaptive, image-based denoising and filtered back projection compared to scanner-based iterative reconstruction for lower dose CT enterography

PURPOSE

The purpose of this study was to compare observer performance for detection of intestinal inflammation for low-dose CT enterography (LD-CTE) using scanner-based iterative reconstruction (IR) vs. vendor-independent, adaptive image-based noise reduction (ANLM) or filtered back projection (FBP).

METHODS

Sixty-two LD-CTE exams were performed. LD-CTE images were reconstructed using IR, ANLM, and FBP. Three readers, blinded to image type, marked intestinal inflammation directly on patient images using a specialized workstation over three sessions, interpreting one image type/patient/session. Reference standard was created by a gastroenterologist and radiologist, who reviewed all available data including dismissal Gastroenterology records, and who marked all inflamed bowel segments on the same workstation. Reader and reference localizations were then compared. Non-inferiority was tested using Jackknife free-response ROC (JAFROC) figures of merit (FOM) for ANLM and FBP compared to IR. Patient-level analyses for the presence or absence of inflammation were also conducted.

RESULTS

There were 46 inflamed bowel segments in 24/62 patients (CTDIvol interquartile range 6.9-10.1 mGy). JAFROC FOM for ANLM and FBP were 0.84 (95% CI 0.75-0.92) and 0.84 (95% CI 0.75-0.92), and were statistically non-inferior to IR (FOM 0.84; 95% CI 0.76-0.93). Patient-level pooled confidence intervals for sensitivity widely overlapped, as did specificities. Image quality was rated as better with IR and AMLM compared to FBP (p < 0.0001), with no difference in reading times (p = 0.89).

CONCLUSIONS

Vendor-independent adaptive image-based noise reduction and FBP provided observer performance that was non-inferior to scanner-based IR methods. Adaptive image-based noise reduction maintained or improved upon image quality ratings compared to FBP when performing CTE at lower dose levels.

Pulmonary Embolism Detection with Three-dimensional Ultrashort Echo Time MR Imaging: Experimental Study in Canines

PURPOSE

To demonstrate the feasibility of free-breathing three-dimensional (3D) radial ultrashort echo time (UTE) magnetic resonance (MR) imaging in the simultaneous detection of pulmonary embolism (PE) and high-quality evaluation of lung parenchyma.

MATERIALS AND METHODS

The institutional animal care committee approved this study. A total of 12 beagles underwent MR imaging and computed tomography (CT) before and after induction of PE with autologous clots. Breath-hold 3D MR angiography and free-breathing 3D radial UTE (1.0-mm isotropic spatial resolution; echo time, 0.08 msec) were performed at 3 T. Two blinded radiologists independently marked and graded all PEs on a four-point scale (1 = low confidence, 4 = absolutely certain) on MR angiographic and UTE images. Image quality of pulmonary arteries and lung parenchyma was scored on a four-point-scale (1 = poor, 4 = excellent). Locations and ratings of emboli were compared with reference standard CT images by using an alternative free-response receiver operating characteristic curve (AFROC) method. Areas under the curve and image quality ratings were compared by using the F test and the Wilcoxon signed-rank test.

RESULTS

A total of 48 emboli were detected with CT. Both readers showed higher sensitivity for PE detection with UTE (83% and 79%) than with MR angiography (75% and 71%). The AFROC area under the curve was higher for UTE than for MR angiography (0.95 vs 0.89), with a significant difference in area under the curve of 0.06 (95% confidence interval: 0.01, 0.11; P = .018). UTE image quality exceeded that of MR angiography for subsegmental arteries (3.5 ± 0.7 vs 2.9 ± 0.5, P = .002) and lung parenchyma (3.8 ± 0.5 vs 2.2 ± 0.2, P < .001). The apparent signal-to-noise ratio in pulmonary arteries and lung parenchyma was significantly higher for UTE than for MR angiography (41.0 ± 5.2 vs 24.5 ± 6.2 [P < .001] and 10.2 ± 1.8 vs 3.5 ± 0.8 [P < .001], respectively). The apparent contrast-to-noise ratio between arteries and PEs was higher for UTE than for MR angiography (20.3 ± 5.2 vs 15.4 ± 6.7, P = .055).

CONCLUSION

In a canine model, free-breathing 3D radial UTE performs better than breath-hold 3D MR angiography in the detection of PE and yields better image quality for visualization of small vessels and lung parenchyma. Free-breathing 3D radial UTE for detection of PE is feasible and warrants evaluation in human subjects.

Observer Performance in the Detection and Classification of Malignant Hepatic Nodules and Masses with CT Image-Space Denoising and Iterative Reconstruction

PURPOSE

To determine if lower-dose computed tomographic (CT) scans obtained with adaptive image-based noise reduction (adaptive nonlocal means [ANLM]) or iterative reconstruction (sinogram-affirmed iterative reconstruction [SAFIRE]) result in reduced observer performance in the detection of malignant hepatic nodules and masses compared with routine-dose scans obtained with filtered back projection (FBP).

MATERIALS AND METHODS

This study was approved by the institutional review board and was compliant with HIPAA. Informed consent was obtained from patients for the retrospective use of medical records for research purposes. CT projection data from 33 abdominal and 27 liver or pancreas CT examinations were collected (median volume CT dose index, 13.8 and 24.0 mGy, respectively). Hepatic malignancy was defined by progression or regression or with histopathologic findings. Lower-dose data were created by using a validated noise insertion method (10.4 mGy for abdominal CT and 14.6 mGy for liver or pancreas CT) and images reconstructed with FBP, ANLM, and SAFIRE. Four readers evaluated routine-dose FBP images and all lower-dose images, circumscribing liver lesions and selecting diagnosis. The jackknife free-response receiver operating characteristic figure of merit (FOM) was calculated on a per-malignant nodule or per-mass basis. Noninferiority was defined by the lower limit of the 95% confidence interval (CI) of the difference between lower-dose and routine-dose FOMs being less than -0.10.

RESULTS

Twenty-nine patients had 62 malignant hepatic nodules and masses. Estimated FOM differences between lower-dose FBP and lower-dose ANLM versus routine-dose FBP were noninferior (difference: -0.041 [95% CI: -0.090, 0.009] and -0.003 [95% CI: -0.052, 0.047], respectively). In patients with dedicated liver scans, lower-dose ANLM images were noninferior (difference: +0.015 [95% CI: -0.077, 0.106]), whereas lower-dose FBP images were not (difference -0.049 [95% CI: -0.140, 0.043]). In 37 patients with SAFIRE reconstructions, the three lower-dose alternatives were found to be noninferior to the routine-dose FBP.

CONCLUSION

At moderate levels of dose reduction, lower-dose FBP images without ANLM or SAFIRE were noninferior to routine-dose images for abdominal CT but not for liver or pancreas CT.

Modeling false positive error making patterns in radiology trainees for improved mammography education

INTRODUCTION

While mammography notably contributes to earlier detection of breast cancer, it has its limitations, including a large number of false positive exams. Improved radiology education could potentially contribute to alleviating this issue. Toward this goal, in this paper we propose an algorithm for modeling of false positive error making among radiology trainees. Identifying troublesome locations for the trainees could focus their training and in turn improve their performance.

METHODS

The algorithm proposed in this paper predicts locations that are likely to result in a false positive error for each trainee based on the previous annotations made by the trainee. The algorithm consists of three steps. First, the suspicious false positive locations are identified in mammograms by Difference of Gaussian filter and suspicious regions are segmented by computer vision-based segmentation algorithms. Second, 133 features are extracted for each suspicious region to describe its distinctive characteristics. Third, a random forest classifier is applied to predict the likelihood of the trainee making a false positive error using the extracted features. The random forest classifier is trained using previous annotations made by the trainee. We evaluated the algorithm using data from a reader study in which 3 experts and 10 trainees interpreted 100 mammographic cases.

RESULTS

The algorithm was able to identify locations where the trainee will commit a false positive error with accuracy higher than an algorithm that selects such locations randomly. Specifically, our algorithm found false positive locations with 40% accuracy when only 1 location was selected for all cases for each trainee and 12% accuracy when 10 locations were selected. The accuracies for randomly identified locations were both 0% for these two scenarios.

CONCLUSIONS

In this first study on the topic, we were able to build computer models that were able to find locations for which a trainee will make a false positive error in images that were not previously seen by the trainee. Presenting the trainees with such locations rather than randomly selected ones may improve their educational outcomes.

Feature selection for the automated detection of metaphase chromosomes: performance comparison using a receiver operating characteristic method

BACKGROUND

The purpose of this study is to identify a set of features for optimizing the performance of metaphase chromosome detection under high throughput scanning microscopy. In the development of computer-aided detection (CAD) scheme, feature selection is critically important, as it directly determines the accuracy of the scheme. Although many features have been examined previously, selecting optimal features is often application oriented.

METHODS

In this experiment, 200 bone marrow cells were first acquired by a high throughput scanning microscope. Then 9 different features were applied individually to group captured images into the clinically analyzable and unanalyzable classes. The performance of these different methods was assessed by a receiving operating characteristic (ROC) method.

RESULTS

The results show that using the number of labeled regions on each acquired image is suitable for the first on-line CAD scheme. For the second off-line CAD scheme, it would be suggested to combine four feature extraction methods including the number of labeled regions, average regions area, average region pixel value, and the standard deviation of either region distance or circularity.

CONCLUSION

This study demonstrates an effective method of feature selection and comparison to facilitate the optimization of the CAD schemes for high throughput scanning microscope in the future.

Multi-dimensional tumor detection in automated whole breast ultrasound using topographic watershed

Automated whole breast ultrasound (ABUS) is becoming a popular screening modality for whole breast examination. Compared to conventional handheld ultrasound, ABUS achieves operator-independent and is feasible for mass screening. However, reviewing hundreds of slices in an ABUS image volume is time-consuming. A computer-aided detection (CADe) system based on watershed transform was proposed in this study to accelerate the reviewing. The watershed transform was applied to gather similar tissues around local minima to be homogeneous regions. The likelihoods of being tumors of the regions were estimated using the quantitative morphology, intensity, and texture features in the 2-D/3-D false positive reduction (FPR). The collected database comprised 68 benign and 65 malignant tumors. As a result, the proposed system achieved sensitivities of 100% (133/133), 90% (121/133), and 80% (107/133) with FPs/pass of 9.44, 5.42, and 3.33, respectively. The figure of merit of the combination of three feature sets is 0.46 which is significantly better than that of other feature sets ( [Formula: see text]). In summary, the proposed CADe system based on the multi-dimensional FPR using the integrated feature set is promising in detecting tumors in ABUS images.

Computer-aided diagnosis improves detection of small intracranial aneurysms on MRA in a clinical setting

BACKGROUND AND PURPOSE

MRA is widely accepted as a noninvasive diagnostic tool for the detection of intracranial aneurysms, but detection is still a challenging task with rather low detection rates. Our aim was to examine the performance of a computer-aided diagnosis algorithm for detecting intracranial aneurysms on MRA in a clinical setting.

MATERIALS AND METHODS

Aneurysm detectability was evaluated retrospectively in 48 subjects with and without computer-aided diagnosis by 6 readers using a clinical 3D viewing system. Aneurysms ranged from 1.1 to 6.0 mm (mean = 3.12 mm, median = 2.50 mm). We conducted a multireader, multicase, double-crossover design, free-response, observer-performance study on sets of images from different MRA scanners by using DSA as the reference standard. Jackknife alternative free-response operating characteristic curve analysis with the figure of merit was used.

RESULTS

For all readers combined, the mean figure of merit improved from 0.655 to 0.759, indicating a change in the figure of merit attributable to computer-aided diagnosis of 0.10 (95% CI, 0.03-0.18), which was statistically significant (F(1,47) = 7.00, P = .011). Five of the 6 radiologists had improved performance with computer-aided diagnosis, primarily due to increased sensitivity.

CONCLUSIONS

In conditions similar to clinical practice, using computer-aided diagnosis significantly improved radiologists' detection of intracranial DSA-confirmed aneurysms of ≤6 mm.

Cluster size statistic and cluster mass statistic: two novel methods for identifying changes in functional connectivity between groups or conditions

Functional connectivity has become an increasingly important area of research in recent years. At a typical spatial resolution, approximately 300 million connections link each voxel in the brain with every other. This pattern of connectivity is known as the functional connectome. Connectivity is often compared between experimental groups and conditions. Standard methods used to control the type 1 error rate are likely to be insensitive when comparisons are carried out across the whole connectome, due to the huge number of statistical tests involved. To address this problem, two new cluster based methods--the cluster size statistic (CSS) and cluster mass statistic (CMS)--are introduced to control the family wise error rate across all connectivity values. These methods operate within a statistical framework similar to the cluster based methods used in conventional task based fMRI. Both methods are data driven, permutation based and require minimal statistical assumptions. Here, the performance of each procedure is evaluated in a receiver operator characteristic (ROC) analysis, utilising a simulated dataset. The relative sensitivity of each method is also tested on real data: BOLD (blood oxygen level dependent) fMRI scans were carried out on twelve subjects under normal conditions and during the hypercapnic state (induced through the inhalation of 6% CO2 in 21% O2 and 73%N2). Both CSS and CMS detected significant changes in connectivity between normal and hypercapnic states. A family wise error correction carried out at the individual connection level exhibited no significant changes in connectivity.

Evaluation of computer-aided detection and diagnosis systems

Computer-aided detection and diagnosis (CAD) systems are increasingly being used as an aid by clinicians for detection and interpretation of diseases. Computer-aided detection systems mark regions of an image that may reveal specific abnormalities and are used to alert clinicians to these regions during image interpretation. Computer-aided diagnosis systems provide an assessment of a disease using image-based information alone or in combination with other relevant diagnostic data and are used by clinicians as a decision support in developing their diagnoses. While CAD systems are commercially available, standardized approaches for evaluating and reporting their performance have not yet been fully formalized in the literature or in a standardization effort. This deficiency has led to difficulty in the comparison of CAD devices and in understanding how the reported performance might translate into clinical practice. To address these important issues, the American Association of Physicists in Medicine (AAPM) formed the Computer Aided Detection in Diagnostic Imaging Subcommittee (CADSC), in part, to develop recommendations on approaches for assessing CAD system performance. The purpose of this paper is to convey the opinions of the AAPM CADSC members and to stimulate the development of consensus approaches and "best practices" for evaluating CAD systems. Both the assessment of a standalone CAD system and the evaluation of the impact of CAD on end-users are discussed. It is hoped that awareness of these important evaluation elements and the CADSC recommendations will lead to further development of structured guidelines for CAD performance assessment. Proper assessment of CAD system performance is expected to increase the understanding of a CAD system's effectiveness and limitations, which is expected to stimulate further research and development efforts on CAD technologies, reduce problems due to improper use, and eventually improve the utility and efficacy of CAD in clinical practice.

Automated pulmonary nodule detection based on three-dimensional shape-based feature descriptor

Computer-aided detection (CAD) can help radiologists to detect pulmonary nodules at an early stage. In pulmonary nodule CAD systems, feature extraction is very important for describing the characteristics of nodule candidates. In this paper, we propose a novel three-dimensional shape-based feature descriptor to detect pulmonary nodules in CT scans. After lung volume segmentation, nodule candidates are detected using multi-scale dot enhancement filtering in the segmented lung volume. Next, we extract feature descriptors from the detected nodule candidates, and these are refined using an iterative wall elimination method. Finally, a support vector machine-based classifier is trained to classify nodules and non-nodules. The performance of the proposed system is evaluated on Lung Image Database Consortium data. The proposed method significantly reduces the number of false positives in nodule candidates. This method achieves 97.5% sensitivity, with only 6.76 false positives per scan.

A brief history of free-response receiver operating characteristic paradigm data analysis

In the receiver operating characteristic paradigm the observer assigns a single rating to each image and the location of the perceived abnormality, if any, is ignored. In the free-response receiver operating characteristic paradigm the observer is free to mark and rate as many suspicious regions as are considered clinically reportable. Credit for a correct localization is given only if a mark is sufficiently close to an actual lesion; otherwise, the observer's mark is scored as a location-level false positive. Until fairly recently there existed no accepted method for analyzing the resulting relatively unstructured data containing random numbers of mark-rating pairs per image. This report reviews the history of work in this field, which has now spanned more than five decades. It introduces terminology used to describe the paradigm, proposed measures of performance (figures of merit), ways of visualizing the data (operating characteristics), and software for analyzing free-response receiver operating characteristic studies.

Subject-centered free-response ROC (FROC) analysis

PURPOSE

To develop an approach of estimating subject-centered free-response receiver operating characteristic (FROC) curve for providing patient-centered inferences regarding detection-localization characteristics of a diagnostic system.

METHODS

The authors examine properties of the conventional, target-centered, FROC curve and demonstrate that in scenarios where the diagnostic performance correlates with the total number of targets on a subject, it may lead to inadequate inferences from the perspective of possible benefits to a patient. Following solutions to patient-centered approaches in other applications, the authors define a subject-centered FROC curve and develop its formulation as a covariate-adjusted FROC curve. The authors also conduct a numerical study illustrating the relative properties of the conventional and subject-centered approach and provide an example.

RESULTS

A simple-to-implement approach for estimating the subject-centered FROC curve and its overall index can be formulated as a type of stratified FROC analysis. The authors demonstrate that when diagnostic performance is associated with the number of targets, the diagnostic system with apparently superior target-centered characteristics (conventional approach) can be actually inferior from the subject-centered perspective. The authors show that under some clinically reasonable conditions the magnitude of disagreement in results could be substantial. An example from an actual observer performance study illustrates the natural setting where the developed approach would be relevant and lead to conclusions that are contradictory to those obtained from conventional analysis.

CONCLUSIONS

The authors developed a subject-centered FROC curve and its overall index provides tools for inferences that may be relevant from a perspective of potential benefits to a patient.

A new ROC analysis method considering the correlation between neighboring pixels

In this paper, we introduce a novel receiver operating characteristic (ROC) analysis method that considers spatial correlation between pixels to evaluate classification algorithms. ROC analysis is one of the most important tools in the evaluation of medical images and computer aided diagnosis (CAD) systems. It provides a comprehensive description of the detection accuracy of the test image. To evaluate the localization performance, operating points of ROC curves are obtained based on the classification results of individual pixels. To this date, the confidence level or intensity value of each pixel is assumed to be independent within the image. However, this assumption is not satisfied in real problems. In this paper, a new ROC analysis algorithm that considers the correlation between neighboring pixels is proposed. Our results show that the new ROC curves provide a more accurate evaluation of the test image.

Whole breast lesion detection using naive bayes classifier for portable ultrasound

In recent years, portable PC-based ultrasound (US) imaging systems developed by some companies can provide an integrated computer environment for computer-aided diagnosis and detection applications. In this article, an automatic whole breast lesion detection system based on the naive Bayes classifier using the PC-based US system Terason t3000 (Terason Ultrasound, Burlington, MA, USA) with a hand-held probe is proposed. To easily retrieve the US images for any regions of the breast, a clock-based storing system is proposed to record the scanned US images. A computer-aided detection (CAD) system is also developed to save the physicians' time for a huge volume of scanned US images. The pixel classification of the US is based on the naive Bayes classifier for the proposed lesion detection system. The pixels of the US are classified into two types: lesions or normal tissues. The connected component labeling is applied to find the suspected lesions in the image. Consequently, the labeled two-dimensional suspected regions are separated into two clusters and further checked by two-phase lesion selection criteria for the determination of the real lesion, while reducing the false-positive rate. The free-response operative characteristics (FROC) curve is used to evaluate the detection performance of the proposed system. According to the experimental results of 31 cases with 33 lesions, the proposed system yields a 93.4% (31/33) sensitivity at 4.22 false positives (FPs) per hundred slices. Moreover, the speed for the proposed detection scheme achieves 12.3 frames per second (fps) with an Intel Dual-Core Quad 3 GHz processor and can be also effectively and efficiently used for other screening systems.

Breast imaging: understanding how accuracy is measured when lesions are the unit of analysis

Medical imaging tests of breast cancer patients can be used to detect and provide information on the location of multiple malignant lesions within a patient. Within this context, it is often the case that one needs to evaluate the accuracy of an imaging test for finding the multiple lesions in a patient rather than simply detecting that a patient has disease. A natural way to approach this task is to estimate the accuracy of the test using a lesion-level analysis. Sensitivity, specificity, and receiver operating characteristic (ROC) curves are analytic measures that are frequently used to quantify the accuracy of medical tests. When the test or radiologist must first locate the lesions, however, it is not possible to directly estimate the specificity or an ROC curve keeping the individual lesions as the unit of analysis. The goal of this study is to demonstrate to clinicians conducting or reviewing studies evaluating breast imaging tests what measures of accuracy can and cannot be calculated in different types of studies and to describe in detail the difficulty with calculating specificity and ROC curves in a lesion-level analysis.

Evaluating imaging and computer-aided detection and diagnosis devices at the FDA

This report summarizes the Joint FDA-MIPS Workshop on Methods for the Evaluation of Imaging and Computer-Assist Devices. The purpose of the workshop was to gather information on the current state of the science and facilitate consensus development on statistical methods and study designs for the evaluation of imaging devices to support US Food and Drug Administration submissions. Additionally, participants expected to identify gaps in knowledge and unmet needs that should be addressed in future research. This summary is intended to document the topics that were discussed at the meeting and disseminate the lessons that have been learned through past studies of imaging and computer-aided detection and diagnosis device performance.

New developments in observer performance methodology in medical imaging

A common task in medical imaging is assessing whether a new imaging system, or a variant of an existing one, is an improvement over an existing imaging technology. Imaging systems are generally quite complex, consisting of several components-for example, image acquisition hardware, image processing and display hardware and software, and image interpretation by radiologists- each of which can affect performance. Although it may appear odd to include the radiologist as a "component" of the imaging chain, because the radiologist's decision determines subsequent patient care, the effect of the human interpretation has to be included. Physical measurements such as modulation transfer function, signal-to-noise ratio, are useful for characterizing the nonhuman parts of the imaging chain under idealized and often unrealistic conditions, such as uniform background phantoms and target objects with sharp edges. Measuring the performance of the entire imaging chain, including the radiologist, and using real clinical images requires different methods that fall under the rubric of observer performance methods or "ROC" analysis, that involve collecting rating data on images. The purpose of this work is to review recent developments in this field, particularly with respect to the free-response method, where location information is also collected.

Nonparametric signal detectability evaluation using an exponential transformation of the FROC curve

PURPOSE

To develop an efficient nonparametric method for evaluation the detectability of signals at unknown locations in images, as a mean for image quality assessment.

METHODS

We use the free-response methodology that allows the image observer to mark and score all locations found as suspicious in an image, summarizing these results in a free-response operating characteristic (FROC) curve. However, unlike the relative (or receiver) operating characteristic (ROC), or the localization ROC (LROC), the FROC curve has an undefined, theoretically infinite, right side limit. Therefore area under the FROC curves cannot be directly used as an overall performance index, as the area under the curve is for ROC or LROC. We circumvent this drawback by using a transformation of the abscissa that leads to a finite integration range. By applying an exponential transformation we derive a nonparametric estimator for such a metric, and we study its properties by deriving analytical expressions for the mean and standard deviation in conditions of scores independence.

RESULTS

A comparative study with other related nonparametric estimators for ROC, LROC, and alternative FROC (AFROC) method is presented.

CONCLUSIONS

The new nonparametric estimator has sensitivity and scalability properties that make it particularly advantageous for signal detectability evaluation in phantom experiments using model observers.

Computer-aided detection scheme for sentinel lymph nodes in lymphoscintigrams using symmetrical property around mapped injection point

It is difficult to detect sentinel lymph nodes (SLNs) around an injection point of radiopharmaceuticals mapped in lymphoscintigrams. The purpose of this study was to develop a computer-aided detection (CAD) scheme for SLNs by a subtraction technique using the symmetrical property in the mapped injection point. Our database consisted of 78 lymphoscintigrams with 86 SLNs. In our CAD scheme, the mapped injection point of radiopharmaceuticals was first segmented from the lymphoscintigram using a gray-level thresholding technique. Lymphoscintigram was then divided into four regions by vertical and horizontal straight lines through the center of the segmented injection point. One of the four divided regions was defined as the target region. The correlation coefficients based on pixel values were calculated between the target region and each of the other three regions. The region with the highest correlation coefficient among three regions was selected as the similar region to the target region. The values of pixels on the target region were subtracted by the values of the corresponding pixels on the similar region. This procedure was repeated until every divided region had been used as target region. SLNs were segmented by applying a gray-level thresholding technique to the subtracted image. With our CAD scheme, sensitivity and the number of false positives were 95.3% (82/86) and 2.51 per image, respectively. Our CAD scheme achieved a high level of detection accuracy, and would have a great potential in assisting physicians to detect SLNs in lymphoscintigrams.

Development and evaluation of a computer-aided diagnostic scheme for lung nodule detection in chest radiographs by means of two-stage nodule enhancement with support vector classification

PURPOSE

To develop a computer-aided detection (CADe) scheme for nodules in chest radiographs (CXRs) with a high sensitivity and a low false-positive (FP) rate.

METHODS

The authors developed a CADe scheme consisting of five major steps, which were developed for improving the overall performance of CADe schemes. First, to segment the lung fields accurately, the authors developed a multisegment active shape model. Then, a two-stage nodule-enhancement technique was developed for improving the conspicuity of nodules. Initial nodule candidates were detected and segmented by using the clustering watershed algorithm. Thirty-one shape-, gray-level-, surface-, and gradient-based features were extracted from each segmented candidate for determining the feature space, including one of the new features based on the Canny edge detector to eliminate a major FP source caused by rib crossings. Finally, a nonlinear support vector machine (SVM) with a Gaussian kernel was employed for classification of the nodule candidates.

RESULTS

To evaluate and compare the scheme to other published CADe schemes, the authors used a publicly available database containing 140 nodules in 140 CXRs and 93 normal CXRs. The CADe scheme based on the SVM classifier achieved sensitivities of 78.6% (110/140) and 71.4% (100/140) with averages of 5.0 (1165/233) FPs/image and 2.0 (466/233) FPs/image, respectively, in a leave-one-out cross-validation test, whereas the CADe scheme based on a linear discriminant analysis classifier had a sensitivity of 60.7% (85/140) at an FP rate of 5.0 FPs/image. For nodules classified as "very subtle" and "extremely subtle," a sensitivity of 57.1% (24/42) was achieved at an FP rate of 5.0 FPs/image. When the authors used a database developed at the University of Chicago, the sensitivities was 83.3% (40/48) and 77.1% (37/48) at an FP rate of 5.0 (240/48) FPs/image and 2.0 (96/48) FPs/image, respectively.

CONCLUSIONS

These results compare favorably to those described for other commercial and non-commercial CADe nodule detection systems.

Biplane correlation imaging: a feasibility study based on phantom and human data

The objective of this study was to implement and evaluate the performance of a biplane correlation imaging (BCI) technique aimed to reduce the effect of anatomic noise and improve the detection of lung nodules in chest radiographs. Seventy-one low-dose posterior-anterior images were acquired from an anthropomorphic chest phantom with 0.28° angular separations over a range of ±10° along the vertical axis within an 11 s interval. Similar data were acquired from 19 human subjects with institutional review board approval and informed consent. The data were incorporated into a computer-aided detection (CAD) algorithm in which suspect lesions were identified by examining the geometrical correlation of the detected signals that remained relatively constant against variable anatomic backgrounds. The data were analyzed to determine the effect of angular separation, and the overall sensitivity and false-positives for lung nodule detection. The best performance was achieved for angular separations of the projection pairs greater than 5°. Within that range, the technique provided an order of magnitude decrease in the number of false-positive reports when compared with CAD analysis of single-view images. Overall, the technique yielded ~1.1 false-positive per patient with an average sensitivity of 75%. The results indicated that the incorporation of angular information can offer a reduction in the number of false-positives without a notable reduction in sensitivity. The findings suggest that the BCI technique has the potential for clinical implementation as a cost-effective technique to improve the detection of subtle lung nodules with lowered rate of false-positives.

Digital magnification mammography with matched incident exposure: physical imaging properties and detectability of simulated microcalcifications

Our purpose was to evaluate the usefulness of digital magnification mammography with matched incident exposure by investigating the physical imaging properties and doing an observer performance test. A computed radiography system and a mammographic unit were used in this study. Contact and magnification radiographies of 1.2-1.8 in combination with focal spot sizes of 0.1 mm without grid and 0.3 mm with grid were performed. Physical imaging properties, namely, scatter fraction, total modulation transfer function (MTF) including the presampled MTF and the MTF of focal spot size, and Wiener spectrum (WS), were measured. Detail visibility was evaluated by use of free-response receiver operating characteristic analysis of the detectability of simulated microcalcifications. Scatter fractions decreased considerably as the magnification factor increased without grid technique. In the grid technique, scatter fractions for all magnification techniques were comparable. The total MTFs of magnification techniques with a focal spot size of 0.1 mm improved significantly compared with the conventional contact technique. However, the improvement of the total MTFs of magnification techniques with the combination of 0.3 mm focal spot size was small. The WSs degraded with an increase of the magnification factor compared with the contact technique due to the maintained exposure incident on the object. The observer performance test indicated that the 1.8 magnification technique with the 0.1 mm focal spot size provided higher detectability than did the contact technique. Digital magnification mammography under the same incident exposure conditions improved the detectability of microcalcifications.

Current perspectives in medical image perception

Medical images constitute a core portion of the information a physician utilizes to render diagnostic and treatment decisions. At a fundamental level, this diagnostic process involves two basic processes: visually inspecting the image (visual perception) and rendering an interpretation (cognition). The likelihood of error in the interpretation of medical images is, unfortunately, not negligible. Errors do occur, and patients' lives are impacted, underscoring our need to understand how physicians interact with the information in an image during the interpretation process. With improved understanding, we can develop ways to further improve decision making and, thus, to improve patient care. The science of medical image perception is dedicated to understanding and improving the clinical interpretation process.

A random-sum Wilcoxon statistic and its application to analysis of ROC and LROC data

The Wilcoxon-Mann-Whitney statistic is commonly used for a distribution-free comparison of two groups. One requirement for its use is that the sample sizes of the two groups are fixed. This is violated in some of the applications such as medical imaging studies and diagnostic marker studies; in the former, the violation occurs since the number of correctly localized abnormal images is random, while in the latter the violation is due to some subjects not having observable measurements. For this reason, we propose here a random-sum Wilcoxon statistic for comparing two groups in the presence of ties, and derive its variance as well as its asymptotic distribution for large sample sizes. The proposed statistic includes the regular Wilcoxon rank-sum statistic. Finally, we apply the proposed statistic for summarizing location response operating characteristic data from a liver computed tomography study, and also for summarizing diagnostic accuracy of biomarker data.

An approach to comparing accuracies of two FLAIR MR sequences in the detection of multiple sclerosis lesions in the brain in the absence of gold standard

RATIONALE AND OBJECTIVES

The purpose of this study was to present a new methodology to compare accuracies of two imaging fluid attenuated inversion recovery (FLAIR) magnetic resonance sequences in detection of multiple sclerosis (MS) lesions in the brain in the absence of ground truth, and to determine whether the two sequences, which differed only in echo time (TE), have the same accuracy.

MATERIALS AND METHODS

We acquired FLAIR images at TE(1) = 90 ms and TE(2) = 155 ms from 46 patients with MS (24-69 years old, mean 45.8, 15 males) and 11 healthy volunteers (23-54 years old, mean 37.1, 6 males). Seven experienced neuroradiologists segmented lesions manually on randomly presented corresponding TE(1) and TE(2) images. For every image pair, a "surrogate ground truth" for each TE was generated by applying probability thresholds, ranging from 0.3 to 0.5, to the weighted average of experts' segmentations. Jackknife alternative free-response receiver operating characteristic analysis was used to compare experts' performance on TE(1) and TE(2) images, using successively the TE(1)- and TE(2)-based ground truths.

RESULTS

Supratentorially, there were significant differences in relative accuracy between the two sequences, ranging from 8.4% to 12.1%. In addition, we found a higher ratio of false positives to true positives for the TE(2) sequence using the TE(2) ground truth, compared to the TE(1) equivalent. Infratentorially, differences in the relative accuracy did not reach statistical significance.

CONCLUSION

The presented methodology may be useful in assessing the value of new clinical imaging protocols or techniques in the context of replacing existing ones, when the absolute ground truth is not available, and in determining changes in disease progression in follow-up studies. Our results suggest that the sequence with shorter TE should be preferred because it generates relatively fewer false positives. The finding is consistent with results of previous computer simulation studies.

A status report on free-response analysis

The purpose of this paper is to summarise recent progress in free-response receiver operating characteristic (FROC) methodology. These are: (1) jackknife alternative FROC analysis including recent extensions and alternative methods; (2) the search-model simulator that enables validation and objective comparison of different methods of analysing the data; (3) case-based analysis that has the potential of greater clinical relevance than conventional free-response analysis; (4) a method for collectively analysing the multiple lesion types in an image (e.g. microcalcifications, masses and architectural distortions); (5) a method for sample-size estimation for FROC studies; and (6) a method for determining an objective proximity criterion, namely how close must a mark be to a true lesion in order to credit the observer for a true localisation. FROC analysis is being increasingly used to evaluate the imaging systems and understanding of recent progress should help researchers conduct better FROC studies.