A nonparametric procedure for comparing the areas under correlated LROC curves

Extending the DeLong algorithm for comparing areas under correlated receiver operating characteristic curves with missing data

A nonparametric method proposed by DeLong et al in 1988 for comparing areas under correlated receiver operating characteristic curves is used widely in practice. However, the DeLong method as implemented in popular software quietly deletes individuals with any missing values, yielding potentially invalid and/or inefficient results. We simplify the DeLong algorithm using ranks and extend it to accommodate missing data by using a mixed model approach for multivariate data. Simulation results demonstrate the validity and efficiency of our procedure for data missing at random. We illustrate our proposed procedure in SAS, Stata, and R using the original DeLong data.

A flexible three-dimensional heterophase computed tomography hepatocellular carcinoma detection algorithm for generalizable and practical screening

Hepatocellular carcinoma (HCC) can be potentially discovered from abdominal computed tomography (CT) studies under varied clinical scenarios (e.g., fully dynamic contrast-enhanced [DCE] studies, noncontrast [NC] plus venous phase [VP] abdominal studies, or NC-only studies). Each scenario presents its own clinical challenges that could benefit from computer-aided detection (CADe) tools. We investigate whether a single CADe model can be made flexible enough to handle different contrast protocols and whether this flexibility imparts performance gains. We developed a flexible three-dimensional deep algorithm, called heterophase volumetric detection (HPVD), that can accept any combination of contrast-phase inputs with adjustable sensitivity depending on the clinical purpose. We trained HPVD on 771 DCE CT scans to detect HCCs and evaluated it on 164 positives and 206 controls. We compared performance against six clinical readers, including two radiologists, two hepatopancreaticobiliary surgeons, and two hepatologists. The area under the curve of the localization receiver operating characteristic for NC-only, NC plus VP, and full DCE CT yielded 0.71 (95% confidence interval [CI], 0.64-0.77), 0.81 (95% CI, 0.75-0.87), and 0.89 (95% CI, 0.84-0.93), respectively. At a high-sensitivity operating point of 80% on DCE CT, HPVD achieved 97% specificity, which is comparable to measured physician performance. We also demonstrated performance improvements over more typical and less flexible nonheterophase detectors. Conclusion: A single deep-learning algorithm can be effectively applied to diverse HCC detection clinical scenarios, indicating that HPVD could serve as a useful clinical aid for at-risk and opportunistic HCC surveillance.

Deep-learning model observer for a low-contrast hepatic metastases localization task in computed tomography

PURPOSE

Conventional model observers (MO) in CT are often limited to a uniform background or varying background that is random and can be modeled in an analytical form. It is unclear if these conventional MOs can be readily generalized to predict human observer performance in clinical CT tasks that involve realistic anatomical background. Deep-learning-based model observers (DL-MO) have recently been developed, but have not been validated for challenging low contrast diagnostic tasks in abdominal CT. We consequently sought to validate a DL-MO for a low-contrast hepatic metastases localization task.

METHODS

We adapted our recently developed DL-MO framework for the liver metastases localization task. Our previously-validated projection-domain lesion-/noise-insertion techniques were used to synthesize realistic positive and low-dose abdominal CT exams, using the archived patient projection data. Ten experimental conditions were generated, which involved different lesion sizes/contrasts, radiation dose levels, and image reconstruction types. Each condition included 100 trials generated from a patient cohort of 7 cases. Each trial was presented as liver image patches (160×160×5 voxels). The DL-MO performance was calculated for each condition and was compared with human observer performance, which was obtained by three sub-specialized radiologists in an observer study. The performance of DL-MO and radiologists was gauged by the area under localization receiver-operating-characteristic curves. The generalization performance of the DL-MO was estimated with the repeated twofold cross-validation method over the same set of trials used in the human observer study. A multi-slice Channelized Hoteling Observers (CHO) was compared with the DL-MO across the same experimental conditions.

RESULTS

The performance of DL-MO was highly correlated to that of radiologists (Pearson's correlation coefficient: 0.987; 95% CI: [0.942, 0.997]). The performance level of DL-MO was comparable to that of the grouped radiologists, that is, the mean performance difference was -3.3%. The CHO performance was poorer than the grouped radiologist performance, before internal noise could be added. The correlation between CHO and radiologists was weaker (Pearson's correlation coefficient: 0.812, and 95% CI: [0.378, 0.955]), and the corresponding performance bias (-29.5%) was statistically significant.

CONCLUSION

The presented study demonstrated the potential of using the DL-MO for image quality assessment in patient abdominal CT tasks.

Deep-learning-based model observer for a lung nodule detection task in computed tomography

Purpose: Task-based image quality assessment using model observers (MOs) is an effective approach to radiation dose and scanning protocol optimization in computed tomography (CT) imaging, once the correlation between MOs and radiologists can be established in well-defined clinically relevant tasks. Conventional MO studies were typically simplified to detection, classification, or localization tasks using tissue-mimicking phantoms, as traditional MOs cannot be readily used in complex anatomical background. However, anatomical variability can affect human diagnostic performance. Approach: To address this challenge, we developed a deep-learning-based MO (DL-MO) for localization tasks and validated in a lung nodule detection task, using previously validated projection-based lesion-/noise-insertion techniques. The DL-MO performance was compared with 4 radiologist readers over 12 experimental conditions, involving varying radiation dose levels, nodule sizes, nodule types, and reconstruction types. Each condition consisted of 100 trials (i.e., 30 images per trial) generated from a patient cohort of 50 cases. DL-MO was trained using small image volume-of-interests extracted across the entire volume of training cases. For each testing trial, the nodule searching of DL-MO was confined to a 3-mm thick volume to improve computational efficiency, and radiologist readers were tasked to review the entire volume. Results: A strong correlation between DL-MO and human readers was observed (Pearson's correlation coefficient: 0.980 with a 95% confidence interval of [0.924, 0.994]). The averaged performance bias between DL-MO and human readers was 0.57%. Conclusion: The experimental results indicated the potential of using the proposed DL-MO for diagnostic image quality assessment in realistic chest CT tasks.

Localization of liver lesions in abdominal CT imaging: I. Correlation of human observer performance between anatomical and uniform backgrounds

The purpose of this study was to determine the correlation between human observer performance for localization of small low contrast lesions within uniform water background versus an anatomical liver background, under the conditions of varying dose, lesion size, and reconstruction algorithm. Liver lesions (5 mm, 7 mm, and 9 mm, contrast:  -21 HU) were digitally inserted into CT projection data of ten normal patients in vessel-free liver regions. Noise was inserted into the projection data to create three image sets: full dose and simulated half and quarter doses. Images were reconstructed with a standard filtered back projection (FBP) and an iterative reconstruction (IR) algorithm. Lesion and noise insertion procedures were repeated with water phantom data. Two-dimensional regions of interest (66 lesion-present, 34 lesion-absent) were selected, randomized, and independently reviewed by three medical physicists to identify the most likely location of the lesion and provide a confidence score. Locations and confidence scores were assessed using the area under the localization receiver operating characteristic curve (Az_LROC). We examined the correlation between human performance for the liver and uniform water backgrounds as dose, lesion size, and reconstruction algorithm varied. As lesion size or dose increased, reader localization performance improved. For full dose IR images, the Az_LROC for 5, 7, and 9 mm lesions were 0.53, 0.91, and 0.97 (liver) and 0.51, 0.96, and 0.99 (uniform water), respectively. Similar trends were seen with other parameters. Performance values for liver and uniform backgrounds were highly correlated for both reconstruction algorithms, with a Spearman correlation of ρ  =  0.97, and an average difference in Az_LROC of 0.05  ±  0.04. For the task of localizing low contrast liver lesions, human observer performance was highly correlated between anatomical and uniform backgrounds, suggesting that lesion localization studies emulating a clinical test of liver lesion detection can be performed using a uniform background.

Localization of liver lesions in abdominal CT imaging: II. Mathematical model observer performance correlates with human observer performance for localization of liver lesions in abdominal CT imaging

Determination of the effect of protocol modifications on diagnostic performance in CT with human observers is extremely time-consuming, limiting the applicability of such methods in routine clinical practice. In this work, we sought to determine whether a channelized Hotelling observer (CHO) could predict human observer performance for the task of liver lesion localization as background, reconstruction algorithm, dose, and lesion size were varied. Liver lesions (5 mm, 7 mm, and 9 mm) were digitally inserted into the CT projection data of patients with normal livers and water phantoms. The projection data were reconstructed with filtered back projection (FBP) and iterative reconstruction (IR) algorithms for three dose levels: full dose (liver CTDIvol  =  10.5  ±  8.5 mGy, water phantom CTDIvol  =  9.6  ±  0.1 mGy) and simulated half and quarter doses. For each of 36 datasets (3 dose levels  ×  2 reconstruction algorithms  ×  2 backgrounds  ×  3 sizes), 66 signal-present and 34 signal-absent 2D images were extracted from the reconstructed volumes. Three medical physicists independently reviewed each dataset and noted the lesion location and a confidence score for each image. A CHO with Gabor channels was calculated to estimate the performance for each of the 36 localization tasks. The CHO performances, quantified using localization receiver operating characteristic (LROC) analysis, were compared to the human observer performances. Performance values between human and model observers were highly correlated for equivalent parameters (same lesion size, dose, background, and reconstruction), with a Spearman's correlation coefficient of 0.93 (95% CI: 0.82-0.98). CHO performance values for the uniform background were strongly correlated (ρ  =  0.94, CI: 0.80-1.0) with the human observer performance values for the liver background. Performance values between human observers and CHO were highly correlated as dose, reconstruction type and object size were varied for the task of localization of patient liver lesions in both uniform and liver backgrounds.

Optimal Joint Detection and Estimation That Maximizes ROC-Type Curves

Combined detection-estimation tasks are frequently encountered in medical imaging. Optimal methods for joint detection and estimation are of interest because they provide upper bounds on observer performance, and can potentially be utilized for imaging system optimization, evaluation of observer efficiency, and development of image formation algorithms. We present a unified Bayesian framework for decision rules that maximize receiver operating characteristic (ROC)-type summary curves, including ROC, localization ROC (LROC), estimation ROC (EROC), free-response ROC (FROC), alternative free-response ROC (AFROC), and exponentially-transformed FROC (EFROC) curves, succinctly summarizing previous results. The approach relies on an interpretation of ROC-type summary curves as plots of an expected utility versus an expected disutility (or penalty) for signal-present decisions. We propose a general utility structure that is flexible enough to encompass many ROC variants and yet sufficiently constrained to allow derivation of a linear expected utility equation that is similar to that for simple binary detection. We illustrate our theory with an example comparing decision strategies for joint detection-estimation of a known signal with unknown amplitude. In addition, building on insights from our utility framework, we propose new ROC-type summary curves and associated optimal decision rules for joint detection-estimation tasks with an unknown, potentially-multiple, number of signals in each observation.

Correlation between model observer and human observer performance in CT imaging when lesion location is uncertain

PURPOSE

The purpose of this study was to investigate the correlation between model observer and human observer performance in CT imaging for the task of lesion detection and localization when the lesion location is uncertain.

METHODS

Two cylindrical rods (3-mm and 5-mm diameters) were placed in a 35×26 cm torso-shaped water phantom to simulate lesions with -15 HU contrast at 120 kV. The phantom was scanned 100 times on a 128-slice CT scanner at each of four dose levels (CTDIvol=5.7, 11.4, 17.1, and 22.8 mGy). Regions of interest (ROIs) around each lesion were extracted to generate images with signal-present, with each ROI containing 128×128 pixels. Corresponding ROIs of signal-absent images were generated from images without lesion mimicking rods. The location of the lesion (rod) in each ROI was randomly distributed by moving the ROIs around each lesion. Human observer studies were performed by having three trained observers identify the presence or absence of lesions, indicating the lesion location in each image and scoring confidence for the detection task on a 6-point scale. The same image data were analyzed using a channelized Hotelling model observer (CHO) with Gabor channels. Internal noise was added to the decision variables for the model observer study. Area under the curve (AUC) of ROC and localization ROC (LROC) curves were calculated using a nonparametric approach. The Spearman's rank order correlation between the average performance of the human observers and the model observer performance was calculated for the AUC of both ROC and LROC curves for both the 3- and 5-mm diameter lesions.

RESULTS

In both ROC and LROC analyses, AUC values for the model observer agreed well with the average values across the three human observers. The Spearman's rank order correlation values for both ROC and LROC analyses for both the 3- and 5-mm diameter lesions were all 1.0, indicating perfect rank ordering agreement of the figures of merit (AUC) between the average performance of the human observers and the model observer performance.

CONCLUSIONS

In CT imaging of different sizes of low-contrast lesions (-15 HU), the performance of CHO with Gabor channels was highly correlated with human observer performance for the detection and localization tasks with uncertain lesion location in CT imaging at four clinically relevant dose levels. This suggests the ability of Gabor CHO model observers to meaningfully assess CT image quality for the purpose of optimizing scan protocols and radiation dose levels in detection and localization tasks for low-contrast lesions.