ROCView: prototype software for data collection in jackknife alternative free-response receiver operating characteristic analysis

The efficacy of preliminary clinical evaluation for emergency department chest radiographs with trauma presentations in pre- and post-training situations

INTRODUCTION

The chest X-ray (CXR) is the most frequently performed radiographic examination. This study evaluates radiographers' ability to localise traumatic CXR pathology and provide a preliminary clinical evaluation (PCE) for these cases.

METHODS

This observer study was performed in a district general hospital in the United Kingdom (UK). A 58-case image bank was used with 20 positive cases. Participants were awarded a maximum of three points, based on abnormality recognition and descriptive accuracy. Localisation data were recorded with ROCView. Training was delivered via short online recorded tutorials covering an introduction of a systematic search strategy for CXR, how to recognise the common abnormalities covered in the tests, how to structure a PCE and multiple practice cases to review at participants' own pace. Pre- and post-training data was recorded.

RESULTS

Nine participants completed the study. Overall, pooled sensitivity remained consistent (78.9%-78.8%) following training, specificity and accuracy showed improvement of 79.0%-89.9% and 78.9%-86.0% respectively. An increase in the number of correct localisations and PCE scores were also evident. Participants performed better at correctly identifying a pneumothorax compared to skeletal abnormalities.

CONCLUSION

Improvements in performance were evident for most participants' abnormality localisations and PCE scores, following the training intervention. The study highlighted areas of CXR PCE that may require further training, such as detecting superimposed or subtle abnormalities.

IMPLICATIONS FOR PRACTICE

This study provides additional support for the development of PCE systems in additional areas of imaging practice.

Human Observer Net: A Platform Tool for Human Observer Studies of Image Data

Background Current software applications for human observer studies of images lack flexibility in study design, platform independence, multicenter use, and assessment methods and are not open source, limiting accessibility and expandability. Purpose To develop a user-friendly software platform that enables efficient human observer studies in medical imaging with flexibility of study design. Materials and Methods Software for human observer imaging studies was designed as an open-source web application to facilitate access, platform-independent usability, and multicenter studies. Different interfaces for study creation, participation, and management of results were implemented. The software was evaluated in human observer experiments between May 2019 and March 2021, in which duration of observer responses was tracked. Fourteen radiologists evaluated and graded software usability using the 100-point system usability scale. The application was tested in Chrome, Firefox, Safari, and Edge browsers. Results Software function was designed to allow visual grading analysis (VGA), multiple-alternative forced-choice (m-AFC), receiver operating characteristic (ROC), localization ROC, free-response ROC, and customized designs. The mean duration of reader responses per image or per image set was 6.2 seconds ± 4.8 (standard deviation), 5.8 seconds ± 4.7, 8.7 seconds ± 5.7, and 6.0 seconds ± 4.5 in four-AFC with 160 image quartets per reader, four-AFC with 640 image quartets per reader, localization ROC, and experimental studies, respectively. The mean system usability scale score was 83 ± 11 (out of 100). The documented code and a demonstration of the application are available online (https://github.com/genskeu/HON, https://hondemo.pythonanywhere.com/). Conclusion A user-friendly and efficient open-source application was developed for human reader experiments that enables study design versatility, as well as platform-independent and multicenter usability. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Thompson in this issue.

A JAFROC study of nodule detection performance in CT images of a thorax acquired during PET/CT

PURPOSE

Two types of CT images (modalities) are acquired in PET/CT: for attenuation correction (AC) and diagnosis. The purpose of the study was to compare nodule detection and localization performance between these two modalities.

METHODS

CT images, using both modalities, of an anthropomorphic chest phantom containing zero or more simulated spherical nodules of 5, 8, 10 and 12 mm diameters and contrasts -800, -630 and 100 HU were acquired. An observer performance study using nine observers interpreting 45 normal (zero nodules) images and 47 abnormal images (1-3 nodules; average 1.26) was conducted using the free-response receiver operating characteristic (FROC) paradigm. Data were analysed using an R software package implemented jackknife alternative FROC (JAFROC) analysis. Both empirical areas under the equally weighted AFROC curve (wAFROC) and under the highest rating inferred ROC (HR-ROC) curve were used as figures of merit (FOM). To control the probability of Type I error test alpha was set at 0.05.

RESULTS

Nodule detection as measured by either FOM was significantly better on the diagnostic quality images (2nd modality), irrespective of the method of analysis, [reader averaged inter-modality wAFROC FOM difference = -0.07 (-0.11,-0.04); reader averaged inter-modality HR-ROC FOM difference = -0.05 (-0.09, -0.01)].

CONCLUSION

Nodule detection was statistically worse on images acquired for AC; suggesting that images acquired for AC should not be used to evaluate pulmonary pathology.

The impact of simulated motion blur on lesion detection performance in full-field digital mammography

OBJECTIVE

Motion blur is a known phenomenon in full-field digital mammography, but the impact on lesion detection is unknown. This is the first study to investigate detection performance with varying magnitudes of simulated motion blur.

METHODS

7 observers (15 ± 5 years' reporting experience) evaluated 248 cases (62 containing malignant masses, 62 containing malignant microcalcifications and 124 normal cases) for 3 conditions: no blurring (0 mm) and 2 magnitudes of simulated blurring (0.7 and 1.5 mm). Abnormal cases were biopsy proven. Mathematical simulation was used to provide a pixel shift in order to simulate motion blur. A free-response observer study was conducted to compare lesion detection performance for the three conditions. The equally weighted jackknife alternative free-response receiver operating characteristic was used as the figure of merit. Test alpha was set at 0.05 to control probability of Type I error.

RESULTS

The equally weighted jackknife alternative free-response receiver operating characteristic analysis found a statistically significant difference in lesion detection performance for both masses [F(2,22) = 6.01, p = 0.0084] and microcalcifications [F(2,49) = 23.14, p < 0.0001]. The figures of merit reduced as the magnitude of simulated blurring increased. Statistical differences were found between some of the pairs investigated for the detection of masses (0.0 vs 0.7 and 0.0 vs 1.5 mm) and all pairs for microcalcifications (0.0 vs 0.7, 0.0 vs 1.5 and 0.7 vs 1.5 mm). No difference was detected between 0.7 and 1.5 mm for masses.

CONCLUSION

The mathematical simulation of motion blur caused a statistically significant reduction in lesion detection performance. These false-negative decisions could have implications for clinical practice. Advances in knowledge: This research demonstrates for the first time that motion blur has a negative and statistically significant impact on lesion detection performance in digital mammography.

The impact of greyscale inversion for nodule detection in an anthropomorphic chest phantom: a free-response observer study

Objective:

The aim of this work was to assess the impact of greyscale inversion on nodule detection on posteroanterior chest X-ray images. Previous work has attempted this, with no consensus opinion formed. We assessed the value of “fast-flicking” between standard and inverted display modes for nodule detection.

Methods:

Six consultant radiologists (with 5–32 years' reporting experience) completed an observer task under the free-response paradigm. An anthropomorphic chest phantom was loaded with 50 different configurations of simulated nodules (1–4 nodules per case) measuring 5, 8, 10 and 12 mm in spherical diameter; each configuration represented a single case. In addition, 25 cases contained no nodules. Images were displayed in three modes: (i) standard, (ii) inverted and (iii) fast-flicking between standard and inverted display modes. Each observer completed the study in a different order of display (i, ii, iii) using a calibrated 5-megapixel monitor. Nodules were localized with mouse clicks and ratings assigned using a 1–10 discrete slider-bar confidence scale. Rjafroc (Pittsburgh, PA) was used for data analysis; differences in nodule detection performance were considered significant at 0.05.

Results:

The observer-averaged weighted jackknife alternative free-response receiver-operating characteristic figures of merit were 0.715 (standard), 0.684 (inverted) and 0.717 (fast-flicking). Random-reader fixed-case analysis revealed no statistically significant difference between any treatment pair [F(2,8) = 1.22; p = 0.345].

Conclusion:

No statistically significant difference in nodule detection was found for the three display conditions.

Advances in knowledge:

We have investigated the impact of fast-flicking between standard and inverted display modes for the detection of nodules. We found no benefit.

VIEWDEX: A STATUS REPORT

ViewDEX (Viewer for Digital Evaluation of X-ray images) is an image viewer and task manager suitable for research and optimisation tasks in medical imaging. The software has undergone continuous development during more than a decade and has during this time period been used in numerous studies. ViewDEX is DICOM compatible, and the features of the interface (tasks, image handling and functionality) are general and flexible. The set-up of a study is determined by altering properties in a text-editable file, enabling easy and flexible configuration. ViewDEX is developed in Java and can run from any disc area connected to a computer. It is free to use for non-commercial purposes and can be downloaded from http://www.vgregion.se/sas/viewdex The purposes of the present article are to give a short overview of the development of ViewDEX and to describe recent updates of the software. In addition, a description on how to configure a viewing session in ViewDEX is provided.

Effect of reconstruction methods and x-ray tube current-time product on nodule detection in an anthropomorphic thorax phantom: A crossed-modality JAFROC observer study

Purpose:

To evaluate nodule detection in an anthropomorphic chest phantom in computed tomography (CT) images reconstructed with adaptive iterative dose reduction 3D (AIDR^3D) and filtered back projection (FBP) over a range of tube current–time product (mAs).

Methods:

Two phantoms were used in this study: (i) an anthropomorphic chest phantom was loaded with spherical simulated nodules of 5, 8, 10, and 12 mm in diameter and +100, −630, and −800 Hounsfield units electron density; this would generate CT images for the observer study; (ii) a whole-body dosimetry verification phantom was used to ultimately estimate effective dose and risk according to the model of the BEIR VII committee. Both phantoms were scanned over a mAs range (10, 20, 30, and 40), while all other acquisition parameters remained constant. Images were reconstructed with both AIDR^3D and FBP. For the observer study, 34 normal cases (no nodules) and 34 abnormal cases (containing 1–3 nodules, mean 1.35 ± 0.54) were chosen. Eleven observers evaluated images from all mAs and reconstruction methods under the free-response paradigm. A crossed-modality jackknife alternative free-response operating characteristic (JAFROC) analysis method was developed for data analysis, averaging data over the two factors influencing nodule detection in this study: mAs and image reconstruction (AIDR^3D or FBP). A Bonferroni correction was applied and the threshold for declaring significance was set at 0.025 to maintain the overall probability of Type I error at α = 0.05. Contrast-to-noise (CNR) was also measured for all nodules and evaluated by a linear least squares analysis.

Results:

For random-reader fixed-case crossed-modality JAFROC analysis, there was no significant difference in nodule detection between AIDR^3D and FBP when data were averaged over mAs [F(1, 10) = 0.08, p = 0.789]. However, when data were averaged over reconstruction methods, a significant difference was seen between multiple pairs of mAs settings [F(3, 30) = 15.96, p < 0.001]. Measurements of effective dose and effective risk showed the expected linear dependence on mAs. Nodule CNR was statistically higher for simulated nodules on images reconstructed with AIDR^3D (p < 0.001).

Conclusions:

No significant difference in nodule detection performance was demonstrated between images reconstructed with FBP and AIDR^3D. mAs was found to influence nodule detection, though further work is required for dose optimization.

A free-response evaluation determining value in the computed tomography attenuation correction image for revealing pulmonary incidental findings: a phantom study

RATIONALE AND OBJECTIVES

The purpose of this study was to compare lesion-detection performance when interpreting computed tomography (CT) images that are acquired for attenuation correction when performing single photon emission computed tomography/computed tomography (SPECT/CT) myocardial perfusion studies. In the United Kingdom, there is a requirement that these images be interpreted; thus, it is necessary to understand observer performance on these images.

MATERIALS AND METHODS

An anthropomorphic chest phantom with inserted spherical lesions of different sizes and contrasts was scanned on five different SPECT/CT systems using site-specific CT protocols for SPECT/CT myocardial perfusion imaging. Twenty-one observers (0-4 years of CT experience) searched 26 image slices (17 abnormal, containing 1-3 lesions, and 9 normal, containing no lesions) for each CT acquisition. The observers marked and rated perceived lesions under the free-response paradigm. Four analyses were conducted using jackknife alternative free-response receiver operating characteristic (JAFROC) analysis: (1) 20-pixel acceptance radius (AR) with all 21 readers, abbreviated to 20/ALL analysis, (2) 40-pixel AR with 21 readers (40/ALL), (3) 20-pixel AR with 14 readers experienced in CT (20/EXP), and (4) 20-pixel AR with 7 readers with no CT experience (20/NOT). The significance level of the test was set so as to conservatively control the overall probability of a type I error to <0.05.

RESULTS

The mean JAFROC figure of merit (FOM) for the five CT acquisitions for the 20/ALL study were 0.602, 0.639, 0.372, 0.475, and 0.719 with a significant difference in lesion-detection performance evident between all individual treatment pairs (P < .0001) with the exception of the 1-2 pairing, which was not significant (these differed only in milliamp seconds). System 5, which had the highest performance, had the smallest slice thickness and the largest matrix size. For the other analyses, the system orderings remained unchanged, and the significance of FOM difference findings remained identical to those for 20/ALL, with one exception: for 20/EXP analysis the 1-2 difference became significant with the higher milliamp seconds superior. Improved detection performance was associated with a smaller slice thickness, increased matrix size, and, to a lesser extent, increased tube charge.

CONCLUSIONS

Protocol variations for CT-based attenuation correction (AC) in SPECT/CT imaging have a measurable impact on lesion-detection performance. The results imply that z-axis resolution and matrix size had the greatest impact on lesion detection, with a weaker but detectable dependence on the product of milliamp and seconds.

Accurate localization of incidental findings on the computed tomography attenuation correction image: the influence of tube current variation

This observer performance study assessed lesion detection in the computed tomography attenuation correction image, as would be produced for myocardial perfusion imaging over a tube current (mA) range. A static anthropomorphic chest phantom containing simulated pulmonary lesions was scanned using the four available mA values (1, 1.5, 2 and 2.5) on a GE Infinia Hawkeye 4. All other computed tomography acquisition parameters remained constant throughout. Twenty-seven cases showing zero to four lesions were produced for a free-response receiver-operating characteristic method. Image observations were completed using our novel web-based ROCView software under controlled conditions. The Jackknife alternative free-response receiver-operating characteristic (JAFROC) figure of merit was used for significance testing, wherein a difference in lesion detection performance was considered significant at P values less than 0.05. Twenty readers with varying computed tomography experience (0-24 years) evaluated 108 images using an ordinal scale to score confidence. The JAFROC analysis showed that there was no statistically significant difference in performance between mA values (P=0.439) for this sample of observers. In conclusion, no significant difference in lesion detection performance was seen between the mA values. This suggests that there is no value in using anything other than the lowest mA value for the investigation of incidental extracardiac findings.