A status report on free-response analysis

The Effect of Clinical History on Diagnostic Imaging Interpretation - A Systematic Review

RATIONALE AND OBJECTIVES

To provide updated information on the effect of clinical history on diagnostic image interpretation and to provide study methodology and design recommendations for future studies assessing the effect of clinical history on diagnostic image performance.

MATERIALS AND METHODS

A literature search of Medline, Embase, Scopus, Web of Science, and the Cochrane Central Register of Controlled Trials (CENTRAL) databases was conducted from database inception to July 21, 2020. Studies comparing diagnostic imaging performance with and without clinical history, using observers reading images under both conditions that used an independent reference standard were included.

RESULTS

Twenty-two studies met the inclusion criteria, with 15 showing clinical history improved diagnostic performance. One study reported a decrease in diagnostic performance with clinical history and the remaining six studies found no significant change in performance. Two studies used the free response paradigm with both reporting clinical history increased location sensitivity, decreased specificity and had no overall change in diagnostic performance. The disease spectrum of included cases was largely unreported and a balanced reading design was not used in 19 studies.

CONCLUSION

Most published studies found that clinical history improved diagnostic performance. More recent studies accounting for abnormality location and multiple abnormalities showed an increase in false positives and no significant change in overall diagnostic performance with clinical history.

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists: A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis

RATIONALE AND OBJECTIVES

Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

METHODS

A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

RESULTS

The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808-0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766-0.811), P < .0001.

CONCLUSIONS

With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

Color radiography in lung nodule detection and characterization: comparison with conventional gray scale radiography

Background

To compare the capability of lung nodule detection and characterization between dual-energy radiography with color-representation (DCR) and conventional gray scale chest radiography (GSR).

Methods

A total of 130 paired chest radiographs (DCR and GSR) obtained from 65 patients (14 with normal scans and 51 with pulmonary nodules) were evaluated. After analysis, 45 non-calcified and 21 calcified nodules were identified. DCR was obtained by adding color space within material-decomposed data (blue for high attenuation and red for low attenuation) and by compounding the manipulated data to one color image. Three radiologists marked suggested nodules on radiographic images and assessed the level of confidence of lesion presence and probability of nodule calcification by using a nine-point rating scale. The jackknife active free-response receiver operating characteristics (JAFROC) analysis was used to evaluate lesion detectability, and multi-reader multi-case receiver operating characteristics (MRMC ROC) analysis was used for the evaluation of the accuracy of nodule calcification prediction.

Results

Figures of merit (FOM) from JAFROC was 0.807 for DCR and 0.811 for GSR, respectively; nodule detectability was not significantly different between DCR and GSR (p = 0.93). Areas under curve (AUC) from MRMC ROC were 0.944 for DCR and 0.828 for GSR, respectively; performance of DCR in predicting lung nodule calcification was significantly higher than that of GSR (p = 0.04).

Conclusions

DCR showed similar performance in terms of lung nodule detection compared with GSR. However, DCR does provide a significant benefit in predicting the presence of nodule calcification.

Comparison of dual-energy subtraction and electronic bone suppression combined with computer-aided detection on chest radiographs: effect on human observers' performance in nodule detection

OBJECTIVE

The objective of our study was to compare the effect of dual-energy subtraction and bone suppression software alone and in combination with computer-aided detection (CAD) on the performance of human observers in lung nodule detection.

MATERIALS AND METHODS

One hundred one patients with from one to five lung nodules measuring 5-29 mm and 42 subjects with no nodules were retrospectively selected and randomized. Three independent radiologists marked suspicious-appearing lesions on the original chest radiographs, dual-energy subtraction images, and bone-suppressed images before and after postprocessing with CAD. Marks of the observers and CAD marks were compared with CT as the reference standard. Data were analyzed using nonparametric tests and the jackknife alternative free-response receiver operating characteristic (JAFROC) method.

RESULTS

Using dual-energy subtraction alone (p = 0.0198) or CAD alone (p = 0.0095) improved the detection rate compared with using the original conventional chest radiograph. The combination of bone suppression and CAD provided the highest sensitivity (51.6%) and the original nonenhanced conventional chest radiograph alone provided the lowest (46.9%; p = 0.0049). Dual-energy subtraction and bone suppression provided the same false-positive (p = 0.2702) and true-positive (p = 0.8451) rates. Up to 22.9% of lesions were found only by the CAD program and were missed by the readers. JAFROC showed no difference in the performance between modalities (p = 0.2742-0.5442).

CONCLUSION

Dual-energy subtraction and the electronic bone suppression program used in this study provided similar detection rates for pulmonary nodules. Additionally, CAD alone or combined with bone suppression can significantly improve the sensitivity of human observers for pulmonary nodule detection.

On the choice of acceptance radius in free-response observer performance studies

OBJECTIVES

Choosing an acceptance radius or proximity criterion is necessary to analyse free-response receiver operating characteristic (FROC) observer performance data. This is currently subjective, with little guidance in the literature about what is an appropriate acceptance radius. We evaluated varying acceptance radii in a nodule detection task in chest radiography and suggest guidelines for determining an acceptance radius.

METHODS

80 chest radiographs were chosen, half of which contained nodules. We determined each nodule's centre. 21 radiologists read the images. We created acceptance radii bins of <5 pixels, <10 pixels, <20 pixels and onwards up to <200 and 200+ pixels. We counted lesion localisations in each bin and visually compared marks with the borders of nodules.

RESULTS

Most reader marks were tightly clustered around nodule centres, with tighter clustering for smaller than for larger nodules. At least 70% of readers' marks were placed within <10 pixels for small nodules, <20 pixels for medium nodules and <30 pixels for large nodules. Of 72 inspected marks that were less than 50 pixels from the centre of a nodule, only 1 fell outside the border of a nodule.

CONCLUSION

The acceptance radius should be based on the larger nodule sizes. For our data, an acceptance radius of 50 pixels would have captured all but 2 reader marks within the borders of a nodule, while excluding only 1 true-positive mark. The choice of an acceptance radius for FROC analysis of observer performance studies should be based on the size of larger abnormalities.

Learning aspects and potential pitfalls regarding detection of pulmonary nodules in chest tomosynthesis and proposed related quality criteria

BACKGROUND

In chest tomosynthesis, low-dose projections collected over a limited angular range are used for reconstruction of an arbitrary number of section images of the chest, resulting in a moderately increased radiation dose compared to chest radiography.

PURPOSE

To investigate the effects of learning with feedback on the detection of pulmonary nodules for observers with varying experience of chest tomosynthesis, to identify pitfalls regarding detection of pulmonary nodules, and present suggestions for how to avoid them, and to adapt the European quality criteria for chest radiography and computed tomography (CT) to chest tomosynthesis.

MATERIAL AND METHODS

Six observers analyzed tomosynthesis cases for presence of nodules in a jackknife alternative free-response receiver-operating characteristics (JAFROC) study. CT was used as reference. The same tomosynthesis cases were analyzed before and after learning with feedback, which included a collective learning session. The difference in performance between the two readings was calculated using the JAFROC figure of merit as principal measure of detectability.

RESULTS

Significant improvement in performance after learning with feedback was found only for observers inexperienced in tomosynthesis. At the collective learning session, localization of pleural and subpleural nodules or structures was identified as the main difficulty in analyzing tomosynthesis images.

CONCLUSION

The results indicate that inexperienced observers can reach a high level of performance regarding nodule detection in tomosynthesis after learning with feedback and that the main problem with chest tomosynthesis is related to the limited depth resolution.