Effect of observer instruction on ROC study of chest images

Improving Image Quality and Reducing Radiation Dose for Pediatric CT by Using Deep Learning Reconstruction

Background CT deep learning reconstruction (DLR) algorithms have been developed to remove image noise. How the DLR affects image quality and radiation dose reduction has yet to be fully investigated. Purpose To investigate a DLR algorithm's dose reduction and image quality improvement for pediatric CT. Materials and Methods DLR was compared with filtered back projection (FBP), statistical-based iterative reconstruction (SBIR), and model-based iterative reconstruction (MBIR) in a retrospective study by using data from CT examinations of pediatric patients (February to December 2018). A comparison of object detectability for 15 objects (diameter, 0.5-10 mm) at four contrast difference levels (50, 150, 250, and 350 HU) was performed by using a non-prewhitening-matched mathematical observer model with eye filter (d'_NPWE), task transfer function, and noise power spectrum analysis. Object detectability was assessed by using area under the curve analysis. Three pediatric radiologists performed an observer study to assess anatomic structures with low object-to-background signal and contrast to noise in the azygos vein, right hepatic vein, common bile duct, and superior mesenteric artery. Observers rated from 1 to 10 (worst to best) for edge definition, quantum noise level, and object conspicuity. Analysis of variance and Tukey honest significant difference post hoc tests were used to analyze differences between reconstruction algorithms. Results Images from 19 patients (mean age, 11 years ± 5 [standard deviation]; 10 female patients) were evaluated. Compared with FBP, SBIR, and MBIR, DLR demonstrated improved object detectability by 51% (16.5 of 10.9), 18% (16.5 of 13.9), and 11% (16.5 of 14.8), respectively. DLR reduced image noise without noise texture effects seen with MBIR. Radiologist ratings were 7 ± 1 (DLR), 6.2 ± 1 (MBIR), 6.2 ± 1 (SBIR), and 4.6 ± 1 (FBP); two-way analysis of variance showed a difference on the basis of reconstruction type (P < .001). Radiologists consistently preferred DLR images (intraclass correlation coefficient, 0.89; 95% CI: 0.83, 0.93). DLR demonstrated 52% (1 of 2.1) greater dose reduction than SBIR. Conclusion The DLR algorithm improved image quality and dose reduction without sacrificing noise texture and spatial resolution. © RSNA, 2020 Online supplemental material is available for this article.

Method for minimizing observer variation for the quantitation of high-resolution computed tomographic signs of lung disease

OBJECTIVES

This study aimed to describe a method of reducing interobserver variation associated with the visual quantitation of high-resolution computed tomographic (HRCT) signs of airways and interstitial lung disease (ILD).

METHODS

The HRCT scans of 2 cohorts of patients with airways disease (n = 144) and ILD (n = 109) were evaluated by 2 observers. Selected signs of airways disease were evaluated: (1) bronchial wall thickness and (2) the extent of the decreased attenuation. In the ILD group, the total extent of disease was scored. These 3 HRCT signs were scored by 2 observers independently using a standard method. The observers rescored the CT scans with a new scoring system (continuous learning method, CLM).

RESULTS

Observer agreement for CT signs was superior for CLM: bronchial wall thickness κw increased from 0.51 to 0.76; for decreased attenuation, κw increased from 0.34 to 0.81; and for ILD extent, κw increased from 0.53 to 0.87.

CONCLUSIONS

The CLM reduces noise from observer variation in studies that require visual quantitation of HRCT signs of lung disease.

Systematic review: bias in imaging studies - the effect of manipulating clinical context, recall bias and reporting intensity

OBJECTIVES

To perform a systematic review of diagnostic test accuracy studies which manipulate or investigate the context of interpretation. In particular, those which modify or conceal sample characteristics (e.g. disease prevalence or reporting intensity) or research setting ("laboratory" versus "field"). We also investigated recall bias.

METHODS

We searched the biomedical literature to March 2010 using 3 complementary strategies. Inclusion criteria were: imaging studies quantifying the effect on diagnosis of modifying the context of observers' interpretations, varying disease prevalence, concealing sample characteristics, reporting intensity and recall bias.

RESULTS

11247 abstracts were reviewed, 201 full texts examined and 12 ultimately included. There were 5 to 9520 patients and 2 to 129 observers per study. Nine studies investigated clinical review bias of sample level information. Only 3 studies investigated prevalence, 2 of which investigated maximum enrichment well below the levels often used by researchers. We identified no research specifically directed at concealing disease prevalence. Available research found no evidence of recall bias or "washout" on study results.

CONCLUSIONS

Several sources of bias central to the design of diagnostic test accuracy studies are poorly researched; the implications for evidence-based-practice remain uncertain. Research is suggested to guide methodological design, particularly in the context of screening.

KEY POINTS

Imaging research studies often ignore the possible effect of disease prevalence It is unclear how the expectation of disease influences radiological interpretation The potential effect of observer recall bias is poorly researched Such factors might introduce bias into radiological research methodology This systematic review attempts to illustrate these points.

Reader performance in radiographic diagnosis of signs of mitral regurgitation in cavalier King Charles spaniels

OBJECTIVES

To measure accuracy and variability of diagnosis by radiography of heart enlargement (HE) and heart failure (HF) in mitral regurgitation (MR).

METHODS

Sixteen readers representing four levels of experience evaluated 50 sets of radiographs with varying severity of MR for presence or absence of HE, left atrial enlargement (LAE) and HF. The performance of the readers was compared with a reference standard, using area under the curve (AUC) of receiver operating characteristic (ROC) curves. The interreader agreement value kappa (K) was calculated. A subset of difficult cases of HF was analysed before and after removing an outlying reader from each group.

RESULTS

AUC for HE was 0.89, for LAE it was 0.93 and for HF it was 0.92. Experience increased certainty of diagnosis but not accuracy. K ranges were HE, 0.53 to 0.67; LAE, 0.61 to 0.69 and HF, 0.49 to 0.58. When only difficult cases of HF were read, accuracy decreased and experienced readers performed better than inexperienced. When outlying readers were excluded, the differences between experienced and inexperienced readers increased.

CLINICAL SIGNIFICANCE

LAE, not HE, should be used to evaluate the heart size and indirectly the severity of MR on radiographs. For HF, agreement among individual readers was only moderate. Studies of reader accuracy should consider the effects of interreader variability.

Binary and multi-category ratings in a laboratory observer performance study: a comparison

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations

PURPOSE

To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study.

MATERIALS AND METHODS

This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates.

RESULTS

On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01).

CONCLUSION

Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well.

Measuring Observer Performance in Chest Radiology: Some Experiences

All decisions made under conditions of uncertainty have error rates. All meaningful decisions are made under conditions of uncertainty. Can this uncertainty be measured? Can variations in how different observers deal with this uncertainty be ascertained? The ability to measure observer performance in diagnostic imaging was one of the issues that initiated the field of medical decision analysis. This article exemplifies an approach and is worth discussing as a preamble to presenting our long-term project of measuring variations in observer performance. The paper focuses on the interpretation of chest x-ray images, although the principles and findings described can be applied to nearly every radiologic modality and interpretation task.

Oleic Acid-induced Lung Injury: Thin-Section CT Evaluation in Dogs

PURPOSE

To validate lung attenuation measurements for quantifying extravascular lung water in oleic acid-induced pulmonary edema, compare subjective assessment with attenuation measurements, and compare this permeability-type pulmonary edema with hydrostatic-type pulmonary edema.

MATERIALS AND METHODS

Thin-section computed tomography (CT) and pulmonary hemodynamic examinations were performed sequentially in six dogs before and after intravenous administration of 0.08 mg of oleic acid per kilogram of body weight. Extravascular lung water and pulmonary capillary pressure were measured. Results were compared with those reported in a canine model of hydrostatic edema.

RESULTS

Oleic acid induced a progressive increase in extravascular lung water without a change in capillary pressure, which indicated pure permeability-type edema. Ground-glass opacification was detected as soon as extravascular lung water increased. Lung attenuation was highly correlated to extravascular lung water (r = 0.76, P<.001), as in hydrostatic edema, but was characterized by an almost absent gravitational gradient.

CONCLUSION

Thin-section CT is sensitive for early detection and quantification of oleic acid-induced pulmonary edema in a canine model. Different from early canine hydrostatic edema, which is characterized by a gravitational gradient, early oleic acid-induced pulmonary edema in a supine dog is characterized by nearly homogeneous distribution, except for ventral sparing.

Pulmonary emphysema: subjective visual grading versus objective quantification with macroscopic morphometry and thin-section CT densitometry

PURPOSE

To compare subjective visual grading of pulmonary emphysema with macroscopic morphometry and computed tomographic (CT) densitometry.

MATERIALS AND METHODS

In 62 consecutive patients who underwent thinsection CT before surgical lung resection, emphysema was objectively quantified with computer-assisted macroscopic morphometry and CT densitometry. The percentage of lung macroscopically occupied by emphysema was compared with the percentage occupied on CT scans by pixels with attenuation values lower than a predefined threshold (CT densitometry). Three readers with varying degrees of expertise subjectively graded emphysema with visual assessment at two reading sessions. Data from objective quantification and subjective grading were analyzed with correlation coefficients, and interobserver and intraobserver agreement were calculated.

RESULTS

Subjective grading of emphysema showed less agreement with the macroscopic reference standard results (r = 0.439-0.505; P < .05) than with objective CT densitometric results (r = 0.555-0.623; P < .001). The 95% CIs for the intercepts of the linear regression lines were suggestive of systematic subjective overestimation of emphysema by all three readers. Interobserver agreement was moderate (kappa = 0.431-0.589). Intraobserver agreement was good to excellent (kappa = 0.738-0.936). The expertise of individual readers did not substantially influence results.

CONCLUSION

Systematic overestimation and moderate interobserver agreement may compromise subjective visual grading of emphysema, which suggests that subjective visual grading should be supplemented with objective methods to achieve precise, reader-independent quantification of emphysema.

Hydrostatic pulmonary edema: evaluation with thin-section CT in dogs

PURPOSE

To identify the hemodynamic determinants of ground-glass opacification on thin-section computed tomographic (CT) scans of hydrostatic pulmonary edema and to compare attenuation and subjective assessments of ground-glass opacification with extravascular lung water.

MATERIALS AND METHODS

Left atrial pressure, pulmonary arterial pressure, effective pulmonary capillary pressure, and extravascular lung water were measured in six dogs before and during progressive increase of effective pulmonary capillary pressure. A thin-section CT scan was obtained at each step. Lung attenuation and subjective assessments of ground-glass opacification were compared with hemodynamic variables and extravascular lung water.

RESULTS

Ground-glass opacification was identified when effective pulmonary capillary pressure equaled critical pulmonary capillary pressure. Extravascular lung water increased, and the distribution curve of lung attenuation coefficients shifted to higher attenuation from the second measurement at an effective pulmonary capillary pressure greater than the critical pulmonary capillary pressure. Attenuation was highly correlated (r = 0.98, P < .001) with extravascular lung water; ground-glass opacification was detected before a significant (P = .615, analysis of variance) increase in extravascular lung water.

CONCLUSION

Thin-section CT depicts ground-glass opacification when effective pulmonary capillary pressure equals critical pulmonary capillary pressure and before a detectable increase in extravascular lung water. Attenuation reflects extravascular lung water.

Forced choice and ordinal discrete rating assessment of image quality: a comparison

This study compared a five-category ordinal scale and a two-alternative forced-choice subjective rating of image quality preferences in a multiabnormality environment. 140 pairs of laser-printed posteroanterior (PA) chest images were evaluated twice by three radiologists who were asked to select during a side-by-side review which image in each pair was the "better" one for the determination of the presence or absence of specific abnormalities. Each pair included one image (the digitized film at 100 microns pixel resolution and laser printed onto film) and a highly compressed (approximately 60:1) and decompressed version of the digitized film that was laser printed onto film. Ratings were performed once with a five-category ordinal scale and once with a two-alternative forced-choice scale. The selection process was significantly affected by the rating scale used. The "comparable" or "equivalent for diagnosis "category was used in 88.5% of the ratings with the ordinal scale. When using the two-alternative forced-choice approach, noncompressed images were selected 66.8% of the time as being the "better" images. This resulted in a significantly lower ability to detect small differences in perceived image quality between the noncompressed and compressed images when the ordinal rating scale is used. Observer behavior can be affected by the type of question asked and the rating scale used. Observers are highly sensitive to small differences in image presentation during a side-by-side review.