Forced choice and ordinal discrete rating assessment of image quality: a comparison

Investigating the Small Pixel Effect in Ultra-High Resolution Photon-Counting CT of the Lung

OBJECTIVES

The aim of this study was to investigate potential benefits of ultra-high resolution (UHR) over standard resolution scan mode in ultra-low dose photon-counting detector CT (PCD-CT) of the lung.

MATERIALS AND METHODS

Six cadaveric specimens were examined with 5 dose settings using tin prefiltration, each in UHR (120 × 0.2 mm) and standard mode (144 × 0.4 mm), on a first-generation PCD-CT scanner. Image quality was evaluated quantitatively by noise comparisons in the trachea and both main bronchi. In addition, 16 readers (14 radiologists and 2 internal medicine physicians) independently completed a browser-based pairwise forced-choice comparison task for assessment of subjective image quality. The Kendall rank coefficient ( W ) was calculated to assess interrater agreement, and Pearson's correlation coefficient ( r ) was used to analyze the relationship between noise measurements and image quality rankings.

RESULTS

Across all dose levels, image noise in UHR mode was lower than in standard mode for scan protocols matched by CTDI vol ( P < 0.001). UHR examinations exhibited noise levels comparable to the next higher dose setting in standard mode ( P ≥ 0.275). Subjective ranking of protocols based on 5760 pairwise tests showed high interrater agreement ( W = 0.99; P ≤ 0.001) with UHR images being preferred by readers in the majority of comparisons. Irrespective of scan mode, a substantial indirect correlation was observed between image noise and subjective image quality ranking ( r = -0.97; P ≤ 0.001).

CONCLUSIONS

In PCD-CT of the lung, UHR scan mode reduces image noise considerably over standard resolution acquisition. Originating from the smaller detector element size in fan direction, the small pixel effect allows for superior image quality in ultra-low dose examinations with considerable potential for radiation dose reduction.

How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

OBJECTIVES

The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans.

METHODS

Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC).

RESULTS

Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396-0.814) to 0.785 (95%CI 0.676-0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034-0.500) to 0.562 (95%CI 0.337-0.729). Intraobserver agreement increased for four out of six observers.

CONCLUSION

The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement.

CLINICAL RELEVANCE STATEMENT

This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms.

KEY POINTS

• Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies.

Image quality evaluation of dual-layer spectral CT in comparison to single-layer CT in a reduced-dose setting

Objectives

To quantitatively and qualitatively evaluate image quality in dual-layer CT (DLCT) compared to single-layer CT (SLCT) in the thorax, abdomen, and pelvis in a reduced-dose setting.

Methods

Intraindividual, retrospective comparisons were performed in 25 patients who received at least one acquisition of all three acquisition protocols SLCT_low (100 kVp), DLCT_high (120 kVp), and DLCT_low (120 kVp), all covering the venous-phase thorax, abdomen, and pelvis with matched CTDI_vol between SLCT_low and DLCT_low. Reconstruction parameters were identical between all scans. Image quality was assessed quantitatively at 10 measurement locations in the thorax, abdomen, and pelvis by two independent observers, and subjectively with an intraindividual forced choice test between the three acquisitions. Dose-length product (DLP) and CTDI_vol were extracted for dose comparison.

Results

Despite matched CTDI_vol in acquisition protocols, CTDI_vol and DLP were lower for SLCT_low compared to DLCT_low and DLCT_high (DLP 408.58, 444.68, 647.08 mGy·cm, respectively; p < 0.0004), as automated tube current modulation for DLCT_low reached the lower limit in the thorax (mean 66.1 mAs vs limit 65 mAs). Noise and CNR were comparable between SLCT_low and DLCT_low (p values, 0.29–0.51 and 0.05–0.20), but CT numbers were significantly higher for organs and vessels in the upper abdomen for SLCT_low compared to DLCT_low. DLCT_high had significantly better image quality (Noise and CNR). Subjective image quality was superior for DLCT_high, but no difference was found between SLCT_low and DLCT_low.

Conclusions

DLCT_low showed comparable image quality to SLCT_low, with the additional possibility of spectral post-processing. Further dose reduction seems possible by decreasing the lower limit of the tube current for the thorax.

Key Points

• Clinical use of reduced-dose DLCT is feasible despite the required higher tube potential.

• DLCT with reduced dose shows comparable objective and subjective image quality to reduced-dose SLCT.

• Further dose reduction in the thorax might be possible by adjusting mAs thresholds.

Variable refocusing flip angle single-shot fast spin echo imaging of liver lesions: increased speed and lesion contrast

PURPOSE

To evaluate acquisition time and clinical image quality of a variable refocusing flip angle (vrf) single-shot fast spin echo (SSFSE) sequence in comparison with a conventional SSFSE sequence for imaging of liver lesions in patients undergoing whole-body PET/MRI for oncologic staging.

METHODS

A vrfSSFSE sequence was acquired in 43 patients with known pancreatic neuroendocrine tumors undergoing ⁶⁸Ga-DOTA-TOC PET on a simultaneous time-of-flight 3.0T PET/MRI. Liver lesions ≥1.5 cm with radionucleotide uptake were analyzed. Contrast-to-noise ratios (CNRs) were measured, and four blinded radiologists assessed overall image quality. Differences in repetition time and CNR were assessed using a paired Student's t test with p < 0.05 considered statistically significant. Inter-reader variability was assessed with Fleiss' kappa statistic.

RESULTS

53 eligible lesions in 27 patients were included for analysis. vrfSSFSE demonstrated higher mean lesion CNR compared to SSFSE (9.9 ± 4.1 vs. 6.7 ± 4.1, p < 0.001). Mean repetition time (TR) was 679 ± 97 ms for the vrfSSFSE sequence compared to 1139 ± 106 ms for SSFSE (p < 0.0001), corresponding to a 1.7-fold decrease in acquisition time. Overall quality of liver lesion and common bile duct images with the vrfSSFSE sequence was graded as superior than or equivalent to the SSFSE sequence for 59% and 67% of patients, respectively.

CONCLUSIONS

Compared to conventional SSFSE, vrfSSFSE resulted in improved lesion contrast on simultaneous PET/MRI in patients with liver metastases. Due to decreased SAR demands, vrfSSFSE significantly decreased TR, allowing coverage of the entire liver in a single twenty-second breath hold. This may have important clinical implications in the setting of PET/MRI, where scan time is limited by the necessity of whole-body image acquisition in addition to bed specific imaging.

Pairwise comparison versus Likert scale for biomedical image assessment

OBJECTIVE

Biomedical imaging research relies heavily on the subjective and semi-quantitative reader analysis of images. Current methods are limited by interreader variability and fixed upper and lower limits. The purpose of this study was to compare the performance of two assessment methods, pairwise comparison and Likert scale, for improved analysis of biomedical images.

MATERIALS AND METHODS

A set of 10 images with varying degrees of image sharpness was created by digitally blurring a normal clinical chest radiograph. Readers assessed the degree of image sharpness using two different methods: pairwise comparison and a 10-point Likert scale. Reader agreement with actual chest radiograph sharpness was calculated for each method by use of the Lin concordance correlation coefficient (CCC).

RESULTS

Reader accuracy was highest for pairwise comparison (CCC, 1.0) and ranked Likert (CCC, 0.99) scores and lowest for nonranked Likert scores (CCC, 0.83). Accuracy improved slightly when readers repeated their assessments (CCC, 0.87) or had reference images available (CCC, 0.91).

CONCLUSION

Pairwise comparison and ranked Likert scores yield more accurate reader assessments than nonranked Likert scores.

Seven Challenges in Image Quality Assessment: Past, Present, and Future Research

Image quality assessment (IQA) has been a topic of intense research over the last several decades. With each year comes an increasing number of new IQA algorithms, extensions of existing IQA algorithms, and applications of IQA to other disciplines. In this article, I first provide an up-to-date review of research in IQA, and then I highlight several open challenges in this field. The first half of this article provides discuss key properties of visual perception, image quality databases, existing full-reference, no-reference, and reduced-reference IQA algorithms. Yet, despite the remarkable progress that has been made in IQA, many fundamental challenges remain largely unsolved. The second half of this article highlights some of these challenges. I specifically discuss challenges related to lack of complete perceptual models for: natural images, compound and suprathreshold distortions, and multiple distortions, and the interactive effects of these distortions on the images. I also discuss challenges related to IQA of images containing nontraditional, and I discuss challenges related to the computational efficiency. The goal of this article is not only to help practitioners and researchers keep abreast of the recent advances in IQA, but to also raise awareness of the key limitations of current IQA knowledge.

The effect of reduced breast compression in breast tomosynthesis: human observer study using clinical cases

The aim of this study was to investigate whether the compression force used with conventional mammography can be reduced with breast tomosynthesis (BT), without adversely affecting the visualisation of normal and pathological structures. Forty-five women were examined with BT using full (same as for 2D mammography) and half compression force. Both examinations were performed with the same acquisition parameters. A total of 103 paired structure images were evaluated according to specified image quality criteria. Three experienced radiologists participated in the study. They had to make a forced choice, i.e. choose the image they felt best fulfilled the image quality criteria. The results showed no evident difference in the image quality, indicating that BT may be performed with substantially less compression force compared with 2D mammography. A majority of the examined women felt that half compression was more comfortable than full compression.

Summation or axial slab average intensity projection of abdominal thin-section CT datasets: can they substitute for the primary reconstruction from raw projection data?

We hypothesized that that the summation or axial slab average intensity projection (AIP) techniques can substitute for the primary reconstruction (PR) from a raw projection data for abdominal applications. To compare with PR datasets (5-mm thick, 20% overlap) in 150 abdominal studies, corresponding summation and AIP datasets were calculated from 2-mm thick images (50% overlap). The root-mean-square error between PR and summation images was significantly greater than that between PR and AIP images (9.55 [median] vs. 7.12, p < 0.0001, Wilcoxon signed-ranks test). Four radiologists independently compared 2,000 test images (PR [as control], summation, or AIP) and their corresponding PR images to prove that the identicalness of summation or AIP images to PR images was not 1% less than the assessed identicalness of PR images to themselves (Wald-type test for clustered matched-pair data in a non-inferiority design). For each reader, both summation and AIP images were not inferior to PR images in terms of being rated identical to PR (p < 0.05). Although summation and AIP techniques produce images that differ from PR images, these differences are not easily perceived by radiologists. Thus, the summation or AIP techniques can substitute for PR for the primary interpretation of abdominal CT.

Image quality, compression and segmentation in medicine

This review considers image quality in the context of the evolving technology of image compression, and the effects image compression has on perceived quality. The concepts of lossless, perceptually lossless, and diagnostically lossless but lossy compression are described, as well as the possibility of segmented images, combining lossy compression with perceptually lossless regions of interest. The different requirements for diagnostic and training images are also discussed. The lack of established methods for image quality evaluation is highlighted and available methods discussed in the light of the information that may be inferred from them. Confounding variables are also identified. Areas requiring further research are illustrated, including differences in perceptual quality requirements for different image modalities, image regions, diagnostic subtleties, and tasks. It is argued that existing tools for measuring image quality need to be refined and new methods developed. The ultimate aim should be the development of standards for image quality evaluation which take into consideration both the task requirements of the images and the acceptability of the images to the users.

Dose efficiency and low-contrast detectability of an amorphous silicon x-ray detector for digital radiography

The effect of dose reduction on low-contrast detectability is investigated theoretically and experimentally for a production grade amorphous silicon (a-Si) x-ray detector and compared with a standard thoracic screen-film combination. A non-prewhitening matched filter observer model modified to include a spatial response function and internal noise for the human visual system (HVS) is used to calculate a signal-to-noise ratio (SNR) related to object detectability. Other inputs to the SNR calculation are the detective quantum efficiency (DQE) and the modulation transfer function (MTF) of the imaging system. Besides threshold detectability, the model predicts the equivalent perception dose ratio (EPDR), which is the fraction of the screen film exposure for which the digital detector provides equal detectability. Images of a contrast-detail phantom are obtained with the digital detector at dose levels corresponding to 27%, 41%, 63% and 100% of the dose used for screen-film. The images are used in a four-alternative forced choice (4-AFC) observer perception study in order to measure threshold detectability. A statistically significant improvement in contrast detectability is measured with the digital detector at 100% and 63% of the screen-film dose. There is no statistical difference between screen-film and digital at 41% of the dose. On average, the experimental EPDR is 44%, which agrees well with the model prediction of 40%.

Multipoint rank-order study methodology: observer issues

RATIONALE AND OBJECTIVES

We performed a multipoint rank-order experiment to evaluate variability in observers' sensitivity to small differences in image presentation and to assess observers' performance as a function of the type and number of tasks included.

METHODS

Five experienced observers were presented with four sets of chest images that had been compressed at five different levels. Each set contained six images ranging from noncompressed to approximately 60:1-compressed images. Observers were asked to review all images of each case side by side and rank-order the "quality" of each to enable determination of the presence or absence of interstitial disease and/or pneumothoraces.

RESULTS

Observers varied significantly in their ability to detect very small differences among the images (P < 0.001). Those who performed well did so regardless of whether they ranked a specific abnormality in a multidisease or a single-disease setting.

CONCLUSIONS

Selected observers can reliably detect very small differences among similar images. These readers could be used to confirm or rule out the need for objective observer-performance-type studies.