Toward objective and quantitative evaluation of imaging systems using images of phantoms

Adaptation of a channelized Hotelling observer model to accommodate anthropomorphic backgrounds and moving test objects in X-ray angiography

BACKGROUND

Prior implementations of the channelized Hotelling observer (CHO) model have succeeded in assessing the performance of X-ray angiography systems under a variety of imaging conditions. However, often times these conditions do not resemble those present in routine clinical imaging scenarios, such as having complex anthropomorphic backgrounds in conjunction with moving test objects.

PURPOSE

This work builds up on prior established CHO methods and introduces a new approach to switch from the already established "multiple-sample" CHO implementation to a "single-sample" technique. The proposed implementation enables the inclusion of moving test objects upon nonuniform backgrounds by allowing only a single sample to represent the test object present condition that is to be used within the statistical test to estimate the detectability index.

METHODS

To assess the proposed method, two image data sets were acquired with a clinical X-ray angiography system. The first set consisted of a uniform background in combination with static test objects while the second consisted of an anthropomorphic chest phantom in conjunction with moving test objects. The first set was used to validate the proposed approach against the multiple-sample method while the second was used to assess the feasibility of the proposed method under a variety of imaging conditions, including seven object sizes and seven detector target dose (DTD) levels.

RESULTS

For the uniform background data set, considering all detectability indices greater or equal than 1, the ratio between the detectability indices of the proposed single-sample approach versus the multiple-sample method was 0.997 ± 0.056 (range 0.884-1.159). The average single-direction width of the 95% confidence intervals (CIs) of the detectability index estimates for the multiple-sample method was 0.38 ± 0.43 (range 0.03-2.20). For the single-sample approach, the average width was 2.52 ± 0.63 (range 1.11-5.44). For the anthropomorphic background image set, the results were consistent with classical quantum-limited signal-to-noise ratio (SNR) theory. The magnitude of the detectability indices varied predictably with changes in both object size and DTD, with the highest value associated with the highest dose and the largest object size. Additionally, the proposed method was able to capture differences in the imaging performance for a given test object across the field of view, which was associated with the attenuation levels provided by different features of the anthropomorphic background.

CONCLUSIONS

A new single-sample variant of the CHO model to assess the performance of X-ray angiography imaging systems is proposed. The new approach is consistent with quantum-limited image quality theory and with a standard implementation of the CHO model. The proposed method enables the assessment of moving test objects in combination with complex, nonuniform image backgrounds, thereby opening the possibility to assess imaging conditions which more closely resemble those used in clinical care.

Accurate and efficient measurement of channelized Hotelling observer-based low-contrast detectability on the ACR CT accreditation phantom

BACKGROUND

Current CT quality control (QC) for low-contrast detectability relies on visual inspection and measurement of contrast-to-noise ratio (CNR). However, CNR numbers become unreliable when it comes to nonlinear methods, such as iterative reconstruction (IR) and deep-learning-based techniques. Image quality metrics using channelized Hotelling observer (CHO) have been validated to be well correlated with human observer performance on phantom-based and patient-based tasks, but it has not been widely used in routine CT QC mainly because the CHO calculation typically requires a large number of repeated scans in order to provide accurate and precise estimate of index of detectability (d').

PURPOSE

The main goal of this work is to optimize channel filters and other CHO parameters and accurately estimate the low-contrast detectability with minimum number of repeated scans for the widely used American College of Radiology (ACR) CT accreditation phantom so that it can become practically feasible for routine CT QC tests.

METHODS

To provide a converged d' value, an ACR phantom was repeatedly scanned 100 times at three dose levels (24, 12, and 6 mGy). Images were reconstructed with two kernels (FBP Br44 and IR Br44-3). d' as a function of number of repeated scans was determined for different number of background regions of interest (ROIs), different number of low-contrast objects, different number of slices per each object, and different channel filter options. A reference d' was established using the optimized CHO setting, and the bias of d' was quantified using the d' calculated from all 100 repeated scans. The variation of d' at each condition was estimated using a resampling method combining random subsampling among 100 repeated scans and bootstrapping of the ensembles of signal and background ROIs.

RESULTS

Optimized parameters in CHO calculation were determined: two background ROIs per object, four objects per low-contrast object size, nine non-overlapping slices per object, and a 4-channel Gabor filter. The bias and uncertainty were estimated at different numbers of repeated scans using these parameters. When only one single scan was used in the CHO calculation, the bias of d' was below 6.2% and the uncertainty 15.6-19.6% for the 6, 5, and 4 mm objects, while with three repeated scans the bias was below 2.0% and uncertainty 8.7-10.9% for the three object sizes.

CONCLUSION

With optimized parameter settings in CHO, efficient and accurate measurement of low-contrast detectability on the commonly used ACR phantom becomes feasible, which could potentially lead to adoption of CHO-based low-contrast evaluation in routine QC tests.

Optimizing constrained reconstruction in magnetic resonance imaging for signal detection

Constrained reconstruction in magnetic resonance imaging (MRI) allows the use of prior information through constraints to improve reconstructed images. These constraints often take the form of regularization terms in the objective function used for reconstruction. Constrained reconstruction leads to images which appear to have fewer artifacts than reconstructions without constraints but because the methods are typically nonlinear, the reconstructed images have artifacts whose structure is hard to predict. In this work, we compared different methods of optimizing the regularization parameter using a total variation (TV) constraint in the spatial domain and sparsity in the wavelet domain for one-dimensional (2.56×) undersampling using variable density undersampling. We compared the mean squared error (MSE), structural similarity (SSIM), L-curve and the area under the receiver operating characteristic (AUC) using a linear discriminant for detecting a small and a large signal. We used a signal-known-exactly task with varying backgrounds in a simulation where the anatomical variation was the major source of clutter for the detection task. Our results show that the AUC dependence on regularization parameters varies with the imaging task (i.e. the signal being detected). The choice of regularization parameters for MSE, SSIM, L-curve and AUC were similar. We also found that a model-based reconstruction including TV and wavelet sparsity did slightly better in terms of AUC than just enforcing data consistency but using these constraints resulted in much better MSE and SSIM. These results suggest that the increased performance in MSE and SSIM over-estimate the improvement in detection performance for the tasks in this paper. The MSE and SSIM metrics show a big difference in performance where the difference in AUC is small. To our knowledge, this is the first time that signal detection with varying backgrounds has been used to optimize constrained reconstruction in MRI.

Selection and evaluation of spherical acquisition trajectories for industrial computed tomography

In conventional industrial computed tomography, the source–detector system rotates in equiangular steps in-plane relative to the part of investigation. While being by far the most frequently used acquisition trajectory today, this method has several drawbacks like the formation of cone beam artefacts or limited usability in case of geometrical restrictions. In such cases, the usage of alternative spherical trajectories can be beneficial to improve image quality and defect visibility. While investigations have been performed to relate the influence of the trajectory choice in the typical metrological case of a high number of available projections, so far barely any work has been done for the case of few source–detector poses, which is more relevant in the field of non-destructive testing. In this work, we provide an overview of quantitative metrics that can be used to assess the image quality of reconstructed computed tomography volumes, discuss their advantages and drawbacks and propose a framework to investigate the performance of several non-standard trajectories with respect to previously defined regions of interest. Inspired by pseudorandom sampling methods for Monte–Carlo-algorithms, we also suggest an entirely new trajectory design, the low-discrepancy spherical trajectory, which extends the concept of equiangular planar trajectories into three dimensions and can be used for benchmarking and comparison with other spherical trajectories. Last, we use an optimization method to calculate task-specific acquisition trajectories and relate their performance to other spherical designs.

Assessment of task-based performance from five clinical DBT systems using an anthropomorphic breast phantom

PURPOSE

Digital breast tomosynthesis (DBT) is a limited-angle tomographic breast imaging modality that can be used for breast cancer screening in conjunction with full-field digital mammography (FFDM) or synthetic mammography (SM). Currently, there are five commercial DBT systems that have been approved by the U.S. FDA for breast cancer screening, all varying greatly in design and imaging protocol. Because the systems are different in technical specifications, there is a need for a quantitative approach for assessing them. In this study, the DBT systems are assessed using a novel methodology with an inkjet-printed anthropomorphic phantom and four alternative forced choice (4AFC) study scheme.

METHOD

A breast phantom was fabricated using inkjet printing and parchment paper. The phantom contained 5-mm spiculated masses fabricated with potassium iodide (KI)-doped ink and microcalcifications (MCs) made with calcium hydroxyapatite. Images of the phantom were acquired on all five systems with DBT, FFDM, and SM modalities where available using beam settings under automatic exposure control. A 4AFC study was conducted to assess reader performance with each signal under each modality. Statistical analysis was performed on the data to determine proportion correct (PC), standard deviations, and levels of significance.

RESULTS

For masses, overall detection was highest with DBT. The difference in PC was statistically significant between DBT and SM for most systems. A relationship was observed between increasing PC and greater gantry span. For MCs, performance was highest with DBT and FFDM compared to SM. The difference between PC of DBT and PC of SM was statistically significant for all manufacturers.

CONCLUSIONS

This methodology represents a novel approach for evaluating systems. This study is the first of its kind to use an inkjet-printed anthropomorphic phantom with realistic signals to assess performance of clinical DBT imaging systems.

Optimization of the difference-of-Gaussian channel sets for the channelized Hotelling observer

The channelized-Hotelling observer (CHO) was investigated as a surrogate of human observers in task-based image quality assessment. The CHO with difference-of-Gaussian (DoG) channels has shown potential for the prediction of human detection performance in digital mammography (DM) images. However, the DoG channels employ parameters that describe the shape of each channel. The selection of these parameters influences the performance of the DoG CHO and needs further investigation. The detection performance of the DoG CHO was calculated and correlated with the detection performance of three humans who evaluated DM images in 2-alternative forced-choice experiments. A set of DM images of an anthropomorphic breast phantom with and without calcification-like signals was acquired at four different dose levels. For each dose level, 200 square regions-of-interest (ROIs) with and without signal were extracted. Signal detectability was assessed on ROI basis using the CHO with various DoG channel parameters and it was compared to that of the human observers. It was found that varying these DoG parameter values affects the correlation (r 2 ) of the CHO with human observers for the detection task investigated. In conclusion, it appears that the the optimal DoG channel sets that maximize the prediction ability of the CHO might be dependent on the type of background and signal of ROIs investigated.

Can a channelized Hotelling observer assess image quality in acquired mammographic images of an anthropomorphic breast phantom including image processing?

PURPOSE

To study the feasibility of a channelized Hotelling observer (CHO) to predict human observer performance in detecting calcification-like signals in mammography images of an anthropomorphic breast phantom, as part of a quality control (QC) framework.

METHODS

A prototype anthropomorphic breast phantom with inserted gold disks of 0.25 mm diameter was imaged with two different digital mammography x-ray systems at four different dose levels. Regions of interest (ROIs) were extracted from the acquired processed and unprocessed images, signal-present and signal-absent. The ROIs were evaluated by a CHO using four different formulations of the difference of Gaussian (DoG) channel sets. Three human observers scored the ROIs in a two-alternative forced-choice experiment. We compared the human and the CHO performance on the simple task to detect calcification-like disks in ROIs with and without postprocessing. The proportion of correct responses of the human reader (PC_H ) and the CHO (PC_CHO ) was calculated and the correlation between the two was analyzed using a mixed-effect regression model. To address the signal location uncertainty, the impact of shifting the DoG channel sets in all directions up to two pixels was evaluated. Correlation results including the goodness of fit (r² ) of PC_H and PC_CHO for all different parameters were evaluated.

RESULTS

Subanalysis by system yielded strong correlations between PC_H and PC_CHO , with r² between PC_H and PC_CHO was found to be between 0.926 and 0.958 for the unshifted and between 0.759 and 0.938 for the shifted channel sets, respectively. However, the linear fit suggested a slight system dependence. PC_CHO with shifted channel sets increased CHO performance but the correlation with humans was decreased. These correlations were not considerably affected by of the DoG channel set used.

CONCLUSIONS

There is potential for the CHO to be used in QC for the evaluation of detectability of calcification-like signals. The CHO can predict the PC of humans in images of calcification-like signals of two different systems. However, a global model to be used for all systems requires further investigation.

Signal template generation from acquired images for model observer-based image quality analysis in mammography

Mammography images undergo vendor-specific processing, which may be nonlinear, before radiologist interpretation. Therefore, to test the entire imaging chain, the effect of image processing should be included in the assessment of image quality, which is not current practice. For this purpose, model observers (MOs), in combination with anthropomorphic breast phantoms, are proposed to evaluate image quality in mammography. In this study, the nonprewhitening MO with eye filter and the channelized Hotelling observer were investigated. The goal of this study was to optimize the efficiency of the procedure to obtain the expected signal template from acquired images for the detection of a 0.25-mm diameter disk. Two approaches were followed: using acquired images with homogeneous backgrounds (approach 1) and images from an anthropomorphic breast phantom (approach 2). For quality control purposes, a straightforward procedure using a single exposure of a single disk was found adequate for both approaches. However, only approach 2 can yield templates from processed images since, due to its nonlinearity, image postprocessing cannot be evaluated using images of homogeneous phantoms. Based on the results of the current study, a phantom should be designed, which can be used for the objective assessment of image quality.

Toward image quality assessment in mammography using model observers: Detection of a calcification-like object

PURPOSE

Model observers (MOs) are of interest in the field of medical imaging to assess image quality. However, before procedures using MOs can be proposed in quality control guidelines for mammography systems, we need to know whether MOs are sensitive to changes in image quality and correlations in background structure. Therefore, as a proof of principle, in this study human and model observer (MO) performance are compared for the detection of calcification-like objects using different background structures and image quality levels of unprocessed mammography images.

METHOD

Three different phantoms, homogeneous polymethyl methacrylate, BR3D slabs with swirled patterns (CIRS, Norfolk, VA, USA), and a prototype anthropomorphic breast phantom (Institute of Medical Physics and Radiation Protection, Technische Hochschule Mittelhessen, Germany) were imaged on an Amulet Innovality (FujiFilm, Tokyo, Japan) mammographic X-ray unit. Because the complexities of the structures of these three phantoms were different and not optimized to match the characteristics of real mammographic images, image processing was not applied in this study. In addition, real mammograms were acquired on the same system. Regions of interest (ROIs) were extracted from each image. In half of the ROIs, a 0.25-mm diameter disk was inserted at four different contrast levels to represent a calcification-like object. Each ROI was then modified, so four image qualities relevant for mammography were simulated. The signal-present and signal-absent ROIs were evaluated by a non-pre-whitening model observer with eye filter (NPWE) and a channelized Hotelling observer (CHO) using dense difference of Gaussian channels. The ROIs were also evaluated by human observers in a two alternative forced choice experiment. Detectability results for the human and model observer experiments were correlated using a mixed-effect regression model. Threshold disk contrasts for human and predicted human observer performance based on the NPWE MO and CHO were estimated.

RESULTS

Global trends in threshold contrast were similar for the different background structures, but absolute contrast threshold levels differed. Contrast thresholds tended to be lower in ROIs from simple phantoms compared with ROIs from real mammographic images. The correlation between human and model observer performance was not affected by the range of image quality levels studied.

CONCLUSIONS

The correlation between human and model observer performance does not depend on image quality. This is a promising outcome for the use of model observers in image quality analysis and allows for subsequent research toward the development of MO-based quality control procedures and guidelines.

Use of a channelized Hotelling observer to assess CT image quality and optimize dose reduction for iteratively reconstructed images

The use of iterative reconstruction (IR) algorithms in CT generally decreases image noise and enables dose reduction. However, the amount of dose reduction possible using IR without sacrificing diagnostic performance is difficult to assess with conventional image quality metrics. Through this investigation, achievable dose reduction using a commercially available IR algorithm without loss of low contrast spatial resolution was determined with a channelized Hotelling observer (CHO) model and used to optimize a clinical abdomen/pelvis exam protocol. A phantom containing 21 low contrast disks-three different contrast levels and seven different diameters-was imaged at different dose levels. Images were created with filtered backprojection (FBP) and IR. The CHO was tasked with detecting the low contrast disks. CHO performance indicated dose could be reduced by 22% to 25% without compromising low contrast detectability (as compared to full-dose FBP images) whereas 50% or more dose reduction significantly reduced detection performance. Importantly, default settings for the scanner and protocol investigated reduced dose by upward of 75%. Subsequently, CHO-based protocol changes to the default protocol yielded images of higher quality and doses more consistent with values from a larger, dose-optimized scanner fleet. CHO assessment provided objective data to successfully optimize a clinical CT acquisition protocol.

Can the channelized Hotelling observer including aspects of the human visual system predict human observer performance in mammography?

PURPOSE

In mammography, images are processed prior to display. Model observers (MO) are candidates to objectively evaluate processed images if they can predict human observer performance for detail detection. The aim of this study was to investigate if the channelized Hotelling observer (CHO) can be configured to predict human observer performance in mammography like images.

METHODS

The performance correlation between human observers and CHO has been evaluated using different channel-sets and by including aspects of the human visual system (HVS). The correlation was investigated for the detection of disk-shaped details in simulated white noise (WN) and clustered lumpy backgrounds (CLB) images, representing respectively quantum noise limited and mammography like images. The images were scored by the MO and five human observers in 2-alternative forced choice experiments.

RESULTS

For WN images the most useful formulation of the CHO to predict human observer performance was obtained using three difference of Gaussian channels without adding HVS aspects (RLR2=0.62). For CLB images the most useful formulation was the partial least square channel-set without adding HVS aspects (RLR2=0.71). The correlation was affected by detail size and background.

CONCLUSIONS

This study has shown that the CHO can predict human observer performance. Due to object size and background dependency it is important that the range of object sizes and allowed variability in background are specified and validated carefully before the CHO can be implemented for objective image quality assessment.

Direct estimation and correction of bias from temporally variable non-stationary noise in a channelized Hotelling model observer

Channelized Hotelling model observer (CHO) methods were developed to assess performance of an x-ray angiography system. The analytical methods included correction for known bias error due to finite sampling. Detectability indices ([Formula: see text]) corresponding to disk-shaped objects with diameters in the range 0.5-4 mm were calculated. Application of the CHO for variable detector target dose (DTD) in the range 6-240 nGy frame(-1) resulted in [Formula: see text] estimates which were as much as 2.9×  greater than expected of a quantum limited system. Over-estimation of [Formula: see text] was presumed to be a result of bias error due to temporally variable non-stationary noise. Statistical theory which allows for independent contributions of 'signal' from a test object (o) and temporally variable non-stationary noise (ns) was developed. The theory demonstrates that the biased [Formula: see text] is the sum of the detectability indices associated with the test object [Formula: see text] and non-stationary noise ([Formula: see text]). Given the nature of the imaging system and the experimental methods, [Formula: see text] cannot be directly determined independent of [Formula: see text]. However, methods to estimate [Formula: see text] independent of [Formula: see text] were developed. In accordance with the theory, [Formula: see text] was subtracted from experimental estimates of [Formula: see text], providing an unbiased estimate of [Formula: see text]. Estimates of [Formula: see text] exhibited trends consistent with expectations of an angiography system that is quantum limited for high DTD and compromised by detector electronic readout noise for low DTD conditions. Results suggest that these methods provide [Formula: see text] estimates which are accurate and precise for [Formula: see text]. Further, results demonstrated that the source of bias was detector electronic readout noise. In summary, this work presents theory and methods to test for the presence of bias in Hotelling model observers due to temporally variable non-stationary noise and correct this bias when the temporally variable non-stationary noise is independent and additive with respect to the test object signal.

Impact of number of repeated scans on model observer performance for a low-contrast detection task in computed tomography

Channelized Hotelling observer (CHO) models have been shown to correlate well with human observers for several phantom-based detection/classification tasks in clinical computed tomography (CT). A large number of repeated scans were used to achieve an accurate estimate of the model's template. The purpose of this study is to investigate how the experimental and CHO model parameters affect the minimum required number of repeated scans. A phantom containing 21 low-contrast objects was scanned on a 128-slice CT scanner at three dose levels. Each scan was repeated 100 times. For each experimental configuration, the low-contrast detectability, quantified as the area under receiver operating characteristic curve, [Formula: see text], was calculated using a previously validated CHO with randomly selected subsets of scans, ranging from 10 to 100. Using [Formula: see text] from the 100 scans as the reference, the accuracy from a smaller number of scans was determined. Our results demonstrated that the minimum number of repeated scans increased when the radiation dose level decreased, object size and contrast level decreased, and the number of channels increased. As a general trend, it increased as the low-contrast detectability decreased. This study provides a basis for the experimental design of task-based image quality assessment in clinical CT using CHO.

Anthropomorphic model observer performance in three-dimensional detection task for low-contrast computed tomography

X-ray medical imaging is increasingly becoming three-dimensional (3-D). The dose to the population and its management are of special concern in computed tomography (CT). Task-based methods with model observers to assess the dose-image quality trade-off are promising tools, but they still need to be validated for real volumetric images. The purpose of the present work is to evaluate anthropomorphic model observers in 3-D detection tasks for low-contrast CT images. We scanned a low-contrast phantom containing four types of signals at three dose levels and used two reconstruction algorithms. We implemented a multislice model observer based on the channelized Hotelling observer (msCHO) with anthropomorphic channels and investigated different internal noise methods. We found a good correlation for all tested model observers. These results suggest that the msCHO can be used as a relevant task-based method to evaluate low-contrast detection for CT and optimize scan protocols to lower dose in an efficient way.

Technical Note: Impact on detective quantum efficiency of edge angle determination method by International Electrotechnical Commission methodology for cardiac x-ray image detectors

PURPOSE

Cardiac x-ray detectors are used to acquire moving images in real-time for angiography and interventional procedures. Detective quantum efficiency (DQE) is not generally measured on these dynamic detectors; the required "for processing" image data and control of x-ray settings have not been accessible. By 2016, USA hospital physicists will have the ability to measure DQE and will likely utilize the International Electrotechnical Commission (IEC) standard for measuring DQE of dynamic x-ray imaging devices. The current IEC standard requires an image of a tilted tungsten edge test object to obtain modulation transfer function (MTF) for DQE calculation. It specifies the range of edge angles to use; however, it does not specify a preferred method to determine this angle for image analysis. The study aimed to answer the question "will my choice in method impact my results?" Four different established edge angle determination methods were compared to investigate the impact on DQE.

METHODS

Following the IEC standard, edge and flat field images were acquired on a cardiac flat-panel detector to calculate MTF and noise power spectrum, respectively, to determine DQE. Accuracy of the methods in determining the correct angle was ascertained using a simulated edge image with known angulations. Precision of the methods was ascertained using variability of MTF and DQE, calculated via bootstrapping.

RESULTS

Three methods provided near equal angles and the same MTF while the fourth, with an angular difference of 6%, had a MTF lower by 3% at 1.5 mm(-1) spatial frequency and 8% at 2.5 mm(-1); corresponding DQE differences were 6% at 1.5 mm(-1) and 17% at 2.5 mm(-1); differences were greater than standard deviations in the measurements.

CONCLUSIONS

DQE measurements may vary by a significant amount, depending on the method used to determine the edge angle when following the IEC standard methodology for a cardiac x-ray detector. The most accurate and precise methods are recommended for absolute assessments and reproducible measurements, respectively.

Exact confidence intervals for channelized Hotelling observer performance in image quality studies

Task-based assessments of image quality constitute a rigorous, principled approach to the evaluation of imaging system performance. To conduct such assessments, it has been recognized that mathematical model observers are very useful, particularly for purposes of imaging system development and optimization. One type of model observer that has been widely applied in the medical imaging community is the channelized Hotelling observer (CHO), which is well-suited to known-location discrimination tasks. In the present work, we address the need for reliable confidence interval estimators of CHO performance. Specifically, we show that the bias associated with point estimates of CHO performance can be overcome by using confidence intervals proposed by Reiser for the Mahalanobis distance. In addition, we find that these intervals are well-defined with theoretically-exact coverage probabilities, which is a new result not proved by Reiser. The confidence intervals are tested with Monte Carlo simulation and demonstrated with two examples comparing X-ray CT reconstruction strategies. Moreover, commonly-used training/testing approaches are discussed and compared to the exact confidence intervals. MATLAB software implementing the estimators described in this work is publicly available at http://code.google.com/p/iqmodelo/.

Implementation of a channelized Hotelling observer model to assess image quality of x-ray angiography systems

Evaluation of flat-panel angiography equipment through conventional image quality metrics is limited by the scope of standard spatial-domain image quality metric(s), such as contrast-to-noise ratio and spatial resolution, or by restricted access to appropriate data to calculate Fourier domain measurements, such as modulation transfer function, noise power spectrum, and detective quantum efficiency. Observer models have been shown capable of overcoming these limitations and are able to comprehensively evaluate medical-imaging systems. We present a spatial domain-based channelized Hotelling observer model to calculate the detectability index (DI) of our different sized disks and compare the performance of different imaging conditions and angiography systems. When appropriate, changes in DIs were compared to expectations based on the classical Rose model of signal detection to assess linearity of the model with quantum signal-to-noise ratio (SNR) theory. For these experiments, the estimated uncertainty of the DIs was less than 3%, allowing for precise comparison of imaging systems or conditions. For most experimental variables, DI changes were linear with expectations based on quantum SNR theory. DIs calculated for the smallest objects demonstrated nonlinearity with quantum SNR theory due to system blur. Two angiography systems with different detector element sizes were shown to perform similarly across the majority of the detection tasks.

Evaluation of clinical full field digital mammography with the task specific system-model-based Fourier Hotelling observer (SMFHO) SNR

PURPOSE

The purpose of this work is to evaluate the performance of the image acquisition chain of clinical full field digital mammography (FFDM) systems by quantifying their image quality, and how well the desired information is captured by the images.

METHODS

The authors present a practical methodology to evaluate FFDM using the task specific system-model-based Fourier Hotelling observer (SMFHO) signal to noise ratio (SNR), which evaluates the signal and noise transfer characteristics of FFDM systems in the presence of a uniform polymethyl methacrylate phantom that models the attenuation of a 6 cm thick 20/80 breast (20% glandular/80% adipose). The authors model the system performance using the generalized modulation transfer function, which accounts for scatter blur and focal spot unsharpness, and the generalized noise power spectrum, both estimated with the phantom placed in the field of view. Using the system model, the authors were able to estimate system detectability for a series of simulated disk signals with various diameters and thicknesses, quantified by a SMFHO SNR map. Contrast-detail (CD) curves were generated from the SNR map and adjusted using an estimate of the human observer efficiency, without performing time-consuming human reader studies. Using the SMFHO method the authors compared two FFDM systems, the GE Senographe DS and Hologic Selenia FFDM systems, which use indirect and direct detectors, respectively.

RESULTS

Even though the two FFDM systems have different resolutions, noise properties, detector technologies, and antiscatter grids, the authors found no significant difference between them in terms of detectability for a given signal detection task. The authors also compared the performance between the two image acquisition modes (fine view and standard) of the GE Senographe DS system, and concluded that there is no significant difference when evaluated by the SMFHO. The estimated human observer efficiency was 30 ± 5% when compared to the SMFHO. The results showed good agreement when compared to other model observers as well as previously published human observer data.

CONCLUSIONS

This method generates CD curves from the SMFHO SNR that can be used as figures of merit for evaluating the image acquisition performance of clinical FFDM systems. It provides a way of creating an empirical model of the FFDM system that accounts for patient scatter, focal spot unsharpness, and detector blur. With the use of simulated signals, this method can predict system performance for a signal known exactly/background known exactly detection task with a limited number of images, therefore, it can be readily applied in a clinical environment.

Confidence intervals for performance assessment of linear observers

PURPOSE

This work seeks to develop exact confidence interval estimators for figures of merit that describe the performance of linear observers, and to demonstrate how these estimators can be used in the context of x-ray computed tomography (CT). The figures of merit are the receiver operating characteristic (ROC) curve and associated summary measures, such as the area under the ROC curve. Linear computerized observers are valuable for optimization of parameters associated with image reconstruction algorithms and data acquisition geometries. They provide a means to perform assessment of image quality with metrics that account not only for shift-variant resolution and nonstationary noise but that are also task-based.

METHODS

We suppose that a linear observer with fixed template has been defined and focus on the problem of assessing the performance of this observer for the task of deciding if an unknown lesion is present at a specific location. We introduce a point estimator for the observer signal-to-noise ratio (SNR) and identify its sampling distribution. Then, we show that exact confidence intervals can be constructed from this distribution. The sampling distribution of our SNR estimator is identified under the following hypotheses: (i) the observer ratings are normally distributed for each class of images and (ii) the variance of the observer ratings is the same for each class of images. These assumptions are, for example, appropriate in CT for ratings produced by linear observers applied to low-contrast lesion detection tasks.

RESULTS

Unlike existing approaches to the estimation of ROC confidence intervals, the new confidence intervals presented here have exactly known coverage probabilities when our data assumptions are satisfied. Furthermore, they are applicable to the most commonly used ROC summary measures, and they may be easily computed (a computer routine is supplied along with this article on the Medical Physics Website). The utility of our exact interval estimators is demonstrated through an image quality evaluation example using real x-ray CT images. Also, strong robustness is shown to potential deviations from the assumption that the ratings for the two classes of images have equal variance. Another aspect of our interval estimators is the fact that we can calculate their mean length exactly for fixed parameter values, which enables precise investigations of sampling effects. We demonstrate this aspect by exploring the potential reduction in statistical variability that can be gained by using additional images from one class, if such images are readily available. We find that when additional images from one class are used for an ROC study, the mean AUC confidence interval length for our estimator can decrease by as much as 35%.

CONCLUSIONS

We have shown that exact confidence intervals can be constructed for ROC curves and for ROC summary measures associated with fixed linear computerized observers applied to binary discrimination tasks at a known location. Although our intervals only apply under specific conditions, we believe that they form a valuable tool for the important problem of optimizing parameters associated with image reconstruction algorithms and data acquisition geometries, particularly in x-ray CT.

Optimisation of radiological protocols for chest imaging using computed radiography and flat-panel X-ray detectors

PURPOSE

Digital radiography technology has replaced conventional screen-film systems in many hospitals. Despite the different characteristics of new detector materials, frequently, the same radiological protocols previously optimised for screen film are still used with digital equipment without any critical review. This study addressed optimisation of exposure settings for chest examinations with digital systems, considering both image quality and patient dose.

MATERIALS AND METHODS

Images acquired with direct digital radiography equipment and a computed radiography system were analysed with specially developed commercial software with a four-alternative forced-choice method: the most promising protocols were then scored by two senior radiologists.

RESULTS

Digital technology offers a wide dynamic range and the ability to postprocess images, allowing use of lower tube potentials in chest examinations. The computed radiography system showed both better image quality and lower dose at lower energies (85 kVp and 95 kVp) than those currently used (125 kVp). Direct digital radiography equipment confirmed both its superior image quality and lower dose requirements compared with the storage phosphor plate system.

CONCLUSIONS

Generally, lowering tube potentials in chest examinations seems to allow better image quality/effective dose ratio when using digital equipment.

Comparison of signal to noise ratios from spatial and frequency domain formulations of nonprewhitening model observers in digital mammography

PURPOSE

Image quality indices based upon model observers are promising alternatives to laborious human readings of contrast-detail images. This is especially appealing in digital mammography as limiting values for contrast thresholds determine, according to some international protocols, the acceptability of these systems in the radiological practice. The objective of the present study was to compare the signal to noise ratios (SNR) obtained with two nonprewhitening matched filter model observer approaches, one in the spatial domain and the other in the frequency domain, and with both of them worked out for disks as present in the CDMAM phantom.

METHODS

The analysis was performed using images acquired with the Siemens Novation and Inspiration digital mammography systems. The spatial domain formulation uses a series of high dose CDMAM images as the signal and a routine exposure of two flood images to calculate the covariance matrix. The frequency domain approach uses the mathematical description of a disk and modulation transfer function (MTF) and noise power spectrum (NPS) calculated from images.

RESULTS

For both systems most of the SNR values calculated in the frequency domain were in very good agreement with the SNR values calculated in the spatial domain. Both the formulations in the frequency domain and in the spatial domain show a linear relationship between SNR and the diameter of the CDMAM discs.

CONCLUSIONS

The results suggest that both formulations of the model observer lead to very similar figures of merit. This is a step forward in the adoption of figures of merit based on NPS and MTF for the acceptance testing of mammography systems.

Use of the cross-correlation component of the multiscale structural similarity metric (R* metric) for the evaluation of medical images

PURPOSE

The aim of the present work is to analyze the potential of the cross-correlation component of the multiscale structural similarity metric (R*) to predict human performance in detail detection tasks closely related with diagnostic x-ray images. To check the effectiveness of R*, the authors have initially applied this metric to a contrast detail detection task.

METHODS

Threshold contrast visibility using the R* metric was determined for two sets of images of a contrast-detail phantom (CDMAM). Results from R* and human observers were compared as far as the contrast threshold was concerned. A comparison between the R* metric and two algorithms currently used to evaluate CDMAM images was also performed.

RESULTS

Similar trends for the CDMAM detection task of human observers and R* were found in this study. Threshold contrast visibility values using R* are statistically indistinguishable from those obtained by human observers (F-test statistics: p > 0.05).

CONCLUSIONS

These results using R* show that it could be used to mimic human observers for certain tasks, such as the determination of contrast detail curves in the presence of uniform random noise backgrounds. The R* metric could also outperform other metrics and algorithms currently used to evaluate CDMAM images and can automate this evaluation task.

Computation of realistic virtual phantom images for an objective lesion detectability assessment in digital mammography

Image quality assessment is required for an optimal use of mammographic units. On the one hand, there are objective image quality assessment methods based on the measurement of technical parameters such as modulation transfer function (MTF), noise power spectrum (NPS) or detection quantum efficiency (DQE) describing performances of digital detectors. These parameters are, however, without direct relationship with lesion detectability in clinical practice. On the other hand, there are image quality assessment methods involving time consuming procedures, but presenting a direct relationship with lesion detectability. This contribution describes an X-ray source/digital detector model leading to the simulation of virtual contrast-detail phantom (CDMAM) images. The virtual image computation method requires the acquisition of only few real images and allows for an objective image quality assessment presenting a direct relationship with lesion detectability. The transfer function of the proposed model takes as input physical parameters (MTF* and noise) measured under clinical conditions on mammographic units. As presented in this contribution, MTF* is a modified MTF taking into account the effects due to X-ray scatter in the breast and magnification. Results obtained with the structural similarity index prove that the simulated images are quite realistic in terms of contrast and noise. Tests using contrast detail curves highlight the fact that the simulated and real images lead to very similar data quality in terms of lesion detectability. Finally, various statistical tests show that quality factors computed for both the simulated images and the real images are very close for the two data sets.

Estimation of channelized hotelling observer performance with known class means or known difference of class means

This paper concerns task-based image quality assessment for the task of discriminating between two classes of images. We address the problem of estimating two widely-used detection performance measures, SNR and AUC, from a finite number of images, assuming that the class discrimination is performed with a channelized Hotelling observer. In particular, we investigate the advantage that can be gained when either 1) the means of the signal-absent and signal-present classes are both known, or 2) when the difference of class means is known. For these two scenarios, we propose uniformly minimum variance unbiased estimators of SNR(2), derive the corresponding sampling distributions and provide variance expressions. In addition, we demonstrate how the bias and variance for the related AUC estimators may be calculated numerically by using the sampling distributions for the SNR(2) estimators. We find that for both SNR(2) and AUC, the new estimators have significantly lower bias and mean-square error than the traditional estimator, which assumes that the class means, and their difference, are unknown.

Does image quality matter? Impact of resolution and noise on mammographic task performance

The purpose of this study was to examine the effects of different resolution and noise levels on task performance in digital mammography. This study created an image set with images at three different resolution levels, corresponding to three digital display devices, and three different noise levels, with noise magnitudes similar to full clinical dose, half clinical dose, and quarter clinical dose. The images were read by five experienced breast imaging radiologists. The data were then analyzed to compute two accuracy statistics (overall classification accuracy and lesion detection accuracy) and performance at four diagnostic tasks (detection of microcalcifications, benign masses, malignant masses, and discrimination of benign and malignant masses). Human observer results showed decreasing display resolution had little effect on overall classification accuracy and individual diagnostic task performance, but increasing noise caused overall classification accuracy to decrease by a statistically significant 21% as the breast dose went to one quarter of its normal clinical value. The noise effects were most prominent for the tasks of microcalcification detection and mass discrimination. When the noise changed from full clinical dose to quarter clinical dose, the microcalcification detection performance fell from 89% to 67% and the mass discrimination performance decreased from 93% to 79%, while malignant mass detection performance remained relatively constant with values of 88% and 84%, respectively. As a secondary aim, the image set was also analyzed by two observer models to examine whether their performance was similar to humans. Observer models differed from human observers and each other in their sensitivity to resolution degradation and noise. The primary conclusions of this study suggest that quantum noise appears to be the dominant image quality factor in digital mammography, affecting radiologist performance much more profoundly than display resolution.

Assessment of a fluorescence-enhanced optical imaging system using the Hotelling observer

This study represents a first attempt to assess the detection capability of a fluorescence-enhanced optical imaging system as quantified by the Hotelling observer. The imaging system is simulated by the diffusion approximation of the time-dependent radiative transfer equation, which describes near infra-red (NIR) light propagation through clinically relevant tissue volumes. The random structures in the background are introduced using a lumpy-object model as a representation of anatomical structure as well as non-uniform distribution of disease markers. The systematic errors and noise associated with the actual experimental conditions are incorporated into the simulated boundary measurements to acquire imaging data sets. A large number of imaging data sets are considered in order to perform Hotelling observer studies. We find that the signal-to-noise ratio (SNR) of Hotelling observer (i) decreases as the strength of lumpy perturbations in the background increases, (ii) decreases as the target depth increases, and (iii) increases as excitation light leakage decreases, and reaches a maximum for filter optical density values of 5 or higher.