MRI quality control: six imagers studied using eleven unified image quality parameters

Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization

Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.

Determining the longitudinal accuracy and reproducibility of T1 and T2 in a 3T MRI scanner

Purpose

To determine baseline accuracy and reproducibility of T₁ and T₂ relaxation times over 12 months on a dedicated radiotherapy MRI scanner.

Methods

An International Society of Magnetic Resonance in Medicine/National Institute of Standards and Technology (ISMRM/NIST) System Phantom was scanned monthly on a 3T MRI scanner for 1 year. T₁ was measured using inversion recovery (T₁‐IR) and variable flip angle (T₁‐VFA) sequences and T₂ was measured using a multi‐echo spin echo (T₂‐SE) sequence. For each vial in the phantom, accuracy errors (%bias) were determined by the relative differences in measured T₁ and T₂ times compared to reference values. Reproducibility was measured by the coefficient of variation (CV) of T₁ and T₂ measurements across monthly scans. Accuracy and reproducibility were mainly assessed on vials with relaxation times expected to be in physiological ranges at 3T.

Results

A strong linear correlation between measured and reference relaxation times was found for all sequences tested (R² > 0.997). Baseline bias (and CV[%]) for T₁‐IR, T₁‐VFA and T₂‐SE sequences were +2.0% (2.1), +6.5% (4.2), and +8.5% (1.9), respectively.

Conclusions

The accuracy and reproducibility of T₁ and T₂ on the scanner were considered sufficient for the sequences tested. No longitudinal trends of variation were deduced, suggesting less frequent measurements are required following the establishment of baselines.

Image quality assessments according to the angle of tilt of a flex tilt coil supporting device: An ACR phantom study

In this study, we assessed how image quality depends on the angle of tilt of a flex tilt coil supporting device during an MRI examination. All measurements were performed with an American College of Radiology (ACR) MRI phantom using a flex tilt coil supporting device. All images were analyzed using an automatic assessment method following the ACR MRI accreditation guidance. Image quality was compared between acquisitions grouped according to the angle of tilt of the coil supporting device: group A (Flat mode), group B (10˚), and group C (18˚). All measured image qualities were within the ACR recommended criteria, regardless of the angle of tilt of the flex tilt coil supporting device. However, statistically significant differences between the three groups were found for slice thickness, position accuracy, image intensity uniformity, and SNR (P < 0.05, ANOVA). The flex tilt coil supporting device can provide sufficient image quality, passing the criteria of the ACR MRI guideline, despite differences in slice thickness, slice position accuracy, image intensity uniformity, and SNR according to the angle of tilt.

An Automated Method for Quality Control in MRI Systems: Methods and Considerations

Objective: The purpose of this study was to develop an automated method for performing quality control (QC) tests in magnetic resonance imaging (MRI) systems, investigate the effect of different definitions of QC parameters and its sensitivity with respect to variations in regions of interest (ROI) positioning, and validate the reliability of the automated method by comparison with results from manual evaluations. Materials and Methods: Magnetic Resonance imaging MRI used for acceptance and routine QC tests from five MRI systems were selected. All QC tests were performed using the American College of Radiology (ACR) MRI accreditation phantom. The only selection criterion was that in the same QC test, images from two identical sequential sequences should be available. The study was focused on four QC parameters: percent signal ghosting (PSG), percent image uniformity (PIU), signal-to-noise ratio (SNR), and SNR uniformity (SNRU), whose values are calculated using the mean signal and the standard deviation of ROIs defined within the phantom image or in the background. The variability of manual ROIs placement was emulated by the software using random variables that follow appropriate normal distributions. Results: Twenty-one paired sequences were employed. The automated test results for PIU were in good agreement with manual results. However, the PSG values were found to vary depending on the selection of ROIs with respect to the phantom. The values of SNR and SNRU also vary significantly, depending on the combination of the two out of the four standard rectangular ROIs. Furthermore, the methodology used for SNR and SNRU calculation also had significant effect on the results. Conclusions: The automated method standardizes the position of ROIs with respect to the ACR phantom image and allows for reproducible QC results.

LEGO-compatible modular mapping phantom for magnetic resonance imaging

Physical phantoms have been widely used for performance evaluation of magnetic resonance imaging (MRI). Although there are many kinds of physical phantoms, most MRI phantoms use fixed configurations with specific sizes that may fit one or a few different types of radio frequency (RF) coils. Therefore, it has limitations for various image quality assessments of scanning areas. In this article, we report a novel design for a truly customizable MRI phantom called the LEGO-compatible Modular Mapping (MOMA) phantom, which not only serves as a general quality assurance phantom for a wide range of RF coils, but also a flexible calibration phantom for quantitative imaging. The MOMA phantom has a modular architecture which includes individual assessment functionality of the modules and LEGO-type assembly compatibility. We demonstrated the feasibility of the MOMA phantom for quantitative evaluation of image quality using customized module assembly compatible with head, breast, spine, knee, and body coil features. This unique approach allows comprehensive image quality evaluation with wide versatility. In addition, we provide detailed MOMA phantom development and imaging characteristics of the modules.

Assessing effects of scanner upgrades for clinical studies

BACKGROUND

Scanner upgrades due to software and hardware changes are an inevitable part of MR research and, without quality assurance protocols, can jeopardize studies.

PURPOSE

To evaluate changes in T₁ relaxation time by inversion recovery (IR) and variable flip angle (VFA) measurements on a 3T system that underwent an "everything but the magnet" upgrade.

STUDY TYPE

Longitudinal.

PHANTOM

An International Society of Magnetic Resonance in Medicine / National Institute of Standards and Technology (ISMRM/NIST) system phantom with repeated measurements across multiple (n = 3) days.

FIELD STRENGTH/SEQUENCE

T₁ IR, VFA at 3T.

ASSESSMENT

The T₁ measurements by IR and VFA were compared with the nuclear magnetic resonance (NMR) measurements, which constitute the known values for the ISMRM/NIST system phantom, to determine the measurement error.

STATISTICAL TESTS

Descriptive.

RESULTS

The T₁ VFA measurement errors were distributed around zero (-15% to +10%) on the original system and then the errors were distributed entirely above zero post-upgrade (+5% to 30%). The T₁ IR results had a dramatic increase in error distribution (±5% original, ±20% post-upgrade) prior to the identification of signal saturation as an issue. Once the signal saturation was accounted for, the T₁ IR errors decreased to ±10% post-upgrade.

DATA CONCLUSION

The T₁ VFA measurement differences between the original and post-upgrade systems can be entirely attributed to contributions from B₁ . The T₁ IR measurements demonstrate the need for quantitative quality assurance (QA) processes. The site under study passed the QA protocols in place, which did not identify the increased T₁ error, nor the signal saturation issue. To improve on this study, we would make systematic, quantitative measurements at intervals less than a year and following any hardware or software upgrade.

LEVEL OF EVIDENCE

1 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2019. J. Magn. Reson. Imaging 2019;50:1948-1954.

A Simple Measure for Acuity in Medical Images

An automatic and objective assessment of image quality is important in an era, where large-scale processing of imaging data from multi-center studies becomes commonplace. Based on a comprehensive statistical image model that includes noise and blur, a measure for image acuity is derived here as the ratio of the maximal gradient magnitude and the intensity difference at a boundary. Acuity may be affected by the object under study, the image acquisition, reconstruction processes, and any post-processing steps. The acuity measure presented here is post-hoc, intuitive to understand, simple to compute, and easily integrates with other standard measures of image quality. Three applications in medical imaging are included where our acuity measure is useful in the objective and automatic assessment of image quality.

An Automatic Image Processing Workflow for Daily Magnetic Resonance Imaging Quality Assurance

The performance of magnetic resonance imaging (MRI) equipment is typically monitored with a quality assurance (QA) program. The QA program includes various tests performed at regular intervals. Users may execute specific tests, e.g., daily, weekly, or monthly. The exact interval of these measurements varies according to the department policies, machine setup and usage, manufacturer's recommendations, and available resources. In our experience, a single image acquired before the first patient of the day offers a low effort and effective system check. When this daily QA check is repeated with identical imaging parameters and phantom setup, the data can be used to derive various time series of the scanner performance. However, daily QA with manual processing can quickly become laborious in a multi-scanner environment. Fully automated image analysis and results output can positively impact the QA process by decreasing reaction time, improving repeatability, and by offering novel performance evaluation methods. In this study, we have developed a daily MRI QA workflow that can measure multiple scanner performance parameters with minimal manual labor required. The daily QA system is built around a phantom image taken by the radiographers at the beginning of day. The image is acquired with a consistent phantom setup and standardized imaging parameters. Recorded parameters are processed into graphs available to everyone involved in the MRI QA process via a web-based interface. The presented automatic MRI QA system provides an efficient tool for following the short- and long-term stability of MRI scanners.

Automated image quality evaluation of T2 -weighted liver MRI utilizing deep learning architecture

PURPOSE

To develop and test a deep learning approach named Convolutional Neural Network (CNN) for automated screening of T₂ -weighted (T₂ WI) liver acquisitions for nondiagnostic images, and compare this automated approach to evaluation by two radiologists.

MATERIALS AND METHODS

We evaluated 522 liver magnetic resonance imaging (MRI) exams performed at 1.5T and 3T at our institution between November 2014 and May 2016 for CNN training and validation. The CNN consisted of an input layer, convolutional layer, fully connected layer, and output layer. 351 T₂ WI were anonymized for training. Each case was annotated with a label of being diagnostic or nondiagnostic for detecting lesions and assessing liver morphology. Another independently collected 171 cases were sequestered for a blind test. These 171 T₂ WI were assessed independently by two radiologists and annotated as being diagnostic or nondiagnostic. These 171 T₂ WI were presented to the CNN algorithm and image quality (IQ) output of the algorithm was compared to that of two radiologists.

RESULTS

There was concordance in IQ label between Reader 1 and CNN in 79% of cases and between Reader 2 and CNN in 73%. The sensitivity and the specificity of the CNN algorithm in identifying nondiagnostic IQ was 67% and 81% with respect to Reader 1 and 47% and 80% with respect to Reader 2. The negative predictive value of the algorithm for identifying nondiagnostic IQ was 94% and 86% (relative to Readers 1 and 2).

CONCLUSION

We demonstrate a CNN algorithm that yields a high negative predictive value when screening for nondiagnostic T₂ WI of the liver.

LEVEL OF EVIDENCE

2 Technical Efficacy: Stage 2 J. Magn. Reson. Imaging 2018;47:723-728.

Ultrasound quality assurance and equipment governance

A case is presented to show the importance of good governance of ultrasound medical imaging equipment. Issues relating to the large numbers and diverse range of users and equipment are identified. Based on experience gained over 25 years, supporting upwards of 1000 systems, discussions consider why and how the testing of ultrasound systems should be approached by both the medical physics expert and end user. The management of the process is presented in the context of professional guidance and monitoring organisations' standards are considered to give a suggested best practice.

MR T1 and T2 relaxations in cysts and abscesses measured by 1.5 T MRI

OBJECTIVES

The main objective of this study was to make a comparison between the relaxation rates in jaw cysts and abscesses. Such a comparison should provide quantitative information for MR image analysis.

METHODS

A phantom containing 20 odontogenic jaw cysts and 11 jaw abscesses was imaged with 1.5 T MR. T(1) measurements were performed by using a mixed sequence of inversion recovery and spin echo, while T(2) measurements were carried out by the Carr-Purcell Meiboom-Gill (CPMG) sequence. Cystic fluids and abscesses were compared statistically.

RESULTS

In cysts and abscesses, respectively, the mean 1/T(1) was 0.9355 s(-1) and 0.8245 s(-1) and the mean 1/T(2) was 2.4575 s(-1) and 4.7073 s(-1). The 1/T(2) in cysts was very highly significantly different from that in abscesses (p = 0.0001). Both T(1) and T(2) were linearly proportional to material contents. T(2) relaxivities [26.458 ml (g s)(-1) for abscesses and 21.455 ml (g s)(-1) for cysts] were higher than T(1) relaxivities [5.4766 ml (g s)(-1) for abscesses and 10.075 ml (g s)(-1) for cysts].

DISCUSSION

Present T(2) measurements differentiate cysts from abscesses with a confidence interval of 95%. Because in vivo and in vitro image contrasts are changed by the same parameters, the T(2) findings should present valuable information for in vivo MRI. Hence the significant difference and the relaxivities may provide quantitative information for clinicians and researchers making image analyses.

CONCLUSION

T(2) may differentiate cysts from abscesses. The difference in T(2) is related to the material content of samples.

Imaging vascular function for early stage clinical trials using dynamic contrast-enhanced magnetic resonance imaging

Many therapeutic approaches to cancer affect the tumour vasculature, either indirectly or as a direct target. Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) has become an important means of investigating this action, both pre-clinically and in early stage clinical trials. For such trials, it is essential that the measurement process (i.e. image acquisition and analysis) can be performed effectively and with consistency among contributing centres. As the technique continues to develop in order to provide potential improvements in sensitivity and physiological relevance, there is considerable scope for between-centre variation in techniques. A workshop was convened by the Imaging Committee of the Experimental Cancer Medicine Centres (ECMC) to review the current status of DCE-MRI and to provide recommendations on how the technique can best be used for early stage trials. This review and the consequent recommendations are summarised here. Key Points • Tumour vascular function is key to tumour development and treatment • Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) can assess tumour vascular function • Thus DCE-MRI with pharmacokinetic models can assess novel treatments • Many recent developments are advancing the accuracy of and information from DCE-MRI • Establishing common methodology across multiple centres is challenging and requires accepted guidelines.

MRI quality assurance using the ACR phantom in a multi-unit imaging center

BACKGROUND

Magnetic resonance imaging (MRI) instrumentation is vulnerable to technical and image quality problems, and quality assurance is essential. In the studied regional imaging center the long-term quality assurance has been based on MagNET phantom measurements. American College of Radiology (ACR) has an accreditation program including a standardized image quality measurement protocol and phantom. The ACR protocol includes recommended acceptance criteria for clinical sequences and thus provides possibility to assess the clinical relevance of quality assurance. The purpose of this study was to test the ACR MRI phantom in quality assurance of a multi-unit imaging center.

MATERIAL AND METHODS

The imaging center operates 11 MRI systems of three major manufacturers with field strengths of 3.0 T, 1.5 T and 1.0 T. Images of the ACR phantom were acquired using a head coil following the ACR scanning instructions. Both ACR T1- and T2-weighted sequences as well as T1- and T2-weighted brain sequences in clinical use at each site were acquired. Measurements were performed twice. The images were analyzed and the results were compared with the ACR acceptance levels.

RESULTS

The acquisition procedure with the ACR phantom was faster than with the MagNET phantoms. On the first and second measurement rounds 91% and 73% of the systems passed the ACR test. Measured slice thickness accuracies were not within the acceptance limits in site T2 sequences. Differences in the high contrast spatial resolution between the ACR and the site sequences were observed. In 3.0 T systems the image intensity uniformity was slightly lower than the ACR acceptance limit.

CONCLUSION

The ACR method was feasible in quality assurance of a multi-unit imaging center and the ACR protocol could replace the MagNET phantom tests. An automatic analysis of the images will further improve cost-effectiveness and objectiveness of the ACR protocol.

Whole brain quantitative T2 MRI across multiple scanners with dual echo FSE: applications to AD, MCI, and normal aging

The ability to pool data from multiple MRI scanners is becoming increasingly important with the influx in multi-site research studies. Fast spin echo (FSE) dual spin echo sequences are often chosen for such studies based principally on their short acquisition time and the clinically useful contrasts they provide for assessing gross pathology. The practicality of measuring FSE-T2 relaxation properties has rarely been assessed. Here, FSE-T2 relaxation properties are examined across the three main scanner vendors (General Electric (GE), Philips, and Siemens). The American College of Radiology (ACR) phantom was scanned on four 1.5T platforms (two GE, one Philips, and one Siemens) to determine if the dual echo pulse sequence is susceptible to vendor-based variance. In addition, data from 85 subjects spanning the spectrum of normal aging, mild cognitive impairment (MCI), and Alzheimer's disease (AD) was obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) to affirm the presence of any phantom based between vendor variance and determine the relationship between this variance and disease. FSE-T2 relaxation properties, including peak FSE-T2 and histogram width, were calculated for each phantom and human subject. Direct correspondence was found between the phantom and human subject data. Peak FSE-T2 of Siemens scanners was consistently at least 20ms prolonged compared to GE and Philips. Siemens scanners showed broader FSE-T2 histograms than the other scanners. Greater variance was observed across GE scanners than either Philips or Siemens. FSE-T2 differences were much greater with scanner vendor than between diagnostic groups, as no significant changes in peak FSE-T2 or histogram width between normal aged, MCI, and AD subject groups were observed. These results indicate that whole brain histogram measures are not sensitive enough to detect FSE-T2 changes between normal aging, MCI, and AD and that FSE-T2 is highly variable across scanner vendors.

Automatic quality assessment in structural brain magnetic resonance imaging

MRI has evolved into an important diagnostic technique in medical imaging. However, reliability of the derived diagnosis can be degraded by artifacts, which challenge both radiologists and automatic computer-aided diagnosis. This work proposes a fully-automatic method for measuring image quality of three-dimensional (3D) structural MRI. Quality measures are derived by analyzing the air background of magnitude images and are capable of detecting image degradation from several sources, including bulk motion, residual magnetization from incomplete spoiling, blurring, and ghosting. The method has been validated on 749 3D T(1)-weighted 1.5T and 3T head scans acquired at 36 Alzheimer's Disease Neuroimaging Initiative (ADNI) study sites operating with various software and hardware combinations. Results are compared against qualitative grades assigned by the ADNI quality control center (taken as the reference standard). The derived quality indices are independent of the MRI system used and agree with the reference standard quality ratings with high sensitivity and specificity (>85%). The proposed procedures for quality assessment could be of great value for both research and routine clinical imaging. It could greatly improve workflow through its ability to rule out the need for a repeat scan while the patient is still in the magnet bore.

Measurement of MRI scanner performance with the ADNI phantom

The objectives of this study are as follows: to describe practical implementation challenges of multisite, multivendor quantitative studies; to describe the MRI phantom and analysis software used in the Alzheimer's Disease Neuroimaging Initiative (ADNI) study, illustrate the utility of the system for measuring scanner performance, the ability to assess gradient field nonlinearity corrections: and to recover human brain images without geometric scaling errors in multisite studies. ADNI is a large multicenter study with each center having its own copy of the phantom. The design of the phantom and analysis software are presented as results from predistribution systematics studies and results from field experience with the phantom at 58 enrolling ADNI sites over a 3 year period. The estimated coefficients of variation intrinsic to measurements of geometry in a single phantom are in the range of 3-5 parts in 10(4). Phantom measurements accurately detect linear and nonlinear scaling in images. Gradient unwarping methods are readily assessed by phantom nonlinearity measurements. Phantom-based scaling correction reduces observed geometric drift in human images by one-third or more. Repair or replacement of phantoms between scans, however, is a confounding factor. The ADNI phantom can be used to assess both scanner performance and the validity of postprocessing image corrections in order to reduce systematic errors in human images. Reduced measurement errors should decrease measurement bias and increase statistical power for measurements of rates of change in the brain structure in AD treatment trials. Perhaps the greatest practical value of incorporating ADNI phantom measurements in a multisite study is to identify scanner errors through central monitoring. This approach has resulted in identification of system errors including sites misidentification of their own gradient hardware and the disabling of autoshim, and a miscalibrated laser alignment light. If undetected, these errors would have contributed to imprecision in quantitative metrics at over 25% of all enrolling ADNI sites.

Automated quality control of brain MR images

PURPOSE

To present a novel fully automated method for assessing the quality of magnetic resonance imaging (MRI) data acquired in a clinical trials environment.

MATERIALS AND METHODS

This work was performed in the context of clinical trials for multiple sclerosis. Quality control (QC) procedures included were: (i) patient brain identity verification, (ii) alphanumeric parameter matching, (iii) signal-to-noise ratio estimation, (iv) gadolinium-enhancement verification, and (v) detection of ghosting due to head motion. Each QC procedure produces a quantitative measurement which is compared against an acceptance threshold that was determined based on receiver operating characteristic analysis of traditional manual and visual QC performed by trained experts.

RESULTS

The automated QC results have high sensitivity and specificity when compared with the visual QC.

CONCLUSION

Our automated objective QC procedure can replace many manual subjective procedures to provide increased data throughput while reducing reader variability.

Establishment and results of a magnetic resonance quality assurance program for the pediatric brain tumor consortium

RATIONALE AND OBJECTIVES

Magnetic resonance (MR) imaging is used to assess brain tumor response to therapies, and a MR quality assurance (QA) program is necessary for multicenter clinical trials employing imaging. This study was performed to determine overall variability of quantitative imaging metrics measured with the American College of Radiology (ACR) phantom among 11 sites participating in the Pediatric Brain Tumor Consortium (PBTC) Neuroimaging Center (NIC) MR QA program.

MATERIALS AND METHODS

An MR QA program was implemented among 11 participating PBTC sites and quarterly evaluations of scanner performance for seven imaging metrics defined by the ACR were sought and subject to statistical evaluation over a 4.5-year period. Overall compliance with the QA program, means, standard deviations, and coefficients of variation (CV) for the quantitative imaging metrics were evaluated.

RESULTS

Quantitative measures of the seven imaging metrics were generally within ACR recommended guidelines for all sites. Compliance improved as the study progressed. Intersite variabilities, as gauged by CV for slice thickness and geometric accuracy, imaging parameters that influence size or positioning measurements in tumor studies, were on the order of 10% and 1%, respectively.

CONCLUSIONS

Although challenging to establish, MR QA programs within the context of PBTC multisite clinical trials when based on the ACR MR phantom program can indicate sites performing below acceptable image quality levels and establish levels of precision through instrumental variabilities that are relevant to quantitative image analyses (eg, tumor volume changes).

Report on a multicenter fMRI quality assurance protocol

Temporal stability during an fMRI acquisition is very important because the blood oxygen level-dependent (BOLD) effects of interest are only a few percent in magnitude. Also, studies involving the collection of groups of subjects over time require stable scanner performance over days, weeks, months, and even years. We describe a protocol designed by one of the authors that has been tested for several years within the context of a large, multicenter collaborative fMRI research project (FIRST-BIRN). A full description of the phantom, the quality assurance (QA) protocol, and the several calculations used to measure performance is provided. The results obtained with this protocol at multiple sites over time are presented. These data can be used as benchmarks for other centers involved in fMRI research. Some issues with the various protocol measures are highlighted and discussed, and possible protocol improvements are also suggested. Overall, we expect that other fMRI centers will find this approach to QA useful and this report may facilitate developing a similar QA protocol locally. Based on the findings reported herein, the authors are convinced that monitoring QA in this way will improve the quality of fMRI data.