Non-rigid image registration: theory and practice

Sensitivity of a deep-learning-based breast cancer risk prediction model

Objective.When it comes to the implementation of deep-learning based breast cancer risk (BCR) prediction models in clinical settings, it is important to be aware that these models could be sensitive to various factors, especially those arising from the acquisition process. In this work, we investigated how sensitive the state-of-the-art BCR prediction model is to realistic image alterations that can occur as a result of different positioning during the acquisition process.Approach.5076 mammograms (1269 exams, 650 participants) from the Slovenian and Belgium (University Hospital Leuven) Breast Cancer Screening Programs were collected. The Original MIRAI model was used for 1-5 year BCR estimation. First, BCR was predicted for the original mammograms, which were not changed. Then, a series of different image alteration techniques was performed, such as swapping left and right breasts, removing tissue below the inframammary fold, translations, cropping, rotations, registration and pectoral muscle removal. In addition, a subset of 81 exams, where at least one of the mammograms had to be retaken due to inadequate image quality, served as an approximation of a test-retest experiment. Bland-Altman plots were used to determine prediction bias and 95% limits of agreement (LOA). Additionally, the mean absolute difference in BCR (Mean AD) was calculated. The impact on the overall discrimination performance was evaluated with the AUC.Results.Swapping left and right breasts had no impact on the predicted BCR. The removal of skin tissue below the inframammary fold had minimal impact on the predicted BCR (1-5 year LOA: [-0.02, 0.01]). The model was sensitive to translation, rotation, registration, and cropping, where LOAs of up to ±0.1 were observed. Partial pectoral muscle removal did not have a major impact on predicted BCR, while complete removal of pectoral muscle introduced substantial prediction bias and LOAs (1 year LOA: [-0.07, 0.04], 5 year LOA: [-0.06, 0.03]). The approximation of a real test-retest experiment resulted in LOAs similar to those of simulated image alterations. None of the alterations impacted the overall BCR discrimination performance; the initial 1 year AUC (0.90 [0.88, 0.92]) and 5 year AUC (0.77 [0.75, 0.80]) remained unchanged.Significance.While tested image alterations do not impact overall BCR discrimination performance, substantial changes in predicted 1-5 year BCR can occur on an individual basis.

The Impact of Navigation in Lumbar Spine Surgery: A Study of Historical Aspects, Current Techniques and Future Directions

Background/Objectives: We sought to improve accuracy while minimizing radiation hazards, improving surgical outcomes, and preventing potential complications. Despite the increasing popularity of these systems, a limited number of papers have been published addressing the historical evolution, detailing the areas of use, and discussing the advantages and disadvantages, of this increasingly popular system in lumbar spine surgery. Our objective was to offer readers a concise overview of navigation system history in lumbar spine surgeries, the techniques involved, the advantages and disadvantages, and suggestions for future enhancements to the system. Methods: A comprehensive review of the literature was conducted, focusing on the development and implementation of navigation systems in lumbar spine surgeries. Our sources include PubMed-indexed peer-reviewed journals, clinical trial data, and case studies involving technologies such as computer-assisted surgery (CAS), image-guided surgery (IGS), and robotic-assisted systems. Results: To develop more practical, effective, and accurate navigation techniques for spine surgery, consistent advancements have been made over the past four decades. This technological progress began in the late 20th century and has since encompassed image-guided surgery, intraoperative imaging, advanced navigation combined with robotic assistance, and artificial intelligence. These technological advancements have significantly improved the accuracy of implant placement, reducing the risk of misplacement and related complications. Navigation has also been found to be particularly useful in tumor resection and minimally invasive surgery (MIS), where conventional anatomic landmarks are lacking or, in the case of MIS, not visible. Additionally, these innovations have led to shorter operative times, decreased radiation exposure for patients and surgical teams, and lower rates of reoperation. As navigation technology continues to evolve, future innovations are anticipated to further enhance the capabilities and accessibility of these systems, ultimately leading to improved patient outcomes in lumbar spine surgery. Conclusions: The initial limited utilization of navigation system in spine surgery has further expanded to encompass almost all fields of lumbar spine surgeries. As the cost-effectiveness and number of trained surgeons improve, a wider use of the system will be ensured so that the navigation system will be an indispensable tool in lumbar spine surgery. However, continued research and development, along with training programs for surgeons, are essential to fully realize the potential of these technologies in clinical practice.

Motion correction of 3D dynamic contrast-enhanced ultrasound imaging without anatomical B-Mode images: Pilot evaluation in eight patients

BACKGROUND

Dynamic contrast-enhanced ultrasound (DCE-US) is highly susceptible to motion artifacts arising from patient movement, respiration, and operator handling and experience. Motion artifacts can be especially problematic in the context of perfusion quantification. In conventional 2D DCE-US, motion correction (MC) algorithms take advantage of accompanying side-by-side anatomical B-Mode images that contain time-stable features. However, current commercial models of 3D DCE-US do not provide side-by-side B-Mode images, which makes MC challenging.

PURPOSE

This work introduces a novel MC algorithm for 3D DCE-US and assesses its efficacy when handling clinical data sets.

METHODS

In brief, the algorithm uses a pyramidal approach whereby short temporal windows consisting of three consecutive frames are created to perform local registrations, which are then registered to a master reference derived from a weighted average of all frames. We applied the algorithm to imaging studies from eight patients with metastatic lesions in the liver and assessed improvements in original versus motion corrected 3D DCE-US cine using: (i) frame-to-frame volumetric overlap of segmented lesions, (ii) normalized correlation coefficient (NCC) between frames (similarity analysis), and (iii) sum of squared errors (SSE), root-mean-squared error (RMSE), and r-squared (R2) quality-of-fit from fitted time-intensity curves (TIC) extracted from a segmented lesion.

RESULTS

We noted improvements in frame-to-frame lesion overlap across all patients, from 68% ± 13% without correction to 83% ± 3% with MC (p = 0.023). Frame-to-frame similarity as assessed by NCC also improved on two different sets of time points from 0.694 ± 0.057 (original cine) to 0.862 ± 0.049 (corresponding MC cine) and 0.723 ± 0.066 to 0.886 ± 0.036 (p ≤ 0.001 for both). TIC analysis displayed a significant decrease in RMSE (p = 0.018) and a significant increase in R2 goodness-of-fit (p = 0.029) for the patient cohort.

CONCLUSIONS

Overall, results suggest decreases in 3D DCE-US motion after applying the proposed algorithm.

Quantifying error introduced by iterative closest point image registration

OBJECTIVES

The aim of this paper was to quantify the analysis error introduced by iterative closest point (ICP) image registration. We also investigated whether a subsequent subtraction process can reduce process error.

METHODS

We tested metrology and two 3D inspection software using calibration standards at 0.39 μm, and 2.64 μm and mathematically perfect defects (softgauges) at 2 and 20 μm, on free form surfaces of increasing complexity and area, both with and without registration. Errors were calculated in percentage relative to the size of the defect being measured. Data were analysed in GraphPad Prism 9, normal and two-way ANOVA with post-hoc Tukey's was applied. Significance was inferred at p < 0.05.

RESULTS

Using ICP registration introduced errors from 0 % to 15.63 % of the defect size depending on the surface complexity and size of the defect. Significant differences were observed in analysis measurements between metrology and 3D inspection software and within different 3D inspection software, however, one did not show clear superiority over another. Even in the absence of registration, defects at 0.39 μm, and 2.64 μm produced substantial measurement error (13.39-77.50 % of defect size) when using 3D inspection software. Adding an additional data subtraction process reduced registration error to negligible levels (<1 % independent of surface complexity or area).

CONCLUSIONS

Commercial 3D inspection software introduces error during direct measurements below 3 μm. When using an ICP registration, errors over 15 % of the defect size can be introduced regardless of the accuracy of adjacent registration surfaces. Analysis output between software are not consistently repeatable or comparable and do not utilise ISO standards. Subtracting the datasets and analysing the residual difference reduced error to negligible levels.

CLINICAL SIGNIFICANCE

This paper quantifies the significant errors and inconsistencies introduced during the registration process even when 3D datasets are true and precise. This may impact on research diagnostics and clinical performance. An additional data processing step of scan subtraction can reduce this error but increases computational complexity.

Intraprocedural assessment of ablation margins using computed tomography co-registration in hepatocellular carcinoma treatment with percutaneous ablation: IAMCOMPLETE study

PURPOSE

The primary objective of this study was to determine the feasibility of ablation margin quantification using a standardized scanning protocol during thermal ablation (TA) of hepatocellular carcinoma (HCC), and a rigid registration algorithm. Secondary objectives were to determine the inter- and intra-observer variability of tumor segmentation and quantification of the minimal ablation margin (MAM).

MATERIALS AND METHODS

Twenty patients who underwent thermal ablation for HCC were included. There were thirteen men and seven women with a mean age of 67.1 ± 10.8 (standard deviation [SD]) years (age range: 49.1-81.1 years). All patients underwent contrast-enhanced computed tomography examination under general anesthesia directly before and after TA, with preoxygenated breath hold. Contrast-enhanced computed tomography examinations were analyzed by radiologists using rigid registration software. Registration was deemed feasible when accurate rigid co-registration could be obtained. Inter- and intra-observer rates of tumor segmentation and MAM quantification were calculated. MAM values were correlated with local tumor progression (LTP) after one year of follow-up.

RESULTS

Co-registration of pre- and post-ablation images was feasible in 16 out of 20 patients (80%) and 26 out of 31 tumors (84%). Mean Dice similarity coefficient for inter- and intra-observer variability of tumor segmentation were 0.815 and 0.830, respectively. Mean MAM was 0.63 ± 3.589 (SD) mm (range: -6.26-6.65 mm). LTP occurred in four out of 20 patients (20%). The mean MAM value for patients who developed LTP was -4.00 mm, as compared to 0.727 mm for patients who did not develop LTP.

CONCLUSION

Ablation margin quantification is feasible using a standardized contrast-enhanced computed tomography protocol. Interpretation of MAM was hampered by the occurrence of tissue shrinkage during TA. Further validation in a larger cohort should lead to meaningful cut-off values for technical success of TA.

Automated marker-free longitudinal infrared breast image registration by GA-PSO

The automated marker-free longitudinal Infrared (IR) breast image registration overcomes several challenges like no anatomic fiducial markers on the body surface, blurry boundaries, heat pattern variation by environmental and physiological factors, nonrigid deformation, etc, has the ability of quantitative pixel-wise analysis with the heat energy and patterns change in a time course study. To achieve the goal, scale-invariant feature transform, Harris corner, and Hessian matrix were employed to generate the feature points as anatomic fiducial markers, and hybrid genetic algorithm and particle swarm optimization minimizing the matching errors was used to find the appropriate corresponding pairs between the 1st IR image and thenth IR image. Moreover, the mechanism of the IR spectrogram hardware system has a high level of reproducibility. The performance of the proposed longitudinal image registration system was evaluated by the simulated experiments and the clinical trial. In the simulated experiments, the mean difference of our system is 1.64 mm, which increases 57.58% accuracy than manual determination and makes a 17.4% improvement than the previous study. In the clinical trial, 80 patients were captured several times of IR breast images during chemotherapy. Most of them were well aligned in the spatiotemporal domain. In the few cases with evident heat pattern dissipation and spatial deviation, it still provided a reliable comparison of vascular variation. Therefore, the proposed system is accurate and robust, which could be considered as a reliable tool for longitudinal approaches to breast cancer diagnosis.

Improved visualization of free-running cardiac magnetic resonance by respiratory phase using principal component analysis

Rationale and objectives

To support cardiac MR acquisitions during breathing without ECG, we developed software to mitigate the effects of respiratory displacement of the heart. The algorithm resolves respiratory motions and cardiac cycles from DICOM files. The new software automatically detects heartbeats from expiration and inspiration to decrease apparent respiratory motion.

Materials and methods

Our software uses principal component analysis to resolve respiratory motions from cardiac cycles. It groups heartbeats from expiration and inspiration to decrease apparent respiratory motion. The respiratory motion correction was evaluated on short-axis views (acquired with compressed sensing) of 11 healthy subjects and 8 cardiac patients. Two expert radiologists, blinded to the processing, assessed the dynamic images in terms of blood-myocardial contrast, endocardial interface definition, and motion artifacts.

Results

The smallest correlation coefficients between end-systolic frames of the original dynamic scans averaged 0.79. After segregation of cardiac cycles by respiratory phase, the mean correlation coefficients between cardiac cycles were 0.94±0.03 at end-expiration and 0.90±0.08 at end-inspiration. The improvements in correlation coefficients were significant in paired t-tests for healthy subjects and heart patients at end-expiration. Clinical assessment preferred cardiac cycles during end-expiration, which maintained or enhanced scores in 90% of healthy subjects and 83% of the heart patients. Performance remained high with arrhythmia and irregular breathing present.

Conclusion

Heartbeats collected from end-expiration mitigate respiratory motion and are accessible by applying the new software to DICOM files from real-time CMR. Inspiratory heartbeats are also accessible for examination of arrhythmias or abnormalities at end-inspiration.

Motion-compensated T1 mapping in cardiovascular magnetic resonance imaging: a technical review

T 1 mapping is becoming a staple magnetic resonance imaging method for diagnosing myocardial diseases such as ischemic cardiomyopathy, hypertrophic cardiomyopathy, myocarditis, and more. Clinically, most T 1 mapping sequences acquire a single slice at a single cardiac phase across a 10 to 15-heartbeat breath-hold, with one to three slices acquired in total. This leaves opportunities for improving patient comfort and information density by acquiring data across multiple cardiac phases in free-running acquisitions and across multiple respiratory phases in free-breathing acquisitions. Scanning in the presence of cardiac and respiratory motion requires more complex motion characterization and compensation. Most clinical mapping sequences use 2D single-slice acquisitions; however newer techniques allow for motion-compensated reconstructions in three dimensions and beyond. To further address confounding factors and improve measurement accuracy, T 1 maps can be acquired jointly with other quantitative parameters such as T 2 , T 2 ∗ , fat fraction, and more. These multiparametric acquisitions allow for constrained reconstruction approaches that isolate contributions to T 1 from other motion and relaxation mechanisms. In this review, we examine the state of the literature in motion-corrected and motion-resolved T 1 mapping, with potential future directions for further technical development and clinical translation.

MICDIR: Multi-scale inverse-consistent deformable image registration using UNetMSS with self-constructing graph latent

Image registration is the process of bringing different images into a common coordinate system - a technique widely used in various applications of computer vision, such as remote sensing, image retrieval, and, most commonly, medical imaging. Deep learning based techniques have been applied successfully to tackle various complex medical image processing problems, including medical image registration. Over the years, several image registration techniques have been proposed using deep learning. Deformable image registration techniques such as Voxelmorph have been successful in capturing finer changes and providing smoother deformations. However, Voxelmorph, as well as ICNet and FIRE, do not explicitly encode global dependencies (i.e. the overall anatomical view of the supplied image) and, therefore, cannot track large deformations. In order to tackle the aforementioned problems, this paper extends the Voxelmorph approach in three different ways. To improve the performance in case of small as well as large deformations, supervision of the model at different resolutions has been integrated using a multi-scale UNet. To support the network to learn and encode the minute structural co-relations of the given image-pairs, a self-constructing graph network (SCGNet) has been used as the latent of the multi-scale UNet - which can improve the learning process of the model and help the model to generalise better. And finally, to make the deformations inverse-consistent, cycle consistency loss has been employed. On the task of registration of brain MRIs, the proposed method achieved significant improvements over ANTs and VoxelMorph, obtaining a Dice score of 0.8013 ± 0.0243 for intramodal and 0.6211 ± 0.0309 for intermodal, while VoxelMorph achieved 0.7747 ± 0.0260 and 0.6071 ± 0.0510, respectively.

Direct estimation of regional lung volume change from paired and single CT images using residual regression neural network

BACKGROUND

Chest computed tomography (CT) enables characterization of pulmonary diseases by producing high-resolution and high-contrast images of the intricate lung structures. Deformable image registration is used to align chest CT scans at different lung volumes, yielding estimates of local tissue expansion and contraction.

PURPOSE

We investigated the utility of deep generative models for directly predicting local tissue volume change from lung CT images, bypassing computationally expensive iterative image registration and providing a method that can be utilized in scenarios where either one or two CT scans are available.

METHODS

A residual regression convolutional neural network, called Reg3DNet+, is proposed for directly regressing high-resolution images of local tissue volume change (i.e., Jacobian) from CT images. Image registration was performed between lung volumes at total lung capacity (TLC) and functional residual capacity (FRC) using a tissue mass- and structure-preserving registration algorithm. The Jacobian image was calculated from the registration-derived displacement field and used as the ground truth for local tissue volume change. Four separate Reg3DNet+ models were trained to predict Jacobian images using a multifactorial study design to compare the effects of network input (i.e., single image vs. paired images) and output space (i.e., FRC vs. TLC). The models were trained and evaluated on image datasets from the COPDGene study. Models were evaluated against the registration-derived Jacobian images using local, regional, and global evaluation metrics.

RESULTS

Statistical analysis revealed that both factors - network input and output space - were significant determinants for change in evaluation metrics. Paired-input models performed better than single-input models, and model performance was better in the output space of FRC rather than TLC. Mean structural similarity index for paired-input models was 0.959 and 0.956 for FRC and TLC output spaces, respectively, and for single-input models was 0.951 and 0.937. Global evaluation metrics demonstrated correlation between registration-derived Jacobian mean and predicted Jacobian mean: coefficient of determination (r2 ) for paired-input models was 0.974 and 0.938 for FRC and TLC output spaces, respectively, and for single-input models was 0.598 and 0.346. After correcting for effort, registration-derived lobar volume change was strongly correlated with the predicted lobar volume change: for paired-input models r2 was 0.899 for both FRC and TLC output spaces, and for single-input models r2 was 0.803 and 0.862, respectively.

CONCLUSIONS

Convolutional neural networks can be used to directly predict local tissue mechanics, eliminating the need for computationally expensive image registration. Networks that use paired CT images acquired at TLC and FRC allow for more accurate prediction of local tissue expansion compared to networks that use a single image. Networks that only require a single input image still show promising results, particularly after correcting for effort, and allow for local tissue expansion estimation in cases where multiple CT scans are not available. For single-input networks, the FRC image is more predictive of local tissue volume change compared to the TLC image.

Virtual alignment of pathology image series for multi-gigapixel whole slide images

Interest in spatial omics is on the rise, but generation of highly multiplexed images remains challenging, due to cost, expertise, methodical constraints, and access to technology. An alternative approach is to register collections of whole slide images (WSI), generating spatially aligned datasets. WSI registration is a two-part problem, the first being the alignment itself and the second the application of transformations to huge multi-gigapixel images. To address both challenges, we developed Virtual Alignment of pathoLogy Image Series (VALIS), software which enables generation of highly multiplexed images by aligning any number of brightfield and/or immunofluorescent WSI, the results of which can be saved in the ome.tiff format. Benchmarking using publicly available datasets indicates VALIS provides state-of-the-art accuracy in WSI registration and 3D reconstruction. Leveraging existing open-source software tools, VALIS is written in Python, providing a free, fast, scalable, robust, and easy-to-use pipeline for registering multi-gigapixel WSI, facilitating downstream spatial analyses.

Non-Rigid Registration for High-Resolution Retinal Imaging

Adaptive optics provides improved resolution in ophthalmic imaging when retinal microstructures need to be identified, counted, and mapped. In general, multiple images are averaged to improve the signal-to-noise ratio or analyzed for temporal dynamics. Image registration by cross-correlation is straightforward for small patches; however, larger images require more sophisticated registration techniques. Strip-based registration has been used successfully for photoreceptor mosaic alignment in small patches; however, if the deformations along strips are not simple displacements, averaging can degrade the final image. We have applied a non-rigid registration technique that improves the quality of processed images for mapping cones over large image patches. In this approach, correction of local deformations compensates for local image stretching, compressing, bending, and twisting due to a number of causes. The main result of this procedure is improved definition of retinal microstructures that can be better identified and segmented. Derived metrics such as cone density, wall-to-lumen ratio, and quantification of structural modification of blood vessel walls have diagnostic value in many retinal diseases, including diabetic retinopathy and age-related macular degeneration, and their improved evaluations may facilitate early diagnostics of retinal diseases.

RHOMBUS DEFORMATION OF RETINAL LATERAL DISPLACEMENT AFTER EPIRETINAL MEMBRANE REMOVAL REVEALED BY DIFFEOMORPHIC IMAGE REGISTRATION

PURPOSE

To establish an analysis method using diffeomorphic image registration and evaluate microvascular displacement through epiretinal membrane (ERM) removal.

METHODS

Medical records of eyes that underwent vitreous surgery for ERM were reviewed. Postoperative optical coherence tomography angiography (OCTA) images were converted to the corresponding preoperative images according to a configured algorithm using diffeomorphism.

RESULTS

Thirty-seven eyes with ERM were examined. Measured changes in the foveal avascular zone (FAZ) area showed a significant negative correlation with central foveal thickness (CFT). The average amplitude of microvascular displacement calculated for each pixel was 69 ± 27 µ m in the nasal area, which was relatively smaller than that in other areas. The vector map, which included both the amplitude and the vector of microvasculature displacement, showed a unique vector flow pattern called the rhombus deformation sign in 17 eyes. Eyes with this deformation sign showed less surgery-induced changes in the FAZ area and CFT and a milder ERM stage than those without this sign.

CONCLUSION

The authors calculated and visualized microvascular displacement using diffeomorphism. The authors found a unique pattern (rhombus deformation) of retinal lateral displacement through ERM removal, which was significantly associated with the severity of ERM.

Evaluation of Different Registration Algorithms to Reduce Motion Artifacts in CT-Thermography (CTT)

Computed tomography (CT)-based Thermography (CTT) is currently being investigated as a non-invasive temperature monitoring method during ablation procedures. Since multiple CT scans with defined time intervals were acquired during this procedure, interscan motion artifacts can occur between the images, so registration is required. The aim of this study was to investigate different registration algorithms and their combinations for minimizing inter-scan motion artifacts during thermal ablation. Four CTT datasets were acquired using microwave ablation (MWA) of normal liver tissue performed in an in vivo porcine model. During each ablation, spectral CT volume scans were sequentially acquired. Based on initial reconstructions, rigid or elastic registration, or a combination of these, were carried out and rated by 15 radiologists. Friedman's test was used to compare rating results in reader assessments and revealed significant differences for the ablation probe movement rating only (p = 0.006; range, 5.3-6.6 points). Regarding this parameter, readers assessed rigid registration as inferior to other registrations. Quantitative analysis of ablation probe movement yielded a significantly decreased distance for combined registration as compared with unregistered data. In this study, registration was found to have the greatest influence on ablation probe movement, with connected registration being superior to only one registration process.

GradICON: Approximate Diffeomorphisms via Gradient Inverse Consistency

We present an approach to learning regular spatial transformations between image pairs in the context of medical image registration. Contrary to optimization-based registration techniques and many modern learning-based methods, we do not directly penalize transformation irregularities but instead promote transformation regularity via an inverse consistency penalty. We use a neural network to predict a map between a source and a target image as well as the map when swapping the source and target images. Different from existing approaches, we compose these two resulting maps and regularize deviations of the Jacobian of this composition from the identity matrix. This regularizer - GradICON - results in much better convergence when training registration models compared to promoting inverse consistency of the composition of maps directly while retaining the desirable implicit regularization effects of the latter. We achieve state-of-the-art registration performance on a variety of real-world medical image datasets using a single set of hyperparameters and a single non-dataset-specific training protocol. Code is available at https://github.com/uncbiag/ICON.

Applicability and usage of dose mapping/accumulation in radiotherapy

Dose mapping/accumulation (DMA) is a topic in radiotherapy (RT) for years, but has not yet found its widespread way into clinical RT routine. During the ESTRO Physics workshop 2021 on "commissioning and quality assurance of deformable image registration (DIR) for current and future RT applications", we built a working group on DMA from which we present the results of our discussions in this article. Our aim in this manuscript is to shed light on the current situation of DMA in RT and to highlight the issues that hinder consciously integrating it into clinical RT routine. As a first outcome of our discussions, we present a scheme where representative RT use cases are positioned, considering expected anatomical variations and the impact of dose mapping uncertainties on patient safety, which we have named the DMA landscape (DMAL). This tool is useful for future reference when DMA applications get closer to clinical day-to-day use. Secondly, we discussed current challenges, lightly touching on first-order effects (related to the impact of DIR uncertainties in dose mapping), and focusing in detail on second-order effects often dismissed in the current literature (as resampling and interpolation, quality assurance considerations, and radiobiological issues). Finally, we developed recommendations, and guidelines for vendors and users. Our main point include: Strive for context-driven DIR (by considering their impact on clinical decisions/judgements) rather than perfect DIR; be conscious of the limitations of the implemented DIR algorithm; and consider when dose mapping (with properly quantified uncertainties) is a better alternative than no mapping.

Detailed analysis of failure patterns using deformable image registration in hypopharyngeal cancer patients treated with sequential boost intensity-modulated radiotherapy

INTRODUCTION

Sequential boost intensity-modulated radiotherapy (SQB-IMRT) uses two different planning CTs (pCTs) and treatment plans. SQB-IMRT is a form of adaptive radiotherapy that allows for responses to changes in the shape of the tumour and organs at risk (OAR). On the other hand, dose accumulation with the two plans can be difficult to evaluate. The purpose of this study was to analyse patterns of loco-regional failure using deformable image registration (DIR) in hypopharyngeal cancer patients treated with SQB-IMRT.

METHODS

Between 2013 and 2019, 102 patients with hypopharyngeal cancer were treated with definitive SQB-IMRT at our institution. Dose accumulation with the 1st and 2nd plans was performed, and the dose to the loco-regional recurrent tumour volume was calculated using the DIR workflow. Failure was classified as follows: (i) in-field (≥95% of the recurrent tumour volume received 95% of the prescribed dose); (ii) marginal (20-95%); or (iii) out-of-field (<20%).

RESULTS

After a median follow-up period of 25 months, loco-regional failure occurred in 34 patients. Dose-volume histogram analysis showed that all loco-regional failures occurred in the field within 95% of the prescribed dose, with no marginal or out-of-field recurrences observed.

CONCLUSION

The dosimetric analysis using DIR showed that all loco-regional failures were within the high-dose region. More aggressive treatment may be required for gross tumours.

A method based on 3D affine alignment for the quantification of palatal expansion

INTRODUCTION

The current methodologies to quantify the palatal expansion are based on a preliminary rigid superimposition of 3D digital models representing the status of a given patient at different times. A new method based on affine alignment is proposed and compared to the gold standard, leading to the automatic analysis of 3-dimensional structural changes and to a simple numeric quantification of overall expansion vector and a better alignment of the digital models.

MATERIALS AND METHODS

40 digital models (timing span delta 25.8 ± 12.5 months) from young patients (mean age 10.7 ± 2.6) treated with two different palatal expansion techniques (20 subjects with RME-Rapid Maxillary Expander, and 20 subjects with NiTiSE, NiTi self-expander) were superimposed with the new affine alignment technique implemented as an extension package of the open-source MeshLab, from a golden standard starting point of rigid alignment. The results were then compared.

RESULTS

The new measurement function indicates a mean expansion expressed in a single numeric value of 9.3%, 10.3% for the RME group and 8.4% for the NiTiSE group respectively. The comparison with the golden standard showed a decrease to the average error from 0.91 mm to 0.58 mm.

CONCLUSIONS

Affine alignment improves the current perspective of structural change quantification in the specific group of growing patients treated with palatal expanders giving the clinician useful information on the 3-dimensional morphological changes.

Estimating medical image registration error and confidence: A taxonomy and scoping review

Given that image registration is a fundamental and ubiquitous task in both clinical and research domains of the medical field, errors in registration can have serious consequences. Since such errors can mislead clinicians during image-guided therapies or bias the results of a downstream analysis, methods to estimate registration error are becoming more popular. To give structure to this new heterogenous field we developed a taxonomy and performed a scoping review of methods that quantitatively and automatically provide a dense estimation of registration error. The taxonomy breaks down error estimation methods into Approach (Image- or Transformation-based), Framework (Machine Learning or Direct) and Measurement (error or confidence) components. Following the PRISMA guidelines for scoping reviews, the 570 records found were reduced to twenty studies that met inclusion criteria, which were then reviewed according to the proposed taxonomy. Trends in the field, advantages and disadvantages of the methods, and potential sources of bias are also discussed. We provide suggestions for best practices and identify areas of future research.

Joint Progressive and Coarse-to-Fine Registration of Brain MRI via Deformation Field Integration and Non-Rigid Feature Fusion

Registration of brain MRI images requires to solve a deformation field, which is extremely difficult in aligning intricate brain tissues, e.g., subcortical nuclei, etc. Existing efforts resort to decomposing the target deformation field into intermediate sub-fields with either tiny motions, i.e., progressive registration stage by stage, or lower resolutions, i.e., coarse-to-fine estimation of the full-size deformation field. In this paper, we argue that those efforts are not mutually exclusive, and propose a unified framework for robust brain MRI registration in both progressive and coarse-to-fine manners simultaneously. Specifically, building on a dual-encoder U-Net, the fixed-moving MRI pair is encoded and decoded into multi-scale sub-fields from coarse to fine. Each decoding block contains two proposed novel modules: i) in Deformation Field Integration (DFI), a single integrated deformation sub-field is calculated, warping by which is equivalent to warping progressively by sub-fields from all previous decoding blocks, and ii) in Non-rigid Feature Fusion (NFF), features of the fixed-moving pair are aligned by DFI-integrated deformation field, and then fused to predict a finer sub-field. Leveraging both DFI and NFF, the target deformation field is factorized into multi-scale sub-fields, where the coarser fields alleviate the estimate of a finer one and the finer field learns to make up those misalignments insolvable by previous coarser ones. The extensive and comprehensive experimental results on both private and two public datasets demonstrate a superior registration performance of brain MRI images over progressive registration only and coarse-to-fine estimation only, with an increase by at most 8% in the average Dice.

BAYESIAN FUNCTIONAL REGISTRATION OF FMRI ACTIVATION MAPS

Functional magnetic resonance imaging (fMRI) has provided invaluable insight into our understanding of human behavior. However, large inter-individual differences in both brain anatomy and functional localization after anatomical alignment remain a major limitation in conducting group analyses and performing population level inference. This paper addresses this problem by developing and validating a new computational technique for reducing misalignment across individuals in functional brain systems by spatially transforming each subjects functional data to a common reference map. Our proposed Bayesian functional registration approach allows us to assess differences in brain function across subjects and individual differences in activation topology. It combines intensity-based and feature-based information into an integrated framework, and allows inference to be performed on the transformation via the posterior samples. We evaluate the method in a simulation study and apply it to data from a study of thermal pain. We find that the proposed approach provides increased sensitivity for group-level inference.

A Novel Method for Observing Tumor Margin in Hepatoblastoma Based on Microstructure 3D Reconstruction

Objective: We investigated three-dimensional (3 D) reconstruction for the assessment of the tumor margin microstructure of hepatoblastoma (HB). Methods: Eleven surgical resections of childhood hepatoblastomas obtained between September 2018 and December 2019 were formalin-fixed, paraffin-embedded, serially sectioned at 4 μm, stained with hematoxylin and eosin (every 19th and 20th section stained with alpha-fetoprotein and glypican 3), and the digital images of all sections were acquired at 100× followed by image registration using the B-spline based method with modified residual complexity. Reconstruction was performed using 3 D Slicer software. Results: The reconstructed orthogonal 3 D images clearly presented the internal microstructure of the tumor margin. The rendered 3 D image could be rotated at any angle. Conclusions: Microstructure 3 D reconstruction is feasible for observing the pathological structure of the HB tumor margin.

Non-rigid registration of medical images based on S 2 1 ⁡ ( Δ mn ( 2 ) ) non-tensor product B-spline

In this study, a non-tensor product B-spline algorithm is applied to the search space of the registration process, and a new method of image non-rigid registration is proposed. The tensor product B-spline is a function defined in the two directions of x and y, while the non-tensor product B-splineS 2 1 ⁡ ( Δ mn ( 2 ) ) is defined in four directions on the 2-type triangulation. For certain problems, using non-tensor product B-splines to describe the non-rigid deformation of an image can more accurately extract the four-directional information of the image, thereby describing the global or local non-rigid deformation of the image in more directions. Indeed, it provides a method to solve the problem of image deformation in multiple directions. In addition, the region of interest of medical images is irregular, and usually no value exists on the boundary triangle. The value of the basis function of the non-tensor product B-spline on the boundary triangle is only 0. The algorithm process is optimized. The algorithm performs completely automatic non-rigid registration of computed tomography and magnetic resonance imaging images of patients. In particular, this study compares the performance of the proposed algorithm with the tensor product B-spline registration algorithm. The results elucidate that the proposed algorithm clearly improves the accuracy.

Comparison of an in-house hybrid DIR method to NiftyReg on CBCT and CT images for head and neck cancer

An in-house hybrid deformable image registration (DIR) method, which combines free-form deformation (FFD) and the viscous fluid registration method, is proposed. Its results on the planning computed tomography (CT) and the day 1 treatment cone-beam CT (CBCT) image from 68 head and neck cancer patients are compared with the results of NiftyReg, which uses B-spline FFD alone. Several similarity metrics, the target registration error (TRE) of annotated points, as well as the Dice similarity coefficient (DSC) and Hausdorff distance (HD) of the propagated organs at risk are employed to analyze their registration accuracy. According to quantitative analysis on mutual information, normalized cross-correlation, and the absolute pixel value differences, the results of the proposed DIR are more similar to the CBCT images than the NiftyReg results. Smaller TRE of the annotated points is observed in the proposed method, and the overall mean TRE for the proposed method and NiftyReg was 2.34 and 2.98 mm, respectively (p < 0.001). The mean DSC in the larynx, spinal cord, oral cavity, mandible, and parotid given by the proposed method ranged from 0.78 to 0.91, significantly higher than the NiftyReg results (ranging from 0.77 to 0.90), and the HD was significantly lower compared to NiftyReg. Furthermore, the proposed method did not suffer from unrealistic deformations as the NiftyReg did in the visual evaluation. Meanwhile, the execution time of the proposed method was much higher than NiftyReg (96.98 ± 11.88 s vs. 4.60 ± 0.49 s). In conclusion, the in-house hybrid method gave better accuracy and more stable performance than NiftyReg.

A guide for single-particle chromatin tracking in live cell nuclei

The emergence of labeling strategies and live cell imaging methods enables the imaging of chromatin in living cells at single digit nanometer resolution as well as milliseconds temporal resolution. These technical breakthroughs revolutionize our understanding of chromatin structure, dynamics and functions. Single molecule tracking algorithms are usually preferred to quantify the movement of these intranucleus elements to interpret the spatiotemporal evolution of the chromatin. In this review, we will first summarize the fluorescent labeling strategy of chromatin in live cells which will be followed by a sys-tematic comparison of live cell imaging instrumentation. With the proper microscope, we will discuss the image analysis pipelines to extract the biophysical properties of the chromatin. Finally, we expect to give practical suggestions to broad biologists on how to select methods and link to the model properly according to different investigation pur-poses. This article is protected by copyright. All rights reserved.

Quantitative Pulmonary Neutrophil Dynamics Using Computer-Vision Stabilized Intravital Imaging

In vivo intravital imaging in animal models in the lung remains challenging owing to respiratory motion artifacts. Here we describe a novel intravital imaging approach based on the computer-vision stabilization algorithm, Computer-Vision Stabilized Intravital Imaging. This method corrects lung movements and deformations at submicron precision in respiring mouse lungs. The precision enables high-throughput quantitative analysis of intravital pulmonary polymorphonuclear neutrophil (PMN) dynamics in lungs. We quantified real-time PMN patrolling dynamics of microvessels in the basal state and PMN recruitment resulting from sequestration in a model of endotoxemia in mice. We focused on determining the marginated pool of PMNs in the lung. Direct visualization of marginated PMNs revealed that they are not static but highly dynamic and undergo repeated cycles of "catch and release." PMNs briefly arrest in larger diameter capillary junction (∼10 μm) and then squeeze into narrower, approximately 5-μm diameter vessels through PMN deformation. We also observed that the sequestered PMNs in lung microvessels lost their migratory capabilities in association with cell morphological change following prolonged endotoxemia. These observations underscore the value of direct visualization and quantitative analysis of PMN dynamics in lungs to study PMN physiology and pathophysiology and role in inflammatory lung injury.

Few-shot learning for deformable image registration in 4DCT images

OBJECTIVES

To develop a rapid and accurate 4D deformable image registration (DIR) approach for online adaptive radiotherapy.

METHODS

We propose a deep learning (DL)-based few-shot registration network (FR-Net) to generate deformation vector fields from each respiratory phase to an implicit reference image, thereby mitigating the bias introduced by the selection of reference images. The proposed FR-Net is pretrained with limited unlabeled 4D data and further optimized by maximizing the intensity similarity of one specific four-dimensional computed tomography (4DCT) scan. Because of the learning ability of DL models, the few-shot learning strategy facilitates the generalization of the model to other 4D data sets and the acceleration of the optimization process.

RESULTS

The proposed FR-Net is evaluated for 4D groupwise and 3D pairwise registration on thoracic 4DCT data sets DIR-Lab and POPI. FR-Net displays an averaged target registration error of 1.48 mm and 1.16 mm between the maximum inhalation and exhalation phases in the 4DCT of DIR-Lab and POPI, respectively, with approximately 2 min required to optimize one 4DCT. Overall, FR-Net outperforms state-of-the-art methods in terms of registration accuracy and exhibits a low computational time.

CONCLUSION

We develop a few-shot groupwise DIR algorithm for 4DCT images. The promising registration performance and computational efficiency demonstrate the prospective applications of this approach in registration tasks for online adaptive radiotherapy.

ADVANCES IN KNOWLEDGE

This work exploits DL models to solve the optimization problem in registering 4DCT scans while combining groupwise registration and few-shot learning strategy to solve the problem of consuming computational time and inferior registration accuracy.

Quantitative assessment of intra- and inter-modality deformable image registration of the heart, left ventricle, and thoracic aorta on longitudinal 4D-CT and MR images

Purpose

Magnetic resonance imaging (MRI)‐based investigations into radiotherapy (RT)‐induced cardiotoxicity require reliable registrations of magnetic resonance (MR) imaging to planning computed tomography (CT) for correlation to regional dose. In this study, the accuracy of intra‐ and inter‐modality deformable image registration (DIR) of longitudinal four‐dimensional CT (4D‐CT) and MR images were evaluated for heart, left ventricle (LV), and thoracic aorta (TA).

Methods and materials

Non‐cardiac‐gated 4D‐CT and T1 volumetric interpolated breath‐hold examination (T1‐VIBE) MRI datasets from five lung cancer patients were obtained at two breathing phases (inspiration/expiration) and two time points (before treatment and 5 weeks after initiating RT). Heart, LV, and TA were manually contoured. Each organ underwent three intramodal DIRs ((A) CT modality over time, (B) MR modality over time, and (C) MR contrast effect at the same time) and two intermodal DIRs ((D) CT/MR multimodality at same time and (E) CT/MR multimodality over time). Hausdorff distance (HD), mean distance to agreement (MDA), and Dice were evaluated and assessed for compliance with American Association of Physicists in Medicine (AAPM) Task Group (TG)‐132 recommendations.

Results

Mean values of HD, MDA, and Dice under all registration scenarios for each region of interest ranged between 8.7 and 16.8 mm, 1.0 and 2.6 mm, and 0.85 and 0.95, respectively, and were within the TG‐132 recommended range (MDA < 3 mm, Dice > 0.8). Intramodal DIR showed slightly better results compared to intermodal DIR. Heart and TA demonstrated higher registration accuracy compared to LV for all scenarios except for HD and Dice values in Group A. Significant differences for each metric and tissue of interest were noted between Groups B and D and between Groups B and E. MDA and Dice significantly differed between LV and heart in all registrations except for MDA in Group E.

Conclusions

DIR of the heart, LV, and TA between non‐cardiac‐gated longitudinal 4D‐CT and MRI across two modalities, breathing phases, and pre/post‐contrast is acceptably accurate per AAPM TG‐132 guidelines. This study paves the way for future evaluation of RT‐induced cardiotoxicity and its related factors using multimodality DIR.

Construction of 3D model of knee joint motion based on MRI image registration

There is a growing demand for information and computational technology for surgeons help with surgical planning as well as prosthetics design. The two-dimensional images are registered to the three-dimensional (3D) model for high efficiency. To reconstruct the 3D model of knee joint including bone structure and main soft tissue structure, the evaluation and analysis of sports injury and rehabilitation treatment are detailed in this study. Mimics 10.0 was used to reconstruct the bone structure, ligament, and meniscus according to the pulse diffusion-weighted imaging sequence (PDWI) and stir sequences of magnetic resonance imaging (MRI). Excluding congenital malformations and diseases of the skeletal muscle system, MRI scanning was performed on bilateral knee joints. Proton weighted sequence (PDWI sequence) and stir pulse sequence were selected for MRI. The models were imported into Geomagic Studio 11 software for refinement and modification, and 3D registration of bone structure and main soft tissue structure was performed to construct a digital model of knee joint bone structure and accessory cartilage and ligament structure. The 3D knee joint model including bone, meniscus, and collateral ligament was established. Reconstruction and image registration based on mimics and Geomagic Studio can build a 3D model of knee joint with satisfactory morphology, which can meet the requirements of teaching, motion simulation, and biomechanical analysis.

Fully automated quality control of rigid and affine registrations of T1w and T2w MRI in big data using machine learning

BACKGROUND

Magnetic resonance imaging (MRI)-based morphometry and relaxometry are proven methods for the structural assessment of the human brain in several neurological disorders. These procedures are generally based on T1-weighted (T1w) and/or T2-weighted (T2w) MRI scans, and rigid and affine registrations to a standard template(s) are essential steps in such studies. Therefore, a fully automatic quality control (QC) of these registrations is necessary in big data scenarios to ensure that they are suitable for subsequent processing.

METHOD

A supervised machine learning (ML) framework is proposed by computing similarity metrics such as normalized cross-correlation, normalized mutual information, and correlation ratio locally. We have used these as candidate features for cross-validation and testing of different ML classifiers. For 5-fold repeated stratified grid search cross-validation, 400 correctly aligned, 2000 randomly generated misaligned images were used from the human connectome project young adult (HCP-YA) dataset. To test the cross-validated models, the datasets from autism brain imaging data exchange (ABIDE I) and information eXtraction from images (IXI) were used.

RESULTS

The ensemble classifiers, random forest, and AdaBoost yielded best performance with F1-scores, balanced accuracies, and Matthews correlation coefficients in the range of 0.95-1.00 during cross-validation. The predictive accuracies reached 0.99 on the Test set #1 (ABIDE I), 0.99 without and 0.96 with noise on Test set #2 (IXI, stratified w.r.t scanner vendor and field strength).

CONCLUSIONS

The cross-validated and tested ML models could be used for QC of both T1w and T2w rigid and affine registrations in large-scale MRI studies.

Adaptive Physics-Based Non-Rigid Registration for Immersive Image-Guided Neuronavigation Systems

Objective: In image-guided neurosurgery, co-registered preoperative anatomical, functional, and diffusion tensor imaging can be used to facilitate a safe resection of brain tumors in eloquent areas of the brain. However, the brain deforms during surgery, particularly in the presence of tumor resection. Non-Rigid Registration (NRR) of the preoperative image data can be used to create a registered image that captures the deformation in the intraoperative image while maintaining the quality of the preoperative image. Using clinical data, this paper reports the results of a comparison of the accuracy and performance among several non-rigid registration methods for handling brain deformation. A new adaptive method that automatically removes mesh elements in the area of the resected tumor, thereby handling deformation in the presence of resection is presented. To improve the user experience, we also present a new way of using mixed reality with ultrasound, MRI, and CT.

Materials and methods: This study focuses on 30 glioma surgeries performed at two different hospitals, many of which involved the resection of significant tumor volumes. An Adaptive Physics-Based Non-Rigid Registration method (A-PBNRR) registers preoperative and intraoperative MRI for each patient. The results are compared with three other readily available registration methods: a rigid registration implemented in 3D Slicer v4.4.0; a B-Spline non-rigid registration implemented in 3D Slicer v4.4.0; and PBNRR implemented in ITKv4.7.0, upon which A-PBNRR was based. Three measures were employed to facilitate a comprehensive evaluation of the registration accuracy: (i) visual assessment, (ii) a Hausdorff Distance-based metric, and (iii) a landmark-based approach using anatomical points identified by a neurosurgeon.

Results: The A-PBNRR using multi-tissue mesh adaptation improved the accuracy of deformable registration by more than five times compared to rigid and traditional physics based non-rigid registration, and four times compared to B-Spline interpolation methods which are part of ITK and 3D Slicer. Performance analysis showed that A-PBNRR could be applied, on average, in <2 min, achieving desirable speed for use in a clinical setting.

Conclusions: The A-PBNRR method performed significantly better than other readily available registration methods at modeling deformation in the presence of resection. Both the registration accuracy and performance proved sufficient to be of clinical value in the operating room. A-PBNRR, coupled with the mixed reality system, presents a powerful and affordable solution compared to current neuronavigation systems.

Deformable image registration to assist clinical decision for radiotherapy treatment adaptation for head and neck cancer patients

Head and neck (H&N) cancer patients often present anatomical and geometrical changes in tumors and organs at risk (OARs) during radiotherapy treatment. These changes may result in the need to adapt the existing treatment planning, using an expert's subjective opinion, for offline adaptive radiotherapy and a new treatment planning before each treatment, for online adaptive radiotherapy. In the present study, a fast methodology is proposed to assist in planning adaptation clinical decision using tumor and parotid glands percentage volume changes during treatment. The proposed approach was applied to 40 Η&Ν cases, with one planning Computed Tomography (pCT) image and CBCT scans for 6 weeks of treatment per case. Deformable registration was used for each patient's pCT image alignment to its weekly CBCT. The calculated transformations were used to align each patient's anatomical structures to the weekly anatomy. Clinical target volume (CTV) and parotid gland volume percentage changes were calculated in each case. The accuracy of the achieved image alignment was validated qualitatively and quantitatively. Furthermore, statistical analysis was performed to test if there is a statistically significant correlation between CTV and parotid glands volume percentage changes. Average MDA for CTV and parotid glands between corresponding structures defined by an expert in CBCTs and automatically calculated through registration was 1.4 ± 0.1 mm and 1.5 ± 0.1 mm, respectively. The mean registration time of the first CBCT image registration for 40 cases was lower than 3.4 min. Five patients show more than 20% tumor volume change. Six patients show more than 30% parotid glands volume change. Ten out of 40 patients proposed for planning adaptation. All the statistical tests performed showed no correlation between CTV/parotid glands percentage volume changes. The aim to assist in clinical decision making on a fast and automatic way was achieved using the proposed methodology, thereby reducing workload in clinical practice.

Deep reinforcement learning in medical imaging: A literature review

Deep reinforcement learning (DRL) augments the reinforcement learning framework, which learns a sequence of actions that maximizes the expected reward, with the representative power of deep neural networks. Recent works have demonstrated the great potential of DRL in medicine and healthcare. This paper presents a literature review of DRL in medical imaging. We start with a comprehensive tutorial of DRL, including the latest model-free and model-based algorithms. We then cover existing DRL applications for medical imaging, which are roughly divided into three main categories: (i) parametric medical image analysis tasks including landmark detection, object/lesion detection, registration, and view plane localization; (ii) solving optimization tasks including hyperparameter tuning, selecting augmentation strategies, and neural architecture search; and (iii) miscellaneous applications including surgical gesture segmentation, personalized mobile health intervention, and computational model personalization. The paper concludes with discussions of future perspectives.

Automatic Registration of the Mass Spectrometry Imaging Data of Sagittal Brain Slices to the Reference Atlas

The registration of the mass spectrometry imaging (MSI) data with mouse brain tissue slices from the atlases could perform automatic anatomical interpretation, and the comparison of MSI data in particular brain regions from different mice could be accelerated. However, the current registration of MSI data with mouse brain tissue slices is mainly focused on the coronal. Although the sagittal plane is able to provide more information about brain regions on a single histological slice than the coronal, it is difficult to directly register the complete sagittal brain slices of a mouse as a result of the more significant individualized differences and more positional shifts of brain regions. Herein, by adding the auxiliary line on the two brain regions of central canal (CC) and cerebral peduncle (CP), the registration accuracy of the MSI data with sagittal brain slices has been improved (∼2-5-folds for different brain regions). Moreover, the histological sections with different degrees deformation and different dyeing effects have been used to verify that this pipeline has a certain universality. Our method facilitates the rapid comparison of sagittal plane MSI data from different animals and accelerates the application in the discovery of disease markers.

Correlation of Automated Computed Tomography Volumetric Analysis Metrics With Motility Disturbances in Thyroid Eye Disease

PURPOSE

The authors sought to examine relationships between CT metrics derived via an automated method and clinical parameters of extraocular muscle changes in thyroid eye disease (TED).

METHODS

CT images of 204 orbits in the setting of TED were analyzed with an automated segmentation tool developed at the institution. Labels were applied to orbital structures of interest on the study images, which were then registered against a previously established atlas of manually indexed orbits derived from 35 healthy individuals. Point-wise correspondences between study and atlas images were then compared via a fusion algorithm to highlight metrics of interest where TED orbits differed from healthy orbits.

RESULTS

Univariate analysis demonstrated several correlations between CT metrics and clinical data. Metrics pertaining to the extraocular muscles-including average diameter, maximum diameter, and muscle volume-were strongly correlated (p < 0.05) with the presence of ocular motility deficits with regards to the superior, inferior, and lateral recti (with exception of superior rectus motility deficits being mildly correlated with muscle volume [p = 0.09]). Motility defects of the medial rectus were strongly correlated with muscle volume, and only weakly correlated with average and maximum muscle diameter.

CONCLUSIONS

The novel method of automated imaging metrics may provide objective, rapid clinical information which may have utility in prevention and recognition of visual impairments in TED before they reach an advanced or irreversible stage and while they are able to be improved with immunomodulatory treatments.

High-resolution three-dimensional visualization of hepatic sinusoids in cirrhotic rats via serial histological sections

AIM

As a specialized intraparenchymal vascular conduit, hepatic sinusoids play a key role in liver microcirculation. This study aimed to explore the three-dimensional (3D) morphological changes of cirrhotic sinusoids by serial histological sections.

METHODS

Cirrhosis was induced by tail vein injection of albumin in Wistar rats with a positive antibody. A total of 356 serial histological sections were prepared from liver tissue blocks of normal and cirrhotic rats. The optical microscope images were registered and reconstructed, and 3D reconstructions of the fine structures of fibrous tissues and sinusoids were subsequently visualized.

RESULTS

The fibrosis area of the cirrhotic sample was 6-16 times that of the normal sample (P<0.001). Cirrhosis led to obvious changes in the distribution and morphology of sinusoids, which were mainly manifested as dilation, increased quantity and disordered distribution. Compared with normal liver, cirrhotic liver has a significantly increased volume ratio, number and volume of sinusoids (1.63-, 0.53-, and 1.75-fold, respectively, P<0.001). Furthermore, the samples were further divided into three zones according to the oxygen supply, and there were significant differences in the morphology of the sinusoids in the normal and cirrhotic samples (P<0.05). In particular, morphological parameters of the cirrhotic sinusoids near the portal area were obviously greater than those in the normal liver (P<0.05).

CONCLUSION

3D morphological structures of hepatic sinusoids were reconstructed, and the adaptive microstructure changes of cirrhotic sinusoids were accurately measured, which has an important implications for the study of hepatic microcirculation and pathological changes of cirrhosis.

Impact of the method chosen for the analysis of recurrences after radiotherapy for head and neck cancers: volume-based, point-based and combined methods

Intensity modulated radiation therapy for head and neck is a complex technique. Inappropriate delineation and/or dose distribution can lead to recurrences. Analysis of these recurrences should lead to improve clinical practice. For several years, different methods of analysis have been described. The purpose of this review is to describe these different methods and to discuss their advantages and limitations. The first published methods used a volume-based approach studying the entire volume of recurrence according to initial target volumes, or dose distribution. The main limitation of these methods was that the volume of recurrence studied was dependent on the delay in diagnosis of that recurrence. Subsequently, other methods used point-based approaches, conceptualizing recurrence either as a spherical expansion from a core of radioresistant cells (center of mass of recurrence volume) or using a more clinical approach, taking into account tumor expansion pathways. More recently, more precise combined methods have been described, combining the different approaches. The choice of method is decisive for conclusions on the origin of recurrence.

Role of Fusion Imaging in Image-Guided Thermal Ablations

Thermal ablation (TA) procedures are effective treatments for several kinds of cancers. In the recent years, several medical imaging advancements have improved the use of image-guided TA. Imaging technique plays a pivotal role in improving the ablation success, maximizing pre-procedure planning efficacy, intraprocedural targeting, post-procedure monitoring and assessing the achieved result. Fusion imaging (FI) techniques allow for information integration of different imaging modalities, improving all the ablation procedure steps. FI concedes exploitation of all imaging modalities’ strengths concurrently, eliminating or minimizing every single modality’s weaknesses. Our work aims to give an overview of FI, explain and analyze FI technical aspects and its clinical applications in ablation therapy and interventional oncology.

Fast GPU 3D diffeomorphic image registration

3D image registration is one of the most fundamental and computationally expensive operations in medical image analysis. Here, we present a mixed-precision, Gauss-Newton-Krylov solver for diffeomorphic registration of two images. Our work extends the publicly available CLAIRE library to GPU architectures. Despite the importance of image registration, only a few implementations of large deformation diffeomorphic registration packages support GPUs. Our contributions are new algorithms to significantly reduce the run time of the two main computational kernels in CLAIRE: calculation of derivatives and scattered-data interpolation. We deploy (i) highly-optimized, mixed-precision GPU-kernels for the evaluation of scattered-data interpolation, (ii) replace Fast-Fourier-Transform (FFT)-based first-order derivatives with optimized 8th-order finite differences, and (iii) compare with state-of-the-art CPU and GPU implementations. As a highlight, we demonstrate that we can register 2563 clinical images in less than 6 seconds on a single NVIDIA Tesla V100. This amounts to over 20× speed-up over the current version of CLAIRE and over 30× speed-up over existing GPU implementations.

miRID: Multi-Modal Image Registration Using Modality-Independent and Rotation-Invariant Descriptor

Axiomatically, symmetry is a fundamental property of mathematical functions defining similarity measures, where similarity measures are important tools in many areas of computer science, including machine learning and image processing. In this paper, we investigate a new technique to measure the similarity between two images, a fixed image and a moving image, in multi-modal image registration (MIR). MIR in medical image processing is essential and useful in diagnosis and therapy guidance, but still a very challenging task due to the lack of robustness against the rotational variance in the image transformation process. Our investigation leads to a novel, local self-similarity descriptor, called the modality-independent and rotation-invariant descriptor (miRID). By relying on the mean of the intensity values, an miRID is simply computable and can effectively handle the complicated intensity relationship between multi-modal images. Moreover, it can also overcome the problem of rotational variance by sorting the numerical values, each of which is the absolute difference between each pixel’s intensity and the mean of all pixel intensities within a patch of the image. The experimental result shows that our method outperforms others in both multi-modal rigid and non-rigid image registrations.

Registration quality filtering improves robustness of voxel-wise analyses to the choice of brain template

Motivation:

Many clinical and scientific conclusions that rely on voxel-wise analyses of neuroimaging depend on the accurate comparison of corresponding anatomical regions. Such comparisons are made possible by registration of the images of subjects of interest onto a common brain template, such as the Johns Hopkins University (JHU) template. However, current image registration algorithms are prone to errors that are distributed in a template-dependent manner. Therefore, the results of voxel-wise analyses can be sensitive to template choice. Despite this problem, the issue of appropriate template choice for voxel-wise analyses is not generally addressed in contemporary neuroimaging studies, which may lead to the reporting of spurious results.

Results:

We present a novel approach to determine the suitability of a brain template for voxel-wise analysis. The approach is based on computing a “distance” between automatically-generated atlases of the subjects of interest and templates that is indicative of the extent of subject-to-template registration errors. This allows for the filtering of subjects and candidate templates based on a quantitative measure of registration quality. We benchmark our approach by evaluating alternative templates for a voxel-wise analysis that reproduces the well-known decline in fractional anisotropy (FA) with age. Our results show that filtering registrations minimizes errors and decreases the sensitivity of voxel-wise analysis to template choice. In addition to carrying important implications for future neuroimaging studies, the developed framework of template induction can be used to evaluate robustness of data analysis methods to template choice.

Rethinking medical image reconstruction via shape prior, going deeper and faster: Deep joint indirect registration and reconstruction

Indirect image registration is a promising technique to improve image reconstruction quality by providing a shape prior for the reconstruction task. In this paper, we propose a novel hybrid method that seeks to reconstruct high quality images from few measurements whilst requiring low computational cost. With this purpose, our framework intertwines indirect registration and reconstruction tasks is a single functional. It is based on two major novelties. Firstly, we introduce a model based on deep nets to solve the indirect registration problem, in which the inversion and registration mappings are recurrently connected through a fixed-point interaction based sparse optimisation. Secondly, we introduce specific inversion blocks, that use the explicit physical forward operator, to map the acquired measurements to the image reconstruction. We also introduce registration blocks based deep nets to predict the registration parameters and warp transformation accurately and efficiently. We demonstrate, through extensive numerical and visual experiments, that our framework outperforms significantly classic reconstruction schemes and other bi-task method; this in terms of both image quality and computational time. Finally, we show generalisation capabilities of our approach by demonstrating their performance on fast Magnetic Resonance Imaging (MRI), sparse view computed tomography (CT) and low dose CT with measurements much below the Nyquist limit.

Esophagus segmentation from planning CT images using an atlas-based deep learning approach

BACKGROUND AND OBJECTIVE

One of the main steps in the planning of radiotherapy (RT) is the segmentation of organs at risk (OARs) in Computed Tomography (CT). The esophagus is one of the most difficult OARs to segment. The boundaries between the esophagus and other surrounding tissues are not well-defined, and it is presented in several slices of the CT. Thus, manually segment the esophagus requires a lot of experience and takes time. This difficulty in manual segmentation combined with fatigue due to the number of slices to segment can cause human errors. To address these challenges, computational solutions for analyzing medical images and proposing automated segmentation have been developed and explored in recent years. In this work, we propose a fully automatic method for esophagus segmentation for better planning of radiotherapy in CT.

METHODS

The proposed method is a fully automated segmentation of the esophagus, consisting of 5 main steps: (a) image acquisition; (b) VOI segmentation; (c) preprocessing; (d) esophagus segmentation; and (e) segmentation refinement.

RESULTS

The method was applied in a database of 36 CT acquired from 3 different institutes. It achieved the best results in literature so far: Dice coefficient value of 82.15%, Jaccard Index of 70.21%, accuracy of 99.69%, sensitivity of 90.61%, specificity of 99.76%, and Hausdorff Distance of 6.1030 mm.

CONCLUSIONS

With the achieved results, we were able to show how promising the method is, and that applying it in large medical centers, where esophagus segmentation is still an arduous and challenging task, can be of great help to the specialists.

Combining Sparse and Dense Features to Improve Multi-Modal Registration for Brain DTI Images

A new solution to overcome the constraints of multimodality medical intra-subject image registration is proposed, using the mutual information (MI) of image histogram-oriented gradients as a new matching criterion. We present a rigid, multi-modal image registration algorithm based on linear transformation and oriented gradients for the alignment of T2-weighted (T2w) images (as a fixed reference) and diffusion tensor imaging (DTI) (b-values of 500 and 1250 s/mm2) as floating images of three patients to compensate for the motion during the acquisition process. Diffusion MRI is very sensitive to motion, especially when the intensity and duration of the gradient pulses (characterized by the b-value) increases. The proposed method relies on the whole brain surface and addresses the variability of anatomical features into an image stack. The sparse features refer to corners detected using the Harris corner detector operator, while dense features use all image pixels through the image histogram of oriented gradients (HOG) as a measure of the degree of statistical dependence between a pair of registered images. HOG as a dense feature is focused on the structure and extracts the oriented gradient image in the x and y directions. MI is used as an objective function for the optimization process. The entropy functions and joint entropy function are determined using the HOGs data. To determine the best image transformation, the fiducial registration error (FRE) measure is used. We compare the results against the MI-based intensities results computed using a statistical intensity relationship between corresponding pixels in source and target images. Our approach, which is devoted to the whole brain, shows improved registration accuracy, robustness, and computational cost compared with the registration algorithms, which use anatomical features or regions of interest areas with specific neuroanatomy. Despite the supplementary HOG computation task, the computation time is comparable for MI-based intensities and MI-based HOG methods.

Relative Regional Air Volume Change Maps at the Acinar Scale Reflect Variable Ventilation in Low Lung Attenuation of COPD patients

OBJECTIVES

The purpose of this study was to investigate regional air volume changes at the acinar scale of the lung in chronic obstructive pulmonary disease (COPD) patients using an image registration technique.

MATERIALS AND METHODS

Thirty-four emphysema patients and 24 subjects with normal chest CT and pulmonary function test (PFT) results were included in this retrospective study for which informed consent was waived by the institutional review board. After lung segmentation, a mass-preserving image registration technique was used to compute relative regional air volume changes (RRAVCs) between inspiration and expiration CT scans. After determining the appropriate thresholds of RRAVCs for low ventilation areas (LVAs), they were displayed and analyzed using color maps on the background inspiration CT image, and compared with the low attenuation area (LAA) map. Correlations between quantitative CT parameters and PFTs were assessed using Pearson's correlation test, and parameters were compared between emphysema and normal-CT patients using the Student's t-test.

RESULTS

LVA percentage with an RRAVC threshold of 0.5 (%LVA_0.5) showed the strongest correlations with FEV₁/FVC (r = -0.566), FEV₁ (r = -0.534), %LAA_-950insp (r = 0.712), and %LAA_-856exp (r = 0.775). %LVA_0.5 was significantly higher (P < 0.001) in COPD patients than normal subjects. Despite the identical appearance of emphysematous lesions on the LAA_-950insp map, the RRAVC map depicted a wide range of ventilation differences between these LAA clusters.

CONCLUSION

RRAVC-based %LVA_0.5 correlated well with FEV₁/FVC, FEV₁, %LAA_-950insp and %LAA_-856exp. RRAVC holds the potential for providing additional acinar scale functional information for emphysematous LAAs in inspiratory CT images, providing the basis for a novel set for emphysematous phenotypes.

Facial Asymmetry in Nonsyndromic and Muenke Syndrome-Associated Unicoronal Synostosis: A 3-Dimensional Study Based on Facial Surfaces Extracted From CT Scans

OBJECTIVE

To quantify soft tissue facial asymmetry (FA) in children with nonsyndromic and Muenke syndrome-associated unicoronal synostosis (NS-UCS and MS-UCS), hypothesizing that MS-UCS presents with significantly larger FA than NS-UCS.

DESIGN

Retrospective cohort study.

PATIENTS AND METHODS

Twenty-one children (mean age: 0.6 years; range: 0.1-1.4 years) were included in the study (NS-UCS = 14; MS-UCS = 7). From presurgical computed tomography scans, facial surfaces were constructed for analysis. A landmark guided atlas was deformed to match each patient's surface, obtaining spatially detailed left-right point correspondence. Facial asymmetry was calculated in each surface point across the face, as the length (mm) of an asymmetry vector, with its Cartesian components providing 3 directions. Mean FA was calculated for the full face, and the forehead, eye, nose, cheek, mouth, and chin regions.

RESULTS

For the full face, a significant difference of 2.4 mm (P = .001) was calculated between the 2 groups, predominately in the transverse direction (1.5 mm; P < .001). The forehead and chin regions presented with the largest significant difference, 3.5 mm (P = .002) and 3.2 mm (P < .001), respectively; followed by the eye (2.4 mm; P = .004), cheek (2.2 mm; P = .004), nose (1.7 mm; P = .001), and mouth (1.4 mm; P = .009) regions. The transverse direction presented with the largest significant difference in the forehead, chin, mouth, and nose regions, the sagittal direction in the cheek region, and the vertical direction in the eye region.

CONCLUSIONS

Muenke syndrome-associated unicoronal synostosis presented with significantly larger FA in all regions compared to NS-UCS. The largest significant differences were found in the forehead and chin regions, predominantly in the transverse direction.

Grayscale representation of infrared microscopy images by extended multiplicative signal correction for registration with histological images

Fourier-transform infrared (FTIR) microspectroscopy is rounding the corner to become a label-free routine method for cancer diagnosis. In order to build infrared-spectral based classifiers, infrared images need to be registered with Hematoxylin and Eosin (H&E) stained histological images. While FTIR images have a deep spectral domain with thousands of channels carrying chemical and scatter information, the H&E images have only three color channels for each pixel and carry mainly morphological information. Therefore, image representations of infrared images are needed that match the morphological information in H&E images. In this paper, we propose a novel approach for representation of FTIR images based on extended multiplicative signal correction highlighting morphological features that showed to correlate well with morphological information in H&E images. Based on the obtained representations, we developed a strategy for global-to-local image registration for FTIR images and H&E stained histological images of parallel tissue sections.