Real-Time Deep Pose Estimation With Geodesic Loss for Image-to-Template Rigid Registration

Towards automatic US-MR fetal brain image registration with learning-based methods

Fetal brain imaging is essential for prenatal care, with ultrasound (US) and magnetic resonance imaging (MRI) providing complementary strengths. While MRI has superior soft tissue contrast, US offers portable and inexpensive screening of neurological abnormalities. Despite the great potential synergy of combined fetal brain US and MR imaging to enhance diagnostic accuracy, little effort has been made to integrate these modalities. An essential step towards this integration is accurate automatic spatial alignment, which is technically very challenging due to the inherent differences in contrast and modality-specific imaging artifacts. In this work, we present a novel atlas-assisted multi-task learning technique to address this problem. Instead of training the registration model solely with intra-subject US-MR image pairs, our approach enables the network to also learn from domain-specific image-to-atlas registration tasks. This leads to an end-to-end multi-task learning framework with superior registration performance. Our proposed method was validated using a dataset of same-day intra-subject 3D US-MR image pairs. The results show that our method outperforms conventional optimization-based methods and recent learning-based techniques for rigid image registration. Specifically, the average target registration error for our method is less than 4 mm, which is significantly better than existing methods. Extensive experiments have also shown that our method has a much wider capture range and is robust to brain abnormalities. Given these advantages over existing techniques, our method is more suitable for deployment in clinical workflows and may contribute to streamlined multimodal imaging pipelines for fetal brain assessment.

A critical assessment of artificial intelligence in magnetic resonance imaging of cancer

Given the enormous output and pace of development of artificial intelligence (AI) methods in medical imaging, it can be challenging to identify the true success stories to determine the state-of-the-art of the field. This report seeks to provide the magnetic resonance imaging (MRI) community with an initial guide into the major areas in which the methods of AI are contributing to MRI in oncology. After a general introduction to artificial intelligence, we proceed to discuss the successes and current limitations of AI in MRI when used for image acquisition, reconstruction, registration, and segmentation, as well as its utility for assisting in diagnostic and prognostic settings. Within each section, we attempt to present a balanced summary by first presenting common techniques, state of readiness, current clinical needs, and barriers to practical deployment in the clinical setting. We conclude by presenting areas in which new advances must be realized to address questions regarding generalizability, quality assurance and control, and uncertainty quantification when applying MRI to cancer to maintain patient safety and practical utility.

Application-driven validation of posteriors in inverse problems

Current deep learning-based solutions for image analysis tasks are commonly incapable of handling problems to which multiple different plausible solutions exist. In response, posterior-based methods such as conditional Diffusion Models and Invertible Neural Networks have emerged; however, their translation is hampered by a lack of research on adequate validation. In other words, the way progress is measured often does not reflect the needs of the driving practical application. Closing this gap in the literature, we present the first systematic framework for the application-driven validation of posterior-based methods in inverse problems. As a methodological novelty, it adopts key principles from the field of object detection validation, which has a long history of addressing the question of how to locate and match multiple object instances in an image. Treating modes as instances enables us to perform mode-centric validation, using well-interpretable metrics from the application perspective. We demonstrate the value of our framework through instantiations for a synthetic toy example and two medical vision use cases: pose estimation in surgery and imaging-based quantification of functional tissue parameters for diagnostics. Our framework offers key advantages over common approaches to posterior validation in all three examples and could thus revolutionize performance assessment in inverse problems.

Scanner-based real-time three-dimensional brain + body slice-to-volume reconstruction for T2-weighted 0.55-T low-field fetal magnetic resonance imaging

BACKGROUND

Motion correction methods based on slice-to-volume registration (SVR) for fetal magnetic resonance imaging (MRI) allow reconstruction of three-dimensional (3-D) isotropic images of the fetal brain and body. However, all existing SVR methods are confined to research settings, which limits clinical integration. Furthermore, there have been no reported SVR solutions for low-field 0.55-T MRI.

OBJECTIVE

Integration of automated SVR motion correction methods directly into fetal MRI scanning process via the Gadgetron framework to enable automated T2-weighted (T2W) 3-D fetal brain and body reconstruction in the low-field 0.55-T MRI scanner within the duration of the scan.

MATERIALS AND METHODS

A deep learning fully automated pipeline was developed for T2W 3-D rigid and deformable (D/SVR) reconstruction of the fetal brain and body of 0.55-T T2W datasets. Next, it was integrated into 0.55-T low-field MRI scanner environment via a Gadgetron workflow that enables launching of the reconstruction process directly during scanning in real-time.

RESULTS

During prospective testing on 12 cases (22-40 weeks gestational age), the fetal brain and body reconstructions were available on average 6:42 ± 3:13 min after the acquisition of the final stack and could be assessed and archived on the scanner console during the ongoing fetal MRI scan. The output image data quality was rated as good to acceptable for interpretation. The retrospective testing of the pipeline on 83 0.55-T datasets demonstrated stable reconstruction quality for low-field MRI.

CONCLUSION

The proposed pipeline allows scanner-based prospective T2W 3-D motion correction for low-field 0.55-T fetal MRI via direct online integration into the scanner environment.

Boosting 2D brain image registration via priors from large model

BACKGROUND

Deformable medical image registration aims to align image pairs with local differences, improving the accuracy of medical analyses and assisting various diagnostic scenarios.

PURPOSE

We aim to overcome these challenges: Deep learning-based registration approaches have greatly enhanced registration speed and accuracy by continuously improving registration networks and processes. However, the lack of extensive medical datasets limits the complexity of registration models. Optimizing registration networks within a fixed dataset often leads to overfitting, hindering further accuracy improvements and reducing generalization capabilities.

METHODS

We explore the application of the foundational model DINOv2 to registration tasks, leveraging its prior knowledge to support learning-based unsupervised registration networks and overcome network bottlenecks to improve accuracy. We investigate three modes of DINOv2-assisted registration, including direct registration architecture, enhanced architecture, and refined architecture. Additionally, we study the applicability of three feature aggregation methods-convolutional interaction, direct fusion, and cross-attention-within the proposed DINOv2-based registration frameworks.

RESULTS

We conducted extensive experiments on the IXI and OASIS public datasets, demonstrating that the enhanced and refined architectures notably improve registration accuracy, reduce data dependency, and maintain strong generalization capabilities.

CONCLUSION

This study offers novel approaches for applying foundational models to deformable image registration tasks.

GMmorph: dynamic spatial matching registration model for 3D medical image based on gated Mamba

Objective.Deformable registration aims to achieve nonlinear alignment of image space by estimating a dense displacement field. It is commonly used as a preprocessing step in clinical and image analysis applications, such as surgical planning, diagnostic assistance, and surgical navigation. We aim to overcome these challenges: Deep learning-based registration methods often struggle with complex displacements and lack effective interaction between global and local feature information. They also neglect the spatial position matching process, leading to insufficient registration accuracy and reduced robustness when handling abnormal tissues.Approach.We propose a dual-branch interactive registration model architecture from the perspective of spatial matching. Implicit regularization is achieved through a consistency loss, enabling the network to balance high accuracy with a low folding rate. We introduced the dynamic matching module between the two branches of the registration, which generates learnable offsets based on all the tokens across the entire resolution range of the base branch features. Using trilinear interpolation, the model adjusts its feature expression range according to the learned offsets, capturing highly flexible positional differences. To facilitate the spatial matching process, we designed the gated mamba layer to globally model pixel-level features by associating all voxel information, while the detail enhancement module, which is based on channel and spatial attention, enhances the richness of local feature details.Main results.Our study explores the model's performance in single-modal and multi-modal image registration, including normal brain, brain tumor, and lung images. We propose unsupervised and semi-supervised registration modes and conduct extensive validation experiments. The results demonstrate that the model achieves state-of-the-art performance across multiple datasets.Significance.By introducing a novel perspective of position matching, the model achieves precise registration of various types of medical data, offering significant clinical value in medical applications.

SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI

Rigid motion tracking is paramount in many medical imaging applications where movements need to be detected, corrected, or accounted for. Modern strategies rely on convolutional neural networks (CNN) and pose this problem as rigid registration. Yet, CNNs do not exploit natural symmetries in this task, as they are equivariant to translations (their outputs shift with their inputs) but not to rotations. Here we propose EquiTrack, the first method that uses recent steerable SE(3)-equivariant CNNs (E-CNN) for motion tracking. While steerable E-CNNs can extract corresponding features across different poses, testing them on noisy medical images reveals that they do not have enough learning capacity to learn noise invariance. Thus, we introduce a hybrid architecture that pairs a denoiser with an E-CNN to decouple the processing of anatomically irrelevant intensity features from the extraction of equivariant spatial features. Rigid transforms are then estimated in closed-form. EquiTrack outperforms state-of-the-art learning and optimisation methods for motion tracking in adult brain MRI and fetal MRI time series. Our code is available at https://github.com/BBillot/EquiTrack.

Fully automated planning for anatomical fetal brain MRI on 0.55T

PURPOSE

Widening the availability of fetal MRI with fully automatic real-time planning of radiological brain planes on 0.55T MRI.

METHODS

Deep learning-based detection of key brain landmarks on a whole-uterus echo planar imaging scan enables the subsequent fully automatic planning of the radiological single-shot Turbo Spin Echo acquisitions. The landmark detection pipeline was trained on over 120 datasets from varying field strength, echo times, and resolutions and quantitatively evaluated. The entire automatic planning solution was tested prospectively in nine fetal subjects between 20 and 37 weeks. A comprehensive evaluation of all steps, the distance between manual and automatic landmarks, the planning quality, and the resulting image quality was conducted.

RESULTS

Prospective automatic planning was performed in real-time without latency in all subjects. The landmark detection accuracy was 4.2± $$ \pm $$ 2.6 mm for the fetal eyes and 6.5± $$ \pm $$ 3.2 for the cerebellum, planning quality was 2.4/3 (compared to 2.6/3 for manual planning) and diagnostic image quality was 2.2 compared to 2.1 for manual planning.

CONCLUSIONS

Real-time automatic planning of all three key fetal brain planes was successfully achieved and will pave the way toward simplifying the acquisition of fetal MRI thereby widening the availability of this modality in nonspecialist centers.

ACSwinNet: A Deep Learning-Based Rigid Registration Method for Head-Neck CT-CBCT Images in Image-Guided Radiotherapy

Accurate and precise rigid registration between head-neck computed tomography (CT) and cone-beam computed tomography (CBCT) images is crucial for correcting setup errors in image-guided radiotherapy (IGRT) for head and neck tumors. However, conventional registration methods that treat the head and neck as a single entity may not achieve the necessary accuracy for the head region, which is particularly sensitive to radiation in radiotherapy. We propose ACSwinNet, a deep learning-based method for head-neck CT-CBCT rigid registration, which aims to enhance the registration precision in the head region. Our approach integrates an anatomical constraint encoder with anatomical segmentations of tissues and organs to enhance the accuracy of rigid registration in the head region. We also employ a Swin Transformer-based network for registration in cases with large initial misalignment and a perceptual similarity metric network to address intensity discrepancies and artifacts between the CT and CBCT images. We validate the proposed method using a head-neck CT-CBCT dataset acquired from clinical patients. Compared with the conventional rigid method, our method exhibits lower target registration error (TRE) for landmarks in the head region (reduced from 2.14 ± 0.45 mm to 1.82 ± 0.39 mm), higher dice similarity coefficient (DSC) (increased from 0.743 ± 0.051 to 0.755 ± 0.053), and higher structural similarity index (increased from 0.854 ± 0.044 to 0.870 ± 0.043). Our proposed method effectively addresses the challenge of low registration accuracy in the head region, which has been a limitation of conventional methods. This demonstrates significant potential in improving the accuracy of IGRT for head and neck tumors.

Empowering brain cancer diagnosis: harnessing artificial intelligence for advanced imaging insights

Artificial intelligence (AI) is increasingly being used in the medical field, specifically for brain cancer imaging. In this review, we explore how AI-powered medical imaging can impact the diagnosis, prognosis, and treatment of brain cancer. We discuss various AI techniques, including deep learning and causality learning, and their relevance. Additionally, we examine current applications that provide practical solutions for detecting, classifying, segmenting, and registering brain tumors. Although challenges such as data quality, availability, interpretability, transparency, and ethics persist, we emphasise the enormous potential of intelligent applications in standardising procedures and enhancing personalised treatment, leading to improved patient outcomes. Innovative AI solutions have the power to revolutionise neuro-oncology by enhancing the quality of routine clinical practice.

Test Platform for Developing New Optical Position Tracking Technology towards Improved Head Motion Correction in Magnetic Resonance Imaging

Optical tracking of head pose via fiducial markers has been proven to enable effective correction of motion artifacts in the brain during magnetic resonance imaging but remains difficult to implement in the clinic due to lengthy calibration and set up times. Advances in deep learning for markerless head pose estimation have yet to be applied to this problem because of the sub-millimetre spatial resolution required for motion correction. In the present work, two optical tracking systems are described for the development and training of a neural network: one marker-based system (a testing platform for measuring ground truth head pose) with high tracking fidelity to act as the training labels, and one markerless deep-learning-based system using images of the markerless head as input to the network. The markerless system has the potential to overcome issues of marker occlusion, insufficient rigid attachment of the marker, lengthy calibration times, and unequal performance across degrees of freedom (DOF), all of which hamper the adoption of marker-based solutions in the clinic. Detail is provided on the development of a custom moiré-enhanced fiducial marker for use as ground truth and on the calibration procedure for both optical tracking systems. Additionally, the development of a synthetic head pose dataset is described for the proof of concept and initial pre-training of a simple convolutional neural network. Results indicate that the ground truth system has been sufficiently calibrated and can track head pose with an error of <1 mm and <1°. Tracking data of a healthy, adult participant are shown. Pre-training results show that the average root-mean-squared error across the 6 DOF is 0.13 and 0.36 (mm or degrees) on a head model included and excluded from the training dataset, respectively. Overall, this work indicates excellent feasibility of the deep-learning-based approach and will enable future work in training and testing on a real dataset in the MRI environment.

PETS-Nets: Joint Pose Estimation and Tissue Segmentation of Fetal Brains Using Anatomy-Guided Networks

Fetal Magnetic Resonance Imaging (MRI) is challenged by fetal movements and maternal breathing. Although fast MRI sequences allow artifact free acquisition of individual 2D slices, motion frequently occurs in the acquisition of spatially adjacent slices. Motion correction for each slice is thus critical for the reconstruction of 3D fetal brain MRI. In this paper, we propose a novel multi-task learning framework that adopts a coarse-to-fine strategy to jointly learn the pose estimation parameters for motion correction and tissue segmentation map of each slice in fetal MRI. Particularly, we design a regression-based segmentation loss as a deep supervision to learn anatomically more meaningful features for pose estimation and segmentation. In the coarse stage, a U-Net-like network learns the features shared for both tasks. In the refinement stage, to fully utilize the anatomical information, signed distance maps constructed from the coarse segmentation are introduced to guide the feature learning for both tasks. Finally, iterative incorporation of the signed distance maps further improves the performance of both regression and segmentation progressively. Experimental results of cross-validation across two different fetal datasets acquired with different scanners and imaging protocols demonstrate the effectiveness of the proposed method in reducing the pose estimation error and obtaining superior tissue segmentation results simultaneously, compared with state-of-the-art methods.

Affine medical image registration with fusion feature mapping in local and global

Objective. Medical image affine registration is a crucial basis before using deformable registration. On the one hand, the traditional affine registration methods based on step-by-step optimization are very time-consuming, so these methods are not compatible with most real-time medical applications. On the other hand, convolutional neural networks are limited in modeling long-range spatial relationships of the features due to inductive biases, such as weight sharing and locality. This is not conducive to affine registration tasks. Therefore, the evolution of real-time and high-accuracy affine medical image registration algorithms is necessary for registration applications.Approach. In this paper, we propose a deep learning-based coarse-to-fine global and local feature fusion architecture for fast affine registration, and we use an unsupervised approach for end-to-end training. We use multiscale convolutional kernels as our elemental convolutional blocks to enhance feature extraction. Then, to learn the long-range spatial relationships of the features, we propose a new affine registration framework with weighted global positional attention that fuses global feature mapping and local feature mapping. Moreover, the fusion regressor is designed to generate the affine parameters.Main results. The additive fusion method can be adaptive to global mapping and local mapping, which improves affine registration accuracy without the center of mass initialization. In addition, the max pooling layer and the multiscale convolutional kernel coding module increase the ability of the model in affine registration.Significance. We validate the effectiveness of our method on the OASIS dataset with 414 3D MRI brain maps. Comprehensive results demonstrate that our method achieves state-of-the-art affine registration accuracy and very efficient runtimes.

Real-time fetal brain tracking for functional fetal MRI

PURPOSE

To improve motion robustness of functional fetal MRI scans by developing an intrinsic real-time motion correction method. MRI provides an ideal tool to characterize fetal brain development and growth. It is, however, a relatively slow imaging technique and therefore extremely susceptible to subject motion, particularly in functional MRI experiments acquiring multiple Echo-Planar-Imaging-based repetitions, for example, diffusion MRI or blood-oxygen-level-dependency MRI.

METHODS

A 3D UNet was trained on 125 fetal datasets to track the fetal brain position in each repetition of the scan in real time. This tracking, inserted into a Gadgetron pipeline on a clinical scanner, allows updating the position of the field of view in a modified echo-planar imaging sequence. The method was evaluated in real-time in controlled-motion phantom experiments and ten fetal MR studies (17 + 4-34 + 3 gestational weeks) at 3T. The localization network was additionally tested retrospectively on 29 low-field (0.55T) datasets.

RESULTS

Our method achieved real-time fetal head tracking and prospective correction of the acquisition geometry. Localization performance achieved Dice scores of 84.4% and 82.3%, respectively for both the unseen 1.5T/3T and 0.55T fetal data, with values higher for cephalic fetuses and increasing with gestational age.

CONCLUSIONS

Our technique was able to follow the fetal brain even for fetuses under 18 weeks GA in real-time at 3T and was successfully applied "offline" to new cohorts on 0.55T. Next, it will be deployed to other modalities such as fetal diffusion MRI and to cohorts of pregnant participants diagnosed with pregnancy complications, for example, pre-eclampsia and congenital heart disease.

Fast fetal head compounding from multi-view 3D ultrasound

The diagnostic value of ultrasound images may be limited by the presence of artefacts, notably acoustic shadows, lack of contrast and localised signal dropout. Some of these artefacts are dependent on probe orientation and scan technique, with each image giving a distinct, partial view of the imaged anatomy. In this work, we propose a novel method to fuse the partially imaged fetal head anatomy, acquired from numerous views, into a single coherent 3D volume of the full anatomy. Firstly, a stream of freehand 3D US images is acquired using a single probe, capturing as many different views of the head as possible. The imaged anatomy at each time-point is then independently aligned to a canonical pose using a recurrent spatial transformer network, making our approach robust to fast fetal and probe motion. Secondly, images are fused by averaging only the most consistent and salient features from all images, producing a more detailed compounding, while minimising artefacts. We evaluated our method quantitatively and qualitatively, using image quality metrics and expert ratings, yielding state of the art performance in terms of image quality and robustness to misalignments. Being online, fast and fully automated, our method shows promise for clinical use and deployment as a real-time tool in the fetal screening clinic, where it may enable unparallelled insight into the shape and structure of the face, skull and brain.

Emerging Trends in Fast MRI Using Deep-Learning Reconstruction on Undersampled k-Space Data: A Systematic Review

Magnetic Resonance Imaging (MRI) is an essential medical imaging modality that provides excellent soft-tissue contrast and high-resolution images of the human body, allowing us to understand detailed information on morphology, structural integrity, and physiologic processes. However, MRI exams usually require lengthy acquisition times. Methods such as parallel MRI and Compressive Sensing (CS) have significantly reduced the MRI acquisition time by acquiring less data through undersampling k-space. The state-of-the-art of fast MRI has recently been redefined by integrating Deep Learning (DL) models with these undersampled approaches. This Systematic Literature Review (SLR) comprehensively analyzes deep MRI reconstruction models, emphasizing the key elements of recently proposed methods and highlighting their strengths and weaknesses. This SLR involves searching and selecting relevant studies from various databases, including Web of Science and Scopus, followed by a rigorous screening and data extraction process using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. It focuses on various techniques, such as residual learning, image representation using encoders and decoders, data-consistency layers, unrolled networks, learned activations, attention modules, plug-and-play priors, diffusion models, and Bayesian methods. This SLR also discusses the use of loss functions and training with adversarial networks to enhance deep MRI reconstruction methods. Moreover, we explore various MRI reconstruction applications, including non-Cartesian reconstruction, super-resolution, dynamic MRI, joint learning of reconstruction with coil sensitivity and sampling, quantitative mapping, and MR fingerprinting. This paper also addresses research questions, provides insights for future directions, and emphasizes robust generalization and artifact handling. Therefore, this SLR serves as a valuable resource for advancing fast MRI, guiding research and development efforts of MRI reconstruction for better image quality and faster data acquisition.

A two-step deep learning method for 3DCT-2DUS kidney registration during breathing

This work proposed KidneyRegNet, a novel deep registration pipeline for 3D CT and 2D U/S kidney scans of free breathing, which comprises a feature network, and a 3D-2D CNN-based registration network. The feature network has handcrafted texture feature layers to reduce the semantic gap. The registration network is an encoder-decoder structure with loss of feature-image-motion (FIM), which enables hierarchical regression at decoder layers and avoids multiple network concatenation. It was first pretrained with a retrospective dataset cum training data generation strategy and then adapted to specific patient data under unsupervised one-cycle transfer learning in onsite applications. The experiment was performed on 132 U/S sequences, 39 multiple-phase CT and 210 public single-phase CT images, and 25 pairs of CT and U/S sequences. This resulted in a mean contour distance (MCD) of 0.94 mm between kidneys on CT and U/S images and MCD of 1.15 mm on CT and reference CT images. Datasets with small transformations resulted in MCDs of 0.82 and 1.02 mm, respectively. Large transformations resulted in MCDs of 1.10 and 1.28 mm, respectively. This work addressed difficulties in 3DCT-2DUS kidney registration during free breathing via novel network structures and training strategies.

2D/3D Deep Registration Along Trajectories With Spatiotemporal Context: Application to Prostate Biopsy Navigation

OBJECTIVE

The accuracy of biopsy targeting is a major issue for prostate cancer diagnosis and therapy. However, navigation to biopsy targets remains challenging due to the limitations of transrectal ultrasound (TRUS) guidance added to prostate motion issues. This article describes a rigid 2D/3D deep registration method, which provides a continuous tracking of the biopsy location w.r.t the prostate for enhanced navigation.

METHODS

A spatiotemporal registration network (SpT-Net) is proposed to localize the live 2D US image relatively to a previously aquired US reference volume. The temporal context relies on prior trajectory information based on previous registration results and probe tracking. Different forms of spatial context were compared through inputs (local, partial or global) or using an additional spatial penalty term. The proposed 3D CNN architecture with all combinations of spatial and temporal context was evaluated in an ablation study. For providing a realistic clinical validation, a cumulative error was computed through series of registrations along trajectories, simulating a complete clinical navigation procedure. We also proposed two dataset generation processes with increasing levels of registration complexity and clinical realism.

RESULTS

The experiments show that a model using local spatial information combined with temporal information performs better than more complex spatiotemporal combination.

CONCLUSION

The best proposed model demonstrates robust real-time 2D/3D US cumulated registration performance on trajectories. Those results respect clinical requirements, application feasibility, and they outperform similar state-of-the-art methods.

SIGNIFICANCE

Our approach seems promising for clinical prostate biopsy navigation assistance or other US image-guided procedure.

Retrospective motion correction in foetal MRI for clinical applications: existing methods, applications and integration into clinical practice

Foetal MRI is a complementary imaging method to antenatal ultrasound. It provides advanced information for detection and characterisation of foetal brain and body anomalies. Even though modern single shot sequences allow fast acquisition of 2D slices with high in-plane image quality, foetal MRI is intrinsically corrupted by motion. Foetal motion leads to loss of structural continuity and corrupted 3D volumetric information in stacks of slices. Furthermore, the arbitrary and constantly changing position of the foetus requires dynamic readjustment of acquisition planes during scanning.

Artificial intelligence applied to fetal MRI: A scoping review of current research

Artificial intelligence (AI) is defined as the development of computer systems to perform tasks normally requiring human intelligence. A subset of AI, known as machine learning (ML), takes this further by drawing inferences from patterns in data to 'learn' and 'adapt' without explicit instructions meaning that computer systems can 'evolve' and hopefully improve without necessarily requiring external human influences. The potential for this novel technology has resulted in great interest from the medical community regarding how it can be applied in healthcare. Within radiology, the focus has mostly been for applications in oncological imaging, although new roles in other subspecialty fields are slowly emerging.In this scoping review, we performed a literature search of the current state-of-the-art and emerging trends for the use of artificial intelligence as applied to fetal magnetic resonance imaging (MRI). Our search yielded several publications covering AI tools for anatomical organ segmentation, improved imaging sequences and aiding in diagnostic applications such as automated biometric fetal measurements and the detection of congenital and acquired abnormalities. We highlight our own perceived gaps in this literature and suggest future avenues for further research. It is our hope that the information presented highlights the varied ways and potential that novel digital technology could make an impact to future clinical practice with regards to fetal MRI.

Accurate volume alignment of arbitrarily oriented tibiae based on a mutual attention network for osteoarthritis analysis

Damage to cartilage is an important indicator of osteoarthritis progression, but manual extraction of cartilage morphology is time-consuming and prone to error. To address this, we hypothesize that automatic labeling of cartilage can be achieved through the comparison of contrasted and non-contrasted Computer Tomography (CT). However, this is non-trivial as the pre-clinical volumes are at arbitrary starting poses due to the lack of standardized acquisition protocols. Thus, we propose an annotation-free deep learning method, D-net, for accurate and automatic alignment of pre- and post-contrasted cartilage CT volumes. D-Net is based on a novel mutual attention network structure to capture large-range translation and full-range rotation without the need for a prior pose template. CT volumes of mice tibiae are used for validation, with synthetic transformation for training and tested with real pre- and post-contrasted CT volumes. Analysis of Variance (ANOVA) was used to compare the different network structures. Our proposed method, D-net, achieves a Dice coefficient of 0.87, and significantly outperforms other state-of-the-art deep learning models, in the real-world alignment of 50 pairs of pre- and post-contrasted CT volumes when cascaded as a multi-stage network.

Clinical application of AI-based PET images in oncological patients

Based on the advantages of revealing the functional status and molecular expression of tumor cells, positron emission tomography (PET) imaging has been performed in numerous types of malignant diseases for diagnosis and monitoring. However, insufficient image quality, the lack of a convincing evaluation tool and intra- and interobserver variation in human work are well-known limitations of nuclear medicine imaging and restrict its clinical application. Artificial intelligence (AI) has gained increasing interest in the field of medical imaging due to its powerful information collection and interpretation ability. The combination of AI and PET imaging potentially provides great assistance to physicians managing patients. Radiomics, an important branch of AI applied in medical imaging, can extract hundreds of abstract mathematical features of images for further analysis. In this review, an overview of the applications of AI in PET imaging is provided, focusing on image enhancement, tumor detection, response and prognosis prediction and correlation analyses with pathology or specific gene mutations in several types of tumors. Our aim is to describe recent clinical applications of AI-based PET imaging in malignant diseases and to focus on the description of possible future developments.

AFFIRM: Affinity Fusion-Based Framework for Iteratively Random Motion Correction of Multi-Slice Fetal Brain MRI

Multi-slice magnetic resonance images of the fetal brain are usually contaminated by severe and arbitrary fetal and maternal motion. Hence, stable and robust motion correction is necessary to reconstruct high-resolution 3D fetal brain volume for clinical diagnosis and quantitative analysis. However, the conventional registration-based correction has a limited capture range and is insufficient for detecting relatively large motions. Here, we present a novel Affinity Fusion-based Framework for Iteratively Random Motion (AFFIRM) correction of the multi-slice fetal brain MRI. It learns the sequential motion from multiple stacks of slices and integrates the features between 2D slices and reconstructed 3D volume using affinity fusion, which resembles the iterations between slice-to-volume registration and volumetric reconstruction in the regular pipeline. The method accurately estimates the motion regardless of brain orientations and outperforms other state-of-the-art learning-based methods on the simulated motion-corrupted data, with a 48.4% reduction of mean absolute error for rotation and 61.3% for displacement. We then incorporated AFFIRM into the multi-resolution slice-to-volume registration and tested it on the real-world fetal MRI scans at different gestation stages. The results indicated that adding AFFIRM to the conventional pipeline improved the success rate of fetal brain super-resolution reconstruction from 77.2% to 91.9%.

A component method to delineate surgical spine implants for proton Monte Carlo dose calculation

PURPOSE

Metallic implants have been correlated to local control failure for spinal sarcoma and chordoma patients due to the uncertainty of implant delineation from computed tomography (CT). Such uncertainty can compromise the proton Monte Carlo dose calculation (MCDC) accuracy. A component method is proposed to determine the dimension and volume of the implants from CT images.

METHODS

The proposed component method leverages the knowledge of surgical implants from medical supply vendors to predefine accurate contours for each implant component, including tulips, screw bodies, lockers, and rods. A retrospective patient study was conducted to demonstrate the feasibility of the method. The reference implant materials and samples were collected from patient medical records and vendors, Medtronic and NuVasive. Additional CT images with extensive features, such as extended Hounsfield units and various reconstruction diameters, were used to quantify the uncertainty of implant contours.

RESULTS

For in vivo patient implant estimation, the reference and the component method differences were 0.35, 0.17, and 0.04 cm3 for tulips, screw bodies, and rods, respectively. The discrepancies by a conventional threshold method were 5.46, 0.76, and 0.05 cm3 , respectively. The mischaracterization of implant materials and dimensions can underdose the clinical target volume coverage by 20 cm3 for a patient with eight lumbar implants. The tulip dominates the dosimetry uncertainty as it can be made from titanium or cobalt-chromium alloys by different vendors.

CONCLUSIONS

A component method was developed and demonstrated using phantom and patient studies with implants. The proposed method provides more accurate implant characterization for proton MCDC and can potentially enhance the treatment quality for proton therapy. The current proof-of-concept study is limited to the implant characterization for lumbar spine. Future investigations could be extended to cervical spine and dental implants for head-and-neck patients where tight margins are required to spare organs at risk.

A review on AI-based medical image computing in head and neck surgery

Head and neck surgery is a fine surgical procedure with a complex anatomical space, difficult operation and high risk. Medical image computing (MIC) that enables accurate and reliable preoperative planning is often needed to reduce the operational difficulty of surgery and to improve patient survival. At present, artificial intelligence, especially deep learning, has become an intense focus of research in MIC. In this study, the application of deep learning-based MIC in head and neck surgery is reviewed. Relevant literature was retrieved on the Web of Science database from January 2015 to May 2022, and some papers were selected for review from mainstream journals and conferences, such as IEEE Transactions on Medical Imaging, Medical Image Analysis, Physics in Medicine and Biology, Medical Physics, MICCAI, etc. Among them, 65 references are on automatic segmentation, 15 references on automatic landmark detection, and eight references on automatic registration. In the elaboration of the review, first, an overview of deep learning in MIC is presented. Then, the application of deep learning methods is systematically summarized according to the clinical needs, and generalized into segmentation, landmark detection and registration of head and neck medical images. In segmentation, it is mainly focused on the automatic segmentation of high-risk organs, head and neck tumors, skull structure and teeth, including the analysis of their advantages, differences and shortcomings. In landmark detection, the focus is mainly on the introduction of landmark detection in cephalometric and craniomaxillofacial images, and the analysis of their advantages and disadvantages. In registration, deep learning networks for multimodal image registration of the head and neck are presented. Finally, their shortcomings and future development directions are systematically discussed. The study aims to serve as a reference and guidance for researchers, engineers or doctors engaged in medical image analysis of head and neck surgery.

SVoRT: Iterative Transformer for Slice-to-Volume Registration in Fetal Brain MRI

Volumetric reconstruction of fetal brains from multiple stacks of MR slices, acquired in the presence of almost unpredictable and often severe subject motion, is a challenging task that is highly sensitive to the initialization of slice-to-volume transformations. We propose a novel slice-to-volume registration method using Transformers trained on synthetically transformed data, which model multiple stacks of MR slices as a sequence. With the attention mechanism, our model automatically detects the relevance between slices and predicts the transformation of one slice using information from other slices. We also estimate the underlying 3D volume to assist slice-to-volume registration and update the volume and transformations alternately to improve accuracy. Results on synthetic data show that our method achieves lower registration error and better reconstruction quality compared with existing state-of-the-art methods. Experiments with real-world MRI data are also performed to demonstrate the ability of the proposed model to improve the quality of 3D reconstruction under severe fetal motion.

Custom 3D fMRI Registration Template Construction Method Based on Time-Series Fusion

As the brain standard template for medical image registration has only been constructed with an MRI template, there is no three-dimensional fMRI standard template for use, and when the subject’s brain structure is quite different from the standard brain structure, the registration to the standard space will lead to large errors. Registration to an individual space can avoid this problem. However, in the current fMRI registration algorithm based on individual space, the reference image is often selected by researchers or randomly selected fMRI images at a certain time point. This makes the quality of the reference image very dependent on the experience and ability of the researchers and has great contingency. Whether the reference image is appropriate and reasonable affects the rationality and accuracy of the registration results to a great extent. Therefore, a method for constructing a 3D custom fMRI template is proposed. First, the data are preprocessed; second, by taking a group of two-dimensional slices corresponding to the same layer of the brain in three-dimensional fMRI images at multiple time points as image sequences, each group of slice sequences are registered and fused; and finally, a group of fused slices corresponding to different layers of the brain are obtained. In the process of registration, in order to make full use of the correlation information between the sequence data, the feature points of each two slices of adjacent time points in the sequence are matched, and then according to the transformation relationship between the adjacent images, they are recursively forwarded and mapped to the same space. Then, the fused slices are stacked in order to form a three-dimensional customized fMRI template with individual pertinence. Finally, in the classic registration algorithm, the difference in the registration accuracy between using a custom fMRI template and different standard spaces is compared, which proves that using a custom template can improve the registration effect to a certain extent.

RGB-D Point Cloud Registration Based on Salient Object Detection

We propose a robust algorithm for aligning rigid, noisy, and partially overlapping red green blue-depth (RGB-D) point clouds. To address the problems of data degradation and uneven distribution, we offer three strategies to increase the robustness of the iterative closest point (ICP) algorithm. First, we introduce a salient object detection (SOD) method to extract a set of points with significant structural variation in the foreground, which can avoid the unbalanced proportion of foreground and background point sets leading to the local registration. Second, registration algorithms that rely only on structural information for alignment cannot establish the correct correspondences when faced with the point set with no significant change in structure. Therefore, a bidirectional color distance (BCD) is designed to build precise correspondence with bidirectional search and color guidance. Third, the maximum correntropy criterion (MCC) and trimmed strategy are introduced into our algorithm to handle with noise and outliers. We experimentally validate that our algorithm is more robust than previous algorithms on simulated and real-world scene data in most scenarios and achieve a satisfying 3-D reconstruction of indoor scenes.

Deep learning solution for medical image localization and orientation detection

Magnetic Resonance (MR) imaging plays an important role in medical diagnosis and biomedical research. Due to the high in-slice resolution and low through-slice resolution nature of MR imaging, the usefulness of the reconstruction highly depends on the positioning of the slice group. Traditional clinical workflow relies on time-consuming manual adjustment that cannot be easily reproduced. Automation of this task can therefore bring important benefits in terms of accuracy, speed and reproducibility. Current auto-slice-positioning methods rely on automatically detected landmarks to derive the positioning, and previous studies suggest that a large, redundant set of landmarks are required to achieve robust results. However, a costly data curation procedure is needed to generate training labels for those landmarks, and the results can still be highly sensitive to landmark detection errors. More importantly, a set of anatomical landmark locations are not naturally produced during the standard clinical workflow, which makes online learning impossible. To address these limitations, we propose a novel framework for auto-slice-positioning that focuses on localizing the canonical planes within a 3D volume. The proposed framework consists of two major steps. A multi-resolution region proposal network is first used to extract a volume-of-interest, after which a V-net-like segmentation network is applied to segment the orientation planes. Importantly, our algorithm also includes a Performance Measurement Index as an indication of the algorithm's confidence. We evaluate the proposed framework on both knee and shoulder MR scans. Our method outperforms state-of-the-art automatic positioning algorithms in terms of accuracy and robustness.

Sensing and Artificial Intelligent Maternal-Infant Health Care Systems: A Review

Currently, information and communication technology (ICT) allows health institutions to reach disadvantaged groups in rural areas using sensing and artificial intelligence (AI) technologies. Applications of these technologies are even more essential for maternal and infant health, since maternal and infant health is vital for a healthy society. Over the last few years, researchers have delved into sensing and artificially intelligent healthcare systems for maternal and infant health. Sensors are exploited to gauge health parameters, and machine learning techniques are investigated to predict the health conditions of patients to assist medical practitioners. Since these healthcare systems deal with large amounts of data, significant development is also noted in the computing platforms. The relevant literature reports the potential impact of ICT-enabled systems for improving maternal and infant health. This article reviews wearable sensors and AI algorithms based on existing systems designed to predict the risk factors during and after pregnancy for both mothers and infants. This review covers sensors and AI algorithms used in these systems and analyzes each approach with its features, outcomes, and novel aspects in chronological order. It also includes discussion on datasets used and extends challenges as well as future work directions for researchers.

Deep learning-based plane pose regression in obstetric ultrasound

Purpose

In obstetric ultrasound (US) scanning, the learner’s ability to mentally build a three-dimensional (3D) map of the fetus from a two-dimensional (2D) US image represents a major challenge in skill acquisition. We aim to build a US plane localisation system for 3D visualisation, training, and guidance without integrating additional sensors.

Methods

We propose a regression convolutional neural network (CNN) using image features to estimate the six-dimensional pose of arbitrarily oriented US planes relative to the fetal brain centre. The network was trained on synthetic images acquired from phantom 3D US volumes and fine-tuned on real scans. Training data was generated by slicing US volumes into imaging planes in Unity at random coordinates and more densely around the standard transventricular (TV) plane.

Results

With phantom data, the median errors are 0.90 mm/1.17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document}∘ and 0.44 mm/1.21\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document}∘ for random planes and planes close to the TV one, respectively. With real data, using a different fetus with the same gestational age (GA), these errors are 11.84 mm/25.17\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$^\circ $$\end{document}∘. The average inference time is 2.97 ms per plane.

Conclusion

The proposed network reliably localises US planes within the fetal brain in phantom data and successfully generalises pose regression for an unseen fetal brain from a similar GA as in training. Future development will expand the prediction to volumes of the whole fetus and assess its potential for vision-based, freehand US-assisted navigation when acquiring standard fetal planes.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11548-022-02609-z.

Deep Regression Neural Networks for Proportion Judgment

Deep regression models are widely employed to solve computer vision tasks, such as human age or pose estimation, crowd counting, object detection, etc. Another possible area of application, which to our knowledge has not been systematically explored so far, is proportion judgment. As a prerequisite for successful decision making, individuals often have to use proportion judgment strategies, with which they estimate the magnitude of one stimulus relative to another (larger) stimulus. This makes this estimation problem interesting for the application of machine learning techniques. In regard to this, we proposed various deep regression architectures, which we tested on three original datasets of very different origin and composition. This is a novel approach, as the assumption is that the model can learn the concept of proportion without explicitly counting individual objects. With comprehensive experiments, we have demonstrated the effectiveness of the proposed models which can predict proportions on real-life datasets more reliably than human experts, considering the coefficient of determination (>0.95) and the amount of errors (MAE < 2, RMSE < 3). If there is no significant number of errors in determining the ground truth, with an appropriate size of the learning dataset, an additional reduction of MAE to 0.14 can be achieved. The used datasets will be publicly available to serve as reference data sources in similar projects.

Contributions of artificial intelligence reported in Obstetrics and Gynecology journals: a systematic review

Background

The applications of artificial intelligence (AI) processes have grown significantly in all medical disciplines during the last decades. Two main types of AI have been applied in medicine: symbolic AI (eg, knowledge base and ontologies) and nonsymbolic AI (eg, machine learning and artificial neural networks). Consequently, AI has also been applied across most obstetrics and gynecology (OB/GYN) domains, including general obstetrics, gynecology surgery, fetal ultrasound, and assisted reproductive medicine, among others.

Objective

The aim of this study was to provide a systematic review to establish the actual contributions of AI reported in OB/GYN discipline journals.

Methods

The PubMed database was searched for citations indexed with “artificial intelligence” and at least one of the following medical subject heading (MeSH) terms between January 1, 2000, and April 30, 2020: “obstetrics”; “gynecology”; “reproductive techniques, assisted”; or “pregnancy.” All publications in OB/GYN core disciplines journals were considered. The selection of journals was based on disciplines defined in Web of Science. The publications were excluded if no AI process was used in the study. Review, editorial, and commentary articles were also excluded. The study analysis comprised (1) classification of publications into OB/GYN domains, (2) description of AI methods, (3) description of AI algorithms, (4) description of data sets, (5) description of AI contributions, and (6) description of the validation of the AI process.

Results

The PubMed search retrieved 579 citations and 66 publications met the selection criteria. All OB/GYN subdomains were covered: obstetrics (41%, 27/66), gynecology (3%, 2/66), assisted reproductive medicine (33%, 22/66), early pregnancy (2%, 1/66), and fetal medicine (21%, 14/66). Both machine learning methods (39/66) and knowledge base methods (25/66) were represented. Machine learning used imaging, numerical, and clinical data sets. Knowledge base methods used mostly omics data sets. The actual contributions of AI were method/algorithm development (53%, 35/66), hypothesis generation (42%, 28/66), or software development (3%, 2/66). Validation was performed on one data set (86%, 57/66) and no external validation was reported. We observed a general rising trend in publications related to AI in OB/GYN over the last two decades. Most of these publications (82%, 54/66) remain out of the scope of the usual OB/GYN journals.

Conclusions

In OB/GYN discipline journals, mostly preliminary work (eg, proof-of-concept algorithm or method) in AI applied to this discipline is reported and clinical validation remains an unmet prerequisite. Improvement driven by new AI research guidelines is expected. However, these guidelines are covering only a part of AI approaches (nonsymbolic) reported in this review; hence, updates need to be considered.

Artificial intelligence with deep learning in nuclear medicine and radiology

The use of deep learning in medical imaging has increased rapidly over the past few years, finding applications throughout the entire radiology pipeline, from improved scanner performance to automatic disease detection and diagnosis. These advancements have resulted in a wide variety of deep learning approaches being developed, solving unique challenges for various imaging modalities. This paper provides a review on these developments from a technical point of view, categorizing the different methodologies and summarizing their implementation. We provide an introduction to the design of neural networks and their training procedure, after which we take an extended look at their uses in medical imaging. We cover the different sections of the radiology pipeline, highlighting some influential works and discussing the merits and limitations of deep learning approaches compared to other traditional methods. As such, this review is intended to provide a broad yet concise overview for the interested reader, facilitating adoption and interdisciplinary research of deep learning in the field of medical imaging.

Automatic Inter-Frame Patient Motion Correction for Dynamic Cardiac PET Using Deep Learning

Patient motion during dynamic PET imaging can induce errors in myocardial blood flow (MBF) estimation. Motion correction for dynamic cardiac PET is challenging because the rapid tracer kinetics of 82Rb leads to substantial tracer distribution change across different dynamic frames over time, which can cause difficulties for image registration-based motion correction, particularly for early dynamic frames. In this paper, we developed an automatic deep learning-based motion correction (DeepMC) method for dynamic cardiac PET. In this study we focused on the detection and correction of inter-frame rigid translational motion caused by voluntary body movement and pattern change of respiratory motion. A bidirectional-3D LSTM network was developed to fully utilize both local and nonlocal temporal information in the 4D dynamic image data for motion detection. The network was trained and evaluated over motion-free patient scans with simulated motion so that the motion ground-truths are available, where one million samples based on 65 patient scans were used in training, and 600 samples based on 20 patient scans were used in evaluation. The proposed method was also evaluated using additional 10 patient datasets with real motion. We demonstrated that the proposed DeepMC obtained superior performance compared to conventional registration-based methods and other convolutional neural networks (CNN), in terms of motion estimation and MBF quantification accuracy. Once trained, DeepMC is much faster than the registration-based methods and can be easily integrated into the clinical workflow. In the future work, additional investigation is needed to evaluate this approach in a clinical context with realistic patient motion.

Rapid head-pose detection for automated slice prescription of fetal-brain MRI

In fetal-brain MRI, head-pose changes between prescription and acquisition present a challenge to obtaining the standard sagittal, coronal and axial views essential to clinical assessment. As motion limits acquisitions to thick slices that preclude retrospective resampling, technologists repeat ~55-second stack-of-slices scans (HASTE) with incrementally reoriented field of view numerous times, deducing the head pose from previous stacks. To address this inefficient workflow, we propose a robust head-pose detection algorithm using full-uterus scout scans (EPI) which take ~5 seconds to acquire. Our ~2-second procedure automatically locates the fetal brain and eyes, which we derive from maximally stable extremal regions (MSERs). The success rate of the method exceeds 94% in the third trimester, outperforming a trained technologist by up to 20%. The pipeline may be used to automatically orient the anatomical sequence, removing the need to estimate the head pose from 2D views and reducing delays during which motion can occur.

MSU-Net: Multi-Scale U-Net for 2D Medical Image Segmentation

Aiming at the limitation of the convolution kernel with a fixed receptive field and unknown prior to optimal network width in U-Net, multi-scale U-Net (MSU-Net) is proposed by us for medical image segmentation. First, multiple convolution sequence is used to extract more semantic features from the images. Second, the convolution kernel with different receptive fields is used to make features more diverse. The problem of unknown network width is alleviated by efficient integration of convolution kernel with different receptive fields. In addition, the multi-scale block is extended to other variants of the original U-Net to verify its universality. Five different medical image segmentation datasets are used to evaluate MSU-Net. A variety of imaging modalities are included in these datasets, such as electron microscopy, dermoscope, ultrasound, etc. Intersection over Union (IoU) of MSU-Net on each dataset are 0.771, 0.867, 0.708, 0.900, and 0.702, respectively. Experimental results show that MSU-Net achieves the best performance on different datasets. Our implementation is available at https://github.com/CN-zdy/MSU_Net.

Towards deep phenotyping pregnancy: a systematic review on artificial intelligence and machine learning methods to improve pregnancy outcomes

Objective

Development of novel informatics methods focused on improving pregnancy outcomes remains an active area of research. The purpose of this study is to systematically review the ways that artificial intelligence (AI) and machine learning (ML), including deep learning (DL), methodologies can inform patient care during pregnancy and improve outcomes.

Materials and methods

We searched English articles on EMBASE, PubMed and SCOPUS. Search terms included ML, AI, pregnancy and informatics. We included research articles and book chapters, excluding conference papers, editorials and notes.

Results

We identified 127 distinct studies from our queries that were relevant to our topic and included in the review. We found that supervised learning methods were more popular (n = 69) than unsupervised methods (n = 9). Popular methods included support vector machines (n = 30), artificial neural networks (n = 22), regression analysis (n = 17) and random forests (n = 16). Methods such as DL are beginning to gain traction (n = 13). Common areas within the pregnancy domain where AI and ML methods were used the most include prenatal care (e.g. fetal anomalies, placental functioning) (n = 73); perinatal care, birth and delivery (n = 20); and preterm birth (n = 13). Efforts to translate AI into clinical care include clinical decision support systems (n = 24) and mobile health applications (n = 9).

Conclusions

Overall, we found that ML and AI methods are being employed to optimize pregnancy outcomes, including modern DL methods (n = 13). Future research should focus on less-studied pregnancy domain areas, including postnatal and postpartum care (n = 2). Also, more work on clinical adoption of AI methods and the ethical implications of such adoption is needed.

Automatic Fetal Middle Sagittal Plane Detection in Ultrasound Using Generative Adversarial Network

Background and Objective: In the first trimester of pregnancy, fetal growth, and abnormalities can be assessed using the exact middle sagittal plane (MSP) of the fetus. However, the ultrasound (US) image quality and operator experience affect the accuracy. We present an automatic system that enables precise fetal MSP detection from three-dimensional (3D) US and provides an evaluation of its performance using a generative adversarial network (GAN) framework. Method: The neural network is designed as a filter and generates masks to obtain the MSP, learning the features and MSP location in 3D space. Using the proposed image analysis system, a seed point was obtained from 218 first-trimester fetal 3D US volumes using deep learning and the MSP was automatically extracted. Results: The experimental results reveal the feasibility and excellent performance of the proposed approach between the automatically and manually detected MSPs. There was no significant difference between the semi-automatic and automatic systems. Further, the inference time in the automatic system was up to two times faster than the semi-automatic approach. Conclusion: The proposed system offers precise fetal MSP measurements. Therefore, this automatic fetal MSP detection and measurement approach is anticipated to be useful clinically. The proposed system can also be applied to other relevant clinical fields in the future.

Deep Predictive Motion Tracking in Magnetic Resonance Imaging: Application to Fetal Imaging

Fetal magnetic resonance imaging (MRI) is challenged by uncontrollable, large, and irregular fetal movements. It is, therefore, performed through visual monitoring of fetal motion and repeated acquisitions to ensure diagnostic-quality images are acquired. Nevertheless, visual monitoring of fetal motion based on displayed slices, and navigation at the level of stacks-of-slices is inefficient. The current process is highly operator-dependent, increases scanner usage and cost, and significantly increases the length of fetal MRI scans which makes them hard to tolerate for pregnant women. To help build automatic MRI motion tracking and navigation systems to overcome the limitations of the current process and improve fetal imaging, we have developed a new real-time image-based motion tracking method based on deep learning that learns to predict fetal motion directly from acquired images. Our method is based on a recurrent neural network, composed of spatial and temporal encoder-decoders, that infers motion parameters from anatomical features extracted from sequences of acquired slices. We compared our trained network on held-out test sets (including data with different characteristics, e.g. different fetuses scanned at different ages, and motion trajectories recorded from volunteer subjects) with networks designed for estimation as well as methods adopted to make predictions. The results show that our method outperformed alternative techniques, and achieved real-time performance with average errors of 3.5 and 8 degrees for the estimation and prediction tasks, respectively. Our real-time deep predictive motion tracking technique can be used to assess fetal movements, to guide slice acquisitions, and to build navigation systems for fetal MRI.

Deep learning in medical image registration: a review

This paper presents a review of deep learning (DL)-based medical image registration methods. We summarized the latest developments and applications of DL-based registration methods in the medical field. These methods were classified into seven categories according to their methods, functions and popularity. A detailed review of each category was presented, highlighting important contributions and identifying specific challenges. A short assessment was presented following the detailed review of each category to summarize its achievements and future potential. We provided a comprehensive comparison among DL-based methods for lung and brain registration using benchmark datasets. Lastly, we analyzed the statistics of all the cited works from various aspects, revealing the popularity and future trend of DL-based medical image registration.

Anatomy-Guided Convolutional Neural Network for Motion Correction in Fetal Brain MRI

Fetal Magnetic Resonance Imaging (MRI) is challenged by the fetal movements and maternal breathing. Although fast MRI sequences allow artifact free acquisition of individual 2D slices, motion commonly occurs in between slices acquisitions. Motion correction for each slice is thus very important for reconstruction of 3D fetal brain MRI, but is highly operator-dependent and time-consuming. Approaches based on convolutional neural networks (CNNs) have achieved encouraging performance on prediction of 3D motion parameters of arbitrarily oriented 2D slices, which, however, does not capitalize on important brain structural information. To address this problem, we propose a new multi-task learning framework to jointly learn the transformation parameters and tissue segmentation map of each slice, for providing brain anatomical information to guide the mapping from 2D slices to 3D volumetric space in a coarse to fine manner. In the coarse stage, the first network learns the features shared for both regression and segmentation tasks. In the refinement stage, to fully utilize the anatomical information, distance maps constructed based on the coarse segmentation are introduced to the second network. Finally, incorporation of the signed distance maps to guide the regression and segmentation together improves the performance in both tasks. Experimental results indicate that the proposed method achieves superior performance in reducing the motion prediction error and obtaining satisfactory tissue segmentation results simultaneously, compared with state-of-the-art methods.

3D Deep Learning on Medical Images: A Review

The rapid advancements in machine learning, graphics processing technologies and the availability of medical imaging data have led to a rapid increase in the use of deep learning models in the medical domain. This was exacerbated by the rapid advancements in convolutional neural network (CNN) based architectures, which were adopted by the medical imaging community to assist clinicians in disease diagnosis. Since the grand success of AlexNet in 2012, CNNs have been increasingly used in medical image analysis to improve the efficiency of human clinicians. In recent years, three-dimensional (3D) CNNs have been employed for the analysis of medical images. In this paper, we trace the history of how the 3D CNN was developed from its machine learning roots, we provide a brief mathematical description of 3D CNN and provide the preprocessing steps required for medical images before feeding them to 3D CNNs. We review the significant research in the field of 3D medical imaging analysis using 3D CNNs (and its variants) in different medical areas such as classification, segmentation, detection and localization. We conclude by discussing the challenges associated with the use of 3D CNNs in the medical imaging domain (and the use of deep learning models in general) and possible future trends in the field.

Deformable Slice-to-Volume Registration for Motion Correction of Fetal Body and Placenta MRI

In in-utero MRI, motion correction for fetal body and placenta poses a particular challenge due to the presence of local non-rigid transformations of organs caused by bending and stretching. The existing slice-to-volume registration (SVR) reconstruction methods are widely employed for motion correction of fetal brain that undergoes only rigid transformation. However, for reconstruction of fetal body and placenta, rigid registration cannot resolve the issue of misregistrations due to deformable motion, resulting in degradation of features in the reconstructed volume. We propose a Deformable SVR (DSVR), a novel approach for non-rigid motion correction of fetal MRI based on a hierarchical deformable SVR scheme to allow high resolution reconstruction of the fetal body and placenta. Additionally, a robust scheme for structure-based rejection of outliers minimises the impact of registration errors. The improved performance of DSVR in comparison to SVR and patch-to-volume registration (PVR) methods is quantitatively demonstrated in simulated experiments and 20 fetal MRI datasets from 28-31 weeks gestational age (GA) range with varying degree of motion corruption. In addition, we present qualitative evaluation of 100 fetal body cases from 20-34 weeks GA range.

Vision-Based Spacecraft Pose Estimation via a Deep Convolutional Neural Network for Noncooperative Docking Operations

The capture of a target spacecraft by a chaser is an on-orbit docking operation that requires an accurate, reliable, and robust object recognition algorithm. Vision-based guided spacecraft relative motion during close-proximity maneuvers has been consecutively applied using dynamic modeling as a spacecraft on-orbit service system. This research constructs a vision-based pose estimation model that performs image processing via a deep convolutional neural network. The pose estimation model was constructed by repurposing a modified pretrained GoogLeNet model with the available Unreal Engine 4 rendered dataset of the Soyuz spacecraft. In the implementation, the convolutional neural network learns from the data samples to create correlations between the images and the spacecraft’s six degrees-of-freedom parameters. The experiment has compared an exponential-based loss function and a weighted Euclidean-based loss function. Using the weighted Euclidean-based loss function, the implemented pose estimation model achieved moderately high performance with a position accuracy of 92.53 percent and an error of 1.2 m. The in-attitude prediction accuracy can reach 87.93 percent, and the errors in the three Euler angles do not exceed 7.6 degrees. This research can contribute to spacecraft detection and tracking problems. Although the finished vision-based model is specific to the environment of synthetic dataset, the model could be trained further to address actual docking operations in the future.

An Unordered Image Stitching Method Based on Binary Tree and Estimated Overlapping Area

Aiming at the complex computation and time-consuming problem during unordered image stitching, we present a method based on the binary tree and the estimated overlapping areas to stitch images without order in this paper. For image registration, the overlapping areas between input images are estimated, so that the extraction and matching of feature points are only performed in these areas. For image stitching, we build a model of the binary tree to stitch each two matched images without sorting. Compared to traditional methods, our method significantly reduces the computational time of matching irrelevant image pairs and improves the efficiency of image registration and stitching. Moreover, the stitching model of the binary tree proposed in this paper further reduces the distortion of the panorama. Experimental results show that the number of extracted feature points in the estimated overlapping area is approximately 0.3∼0.6 times of that in the entire image by using the same method, which greatly reduces the computational time of feature extraction and matching. Compared to the exhaustive image matching method, our approach only takes about 1/3 of the time to find all matching images.