A review of biomechanically informed breast image registration

MUsculo-Skeleton-Aware (MUSA) deep learning for anatomically guided head-and-neck CT deformable registration

Deep-learning-based deformable image registration (DL-DIR) has demonstrated improved accuracy compared to time-consuming non-DL methods across various anatomical sites. However, DL-DIR is still challenging in heterogeneous tissue regions with large deformation. In fact, several state-of-the-art DL-DIR methods fail to capture the large, anatomically plausible deformation when tested on head-and-neck computed tomography (CT) images. These results allude to the possibility that such complex head-and-neck deformation may be beyond the capacity of a single network structure or a homogeneous smoothness regularization. To address the challenge of combined multi-scale musculoskeletal motion and soft tissue deformation in the head-and-neck region, we propose a MUsculo-Skeleton-Aware (MUSA) framework to anatomically guide DL-DIR by leveraging the explicit multiresolution strategy and the inhomogeneous deformation constraints between the bony structures and soft tissue. The proposed method decomposes the complex deformation into a bulk posture change and residual fine deformation. It can accommodate both inter- and intra- subject registration. Our results show that the MUSA framework can consistently improve registration accuracy and, more importantly, the plausibility of deformation for various network architectures. The code will be publicly available at https://github.com/HengjieLiu/DIR-MUSA.

Ultra-dense Motion Capture: An exploratory full-automatic approach for dense tracking of breast motion in 4D

Understanding the dynamic deformation pattern and biomechanical properties of breasts is crucial in various fields, including designing ergonomic bras and customized prostheses, as well as in clinical practice. Previous studies have recorded and analyzed the dynamic behaviors of the breast surface using 4D scanning, which provides a sequence of 3D meshes during movement with high spatial and temporal resolutions. However, these studies are limited by the lack of robust and automated data processing methods which result in limited data coverage or error-prone analysis results. To address this issue, we identify revealing inter-frame dense correspondence as the core challenge towards conducting reliable and consistent analysis of the 4D scanning data. We proposed a fully-automatic approach named Ulta-dense Motion Capture (UdMC) using Thin-plate Spline (TPS) to augment the sparse landmarks recorded via motion capture (MoCap) as initial dense correspondence and then rectified it with a sophisticated post-alignment scheme. Two downstream tasks are demonstrated to validate its applicability: virtual landmark tracking and deformation intensity analysis. For evaluation, a dynamic 4D human breast anthropometric dataset DynaBreastLite was constructed. The results show that our approach can robustly capture the dynamic deformation characteristics of the breast surfaces, significantly outperforms baselines adapted from previous works in terms of accuracy, consistency, and efficiency. For 10 fps dataset, average error of 0.25 cm on control-landmarks and 0.33 cm on non-control (arbitrary) landmarks were achieved, with 17-70 times faster computation time. Evaluation was also carried out on 60 fps and 120 fps datasets, with consistent and large performance gaining being observed. The proposed method may contribute to advancing research in breast anthropometry, biomechanics, and ergonomics by enabling more accurate tracking of the breast surface deformation patterns and dynamic characteristics.

Estimation of the biomechanical mammographic deformation of the breast using machine learning models

BACKGROUND

A typical problem in the registration of MRI and X-ray mammography is the nonlinear deformation applied to the breast during mammography. We have developed a method for virtual deformation of the breast using a biomechanical model automatically constructed from MRI. The virtual deformation is applied in two steps: unloaded state estimation and compression simulation. The finite element method is used to solve the deformation process. However, the extensive computational cost prevents its usage in clinical routine.

METHODS

We propose three machine learning models to overcome this problem: an extremely randomized tree (first model), extreme gradient boosting (second model), and deep learning-based bidirectional long short-term memory with an attention layer (third model) to predict the deformation of a biomechanical model. We evaluated our methods with 516 breasts with realistic compression ratios up to 76%.

FINDINGS

We first applied one-fold validation, in which the second and third models performed better than the first model. We then applied ten-fold validation. For the unloaded state estimation, the median RMSE for the second and third models is 0.8 mm and 1.2 mm, respectively. For the compression, the median RMSE is 3.4 mm for both models. We evaluated correlations between model accuracy and characteristics of the clinical datasets such as compression ratio, breast volume, and tissue types.

INTERPRETATION

Using the proposed models, we achieved accurate results comparable to the finite element model, with a speedup of factor 240 using the extreme gradient boosting model. These proposed models can replace the finite element model simulation, enabling clinically relevant real-time application.

Improvement of semantic segmentation through transfer learning of multi-class regions with convolutional neural networks on supine and prone breast MRI images

Semantic segmentation of breast and surrounding tissues in supine and prone breast magnetic resonance imaging (MRI) is required for various kinds of computer-assisted diagnoses for surgical applications. Variability of breast shape in supine and prone poses along with various MRI artifacts makes it difficult to determine robust breast and surrounding tissue segmentation. Therefore, we evaluated semantic segmentation with transfer learning of convolutional neural networks to create robust breast segmentation in supine breast MRI without considering supine or prone positions. Total 29 patients with T1-weighted contrast-enhanced images were collected at Asan Medical Center and two types of breast MRI were performed in the prone position and the supine position. The four classes, including lungs and heart, muscles and bones, parenchyma with cancer, and skin and fat, were manually drawn by an expert. Semantic segmentation on breast MRI scans with supine, prone, transferred from prone to supine, and pooled supine and prone MRI were trained and compared using 2D U-Net, 3D U-Net, 2D nnU-Net and 3D nnU-Net. The best performance was 2D models with transfer learning. Our results showed excellent performance and could be used for clinical purposes such as breast registration and computer-aided diagnosis.

Incorporating heterogeneity and anisotropy for surgical applications in breast deformation modeling

BACKGROUND

Simulating soft-tissue breast deformations is of interest for many applications including image fusion, longitudinal registration, and image-guided surgery. For the surgical use case, positional changes cause breast deformations that compromise the use of preoperative imaging to inform tumor excision. Even when acquiring imaging in the supine position, which better reflects surgical presentation, deformations still occur due to arm motion and orientation changes. A biomechanical modeling approach to simulate supine breast deformations for surgical applications must be both accurate and compatible with the clinical workflow.

METHODS

A supine MR breast imaging dataset from n = 11 healthy volunteers was used to simulate surgical deformations by acquiring images in arm-down and arm-up positions. Three linear-elastic modeling approaches with varying levels of complexity were used to predict deformations caused by this arm motion: a homogeneous isotropic model, a heterogeneous isotropic model, and a heterogeneous anisotropic model using a transverse-isotropic constitutive model.

FINDINGS

The average target registration errors for subsurface anatomical features were 5.4 ± 1.5 mm for the homogeneous isotropic model, 5.3 ± 1.5 mm for the heterogeneous isotropic model, and 4.7 ± 1.4 mm for the heterogeneous anisotropic model. A statistically significant improvement in target registration error was observed between the heterogeneous anisotropic model and both the homogeneous and the heterogeneous isotropic models (P < 0.01).

INTERPRETATION

While a model that fully incorporates all constitutive complexities of anatomical structure likely achieves the best accuracy, a computationally tractable heterogeneous anisotropic model provided significant improvement and may be applicable for image-guided breast surgeries.

Image registration method using representative feature detection and iterative coherent spatial mapping for infrared medical images with flat regions

In the registration of medical images, nonrigid registration targets, images with large displacement caused by different postures of the human body, and frequent variations in image intensity due to physiological phenomena are substantial problems that make medical images less suitable for intensity-based image registration modes. These problems also greatly increase the difficulty and complexity of feature detection and matching for feature-based image registration modes. This research introduces an automatic image registration algorithm for infrared medical images that offers the following benefits: effective detection of feature points in flat regions (cold patterns) that appear due to changes in the human body’s thermal patterns, improved mismatch removal through coherent spatial mapping for improved feature point matching, and large-displacement optical flow for optimal transformation. This method was compared with various classical gold standard image registration methods to evaluate its performance. The models were compared for the three key steps of the registration process—feature detection, feature point matching, and image transformation—and the results are presented visually and quantitatively. The results demonstrate that the proposed method outperforms existing methods in all tasks, including in terms of the features detected, uniformity of feature points, matching accuracy, and control point sparsity, and achieves optimal image transformation. The performance of the proposed method with four common image types was also evaluated, and the results verify that the proposed method has a high degree of stability and can effectively register medical images under a variety of conditions.

A Medical Image Registration Method Based on Progressive Images

Background

Medical image registration is an essential task for medical image analysis in various applications. In this work, we develop a coarse-to-fine medical image registration method based on progressive images and SURF algorithm (PI-SURF) for higher registration accuracy.

Methods

As a first step, the reference image and the floating image are fused to generate multiple progressive images. Thereafter, the floating image and progressive image are registered to get the coarse registration result based on the SURF algorithm. For further improvement, the coarse registration result and the reference image are registered to perform fine image registration. The appropriate progressive image has been investigated by experiments. The mutual information (MI), normal mutual information (NMI), normalized correlation coefficient (NCC), and mean square difference (MSD) similarity metrics are used to demonstrate the potential of the PI-SURF method.

Results

For the unimodal and multimodal registration, the PI-SURF method achieves the best results compared with the mutual information method, Demons method, Demons+B-spline method, and SURF method. The MI, NMI, and NCC of PI-SURF are improved by 15.5%, 1.31%, and 7.3%, respectively, while MSD decreased by 13.2% for the multimodal registration compared with the optimal result of the state-of-the-art methods.

Conclusions

The extensive experiments show that the proposed PI-SURF method achieves higher quality of registration.

Breast Tumor Diagnosis Using Finite-Element Modeling Based on Clinical in vivo Elastographic Data

OBJECTIVES

This study exploited finite-element modeling (FEM) to simulate breast tissue multicompression during ultrasound elastography to classify breast tumors based on their nonlinear biomechanical properties.

METHODS

Numeric simulations were first calculated by using 3-dimensional (3D) virtual models with an assumed tumor's geometric dimensions but with actual material properties to test and validate the FEM. Further numeric simulations were used to construct 3D models based on in vivo experimental data to verify our models. The models were designed for each individual in vivo case, emphasizing the geometry, position, and biomechanical properties of the breast tissue. At different compression levels, tissue strains were analyzed between the tumors and the background normal tissues to explore their nonlinearity and classify the tumor type. Tumor classification parameters were deduced by using a power-law relationship between the applied compressive forces and strain differences.

RESULTS

Classification parameters were compared between benign and malignant tumors, for which they were found to be statistically significant in classifying the tumor types (P < .05) by both the validation and verification of FEM. We compared the classification parameters between the in vivo and FEM classifications, for which they were found to be strongly correlated (R = 0.875; P < .001), with no statistical differences between their outcomes (P = .909).

CONCLUSIONS

Good agreement between the model outcomes and the in vivo diagnostics was reported. The implemented models were validated and verified. The introduced 3D modeling method may augment elastographic methods to preliminary classify breast tumors at an early stage.

Deformable Mapping Method to Relate Lesions in Dedicated Breast CT Images to Those in Automated Breast Ultrasound and Digital Breast Tomosynthesis Images

This work demonstrates the potential for using a deformable mapping method to register lesions between dedicated breast computed tomography (bCT) and both automated breast ultrasound (ABUS) and digital breast tomosynthesis (DBT) images (craniocaudal [CC] and mediolateral oblique [MLO] views). Two multi-modality breast phantoms with external fiducial markers attached were imaged by the three modalities. The DBT MLO view was excluded for the second phantom. The automated deformable mapping algorithm uses biomechanical modeling to determine corresponding lesions based on distances between their centers of mass (d_COM) in the deformed bCT model and the reference model (DBT or ABUS). For bCT to ABUS, the mean d_COM was 5.2 ± 2.6 mm. For bCT to DBT (CC), the mean d_COM was 5.1 ± 2.4 mm. For bCT to DBT (MLO), the mean d_COM was 4.7 ± 2.5 mm. This application could help improve a radiologist's efficiency and accuracy in breast lesion characterization, using multiple imaging modalities.

Finite Element Breast Simulation for Sonography Image Registration

In case of female breast cancer, a breast conserving excision is often necessary. For this purpose, information from multiple medical imaging techniques have to be combined. Sonography imaging is essential for dense breast tissue and the only medical imaging technique available during surgery. During sonography of the outer breast quadrants the woman is usually in contralateral posterior oblique position, being in supine orientation while holding her ipsilateral arm over the head. Thus, these images cannot be directly registered with MRI or mammography images because these imaging technologies are performed in other patient positions with hands on the side of the body. Thus, we present a novel Finite Element approach how to enable a sonography image registration by showing the first time how to transfer the supine position with the arm straight on side into a supine position with the ipsilateral arm over the head which can be used to include information from MRI or mammography images. This approach is shown and validated with 3D scanner breast surface data as proof of concept. When comparing the simulation result with a 3D surface scan in supine orientation with the arm over the head, a mean surface distance error of 1.57 mm is achieved.

3D digital breast cancer models with multimodal fusion algorithms

Breast cancer image fusion consists of registering and visualizing different sets of a patient synchronized torso and radiological images into a 3D model. Breast spatial interpretation and visualization by the treating physician can be augmented with a patient-specific digital breast model that integrates radiological images. But the absence of a ground truth for a good correlation between surface and radiological information has impaired the development of potential clinical applications.

A new image acquisition protocol was designed to acquire breast Magnetic Resonance Imaging (MRI) and 3D surface scan data with surface markers on the patient’s breasts and torso. A patient-specific digital breast model integrating the real breast torso and the tumor location was created and validated with a MRI/3D surface scan fusion algorithm in 16 breast cancer patients.

This protocol was used to quantify breast shape differences between different modalities, and to measure the target registration error of several variants of the MRI/3D scan fusion algorithm. The fusion of single breasts without the biomechanical model of pose transformation had acceptable registration errors and accurate tumor locations. The performance of the fusion algorithm was not affected by breast volume. Further research and virtual clinical interfaces could lead to fast integration of this fusion technology into clinical practice.

•

MRI/3D surface scan fusion algorithm to create 3D breast cancer models.

•

A replicable clinical validation protocol for MRI/3D surface scan fusion algorithms.

•

Anthropometric study that quantifies breast deformations by area in MRI and 3D scans.

An automated computational biomechanics workflow for improving breast cancer diagnosis and treatment

Clinicians face many challenges when diagnosing and treating breast cancer. These challenges include interpreting and co-locating information between different medical imaging modalities that are used to identify tumours and predicting where these tumours move to during different treatment procedures. We have developed a novel automated breast image analysis workflow that integrates state-of-the-art image processing and machine learning techniques, personalized three-dimensional biomechanical modelling and population-based statistical analysis to assist clinicians during breast cancer detection and treatment procedures. This paper summarizes our recent research to address the various technical and implementation challenges associated with creating a fully automated system. The workflow is applied to predict the repositioning of tumours from the prone position, where diagnostic magnetic resonance imaging is performed, to the supine position where treatment procedures are performed. We discuss our recent advances towards addressing challenges in identifying the mechanical properties of the breast and evaluating the accuracy of the biomechanical models. We also describe our progress in implementing a prototype of this workflow in clinical practice. Clinical adoption of these state-of-the-art modelling techniques has significant potential for reducing the number of misdiagnosed breast cancers, while also helping to improve the treatment of patients.

Position-based modeling of lesion displacement in ultrasound-guided breast biopsy

PURPOSE

Although ultrasound (US) images represent the most popular modality for guiding breast biopsy, malignant regions are often missed by sonography, thus preventing accurate lesion localization which is essential for a successful procedure. Biomechanical models can support the localization of suspicious areas identified on a preoperative image during US scanning since they are able to account for anatomical deformations resulting from US probe pressure. We propose a deformation model which relies on position-based dynamics (PBD) approach to predict the displacement of internal targets induced by probe interaction during US acquisition.

METHODS

The PBD implementation available in NVIDIA FleX is exploited to create an anatomical model capable of deforming online. Simulation parameters are initialized on a calibration phantom under different levels of probe-induced deformations; then, they are fine-tuned by minimizing the localization error of a US-visible landmark of a realistic breast phantom. The updated model is used to estimate the displacement of other internal lesions due to probe-tissue interaction.

RESULTS

The localization error obtained when applying the PBD model remains below 11 mm for all the tumors even for input displacements in the order of 30 mm. This proposed method obtains results aligned with FE models with faster computational performance, suitable for real-time applications. In addition, it outperforms rigid model used to track lesion position in US-guided breast biopsies, at least halving the localization error for all the displacement ranges considered.

CONCLUSION

Position-based dynamics approach has proved to be successful in modeling breast tissue deformations during US acquisition. Its stability, accuracy and real-time performance make such model suitable for tracking lesions displacement during US-guided breast biopsy.

A novel finite element model-based navigation system-supported workflow for breast tumor excision

In the case of female breast cancer, a breast-conserving excision is often desirable. This surgery is based on preoperatively gathered MRI, mammography, and sonography images. These images are recorded in multiple patient positions, e. g., 2D mammography images in standing position with a compressed breast and 3D MRI images in prone position. In contrast, the surgery happens in supine or beach chair position. Due to these different perspectives and the flexible, thus challenging, breast tissue, the excision puts high demands on the physician. Therefore, this publication presents a novel eight-step excision support workflow that can be used to include information captured preoperatively through medical imaging based on a finite element (FE) model. In addition, an indoor positioning system is integrated in the workflow in order to track surgical devices and the sonography transducer during surgery. The preoperative part of the navigation system-supported workflow is outlined exemplarily based on first experimental results including 3D scans of a patient in different patient positions and her MRI images. Graphical Abstract Finite Element model based navigation system supported workflow for breast tumor excision is based on eight steps and allows inclusion of information from medical images recorded in multiple patient positions.

A biomechanical breast model evaluated with respect to MRI data collected in three different positions

BACKGROUND

Mammography is a specific type of breast imaging that uses low-dose X-rays to detect cancer in early stage. During the exam, the women breast is compressed between two plates in order to even out the breast thickness and to spread out the soft tissues. This technique improves exam quality but can be uncomfortable for the patient. The perceived discomfort can be assessed by the means of a breast biomechanical model. Alternative breast compression techniques may be computationally investigated trough finite elements simulations.

METHODS

The aim of this work is to develop and evaluate a new biomechanical Finite Element (FE) breast model. The complex breast anatomy is considered including adipose and glandular tissues, muscle, skin, suspensory ligaments and pectoral fascias. Material hyper-elasticity is modeled using the Neo-Hookean material models. The stress-free breast geometry and subject-specific constitutive models are derived using tissues deformations measurements from MR images.

FINDINGS

The breast geometry in three breast configurations were computed using the breast stress-free geometry together with the estimated set of equivalent Young's modulus (E_breast^r = 0.3 kPa, E_breast^l = 0.2 kPa, E_skin = 4 kPa, E_fascia = 120 kPa). The Hausdorff distance between estimated and measured breast geometries for prone, supine and supine tilted configurations is equal to 2.17 mm, 1.72 mm and 5.90 mm respectively.

INTERPRETATION

A subject-specific breast model allows a better characterization of breast mechanics. However, the model presents some limitations when estimating the supine tilted breast configuration. The results show clearly the difficulties to characterize soft tissues mechanics at large strain ranges with Neo-Hookean material models.

Automatic and fast segmentation of breast region-of-interest (ROI) and density in MRIs

Accurate segmentation of the breast region of interest (BROI) and breast density (BD) is a significant challenge during the analysis of breast MR images. Most of the existing methods for breast segmentation are semi-automatic and limited in their ability to achieve accurate results. This is because of difficulties in removing landmarks from noisy magnetic resonance images (MRI) due to similar intensity levels and the close connection to BROI. This study proposes an innovative, fully automatic and fast segmentation approach to identify and remove landmarks such as the heart and pectoral muscles. The BROI segmentation is carried out with a framework consisting of three major steps. Firstly, we use adaptive wiener filtering and k-means clustering to minimize the influence of noises, preserve edges and remove unwanted artefacts. The second step systematically excludes the heart area by utilizing active contour based level sets where initial contour points are determined by the maximum entropy thresholding and convolution method. Finally, a pectoral muscle is removed by using morphological operations and local adaptive thresholding on MR images. Prior to the elimination of the pectoral muscle, the MR image is sub divided into three sections: left, right, and central based on the geometrical information. Subsequently, a BD segmentation is achieved with 4 level fuzzy c-means (FCM) thresholding on the denoised BROI segmentation. The proposed method is validated using the 1350 breast images from 15 female subjects. The pixel-based quantitative analysis showed excellent segmentation results when compared with manually drawn BROI and BD. Furthermore, the presented results in terms of evaluation matrices: Acc, Sp, AUC, MR, P, Se and DSC demonstrate the high quality of segmentations using the proposed method. The average computational time for the segmentation of BROI and BD is 1 minute and 50 seconds.

Deformable mapping technique to correlate lesions in digital breast tomosynthesis and automated breast ultrasound images

PURPOSE

To develop a deformable mapping technique to match corresponding lesions between digital breast tomosynthesis (DBT) and automated breast ultrasound (ABUS) images.

METHODS

External fiducial markers were attached to the surface of two CIRS multi-modality compressible breast phantoms (A and B) containing multiple simulated lesions. Both phantoms were imaged with DBT (upright positioning with cranial-caudal compression) and ABUS (supine positioning with anterior-to-chest wall compression). The lesions and markers were manually segmented by three different readers. Reader segmentation similarity and reader reproducibility were assessed using Dice similarity coefficients (DSC) and distances between centers of mass (d_COM ). For deformable mapping between the modalities each reader's segmented dataset was processed with an automated deformable mapping algorithm as follows: First, Morfeus, a finite element (FE) based multi-organ deformable image registration platform, converted segmentations into triangular surface meshes. Second, Altair HyperMesh, a FE pre-processor, created base FE models for the ABUS and DBT data sets. All deformation is performed on the DBT image data; the ABUS image sets remain fixed throughout the process. Deformation was performed on the external skin contour (DBT image set) to match the external skin contour on the ABUS set, and the locations of the external markers were used to morph the skin contours to be within a user-defined distance. Third, the base DBT-FE model was deformed with the FE analysis solver, Optistruct. Deformed DBT lesions were correlated with matching lesions in the base ABUS FE model. Performance (lesion correlation) was assessed with d_COM for all corresponding lesions and lesion overlap. Analysis was performed to determine the minimum number of external fiducial markers needed to create the desired correlation and the improvement of correlation with the use of external markers.

RESULTS

Average DSC for reader similarity ranged from 0.88 to 0.91 (ABUS) and 0.57 to 0.83 (DBT). Corresponding d_COM ranged from 0.20 to 0.36 mm (ABUS) and 0.11 to 1.16 mm (DBT). Lesion correlation is maximized when all corresponding markers are within a maximum distance of 5 mm. For deformable mapping of phantom A, without the use of external markers, only two of six correlated lesions showed overlap with an average lesion d_COM of 6.8 ± 2.8 mm. With use of three external fiducial markers, five of six lesions overlapped and average d_COM improved to 4.9 ± 2.4 mm. For deformable mapping of Phantom B without external markers analysis, four lesions were correlated of seven with overlap between only one of seven lesions, and an average lesion d_COM of 9.7 ± 3.5 mm. With three external markers, all seven possible lesions were correlated with overlap between four of seven lesions. The average d_COM was 8.5 ± 4.0 mm.

CONCLUSION

This work demonstrates the potential for a deformable mapping technique to relate corresponding lesions in DBT and ABUS images by showing improved lesion correspondence and reduced lesion registration errors with the use of external fiducial markers. The technique should improve radiologists' characterization of breast lesions which can reduce patient callbacks, misdiagnoses and unnecessary biopsies.

Enhanced Frequency Difference of Tumor inside Vibrated Tissue by a Compression Cylinder

Breast cancer diagnosis has been mostly accomplished by imaging technologies. These methods have the great advantages of detecting the presence and location of breast cancer. However, it's difficult to distinguish between a benign and malignant tumor in a deep position because both tumor types look similar. In this paper, we vibrated the tissue including tumor from skin with a compression cylinder to analyze the frequency difference for distinguishing the tissue type. Before distinguishing a benign and malignant tumor, it's necessary to validate to distinguish between normal tissue and tumor. The objective is to validate the feasibility of using a compression cylinder that emphasizes the differences in frequency between normal tissue and tumor. In two experiments, we measured the displacement on the surface of a breast phantom vibrated by an impulse hammer. We compared the frequency difference with and without a cylinder. We also studied the frequency changes in the relationship between tumor and cylinder position. We found a 5.0 Hz difference in compliance between normal tissue and the simulated tumor using a compression cylinder. The difference in frequency correlated negatively with distance from the simulated tumor to a compression cylinder. We concluded that a compression cylinder would enhance the frequency difference between normal tissue and a simulated tumor with appropriate configuration.

A Regression Model for Predicting Shape Deformation after Breast Conserving Surgery

Breast cancer treatments can have a negative impact on breast aesthetics, in case when surgery is intended to intersect tumor. For many years mastectomy was the only surgical option, but more recently breast conserving surgery (BCS) has been promoted as a liable alternative to treat cancer while preserving most part of the breast. However, there is still a significant number of BCS intervened patients who are unpleasant with the result of the treatment, which leads to self-image issues and emotional overloads. Surgeons recognize the value of a tool to predict the breast shape after BCS to facilitate surgeon/patient communication and allow more educated decisions; however, no such tool is available that is suited for clinical usage. These tools could serve as a way of visually sensing the aesthetic consequences of the treatment. In this research, it is intended to propose a methodology for predict the deformation after BCS by using machine learning techniques. Nonetheless, there is no appropriate dataset containing breast data before and after surgery in order to train a learning model. Therefore, an in-house semi-synthetic dataset is proposed to fulfill the requirement of this research. Using the proposed dataset, several learning methodologies were investigated, and promising outcomes are obtained.

Ultrasound Imaging Technologies for Breast Cancer Detection and Management: A Review

Ultrasound imaging is a commonly used modality for breast cancer detection and diagnosis. In this review, we summarize ultrasound imaging technologies and their clinical applications for the management of breast cancer patients. The technologies include ultrasound elastography, contrast-enhanced ultrasound, 3-D ultrasound, automatic breast ultrasound and computer-aided detection of breast ultrasound. We summarize the study results seen in the literature and discuss their future directions. We also provide a review of ultrasound-guided, breast biopsy and the fusion of ultrasound with other imaging modalities, especially magnetic resonance imaging (MRI). For comparison, we also discuss the diagnostic performance of mammography, MRI, positron emission tomography and computed tomography for breast cancer diagnosis at the end of this review. New ultrasound imaging techniques, ultrasound-guided biopsy and the fusion of ultrasound with other modalities provide important tools for the management of breast patients.

Review of quantitative multiscale imaging of breast cancer

Breast cancer is the most common cancer among women worldwide and ranks second in terms of overall cancer deaths. One of the difficulties associated with treating breast cancer is that it is a heterogeneous disease with variations in benign and pathologic tissue composition, which contributes to disease development, progression, and treatment response. Many of these phenotypes are uncharacterized and their presence is difficult to detect, in part due to the sparsity of methods to correlate information between the cellular microscale and the whole-breast macroscale. Quantitative multiscale imaging of the breast is an emerging field concerned with the development of imaging technology that can characterize anatomic, functional, and molecular information across different resolutions and fields of view. It involves a diverse collection of imaging modalities, which touch large sections of the breast imaging research community. Prospective studies have shown promising results, but there are several challenges, ranging from basic physics and engineering to data processing and quantification, that must be met to bring the field to maturity. This paper presents some of the challenges that investigators face, reviews currently used multiscale imaging methods for preclinical imaging, and discusses the potential of these methods for clinical breast imaging.

Towards an in-plane methodology to track breast lesions using mammograms and patient-specific finite-element simulations

In breast cancer screening or diagnosis, it is usual to combine different images in order to locate a lesion as accurately as possible. These images are generated using a single or several imaging techniques. As x-ray-based mammography is widely used, a breast lesion is located in the same plane of the image (mammogram), but tracking it across mammograms corresponding to different views is a challenging task for medical physicians. Accordingly, simulation tools and methodologies that use patient-specific numerical models can facilitate the task of fusing information from different images. Additionally, these tools need to be as straightforward as possible to facilitate their translation to the clinical area. This paper presents a patient-specific, finite-element-based and semi-automated simulation methodology to track breast lesions across mammograms. A realistic three-dimensional computer model of a patient's breast was generated from magnetic resonance imaging to simulate mammographic compressions in cranio-caudal (CC, head-to-toe) and medio-lateral oblique (MLO, shoulder-to-opposite hip) directions. For each compression being simulated, a virtual mammogram was obtained and posteriorly superimposed to the corresponding real mammogram, by sharing the nipple as a common feature. Two-dimensional rigid-body transformations were applied, and the error distance measured between the centroids of the tumors previously located on each image was 3.84 mm and 2.41 mm for CC and MLO compression, respectively. Considering that the scope of this work is to conceive a methodology translatable to clinical practice, the results indicate that it could be helpful in supporting the tracking of breast lesions.

A finite element-based machine learning approach for modeling the mechanical behavior of the breast tissues under compression in real-time

This work presents a data-driven method to simulate, in real-time, the biomechanical behavior of the breast tissues in some image-guided interventions such as biopsies or radiotherapy dose delivery as well as to speed up multimodal registration algorithms. Ten real breasts were used for this work. Their deformation due to the displacement of two compression plates was simulated off-line using the finite element (FE) method. Three machine learning models were trained with the data from those simulations. Then, they were used to predict in real-time the deformation of the breast tissues during the compression. The models were a decision tree and two tree-based ensemble methods (extremely randomized trees and random forest). Two different experimental setups were designed to validate and study the performance of these models under different conditions. The mean 3D Euclidean distance between nodes predicted by the models and those extracted from the FE simulations was calculated to assess the performance of the models in the validation set. The experiments proved that extremely randomized trees performed better than the other two models. The mean error committed by the three models in the prediction of the nodal displacements was under 2 mm, a threshold usually set for clinical applications. The time needed for breast compression prediction is sufficiently short to allow its use in real-time (<0.2 s).

The compressed breast during mammography and breast tomosynthesis: in vivo shape characterization and modeling

To characterize and develop a patient-based 3D model of the compressed breast undergoing mammography and breast tomosynthesis. During this IRB-approved, HIPAA-compliant study, 50 women were recruited to undergo 3D breast surface imaging with structured light (SL) during breast compression, along with simultaneous acquisition of a tomosynthesis image. A pair of SL systems were used to acquire 3D surface images by projecting 24 different patterns onto the compressed breast and capturing their reflection off the breast surface in approximately 12-16 s. The 3D surface was characterized and modeled via principal component analysis. The resulting surface model was combined with a previously developed 2D model of projected compressed breast shapes to generate a full 3D model. Data from ten patients were discarded due to technical problems during image acquisition. The maximum breast thickness (found at the chest-wall) had an average value of 56 mm, and decreased 13% towards the nipple (breast tilt angle of 5.2°). The portion of the breast not in contact with the compression paddle or the support table extended on average 17 mm, 18% of the chest-wall to nipple distance. The outermost point along the breast surface lies below the midline of the total thickness. A complete 3D model of compressed breast shapes was created and implemented as a software application available for download, capable of generating new random realistic 3D shapes of breasts undergoing compression. Accurate characterization and modeling of the breast curvature and shape was achieved and will be used for various image processing and clinical tasks.

A hybrid patient-specific biomechanical model based image registration method for the motion estimation of lungs

This paper presents a new hybrid biomechanical model-based non-rigid image registration method for lung motion estimation. In the proposed method, a patient-specific biomechanical modelling process captures major physically realistic deformations with explicit physical modelling of sliding motion, whilst a subsequent non-rigid image registration process compensates for small residuals. The proposed algorithm was evaluated with 10 4D CT datasets of lung cancer patients. The target registration error (TRE), defined as the Euclidean distance of landmark pairs, was significantly lower with the proposed method (TRE = 1.37 mm) than with biomechanical modelling (TRE = 3.81 mm) and intensity-based image registration without specific considerations for sliding motion (TRE = 4.57 mm). The proposed method achieved a comparable accuracy as several recently developed intensity-based registration algorithms with sliding handling on the same datasets. A detailed comparison on the distributions of TREs with three non-rigid intensity-based algorithms showed that the proposed method performed especially well on estimating the displacement field of lung surface regions (mean TRE = 1.33 mm, maximum TRE = 5.3 mm). The effects of biomechanical model parameters (such as Poisson's ratio, friction and tissue heterogeneity) on displacement estimation were investigated. The potential of the algorithm in optimising biomechanical models of lungs through analysing the pattern of displacement compensation from the image registration process has also been demonstrated.

The feasibility of manual parameter tuning for deformable breast MR image registration from a multi-objective optimization perspective

Deformable image registration is typically formulated as an optimization problem involving a linearly weighted combination of terms that correspond to objectives of interest (e.g. similarity, deformation magnitude). The weights, along with multiple other parameters, need to be manually tuned for each application, a task currently addressed mainly via trial-and-error approaches. Such approaches can only be successful if there is a sensible interplay between parameters, objectives, and desired registration outcome. This, however, is not well established. To study this interplay, we use multi-objective optimization, where multiple solutions exist that represent the optimal trade-offs between the objectives, forming a so-called Pareto front. Here, we focus on weight tuning. To study the space a user has to navigate during manual weight tuning, we randomly sample multiple linear combinations. To understand how these combinations relate to desirability of registration outcome, we associate with each outcome a mean target registration error (TRE) based on expert-defined anatomical landmarks. Further, we employ a multi-objective evolutionary algorithm that optimizes the weight combinations, yielding a Pareto front of solutions, which can be directly navigated by the user. To study how the complexity of manual weight tuning changes depending on the registration problem, we consider an easy problem, prone-to-prone breast MR image registration, and a hard problem, prone-to-supine breast MR image registration. Lastly, we investigate how guidance information as an additional objective influences the prone-to-supine registration outcome. Results show that the interplay between weights, objectives, and registration outcome makes manual weight tuning feasible for the prone-to-prone problem, but very challenging for the harder prone-to-supine problem. Here, patient-specific, multi-objective weight optimization is needed, obtaining a mean TRE of 13.6 mm without guidance information reduced to 7.3 mm with guidance information, but also providing a Pareto front that exhibits an intuitively sensible interplay between weights, objectives, and registration outcome, allowing outcome selection.

Improvements of an objective model of compressed breasts undergoing mammography: Generation and characterization of breast shapes

PURPOSE

To develop a set of accurate 2D models of compressed breasts undergoing mammography or breast tomosynthesis, based on objective analysis, to accurately characterize mammograms with few linearly independent parameters, and to generate novel clinically realistic paired cranio-caudal (CC) and medio-lateral oblique (MLO) views of the breast.

METHODS

We seek to improve on an existing model of compressed breasts by overcoming detector size bias, removing the nipple and non-mammary tissue, pairing the CC and MLO views from a single breast, and incorporating the pectoralis major muscle contour into the model. The outer breast shapes in 931 paired CC and MLO mammograms were automatically detected with an in-house developed segmentation algorithm. From these shapes three generic models (CC-only, MLO-only, and joint CC/MLO) with linearly independent components were constructed via principal component analysis (PCA). The ability of the models to represent mammograms not used for PCA was tested via leave-one-out cross-validation, by measuring the average distance error (ADE).

RESULTS

The individual models based on six components were found to depict breast shapes with accuracy (mean ADE-CC = 0.81 mm, ADE-MLO = 1.64 mm, ADE-Pectoralis = 1.61 mm), outperforming the joint CC/MLO model (P ≤ 0.001). The joint model based on 12 principal components contains 99.5% of the total variance of the data, and can be used to generate new clinically realistic paired CC and MLO breast shapes. This is achieved by generating random sets of 12 principal components, following the Gaussian distributions of the histograms of each component, which were obtained from the component values determined from the images in the mammography database used.

CONCLUSION

Our joint CC/MLO model can successfully generate paired CC and MLO view shapes of the same simulated breast, while the individual models can be used to represent with high accuracy clinical acquired mammograms with a small set of parameters. This is the first step toward objective 3D compressed breast models, useful for dosimetry and scatter correction research, among other applications.

From clinical imaging and computational models to personalised medicine and image guided interventions

This short paper describes the development of the UCL Centre for Medical Image Computing (CMIC) from 2006 to 2016, together with reference to historical developments of the Computational Imaging sciences Group (CISG) at Guy's Hospital. Key early work in automated image registration led to developments in image guided surgery and improved cancer diagnosis and therapy. The work is illustrated with examples from neurosurgery, laparoscopic liver and gastric surgery, diagnosis and treatment of prostate cancer and breast cancer, and image guided radiotherapy for lung cancer.