Liver segmentation and metastases detection in MR images using convolutional neural networks

Challenges and solutions of deep learning-based automated liver segmentation: A systematic review

The liver is one of the vital organs in the body. Precise liver segmentation in medical images is essential for liver disease treatment. The deep learning-based liver segmentation process faces several challenges. This research aims to analyze the challenges of liver segmentation in prior studies and identify the modifications made to network models and other enhancements implemented by researchers to tackle each challenge. In total, 88 articles from Scopus and ScienceDirect databases published between January 2016 and January 2022 have been studied. The liver segmentation challenges are classified into five main categories, each containing some subcategories. For each challenge, the proposed technique to overcome the challenge is investigated. The provided report details the authors, publication years, dataset types, imaging technologies, and evaluation metrics of all references for comparison. Additionally, a summary table outlines the challenges and solutions.

Liver Observation Segmentation on Contrast-Enhanced MRI: SAM and MedSAM Performance in Patients With Probable or Definite Hepatocellular Carcinoma

Purpose: To evaluate factors impacting the Segment Anything Model (SAM) and variant MedSAM performance for segmenting liver observations on contrast-enhanced (CE) magnetic resonance imaging (MRI) in high-risk patients with probable hepatocellular carcinoma (HCC) (LR-4) and definite HCC (LR-5). Methods: A retrospective cohort of liver observations (LR-4/LR-5) on CE-MRI from 97 patients at high-risk for HCC was derived (2013-2018). Using bounding-boxes as prompts under 5-fold cross-validation, segmentation performance was evaluated at the model and liver observation-levels for: (1) model types: SAM versus MedSAM, (2) image sizes: 256 × 256 versus 512 × 512, (3) image channel composition: CE sequences at 3 phases of enhancement independently and combined, (4) liver observation size: >10 mm versus >20 mm, (5) certainty of diagnosis: LR-4 versus LR-5, and (6) contrast-agent type: hepatobiliary versus extracellular. Segmentation performance, quantified using Dice coefficient, were compared using univariate (Wilcoxon signed-rank and t-test) and multivariable analyses (multiple correspondence analysis and subsequent linear modelling). Results: MedSAM trained on 512 × 512 combined CE sequences performed best with mean Dice coefficient 0.68 (95% confidence interval 0.66, 0.69). Overall, all factors except contrast-agent type affected performance, with larger image size resulting in the highest performance improvement (512 × 512: 0.57, 256 × 256: 0.26, P < .001) at the model-level. Contrast-agents affected performance for patients with LR-4 observations using MedSAM-based models (P < .03). Larger observation size, image size, and higher certainty of diagnosis were associated with better segmentation on multivariable analysis. Conclusion: A variety of factors were found to impact SAM/MedSAM performance for segmenting liver observations in patients with probable and definite HCC on CE-MRI. Future models may be optimized by accounting for these factors.

Training robust T1-weighted magnetic resonance imaging liver segmentation models using ensembles of datasets with different contrast protocols and liver disease etiologies

Image segmentation of the liver is an important step in treatment planning for liver cancer. However, manual segmentation at a large scale is not practical, leading to increasing reliance on deep learning models to automatically segment the liver. This manuscript develops a generalizable deep learning model to segment the liver on T1-weighted MR images. In particular, three distinct deep learning architectures (nnUNet, PocketNet, Swin UNETR) were considered using data gathered from six geographically different institutions. A total of 819 T1-weighted MR images were gathered from both public and internal sources. Our experiments compared each architecture's testing performance when trained both intra-institutionally and inter-institutionally. Models trained using nnUNet and its PocketNet variant achieved mean Dice-Sorensen similarity coefficients>0.9 on both intra- and inter-institutional test set data. The performance of these models suggests that nnUNet and PocketNet liver segmentation models trained on a large and diverse collection of T1-weighted MR images would on average achieve good intra-institutional segmentation performance.

Segmentation of liver and liver lesions using deep learning

Segmentation of organs and lesions could be employed for the express purpose of dosimetry in nuclear medicine, assisted image interpretations, and mass image processing studies. Deep leaning created liver and liver lesion segmentation on clinical 3D MRI data has not been fully addressed in previous experiments. To this end, the required data were collected from 128 patients, including their T1w and T2w MRI images, and ground truth labels of the liver and liver lesions were generated. The collection of 110 T1w-T2w MRI image sets was divided, with 94 designated for training and 16 for validation. Furthermore, 18 more datasets were separately allocated for use as hold-out test datasets. The T1w and T2w MRI images were preprocessed into a two-channel format so that they were used as inputs to the deep learning model based on the Isensee 2017 network. To calculate the final Dice coefficient of the network performance on test datasets, the binary average of T1w and T2w predicted images was used. The deep learning model could segment all 18 test cases, with an average Dice coefficient of 88% for the liver and 53% for the liver tumor. Liver segmentation was carried out with rather a high accuracy; this could be achieved for liver dosimetry during systemic or selective radiation therapies as well as for attenuation correction in PET/MRI scanners. Nevertheless, the delineation of liver lesions was not optimal; therefore, tumor detection was not practical by the proposed method on clinical data.

Anatomically Accurate, High-Resolution Modeling of the Human Index Finger Using In Vivo Magnetic Resonance Imaging

Anatomically accurate models of a human finger can be useful in simulating various disorders. In order to have potential clinical value, such models need to include a large number of tissue types, identified by an experienced professional, and should be versatile enough to be readily tailored to specific pathologies. Magnetic resonance images were acquired at ultrahigh magnetic field (7 T) with a radio-frequency coil specially designed for finger imaging. Segmentation was carried out under the supervision of an experienced radiologist to accurately capture various tissue types (TTs). The final segmented model of the human index finger had a spatial resolution of 0.2 mm and included 6,809,600 voxels. In total, 15 TTs were identified: subcutis, Pacinian corpuscle, nerve, vein, artery, tendon, collateral ligament, volar plate, pulley A4, bone, cartilage, synovial cavity, joint capsule, epidermis and dermis. The model was applied to the conditions of arthritic joint, ruptured tendon and variations in the geometry of a finger. High-resolution magnetic resonance images along with careful segmentation proved useful in the construction of an anatomically accurate model of the human index finger. An example illustrating the utility of the model in biomedical applications is shown. As the model includes a number of tissue types, it may present a solid foundation for future simulations of various musculoskeletal disease processes in human joints.

Improving automatic liver tumor segmentation in late-phase MRI using multi-model training and 3D convolutional neural networks

Automatic liver tumor segmentation can facilitate the planning of liver interventions. For diagnosis of hepatocellular carcinoma, dynamic contrast-enhanced MRI (DCE-MRI) can yield a higher sensitivity than contrast-enhanced CT. However, most studies on automatic liver lesion segmentation have focused on CT. In this study, we present a deep learning-based approach for liver tumor segmentation in the late hepatocellular phase of DCE-MRI, using an anisotropic 3D U-Net architecture and a multi-model training strategy. The 3D architecture improves the segmentation performance compared to a previous study using a 2D U-Net (mean Dice 0.70 vs. 0.65). A further significant improvement is achieved by a multi-model training approach (0.74), which is close to the inter-rater agreement (0.78). A qualitative expert rating of the automatically generated contours confirms the benefit of the multi-model training strategy, with 66 % of contours rated as good or very good, compared to only 43 % when performing a single training. The lesion detection performance with a mean F1-score of 0.59 is inferior to human raters (0.76). Overall, this study shows that correctly detected liver lesions in late-phase DCE-MRI data can be automatically segmented with high accuracy, but the detection, in particular of smaller lesions, can still be improved.

A Fast Method for Whole Liver- and Colorectal Liver Metastasis Segmentations from MRI Using 3D FCNN Networks

The liver is the most frequent organ for metastasis from colorectal cancer, one of the most common tumor types with a poor prognosis. Despite reducing surgical planning time and providing better spatial representation, current methods of 3D modeling of patient-specific liver anatomy are extremely time-consuming. The purpose of this study was to develop a deep learning model trained on an in-house dataset of 84 MRI volumes to rapidly provide fully automated whole liver and liver lesions segmentation from volumetric MRI series. A cascade approach was utilized to address the problem of class imbalance. The trained model achieved an average Dice score for whole liver segmentation of 0.944 ± 0.009 and 0.780 ± 0.119 for liver lesion segmentation. Furthermore, applying this method to a not-annotated dataset creates a complete 3D segmentation in less than 6 s per MRI volume, with a mean segmentation Dice score of 0.994 ± 0.003 for the liver and 0.709 ± 0.171 for tumors compared to manual corrections applied after the inference was achieved. Availability and integration of our method in clinical practice may improve diagnosis and treatment planning in patients with colorectal liver metastasis and open new possibilities for research into liver tumors.

Assessment of Liver Function With MRI: Where Do We Stand?

Liver disease and hepatocellular carcinoma (HCC) have become a global health burden. For this reason, the determination of liver function plays a central role in the monitoring of patients with chronic liver disease or HCC. Furthermore, assessment of liver function is important, e.g., before surgery to prevent liver failure after hepatectomy or to monitor the course of treatment. Liver function and disease severity are usually assessed clinically based on clinical symptoms, biopsy, and blood parameters. These are rather static tests that reflect the current state of the liver without considering changes in liver function. With the development of liver-specific contrast agents for MRI, noninvasive dynamic determination of liver function based on signal intensity or using T1 relaxometry has become possible. The advantage of this imaging modality is that it provides additional information about the vascular structure, anatomy, and heterogeneous distribution of liver function. In this review, we summarized and discussed the results published in recent years on this technique. Indeed, recent data show that the T1 reduction rate seems to be the most appropriate value for determining liver function by MRI. Furthermore, attention has been paid to the development of automated tools for image analysis in order to uncover the steps necessary to obtain a complete process flow from image segmentation to image registration to image analysis. In conclusion, the published data show that liver function values obtained from contrast-enhanced MRI images correlate significantly with the global liver function parameters, making it possible to obtain both functional and anatomic information with a single modality.

Artificial intelligence in the diagnosis and management of colorectal cancer liver metastases

Colorectal cancer (CRC) is the third most common malignancy worldwide, with approximately 50% of patients developing colorectal cancer liver metastasis (CRLM) during the follow-up period. Management of CRLM is best achieved via a multidisciplinary approach and the diagnostic and therapeutic decision-making process is complex. In order to optimize patients' survival and quality of life, there are several unsolved challenges which must be overcome. These primarily include a timely diagnosis and the identification of reliable prognostic factors. Furthermore, to allow optimal treatment options, a precision-medicine, personalized approach is required. The widespread digitalization of healthcare generates a vast amount of data and together with accessible high-performance computing, artificial intelligence (AI) technologies can be applied. By increasing diagnostic accuracy, reducing timings and costs, the application of AI could help mitigate the current shortcomings in CRLM management. In this review we explore the available evidence of the possible role of AI in all phases of the CRLM natural history. Radiomics analysis and convolutional neural networks (CNN) which combine computed tomography (CT) images with clinical data have been developed to predict CRLM development in CRC patients. AI models have also proven themselves to perform similarly or better than expert radiologists in detecting CRLM on CT and magnetic resonance scans or identifying them from the noninvasive analysis of patients' exhaled air. The application of AI and machine learning (ML) in diagnosing CRLM has also been extended to histopathological examination in order to rapidly and accurately identify CRLM tissue and its different histopathological growth patterns. ML and CNN have shown good accuracy in predicting response to chemotherapy, early local tumor progression after ablation treatment, and patient survival after surgical treatment or chemotherapy. Despite the initial enthusiasm and the accumulating evidence, AI technologies' role in healthcare and CRLM management is not yet fully established. Its limitations mainly concern safety and the lack of regulation and ethical considerations. AI is unlikely to fully replace any human role but could be actively integrated to facilitate physicians in their everyday practice. Moving towards a personalized and evidence-based patient approach and management, further larger, prospective and rigorous studies evaluating AI technologies in patients at risk or affected by CRLM are needed.

Improved performance and consistency of deep learning 3D liver segmentation with heterogeneous cancer stages in magnetic resonance imaging

PURPOSE

Accurate liver segmentation is key for volumetry assessment to guide treatment decisions. Moreover, it is an important pre-processing step for cancer detection algorithms. Liver segmentation can be especially challenging in patients with cancer-related tissue changes and shape deformation. The aim of this study was to assess the ability of state-of-the-art deep learning 3D liver segmentation algorithms to generalize across all different Barcelona Clinic Liver Cancer (BCLC) liver cancer stages.

METHODS

This retrospective study, included patients from an institutional database that had arterial-phase T1-weighted magnetic resonance images with corresponding manual liver segmentations. The data was split into 70/15/15% for training/validation/testing each proportionally equal across BCLC stages. Two 3D convolutional neural networks were trained using identical U-net-derived architectures with equal sized training datasets: one spanning all BCLC stages ("All-Stage-Net": AS-Net), and one limited to early and intermediate BCLC stages ("Early-Intermediate-Stage-Net": EIS-Net). Segmentation accuracy was evaluated by the Dice Similarity Coefficient (DSC) on a dataset spanning all BCLC stages and a Wilcoxon signed-rank test was used for pairwise comparisons.

RESULTS

219 subjects met the inclusion criteria (170 males, 49 females, 62.8±9.1 years) from all BCLC stages. Both networks were trained using 129 subjects: AS-Net training comprised 19, 74, 18, 8, and 10 BCLC 0, A, B, C, and D patients, respectively; EIS-Net training comprised 21, 86, and 22 BCLC 0, A, and B patients, respectively. DSCs (mean±SD) were 0.954±0.018 and 0.946±0.032 for AS-Net and EIS-Net (p<0.001), respectively. The AS-Net 0.956±0.014 significantly outperformed the EIS-Net 0.941±0.038 on advanced BCLC stages (p<0.001) and yielded similarly good segmentation performance on early and intermediate stages (AS-Net: 0.952±0.021; EIS-Net: 0.949±0.027; p = 0.107).

CONCLUSION

To ensure robust segmentation performance across cancer stages that is independent of liver shape deformation and tumor burden, it is critical to train deep learning models on heterogeneous imaging data spanning all BCLC stages.

Emerging artificial intelligence applications in liver magnetic resonance imaging

Chronic liver diseases (CLDs) are becoming increasingly more prevalent in modern society. The use of imaging techniques for early detection, such as magnetic resonance imaging (MRI), is crucial in reducing the impact of these diseases on healthcare systems. Artificial intelligence (AI) algorithms have been shown over the past decade to excel at image-based analysis tasks such as detection and segmentation. When applied to liver MRI, they have the potential to improve clinical decision making, and increase throughput by automating analyses. With Liver diseases becoming more prevalent in society, the need to implement these techniques to utilize liver MRI to its full potential, is paramount. In this review, we report on the current methods and applications of AI methods in liver MRI, with a focus on machine learning and deep learning methods. We assess four main themes of segmentation, classification, image synthesis and artefact detection, and their respective potential in liver MRI and the wider clinic. We provide a brief explanation of some of the algorithms used and explore the current challenges affecting the field. Though there are many hurdles to overcome in implementing AI methods in the clinic, we conclude that AI methods have the potential to positively aid healthcare professionals for years to come.

An automated liver segmentation in liver iron concentration map using fuzzy c-means clustering combined with anatomical landmark data

Background

To estimate median liver iron concentration (LIC) calculated from magnetic resonance imaging, excluded vessels of the liver parenchyma region were defined manually. Previous works proposed the automated method for excluding vessels from the liver region. However, only user-defined liver region remained a manual process. Therefore, this work aimed to develop an automated liver region segmentation technique to automate the whole process of median LIC calculation.

Methods

553 MR examinations from 471 thalassemia major patients were used in this study. LIC maps (in mg/g dry weight) were calculated and used as the input of segmentation procedures. Anatomical landmark data were detected and used to restrict ROI. After that, the liver region was segmented using fuzzy c-means clustering and reduced segmentation errors by morphological processes. According to the clinical application, erosion with a suitable size of the structuring element was applied to reduce the segmented liver region to avoid uncertainty around the edge of the liver. The segmentation results were evaluated by comparing with manual segmentation performed by a board-certified radiologist.

Results

The proposed method was able to produce a good grade output in approximately 81% of all data. Approximately 11% of all data required an easy modification step. The rest of the output, approximately 8%, was an unsuccessful grade and required manual intervention by a user. For the evaluation matrices, percent dice similarity coefficient (%DSC) was in the range 86–92, percent Jaccard index (%JC) was 78–86, and Hausdorff distance (H) was 14–28 mm, respectively. In this study, percent false positive (%FP) and percent false negative (%FN) were applied to evaluate under- and over-segmentation that other evaluation matrices could not handle. The average of operation times could be reduced from 10 s per case using traditional method, to 1.5 s per case using our proposed method.

Conclusion

The experimental results showed that the proposed method provided an effective automated liver segmentation technique, which can be applied clinically for automated median LIC calculation in thalassemia major patients.

Multichannel three-dimensional fully convolutional residual network-based focal liver lesion detection and classification in Gd-EOB-DTPA-enhanced MRI

PURPOSE

Gadolinium ethoxybenzyl diethylenetriamine pentaacetic acid (Gd-EOB-DTPA)-enhanced magnetic resonance imaging (MRI) has high diagnostic accuracy in the detection of liver lesions. There is a demand for computer-aided detection/diagnosis software for Gd-EOB-DTPA-enhanced MRI. We propose a deep learning-based method using one three-dimensional fully convolutional residual network (3D FC-ResNet) for liver segmentation and another 3D FC-ResNet for simultaneous detection and classification of a focal liver lesion in Gd-EOB-DTPA-enhanced MRI.

METHODS

We prepared a five-phase (unenhanced, arterial, portal venous, equilibrium, and hepatobiliary phases) series as the input image sets and labeled focal liver lesion (hepatocellular carcinoma, metastasis, hemangiomas, cysts, and scars) images as the output image sets. We used 100 cases to train our model, 42 cases to determine the hyperparameters of our model, and 42 cases to evaluate our model. We evaluated our model by free-response receiver operating characteristic curve analysis and using a confusion matrix.

RESULTS

Our model simultaneously detected and classified focal liver lesions. In the test cases, the detection accuracy for whole focal liver lesions had a true-positive ratio of 0.6 at an average of 25 false positives per case. The classification accuracy was 0.790.

CONCLUSION

We proposed the simultaneous detection and classification of a focal liver lesion in Gd-EOB-DTPA-enhanced MRI using multichannel 3D FC-ResNet. Our results indicated simultaneous detection and classification are possible using a single network. It is necessary to further improve detection sensitivity to help radiologists.

Artificial intelligence performance in detecting tumor metastasis from medical radiology imaging: A systematic review and meta-analysis

BACKGROUND

Early diagnosis of tumor metastasis is crucial for clinical treatment. Artificial intelligence (AI) has shown great promise in the field of medicine. We therefore aimed to evaluate the diagnostic accuracy of AI algorithms in detecting tumor metastasis using medical radiology imaging.

METHODS

We searched PubMed and Web of Science for studies published from January 1, 1997, to January 30, 2020. Studies evaluating an AI model for the diagnosis of tumor metastasis from medical images were included. We excluded studies that used histopathology images or medical wave-form data and those focused on the region segmentation of interest. Studies providing enough information to construct contingency tables were included in a meta-analysis.

FINDINGS

We identified 2620 studies, of which 69 were included. Among them, 34 studies were included in a meta-analysis with a pooled sensitivity of 82% (95% CI 79-84%), specificity of 84% (82-87%) and AUC of 0·90 (0·87-0·92). Analysis for different AI algorithms showed a pooled sensitivity of 87% (83-90%) for machine learning and 86% (82-89%) for deep learning, and a pooled specificity of 89% (82-93%) for machine learning, and 87% (82-91%) for deep learning.

INTERPRETATION

AI algorithms may be used for the diagnosis of tumor metastasis using medical radiology imaging with equivalent or even better performance to health-care professionals, in terms of sensitivity and specificity. At the same time, rigorous reporting standards with external validation and comparison to health-care professionals are urgently needed for AI application in the medical field.

FUNDING

College students' innovative entrepreneurial training plan program .

Patient-specific fine-tuning of convolutional neural networks for follow-up lesion quantification

Purpose: Convolutional neural network (CNN) methods have been proposed to quantify lesions in medical imaging. Commonly, more than one imaging examination is available for a patient, but the serial information in these images often remains unused. CNN-based methods have the potential to extract valuable information from previously acquired imaging to better quantify lesions on current imaging of the same patient.

Approach: A pretrained CNN can be updated with a patient’s previously acquired imaging: patient-specific fine-tuning (FT). In this work, we studied the improvement in performance of lesion quantification methods on magnetic resonance images after FT compared to a pretrained base CNN. We applied the method to two different approaches: the detection of liver metastases and the segmentation of brain white matter hyperintensities (WMH).

Results: The patient-specific fine-tuned CNN has a better performance than the base CNN. For the liver metastases, the median true positive rate increases from 0.67 to 0.85. For the WMH segmentation, the mean Dice similarity coefficient increases from 0.82 to 0.87.

Conclusions: We showed that patient-specific FT has the potential to improve the lesion quantification performance of general CNNs by exploiting a patient’s previously acquired imaging.

Channel width optimized neural networks for liver and vessel segmentation in liver iron quantification

INTRODUCTION

MRI T2* relaxometry protocols are often used for Liver Iron Quantification in patients with hemochromatosis. Several methods exist to semi-automatically segment parenchyma and exclude vessels for this calculation.

PURPOSE

To determine if inclusion of multiple echoes inputs to Convolutional Neural Networks (CNN) improves automated liver and vessel segmentation in MRI T2* relaxometry protocols and to determine if the resultant segmentations agree with manual segmentations for liver iron quantification analysis.

METHODS

Multi echo Gradient Recalled Echo (GRE) MRI sequence for T2* relaxometry was performed for 79 exams on 31 patients with hemochromatosis for iron quantification analysis. 275 axial liver slices were manually segmented as ground truth masks. A batch normalized U-Net with variable input width to incorporate multiple echoes is used for segmentation, using DICE as the accuracy metric. ANOVA is used to evaluate significance of channel width changes in segmentation accuracy. Linear regression is used to model the relationship of channel width on segmentation accuracy. Liver segmentations are applied to relaxometry data to calculate liver T2* yielding liver iron concentration(LIC) derived from literature based calibration curves. Manual and CNN based LIC values are compared with Pearson correlation. Bland altman plots are used to visualize differences between manual and CNN based LIC values.

RESULTS

Performance metrics are tested on 55 hold out slices. Linear regression indicates that there is a monotonic increase of DICE with increasing channel depth (p = 0.001) with a slope of 3.61e-3. ANOVA indicates a significant increase segmentation accuracy over single channel starting at 3 channels. Incorporation of all channels results in an average DICE of 0.86, an average increase of 0.07 over single channel. The calculated LIC from CNN segmented livers agrees well with manual segmentation (R = 0.998, slope = 0.914, p«0.001), with an average absolute difference 0.27 ± 0.99 mg Fe/g or 1.34 ± 4.3%.

CONCLUSION

More input echoes yields higher model accuracy until the noise floor. Echos beyond the first three echo times in GRE based T2* relaxometry do not contribute significant information for segmentation of liver for LIC calculation. Deep learning models with three channel width allow for generalization of model to protocols of more than three echoes, effectively a universal requirement for relaxometry. Deep learning segmentations achieve a good accuracy compared with manual segmentations with minimal preprocessing. Liver iron values calculated from hand segmented liver and Neural network segmented liver were not statistically different from each other.