Multi-task deep learning network to predict future macrovascular invasion in hepatocellular carcinoma

Hepatic-associated vascular morphological assessment to predict overt hepatic encephalopathy before TIPS: a multicenter study

BACKGROUND

To provide patients the chance of accepting curative transjugular intrahepatic portosystemic shunt (TIPS) rather than palliative treatments for portal hypertension-related variceal bleeding and ascites, we aimed to assess hepatic-associated vascular morphological change to improve the predictive accuracy of overt hepatic encephalopathy (HE) risks.

METHODS

In this multicenter study, 621 patients undergoing TIPS were subdivided into training (413 cases from 3 hospitals) and external validation datasets (208 cases from another 3 hospitals). In addition to traditional clinical factors, we assessed hepatic-associated vascular morphological changes using maximum diameter (including absolute and ratio values). Three predictive models (clinical, hepatic-associated vascular, and combined) were constructed using logistic regression. Their discrimination and calibration were compared to test the necessity of hepatic-associated vascular assessment and identify the optimal model. Furthermore, to verify the improved performance of ModelC-V, we compared it with four previous models, both in discrimination and calibration.

RESULTS

The combined model outperformed the clinical and hepatic-associated vascular models (training: 0.814, 0.754, 0.727; validation: 0.781, 0.679, 0.776; p < 0.050) and had the best calibration. Compared to previous models, ModelC-V showed superior performance in discrimination. The high-, middle-, and low-risk populations displayed significantly different overt HE incidence (p < 0.001). Despite the limited ability of pre-TIPS ammonia to predict overt HE risks, the combined model displayed a satisfactory ability to predict overt HE risks, both in the low- and high-ammonia subgroups.

CONCLUSION

Hepatic-associated vascular assessment improved the predictive accuracy of overt HE, ensuring curative chances by TIPS for suitable patients and providing insights for cirrhosis-related studies.

A Deep Learning Model Combining Multimodal Factors to Predict the Overall Survival of Transarterial Chemoembolization

Background

To develop and validate an overall survival (OS) prediction model for transarterial chemoembolization (TACE).

Methods

In this retrospective study, 301 patients with hepatocellular carcinoma (HCC) who received TACE from 2012 to 2015 were collected. The residual network was used to extract prognostic information from CT images, which was then combined with the clinical factors adjusted by COX regression to predict survival using a modified deep learning model (DLOPCombin). The DLOPCombin model was compared with the residual network model (DLOPCTR), multiple COX regression model (DLOPCox), Radiomic model (Radiomic), and clinical model.

Results

In the validation cohort, DLOPCombin shows the highest TD AUC of all cohorts, which compared with Radiomic (TD AUC: 0.96vs 0.63) and clinical model (TD AUC: 0.96 vs 0.62) model. DLOPCombin showed significant difference in C index compared with DLOPCTR and DLOPCox models (P < 0.05). Moreover, the DLOPCombin showed good calibration and overall net benefit. Patients with DLOPCombin model score ≤ 0.902 had better OS (33 months vs 15.5 months, P < 0.0001).

Conclusion

The deep learning model can effectively predict the patients' overall survival of TACE.

Whole-orbit radiomics: machine learning-based multi- and fused- region radiomics signatures for intravenous glucocorticoid response prediction in thyroid eye disease

BACKGROUND

Radiomics analysis of orbital magnetic resonance imaging (MRI) shows preliminary potential for intravenous glucocorticoid (IVGC) response prediction of thyroid eye disease (TED). The current region of interest segmentation contains only a single organ as extraocular muscles (EOMs). It would be of great value to consider all orbital soft tissues and construct a better prediction model.

METHODS

In this retrospective study, we enrolled 127 patients with TED that received 4·5 g IVGC therapy and had complete follow-up examinations. Pre-treatment orbital T2-weighted imaging (T2WI) was acquired for all subjects. Using multi-organ segmentation (MOS) strategy, we contoured the EOMs, lacrimal gland (LG), orbital fat (OF), and optic nerve (ON), respectively. By fused-organ segmentation (FOS), we contoured the aforementioned structures as a cohesive unit. Whole-orbit radiomics (WOR) models consisting of a multi-regional radiomics (MRR) model and a fused-regional radiomics (FRR) model were further constructed using six machine learning (ML) algorithms.

RESULTS

The support vector machine (SVM) classifier had the best performance on the MRR model (AUC = 0·961). The MRR model outperformed the single-regional radiomics (SRR) models (highest AUC = 0·766, XGBoost on EOMs, or LR on OF) and conventional semiquantitative imaging model (highest AUC = 0·760, NaiveBayes). The application of different ML algorithms for the comparison between the MRR model and the FRR model (highest AUC = 0·916, LR) led to different conclusions.

CONCLUSIONS

The WOR models achieved a satisfactory result in IVGC response prediction of TED. It would be beneficial to include more orbital structures and implement ML algorithms while constructing radiomics models. The selection of separate or overall segmentation of orbital soft tissues has not yet attained its final optimal result.

Multimodal deep learning for liver cancer applications: a scoping review

Background

Hepatocellular carcinoma is a malignant neoplasm of the liver and a leading cause of cancer-related deaths worldwide. The multimodal data combines several modalities, such as medical images, clinical parameters, and electronic health record (EHR) reports, from diverse sources to accomplish the diagnosis of liver cancer. The introduction of deep learning models with multimodal data can enhance the diagnosis and improve physicians' decision-making for cancer patients.

Objective

This scoping review explores the use of multimodal deep learning techniques (i.e., combining medical images and EHR data) in diagnosing and prognosis of hepatocellular carcinoma (HCC) and cholangiocarcinoma (CCA).

Methodology

A comprehensive literature search was conducted in six databases along with forward and backward references list checking of the included studies. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for scoping review guidelines were followed for the study selection process. The data was extracted and synthesized from the included studies through thematic analysis.

Results

Ten studies were included in this review. These studies utilized multimodal deep learning to predict and diagnose hepatocellular carcinoma (HCC), but no studies examined cholangiocarcinoma (CCA). Four imaging modalities (CT, MRI, WSI, and DSA) and 51 unique EHR records (clinical parameters and biomarkers) were used in these studies. The most frequently used medical imaging modalities were CT scans followed by MRI, whereas the most common EHR parameters used were age, gender, alpha-fetoprotein AFP, albumin, coagulation factors, and bilirubin. Ten unique deep-learning techniques were applied to both EHR modalities and imaging modalities for two main purposes, prediction and diagnosis.

Conclusion

The use of multimodal data and deep learning techniques can help in the diagnosis and prediction of HCC. However, there is a limited number of works and available datasets for liver cancer, thus limiting the overall advancements of AI for liver cancer applications. Hence, more research should be undertaken to explore further the potential of multimodal deep learning in liver cancer applications.

Multitask Learning with Convolutional Neural Networks and Vision Transformers Can Improve Outcome Prediction for Head and Neck Cancer Patients

Neural-network-based outcome predictions may enable further treatment personalization of patients with head and neck cancer. The development of neural networks can prove challenging when a limited number of cases is available. Therefore, we investigated whether multitask learning strategies, implemented through the simultaneous optimization of two distinct outcome objectives (multi-outcome) and combined with a tumor segmentation task, can lead to improved performance of convolutional neural networks (CNNs) and vision transformers (ViTs). Model training was conducted on two distinct multicenter datasets for the endpoints loco-regional control (LRC) and progression-free survival (PFS), respectively. The first dataset consisted of pre-treatment computed tomography (CT) imaging for 290 patients and the second dataset contained combined positron emission tomography (PET)/CT data of 224 patients. Discriminative performance was assessed by the concordance index (C-index). Risk stratification was evaluated using log-rank tests. Across both datasets, CNN and ViT model ensembles achieved similar results. Multitask approaches showed favorable performance in most investigations. Multi-outcome CNN models trained with segmentation loss were identified as the optimal strategy across cohorts. On the PET/CT dataset, an ensemble of multi-outcome CNNs trained with segmentation loss achieved the best discrimination (C-index: 0.29, 95% confidence interval (CI): 0.22-0.36) and successfully stratified patients into groups with low and high risk of disease progression (p=0.003). On the CT dataset, ensembles of multi-outcome CNNs and of single-outcome ViTs trained with segmentation loss performed best (C-index: 0.26 and 0.26, CI: 0.18-0.34 and 0.18-0.35, respectively), both with significant risk stratification for LRC in independent validation (p=0.002 and p=0.011). Further validation of the developed multitask-learning models is planned based on a prospective validation study, which has recently completed recruitment.

Development and validation of convolutional neural network-based model to predict the risk of sentinel or non-sentinel lymph node metastasis in patients with breast cancer: a machine learning study

Background

For patients with sentinel lymph node (SLN) metastasis and low risk of residual non-SLN (NSLN) metastasis, axillary lymph node (ALN) dissection could lead to overtreatment. This study aimed to develop and validate an automated preoperative deep learning-based tool to predict the risk of SLN and NSLN metastasis in patients with breast cancer (BC) using dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) images.

Methods

In this machine learning study, we retrospectively enrolled 988 women with BC from three hospitals in Zhejiang, China between June 1, 2013 to December 31, 2021, June 1, 2017 to December 31, 2021, and January 1, 2019 to June 30, 2023, respectively. Patients were divided into the training set (n = 519), internal validation set (n = 129), external test set 1 (n = 296), and external test set 2 (n = 44). A convolutional neural network (CNN) model was proposed to predict the SLN and NSLN metastasis and was compared with clinical and radiomics approaches. The performance of different models to detect ALN metastasis was measured by the area under the curve (AUC), accuracy, sensitivity, and specificity. This study is registered at ChiCTR, ChiCTR2300070740.

Findings

For SLN prediction, the top-performing model (i.e., the CNN algorithm) achieved encouraging predictive performance in the internal validation set (AUC 0.899, 95% CI, 0.887-0.911), external test set 1 (AUC 0.885, 95% CI, 0.867-0.903), and external test set 2 (AUC 0.768, 95% CI, 0.738-0.798). For NSLN prediction, the CNN-based model also exhibited satisfactory performance in the internal validation set (AUC 0.800, 95% CI, 0.783-0.817), external test set 1 (AUC 0.763, 95% CI, 0.732-0.794), and external test set 2 (AUC 0.728, 95% CI, 0.719-0.738). Based on the subgroup analysis, the CNN model performed well in tumour group smaller than 2.0 cm, with the AUC of 0.801 (internal validation set) and 0.823 (external test set 1). Of 469 patients with BC, the false positive rate of SLN prediction declined from 77.9% to 32.9% using CNN model.

Interpretation

The CNN model can predict the SLN status of any detectable lesion size and condition of NSLN in patients with BC. Overall, the CNN model, employing ready DCE-MRI images could serve as a potential technique to assist surgeons in the personalized axillary treatment of in patients with BC non-invasively.

Funding

National Key Research and Development projects intergovernmental cooperation in science and technology of China, National Natural Science Foundation of China, Natural Science Foundation of Zhejiang Province, and Zhejiang Medical and Health Science Project.

Data-Driven Assisted Decision Making for Surgical Procedure of Hepatocellular Carcinoma Resection and Prognostic Prediction: Development and Validation of Machine Learning Models

Background: Currently, surgical decisions for hepatocellular carcinoma (HCC) resection are difficult and not sufficiently personalized. We aimed to develop and validate data driven prediction models to assist surgeons in selecting the optimal surgical procedure for patients. Methods: Retrospective data from 361 HCC patients who underwent radical resection in two institutions were included. End-to-end deep learning models were built to automatically segment lesions from the arterial phase (AP) of preoperative dynamic contrast enhanced magnetic resonance imaging (DCE-MRI). Clinical baseline characteristics and radiomic features were rigorously screened. The effectiveness of radiomic features and radiomic-clinical features was also compared. Three ensemble learning models were proposed to perform the surgical procedure decision and the overall survival (OS) and recurrence-free survival (RFS) predictions after taking different solutions, respectively. Results: SegFormer performed best in terms of automatic segmentation, achieving a Mean Intersection over Union (mIoU) of 0.8860. The five-fold cross-validation results showed that inputting radiomic-clinical features outperformed using only radiomic features. The proposed models all outperformed the other mainstream ensemble models. On the external test set, the area under the receiver operating characteristic curve (AUC) of the proposed decision model was 0.7731, and the performance of the prognostic prediction models was also relatively excellent. The application web server based on automatic lesion segmentation was deployed and is available online. Conclusions: In this study, we developed and externally validated the surgical decision-making procedures and prognostic prediction models for HCC for the first time, and the results demonstrated relatively accurate predictions and strong generalizations, which are expected to help clinicians optimize surgical procedures.

Radiomics: A radiological evidence-based artificial intelligence technique to facilitate personalized precision medicine in hepatocellular carcinoma

The high postoperative recurrence rates in hepatocellular carcinoma (HCC) remain a major hurdle in its management. Appropriate staging and treatment selection may alleviate the extent of fatal recurrence. However, effective methods to preoperatively evaluate pathophysiologic and molecular characteristics of HCC are lacking. Imaging plays a central role in HCC diagnosis and stratification due to the non-invasive diagnostic criteria. Vast and crucial information is hidden within image data. Other than providing a morphological sketch for lesion diagnosis, imaging could provide new insights to describe the pathophysiological and genetic landscape of HCC. Radiomics aims to facilitate diagnosis and prognosis of HCC using artificial intelligence techniques to harness the immense information contained in medical images. Radiomics produces a set of archetypal and robust imaging features that are correlated to key pathological or molecular biomarkers to preoperatively risk-stratify HCC patients. Inferred with outcome data, comprehensive combination of radiomic, clinical and/or multi-omics data could also improve direct prediction of response to treatment and prognosis. The evolution of radiomics is changing our understanding of personalized precision medicine in HCC management. Herein, we review the key techniques and clinical applications in HCC radiomics and discuss current limitations and future opportunities to improve clinical decision making.

Technological Advancements in Interventional Oncology

Interventional radiology, and particularly interventional oncology, represents one of the medical subspecialties in which technological advancements and innovations play an utterly fundamental role. Artificial intelligence, consisting of big data analysis and feature extrapolation through computational algorithms for disease diagnosis and treatment response evaluation, is nowadays playing an increasingly important role in various healthcare fields and applications, from diagnosis to treatment response prediction. One of the fields which greatly benefits from artificial intelligence is interventional oncology. In addition, digital health, consisting of practical technological applications, can assist healthcare practitioners in their daily activities. This review aims to cover the most useful, established, and interesting artificial intelligence and digital health innovations and updates, to help physicians become more and more involved in their use in clinical practice, particularly in the field of interventional oncology.

Deep learning based radiomics for gastrointestinal cancer diagnosis and treatment: A minireview

Gastrointestinal (GI) cancers are the major cause of cancer-related mortality globally. Medical imaging is an important auxiliary means for the diagnosis, assessment and prognostic prediction of GI cancers. Radiomics is an emerging and effective technology to decipher the encoded information within medical images, and traditional machine learning is the most commonly used tool. Recent advances in deep learning technology have further promoted the development of radiomics. In the field of GI cancer, although there are several surveys on radiomics, there is no specific review on the application of deep-learning-based radiomics (DLR). In this review, a search was conducted on Web of Science, PubMed, and Google Scholar with an emphasis on the application of DLR for GI cancers, including esophageal, gastric, liver, pancreatic, and colorectal cancers. Besides, the challenges and recommendations based on the findings of the review are comprehensively analyzed to advance DLR.

Development and validation of a deep learning model for survival prognosis of transcatheter arterial chemoembolization in patients with intermediate-stage hepatocellular carcinoma

PURPOSE

We aimed to develop a deep learning-based approach to evaluate both time-to-progression (TTP) and overall survival (OS) prognosis of transcatheter arterial chemoembolization (TACE) in treatment-naïve patients with intermediate-stage hepatocellular carcinoma (HCC) and compare the approach's performance with those of radiomics and clinical models.

METHODS

EfficientNetV2 was used to build a prognosis model for treatment-naïve patients with HCC. Data of 414 intermediate-stage HCC patients from one participant center were collected to construct the training and validation datasets (70%:30%) for TTP prognosis, while data of 129 intermediate-stage HCC patients from another participant center were collected as the test dataset for both TTP and OS prognosis. Three radiomics and three clinical models were then constructed for comparison.

RESULTS

Patients with EfficientNetV2-based model score ≤ 0.5 had better TTP than those with higher scores (hazard ratio [HR]: 0.32, 95%CI: 0.22-0.46, P < 0.0001; HR: 0.28, 95%CI: 0.20-0.41, P < 0.0001; and HR: 0.55, 95%CI: 0.36-0.88, P = 0.005 in the training, validation, and test datasets, respectively). Patients with model score ≤ 0.5 had better OS (38.8 months vs 20.9 months, HR: 0.58, 95%CI: 0.37-0.90, P = 0.008). Compared with the radiomics (intra-tumoral and peri-tumoral) and three clinical models, the EfficientNetV2-based model showed better survival prognosis for TACE (P < 0.05) in the test dataset.

CONCLUSIONS

The EfficientNetV2-based model enables assessment of both TTP and OS prognosis of TACE in treatment-naïve, intermediate-stage HCC. Patients with lower scores will benefit from TACE. The model can potentially be used by clinicians to improve decision making regarding TACE treatment choices.

Early prediction of treatment response to neoadjuvant chemotherapy based on longitudinal ultrasound images of HER2-positive breast cancer patients by Siamese multi-task network: A multicentre, retrospective cohort study

BACKGROUND

Early prediction of treatment response to neoadjuvant chemotherapy (NACT) in patients with human epidermal growth factor receptor 2 (HER2)-positive breast cancer can facilitate timely adjustment of treatment regimens. We aimed to develop and validate a Siamese multi-task network (SMTN) for predicting pathological complete response (pCR) based on longitudinal ultrasound images at the early stage of NACT.

METHODS

In this multicentre, retrospective cohort study, a total of 393 patients with biopsy-proven HER2-positive breast cancer were retrospectively enrolled from three hospitals in china between December 16, 2013 and March 05, 2021, and allocated into a training cohort and two external validation cohorts. Patients receiving full cycles of NACT and with surgical pathological results available were eligible for inclusion. The key exclusion criteria were missing ultrasound images and/or clinicopathological characteristics. The proposed SMTN consists of two subnetworks that could be joined at multiple layers, which allowed for the integration of multi-scale features and extraction of dynamic information from longitudinal ultrasound images before and after the first /second cycles of NACT. We constructed the clinical model as a baseline using multivariable logistic regression analysis. Then the performance of SMTN was evaluated and compared with the clinical model.

FINDINGS

The training cohort, comprising 215 patients, were selected from Yunnan Cancer Hospital. The two independent external validation cohorts, comprising 95 and 83 patients, were selected from Guangdong Provincial People's Hospital, and Shanxi Cancer Hospital, respectively. The SMTN yielded an area under the receiver operating characteristic curve (AUC) values of 0.986 (95% CI: 0.977-0.995), 0.902 (95%CI: 0.856-0.948), and 0.957 (95%CI: 0.924-0.990) in the training cohort and two external validation cohorts, respectively, which were significantly higher than that those of the clinical model (AUC: 0.524-0.588, P _all < 0.05). The AUCs values of the SMTN within the anti-HER2 therapy subgroups were 0.833-0.972 in the two external validation cohorts. Moreover, 272 of 279 (97.5%) non-pCR patients (159 of 160 (99.4%), 53 of 54 (98.1%), and 60 of 65 (92.3%) in the training and two external validation cohorts, respectively) were successfully identified by the SMTN, suggesting that they could benefit from regime adjustment at the early-stage of NACT.

INTERPRETATION

The SMTN was able to predict pCR in the early-stage of NACT for HER2-positive breast cancer patients, which could guide clinicians in adjusting treatment regimes.

FUNDING

Key-Area Research and Development Program of Guangdong Province (No.2021B0101420006); National Natural Science Foundation of China (No.82071892, 82171920); Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application (No.2022B1212010011); the National Science Foundation for Young Scientists of China (No.82102019, 82001986); Project Funded by China Postdoctoral Science Foundation (No.2020M682643); the Outstanding Youth Science Foundation of Yunnan Basic Research Project (202101AW070001); Scientific research fund project of Department of Education of Yunnan Province(2022J0249). Science and technology Projects in Guangzhou (202201020001;202201010513); High-level Hospital Construction Project (DFJH201805, DFJHBF202105).