Deep Learning-Based Model for Identifying Tumors in Endoscopic Images From Patients With Locally Advanced Rectal Cancer Treated With Total Neoadjuvant Therapy

ResNet-Vision Transformer based MRI-endoscopy fusion model for predicting treatment response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer: A multicenter study

Background:

Neoadjuvant chemoradiotherapy followed by radical surgery has been a common practice for patients with locally advanced rectal cancer, but the response rate varies among patients. This study aimed to develop a ResNet-Vision Transformer based magnetic resonance imaging (MRI)-endoscopy fusion model to precisely predict treatment response and provide personalized treatment.

Methods:

In this multicenter study, 366 eligible patients who had undergone neoadjuvant chemoradiotherapy followed by radical surgery at eight Chinese tertiary hospitals between January 2017 and June 2024 were recruited, with 2928 pretreatment colonic endoscopic images and 366 pelvic MRI images. An MRI-endoscopy fusion model was constructed based on the ResNet backbone and Transformer network using pretreatment MRI and endoscopic images. Treatment response was defined as good response or non-good response based on the tumor regression grade. The Delong test and the Hanley–McNeil test were utilized to compare prediction performance among different models and different subgroups, respectively. The predictive performance of the MRI-endoscopy fusion model was comprehensively validated in the test sets and was further compared to that of the single-modal MRI model and single-modal endoscopy model.

Results:

The MRI-endoscopy fusion model demonstrated favorable prediction performance. In the internal validation set, the area under the curve (AUC) and accuracy were 0.852 (95% confidence interval [CI]: 0.744–0.940) and 0.737 (95% CI: 0.712–0.844), respectively. Moreover, the AUC and accuracy reached 0.769 (95% CI: 0.678–0.861) and 0.729 (95% CI: 0.628–0.821), respectively, in the external test set. In addition, the MRI-endoscopy fusion model outperformed the single-modal MRI model (AUC: 0.692 [95% CI: 0.609–0.783], accuracy: 0.659 [95% CI: 0.565–0.775]) and the single-modal endoscopy model (AUC: 0.720 [95% CI: 0.617–0.823], accuracy: 0.713 [95% CI: 0.612–0.809]) in the external test set.

Conclusion:

The MRI-endoscopy fusion model based on ResNet-Vision Transformer achieved favorable performance in predicting treatment response to neoadjuvant chemoradiotherapy and holds tremendous potential for enabling personalized treatment regimens for locally advanced rectal cancer patients.

Assessing Endoscopic Response in Locally Advanced Rectal Cancer Treated with Total Neoadjuvant Therapy: Development and Validation of a Highly Accurate Convolutional Neural Network

BACKGROUND

Rectal tumors display varying degrees of response to total neoadjuvant therapy (TNT). We evaluated the performance of a convolutional neural network (CNN) in interpreting endoscopic images of either a non-complete response to TNT or local regrowth during watch-and-wait surveillance.

METHODS

Endoscopic images from stage II/III rectal cancers treated with TNT from 2012 to 2020 at a single institution were retrospectively reviewed. Images were labelled as Tumor or No Tumor based on endoscopy timing (before, during, or after treatment) and the tumor's endoluminal response. A CNN was trained using ResNet-50 architecture. The area under the curve (AUC) was analyzed during training and for two test sets. The main test set included images of tumors treated with TNT. The other contained images of local regrowth. The model's performance was compared to sixteen surgeons and surgical trainees who evaluated 119 images for evidence of tumor. Fleiss' kappa was calculated by respondent experience level.

RESULTS

A total of 2717 images from 288 patients were included; 1407 (51.8%) contained tumor. The AUC was 0.99, 0.98, and 0.92 for training, main test, and local regrowth test sets. The model performed on par with surgeons of all experience levels for the main test set. Interobserver agreement was good ( k = 0.71-0.81). All groups outperformed the model in identifying tumor from images of local regrowth. Interobserver agreement was fair to moderate ( k = 0.24-0.52).

CONCLUSIONS

A highly accurate CNN matched the performance of colorectal surgeons in identifying a noncomplete response to TNT. However, the model demonstrated suboptimal accuracy when analyzing images of local regrowth.

Deep learning model based on endoscopic images predicting treatment response in locally advanced rectal cancer undergo neoadjuvant chemoradiotherapy: a multicenter study

PURPOSE

Neoadjuvant chemoradiotherapy has been the standard practice for patients with locally advanced rectal cancer. However, the treatment response varies greatly among individuals, how to select the optimal candidates for neoadjuvant chemoradiotherapy is crucial. This study aimed to develop an endoscopic image-based deep learning model for predicting the response to neoadjuvant chemoradiotherapy in locally advanced rectal cancer.

METHODS

In this multicenter observational study, pre-treatment endoscopic images of patients from two Chinese medical centers were retrospectively obtained and a deep learning-based tumor regression model was constructed. Treatment response was evaluated based on the tumor regression grade and was defined as good response and non-good response. The prediction performance of the deep learning model was evaluated in the internal and external test sets. The main outcome was the accuracy of the treatment prediction model, measured by the AUC and accuracy.

RESULTS

This deep learning model achieved favorable prediction performance. In the internal test set, the AUC and accuracy were 0.867 (95% CI: 0.847-0.941) and 0.836 (95% CI: 0.818-0.896), respectively. The prediction performance was fully validated in the external test set, and the model had an AUC of 0.758 (95% CI: 0.724-0.834) and an accuracy of 0.807 (95% CI: 0.774-0.843).

CONCLUSION

The deep learning model based on endoscopic images demonstrated exceptional predictive power for neoadjuvant treatment response, highlighting its potential for guiding personalized therapy.

Clinical Tools for Rectal Cancer Response Assessment following Neoadjuvant Treatment in the Era of Organ Preservation

Local tumor response evaluation following neoadjuvant treatment(s) in rectal adenocarcinoma requires a multi-modality approach including physical and endoscopic evaluations, rectal protocoled MRI, and cross-sectional imaging. Clinical tumor response exists on a spectrum from complete clinical response (cCR), defined as the absence of clinical evidence of residual tumor, to near-complete response (nCR), which assumes a significant reduction in tumor burden but with increased uncertainty of residual microscopic disease, to incomplete clinical response (iCR), which incorporates all responses less than nCR that is not progressive disease. This article aims to review the clinical tools currently routinely available to evaluate treatment response and offers a potential management approach based on the extent of local tumor response.

Endoscopic Evaluation of Pathological Complete Response Using Deep Neural Network in Esophageal Cancer Patients Who Received Neoadjuvant Chemotherapy-Multicenter Retrospective Study from Four Japanese Esophageal Centers

BACKGROUND

Detecting pathological complete response (pCR) before surgery would facilitate nonsurgical approach after neoadjuvant chemotherapy (NAC). We developed an artificial intelligence (AI)-guided pCR evaluation using a deep neural network to identify pCR before surgery.

METHODS

This study examined resectable esophageal squamous cell carcinoma (ESCC) patients who underwent esophagectomy after NAC. The same number of histological responders without pCR and non-responders were randomly selected based on the number of pCR patients. Endoscopic images were analyzed using a deep neural network. A test dataset consisting of 20 photos was used for validation. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of AI and four experienced endoscopists' pCR evaluations were calculated. For pathological response evaluation, Japanese Classification of Esophageal Cancer was used.

RESULTS

The study enrolled 123 patients, including 41 patients with pCR, the same number of histological responders without pCR, and non-responders [grade 0, 5 (4%); grade 1a, 36 (30%); grade 1b, 21 (17%); grade 2, 20 (16%); grade 3, 41 (33%)]. In 20 models, the median values of sensitivity, specificity, PPV, NPV, and accuracy for endoscopic response (ER) detection were 60%, 81%, 77%, 67%, and 70%, respectively. Similarly, the endoscopists' median of these was 43%, 90%, 85%, 65%, and 66%, respectively.

CONCLUSIONS

This proof-of-concept study demonstrated that the AI-guided endoscopic response evaluation after NAC could identify pCR with moderate accuracy. The current AI algorithm might guide an individualized treatment strategy including nonsurgical approach in ESCC patients through prospective studies with careful external validation to demonstrate the clinical value of this diagnostic approach including primary tumor and lymph node.

Radiomics in colorectal cancer

Colorectal cancer (CRC) is a global health challenge with high morbidity and mortality. Radiomics, an emerging field, utilizes quantitative imaging features extracted from medical images for CRC diagnosis, staging, treatment response assessment, and prognostication. This review highlights the potential of radiomics for personalized CRC management. Radiomics enables noninvasive tumor characterization, aiding in early detection and accurate diagnosis, and it can be used to predict tumor stage, lymph node involvement, and prognosis. Furthermore, radiomics guides personalized therapies by assessing the treatment response and identifying patients who could benefit. Challenges include standardizing imaging protocols and analysis techniques. Robust validation frameworks and user‐friendly software are needed for the integration of radiomics into clinical practice. Despite challenges, radiomics offers valuable insights into tumor biology, treatment response, and prognosis in CRC. Overcoming technical and clinical hurdles will unlock its full potential in CRC management.

Prognostic impact of endoscopic response evaluation after neoadjuvant chemotherapy for esophageal squamous cell carcinoma: a nationwide validation study

BACKGROUND

Our previous study reported the prognostic significance of endoscopic response (ER) evaluation, defined ER, and revealed ER as an independent prognostic factor of overall survival (OS) and recurrence-free survival (RFS) for esophageal squamous cell carcinoma (ESCC) treated with neoadjuvant chemotherapy (NAC) and surgery. The present study aimed to validate the prognostic impact of ER using a nationwide database from the authorized institute for board-certified esophageal surgeons by the Japan Esophageal Society.

METHODS

This study retrospectively reviewed patients with ESCC who underwent subtotal esophagectomy at 85 authorized institutes for esophageal cancer from 2010 to 2015. Patients were classified as ER when the tumor size was markedly reduced post-NAC compared to pre-NAC. The correlation between OS and RFS was investigated.

RESULTS

Of 4781 patients initially enrolled, 3636 were selected for subsequent analysis. Of them, 642 (17.7%) patients were classified as the ER group. Patients with ER showed significantly better OS and RFS. Subgroup analysis revealed the statistical difference in OS and RFS in cStage II and III, while the magnitude of survival difference between ER and non-ER was not evident in cStage I and IV. The percentage of ER varied from 46 to 87% among groups when institutions were classified into 3 subgroups based on the hospital volume, which would indicate the interinstitutional inconsistency.

CONCLUSIONS

The prognostic impact of ER was validated using a nationwide database. Standardization of ER evaluation is required to improve the interinstitutional consistency and clinical validity of the ER evaluation.

Evaluation of Endoscopic Response Using Deep Neural Network in Esophageal Cancer Patients Who Received Neoadjuvant Chemotherapy

BACKGROUND

We previously reported that endoscopic response evaluation can preoperatively predict the prognosis and distribution of residual tumors after neoadjuvant chemotherapy (NAC). In this study, we developed artificial intelligence (AI)-guided endoscopic response evaluation using a deep neural network to discriminate endoscopic responders (ERs) in patients with esophageal squamous cell carcinoma (ESCC) after NAC.

METHOD

Surgically resectable ESCC patients who underwent esophagectomy following NAC were retrospectively analyzed in this study. Endoscopic images of the tumors were analyzed using a deep neural network. The model was validated with a test data set using 10 newly collected ERs and 10 newly collected non-ER images. The sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of the endoscopic response evaluation by AI and endoscopists were calculated and compared.

RESULTS

Of 193 patients, 40 (21%) were diagnosed as ERs. The median sensitivity, specificity, PPV, and NPV values for ER detection in 10 models were 60%, 100%, 100%, and 71%, respectively. Similarly, the median values by the endoscopist were 80%, 80%, 81%, and 81%, respectively.

CONCLUSION

This proof-of-concept study using a deep learning algorithm demonstrated that the constructed AI-guided endoscopic response evaluation after NAC could identify ER with high specificity and PPV. It would appropriately guide an individualized treatment strategy that includes an organ preservation approach in ESCC patients.

Automatic treatment outcome prediction with DeepInteg based on multimodal radiological images in rectal cancer

Neoadjuvant systemic treatment before surgery is a prevalent regimen in the patients with advanced-stage or high-risk tumor, which has shaped the treatment strategies and cancer survival in the past decades. However, some patients present with poor response to the neoadjuvant treatment. Therefore, it is of great significance to develop tools to help distinguish the patients that could achieve pathological complete response before surgery to avoid inappropriate treatment. Here, this study demonstrated a multi-task deep learning tool called DeepInteg. In the DeepInteg framework, the segmentation module was constructed based on the CE-Net with a context extractor to achieve end-to-end delineation of region of interest (ROI) from radiological images, then the features of segmented Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) images of each case were fused and input to the classification module based on a convolution neural network for treatment outcome prediction. The dataset with 1700 MRI and CT slices collected from the prospectively randomized clinical trial (NCT01211210) on systemic treatment for rectal cancer was used to develop and systematically optimize DeepInteg. As a result, DeepInteg achieved automatic segmentation of tumoral ROI with Dices of 0.766 and 0.719 and mIoUs of 0.788 and 0.756 in CT and MRI images, respectively. In addition, DeepInteg achieved AUC of 0.833, accuracy of 0.826 and specificity of 0.856 in the prediction for pathological complete response after treatment, which showed better performance compared with the model based on CT or MRI alone. This study provide a robust framework to develop disease-specific tools for automatic delineation of ROI and clinical outcome prediction. The well-trained DeepInteg could be readily applied in clinic to predict pathological complete response after neoadjuvant therapy in rectal cancer patients.