Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study

Masked hypergraph learning for weakly supervised histopathology whole slide image classification

BACKGROUND AND OBJECTIVES

Graph neural network (GNN) has been extensively used in histopathology whole slide image (WSI) analysis due to the efficiency and flexibility in modelling relationships among entities. However, most existing GNN-based WSI analysis methods only consider the pairwise correlation of patches from one single perspective (e.g. spatial affinity or embedding similarity) yet ignore the intrinsic non-pairwise relationships present in gigapixel WSI, which are likely to contribute to feature learning and downstream tasks. The objective of this study is therefore to explore the non-pairwise relationships in histopathology WSI and exploit them to guide the learning of slide-level representations for better classification performance.

METHODS

In this paper, we propose a novel Masked HyperGraph Learning (MaskHGL) framework for weakly supervised histopathology WSI classification. Compared with most GNN-based WSI classification methods, MaskHGL exploits the non-pairwise correlations between patches with hypergraph and global message passing conducted by hypergraph convolution. Concretely, multi-perspective hypergraphs are first built for each WSI, then hypergraph attention is introduced into the jointed hypergraph to propagate the non-pairwise relationships and thus yield more discriminative node representation. More importantly, a masked hypergraph reconstruction module is devised to guide the hypergraph learning which can generate more powerful robustness and generalization than the method only using hypergraph modelling. Additionally, a self-attention-based node aggregator is also applied to explore the global correlation of patches in WSI and produce the slide-level representation for classification.

RESULTS

The proposed method is evaluated on two public TCGA benchmark datasets and one in-house dataset. On the public TCGA-LUNG (1494 WSIs) and TCGA-EGFR (696 WSIs) test set, the area under receiver operating characteristic (ROC) curve (AUC) were 0.9752±0.0024 and 0.7421±0.0380, respectively. On the USTC-EGFR (754 WSIs) dataset, MaskHGL achieved significantly better performance with an AUC of 0.8745±0.0100, which surpassed the second-best state-of-the-art method SlideGraph+ 2.64%.

CONCLUSIONS

MaskHGL shows a great improvement, brought by considering the intrinsic non-pairwise relationships within WSI, in multiple downstream WSI classification tasks. In particular, the designed masked hypergraph reconstruction module promisingly alleviates the data scarcity and greatly enhances the robustness and classification ability of our MaskHGL. Notably, it has shown great potential in cancer subtyping and fine-grained lung cancer gene mutation prediction from hematoxylin and eosin (H&E) stained WSIs.

Deep neural network for the prediction of KRAS, NRAS, and BRAF genotypes in left-sided colorectal cancer based on histopathologic images

BACKGROUND

The KRAS, NRAS, and BRAF genotypes are critical for selecting targeted therapies for patients with metastatic colorectal cancer (mCRC). Here, we aimed to develop a deep learning model that utilizes pathologic whole-slide images (WSIs) to accurately predict the status of KRAS, NRAS, and BRAFV600E.

METHODS

129 patients with left-sided colon cancer and rectal cancer from the Third Affiliated Hospital of Sun Yat-sen University were assigned to the training and testing cohorts. Utilizing three convolutional neural networks (ResNet18, ResNet50, and Inception v3), we extracted 206 pathological features from H&E-stained WSIs, serving as the foundation for constructing specific pathological models. A clinical feature model was then developed, with carcinoembryonic antigen (CEA) identified through comprehensive multiple regression analysis as the key biomarker. Subsequently, these two models were combined to create a clinical-pathological integrated model, resulting in a total of three genetic prediction models.

RESULT

103 patients were evaluated in the training cohort (1782,302 image tiles), while the remaining 26 patients were enrolled in the testing cohort (489,481 image tiles). Compared with the clinical model and the pathology model, the combined model which incorporated CEA levels and pathological signatures, showed increased predictive ability, with an area under the curve (AUC) of 0.96 in the training and an AUC of 0.83 in the testing cohort, accompanied by a high positive predictive value (PPV 0.92).

CONCLUSION

The combined model demonstrated a considerable ability to accurately predict the status of KRAS, NRAS, and BRAFV600E in patients with left-sided colorectal cancer, with potential application to assist doctors in developing targeted treatment strategies for mCRC patients, and effectively identifying mutations and eliminating the need for confirmatory genetic testing.

Prediction of epidermal growth factor receptor mutation subtypes in non-small cell lung cancer from hematoxylin and eosin-stained slides using deep learning

Accurate assessment of epidermal growth factor receptor (EGFR) mutation status and subtype are critical for the treatment of non-small cell lung cancer (NSCLC) patients. Conventional molecular testing methods for detecting EGFR mutations have limitations. In this study, an artificial intelligence-powered deep learning framework was developed for weakly supervised prediction of EGFR mutations in NSCLC from hematoxylin and eosin (H&E)-stained histopathology whole-slide images (WSIs). The study cohort was partitioned into training and validation subsets. Foreground regions containing tumor tissue were extracted from WSIs. A convolutional neural network (CNN) employing a contrastive learning paradigm was implemented to extract patch-level morphological features. These features were aggregated using a vision-transformer-based model to predict EGFR mutation status and classify patient cases. The established prediction model was validated on unseen datasets. In internal validation with a cohort from (USTC)(n=172), the model achieved patient-level areas under the receiver operating characteristic (ROC) curve (AUCs) of 0.927 and 0.907, sensitivities of 81.6% and 93.0%, and specificities of 83.3% and 92.3%, for surgical resection and biopsy specimens in EGFR mutation subtype prediction, respectively. External validation with cohorts from the Second Affiliated Hospital of Anhui Medical University (AMU) and the First Affiliated Hospital of Wannan Medical College (WMC) (n=193) yielded patient-level AUCs of 0.849 and 0.871, sensitivities of 75.7% and 72.1%, and specificities of 90.5% and 90.3% for surgical and biopsy specimens, respectively. Further validation with The Cancer Genome Atlas (TCGA) dataset (n=81) showed an AUC of 0.861, sensitivity of 84.6%, and specificity of 90.5%. Deep learning solutions demonstrate potential advantages for automated, non-invasive, fast, cost-effective, and accurate inference of EGFR alterations from histomorphology. Integration of such artificial intelligence frameworks into routine digital pathology workflows could augment existing molecular testing pipelines.

Prognostic Analysis Combining Histopathological Features and Clinical Information to Predict Colorectal Cancer Survival from Whole-Slide Images

BACKGROUND

Colorectal cancer (CRC) is a malignant tumor within the digestive tract with both a high incidence rate and mortality. Early detection and intervention could improve patient clinical outcomes and survival.

METHODS

This study computationally investigates a set of prognostic tissue and cell features from diagnostic tissue slides. With the combination of clinical prognostic variables, the pathological image features could predict the prognosis in CRC patients. Our CRC prognosis prediction pipeline sequentially consisted of three modules: (1) A MultiTissue Net to delineate outlines of different tissue types within the WSI of CRC for further ROI selection by pathologists. (2) Development of three-level quantitative image metrics related to tissue compositions, cell shape, and hidden features from a deep network. (3) Fusion of multi-level features to build a prognostic CRC model for predicting survival for CRC.

RESULTS

Experimental results suggest that each group of features has a particular relationship with the prognosis of patients in the independent test set. In the fusion features combination experiment, the accuracy rate of predicting patients' prognosis and survival status is 81.52%, and the AUC value is 0.77.

CONCLUSION

This paper constructs a model that can predict the postoperative survival of patients by using image features and clinical information. Some features were found to be associated with the prognosis and survival of patients.

A guide to artificial intelligence for cancer researchers

Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.

Precision medicine in colorectal cancer: Leveraging multi-omics, spatial omics, and artificial intelligence

Colorectal cancer (CRC) is a leading cause of cancer-related deaths. Recent advancements in genomic technologies and analytical approaches have revolutionized CRC research, enabling precision medicine. This review highlights the integration of multi-omics, spatial omics, and artificial intelligence (AI) in advancing precision medicine for CRC. Multi-omics approaches have uncovered molecular mechanisms driving CRC progression, while spatial omics have provided insights into the spatial heterogeneity of gene expression in CRC tissues. AI techniques have been utilized to analyze complex datasets, identify new treatment targets, and enhance diagnosis and prognosis. Despite the tumor's heterogeneity and genetic and epigenetic complexity, the fusion of multi-omics, spatial omics, and AI shows the potential to overcome these challenges and advance precision medicine in CRC. The future lies in integrating these technologies to provide deeper insights and enable personalized therapies for CRC patients.

Automated curation of large-scale cancer histopathology image datasets using deep learning

BACKGROUND

Artificial intelligence (AI) has numerous applications in pathology, supporting diagnosis and prognostication in cancer. However, most AI models are trained on highly selected data, typically one tissue slide per patient. In reality, especially for large surgical resection specimens, dozens of slides can be available for each patient. Manually sorting and labelling whole-slide images (WSIs) is a very time-consuming process, hindering the direct application of AI on the collected tissue samples from large cohorts. In this study we addressed this issue by developing a deep-learning (DL)-based method for automatic curation of large pathology datasets with several slides per patient.

METHODS

We collected multiple large multicentric datasets of colorectal cancer histopathological slides from the United Kingdom (FOXTROT, N = 21,384 slides; CR07, N = 7985 slides) and Germany (DACHS, N = 3606 slides). These datasets contained multiple types of tissue slides, including bowel resection specimens, endoscopic biopsies, lymph node resections, immunohistochemistry-stained slides, and tissue microarrays. We developed, trained, and tested a deep convolutional neural network model to predict the type of slide from the slide overview (thumbnail) image. The primary statistical endpoint was the macro-averaged area under the receiver operating curve (AUROCs) for detection of the type of slide.

RESULTS

In the primary dataset (FOXTROT), with an AUROC of 0.995 [95% confidence interval [CI]: 0.994-0.996] the algorithm achieved a high classification performance and was able to accurately predict the type of slide from the thumbnail image alone. In the two external test cohorts (CR07, DACHS) AUROCs of 0.982 [95% CI: 0.979-0.985] and 0.875 [95% CI: 0.864-0.887] were observed, which indicates the generalizability of the trained model on unseen datasets. With a confidence threshold of 0.95, the model reached an accuracy of 94.6% (7331 classified cases) in CR07 and 85.1% (2752 classified cases) for the DACHS cohort.

CONCLUSION

Our findings show that using the low-resolution thumbnail image is sufficient to accurately classify the type of slide in digital pathology. This can support researchers to make the vast resource of existing pathology archives accessible to modern AI models with only minimal manual annotations.

Predictive modeling of deep vein thrombosis risk in hospitalized patients: A Q-learning enhanced feature selection model

Deep vein thrombosis (DVT) represents a critical health concern due to its potential to lead to pulmonary embolism, a life-threatening complication. Early identification and prediction of DVT are crucial to prevent thromboembolic events and implement timely prophylactic measures in high-risk individuals. This study aims to examine the risk determinants associated with acute lower extremity DVT in hospitalized individuals. Additionally, it introduces an innovative approach by integrating Q-learning augmented colony predation search ant colony optimizer (QL-CPSACO) into the analysis. This algorithm, then combined with support vector machines (SVM), forms a bQL-CPSACO-SVM feature selection model dedicated to crafting a clinical risk prognostication model for DVT. The effectiveness of the proposed algorithm's optimization and the model's accuracy are assessed through experiments utilizing the CEC 2017 benchmark functions and predictive analyses on the DVT dataset. The experimental results reveal that the proposed model achieves an outstanding accuracy of 95.90% in predicting DVT. Key parameters such as D-dimer, normal plasma prothrombin time, prothrombin percentage activity, age, previously documented DVT, leukocyte count, and thrombocyte count demonstrate significant value in the prognostication of DVT. The proposed method provides a basis for risk assessment at the time of patient admission and offers substantial guidance to physicians in making therapeutic decisions.

Deep learning for dual detection of microsatellite instability and POLE mutations in colorectal cancer histopathology

In the spectrum of colorectal tumors, microsatellite-stable (MSS) tumors with DNA polymerase ε (POLE) mutations exhibit a hypermutated profile, holding the potential to respond to immunotherapy similarly to their microsatellite-instable (MSI) counterparts. Yet, due to their rarity and the associated testing costs, systematic screening for these mutations is not commonly pursued. Notably, the histopathological phenotype resulting from POLE mutations is theorized to resemble that of MSI. This resemblance not only could facilitate their detection by a transformer-based Deep Learning (DL) system trained on MSI pathology slides, but also indicates the possibility for MSS patients with POLE mutations to access enhanced treatment options, which might otherwise be overlooked. To harness this potential, we trained a Deep Learning classifier on a large dataset with the ground truth for microsatellite status and subsequently validated its capabilities for MSI and POLE detection across three external cohorts. Our model accurately identified MSI status in both the internal and external resection cohorts using pathology images alone. Notably, with a classification threshold of 0.5, over 75% of POLE driver mutant patients in the external resection cohorts were flagged as "positive" by a DL system trained on MSI status. In a clinical setting, deploying this DL model as a preliminary screening tool could facilitate the efficient identification of clinically relevant MSI and POLE mutations in colorectal tumors, in one go.

Prediction of Mismatch Repair Status in Endometrial Cancer from Histological Slide Images Using Various Deep Learning-Based Algorithms

The application of deep learning algorithms to predict the molecular profiles of various cancers from digital images of hematoxylin and eosin (H&E)-stained slides has been reported in recent years, mainly for gastric and colon cancers. In this study, we investigated the potential use of H&E-stained endometrial cancer slide images to predict the associated mismatch repair (MMR) status. H&E-stained slide images were collected from 127 cases of the primary lesion of endometrial cancer. After digitization using a Nanozoomer virtual slide scanner (Hamamatsu Photonics), we segmented the scanned images into 5397 tiles of 512 × 512 pixels. The MMR proteins (PMS2, MSH6) were immunohistochemically stained, classified into MMR proficient/deficient, and annotated for each case and tile. We trained several neural networks, including convolutional and attention-based networks, using tiles annotated with the MMR status. Among the tested networks, ResNet50 exhibited the highest area under the receiver operating characteristic curve (AUROC) of 0.91 for predicting the MMR status. The constructed prediction algorithm may be applicable to other molecular profiles and useful for pre-screening before implementing other, more costly genetic profiling tests.

Classification and prediction of chemoradiotherapy response and survival from esophageal carcinoma histopathology images

Whole slide imaging (WSI) of Hematoxylin and Eosin-stained biopsy specimens has been used to predict chemoradiotherapy (CRT) response and overall survival (OS) of esophageal squamous cell carcinoma (ESCC) patients. This retrospective study collected 279 specimens in 89 non-surgical ESCC patients through endoscopic biopsy between January 2010 and January 2019. These patients were divided into a CRT response group (CR + PR group) and a CRT non-response group (SD + PD group). The WSIs have segmented approximately 1,206,000 non-overlapping patches. Two experienced pathologists manually delineated the eight types of tissues on 32 WSIs, including esophagus tumor cell (TUM), cancer-associated stroma (CAS), normal epithelium layer (NEL), smooth muscle (MUS), lymphocytes (LYM), Red cells (RED), debris (DEB), uneven areas (UNE). The chemoradiotherapy response prediction models were built using maximum relevance-minimum redundancy (MRMR) feature selection and least absolute shrinkage and selection operator (LASSO) regression. However, pathological features with p < 0.1 were selected and integrated to be further screened using a LASSO Cox regression model to build a multivariate Cox proportional hazards model for predicting the OS. The testing accuracy of the tissue classification model was 91.3 %. The pathological model created using two CAS in-depth features and eight TUM in-depth features performed best for the prediction of treatment response and achieved an AUC of 0.744. For the prediction of OS, the testing AUC of this model at one year and three years were 0.675 and 0.870, respectively. The TUM model showed the highest AUC at one year (0.712). With its high accuracy rate, the deep learning model has the potential to transform from bench to bedside in clinical practice, improve patient's quality of life, and prolong the OS rate.

A systematic review of machine learning-based tumor-infiltrating lymphocytes analysis in colorectal cancer: Overview of techniques, performance metrics, and clinical outcomes

The incidence of colorectal cancer (CRC), one of the deadliest cancers around the world, is increasing. Tissue microenvironment (TME) features such as tumor-infiltrating lymphocytes (TILs) can have a crucial impact on diagnosis or decision-making for treating patients with CRC. While clinical studies showed that TILs improve the host immune response, leading to a better prognosis, inter-observer agreement for quantifying TILs is not perfect. Incorporating machine learning (ML) based applications in clinical routine may promote diagnosis reliability. Recently, ML has shown potential for making progress in routine clinical procedures. We aim to systematically review the TILs analysis based on ML in CRC histological images. Deep learning (DL) and non-DL techniques can aid pathologists in identifying TILs, and automated TILs are associated with patient outcomes. However, a large multi-institutional CRC dataset with a diverse and multi-ethnic population is necessary to generalize ML methods.

Pros and cons of artificial intelligence implementation in diagnostic pathology

The rapid introduction of digital pathology has greatly facilitated development of artificial intelligence (AI) models in pathology that have shown great promise in assisting morphological diagnostics and quantitation of therapeutic targets. We are now at a tipping point where companies have started to bring algorithms to the market, and questions arise whether the pathology community is ready to implement AI in routine workflow. However, concerns also arise about the use of AI in pathology. This article reviews the pros and cons of introducing AI in diagnostic pathology.

Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer

Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.

Integrating AI-Powered Digital Pathology and Imaging Mass Cytometry Identifies Key Classifiers of Tumor Cells, Stroma, and Immune Cells in Non-Small Cell Lung Cancer

Artificial intelligence (AI)-powered approaches are becoming increasingly used as histopathologic tools to extract subvisual features and improve diagnostic workflows. On the other hand, hi-plex approaches are widely adopted to analyze the immune ecosystem in tumor specimens. Here, we aimed at combining AI-aided histopathology and imaging mass cytometry (IMC) to analyze the ecosystem of non-small cell lung cancer (NSCLC). An AI-based approach was used on hematoxylin and eosin (H&E) sections from 158 NSCLC specimens to accurately identify tumor cells, both adenocarcinoma and squamous carcinoma cells, and to generate a classifier of tumor cell spatial clustering. Consecutive tissue sections were stained with metal-labeled antibodies and processed through the IMC workflow, allowing quantitative detection of 24 markers related to tumor cells, tissue architecture, CD45+ myeloid and lymphoid cells, and immune activation. IMC identified 11 macrophage clusters that mainly localized in the stroma, except for S100A8+ cells, which infiltrated tumor nests. T cells were preferentially localized in peritumor areas or in tumor nests, the latter being associated with better prognosis, and they were more abundant in highly clustered tumors. Integrated tumor and immune classifiers were validated as prognostic on whole slides. In conclusion, integration of AI-powered H&E and multiparametric IMC allows investigation of spatial patterns and reveals tissue relevant features with clinical relevance.

SIGNIFICANCE

Leveraging artificial intelligence-powered H&E analysis integrated with hi-plex imaging mass cytometry provides insights into the tumor ecosystem and can translate tumor features into classifiers to predict prognosis, genotype, and therapy response.

Predicting 5-year recurrence risk in colorectal cancer: development and validation of a histology-based deep learning approach

BACKGROUND

Accurate estimation of the long-term risk of recurrence in patients with non-metastatic colorectal cancer (CRC) is crucial for clinical management. Histology-based deep learning is expected to provide more abundant information for risk stratification.

METHODS

We developed and validated a weakly supervised deep-learning model for predicting 5-year relapse-free survival (RFS) to stratify patients with different risks based on histological images from three hospitals of 614 cases with non-metastatic CRC. A deep prognostic factor (DL-RRS) was established to stratify patients into high and low-risk group. The areas under the curve (AUCs) were calculated to evaluate the performances of models.

RESULTS

Our proposed model achieves the AUCs of 0.833 (95% CI: 0.736-0.905) and 0.715 (95% CI: 0.647-0.776) on validation cohort and external test cohort, respectively. The 5-year RFS rate was 45.7% for high DL-RRS patients, and 82.5% for low DL-RRS patients respectively in the external test cohort (HR: 3.89, 95% CI: 2.51-6.03, P < 0.001). Adjuvant chemotherapy was associated with improved RFS in Stage II patients with high DL-RRS (HR: 0.15, 95% CI: 0.06-0.38, P < 0.001).

CONCLUSIONS

DL-RRS has a good predictive performance of 5-year recurrence risk in CRC, and will better serve the clinical decision-making.

The Use of Artificial Intelligence in Head and Neck Cancers: A Multidisciplinary Survey

Artificial intelligence (AI) approaches have been introduced in various disciplines but remain rather unused in head and neck (H&N) cancers. This survey aimed to infer the current applications of and attitudes toward AI in the multidisciplinary care of H&N cancers. From November 2020 to June 2022, a web-based questionnaire examining the relationship between AI usage and professionals' demographics and attitudes was delivered to different professionals involved in H&N cancers through social media and mailing lists. A total of 139 professionals completed the questionnaire. Only 49.7% of the respondents reported having experience with AI. The most frequent AI users were radiologists (66.2%). Significant predictors of AI use were primary specialty (V = 0.455; p < 0.001), academic qualification and age. AI's potential was seen in the improvement of diagnostic accuracy (72%), surgical planning (64.7%), treatment selection (57.6%), risk assessment (50.4%) and the prediction of complications (45.3%). Among participants, 42.7% had significant concerns over AI use, with the most frequent being the 'loss of control' (27.6%) and 'diagnostic errors' (57.0%). This survey reveals limited engagement with AI in multidisciplinary H&N cancer care, highlighting the need for broader implementation and further studies to explore its acceptance and benefits.

Generation of synthetic whole-slide image tiles of tumours from RNA-sequencing data via cascaded diffusion models

Training machine-learning models with synthetically generated data can alleviate the problem of data scarcity when acquiring diverse and sufficiently large datasets is costly and challenging. Here we show that cascaded diffusion models can be used to synthesize realistic whole-slide image tiles from latent representations of RNA-sequencing data from human tumours. Alterations in gene expression affected the composition of cell types in the generated synthetic image tiles, which accurately preserved the distribution of cell types and maintained the cell fraction observed in bulk RNA-sequencing data, as we show for lung adenocarcinoma, kidney renal papillary cell carcinoma, cervical squamous cell carcinoma, colon adenocarcinoma and glioblastoma. Machine-learning models pretrained with the generated synthetic data performed better than models trained from scratch. Synthetic data may accelerate the development of machine-learning models in scarce-data settings and allow for the imputation of missing data modalities.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review

BACKGROUND

Generative artificial intelligence tools and applications (GenAI) are being increasingly used in health care. Physicians, specialists, and other providers have started primarily using GenAI as an aid or tool to gather knowledge, provide information, train, or generate suggestive dialogue between physicians and patients or between physicians and patients' families or friends. However, unless the use of GenAI is oriented to be helpful in clinical service encounters that can improve the accuracy of diagnosis, treatment, and patient outcomes, the expected potential will not be achieved. As adoption continues, it is essential to validate the effectiveness of the infusion of GenAI as an intelligent technology in service encounters to understand the gap in actual clinical service use of GenAI.

OBJECTIVE

This study synthesizes preliminary evidence on how GenAI assists, guides, and automates clinical service rendering and encounters in health care The review scope was limited to articles published in peer-reviewed medical journals.

METHODS

We screened and selected 0.38% (161/42,459) of articles published between January 1, 2020, and May 31, 2023, identified from PubMed. We followed the protocols outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to select highly relevant studies with at least 1 element on clinical use, evaluation, and validation to provide evidence of GenAI use in clinical services. The articles were classified based on their relevance to clinical service functions or activities using the descriptive and analytical information presented in the articles.

RESULTS

Of 161 articles, 141 (87.6%) reported using GenAI to assist services through knowledge access, collation, and filtering. GenAI was used for disease detection (19/161, 11.8%), diagnosis (14/161, 8.7%), and screening processes (12/161, 7.5%) in the areas of radiology (17/161, 10.6%), cardiology (12/161, 7.5%), gastrointestinal medicine (4/161, 2.5%), and diabetes (6/161, 3.7%). The literature synthesis in this study suggests that GenAI is mainly used for diagnostic processes, improvement of diagnosis accuracy, and screening and diagnostic purposes using knowledge access. Although this solves the problem of knowledge access and may improve diagnostic accuracy, it is oriented toward higher value creation in health care.

CONCLUSIONS

GenAI informs rather than assisting or automating clinical service functions in health care. There is potential in clinical service, but it has yet to be actualized for GenAI. More clinical service-level evidence that GenAI is used to streamline some functions or provides more automated help than only information retrieval is needed. To transform health care as purported, more studies related to GenAI applications must automate and guide human-performed services and keep up with the optimism that forward-thinking health care organizations will take advantage of GenAI.

An interpretable deep learning model for identifying the morphological characteristics of dMMR/MSI-H gastric cancer

Accurate tumor diagnosis by pathologists relies on identifying specific morphological characteristics. However, summarizing these unique morphological features in tumor classifications can be challenging. Although deep learning models have been extensively studied for tumor classification, their indirect and subjective interpretation obstructs pathologists from comprehending the model and discerning the morphological features accountable for classifications. In this study, we introduce a new approach utilizing Style Generative Adversarial Networks, which enables a direct interpretation of deep learning models to detect significant morphological characteristics within datasets representing patients with deficient mismatch repair/microsatellite instability-high gastric cancer. Our approach effectively identifies distinct morphological features crucial for tumor classification, offering valuable insights for pathologists to enhance diagnostic accuracy and foster professional growth.

Informing immunotherapy with multi-omics driven machine learning

Progress in sequencing technologies and clinical experiments has revolutionized immunotherapy on solid and hematologic malignancies. However, the benefits of immunotherapy are limited to specific patient subsets, posing challenges for broader application. To improve its effectiveness, identifying biomarkers that can predict patient response is crucial. Machine learning (ML) play a pivotal role in harnessing multi-omic cancer datasets and unlocking new insights into immunotherapy. This review provides an overview of cutting-edge ML models applied in omics data for immunotherapy analysis, including immunotherapy response prediction and immunotherapy-relevant tumor microenvironment identification. We elucidate how ML leverages diverse data types to identify significant biomarkers, enhance our understanding of immunotherapy mechanisms, and optimize decision-making process. Additionally, we discuss current limitations and challenges of ML in this rapidly evolving field. Finally, we outline future directions aimed at overcoming these barriers and improving the efficiency of ML in immunotherapy research.

Unsupervised Joint Domain Adaptation for Decoding Brain Cognitive States From tfMRI Images

Recent advances in large model and neuroscience have enabled exploration of the mechanism of brain activity by using neuroimaging data. Brain decoding is one of the most promising researches to further understand the human cognitive function. However, current methods excessively depends on high-quality labeled data, which brings enormous expense of collection and annotation of neural images by experts. Besides, the performance of cross-individual decoding suffers from inconsistency in data distribution caused by individual variation and different collection equipments. To address mentioned above issues, a Join Domain Adapative Decoding (JDAD) framework is proposed for unsupervised decoding specific brain cognitive state related to behavioral task. Based on the volumetric feature extraction from task-based functional Magnetic Resonance Imaging (tfMRI) data, a novel objective loss function is designed by the combination of joint distribution regularizer, which aims to restrict the distance of both the conditional and marginal probability distribution of labeled and unlabeled samples. Experimental results on the public Human Connectome Project (HCP) S1200 dataset show that JDAD achieves superior performance than other prevalent methods, especially for fine-grained task with 11.5%-21.6% improvements of decoding accuracy. The learned 3D features are visualized by Grad-CAM to build a combination with brain functional regions, which provides a novel path to learn the function of brain cortex regions related to specific cognitive task in group level.

Artificial Intelligence in the Diagnosis of Colorectal Cancer: A Literature Review

BACKGROUND

The aim of this review is to explore the role of artificial intelligence in the diagnosis of colorectal cancer, how it impacts CRC morbidity and mortality, and why its role in clinical medicine is limited.

METHODS

A targeted, non-systematic review of the published literature relating to colorectal cancer diagnosis was performed with PubMed databases that were scouted to help provide a more defined understanding of the recent advances regarding artificial intelligence and their impact on colorectal-related morbidity and mortality. Articles were included if deemed relevant and including information associated with the keywords.

RESULTS

The advancements in artificial intelligence have been significant in facilitating an earlier diagnosis of CRC. In this review, we focused on evaluating genomic biomarkers, the integration of instruments with artificial intelligence, MR and hyperspectral imaging, and the architecture of neural networks. We found that these neural networks seem practical and yield positive results in initial testing. Furthermore, we explored the use of deep-learning-based majority voting methods, such as bag of words and PAHLI, in improving diagnostic accuracy in colorectal cancer detection. Alongside this, the autonomous and expansive learning ability of artificial intelligence, coupled with its ability to extract increasingly complex features from images or videos without human reliance, highlight its impact in the diagnostic sector. Despite this, as most of the research involves a small sample of patients, a diversification of patient data is needed to enhance cohort stratification for a more sensitive and specific neural model. We also examined the successful application of artificial intelligence in predicting microsatellite instability, showcasing its potential in stratifying patients for targeted therapies.

CONCLUSIONS

Since its commencement in colorectal cancer, artificial intelligence has revealed a multitude of functionalities and augmentations in the diagnostic sector of CRC. Given its early implementation, its clinical application remains a fair way away, but with steady research dedicated to improving neural architecture and expanding its applicational range, there is hope that these advanced neural software could directly impact the early diagnosis of CRC. The true promise of artificial intelligence, extending beyond the medical sector, lies in its potential to significantly influence the future landscape of CRC's morbidity and mortality.

A journey from omics to clinicomics in solid cancers: Success stories and challenges

The word 'cancer' encompasses a heterogenous group of distinct disease types characterized by a spectrum of pathological features, genetic alterations and response to therapies. According to the World Health Organization, cancer is the second leading cause of death worldwide, responsible for one in six deaths and hence imposes a significant burden on global healthcare systems. High-throughput omics technologies combined with advanced imaging tools, have revolutionized our ability to interrogate the molecular landscape of tumors and has provided unprecedented understanding of the disease. Yet, there is a gap between basic research discoveries and their translation into clinically meaningful therapies for improving patient care. To bridge this gap, there is a need to analyse the vast amounts of high dimensional datasets from multi-omics platforms. The integration of multi-omics data with clinical information like patient history, histological examination and imaging has led to the novel concept of clinicomics and may expedite the bench-to-bedside transition in cancer. The journey from omics to clinicomics has gained momentum with development of radiomics which involves extracting quantitative features from medical imaging data with the help of deep learning and artificial intelligence (AI) tools. These features capture detailed information about the tumor's shape, texture, intensity, and spatial distribution. Together, the related fields of multiomics, translational bioinformatics, radiomics and clinicomics may provide evidence-based recommendations tailored to the individual cancer patient's molecular profile and clinical characteristics. In this chapter, we summarize multiomics studies in solid cancers with a specific focus on breast cancer. We also review machine learning and AI based algorithms and their use in cancer diagnosis, subtyping, prognosis and predicting treatment resistance and relapse.

Predicting Mismatch Repair Deficiency Status in Endometrial Cancer through Multi-Resolution Ensemble Learning in Digital Pathology

For molecular classification of endometrial carcinoma, testing for mismatch repair (MMR) status is becoming a routine process. Mismatch repair deficiency (MMR-D) is caused by loss of expression in one or more of the 4 major MMR proteins: MLH1, MSH2, MSH6, PHS2. Over 30% of patients with endometrial cancer have MMR-D. Determining the MMR status holds significance as individuals with MMR-D are potential candidates for immunotherapy. Pathological whole slide image (WSI) of endometrial cancer with immunohistochemistry results of MMR proteins were gathered. Color normalization was applied to the tiles using a CycleGAN-based network. The WSI was divided into tiles at three different magnifications (2.5 × , 5 × , and 10 ×). Three distinct networks of the same architecture were employed to include features from all three magnification levels and were stacked for ensemble learning. Three architectures, InceptionResNetV2, EfficientNetB2, and EfficientNetB3 were employed and subjected to comparison. The per-tile results were gathered to classify MMR status in the WSI, and prediction accuracy was evaluated using the following performance metrics: AUC, accuracy, sensitivity, and specificity. The EfficientNetB2 was able to make predictions with an AUC of 0.821, highest among the three architectures, and an overall AUC range of 0.767 - 0.821 was reported across the three architectures. In summary, our study successfully predicted MMR classification from pathological WSIs in endometrial cancer through a multi-resolution ensemble learning approach, which holds the potential to facilitate swift decisions on tailored treatment, such as immunotherapy, in clinical settings.

Regression-based Deep-Learning predicts molecular biomarkers from pathology slides

Deep Learning (DL) can predict biomarkers from cancer histopathology. Several clinically approved applications use this technology. Most approaches, however, predict categorical labels, whereas biomarkers are often continuous measurements. We hypothesize that regression-based DL outperforms classification-based DL. Therefore, we develop and evaluate a self-supervised attention-based weakly supervised regression method that predicts continuous biomarkers directly from 11,671 images of patients across nine cancer types. We test our method for multiple clinically and biologically relevant biomarkers: homologous recombination deficiency score, a clinically used pan-cancer biomarker, as well as markers of key biological processes in the tumor microenvironment. Using regression significantly enhances the accuracy of biomarker prediction, while also improving the predictions' correspondence to regions of known clinical relevance over classification. In a large cohort of colorectal cancer patients, regression-based prediction scores provide a higher prognostic value than classification-based scores. Our open-source regression approach offers a promising alternative for continuous biomarker analysis in computational pathology.

Interpretable machine learning model to predict surgical difficulty in laparoscopic resection for rectal cancer

Background

Laparoscopic total mesorectal excision (LaTME) is standard surgical methods for rectal cancer, and LaTME operation is a challenging procedure. This study is intended to use machine learning to develop and validate prediction models for surgical difficulty of LaTME in patients with rectal cancer and compare these models' performance.

Methods

We retrospectively collected the preoperative clinical and MRI pelvimetry parameter of rectal cancer patients who underwent laparoscopic total mesorectal resection from 2017 to 2022. The difficulty of LaTME was defined according to the scoring criteria reported by Escal. Patients were randomly divided into training group (80%) and test group (20%). We selected independent influencing features using the least absolute shrinkage and selection operator (LASSO) and multivariate logistic regression method. Adopt synthetic minority oversampling technique (SMOTE) to alleviate the class imbalance problem. Six machine learning model were developed: light gradient boosting machine (LGBM); categorical boosting (CatBoost); extreme gradient boost (XGBoost), logistic regression (LR); random forests (RF); multilayer perceptron (MLP). The area under receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity and F1 score were used to evaluate the performance of the model. The Shapley Additive Explanations (SHAP) analysis provided interpretation for the best machine learning model. Further decision curve analysis (DCA) was used to evaluate the clinical manifestations of the model.

Results

A total of 626 patients were included. LASSO regression analysis shows that tumor height, prognostic nutrition index (PNI), pelvic inlet, pelvic outlet, sacrococcygeal distance, mesorectal fat area and angle 5 (the angle between the apex of the sacral angle and the lower edge of the pubic bone) are the predictor variables of the machine learning model. In addition, the correlation heatmap shows that there is no significant correlation between these seven variables. When predicting the difficulty of LaTME surgery, the XGBoost model performed best among the six machine learning models (AUROC=0.855). Based on the decision curve analysis (DCA) results, the XGBoost model is also superior, and feature importance analysis shows that tumor height is the most important variable among the seven factors.

Conclusions

This study developed an XGBoost model to predict the difficulty of LaTME surgery. This model can help clinicians quickly and accurately predict the difficulty of surgery and adopt individualized surgical methods.

Detecting microsatellite instability in colorectal cancer using Transformer-based colonoscopy image classification and retrieval

Colorectal cancer (CRC) is a major global health concern, with microsatellite instability-high (MSI-H) being a defining characteristic of hereditary nonpolyposis colorectal cancer syndrome and affecting 15% of sporadic CRCs. Tumors with MSI-H have unique features and better prognosis compared to MSI-L and microsatellite stable (MSS) tumors. This study proposed establishing a MSI prediction model using more available and low-cost colonoscopy images instead of histopathology. The experiment utilized a database of 427 MSI-H and 1590 MSS colonoscopy images and vision Transformer (ViT) with different feature training approaches to establish the MSI prediction model. The accuracy of combining pre-trained ViT features was 84% with an area under the receiver operating characteristic curve of 0.86, which was better than that of DenseNet201 (80%, 0.80) in the experiment with support vector machine. The content-based image retrieval (CBIR) approach showed that ViT features can obtain a mean average precision of 0.81 compared to 0.79 of DenseNet201. ViT reduced the issues that occur in convolutional neural networks, including limited receptive field and gradient disappearance, and may be better at interpreting diagnostic information around tumors and surrounding tissues. By using CBIR, the presentation of similar images with the same MSI status would provide more convincing deep learning suggestions for clinical use.

Deep learning predicts therapy-relevant genetics in acute myeloid leukemia from Pappenheim-stained bone marrow smears

ABSTRACT

The detection of genetic aberrations is crucial for early therapy decisions in acute myeloid leukemia (AML) and recommended for all patients. Because genetic testing is expensive and time consuming, a need remains for cost-effective, fast, and broadly accessible tests to predict these aberrations in this aggressive malignancy. Here, we developed a novel fully automated end-to-end deep learning pipeline to predict genetic aberrations directly from single-cell images from scans of conventionally stained bone marrow smears already on the day of diagnosis. We used this pipeline to compile a multiterabyte data set of >2 000 000 single-cell images from diagnostic samples of 408 patients with AML. These images were then used to train convolutional neural networks for the prediction of various therapy-relevant genetic alterations. Moreover, we created a temporal test cohort data set of >444 000 single-cell images from further 71 patients with AML. We show that the models from our pipeline can significantly predict these subgroups with high areas under the curve of the receiver operating characteristic. Potential genotype-phenotype links were visualized with 2 different strategies. Our pipeline holds the potential to be used as a fast and inexpensive automated tool to screen patients with AML for therapy-relevant genetic aberrations directly from routine, conventionally stained bone marrow smears already on the day of diagnosis. It also creates a foundation to develop similar approaches for other bone marrow disorders in the future.

Deficient mismatch repair/microsatellite unstable colorectal cancer: therapeutic advances and questions

The microsatellite instability (MSI) phenotype is related to a deficiency of the DNA mismatch repair (dMMR) system and is observed in 5% of metastatic colorectal cancers (mCRCs). MSI/dMMR phenotype testing should be routine for all CRCs regardless of stage. Two complementary techniques with a high concordance (90-97%) allow us to determine the MSI/dMMR status of a tumor: immunohistochemistry and polymerase chain reaction. Since 2020 and the results of the phase III KEYNOTE 177 trial, pembrolizumab [anti-programmed cell death protein 1 (PD1)] is the new standard of care in first-line MSI/dMMR mCRC. To date, no combination of chemtotherapy ± targeted therapy with immune checkpoint inhibitors (ICIs) has been validated in the management of MSI/dMMR mCRC, and it is not known whether this combination would be beneficial. It is also unclear whether dual therapy with two ICIs is more effective than monotherapy. Several phase III trials are ongoing to answer these questions. Despite a high response rate and long-term benefit of a first line by anti-PD1, 30-50% of patients with MSI/dMMR mCRC experience an early or secondary progression. There are currently no validated predictive biomarkers of anti-PD1 ± anti-cytotoxic T lymphocyte antigen-4 resistance in patients with MSI/dMMR mCRC. In case of early progression on ICIs, the first two questions to consider are the possibility of pseudoprogression and the correct diagnosis of MSI/dMMR status. To date, there are no data on the use of adjuvant ICIs for MSI/dMMR resected colon cancers. By contrast, data are accumulating regarding the efficacy of neoadjuvant ICIs, with at least two-thirds of patients in the different trials in pathological complete response, making it possible to envisage 'Watch and wait' strategies in future.

Development and validation of a clinical prediction model for the risk of distal metastasis in intrahepatic cholangiocarcinoma: a real-world study

BACKGROUND

Cholangiocarcinoma (CCA) is a highly malignant and easily metastatic bile duct tumor with poor prognosis. We aimed at studying the associated risk factors affecting distal metastasis of CCA and using nomogram to guide clinicians in predicting distal metastasis of CCA.

METHODS

Based on inclusion and exclusion criteria, 345 patients with CCA were selected from the Fifth Medical Center of Chinese PLA General Hospital and were divided into distal metastases (N = 21) and non-distal metastases (N = 324). LASSO regression models were used to screen for relevant parameters and to compare basic clinical information between the two groups of patients. Risk factors for distal metastasis were identified based on the results of univariate and multivariate logistic regression analyses. The nomogram was established based on the results of multivariate logistic regression, and we drawn the corresponding correlation heat map. The predictive accuracy of the nomogram was evaluated by receiver operating characteristic (ROC) curves and calibration plots. The utility of the model in clinical applications was illustrated by applying decision curve analysis (DCA), and overall survival(OS) analysis was performed using the method of Kaplan-meier.

RESULTS

This study identified 4 independent risk factors for distal metastasis of CCA, including CA199, cholesterol, hypertension and margin invasion, and developed the nomogram based on this. The result of validation showed that the model had significant accuracy for diagnosis with the area under ROC (AUC) of 0.882 (95% CI: 0.843-0.914). Calibration plots and DCA showed that the model had high clinical utility.

CONCLUSIONS

This study established and validated a model of nomogram for predicting distal metastasis in patients with CCA. Based on this, it could guide clinicians to make better decisions and provide more accurate prognosis and treatment for patients with CCA.

Non-coding RNAs as Biomarkers for Survival in Colorectal Cancer Patients

Colorectal cancer (CRC) has a high incidence and fatality rate worldwide. It ranks second concerning death worldwide. Cancer patients are diagnosed with the disease at a later stage due to the absence of early diagnostic methods, which leads to increased death. With the help of recent advancements in the fields of diagnosis and therapy, the development of novel methods using new targets could be helpful for the long-term survival of CRC patients when CRC is detected early. However, the prognosis for the advanced stage of CRC is abysmal. New biomarkers are emerging as promising alternatives since they can be utilized for early detection of CRC, are simple to use, and non-invasive. Non-coding RNAs (ncRNAs) have been seen to have an aberrant expression in the development of many malignancies, including CRC. In the past two decades, much research has been done on non-coding RNAs, which may be valuable as biomarkers and targets for antitumor therapy. Non-coding RNAs can be employed in detecting and treating CRC. Non-coding RNAs play an essential role in regulating gene expression. This article reviews ncRNAs and their expression levels in CRC patients that could be used as potential biomarkers. Various ncRNAs have been associated with CRC, such as microRNAs, long non-coding RNAs, circular RNAs, etc. The expression of these non-coding RNAs may provide insights into the stages of cancer and the prognosis of cancer patients and therefore proper precautionary measures can be taken to decrease cancer-related deaths.

Use of artificial intelligence in the management of T1 colorectal cancer: a new tool in the arsenal or is deep learning out of its depth?

The field of artificial intelligence is rapidly evolving, and there has been an interest in its use to predict the risk of lymph node metastasis in T1 colorectal cancer. Accurately predicting lymph node invasion may result in fewer patients undergoing unnecessary surgeries; conversely, inadequate assessments will result in suboptimal oncological outcomes. This narrative review aims to summarize the current literature on deep learning for predicting the probability of lymph node metastasis in T1 colorectal cancer, highlighting areas of potential application and barriers that may limit its generalizability and clinical utility.

Artificial Intelligence in Predicting Microsatellite Instability and KRAS, BRAF Mutations from Whole-Slide Images in Colorectal Cancer: A Systematic Review

Mismatch repair deficiency (d-MMR)/microsatellite instability (MSI), KRAS, and BRAF mutational status are crucial for treating advanced colorectal cancer patients. Traditional methods like immunohistochemistry or polymerase chain reaction (PCR) can be challenged by artificial intelligence (AI) based on whole slide images (WSI) to predict tumor status. In this systematic review, we evaluated the role of AI in predicting MSI status, KRAS, and BRAF mutations in colorectal cancer. Studies published in PubMed up to June 2023 were included (n = 17), and we reported the risk of bias and the performance for each study. Some studies were impacted by the reduced number of slides included in the data set and the lack of external validation cohorts. Deep learning models for the d-MMR/MSI status showed a good performance in training cohorts (mean AUC = 0.89, [0.74-0.97]) but slightly less than expected in the validation cohort when available (mean AUC = 0.82, [0.63-0.98]). Contrary to the MSI status, the prediction of KRAS and BRAF mutations was less explored with a less robust methodology. The performance was lower, with a maximum of 0.77 in the training cohort, 0.58 in the validation cohort for KRAS, and 0.82 AUC in the training cohort for BRAF.

An efficient context-aware approach for whole-slide image classification

Computational pathology for gigapixel whole-slide images (WSIs) at slide level is helpful in disease diagnosis and remains challenging. We propose a context-aware approach termed WSI inspection via transformer (WIT) for slide-level classification via holistically modeling dependencies among patches on WSI. WIT automatically learns feature representation of WSI by aggregating features of all image patches. We evaluate classification performance of WIT and state-of-the-art baseline method. WIT achieved an accuracy of 82.1% (95% CI, 80.7%-83.3%) in the detection of 32 cancer types on the TCGA dataset, 0.918 (0.910-0.925) in diagnosis of cancer on the CPTAC dataset, and 0.882 (0.87-0.890) in the diagnosis of prostate cancer from needle biopsy slide, outperforming the baseline by 31.6%, 5.4%, and 9.3%, respectively. WIT can pinpoint the WSI regions that are most influential for its decision. WIT represents a new paradigm for computational pathology, facilitating the development of digital pathology tools.

Deep Learning and Colon Cancer Interpretation: Rise of the Machine

The rapidly evolving development of artificial intelligence (AI) has spurred the development of numerous algorithms that augment information obtained from routine pathologic review of hematoxylin and eosin-stained slides. AI tools that predict prognosis and underlying molecular alterations have been the focus of much of the research to date. The results of these studies highlight the tremendous potential of AI to enhance our pathology reports by providing rapid predictions of key features that influence therapy and outcomes.

Types of DNA damage and research progress

DNA damage is a modification in the structure of DNA under the influence of endogenous or exogenous factors. DNA damage can cause different types of diseases and is closely related to genetic mutations, cancer, and aging. The cause of the corresponding reaction process is essential for the study of related cancers and other genetically related diseases. Therefore, it is essential to gain a deeper understanding of the various types of DNA damage. This paper provides a comprehensive review of recent advances in the types of DNA damage and associated reaction processes, including damage to DNA bases, nucleotides, and strands, as well as the biological implications of the damage.

Validation of MSIntuit as an AI-based pre-screening tool for MSI detection from colorectal cancer histology slides

Mismatch Repair Deficiency (dMMR)/Microsatellite Instability (MSI) is a key biomarker in colorectal cancer (CRC). Universal screening of CRC patients for MSI status is now recommended, but contributes to increased workload for pathologists and delayed therapeutic decisions. Deep learning has the potential to ease dMMR/MSI testing and accelerate oncologist decision making in clinical practice, yet no comprehensive validation of a clinically approved tool has been conducted. We developed MSIntuit, a clinically approved artificial intelligence (AI) based pre-screening tool for MSI detection from haematoxylin-eosin (H&E) stained slides. After training on samples from The Cancer Genome Atlas (TCGA), a blind validation is performed on an independent dataset of 600 consecutive CRC patients. Inter-scanner reliability is studied by digitising each slide using two different scanners. MSIntuit yields a sensitivity of 0.96-0.98, a specificity of 0.47-0.46, and an excellent inter-scanner agreement (Cohen's κ: 0.82). By reaching high sensitivity comparable to gold standard methods while ruling out almost half of the non-MSI population, we show that MSIntuit can effectively serve as a pre-screening tool to alleviate MSI testing burden in clinical practice.

Prognostic significance of cyclin D1 expression pattern in HPV-negative oral and oropharyngeal carcinoma: A deep-learning approach

BACKGROUND

We aimed to establish image recognition and survival prediction models using a novel scoring system of cyclin D1 expression pattern in patients with human papillomavirus-negative oral or oropharyngeal squamous cell carcinoma.

METHODS

The clinicopathological data of 610 patients with human papillomavirus-negative oral/oropharyngeal squamous cell carcinoma were analyzed retrospectively. Cox univariate and multivariate risk regression analyses were performed to compare cyclin D1 expression pattern scoring with the traditional scoring method-cyclin D1 expression level scoring-in relation to patients' overall and progression-free survival. An image recognition model employing the cyclin D1 expression pattern scoring system was established by YOLOv5 algorithms. From this model, two independent survival prediction models were established using the DeepHit and DeepSurv algorithms.

RESULTS

Cyclin D1 had three expression patterns in oral and oropharyngeal squamous cell carcinoma cancer nests. Superior to cyclin D1 expression level scoring, cyclin D1 expression pattern scoring was significantly correlated with the prognosis of patients with oral squamous cell carcinoma (p < 0.0001) and oropharyngeal squamous cell carcinoma (p < 0.05). Moreover, it was an independent prognostic risk factor in both oral squamous cell carcinoma (p < 0.0001) and oropharyngeal squamous cell carcinoma (p < 0.05). The cyclin D1 expression pattern-derived image recognition model showed an average test set accuracy of 78.48% ± 4.31%. In the overall survival prediction models, the average concordance indices of the test sets established by DeepSurv and DeepHit were 0.71 ± 0.02 and 0.70 ± 0.01, respectively.

CONCLUSION

Combined with the image recognition model of the cyclin D1 expression pattern, the survival prediction model had a relatively good prediction effect on the overall survival prognosis of patients with human papillomavirus-negative oral or oropharyngeal squamous cell carcinoma.

Value of the Tumor-Stroma Ratio and Structural Heterogeneity Measured by a Novel Semiautomatic Image Analysis Technique for Predicting Survival in Patients With Colon Cancer

BACKGROUND

The tumor-stroma ratio and intratumor stromal heterogeneity have been identified as prognostic factors for several carcinomas. Recent advancements in image analysis technologies and their application to medicine have enabled detailed analysis of clinical data beyond human cognition.

OBJECTIVE

This study aimed to investigate the tumor-stroma ratio and intratumor stromal heterogeneity measured using a novel objective and semiautomatic method with image analysis.

DESIGN

A retrospective cohort design.

SETTINGS

Single institution.

PATIENTS

This study included patients who underwent curative colectomy for colon cancer.

MAIN OUTCOME MEASURES

The survival analyses between tumor-stroma ratio or intratumor stromal heterogeneity high and low groups after colectomy were assessed in multivariate analyses.

RESULTS

Two hundred patients were divided into 2 groups based on the median tumor-stroma ratio and intratumor stromal heterogeneity values. The 5-year overall survival and relapse-free survival rates after colectomy significantly differed between the high and low tumor-stroma ratio or intratumor stromal heterogeneity groups. Multivariate analysis identified low tumor-stroma ratio (HR: 1.90, p = 0.03) and high intratumor stromal heterogeneity (HR: 2.44, p = 0.002) as independent poor prognostic factors for relapse-free survival. The tumor-stroma ratio and intratumor stromal heterogeneity correlated with the duration from curative surgery to recurrence. Furthermore, postoperative recurrence within 2 years was predicted with higher accuracy by using the tumor-stroma ratio or intratumor stromal heterogeneity than by using the pathological stage. In a validation cohort, interobserver agreement was assessed by 2 observers, and Cohen's κ coefficient for the tumor-stroma ratio (κ value: 0.70) and intratumor stromal heterogeneity (κ value: 0.60) revealed a substantial interobserver agreement.

LIMITATIONS

This study was limited by its retrospective, single-institution design.

CONCLUSIONS

Tumor-stroma ratio and intratumor stromal heterogeneity calculated using image analysis software have potential as imaging biomarkers for predicting the survival of patients with colon cancer after colectomy. See Video Abstract at http://links.lww.com/DCR/C114 .

VALOR DE LA PROPORCIN DE ESTROMA TUMORAL Y LA HETEROGENEIDAD ESTRUCTURAL MEDIDOS POR UNA NUEVA TCNICA DE ANLISIS DE IMGENES SEMIAUTOMTICA PARA PREDECIR LA SUPERVIVENCIA EN PACIENTES CON CNCER DE COLON

ANTECEDENTES:La proporción de estroma tumoral y la heterogeneidad del estroma intratumoral han sido identificados como factores pronósticos para varios tipos de carcinomas. Los avances recientes en cuanto a las tecnologías de análisis de imágenes y sus aplicaciones en la medicina, han permitido un análisis detallado de los datos clínicos más allá del conocimiento humano.OBJETIVO:Investigar la relación del estroma tumoral y la heterogeneidad del estroma intratumoral calculados mediante un nuevo método objetivo y semiautomático para el análisis de imágenes.DISEÑO:Diseño de cohorte retrospectivo.AJUSTES:Institución única.PACIENTES:Pacientes sometidos a colectomía curativa por cáncer de colon.PRINCIPALES MEDIDAS DE RESULTADO:Los análisis de supervivencia entre la relación del estroma tumoral o la heterogeneidad del estroma intratumoral entre los grupos con valores altos y bajos tras la colectomía, fueron evaluados en análisis multivariados.RESULTADOS:Fueron divididos 200 pacientes en dos grupos basados en la mediana de la proporción con respecto a los valores del estroma tumoral y la heterogeneidad del estroma intratumoral. Las tasas de supervivencia general a los 5 años y de supervivencia libre de recaídas después de la colectomía, difirieron significativamente entre los grupos con índice de estroma tumoral o heterogeneidad del estroma intratumoral altos y bajos. El análisis multivariante identificó una proporción de estroma tumoral baja (cociente de riesgos instantáneos: 1.90, p = 0.03) y una heterogeneidad estromal intratumoral alta (cociente de riesgos instantáneos: 2.44, p = 0.002) como factores independientes de mal pronóstico para la supervivencia libre de recaídas. La proporción de estroma tumoral y la heterogeneidad del estroma intratumoral se correlacionaron con la duración de la recurrencia desde la cirugía.Además, la recurrencia posoperatoria dentro de los 2 años se predijo con mayor precisión mediante el uso del índice de estroma tumoral o la heterogeneidad del estroma intratumoral que mediante el uso del estadio patológico. En una cohorte de validación, la concordancia interobservador fue evaluada por dos observadores, y el coeficiente Kappa de Cohen para la proporción de estroma tumoral y la heterogeneidad estromal intratumoral reveló una concordancia interobservador sustancial (valor Kappa: 0.70, 0.60, respectivamente).LIMITACIONES:Este estudio estuvo limitado por su diseño retrospectivo de una sola institución.CONCLUSIONES:La proporción del estroma tumoral y la heterogeneidad del estroma intratumoral calculadas mediante software de análisis de imágenes tienen potencial como biomarcadores de imagen para predecir la supervivencia de los pacientes con cáncer de colon tras la colectomía. Consulte Video Resumen en http://links.lww.com/DCR/C114 . (Traducción-Dr. Osvaldo Gauto ).

Deep Learning-Enabled Diagnosis of Liver Adenocarcinoma

BACKGROUND & AIMS

Diagnosis of adenocarcinoma in the liver is a frequent scenario in routine pathology and has a critical impact on clinical decision making. However, rendering a correct diagnosis can be challenging, and often requires the integration of clinical, radiologic, and immunohistochemical information. We present a deep learning model (HEPNET) to distinguish intrahepatic cholangiocarcinoma from colorectal liver metastasis, as the most frequent primary and secondary forms of liver adenocarcinoma, with clinical grade accuracy using H&E-stained whole-slide images.

METHODS

HEPNET was trained on 714,589 image tiles from 456 patients who were randomly selected in a stratified manner from a pool of 571 patients who underwent surgical resection or biopsy at Heidelberg University Hospital. Model performance was evaluated on a hold-out internal test set comprising 115 patients and externally validated on 159 patients recruited at Mainz University Hospital.

RESULTS

On the hold-out internal test set, HEPNET achieved an area under the receiver operating characteristic curve of 0.994 (95% CI, 0.989-1.000) and an accuracy of 96.522% (95% CI, 94.521%-98.694%) at the patient level. Validation on the external test set yielded an area under the receiver operating characteristic curve of 0.997 (95% CI, 0.995-1.000), corresponding to an accuracy of 98.113% (95% CI, 96.907%-100.000%). HEPNET surpassed the performance of 6 pathology experts with different levels of experience in a reader study of 50 patients (P = .0005), boosted the performance of resident pathologists to the level of senior pathologists, and reduced potential downstream analyses.

CONCLUSIONS

We provided a ready-to-use tool with clinical grade performance that may facilitate routine pathology by rendering a definitive diagnosis and guiding ancillary testing. The incorporation of HEPNET into pathology laboratories may optimize the diagnostic workflow, complemented by test-related labor and cost savings.

Treatment response prediction of neoadjuvant chemotherapy for rectal cancer by deep learning of colonoscopy images

In current clinical practice, several treatment methods, including neoadjuvant therapy, are being developed to improve overall survival or local recurrence rates for locally advanced rectal cancer. The response to neoadjuvant therapy is usually evaluated using imaging data collected before and after preoperative treatment or postsurgical pathological diagnosis. However, there is a need to accurately predict the response to preoperative treatment before treatment is administered. The present study used a deep learning network to examine colonoscopy images and construct a model to predict the response of rectal cancer to neoadjuvant chemotherapy. A total of 53 patients who underwent preoperative chemotherapy followed by radical resection for advanced rectal cancer at the Osaka University Hospital between January 2011 and August 2019 were retrospectively analyzed. A convolutional neural network model was constructed using 403 images from 43 patients as the learning set. The diagnostic accuracy of the deep learning model was evaluated using 84 images from 10 patients as the validation set. The model demonstrated a sensitivity, specificity, accuracy, positive predictive value and area under the curve of 77.6% (38/49), 62.9% (22/33), 71.4% (60/84), 74.5% (38/51) and 0.713, respectively, in predicting a poor response to neoadjuvant therapy. Overall, deep learning of colonoscopy images may contribute to an accurate prediction of the response of rectal cancer to neoadjuvant chemotherapy.

A competing risk nomogram to predict cancer-specific mortality of patients with late-onset colorectal cancer

OBJECTIVE

This study aimed to compare the clinical characteristics and survival differences between early-onset colorectal cancer (EOCRC) patients and late-onset colorectal cancer (LOCRC) patients, identify the risk factors for cancer-specific mortality (CSM) in LOCRC patients and construct a mortality risk assessment nomogram.

METHODS

CRC patients diagnosed pathologically between 2010 and 2019 in the SEER database were included and divided into the early-onset group and the late-onset group, and the late-onset group was divided into the training and validation sets. The Fine-Gray competing risk model was applied to analyze the prognostic factors of LOCRC patients and establish a competing risk nomogram for CSM.

RESULTS

There are differences in the distribution of multiple clinical features between the early-onset group and the late-onset group. Age, tumor size, histological type, pathological grading, T stage, N stage, M stage, SEER stage, primary tumor surgery, metastatic lesion surgery, radiotherapy, chemotherapy, neural invasion, and carcinoembryonic antigen (CEA) were independent influencing factors of the CSM rate in LOCRC patients. The C-index of the prognosis model outweighed 0.8, and the calibration curves fitted the reference line well.

CONCLUSION

The CSM competing risk nomogram for LOCRC patients in this study had acceptable predictive performance that could be applied to the clinic.

Preparing Data for Artificial Intelligence in Pathology with Clinical-Grade Performance

The pathology is decisive for disease diagnosis but relies heavily on experienced pathologists. In recent years, there has been growing interest in the use of artificial intelligence in pathology (AIP) to enhance diagnostic accuracy and efficiency. However, the impressive performance of deep learning-based AIP in laboratory settings often proves challenging to replicate in clinical practice. As the data preparation is important for AIP, the paper has reviewed AIP-related studies in the PubMed database published from January 2017 to February 2022, and 118 studies were included. An in-depth analysis of data preparation methods is conducted, encompassing the acquisition of pathological tissue slides, data cleaning, screening, and subsequent digitization. Expert review, image annotation, dataset division for model training and validation are also discussed. Furthermore, we delve into the reasons behind the challenges in reproducing the high performance of AIP in clinical settings and present effective strategies to enhance AIP's clinical performance. The robustness of AIP depends on a randomized collection of representative disease slides, incorporating rigorous quality control and screening, correction of digital discrepancies, reasonable annotation, and sufficient data volume. Digital pathology is fundamental in clinical-grade AIP, and the techniques of data standardization and weakly supervised learning methods based on whole slide image (WSI) are effective ways to overcome obstacles of performance reproduction. The key to performance reproducibility lies in having representative data, an adequate amount of labeling, and ensuring consistency across multiple centers. Digital pathology for clinical diagnosis, data standardization and the technique of WSI-based weakly supervised learning will hopefully build clinical-grade AIP.

CT-based radiomics nomogram for the preoperative prediction of microsatellite instability and clinical outcomes in colorectal cancer: a multicentre study

AIM

To develop and validate a computed tomography (CT)-based radiomics nomogram for preoperative prediction of microsatellite instability (MSI) status and clinical outcomes in colorectal cancer (CRC) patients.

MATERIALS AND METHODS

This retrospective study enrolled 497 CRC patients from three centres. Least absolute shrinkage and selection operator regression was utilised for feature selection and constructing the radiomics signature. Univariate and multivariate logistic regression analyses were employed to identify significant clinical variables. The radiomics nomogram was constructed by integrating the radiomics signature and the identified clinical variables. The performance of the nomogram was evaluated through receiver operating characteristic curves, calibration curves, and decision curve analysis. Kaplan-Meier analysis was performed to investigate the prognostic value of the nomogram.

RESULTS

The radiomics signature comprised 10 radiomics features associated with MSI status. The nomogram, integrating the radiomics signature and independent predictors (age, location, and thickness), demonstrated favourable calibration and discrimination, achieving areas under the receiver operating characteristic (ROC) curves (AUCs) of 0.89 (95% confidence interval [CI]: 0.83-0.95), 0.87 (95% CI: 0.79-0.95), 0.88 (95% CI: 0.81-0.96), and 0.86 (95% CI: 0.78-0.93) in the training cohort, internal validation cohort, and two external validation cohorts, respectively. The nomogram exhibited superior performance compared to the clinical model (p<0.05). Additionally, survival analysis demonstrated that the nomogram successfully stratified stage II CRC patients based on prognosis (hazard ratio [HR]: 0.357, p=0.022).

CONCLUSION

The radiomics nomogram demonstrated promising performance in predicting MSI status and stratifying the prognosis of patients with CRC.

Predicting Prostate Cancer Molecular Subtype With Deep Learning on Histopathologic Images

Microscopic examination of prostate cancer has failed to reveal a reproducible association between molecular and morphologic features. However, deep-learning algorithms trained on hematoxylin and eosin (H&E)-stained whole slide images (WSI) may outperform the human eye and help to screen for clinically-relevant genomic alterations. We created deep-learning algorithms to identify prostate tumors with underlying ETS-related gene (ERG) fusions or PTEN deletions using the following 4 stages: (1) automated tumor identification, (2) feature representation learning, (3) classification, and (4) explainability map generation. A novel transformer-based hierarchical architecture was trained on a single representative WSI of the dominant tumor nodule from a radical prostatectomy (RP) cohort with known ERG/PTEN status (n = 224 and n = 205, respectively). Two distinct vision transformer-based networks were used for feature extraction, and a distinct transformer-based model was used for classification. The ERG algorithm performance was validated across 3 RP cohorts, including 64 WSI from the pretraining cohort (AUC, 0.91) and 248 and 375 WSI from 2 independent RP cohorts (AUC, 0.86 and 0.89, respectively). In addition, we tested the ERG algorithm performance in 2 needle biopsy cohorts comprised of 179 and 148 WSI (AUC, 0.78 and 0.80, respectively). Focusing on cases with homogeneous (clonal) PTEN status, PTEN algorithm performance was assessed using 50 WSI reserved from the pretraining cohort (AUC, 0.81), 201 and 337 WSI from 2 independent RP cohorts (AUC, 0.72 and 0.80, respectively), and 151 WSI from a needle biopsy cohort (AUC, 0.75). For explainability, the PTEN algorithm was also applied to 19 WSI with heterogeneous (subclonal) PTEN loss, where the percentage tumor area with predicted PTEN loss correlated with that based on immunohistochemistry (r = 0.58, P = .0097). These deep-learning algorithms to predict ERG/PTEN status prove that H&E images can be used to screen for underlying genomic alterations in prostate cancer.

PAIP 2020: Microsatellite instability prediction in colorectal cancer

Microsatellite instability (MSI) refers to alterations in the length of simple repetitive genomic sequences. MSI status serves as a prognostic and predictive factor in colorectal cancer. The MSI-high status is a good prognostic factor in stage II/III cancer, and predicts a lack of benefit to adjuvant fluorouracil chemotherapy in stage II cancer but a good response to immunotherapy in stage IV cancer. Therefore, determining MSI status in patients with colorectal cancer is important for identifying the appropriate treatment protocol. In the Pathology Artificial Intelligence Platform (PAIP) 2020 challenge, artificial intelligence researchers were invited to predict MSI status based on colorectal cancer slide images. Participants were required to perform two tasks. The primary task was to classify a given slide image as belonging to either the MSI-high or the microsatellite-stable group. The second task was tumor area segmentation to avoid ties with the main task. A total of 210 of the 495 participants enrolled in the challenge downloaded the images, and 23 teams submitted their final results. Seven teams from the top 10 participants agreed to disclose their algorithms, most of which were convolutional neural network-based deep learning models, such as EfficientNet and UNet. The top-ranked system achieved the highest F1 score (0.9231). This paper summarizes the various methods used in the PAIP 2020 challenge. This paper supports the effectiveness of digital pathology for identifying the relationship between colorectal cancer and the MSI characteristics.

Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study

Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.

Generalizability of Self-Supervised Training Models for Digital Pathology: A Multicountry Comparison in Colorectal Cancer

PURPOSE

In this multicountry study, we aim to explore the effectiveness of self-supervised learning (SSL) in colorectal cancer (CRC)-related predictive tasks using large amount of unlabeled digital pathology imaging data.

METHODS

We adopted SimSiam to conduct self-supervised pretraining on two large whole-slide image CRC data sets from the United States and Australia. The SSL pretrained encoder is then used in several predictive tasks, including supervised predictive tasks (tissue classification, microsatellite instability v microsatellite stability classification), and weakly supervised predictive tasks (polyp type classification and adenoma grading, and 5-year survival prediction). Performance on the tasks was compared between models using SSL pretraining and those using ImageNet pretraining, and performance for one-country pretraining was compared with two-country pretraining.

RESULTS

We demonstrate that SSL pretraining outperforms ImageNet pretraining in predictive tasks, that is, SSL pretraining outperforms the ImageNet pretraining by 3.01% of F 1 score on average over supervised predictive tasks and 1.53% of AUC on average over weakly supervised predictive tasks. Furthermore, two-country SSL pretraining has shown more stable performance than single-country pretraining, that is, two-country pretraining outperforms at least one of the single-country pretrainings by 1.93% of F 1 on average over supervised predictive tasks and 1.36% of AUC on average over weakly-supervised predictive tasks.

CONCLUSION

We find that using unlabeled image data for SSL pretraining in CRC related tasks is more effective than using ImageNet pretraining. Furthermore, SSL pretraining using data from multiple countries achieve more stable performance and better generalization than single-country pretraining.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

BACKGROUND AND OBJECTIVE

Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers.

METHODS

Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model.

RESULTS

The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision.

CONCLUSIONS

Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.