Harnessing multimodal data integration to advance precision oncology

Benchmarking feature selection and feature extraction methods to improve the performances of machine-learning algorithms for patient classification using metabolomics biomedical data

Objective

Classification tasks are an open challenge in the field of biomedicine. While several machine-learning techniques exist to accomplish this objective, several peculiarities associated with biomedical data, especially when it comes to omics measurements, prevent their use or good performance achievements. Omics approaches aim to understand a complex biological system through systematic analysis of its content at the molecular level. On the other hand, omics data are heterogeneous, sparse and affected by the classical "curse of dimensionality" problem, i.e. having much fewer observation, samples (n) than omics features (p). Furthermore, a major problem with multi-omics data is the imbalance either at the class or feature level. The objective of this work is to study whether feature extraction and/or feature selection techniques can improve the performances of classification machine-learning algorithms on omics measurements.

Methods

Among all omics, metabolomics has emerged as a powerful tool in cancer research, facilitating a deeper understanding of the complex metabolic landscape associated with tumorigenesis and tumor progression. Thus, we selected three publicly available metabolomics datasets, and we applied several feature extraction techniques both linear and non-linear, coupled or not with feature selection methods, and evaluated the performances regarding patient classification in the different configurations for the three datasets.

Results

We provide general workflow and guidelines on when to use those techniques depending on the characteristics of the data available. To further test the extension of our approach to other omics data, we have included a transcriptomics and a proteomics data. Overall, for all datasets, we showed that applying supervised feature selection improves the performances of feature extraction methods for classification purposes. Scripts used to perform all analyses are available at: https://github.com/Plant-Net/Metabolomic_project/.

Enhancing psychiatric rehabilitation outcomes through a multimodal multitask learning model based on BERT and TabNet: An approach for personalized treatment and improved decision-making

Evaluating the rehabilitation status of individuals with serious mental illnesses (SMI) necessitates a comprehensive analysis of multimodal data, including unstructured text records and structured diagnostic data. However, progress in the effective assessment of rehabilitation status remains limited. Our study develops a deep learning model integrating Bidirectional Encoder Representations from Transformers (BERT) and TabNet through a late fusion strategy to enhance rehabilitation prediction, including referral risk, dangerous behaviors, self-awareness, and medication adherence, in patients with SMI. BERT processes unstructured textual data, such as doctor's notes, whereas TabNet manages structured diagnostic information. The model's interpretability function serves to assist healthcare professionals in understanding the model's predictive decisions, improving patient care. Our model exhibited excellent predictive performance for all four tasks, with an accuracy exceeding 0.78 and an area under the curve of 0.70. In addition, a series of tests proved the model's robustness, fairness, and interpretability. This study combines multimodal and multitask learning strategies into a model and applies it to rehabilitation assessment tasks, offering a promising new tool that can be seamlessly integrated with the clinical workflow to support the provision of optimized patient care.

A guide to artificial intelligence for cancer researchers

Artificial intelligence (AI) has been commoditized. It has evolved from a specialty resource to a readily accessible tool for cancer researchers. AI-based tools can boost research productivity in daily workflows, but can also extract hidden information from existing data, thereby enabling new scientific discoveries. Building a basic literacy in these tools is useful for every cancer researcher. Researchers with a traditional biological science focus can use AI-based tools through off-the-shelf software, whereas those who are more computationally inclined can develop their own AI-based software pipelines. In this article, we provide a practical guide for non-computational cancer researchers to understand how AI-based tools can benefit them. We convey general principles of AI for applications in image analysis, natural language processing and drug discovery. In addition, we give examples of how non-computational researchers can get started on the journey to productively use AI in their own work.

Integrating spatial and single-cell transcriptomics reveals tumor heterogeneity and intercellular networks in colorectal cancer

Single cell RNA sequencing (scRNA-seq), a powerful tool for studying the tumor microenvironment (TME), does not preserve/provide spatial information on tissue morphology and cellular interactions. To understand the crosstalk between diverse cellular components in proximity in the TME, we performed scRNA-seq coupled with spatial transcriptomic (ST) assay to profile 41,700 cells from three colorectal cancer (CRC) tumor-normal-blood pairs. Standalone scRNA-seq analyses revealed eight major cell populations, including B cells, T cells, Monocytes, NK cells, Epithelial cells, Fibroblasts, Mast cells, Endothelial cells. After the identification of malignant cells from epithelial cells, we observed seven subtypes of malignant cells that reflect heterogeneous status in tumor, including tumor_CAV1, tumor_ATF3_JUN | FOS, tumor_ZEB2, tumor_VIM, tumor_WSB1, tumor_LXN, and tumor_PGM1. By transferring the cellular annotations obtained by scRNA-seq to ST spots, we annotated four regions in a cryosection from CRC patients, including tumor, stroma, immune infiltration, and colon epithelium regions. Furthermore, we observed intensive intercellular interactions between stroma and tumor regions which were extremely proximal in the cryosection. In particular, one pair of ligands and receptors (C5AR1 and RPS19) was inferred to play key roles in the crosstalk of stroma and tumor regions. For the tumor region, a typical feature of TMSB4X-high expression was identified, which could be a potential marker of CRC. The stroma region was found to be characterized by VIM-high expression, suggesting it fostered a stromal niche in the TME. Collectively, single cell and spatial analysis in our study reveal the tumor heterogeneity and molecular interactions in CRC TME, which provides insights into the mechanisms underlying CRC progression and may contribute to the development of anticancer therapies targeting on non-tumor components, such as the extracellular matrix (ECM) in CRC. The typical genes we identified may facilitate to new molecular subtypes of CRC.

Artificial Intelligence in Oncology: Current Landscape, Challenges, and Future Directions

Artificial intelligence (AI) in oncology is advancing beyond algorithm development to integration into clinical practice. This review describes the current state of the field, with a specific focus on clinical integration. AI applications are structured according to cancer type and clinical domain, focusing on the four most common cancers and tasks of detection, diagnosis, and treatment. These applications encompass various data modalities, including imaging, genomics, and medical records. We conclude with a summary of existing challenges, evolving solutions, and potential future directions for the field.

SIGNIFICANCE

AI is increasingly being applied to all aspects of oncology, where several applications are maturing beyond research and development to direct clinical integration. This review summarizes the current state of the field through the lens of clinical translation along the clinical care continuum. Emerging areas are also highlighted, along with common challenges, evolving solutions, and potential future directions for the field.

Artificial intelligence in surgery

Artificial intelligence (AI) is rapidly emerging in healthcare, yet applications in surgery remain relatively nascent. Here we review the integration of AI in the field of surgery, centering our discussion on multifaceted improvements in surgical care in the preoperative, intraoperative and postoperative space. The emergence of foundation model architectures, wearable technologies and improving surgical data infrastructures is enabling rapid advances in AI interventions and utility. We discuss how maturing AI methods hold the potential to improve patient outcomes, facilitate surgical education and optimize surgical care. We review the current applications of deep learning approaches and outline a vision for future advances through multimodal foundation models.

Exploratory drug discovery in breast cancer patients: A multimodal deep learning approach to identify novel drug candidates targeting RTK signaling

Breast cancer, a highly formidable and diverse malignancy predominantly affecting women globally, poses a significant threat due to its intricate genetic variability, rendering it challenging to diagnose accurately. Various therapies such as immunotherapy, radiotherapy, and diverse chemotherapy approaches like drug repurposing and combination therapy are widely used depending on cancer subtype and metastasis severity. Our study revolves around an innovative drug discovery strategy targeting potential drug candidates specific to RTK signalling, a prominently targeted receptor class in cancer. To accomplish this, we have developed a multimodal deep neural network (MM-DNN) based QSAR model integrating omics datasets to elucidate genomic, proteomic expression data, and drug responses, validated rigorously. The results showcase an R2 value of 0.917 and an RMSE value of 0.312, affirming the model's commendable predictive capabilities. Structural analogs of drug molecules specific to RTK signalling were sourced from the PubChem database, followed by meticulous screening to eliminate dissimilar compounds. Leveraging the MM-DNN-based QSAR model, we predicted the biological activity of these molecules, subsequently clustering them into three distinct groups. Feature importance analysis was performed. Consequently, we successfully identified prime drug candidates tailored for each potential downstream regulatory protein within the RTK signalling pathway. This method makes the early stages of drug development faster by removing inactive compounds, providing a hopeful path in combating breast cancer.

A comprehensive review of machine learning techniques for multi-omics data integration: challenges and applications in precision oncology

Multi-omics data play a crucial role in precision medicine, mainly to understand the diverse biological interaction between different omics. Machine learning approaches have been extensively employed in this context over the years. This review aims to comprehensively summarize and categorize these advancements, focusing on the integration of multi-omics data, which includes genomics, transcriptomics, proteomics and metabolomics, alongside clinical data. We discuss various machine learning techniques and computational methodologies used for integrating distinct omics datasets and provide valuable insights into their application. The review emphasizes both the challenges and opportunities present in multi-omics data integration, precision medicine and patient stratification, offering practical recommendations for method selection in various scenarios. Recent advances in deep learning and network-based approaches are also explored, highlighting their potential to harmonize diverse biological information layers. Additionally, we present a roadmap for the integration of multi-omics data in precision oncology, outlining the advantages, challenges and implementation difficulties. Hence this review offers a thorough overview of current literature, providing researchers with insights into machine learning techniques for patient stratification, particularly in precision oncology. Contact:  anirban@klyuniv.ac.in.

Deep Multimodal Fusion of Data With Heterogeneous Dimensionality via Projective Networks

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D + 3D, 2D + 2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D + 2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer

Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.

Best practices for multimodal clinical data management and integration: An atopic dermatitis research case

BACKGROUND

In clinical research on multifactorial diseases such as atopic dermatitis, data-driven medical research has become more widely used as means to clarify diverse pathological conditions and to realize precision medicine. However, modern clinical data, characterized as large-scale, multimodal, and multi-center, causes difficulties in data integration and management, which limits productivity in clinical data science.

METHODS

We designed a generic data management flow to collect, cleanse, and integrate data to handle different types of data generated at multiple institutions by 10 types of clinical studies. We developed MeDIA (Medical Data Integration Assistant), a software to browse the data in an integrated manner and extract subsets for analysis.

RESULTS

MeDIA integrates and visualizes data and information on research participants obtained from multiple studies. It then provides a sophisticated interface that supports data management and helps data scientists retrieve the data sets they need. Furthermore, the system promotes the use of unified terms such as identifiers or sampling dates to reduce the cost of pre-processing by data analysts. We also propose best practices in clinical data management flow, which we learned from the development and implementation of MeDIA.

CONCLUSIONS

The MeDIA system solves the problem of multimodal clinical data integration, from complex text data such as medical records to big data such as omics data from a large number of patients. The system and the proposed best practices can be applied not only to allergic diseases but also to other diseases to promote data-driven medical research.

Artificial intelligence in neuro-oncology: advances and challenges in brain tumor diagnosis, prognosis, and precision treatment

This review delves into the most recent advancements in applying artificial intelligence (AI) within neuro-oncology, specifically emphasizing work on gliomas, a class of brain tumors that represent a significant global health issue. AI has brought transformative innovations to brain tumor management, utilizing imaging, histopathological, and genomic tools for efficient detection, categorization, outcome prediction, and treatment planning. Assessing its influence across all facets of malignant brain tumor management- diagnosis, prognosis, and therapy- AI models outperform human evaluations in terms of accuracy and specificity. Their ability to discern molecular aspects from imaging may reduce reliance on invasive diagnostics and may accelerate the time to molecular diagnoses. The review covers AI techniques, from classical machine learning to deep learning, highlighting current applications and challenges. Promising directions for future research include multimodal data integration, generative AI, large medical language models, precise tumor delineation and characterization, and addressing racial and gender disparities. Adaptive personalized treatment strategies are also emphasized for optimizing clinical outcomes. Ethical, legal, and social implications are discussed, advocating for transparency and fairness in AI integration for neuro-oncology and providing a holistic understanding of its transformative impact on patient care.

Transfer learning-based PET/CT three-dimensional convolutional neural network fusion of image and clinical information for prediction of EGFR mutation in lung adenocarcinoma

BACKGROUND

To introduce a three-dimensional convolutional neural network (3D CNN) leveraging transfer learning for fusing PET/CT images and clinical data to predict EGFR mutation status in lung adenocarcinoma (LADC).

METHODS

Retrospective data from 516 LADC patients, encompassing preoperative PET/CT images, clinical information, and EGFR mutation status, were divided into training (n = 404) and test sets (n = 112). Several deep learning models were developed utilizing transfer learning, involving CT-only and PET-only models. A dual-stream model fusing PET and CT and a three-stream transfer learning model (TS_TL) integrating clinical data were also developed. Image preprocessing includes semi-automatic segmentation, resampling, and image cropping. Considering the impact of class imbalance, the performance of the model was evaluated using ROC curves and AUC values.

RESULTS

TS_TL model demonstrated promising performance in predicting the EGFR mutation status, with an AUC of 0.883 (95%CI = 0.849-0.917) in the training set and 0.730 (95%CI = 0.629-0.830) in the independent test set. Particularly in advanced LADC, the model achieved an AUC of 0.871 (95%CI = 0.823-0.919) in the training set and 0.760 (95%CI = 0.638-0.881) in the test set. The model identified distinct activation areas in solid or subsolid lesions associated with wild and mutant types. Additionally, the patterns captured by the model were significantly altered by effective tyrosine kinase inhibitors treatment, leading to notable changes in predicted mutation probabilities.

CONCLUSION

PET/CT deep learning model can act as a tool for predicting EGFR mutation in LADC. Additionally, it offers clinicians insights for treatment decisions through evaluations both before and after treatment.

Building Flexible, Scalable, and Machine Learning-Ready Multimodal Oncology Datasets

The advancements in data acquisition, storage, and processing techniques have resulted in the rapid growth of heterogeneous medical data. Integrating radiological scans, histopathology images, and molecular information with clinical data is essential for developing a holistic understanding of the disease and optimizing treatment. The need for integrating data from multiple sources is further pronounced in complex diseases such as cancer for enabling precision medicine and personalized treatments. This work proposes Multimodal Integration of Oncology Data System (MINDS)-a flexible, scalable, and cost-effective metadata framework for efficiently fusing disparate data from public sources such as the Cancer Research Data Commons (CRDC) into an interconnected, patient-centric framework. MINDS consolidates over 41,000 cases from across repositories while achieving a high compression ratio relative to the 3.78 PB source data size. It offers sub-5-s query response times for interactive exploration. MINDS offers an interface for exploring relationships across data types and building cohorts for developing large-scale multimodal machine learning models. By harmonizing multimodal data, MINDS aims to potentially empower researchers with greater analytical ability to uncover diagnostic and prognostic insights and enable evidence-based personalized care. MINDS tracks granular end-to-end data provenance, ensuring reproducibility and transparency. The cloud-native architecture of MINDS can handle exponential data growth in a secure, cost-optimized manner while ensuring substantial storage optimization, replication avoidance, and dynamic access capabilities. Auto-scaling, access controls, and other mechanisms guarantee pipelines' scalability and security. MINDS overcomes the limitations of existing biomedical data silos via an interoperable metadata-driven approach that represents a pivotal step toward the future of oncology data integration.

[Explainable artificial intelligence in pathology]

With the advancements in precision medicine, the demands on pathological diagnostics have increased, requiring standardized, quantitative, and integrated assessments of histomorphological and molecular pathological data. Great hopes are placed in artificial intelligence (AI) methods, which have demonstrated the ability to analyze complex clinical, histological, and molecular data for disease classification, biomarker quantification, and prognosis estimation. This paper provides an overview of the latest developments in pathology AI, discusses the limitations, particularly concerning the black box character of AI, and describes solutions to make decision processes more transparent using methods of so-called explainable AI (XAI).

Management of patients with advanced-stage HER2-positive breast cancer: current evidence and future perspectives

Amplification and/or overexpression of ERBB2, the gene encoding HER2, can be found in 15-20% of invasive breast cancers and is associated with an aggressive phenotype and poor clinical outcomes. Relentless research efforts in molecular biology and drug development have led to the implementation of several HER2-targeted therapies, including monoclonal antibodies, tyrosine-kinase inhibitors and antibody-drug conjugates, constituting one of the best examples of bench-to-bedside translation in oncology. Each individual drug class has improved patient outcomes and, importantly, the combinatorial and sequential use of different HER2-targeted therapies has increased cure rates in the early stage disease setting and substantially prolonged survival for patients with advanced-stage disease. In this Review, we describe key steps in the development of the modern paradigm for the treatment of HER2-positive advanced-stage breast cancer, including selecting and sequencing new-generation HER2-targeted therapies, and summarize efficacy and safety outcomes from pivotal studies. We then outline the factors that are currently known to be related to resistance to HER2-targeted therapies, such as HER2 intratumoural heterogeneity, activation of alternative signalling pathways and immune escape mechanisms, as well as potential strategies that might be used in the future to overcome this resistance and further improve patient outcomes.

Integrating imaging and genomic data for the discovery of distinct glioblastoma subtypes: a joint learning approach

Glioblastoma is a highly heterogeneous disease, with variations observed at both phenotypical and molecular levels. Personalized therapies would be facilitated by non-invasive in vivo approaches for characterizing this heterogeneity. In this study, we developed unsupervised joint machine learning between radiomic and genomic data, thereby identifying distinct glioblastoma subtypes. A retrospective cohort of 571 IDH-wildtype glioblastoma patients were included in the study, and pre-operative multi-parametric MRI scans and targeted next-generation sequencing (NGS) data were collected. L21-norm minimization was used to select a subset of 12 radiomic features from the MRI scans, and 13 key driver genes from the five main signal pathways most affected in glioblastoma were selected from the genomic data. Subtypes were identified using a joint learning approach called Anchor-based Partial Multi-modal Clustering on both radiomic and genomic modalities. Kaplan-Meier analysis identified three distinct glioblastoma subtypes: high-risk, medium-risk, and low-risk, based on overall survival outcome (p < 0.05, log-rank test; Hazard Ratio = 1.64, 95% CI 1.17-2.31, Cox proportional hazard model on high-risk and low-risk subtypes). The three subtypes displayed different phenotypical and molecular characteristics in terms of imaging histogram, co-occurrence of genes, and correlation between the two modalities. Our findings demonstrate the synergistic value of integrated radiomic signatures and molecular characteristics for glioblastoma subtyping. Joint learning on both modalities can aid in better understanding the molecular basis of phenotypical signatures of glioblastoma, and provide insights into the biological underpinnings of tumor formation and progression.

A machine learning and deep learning-based integrated multi-omics technique for leukemia prediction

In recent years, scientific data on cancer has expanded, providing potential for a better understanding of malignancies and improved tailored care. Advances in Artificial Intelligence (AI) processing power and algorithmic development position Machine Learning (ML) and Deep Learning (DL) as crucial players in predicting Leukemia, a blood cancer, using integrated multi-omics technology. However, realizing these goals demands novel approaches to harness this data deluge. This study introduces a novel Leukemia diagnosis approach, analyzing multi-omics data for accuracy using ML and DL algorithms. ML techniques, including Random Forest (RF), Naive Bayes (NB), Decision Tree (DT), Logistic Regression (LR), Gradient Boosting (GB), and DL methods such as Recurrent Neural Networks (RNN) and Feedforward Neural Networks (FNN) are compared. GB achieved 97 % accuracy in ML, while RNN outperformed by achieving 98 % accuracy in DL. This approach filters unclassified data effectively, demonstrating the significance of DL for leukemia prediction. The testing validation was based on 17 different features such as patient age, sex, mutation type, treatment methods, chromosomes, and others. Our study compares ML and DL techniques and chooses the best technique that gives optimum results. The study emphasizes the implications of high-throughput technology in healthcare, offering improved patient care.

Navigating Precision Oncology: Insights from an Integrated Clinical Data and Biobank Repository Initiative across a Network Cancer Program

Advancing cancer treatment relies on the rapid translation of new scientific discoveries to patient care. To facilitate this, an oncology biobank and data repository program, also referred to as the "Moonshot" program, was launched in 2021 within the Integrated Network Cancer Program of the Allegheny Health Network. A clinical data program (CDP) and biospecimen repository were established, and patient data and blood and tissue samples have been collected prospectively. To date, the study has accrued 2920 patients, predominantly female (61%) and Caucasian (90%), with a mean age of 64 ± 13 years. The most common cancer sites were the endometrium/uterus (12%), lung/bronchus (12%), breast (11%), and colon/rectum (11%). Of patients diagnosed with cancer, 34% were diagnosed at stage I, 25% at stage II, 26% at stage III, and 15% at stage IV. The CDP is designed to support our initiative in advancing personalized cancer research by providing a comprehensive array of patient data, encompassing demographic characteristics, diagnostic details, and treatment responses. The "Moonshot" initiative aims to predict therapy responses and clinical outcomes through cancer-related biomarkers. The CDP facilitates this initiative by fostering data sharing, enabling comparative analyses, and informing the development of novel diagnostic and therapeutic methods.

Multimodal machine learning models identify chemotherapy drugs with prospective clinical efficacy in dogs with relapsed B-cell lymphoma

Dogs with B-cell lymphoma typically respond well to first-line CHOP-based chemotherapy, but there is no standard of care for relapsed patients. To help veterinary oncologists select effective drugs for dogs with lymphoid malignancies such as B-cell lymphoma, we have developed multimodal machine learning models that integrate data from multiple tumor profiling modalities and predict the likelihood of a positive clinical response for 10 commonly used chemotherapy drugs. Here we report on clinical outcomes that occurred after oncologists received a prediction report generated by our models. Remarkably, we found that dogs that received drugs predicted to be effective by the models experienced better clinical outcomes by every metric we analyzed (overall response rate, complete response rate, duration of complete response, patient survival times) relative to other dogs in the study and relative to historical controls.

Standardizing digital biobanks: integrating imaging, genomic, and clinical data for precision medicine

Advancements in data acquisition and computational methods are generating a large amount of heterogeneous biomedical data from diagnostic domains such as clinical imaging, pathology, and next-generation sequencing (NGS), which help characterize individual differences in patients. However, this information needs to be available and suitable to promote and support scientific research and technological development, supporting the effective adoption of the precision medicine approach in clinical practice. Digital biobanks can catalyze this process, facilitating the sharing of curated and standardized imaging data, clinical, pathological and molecular data, crucial to enable the development of a comprehensive and personalized data-driven diagnostic approach in disease management and fostering the development of computational predictive models. This work aims to frame this perspective, first by evaluating the state of standardization of individual diagnostic domains and then by identifying challenges and proposing a possible solution towards an integrative approach that can guarantee the suitability of information that can be shared through a digital biobank. Our analysis of the state of the art shows the presence and use of reference standards in biobanks and, generally, digital repositories for each specific domain. Despite this, standardization to guarantee the integration and reproducibility of the numerical descriptors generated by each domain, e.g. radiomic, pathomic and -omic features, is still an open challenge. Based on specific use cases and scenarios, an integration model, based on the JSON format, is proposed that can help address this problem. Ultimately, this work shows how, with specific standardization and promotion efforts, the digital biobank model can become an enabling technology for the comprehensive study of diseases and the effective development of data-driven technologies at the service of precision medicine.

Pathogenomics for accurate diagnosis, treatment, prognosis of oncology: a cutting edge overview

The capability to gather heterogeneous data, alongside the increasing power of artificial intelligence to examine it, leading a revolution in harnessing multimodal data in the life sciences. However, most approaches are limited to unimodal data, leaving integrated approaches across modalities relatively underdeveloped in computational pathology. Pathogenomics, as an invasive method to integrate advanced molecular diagnostics from genomic data, morphological information from histopathological imaging, and codified clinical data enable the discovery of new multimodal cancer biomarkers to propel the field of precision oncology in the coming decade. In this perspective, we offer our opinions on synthesizing complementary modalities of data with emerging multimodal artificial intelligence methods in pathogenomics. It includes correlation between the pathological and genomic profile of cancer, fusion of histology, and genomics profile of cancer. We also present challenges, opportunities, and avenues for future work.

Big data and artificial intelligence in cancer research

The field of oncology has witnessed an extraordinary surge in the application of big data and artificial intelligence (AI). AI development has made multiscale and multimodal data fusion and analysis possible. A new era of extracting information from complex big data is rapidly evolving. However, challenges related to efficient data curation, in-depth analysis, and utilization remain. We provide a comprehensive overview of the current state of the art in big data and computational analysis, highlighting key applications, challenges, and future opportunities in cancer research. By sketching the current landscape, we seek to foster a deeper understanding and facilitate the advancement of big data utilization in oncology, call for interdisciplinary collaborations, ultimately contributing to improved patient outcomes and a profound understanding of cancer.

Encrypted federated learning for secure decentralized collaboration in cancer image analysis

Artificial intelligence (AI) has a multitude of applications in cancer research and oncology. However, the training of AI systems is impeded by the limited availability of large datasets due to data protection requirements and other regulatory obstacles. Federated and swarm learning represent possible solutions to this problem by collaboratively training AI models while avoiding data transfer. However, in these decentralized methods, weight updates are still transferred to the aggregation server for merging the models. This leaves the possibility for a breach of data privacy, for example by model inversion or membership inference attacks by untrusted servers. Somewhat-homomorphically-encrypted federated learning (SHEFL) is a solution to this problem because only encrypted weights are transferred, and model updates are performed in the encrypted space. Here, we demonstrate the first successful implementation of SHEFL in a range of clinically relevant tasks in cancer image analysis on multicentric datasets in radiology and histopathology. We show that SHEFL enables the training of AI models which outperform locally trained models and perform on par with models which are centrally trained. In the future, SHEFL can enable multiple institutions to co-train AI models without forsaking data governance and without ever transmitting any decryptable data to untrusted servers.

New regulatory thinking is needed for AI-based personalised drug and cell therapies in precision oncology

Until recently the application of artificial intelligence (AI) in precision oncology was confined to activities in drug development and had limited impact on the personalisation of therapy. Now, a number of approaches have been proposed for the personalisation of drug and cell therapies with AI applied to therapy design, planning and delivery at the patient's bedside. Some drug and cell-based therapies are already tuneable to the individual to optimise efficacy, to reduce toxicity, to adapt the dosing regime, to design combination therapy approaches and, preclinically, even to personalise the receptor design of cell therapies. Developments in AI-based healthcare are accelerating through the adoption of foundation models, and generalist medical AI models have been proposed. The application of these approaches in therapy design is already being explored and realistic short-term advances include the application to the personalised design and delivery of drugs and cell therapies. With this pace of development, the limiting step to adoption will likely be the capacity and appropriateness of regulatory frameworks. This article explores emerging concepts and new ideas for the regulation of AI-enabled personalised cancer therapies in the context of existing and in development governance frameworks.

Toward Explainable Artificial Intelligence for Precision Pathology

The rapid development of precision medicine in recent years has started to challenge diagnostic pathology with respect to its ability to analyze histological images and increasingly large molecular profiling data in a quantitative, integrative, and standardized way. Artificial intelligence (AI) and, more precisely, deep learning technologies have recently demonstrated the potential to facilitate complex data analysis tasks, including clinical, histological, and molecular data for disease classification; tissue biomarker quantification; and clinical outcome prediction. This review provides a general introduction to AI and describes recent developments with a focus on applications in diagnostic pathology and beyond. We explain limitations including the black-box character of conventional AI and describe solutions to make machine learning decisions more transparent with so-called explainable AI. The purpose of the review is to foster a mutual understanding of both the biomedical and the AI side. To that end, in addition to providing an overview of the relevant foundations in pathology and machine learning, we present worked-through examples for a better practical understanding of what AI can achieve and how it should be done.

AI-Enhanced Detection of Clinically Relevant Structural and Functional Anomalies in MRI: Traversing the Landscape of Conventional to Explainable Approaches

Anomaly detection in medical imaging, particularly within the realm of magnetic resonance imaging (MRI), stands as a vital area of research with far-reaching implications across various medical fields. This review meticulously examines the integration of artificial intelligence (AI) in anomaly detection for MR images, spotlighting its transformative impact on medical diagnostics. We delve into the forefront of AI applications in MRI, exploring advanced machine learning (ML) and deep learning (DL) methodologies that are pivotal in enhancing the precision of diagnostic processes. The review provides a detailed analysis of preprocessing, feature extraction, classification, and segmentation techniques, alongside a comprehensive evaluation of commonly used metrics. Further, this paper explores the latest developments in ensemble methods and explainable AI, offering insights into future directions and potential breakthroughs. This review synthesizes current insights, offering a valuable guide for researchers, clinicians, and medical imaging experts. It highlights AI's crucial role in improving the precision and speed of detecting key structural and functional irregularities in MRI. Our exploration of innovative techniques and trends furthers MRI technology development, aiming to refine diagnostics, tailor treatments, and elevate patient care outcomes. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 1.

Multi-modal cell-free DNA genomic and fragmentomic patterns enhance cancer survival and recurrence analysis

The structure of cell-free DNA (cfDNA) is altered in the blood of patients with cancer. From whole-genome sequencing, we retrieve the cfDNA fragment-end composition using a new software (FrEIA [fragment end integrated analysis]), as well as the cfDNA size and tumor fraction in three independent cohorts (n = 925 cancer from >10 types and 321 control samples). At 95% specificity, we detect 72% cancer samples using at least one cfDNA measure, including 64% early-stage cancer (n = 220). cfDNA detection correlates with a shorter overall (p = 0.0086) and recurrence-free (p = 0.017) survival in patients with resectable esophageal adenocarcinoma. Integrating cfDNA measures with machine learning in an independent test set (n = 396 cancer, 90 controls) achieve a detection accuracy of 82% and area under the receiver operating characteristic curve of 0.96. In conclusion, harnessing the biological features of cfDNA can improve, at no extra cost, the diagnostic performance of liquid biopsies.

The Cross-Scale Association between Pathomics and Radiomics Features in Immunotherapy-Treated NSCLC Patients: A Preliminary Study

BACKGROUND

Recent advances in cancer biomarker development have led to a surge of distinct data modalities, such as medical imaging and histopathology. To develop predictive immunotherapy biomarkers, these modalities are leveraged independently, despite their orthogonality. This study aims to explore the cross-scale association between radiological scans and digitalized pathology images for immunotherapy-treated non-small cell lung cancer (NSCLC) patients.

METHODS

This study involves 36 NSCLC patients who were treated with immunotherapy and for whom both radiology and pathology images were available. A total of 851 and 260 features were extracted from CT scans and cell density maps of histology images at different resolutions. We investigated the radiopathomics relationship and their association with clinical and biological endpoints. We used the Kolmogorov-Smirnov (KS) method to test the differences between the distributions of correlation coefficients with the two imaging modality features. Unsupervised clustering was done to identify which imaging modality captures poor and good survival patients.

RESULTS

Our results demonstrated a significant correlation between cell density pathomics and radiomics features. Furthermore, we also found a varying distribution of correlation values between imaging-derived features and clinical endpoints. The KS test revealed that the two imaging feature distributions were different for PFS and CD8 counts, while similar for OS. In addition, clustering analysis resulted in significant differences in the two clusters generated from the radiomics and pathomics features with respect to patient survival and CD8 counts.

CONCLUSION

The results of this study suggest a cross-scale association between CT scans and pathology H&E slides among ICI-treated patients. These relationships can be further explored to develop multimodal immunotherapy biomarkers to advance personalized lung cancer care.

Historical perspective and future directions: computational science in immuno-oncology

Immuno-oncology holds promise for transforming patient care having achieved durable clinical response rates across a variety of advanced and metastatic cancers. Despite these achievements, only a minority of patients respond to immunotherapy, underscoring the importance of elucidating molecular mechanisms responsible for response and resistance to inform the development and selection of treatments. Breakthroughs in molecular sequencing technologies have led to the generation of an immense amount of genomic and transcriptomic sequencing data that can be mined to uncover complex tumor-immune interactions using computational tools. In this review, we discuss existing and emerging computational methods that contextualize the composition and functional state of the tumor microenvironment, infer the reactivity and clonal dynamics from reconstructed immune cell receptor repertoires, and predict the antigenic landscape for immune cell recognition. We further describe the advantage of multi-omics analyses for capturing multidimensional relationships and artificial intelligence techniques for integrating omics data with histopathological and radiological images to encapsulate patterns of treatment response and tumor-immune biology. Finally, we discuss key challenges impeding their widespread use and clinical application and conclude with future perspectives. We are hopeful that this review will both serve as a guide for prospective researchers seeking to use existing tools for scientific discoveries and inspire the optimization or development of novel tools to enhance precision, ultimately expediting advancements in immunotherapy that improve patient survival and quality of life.

The role of large language models in medical image processing: a narrative review

Background and Objective

The rapid advancement of artificial intelligence (AI) has ushered in a new era in natural language processing (NLP), with large language models (LLMs) like ChatGPT leading the way. This paper explores the profound impact of AI, particularly LLMs, in the field of medical image processing. The objective is to provide insights into the transformative potential of AI in improving healthcare by addressing historical challenges associated with manual image interpretation.

Methods

A comprehensive literature search was conducted on the Web of Science and PubMed databases from 2013 to 2023, focusing on the transformations of LLMs in Medical Imaging Processing. Recent publications on the arXiv database were also reviewed. Our search criteria included all types of articles, including abstracts, review articles, letters, and editorials. The language of publications was restricted to English to facilitate further content analysis.

Key Content and Findings

The review reveals that AI, driven by LLMs, has revolutionized medical image processing by streamlining the interpretation process, traditionally characterized by time-intensive manual efforts. AI's impact on medical care quality and patient well-being is substantial. With their robust interactivity and multimodal learning capabilities, LLMs offer immense potential for enhancing various aspects of medical image processing. Additionally, the Transformer architecture, foundational to LLMs, is gaining prominence in this domain.

Conclusions

In conclusion, this review underscores the pivotal role of AI, especially LLMs, in advancing medical image processing. These technologies have the capacity to enhance transfer learning efficiency, integrate multimodal data, facilitate clinical interactivity, and optimize cost-efficiency in healthcare. The potential applications of LLMs in clinical settings are promising, with far-reaching implications for future research, clinical practice, and healthcare policy. The transformative impact of AI in medical image processing is undeniable, and its continued development and implementation are poised to reshape the healthcare landscape for the better.

Artificial intelligence in fracture detection with different image modalities and data types: A systematic review and meta-analysis

Artificial Intelligence (AI), encompassing Machine Learning and Deep Learning, has increasingly been applied to fracture detection using diverse imaging modalities and data types. This systematic review and meta-analysis aimed to assess the efficacy of AI in detecting fractures through various imaging modalities and data types (image, tabular, or both) and to synthesize the existing evidence related to AI-based fracture detection. Peer-reviewed studies developing and validating AI for fracture detection were identified through searches in multiple electronic databases without time limitations. A hierarchical meta-analysis model was used to calculate pooled sensitivity and specificity. A diagnostic accuracy quality assessment was performed to evaluate bias and applicability. Of the 66 eligible studies, 54 identified fractures using imaging-related data, nine using tabular data, and three using both. Vertebral fractures were the most common outcome (n = 20), followed by hip fractures (n = 18). Hip fractures exhibited the highest pooled sensitivity (92%; 95% CI: 87-96, p< 0.01) and specificity (90%; 95% CI: 85-93, p< 0.01). Pooled sensitivity and specificity using image data (92%; 95% CI: 90-94, p< 0.01; and 91%; 95% CI: 88-93, p < 0.01) were higher than those using tabular data (81%; 95% CI: 77-85, p< 0.01; and 83%; 95% CI: 76-88, p < 0.01), respectively. Radiographs demonstrated the highest pooled sensitivity (94%; 95% CI: 90-96, p < 0.01) and specificity (92%; 95% CI: 89-94, p< 0.01). Patient selection and reference standards were major concerns in assessing diagnostic accuracy for bias and applicability. AI displays high diagnostic accuracy for various fracture outcomes, indicating potential utility in healthcare systems for fracture diagnosis. However, enhanced transparency in reporting and adherence to standardized guidelines are necessary to improve the clinical applicability of AI. Review Registration: PROSPERO (CRD42021240359).

Artificial intelligence for predictive biomarker discovery in immuno-oncology: a systematic review

BACKGROUND

The widespread use of immune checkpoint inhibitors (ICIs) has revolutionised treatment of multiple cancer types. However, selecting patients who may benefit from ICI remains challenging. Artificial intelligence (AI) approaches allow exploitation of high-dimension oncological data in research and development of precision immuno-oncology.

MATERIALS AND METHODS

We conducted a systematic literature review of peer-reviewed original articles studying the ICI efficacy prediction in cancer patients across five data modalities: genomics (including genomics, transcriptomics, and epigenomics), radiomics, digital pathology (pathomics), and real-world and multimodality data.

RESULTS

A total of 90 studies were included in this systematic review, with 80% published in 2021-2022. Among them, 37 studies included genomic, 20 radiomic, 8 pathomic, 20 real-world, and 5 multimodal data. Standard machine learning (ML) methods were used in 72% of studies, deep learning (DL) methods in 22%, and both in 6%. The most frequently studied cancer type was non-small-cell lung cancer (36%), followed by melanoma (16%), while 25% included pan-cancer studies. No prospective study design incorporated AI-based methodologies from the outset; rather, all implemented AI as a post hoc analysis. Novel biomarkers for ICI in radiomics and pathomics were identified using AI approaches, and molecular biomarkers have expanded past genomics into transcriptomics and epigenomics. Finally, complex algorithms and new types of AI-based markers, such as meta-biomarkers, are emerging by integrating multimodal/multi-omics data.

CONCLUSION

AI-based methods have expanded the horizon for biomarker discovery, demonstrating the power of integrating multimodal data from existing datasets to discover new meta-biomarkers. While most of the included studies showed promise for AI-based prediction of benefit from immunotherapy, none provided high-level evidence for immediate practice change. A priori planned prospective trial designs are needed to cover all lifecycle steps of these software biomarkers, from development and validation to integration into clinical practice.

Multimodal Data Integration Advances Longitudinal Prediction of the Naturalistic Course of Depression and Reveals a Multimodal Signature of Remission During 2-Year Follow-up

BACKGROUND

The ability to predict the disease course of individuals with major depressive disorder (MDD) is essential for optimal treatment planning. Here, we used a data-driven machine learning approach to assess the predictive value of different sets of biological data (whole-blood proteomics, lipid metabolomics, transcriptomics, genetics), both separately and added to clinical baseline variables, for the longitudinal prediction of 2-year remission status in MDD at the individual-subject level.

METHODS

Prediction models were trained and cross-validated in a sample of 643 patients with current MDD (2-year remission n = 325) and subsequently tested for performance in 161 individuals with MDD (2-year remission n = 82).

RESULTS

Proteomics data showed the best unimodal data predictions (area under the receiver operating characteristic curve = 0.68). Adding proteomic to clinical data at baseline significantly improved 2-year MDD remission predictions (area under the receiver operating characteristic curve = 0.63 vs. 0.78, p = .013), while the addition of other omics data to clinical data did not yield significantly improved model performance. Feature importance and enrichment analysis revealed that proteomic analytes were involved in inflammatory response and lipid metabolism, with fibrinogen levels showing the highest variable importance, followed by symptom severity. Machine learning models outperformed psychiatrists' ability to predict 2-year remission status (balanced accuracy = 71% vs. 55%).

CONCLUSIONS

This study showed the added predictive value of combining proteomic data, but not other omics data, with clinical data for the prediction of 2-year remission status in MDD. Our results reveal a novel multimodal signature of 2-year MDD remission status that shows clinical potential for individual MDD disease course predictions from baseline measurements.

Accurate integration of single-cell DNA and RNA for analyzing intratumor heterogeneity using MaCroDNA

Cancers develop and progress as mutations accumulate, and with the advent of single-cell DNA and RNA sequencing, researchers can observe these mutations and their transcriptomic effects and predict proteomic changes with remarkable temporal and spatial precision. However, to connect genomic mutations with their transcriptomic and proteomic consequences, cells with either only DNA data or only RNA data must be mapped to a common domain. For this purpose, we present MaCroDNA, a method that uses maximum weighted bipartite matching of per-gene read counts from single-cell DNA and RNA-seq data. Using ground truth information from colorectal cancer data, we demonstrate the advantage of MaCroDNA over existing methods in accuracy and speed. Exemplifying the utility of single-cell data integration in cancer research, we suggest, based on results derived using MaCroDNA, that genomic mutations of large effect size increasingly contribute to differential expression between cells as Barrett's esophagus progresses to esophageal cancer, reaffirming the findings of the previous studies.

Development and Validation of a Post-Radiotherapy Prediction Model for Bowel Dysfunction After Rectal Cancer Resection

BACKGROUND & AIMS

The benefit of radiotherapy for rectal cancer is based largely on a balance between a decrease in local recurrence and an increase in bowel dysfunction. Predicting postoperative disability is helpful for recovery plans and early intervention. We aimed to develop and validate a risk model to improve the prediction of major bowel dysfunction after restorative rectal cancer resection with neoadjuvant radiotherapy using perioperative features.

METHODS

Eligible patients more than 1 year after restorative resection following radiotherapy were invited to complete the low anterior resection syndrome (LARS) score at 3 national hospitals in China. Clinical characteristics and imaging parameters were assessed with machine learning algorithms. The post-radiotherapy LARS prediction model (PORTLARS) was constructed by means of logistic regression on the basis of key factors with proportional weighs. The accuracy of the model for major LARS prediction was internally and externally validated.

RESULTS

A total of 868 patients reported a mean LARS score of 28.4 after an average time of 4.7 years since surgery. Key predictors for major LARS included the length of distal rectum, anastomotic leakage, proximal colon of neorectum, and pathologic nodal stage. PORTLARS had a favorable area under the curve for predicting major LARS in the internal dataset (0.835; 95% CI, 0.800-0.870, n = 521) and external dataset (0.884; 95% CI, 0.848-0.921, n = 347). The model achieved both sensitivity and specificity >0.83 in the external validation. In addition, PORTLARS outperformed the preoperative LARS score for prediction of major events.

CONCLUSIONS

PORTLARS could predict major bowel dysfunction after rectal cancer resection following radiotherapy with high accuracy and robustness. It may serve as a useful tool to identify patients who need additional support for long-term dysfunction in the early stage.

CLINICALTRIALS

gov, number NCT05129215.

M-MSSEU: source-free domain adaptation for multi-modal stroke lesion segmentation using shadowed sets and evidential uncertainty

Due to the unavailability of source domain data encountered in unsupervised domain adaptation, there has been an increasing number of studies on source-free domain adaptation (SFDA) in recent years. To better solve the SFDA problem and effectively leverage the multi-modal information in medical images, this paper presents a novel SFDA method for multi-modal stroke lesion segmentation in which evidential deep learning instead of convolutional neural network. Specifically, for multi-modal stroke images, we design a multi-modal opinion fusion module which uses Dempster-Shafer evidence theory for decision fusion of different modalities. Besides, for the SFDA problem, we use the pseudo label learning method, which obtains pseudo labels from the pre-trained source model to perform the adaptation process. To solve the unreliability of pseudo label caused by domain shift, we propose a pseudo label filtering scheme using shadowed sets theory and a pseudo label refining scheme using evidential uncertainty. These two schemes can automatically extract unreliable parts in pseudo labels and jointly improve the quality of pseudo labels with low computational costs. Experiments on two multi-modal stroke lesion datasets demonstrate the superiority of our method over other state-of-the-art SFDA methods.

Prediction of cancer recurrence based on compact graphs of whole slide images

Cancer recurrence is one of the primary causes of patient mortality following treatment, indicating increased aggressiveness of cancer cells and difficulties in achieving a cure. A critical step to improve patients' survival is accurately predicting recurrence status and giving appropriate treatment. Whole Slide Images (WSIs) are a common type of image data in the field of digital pathology, containing high-resolution tissue information. Furthermore, WSIs of primary tumors contain microenvironmental information directly associated with the growth of tumor cells. To effectively utilize this microenvironmental information. Firstly, we represented microenvironmental features of histopathological images as compact graphs. Secondly, this work aims to develop an enhanced lightweight graph neural network called the Adaptive Graph Clustering Network (AGCNet) for predicting cancer recurrence. Experiments are conducted on three cancer datasets from The Cancer Genome Atlas (TCGA), and AGCNet achieved an accuracy of 81.81% in BLCA, 69.66% in PAAD, and 81.96% in STAD. These results indicated that AGCNet is an effective model for predicting cancer recurrence and is expected to be applied in clinical applications.

Imaging phenotypes predict overall survival in glioma more accurate than basic demographic and cell mutation profiles

BACKGROUND

Magnetic resonance imaging (MRI), digital pathology imaging (PATH), demographics, and IDH mutation status predict overall survival (OS) in glioma. Identifying and characterizing predictive features in the different modalities may improve OS prediction accuracy.

PURPOSE

To evaluate the OS prediction accuracy of combinations of prognostic markers in glioma patients.

MATERIALS AND METHODS

Multi-contrast MRI, comprising T1-weighted, T1-weighted post-contrast, T2-weighted, T2 fluid-attenuated-inversion-recovery, and pathology images from glioma patients (n = 160) were retrospectively collected (1983-2008) from TCGA alongside age and sex. Phenotypic profiling of tumors was performed by quantifying the radiographic and histopathologic descriptors extracted from the delineated region-of-interest in MRI and PATH images. A Cox proportional hazard model was trained with the MRI and PATH features, IDH mutation status, and basic demographic variables (age and sex) to predict OS. The performance was evaluated in a split-train-test configuration using the concordance-index, computed between the predicted risk score and observed OS.

RESULTS

The average age of patients was 51.2years (women: n = 77, age-range=18-84years; men: n = 83, age-range=21-80years). The median OS of the participants was 494.5 (range,3-4752), 481 (range,7-4752), and 524.5 days (range,3-2869), respectively, in complete dataset, training, and test datasets. The addition of MRI or PATH features improved prediction of OS when compared to models based on age, sex, and mutation status alone or their combination (p < 0.001). The full multi-omics model integrated MRI, PATH, clinical, and genetic profiles and predicted the OS best (c-index= 0.87).

CONCLUSION

The combination of imaging, genetic, and clinical profiles leads to a more accurate prognosis than the clinical and/or mutation status.

Development of the ehive Digital Health App: Protocol for a Centralized Research Platform

BACKGROUND

The increasing use of smartphones, wearables, and connected devices has enabled the increasing application of digital technologies for research. Remote digital study platforms comprise a patient-interfacing digital application that enables multimodal data collection from a mobile app and connected sources. They offer an opportunity to recruit at scale, acquire data longitudinally at a high frequency, and engage study participants at any time of the day in any place. Few published descriptions of centralized digital research platforms provide a framework for their development.

OBJECTIVE

This study aims to serve as a road map for those seeking to develop a centralized digital research platform. We describe the technical and functional aspects of the ehive app, the centralized digital research platform of the Hasso Plattner Institute for Digital Health at Mount Sinai Hospital, New York, New York. We then provide information about ongoing studies hosted on ehive, including usership statistics and data infrastructure. Finally, we discuss our experience with ehive in the broader context of the current landscape of digital health research platforms.

METHODS

The ehive app is a multifaceted and patient-facing central digital research platform that permits the collection of e-consent for digital health studies. An overview of its development, its e-consent process, and the tools it uses for participant recruitment and retention are provided. Data integration with the platform and the infrastructure supporting its operations are discussed; furthermore, a description of its participant- and researcher-facing dashboard interfaces and the e-consent architecture is provided.

RESULTS

The ehive platform was launched in 2020 and has successfully hosted 8 studies, namely 6 observational studies and 2 clinical trials. Approximately 1484 participants downloaded the app across 36 states in the United States. The use of recruitment methods such as bulk messaging through the EPIC electronic health records and standard email portals enables broad recruitment. Light-touch engagement methods, used in an automated fashion through the platform, maintain high degrees of engagement and retention. The ehive platform demonstrates the successful deployment of a central digital research platform that can be modified across study designs.

CONCLUSIONS

Centralized digital research platforms such as ehive provide a novel tool that allows investigators to expand their research beyond their institution, engage in large-scale longitudinal studies, and combine multimodal data streams. The ehive platform serves as a model for groups seeking to develop similar digital health research programs.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/49204.

Biomarkers for immunotherapy of hepatocellular carcinoma

Immune-checkpoint inhibitors (ICIs) are now widely used for the treatment of patients with advanced-stage hepatocellular carcinoma (HCC). Two different ICI-containing regimens, atezolizumab plus bevacizumab and tremelimumab plus durvalumab, are now approved standard-of-care first-line therapies in this setting. However, and despite substantial improvements in survival outcomes relative to sorafenib, most patients with advanced-stage HCC do not derive durable benefit from these regimens. Advances in genome sequencing including the use of single-cell RNA sequencing (both of tumour material and blood samples), as well as immune cell identification strategies and other techniques such as radiomics and analysis of the microbiota, have created considerable potential for the identification of novel predictive biomarkers enabling the accurate selection of patients who are most likely to derive benefit from ICIs. In this Review, we summarize data on the immunology of HCC and the outcomes in patients receiving ICIs for the treatment of this disease. We then provide an overview of current biomarker use and developments in the past 5 years, including gene signatures, circulating tumour cells, high-dimensional flow cytometry, single-cell RNA sequencing as well as approaches involving the microbiome, radiomics and clinical markers. Novel concepts for further biomarker development in HCC are then discussed including biomarker-driven trials, spatial transcriptomics and integrated 'big data' analysis approaches. These concepts all have the potential to better identify patients who are most likely to benefit from ICIs and to promote the development of new treatment approaches.

A comprehensive review of machine learning algorithms and their application in geriatric medicine: present and future

The increasing access to health data worldwide is driving a resurgence in machine learning research, including data-hungry deep learning algorithms. More computationally efficient algorithms now offer unique opportunities to enhance diagnosis, risk stratification, and individualised approaches to patient management. Such opportunities are particularly relevant for the management of older patients, a group that is characterised by complex multimorbidity patterns and significant interindividual variability in homeostatic capacity, organ function, and response to treatment. Clinical tools that utilise machine learning algorithms to determine the optimal choice of treatment are slowly gaining the necessary approval from governing bodies and being implemented into healthcare, with significant implications for virtually all medical disciplines during the next phase of digital medicine. Beyond obtaining regulatory approval, a crucial element in implementing these tools is the trust and support of the people that use them. In this context, an increased understanding by clinicians of artificial intelligence and machine learning algorithms provides an appreciation of the possible benefits, risks, and uncertainties, and improves the chances for successful adoption. This review provides a broad taxonomy of machine learning algorithms, followed by a more detailed description of each algorithm class, their purpose and capabilities, and examples of their applications, particularly in geriatric medicine. Additional focus is given on the clinical implications and challenges involved in relying on devices with reduced interpretability and the progress made in counteracting the latter via the development of explainable machine learning.

The long-term health outcomes, pathophysiological mechanisms and multidisciplinary management of long COVID

There have been hundreds of millions of cases of coronavirus disease 2019 (COVID-19), which is caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). With the growing population of recovered patients, it is crucial to understand the long-term consequences of the disease and management strategies. Although COVID-19 was initially considered an acute respiratory illness, recent evidence suggests that manifestations including but not limited to those of the cardiovascular, respiratory, neuropsychiatric, gastrointestinal, reproductive, and musculoskeletal systems may persist long after the acute phase. These persistent manifestations, also referred to as long COVID, could impact all patients with COVID-19 across the full spectrum of illness severity. Herein, we comprehensively review the current literature on long COVID, highlighting its epidemiological understanding, the impact of vaccinations, organ-specific sequelae, pathophysiological mechanisms, and multidisciplinary management strategies. In addition, the impact of psychological and psychosomatic factors is also underscored. Despite these crucial findings on long COVID, the current diagnostic and therapeutic strategies based on previous experience and pilot studies remain inadequate, and well-designed clinical trials should be prioritized to validate existing hypotheses. Thus, we propose the primary challenges concerning biological knowledge gaps and efficient remedies as well as discuss the corresponding recommendations.

Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology

As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.

Revolutionizing Digital Pathology With the Power of Generative Artificial Intelligence and Foundation Models

Digital pathology has transformed the traditional pathology practice of analyzing tissue under a microscope into a computer vision workflow. Whole-slide imaging allows pathologists to view and analyze microscopic images on a computer monitor, enabling computational pathology. By leveraging artificial intelligence (AI) and machine learning (ML), computational pathology has emerged as a promising field in recent years. Recently, task-specific AI/ML (eg, convolutional neural networks) has risen to the forefront, achieving above-human performance in many image-processing and computer vision tasks. The performance of task-specific AI/ML models depends on the availability of many annotated training datasets, which presents a rate-limiting factor for AI/ML development in pathology. Task-specific AI/ML models cannot benefit from multimodal data and lack generalization, eg, the AI models often struggle to generalize to new datasets or unseen variations in image acquisition, staining techniques, or tissue types. The 2020s are witnessing the rise of foundation models and generative AI. A foundation model is a large AI model trained using sizable data, which is later adapted (or fine-tuned) to perform different tasks using a modest amount of task-specific annotated data. These AI models provide in-context learning, can self-correct mistakes, and promptly adjust to user feedback. In this review, we provide a brief overview of recent advances in computational pathology enabled by task-specific AI, their challenges and limitations, and then introduce various foundation models. We propose to create a pathology-specific generative AI based on multimodal foundation models and present its potentially transformative role in digital pathology. We describe different use cases, delineating how it could serve as an expert companion of pathologists and help them efficiently and objectively perform routine laboratory tasks, including quantifying image analysis, generating pathology reports, diagnosis, and prognosis. We also outline the potential role that foundation models and generative AI can play in standardizing the pathology laboratory workflow, education, and training.

Enhancing head and neck tumor management with artificial intelligence: Integration and perspectives

Head and neck tumors (HNTs) constitute a multifaceted ensemble of pathologies that primarily involve regions such as the oral cavity, pharynx, and nasal cavity. The intricate anatomical structure of these regions poses considerable challenges to efficacious treatment strategies. Despite the availability of myriad treatment modalities, the overall therapeutic efficacy for HNTs continues to remain subdued. In recent years, the deployment of artificial intelligence (AI) in healthcare practices has garnered noteworthy attention. AI modalities, inclusive of machine learning (ML), neural networks (NNs), and deep learning (DL), when amalgamated into the holistic management of HNTs, promise to augment the precision, safety, and efficacy of treatment regimens. The integration of AI within HNT management is intricately intertwined with domains such as medical imaging, bioinformatics, and medical robotics. This article intends to scrutinize the cutting-edge advancements and prospective applications of AI in the realm of HNTs, elucidating AI's indispensable role in prevention, diagnosis, treatment, prognostication, research, and inter-sectoral integration. The overarching objective is to stimulate scholarly discourse and invigorate insights among medical practitioners and researchers to propel further exploration, thereby facilitating superior therapeutic alternatives for patients.

Application of colloidal photonic crystals in study of organoids

As alternative disease models, other than 2D cell lines and patient-derived xenografts, organoids have preferable in vivo physiological relevance. However, both endogenous and exogenous limitations impede the development and clinical translation of these organoids. Fortunately, colloidal photonic crystals (PCs), which benefit from favorable biocompatibility, brilliant optical manipulation, and facile chemical decoration, have been applied to the engineering of organoids and have achieved the desirable recapitulation of the ECM niche, well-defined geometrical onsets for initial culture, in situ multiphysiological parameter monitoring, single-cell biomechanical sensing, and high-throughput drug screening with versatile functional readouts. Herein, we review the latest progress in engineering organoids fabricated from colloidal PCs and provide inputs for future research.

An Integrated Multi-Omics and Artificial Intelligence Framework for Advance Plant Phenotyping in Horticulture

This review discusses the transformative potential of integrating multi-omics data and artificial intelligence (AI) in advancing horticultural research, specifically plant phenotyping. The traditional methods of plant phenotyping, while valuable, are limited in their ability to capture the complexity of plant biology. The advent of (meta-)genomics, (meta-)transcriptomics, proteomics, and metabolomics has provided an opportunity for a more comprehensive analysis. AI and machine learning (ML) techniques can effectively handle the complexity and volume of multi-omics data, providing meaningful interpretations and predictions. Reflecting the multidisciplinary nature of this area of research, in this review, readers will find a collection of state-of-the-art solutions that are key to the integration of multi-omics data and AI for phenotyping experiments in horticulture, including experimental design considerations with several technical and non-technical challenges, which are discussed along with potential solutions. The future prospects of this integration include precision horticulture, predictive breeding, improved disease and stress response management, sustainable crop management, and exploration of plant biodiversity. The integration of multi-omics and AI holds immense promise for revolutionizing horticultural research and applications, heralding a new era in plant phenotyping.