Multimodal integration of image, epigenetic and clinical data to predict BRAF mutation status in melanoma

AI in Histopathology Explorer for comprehensive analysis of the evolving AI landscape in histopathology

Digital pathology and artificial intelligence (AI) hold immense transformative potential to revolutionize cancer diagnostics, treatment outcomes, and biomarker discovery. Gaining a deeper understanding of deep learning algorithm methods applied to histopathological data and evaluating their performance on different tasks is crucial for developing the next generation of AI technologies. To this end, we developed AI in Histopathology Explorer (HistoPathExplorer); an interactive dashboard with intelligent tools available at www.histopathexpo.ai . This real-time online resource enables users, including researchers, decision-makers, and various stakeholders, to assess the current landscape of AI applications for specific clinical tasks, analyze their performance, and explore the factors influencing their translation into practice. Moreover, a quality index was defined for evaluating the comprehensiveness of methodological details in published AI methods. HistoPathExplorer highlights opportunities and challenges for AI in histopathology, and offers a valuable resource for creating more effective methods and shaping strategies and guidelines for translating digital pathology applications into clinical practice.

[State of digitalization in dermatopathology]

As in general pathology, digitalization is also inexorably making its way into dermatopathology. This article examines the current state of digitalization in German dermatopathology laboratories based on the authors' own experiences, the current study situation, and a survey of members of the Dermatological Histology Working Group (ADH). Experiences with the establishment of a digital laboratory workflow, artificial intelligence (AI)-based assistance systems, and whole slide images (WSI)-based training programs are discussed. Digitalization in dermatopathology is an opportunity to simplify and accelerate processes, but there are some hurdles to overcome.

Predicting benefit from PARP inhibitors using deep learning on H&E-stained ovarian cancer slides

PURPOSE

Ovarian cancer patients with a Homologous Recombination Deficiency (HRD) often benefit from polyadenosine diphosphate-ribose polymerase (PARP) inhibitor maintenance therapy after response to platinum-based chemotherapy. HR status is currently analyzed via complex molecular tests. Predicting benefit from PARP inhibitors directly on histological whole slide images (WSIs) could be a fast and cheap alternative.

PATIENTS AND METHODS

We trained a Deep Learning (DL) model on H&E stained WSIs with "shrunken centroid" (SC) based HRD ground truth using the AGO-TR1 cohort (n = 208: 108 training, 100 test) and tested its ability to predict HRD as evaluated by the Myriad classifier and the benefit from olaparib in the PAOLA-1 cohort (n = 447) in a blinded manner.

RESULTS

In contrast to the HRD prediction AUROC of 72 % on hold-out, our model only yielded an AUROC of 57 % external. Kaplan-Meier analysis showed that progression free survival (PFS) in the PARP inhibitor treated PAOLA-1 patients was significantly improved in the HRD positive group as defined by our model, but not in the HRD negative group. PFS improvement in PARP inhibitor-treated patients was substantially longer in our HRD positive group, hinting at a biologically meaningful prediction of benefit from PARP inhibitors.

CONCLUSION

Together, our results indicate that it might be possible to generate a predictor of benefit from PARP inhibitors based on the DL-mediated analysis of WSIs. However, further studies with larger cohorts and further methodological improvements will be necessary to generate a predictor with clinically useful accuracy across independent patient cohorts.

Predicting BRAF Mutations in Cutaneous Melanoma Patients Using Neural Network Analysis

Point mutations at codon 600 of the BRAF oncogene are the most common alterations in cutaneous melanoma (CM). Assessment of BRAF status allows to personalize patient management, though the affordability of molecular testing is limited in some countries. This study aimed to develop a model for predicting alteration in BRAF based on routinely available clinical and histological data. Methods: For identifying the key factors associated with point mutations in BRAF, 2041 patients with CM were recruited in the study. The presence of BRAF mutations was an endpoint. The variables included demographic data (gender and age), anatomic location, stage, histological subtype, number of mitosis, and also such features as ulceration, Clark level, Breslow thickness, infiltration by lymphocytes, invasiveness, regression, microsatellites, and association with nevi. Results: A relatively high rate of BRAF mutation was revealed in the Ukrainian cohort of patients with CM. BRAF-mutant melanoma was associated with younger age and location of nonsun-exposed skin. Besides, sex-specific differences were found between CM of various anatomic distributions and the frequency of distinct BRAF mutation subtypes. A minimal set of variables linked to BRAF mutations, defined by the genetic input selection algorithm, included patient age, primary tumor location, histological type, lymphovascular invasion, ulceration, and association with nevi. To encounter nonlinear links, neural network modeling was applied resulting in a multilayer perceptron (MLP) with one hidden layer. Its architecture included four neurons with a logistic activation function. The AUROCMLP6 of the MLP model comprised 0.79 (95% CІ: 0.74-0.84). Under the optimal threshold, the model demonstrated the following parameters: sensitivity: 89.4% (95% CІ: 84.5%-93.1%), specificity: 50.7% (95% CІ: 42.2%-59.1%), positive predictive value: 73.1% (95% CІ: 69.6%-76.3%), and negative predictive value: 76.0% (95% CІ: 67.6%-82.8%). The developed MLP model enables the prediction of the mutation in BRAF oncogene in CM, alleviating decisions on personalized management of patients with CM. In conclusion, the developed MLP model, which relies on the assessment of 6 variables, can predict the BRAF mutation status in patients with CM, supporting decisions on patient management.

Imaging Analytics using Artificial Intelligence in Oncology: A Comprehensive Review

The present era has seen a surge in artificial intelligence-related research in oncology, mainly using deep learning, because of powerful computer hardware, improved algorithms and the availability of large amounts of data from open-source domains and the use of transfer learning. Here we discuss the multifaceted role of deep learning in cancer care, ranging from risk stratification, the screening and diagnosis of cancer, to the prediction of genomic mutations, treatment response and survival outcome prediction, through the use of convolutional neural networks. Another role of artificial intelligence is in the generation of automated radiology reports, which is a boon in high-volume centres to minimise report turnaround time. Although a validated and deployable deep-learning model for clinical use is still in its infancy, there is ongoing research to overcome the barriers for its universal implementation and we also delve into this aspect. We also briefly describe the role of radiomics in oncoimaging. Artificial intelligence can provide answers pertaining to cancer management at baseline imaging, saving cost and time. Imaging biobanks, which are repositories of anonymised images, are also briefly described. We also discuss the commercialisation and ethical issues pertaining to artificial intelligence. The latest generation generalist artificial intelligence model is also briefly described at the end of the article. We believe this article will not only enrich knowledge, but also promote research acumen in the minds of readers to take oncoimaging to another level using artificial intelligence and also work towards clinical translation of such research.

The Promises and Perils of Foundation Models in Dermatology

Foundation models (FM), which are large-scale artificial intelligence (AI) models that can complete a range of tasks, represent a paradigm shift in AI. These versatile models encompass large language models, vision-language models, and multimodal models. Although these models are often trained for broad tasks, they have been applied either out of the box or after additional fine tuning to tasks in medicine, including dermatology. From addressing administrative tasks to answering dermatology questions, these models are poised to have an impact on dermatology care delivery. As FMs become more ubiquitous in health care, it is important for clinicians and dermatologists to have a basic understanding of how these models are developed, what they are capable of, and what pitfalls exist. In this paper, we present a comprehensive yet accessible overview of the current state of FMs and summarize their current applications in dermatology, highlight their limitations, and discuss future developments in the field.

Multimodal MALDI imaging mass spectrometry for improved diagnosis of melanoma

Imaging mass spectrometry (IMS) provides promising avenues to augment histopathological investigation with rich spatio-molecular information. We have previously developed a classification model to differentiate melanoma from nevi lesions based on IMS protein data, a task that is challenging solely by histopathologic evaluation. Most IMS-focused studies collect microscopy in tandem with IMS data, but this microscopy data is generally omitted in downstream data analysis. Microscopy, nevertheless, forms the basis for traditional histopathology and thus contains invaluable morphological information. In this work, we developed a multimodal classification pipeline that uses deep learning, in the form of a pre-trained artificial neural network, to extract the meaningful morphological features from histopathological images, and combine it with the IMS data. To test whether this deep learning-based classification strategy can improve on our previous results in classification of melanocytic neoplasia, we utilized MALDI IMS data with collected serial H&E stained sections for 331 patients, and compared this multimodal classification pipeline to classifiers using either exclusively microscopy or IMS data. The multimodal pipeline achieved the best performance, with ROC-AUCs of 0.968 vs. 0.938 vs. 0.931 for the multimodal, unimodal microscopy and unimodal IMS pipelines respectively. Due to the use of a pre-trained network to perform the morphological feature extraction, this pipeline does not require any training on large amounts of microscopy data. As such, this framework can be readily applied to improve classification performance in other experimental settings where microscopy data is acquired in tandem with IMS experiments.

Evaluating deep learning-based melanoma classification using immunohistochemistry and routine histology: A three center study

Pathologists routinely use immunohistochemical (IHC)-stained tissue slides against MelanA in addition to hematoxylin and eosin (H&E)-stained slides to improve their accuracy in diagnosing melanomas. The use of diagnostic Deep Learning (DL)-based support systems for automated examination of tissue morphology and cellular composition has been well studied in standard H&E-stained tissue slides. In contrast, there are few studies that analyze IHC slides using DL. Therefore, we investigated the separate and joint performance of ResNets trained on MelanA and corresponding H&E-stained slides. The MelanA classifier achieved an area under receiver operating characteristics curve (AUROC) of 0.82 and 0.74 on out of distribution (OOD)-datasets, similar to the H&E-based benchmark classification of 0.81 and 0.75, respectively. A combined classifier using MelanA and H&E achieved AUROCs of 0.85 and 0.81 on the OOD datasets. DL MelanA-based assistance systems show the same performance as the benchmark H&E classification and may be improved by multi stain classification to assist pathologists in their clinical routine.

A 3-dimensional histology computer model of malignant melanoma and its implications for digital pathology

BACKGROUND

Historically, cancer diagnoses have been made by pathologists using two-dimensional histological slides. However, with the advent of digital pathology and artificial intelligence, slides are being digitised, providing new opportunities to integrate their information. Since nature is 3-dimensional (3D), it seems intuitive to digitally reassemble the 3D structure for diagnosis.

OBJECTIVE

To develop the first human-3D-melanoma-histology-model with full data and code availability. Further, to evaluate the 3D-simulation together with experienced pathologists in the field and discuss the implications of digital 3D-models for the future of digital pathology.

METHODS

A malignant melanoma of the skin was digitised via 3 µm cuts by a slide scanner; an open-source software was then leveraged to construct the 3D model. A total of nine pathologists from four different countries with at least 10 years of experience in the histologic diagnosis of melanoma tested the model and discussed their experiences as well as implications for future pathology.

RESULTS

We successfully constructed and tested the first 3D-model of human melanoma. Based on testing, 88.9% of pathologists believe that the technology is likely to enter routine pathology within the next 10 years; advantages include a better reflectance of anatomy, 3D assessment of symmetry and the opportunity to simultaneously evaluate different tissue levels at the same time; limitations include the high consumption of tissue and a yet inferior resolution due to computational limitations.

CONCLUSIONS

3D-histology-models are promising for digital pathology of cancer and melanoma specifically, however, there are yet limitations which need to be carefully addressed.

Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal

Artificial Intelligence (AI) is progressively permeating medicine, notably in the realm of assisted diagnosis. However, the traditional unimodal AI models, reliant on large volumes of accurately labeled data and single data type usage, prove insufficient to assist dermatological diagnosis. Augmenting these models with text data from patient narratives, laboratory reports, and image data from skin lesions, dermoscopy, and pathologies could significantly enhance their diagnostic capacity. Large-scale pre-training multimodal models offer a promising solution, exploiting the burgeoning reservoir of clinical data and amalgamating various data types. This paper delves into unimodal models' methodologies, applications, and shortcomings while exploring how multimodal models can enhance accuracy and reliability. Furthermore, integrating cutting-edge technologies like federated learning and multi-party privacy computing with AI can substantially mitigate patient privacy concerns in dermatological datasets and further fosters a move towards high-precision self-diagnosis. Diagnostic systems underpinned by large-scale pre-training multimodal models can facilitate dermatology physicians in formulating effective diagnostic and treatment strategies and herald a transformative era in healthcare.