The Digital Slide Archive: A Software Platform for Management, Integration, and Analysis of Histology for Cancer Research

NuCLS: A scalable crowdsourcing approach and dataset for nucleus classification and segmentation in breast cancer

BACKGROUND

Deep learning enables accurate high-resolution mapping of cells and tissue structures that can serve as the foundation of interpretable machine-learning models for computational pathology. However, generating adequate labels for these structures is a critical barrier, given the time and effort required from pathologists.

RESULTS

This article describes a novel collaborative framework for engaging crowds of medical students and pathologists to produce quality labels for cell nuclei. We used this approach to produce the NuCLS dataset, containing >220,000 annotations of cell nuclei in breast cancers. This builds on prior work labeling tissue regions to produce an integrated tissue region- and cell-level annotation dataset for training that is the largest such resource for multi-scale analysis of breast cancer histology. This article presents data and analysis results for single and multi-rater annotations from both non-experts and pathologists. We present a novel workflow that uses algorithmic suggestions to collect accurate segmentation data without the need for laborious manual tracing of nuclei. Our results indicate that even noisy algorithmic suggestions do not adversely affect pathologist accuracy and can help non-experts improve annotation quality. We also present a new approach for inferring truth from multiple raters and show that non-experts can produce accurate annotations for visually distinctive classes.

CONCLUSIONS

This study is the most extensive systematic exploration of the large-scale use of wisdom-of-the-crowd approaches to generate data for computational pathology applications.

Narrative online guides for the interpretation of digital-pathology images and tissue-atlas data

Multiplexed tissue imaging facilitates the diagnosis and understanding of complex disease traits. However, the analysis of such digital images heavily relies on the experience of anatomical pathologists for the review, annotation and description of tissue features. In addition, the wider use of data from tissue atlases in basic and translational research and in classrooms would benefit from software that facilitates the easy visualization and sharing of the images and the results of their analyses. In this Perspective, we describe the ecosystem of software available for the analysis of tissue images and discuss the need for interactive online guides that help histopathologists make complex images comprehensible to non-specialists. We illustrate this idea via a software interface (Minerva), accessible via web browsers, that integrates multi-omic and tissue-atlas features. We argue that such interactive narrative guides can effectively disseminate digital histology data and aid their interpretation.

Developing image analysis methods for digital pathology

The potential to use quantitative image analysis and artificial intelligence is one of the driving forces behind digital pathology. However, despite novel image analysis methods for pathology being described across many publications, few become widely adopted and many are not applied in more than a single study. The explanation is often straightforward: software implementing the method is simply not available, or is too complex, incomplete, or dataset‐dependent for others to use. The result is a disconnect between what seems already possible in digital pathology based upon the literature, and what actually is possible for anyone wishing to apply it using currently available software. This review begins by introducing the main approaches and techniques involved in analysing pathology images. I then examine the practical challenges inherent in taking algorithms beyond proof‐of‐concept, from both a user and developer perspective. I describe the need for a collaborative and multidisciplinary approach to developing and validating meaningful new algorithms, and argue that openness, implementation, and usability deserve more attention among digital pathology researchers. The review ends with a discussion about how digital pathology could benefit from interacting with and learning from the wider bioimage analysis community, particularly with regard to sharing data, software, and ideas. © 2022 The Author. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Semantic annotation for computational pathology: multidisciplinary experience and best practice recommendations

Recent advances in whole-slide imaging (WSI) technology have led to the development of a myriad of computer vision and artificial intelligence-based diagnostic, prognostic, and predictive algorithms. Computational Pathology (CPath) offers an integrated solution to utilise information embedded in pathology WSIs beyond what can be obtained through visual assessment. For automated analysis of WSIs and validation of machine learning (ML) models, annotations at the slide, tissue, and cellular levels are required. The annotation of important visual constructs in pathology images is an important component of CPath projects. Improper annotations can result in algorithms that are hard to interpret and can potentially produce inaccurate and inconsistent results. Despite the crucial role of annotations in CPath projects, there are no well-defined guidelines or best practices on how annotations should be carried out. In this paper, we address this shortcoming by presenting the experience and best practices acquired during the execution of a large-scale annotation exercise involving a multidisciplinary team of pathologists, ML experts, and researchers as part of the Pathology image data Lake for Analytics, Knowledge and Education (PathLAKE) consortium. We present a real-world case study along with examples of different types of annotations, diagnostic algorithm, annotation data dictionary, and annotation constructs. The analyses reported in this work highlight best practice recommendations that can be used as annotation guidelines over the lifecycle of a CPath project.

MITI minimum information guidelines for highly multiplexed tissue images

The imminent release of tissue atlases combining multi-channel microscopy with single cell sequencing and other omics data from normal and diseased specimens creates an urgent need for data and metadata standards that guide data deposition, curation and release. We describe a Minimum Information about highly multiplexed Tissue Imaging (MITI) standard that applies best practices developed for genomics and other microscopy data to highly multiplexed tissue images and traditional histology.

A cloud-based tool for federated segmentation of whole slide images

It is commonly known that diverse datasets of WSIs are beneficial when training convolutional neural networks, however sharing medical data between institutions is often hindered by regulatory concerns. We have developed a cloud-based tool for federated WSI segmentation, allowing collaboration between institutions without the need to directly share data. To show the feasibility of federated learning on pathology data in the real world, We demonstrate this tool by segmenting IFTA from three institutions and show that keeping the three datasets separate does not hinder segmentation performance. This pipeline is deployed in the cloud for easy access for data viewing and annotation by each site's respective constituents.

Advances in Deep Neuropathological Phenotyping of Alzheimer Disease: Past, Present, and Future

Alzheimer disease (AD) is a neurodegenerative disorder characterized pathologically by the presence of neurofibrillary tangles and amyloid beta (Aβ) plaques in the brain. The disease was first described in 1906 by Alois Alzheimer, and since then, there have been many advancements in technologies that have aided in unlocking the secrets of this devastating disease. Such advancements include improving microscopy and staining techniques, refining diagnostic criteria for the disease, and increased appreciation for disease heterogeneity both in neuroanatomic location of abnormalities as well as overlap with other brain diseases; for example, Lewy body disease and vascular dementia. Despite numerous advancements, there is still much to achieve as there is not a cure for AD and postmortem histological analyses is still the gold standard for appreciating AD neuropathologic changes. Recent technological advances such as in-vivo biomarkers and machine learning algorithms permit great strides in disease understanding, and pave the way for potential new therapies and precision medicine approaches. Here, we review the history of human AD neuropathology research to include the notable advancements in understanding common co-pathologies in the setting of AD, and microscopy and staining methods. We also discuss future approaches with a specific focus on deep phenotyping using machine learning.

Explainable nucleus classification using Decision Tree Approximation of Learned Embeddings

MOTIVATION

Nucleus detection, segmentation and classification are fundamental to high-resolution mapping of the tumor microenvironment using whole-slide histopathology images. The growing interest in leveraging the power of deep learning to achieve state-of-the-art performance often comes at the cost of explainability, yet there is general consensus that explainability is critical for trustworthiness and widespread clinical adoption. Unfortunately, current explainability paradigms that rely on pixel saliency heatmaps or superpixel importance scores are not well-suited for nucleus classification. Techniques like Grad-CAM or LIME provide explanations that are indirect, qualitative and/or nonintuitive to pathologists.

RESULTS

In this article, we present techniques to enable scalable nuclear detection, segmentation and explainable classification. First, we show how modifications to the widely used Mask R-CNN architecture, including decoupling the detection and classification tasks, improves accuracy and enables learning from hybrid annotation datasets like NuCLS, which contain mixtures of bounding boxes and segmentation boundaries. Second, we introduce an explainability method called Decision Tree Approximation of Learned Embeddings (DTALE), which provides explanations for classification model behavior globally, as well as for individual nuclear predictions. DTALE explanations are simple, quantitative, and can flexibly use any measurable morphological features that make sense to practicing pathologists, without sacrificing model accuracy. Together, these techniques present a step toward realizing the promise of computational pathology in computer-aided diagnosis and discovery of morphologic biomarkers.

AVAILABILITY AND IMPLEMENTATION

Relevant code can be found at github.com/CancerDataScience/NuCLS.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Building tools for machine learning and artificial intelligence in cancer research: best practices and a case study with the PathML toolkit for computational pathology

Imaging datasets in cancer research are growing exponentially in both quantity and information density. These massive datasets may enable derivation of insights for cancer research and clinical care, but only if researchers are equipped with the tools to leverage advanced computational analysis approaches such as machine learning and artificial intelligence. In this work, we highlight three themes to guide development of such computational tools: scalability, standardization, and ease of use. We then apply these principles to develop PathML, a general-purpose research toolkit for computational pathology. We describe the design of the PathML framework and demonstrate applications in diverse use-cases. PathML is publicly available at www.pathml.com.

Profiling Glioblastoma Cases with an Expression of DCX, OLIG2 and NES

Glioblastoma (GBM) remains the leading cause of cancer-related deaths with the lowest five-year survival rates among all of the human cancers. Multiple factors contribute to its poor outcome, including intratumor heterogeneity, along with migratory and invasive capacities of tumour cells. Over the last several years Doublecortin (DCX) has been one of the debatable factors influencing GBM cells’ migration. To resolve DCX’s ambiguous role in GBM cells’ migration, we set to analyse the expression patterns of DCX along with Nestin (NES) and Oligodendrocyte lineage transcription factor 2 (OLIG2) in 17 cases of GBM, using immunohistochemistry, followed by an analysis of single-cell RNA-seq data. Our results showed that only a small subset of DCX positive (DCX⁺) cells was present in the tumour. Moreover, no particular pattern emerged when analysing DCX⁺ cells relative position to the tumour margin. By looking into single-cell RNA-seq data, the majority of DCX⁺ cells were classified as non-cancerous, with a small subset of cells that could be regarded as glioma stem cells. In conclusion, our findings support the notion that glioma cells express DCX; however, there is no clear evidence to prove that DCX participates in GBM cell migration.

PodoSighter: A Cloud-Based Tool for Label-Free Podocyte Detection in Kidney Whole-Slide Images

BACKGROUND

Podocyte depletion precedes progressive glomerular damage in several kidney diseases. However, the current standard of visual detection and quantification of podocyte nuclei from brightfield microscopy images is laborious and imprecise.

METHODS

We have developed PodoSighter, an online cloud-based tool, to automatically identify and quantify podocyte nuclei from giga-pixel brightfield whole-slide images (WSIs) using deep learning. Ground-truth to train the tool used immunohistochemically or immunofluorescence-labeled images from a multi-institutional cohort of 122 histologic sections from mouse, rat, and human kidneys. To demonstrate the generalizability of our tool in investigating podocyte loss in clinically relevant samples, we tested it in rodent models of glomerular diseases, including diabetic kidney disease, crescentic GN, and dose-dependent direct podocyte toxicity and depletion, and in human biopsies from steroid-resistant nephrotic syndrome and from human autopsy tissues.

RESULTS

The optimal model yielded high sensitivity/specificity of 0.80/0.80, 0.81/0.86, and 0.80/0.91, in mouse, rat, and human images, respectively, from periodic acid-Schiff-stained WSIs. Furthermore, the podocyte nuclear morphometrics extracted using PodoSighter were informative in identifying diseased glomeruli. We have made PodoSighter freely available to the general public as turnkey plugins in a cloud-based web application for end users.

CONCLUSIONS

Our study demonstrates an automated computational approach to detect and quantify podocyte nuclei in standard histologically stained WSIs, facilitating podocyte research, and enabling possible future clinical applications.

Assessment of Digital Pathology Imaging Biomarkers Associated with Breast Cancer Histologic Grade

Background: Evaluating histologic grade for breast cancer diagnosis is standard and associated with prognostic outcomes. Current challenges include the time required for manual microscopic evaluation and interobserver variability. This study proposes a computer-aided diagnostic (CAD) pipeline for grading tumors using artificial intelligence. Methods: There were 138 patients included in this retrospective study. Breast core biopsy slides were prepared using standard laboratory techniques, digitized, and pre-processed for analysis. Deep convolutional neural networks (CNNs) were developed to identify the regions of interest containing malignant cells and to segment tumor nuclei. Imaging-based features associated with spatial parameters were extracted from the segmented regions of interest (ROIs). Clinical datasets and pathologic biomarkers (estrogen receptor, progesterone receptor, and human epidermal growth factor 2) were collected from all study subjects. Pathologic, clinical, and imaging-based features were input into machine learning (ML) models to classify histologic grade, and model performances were tested against ground-truth labels at the patient-level. Classification performances were evaluated using receiver-operating characteristic (ROC) analysis. Results: Multiparametric feature sets, containing both clinical and imaging-based features, demonstrated high classification performance. Using imaging-derived markers alone, the classification performance demonstrated an area under the curve (AUC) of 0.745, while modeling these features with other pathologic biomarkers yielded an AUC of 0.836. Conclusion: These results demonstrate an association between tumor nuclear spatial features and tumor grade. If further validated, these systems may be implemented into pathology CADs and can assist pathologists to expeditiously grade tumors at the time of diagnosis and to help guide clinical decisions.

Image Analysis Pipeline for Renal Allograft Evaluation and Fibrosis Quantification

INTRODUCTION

Digital pathology improves the standardization and reproducibility of kidney biopsy specimen assessment. We developed a pipeline allowing the analysis of many images without requiring human preprocessing and illustrate its use with a simple algorithm for quantification of interstitial fibrosis on a large dataset of kidney allograft biopsy specimens.

METHODS

Masson trichrome-stained images from kidney allograft biopsy specimens were used to train and validate a glomeruli detection algorithm using a VGG19 convolutional neural network and an automatic cortical region of interest (ROI) selection algorithm including cortical regions containing all predicted glomeruli. A positive-pixel count algorithm was used to quantify interstitial fibrosis on the ROIs and the association between automatic fibrosis and pathologist evaluation, estimated glomerular filtration rate (GFR) and allograft survival was assessed.

RESULTS

The glomeruli detection (F1 score of 0.87) and ROIs selection (F1 score 0.83 [SD 0.13]) algorithms displayed high accuracy. The correlation between the automatic fibrosis quantification on manually and automatically selected ROIs was high (r = 1.00 [0.99-1.00]). Automatic fibrosis quantification was only moderately correlated with pathologists' assessment and was not significantly associated with eGFR or allograft survival.

CONCLUSION

This pipeline can automatically and accurately detect glomeruli and select cortical ROIs that can easily be used to develop, validate, and apply image analysis algorithms.

Automated Computational Detection of Interstitial Fibrosis, Tubular Atrophy, and Glomerulosclerosis

BACKGROUND

Interstitial fibrosis, tubular atrophy (IFTA), and glomerulosclerosis are indicators of irrecoverable kidney injury. Modern machine learning (ML) tools have enabled robust, automated identification of image structures that can be comparable with analysis by human experts. ML algorithms were developed and tested for the ability to replicate the detection and quantification of IFTA and glomerulosclerosis that renal pathologists perform.

METHODS

A renal pathologist annotated renal biopsy specimens from 116 whole-slide images (WSIs) for IFTA and glomerulosclerosis. A total of 79 WSIs were used for training different configurations of a convolutional neural network (CNN), and 17 and 20 WSIs were used as internal and external testing cases, respectively. The best model was compared against the input of four renal pathologists on 20 new testing slides. Further, for 87 testing biopsy specimens, IFTA and glomerulosclerosis measurements made by pathologists and the CNN were correlated to patient outcome using classic statistical tools.

RESULTS

The best average performance across all image classes came from a DeepLab version 2 network trained at 40× magnification. IFTA and glomerulosclerosis percentages derived from this CNN achieved high levels of agreement with four renal pathologists. The pathologist- and CNN-based analyses of IFTA and glomerulosclerosis showed statistically significant and equivalent correlation with all patient-outcome variables.

CONCLUSIONS

ML algorithms can be trained to replicate the IFTA and glomerulosclerosis assessment performed by renal pathologists. This suggests computational methods may be able to provide a standardized approach to evaluate the extent of chronic kidney injury in situations in which renal-pathologist time is restricted or unavailable.

CytoBrowser: a browser-based collaborative annotation platform for whole slide images

We present CytoBrowser, an open-source (GPLv3) JavaScript and Node.js driven environment for fast and accessible collaborative online visualization, assessment, and annotation of very large microscopy images, including, but not limited to, z-stacks (focus stacks) of cytology or histology whole slide images. CytoBrowser provides a web-based viewer for high-resolution zoomable images and facilitates easy remote collaboration, with options for joint-view visualization and simultaneous collaborative annotation of very large datasets. It delivers a unique combination of functionalities not found in other software solutions, making it a preferred tool for large scale annotation of whole slide image data. The web browser interface is directly accessible on any modern computer or even on a mobile phone, without need for additional software. By sharing a "session", several remote users can interactively explore and jointly annotate whole slide image data, thereby reaching improved data understanding and annotation quality, effortless project scaling and distribution of resources to/from remote locations, efficient creation of "ground truth" annotations for methods' evaluation and training of machine learning-based approaches, a user-friendly learning environment for medical students, to just name a few. Rectangle and polygon region annotations complement point-based annotations, each with a selectable annotation-class as well as free-form text fields. The default setting of CytoBrowser presents an interface for the Bethesda cancer grading system, while other annotation schemes can easily be incorporated. Automatic server side storage of annotations is complemented by JSON-based import/export options facilitating easy interoperability with other tools. CytoBrowser is available here: https://mida-group.github.io/CytoBrowser/.

Artificial intelligence and algorithmic computational pathology: an introduction with renal allograft examples

Whole slide imaging, which is an important technique in the field of digital pathology, has recently been the subject of increased interest and avenues for utilisation, and with more widespread whole slide image (WSI) utilisation, there will also be increased interest in and implementation of image analysis (IA) techniques. IA includes artificial intelligence (AI) and targeted or hypothesis-driven algorithms. In the overall pathology field, the number of citations related to these topics has increased in recent years. Renal pathology is one anatomical pathology subspecialty that has utilised WSIs and IA algorithms; it can be argued that renal transplant pathology could be particularly suited for whole slide imaging and IA, as renal transplant pathology is frequently classified by use of the semiquantitative Banff classification of renal allograft pathology. Hypothesis-driven/targeted algorithms have been used in the past for the assessment of a variety of features in the kidney (e.g. interstitial fibrosis, tubular atrophy, inflammation); in recent years, the amount of research has particularly increased in the area of AI/machine learning for the identification of glomeruli, for histological segmentation, and for other applications. Deep learning is the form of machine learning that is most often used for such AI approaches to the 'big data' of pathology WSIs, and deep learning methods such as artificial neural networks (ANNs)/convolutional neural networks (CNNs) are utilised. Unsupervised and supervised AI algorithms can be employed to accomplish image or semantic classification. In this review, AI and other IA algorithms applied to WSIs are discussed, and examples from renal pathology are covered, with an emphasis on renal transplant pathology.

Interactive Classification of Whole-Slide Imaging Data for Cancer Researchers

Whole-slide histology images contain information that is valuable for clinical and basic science investigations of cancer but extracting quantitative measurements from these images is challenging for researchers who are not image analysis specialists. In this article, we describe HistomicsML2, a software tool for learn-by-example training of machine learning classifiers for histologic patterns in whole-slide images. This tool improves training efficiency and classifier performance by guiding users to the most informative training examples for labeling and can be used to develop classifiers for prospective application or as a rapid annotation tool that is adaptable to different cancer types. HistomicsML2 runs as a containerized server application that provides web-based user interfaces for classifier training, validation, exporting inference results, and collaborative review, and that can be deployed on GPU servers or cloud platforms. We demonstrate the utility of this tool by using it to classify tumor-infiltrating lymphocytes in breast carcinoma and cutaneous melanoma. SIGNIFICANCE: An interactive machine learning tool for analyzing digital pathology images enables cancer researchers to apply this tool to measure histologic patterns for clinical and basic science studies.

A Distributed System Improves Inter-Observer and AI Concordance in Annotating Interstitial Fibrosis and Tubular Atrophy

Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist's visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff's alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other's annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

User friendly, cloud based, whole slide image segmentation

Convolutional neural networks, the state of the art for image segmentation, have been successfully applied to histology images by many computational researchers. However, the translatability of this technology to clinicians and biological researchers is limited due to the complex and undeveloped user interface of the code, as well as the extensive computer setup required. We have developed a plugin for segmentation of whole slide images (WSIs) with an easy to use graphical user interface. This plugin runs a state-of-the-art convolutional neural network for segmentation of WSIs in the cloud. Our plugin is built on the open source tool HistomicsTK by Kitware Inc. (Clifton Park, NY), which provides remote data management and viewing abilities for WSI datasets. The ability to access this tool over the internet will facilitate widespread use by computational non-experts. Users can easily upload slides to a server where our plugin is installed and perform the segmentation analysis remotely. This plugin is open source and once trained, has the ability to be applied to the segmentation of any pathological structure. For a proof of concept, we have trained it to segment glomeruli from renal tissue images, demonstrating it on holdout tissue slides.

Analysis of tumor nuclear features using artificial intelligence to predict response to neoadjuvant chemotherapy in high-risk breast cancer patients

PURPOSE

Neoadjuvant chemotherapy (NAC) is used to treat patients with high-risk breast cancer. The tumor response to NAC can be classified as either a pathological partial response (pPR) or pathological complete response (pCR), defined as complete eradication of invasive tumor cells, with a pCR conferring a significantly lower risk of recurrence. Predicting the response to NAC, however, remains a significant clinical challenge. The objective of this study was to determine if analysis of nuclear features on core biopsies using artificial intelligence (AI) can predict response to NAC.

METHODS

Fifty-eight HER2-positive or triple-negative breast cancer patients were included in this study (pCR n = 37, pPR n = 21). Multiple deep convolutional neural networks were developed to automate tumor detection and nuclear segmentation. Nuclear count, area, and circularity, as well as image-based first- and second-order features including mean pixel intensity and correlation of the gray-level co-occurrence matrix (GLCM-COR) were determined.

RESULTS

In univariate analysis, the pCR group had fewer multifocal/multicentric tumors, higher nuclear intensity, and lower GLCM-COR compared to the pPR group. In multivariate binary logistic regression, tumor multifocality/multicentricity (OR = 0.14, p = 0.012), nuclear intensity (OR = 1.23, p = 0.018), and GLCM-COR (OR = 0.96, p = 0.043) were each independently associated with likelihood of achieving a pCR, and the model was able to successful classify 79% of cases (62% for pPR and 89% for pCR).

CONCLUSION

Analysis of tumor nuclear features using digital pathology/AI can significantly improve models to predict pathological response to NAC.

OncoMX: A Knowledgebase for Exploring Cancer Biomarkers in the Context of Related Cancer and Healthy Data

PURPOSE

The purpose of OncoMX¹ knowledgebase development was to integrate cancer biomarker and relevant data types into a meta-portal, enabling the research of cancer biomarkers side by side with other pertinent multidimensional data types.

METHODS

Cancer mutation, cancer differential expression, cancer expression specificity, healthy gene expression from human and mouse, literature mining for cancer mutation and cancer expression, and biomarker data were integrated, unified by relevant biomedical ontologies, and subjected to rule-based automated quality control before ingestion into the database.

RESULTS

OncoMX provides integrated data encompassing more than 1,000 unique biomarker entries (939 from the Early Detection Research Network [EDRN] and 96 from the US Food and Drug Administration) mapped to 20,576 genes that have either mutation or differential expression in cancer. Sentences reporting mutation or differential expression in cancer were extracted from more than 40,000 publications, and healthy gene expression data with samples mapped to organs are available for both human genes and their mouse orthologs.

CONCLUSION

OncoMX has prioritized user feedback as a means of guiding development priorities. By mapping to and integrating data from several cancer genomics resources, it is hoped that OncoMX will foster a dynamic engagement between bioinformaticians and cancer biomarker researchers. This engagement should culminate in a community resource that substantially improves the ability and efficiency of exploring cancer biomarker data and related multidimensional data.

FalseColor-Python: A rapid intensity-leveling and digital-staining package for fluorescence-based slide-free digital pathology

Slide-free digital pathology techniques, including nondestructive 3D microscopy, are gaining interest as alternatives to traditional slide-based histology. In order to facilitate clinical adoption of these fluorescence-based techniques, software methods have been developed to convert grayscale fluorescence images into color images that mimic the appearance of standard absorptive chromogens such as hematoxylin and eosin (H&E). However, these false-coloring algorithms often require manual and iterative adjustment of parameters, with results that can be inconsistent in the presence of intensity nonuniformities within an image and/or between specimens (intra- and inter-specimen variability). Here, we present an open-source (Python-based) rapid intensity-leveling and digital-staining package that is specifically designed to render two-channel fluorescence images (i.e. a fluorescent analog of H&E) to the traditional H&E color space for 2D and 3D microscopy datasets. However, this method can be easily tailored for other false-coloring needs. Our package offers (1) automated and uniform false coloring in spite of uneven staining within a large thick specimen, (2) consistent color-space representations that are robust to variations in staining and imaging conditions between different specimens, and (3) GPU-accelerated data processing to allow these methods to scale to large datasets. We demonstrate this platform by generating H&E-like images from cleared tissues that are fluorescently imaged in 3D with open-top light-sheet (OTLS) microscopy, and quantitatively characterizing the results in comparison to traditional slide-based H&E histology.

Artificial Intelligence and Mechanistic Modeling for Clinical Decision Making in Oncology

The amount of "big" data generated in clinical oncology, whether from molecular, imaging, pharmacological, or biological origin, brings novel challenges. To mine efficiently this source of information, mathematical models able to produce predictive algorithms and simulations are required, with applications for diagnosis, prognosis, drug development, or prediction of the response to therapy. Such mathematical and computational constructs can be subdivided into two broad classes: biologically agnostic, statistical models using artificial intelligence techniques, and physiologically based, mechanistic models. In this review, recent advances in the applications of such methods in clinical oncology are outlined. These include machine learning applied to big data (omics, imaging, or electronic health records), pharmacometrics and quantitative systems pharmacology, as well as tumor kinetics and metastasis modeling. Focus is set on studies with high potential of clinical translation, and particular attention is given to cancer immunotherapy. Perspectives are given in terms of combinations of the two approaches: "mechanistic learning."

AI in Medical Imaging Informatics: Current Challenges and Future Directions

This paper reviews state-of-the-art research solutions across the spectrum of medical imaging informatics, discusses clinical translation, and provides future directions for advancing clinical practice. More specifically, it summarizes advances in medical imaging acquisition technologies for different modalities, highlighting the necessity for efficient medical data management strategies in the context of AI in big healthcare data analytics. It then provides a synopsis of contemporary and emerging algorithmic methods for disease classification and organ/ tissue segmentation, focusing on AI and deep learning architectures that have already become the de facto approach. The clinical benefits of in-silico modelling advances linked with evolving 3D reconstruction and visualization applications are further documented. Concluding, integrative analytics approaches driven by associate research branches highlighted in this study promise to revolutionize imaging informatics as known today across the healthcare continuum for both radiology and digital pathology applications. The latter, is projected to enable informed, more accurate diagnosis, timely prognosis, and effective treatment planning, underpinning precision medicine.

Comparative analysis of prognostic histopathologic parameters in subtypes of epithelioid pleural mesothelioma

AIMS

Malignant pleural mesothelioma (MPM) is a rare malignancy with a dismal prognosis. While the epithelioid type is associated with a more favourable outcome, additional factors are needed to further stratify prognosis and to identify patients who can benefit from multimodal treatment. As epithelioid MPM shows remarkable morphological variability, the prognostic role of the five defined morphologies, the impact of the nuclear grading system and the mitosis-necrosis score were investigated in this study.

METHODS AND RESULTS

Tumour specimens of 192 patients with epithelioid MPM from five European centres were histologically subtyped. Nuclear grading and mitosis-necrosis score were determined and correlated with clinicopathological parameters and overall survival (OS). Digital slides of 55 independent cases from The Cancer Genome Atlas (TCGA) database were evaluated for external validation. Histological subtypes were collapsed into three groups based on their overlapping survival curves. The tubulopapillary/microcystic group had a significantly longer OS than the solid/trabecular group (732 days versus 397 days, P = 0.0013). Pleomorphic tumours had the shortest OS (173 days). The solid/trabecular variants showed a significant association with high nuclear grade and mitosis-necrosis score. The mitosis-necrosis score was a robust and independent prognostic factor in our patient cohort. The prognostic significance of all three parameters was externally validated in the TCGA cohort. Patients with tubulopapillary or microcystic tumours showed a greater improvement in OS after receiving multimodal therapy than those with solid or trabecular tumours.

CONCLUSIONS

Histological subtypes of epithelioid MPM have a prognostic impact, and might help to select patients for intensive multimodal treatment approaches.

Radiomics Features Predict CIC Mutation Status in Lower Grade Glioma

MRI in combination with genomic markers are critical in the management of gliomas. Radiomics and radiogenomics analysis facilitate the quantitative assessment of tumor properties which can be used to model both molecular subtype and predict disease progression. In this work, we report on the Drosophila gene capicua (CIC) mutation biomarker effects alongside radiomics features on the predictive ability of CIC mutation status in lower-grade gliomas (LGG). Genomic data of lower grade glioma (LGG) patients from The Cancer Genome Atlas (TCGA) (n = 509) and corresponding MR images from TCIA (n = 120) were utilized. Following tumor segmentation, radiomics features were extracted from T1, T2, T2 Flair, and T1 contrast enhanced (CE) images. Lasso feature reduction was used to obtain the most important MR image features and then logistic regression used to predict CIC mutation status. In our study, CIC mutation rarely occurred in Astrocytoma but has a high probability of occurrence in Oligodendroglioma. The presence of CIC mutation was found to be associated with better survival of glioma patients (p < 1e−4, HR: 0.2445), even with co-occurrence of IDH mutation and 1p/19q co-deletion (p = 0.0362, HR: 0.3674). An eleven-feature model achieved glioma prediction accuracy of 94.2% (95% CI, 94.03–94.38%), a six-feature model achieved oligodendroglioma prediction accuracy of 92.3% (95% CI, 91.70–92.92%). MR imaging and its derived image of gliomas with CIC mutation appears more complex and non-uniform but are associated with lower malignancy. Our study identified CIC as a potential prognostic factor in glioma which has close associations with survival. MRI radiomic features could predict CIC mutation, and reflect less malignant manifestations such as milder necrosis and larger tumor volume in MRI and its derived images that could help clinical judgment.

In silico analysis reveals EP300 as a panCancer inhibitor of anti-tumor immune response via metabolic modulation

The tumor immune microenvironment (TIME) of head and neck squamous cell carcinomas (HNSCC) and other solid malignancies is a key determinant of therapy response and prognosis. Among other factors, it is shaped by the tumor mutational burden and defects in DNA repair enzymes. Based on the TCGA database we aimed to define specific, altered genes associated with different TIME types, which might represent new predictive markers or targets for immuno-therapeutic approaches. The HNSCC cohort of the TCGA database was used to define 3 TIME types (immune-activated, immune-suppressed, immune-absent) according to expression of immune-related genes. Mutation frequencies were correlated to the 3 TIME types. Overall survival was best in the immune-activated group. 9 genes were significantly differentially mutated in the 3 TIME types with strongest differences for TP53 and the histone-acetyltransferase EP300. Mutations in EP300 correlated with an immune-activated TIME. In panCancer analyses anti-tumor immune activity was increased in EP300 mutated esophageal, stomach and prostate cancers. Downregulation of EP300 gene expression was associated with higher anti-tumor immunity in most solid malignancies. Since EP300 is a promoter of glycolysis, which negatively affects anti-tumor immune response, we analyzed the association of EP300 with tumor metabolism. PanCancer tumor metabolism was strongly shifted towards oxidative phosphorylation in EP300 downregulated tumors. In silico analyses of of publicly available in vitro data showed a decrease of glycolysis-associated genes after treatment with the EP300 inhibitor C646. Our study reveals associations of specific gene alterations with different TIME types. In detail, we defined EP300 as a panCancer inhibitor of the TIME most likely via metabolic modulation. In this context EP300 represents a promising predictive biomarker and an immuno-therapeutic target.

Enhancing the Value of Histopathological Assessment of Allograft Biopsy Monitoring

Traditional histopathological allograft biopsy evaluation provides, within hours, diagnoses, prognostic information, and mechanistic insights into disease processes. However, proponents of an array of alternative monitoring platforms, broadly classified as "invasive" or "noninvasive" depending on whether allograft tissue is needed, question the value proposition of tissue histopathology. The authors explore the pros and cons of current analytical methods relative to the value of traditional and illustrate advancements of next-generation histopathological evaluation of tissue biopsies. We describe the continuing value of traditional histopathological tissue assessment and "next-generation pathology (NGP)," broadly defined as staining/labeling techniques coupled with digital imaging and automated image analysis. Noninvasive imaging and fluid (blood and urine) analyses promote low-risk, global organ assessment, and "molecular" data output, respectively; invasive alternatives promote objective, "mechanistic" insights by creating gene lists with variably increased/decreased expression compared with steady state/baseline. Proponents of alternative approaches contrast their preferred methods with traditional histopathology and: (1) fail to cite the main value of traditional and NGP-retention of spatial and inferred temporal context available for innumerable objective analyses and (2) belie an unfamiliarity with the impact of advances in imaging and software-guided analytics on emerging histopathology practices. Illustrative NGP examples demonstrate the value of multidimensional data that preserve tissue-based spatial and temporal contexts. We outline a path forward for clinical NGP implementation where "software-assisted sign-out" will enable pathologists to conduct objective analyses that can be incorporated into their final reports and improve patient care.

Validation of machine learning models to detect amyloid pathologies across institutions

Semi-quantitative scoring schemes like the Consortium to Establish a Registry for Alzheimer’s Disease (CERAD) are the most commonly used method in Alzheimer’s disease (AD) neuropathology practice. Computational approaches based on machine learning have recently generated quantitative scores for whole slide images (WSIs) that are highly correlated with human derived semi-quantitative scores, such as those of CERAD, for Alzheimer’s disease pathology. However, the robustness of such models have yet to be tested in different cohorts. To validate previously published machine learning algorithms using convolutional neural networks (CNNs) and determine if pathological heterogeneity may alter algorithm derived measures, 40 cases from the Goizueta Emory Alzheimer’s Disease Center brain bank displaying an array of pathological diagnoses (including AD with and without Lewy body disease (LBD), and / or TDP-43-positive inclusions) and levels of Aβ pathologies were evaluated. Furthermore, to provide deeper phenotyping, amyloid burden in gray matter vs whole tissue were compared, and quantitative CNN scores for both correlated significantly to CERAD-like scores. Quantitative scores also show clear stratification based on AD pathologies with or without additional diagnoses (including LBD and TDP-43 inclusions) vs cases with no significant neurodegeneration (control cases) as well as NIA Reagan scoring criteria. Specifically, the concomitant diagnosis group of AD + TDP-43 showed significantly greater CNN-score for cored plaques than the AD group. Finally, we report that whole tissue computational scores correlate better with CERAD-like categories than focusing on computational scores from a field of view with densest pathology, which is the standard of practice in neuropathological assessment per CERAD guidelines. Together these findings validate and expand CNN models to be robust to cohort variations and provide additional proof-of-concept for future studies to incorporate machine learning algorithms into neuropathological practice.

MLCD: A Unified Software Package for Cancer Diagnosis

PURPOSE

Machine Learning Package for Cancer Diagnosis (MLCD) is the result of a National Institutes of Health/National Cancer Institute (NIH/NCI)-sponsored project for developing a unified software package from state-of-the-art breast cancer biopsy diagnosis and machine learning algorithms that can improve the quality of both clinical practice and ongoing research.

METHODS

Whole-slide images of 240 well-characterized breast biopsy cases, initially assembled under R01 CA140560, were used for developing the algorithms and training the machine learning models. This software package is based on the methodology developed and published under our recent NIH/NCI-sponsored research grant (R01 CA172343) for finding regions of interest (ROIs) in whole-slide breast biopsy images, for segmenting ROIs into histopathologic tissue types and for using this segmentation in classifiers that can suggest final diagnoses.

RESULT

The package provides an ROI detector for whole-slide images and modules for semantic segmentation into tissue classes and diagnostic classification into 4 classes (benign, atypia, ductal carcinoma in situ, invasive cancer) of the ROIs. It is available through the GitHub repository under the Massachusetts Institute of Technology license and will later be distributed with the Pathology Image Informatics Platform system. A Web page provides instructions for use.

CONCLUSION

Our tools have the potential to provide help to other cancer researchers and, ultimately, to practicing physicians and will motivate future research in this field. This article describes the methodology behind the software development and gives sample outputs to guide those interested in using this package.

Open access image repositories: high-quality data to enable machine learning research

Originally motivated by the need for research reproducibility and data reuse, large-scale, open access information repositories have become key resources for training and testing of advanced machine learning applications in biomedical and clinical research. To be of value, such repositories must provide large, high-quality data sets, where quality is defined as minimising variance due to data collection protocols and data misrepresentations. Curation is the key to quality. We have constructed a large public access image repository, The Cancer Imaging Archive, dedicated to the promotion of open science to advance the global effort to diagnose and treat cancer. Drawing on this experience and our experience in applying machine learning techniques to the analysis of radiology and pathology image data, we will review the requirements placed on such information repositories by state-of-the-art machine learning applications and how these requirements can be met.

FalseColor-Python: A rapid intensity-leveling and digital-staining package for fluorescence-based slide-free digital pathology

Slide-free digital pathology techniques, including nondestructive 3D microscopy, are gaining interest as alternatives to traditional slide-based histology. In order to facilitate clinical adoption of these fluorescence-based techniques, software methods have been developed to convert grayscale fluorescence images into color images that mimic the appearance of standard absorptive chromogens such as hematoxylin and eosin (H&E). However, these false-coloring algorithms often require manual and iterative adjustment of parameters, with results that can be inconsistent in the presence of intensity nonuniformities within an image and/or between specimens (intra- and inter-specimen variability). Here, we present an open-source (Python-based) rapid intensity-leveling and digital-staining package that is specifically designed to render two-channel fluorescence images (i.e. a fluorescent analog of H&E) to the traditional H&E color space for 2D and 3D microscopy datasets. However, this method can be easily tailored for other false-coloring needs. Our package offers (1) automated and uniform false coloring in spite of uneven staining within a large thick specimen, (2) consistent color-space representations that are robust to variations in staining and imaging conditions between different specimens, and (3) GPU-accelerated data processing to allow these methods to scale to large datasets. We demonstrate this platform by generating H&E-like images from cleared tissues that are fluorescently imaged in 3D with open-top light-sheet (OTLS) microscopy, and quantitatively characterizing the results in comparison to traditional slide-based H&E histology.

Machine-based detection and classification for bone marrow aspirate differential counts: initial development focusing on nonneoplastic cells

Bone marrow aspirate (BMA) differential cell counts (DCCs) are critical for the classification of hematologic disorders. While manual counts are considered the gold standard, they are labor intensive, time consuming, and subject to bias. A reliable automated counter has yet to be developed, largely due to the inherent complexity of bone marrow specimens. Digital pathology imaging coupled with machine learning algorithms represents a highly promising emerging technology for this purpose. Yet, training datasets for BMA cellular constituents, critical for building and validating machine learning algorithms, are lacking. Herein, we report our experience creating and employing such datasets to develop a machine learning algorithm to detect and classify BMA cells. Utilizing a web-based system that we developed for annotating and managing digital pathology images, over 10,000 cells from scanned whole slide images of BMA smears were manually annotated, including all classes that comprise the standard clinical DCC. We implemented a two-stage, detection and classification approach that allows design flexibility and improved classification accuracy. In a sixfold cross-validation, our algorithms achieved high overall accuracy in detection (0.959 ± 0.008 precision-recall AUC) and classification (0.982 ± 0.03 ROC AUC) using nonneoplastic samples. Testing on a small set of acute myeloid leukemia and multiple myeloma samples demonstrated similar detection and classification performance. In summary, our algorithms showed promising early results and represent an important initial step in the effort to devise a reliable, objective method to automate DCCs. With further development to include formal clinical validation, such a system has the potential to assist in disease diagnosis and prognosis, and significantly impact clinical practice.

PathFlowAI: A High-Throughput Workflow for Preprocessing, Deep Learning and Interpretation in Digital Pathology

The diagnosis of disease often requires analysis of a biopsy. Many diagnoses depend not only on the presence of certain features but on their location within the tissue. Recently, a number of deep learning diagnostic aids have been developed to classify digitized biopsy slides. Clinical workflows often involve processing of more than 500 slides per day. But, clinical use of deep learning diagnostic aids would require a preprocessing workflow that is cost-effective, flexible, scalable, rapid, interpretable, and transparent. Here, we present such a workflow, optimized using Dask and mixed precision training via APEX, capable of handling any patch-level or slide level classification and prediction problem. The workflow uses a flexible and fast preprocessing and deep learning analytics pipeline, incorporates model interpretation and has a highly storage-efficient audit trail. We demonstrate the utility of this package on the analysis of a prototypical anatomic pathology specimen, liver biopsies for evaluation of hepatitis from a prospective cohort. The preliminary data indicate that PathFlowAI may become a cost-effective and time-efficient tool for clinical use of Artificial Intelligence (AI) algorithms.

Open Practices and Resources for Collaborative Digital Pathology

In this paper, we describe open practices and open resources in the field of digital pathology with a specific focus on approaches that ease collaboration in research and education settings. Our review includes open access journals and open peer review, open-source software (libraries, desktop tools, and web applications), and open access collections. We illustrate applications and discuss current limitations and perspectives.

Structured crowdsourcing enables convolutional segmentation of histology images

MOTIVATION

While deep-learning algorithms have demonstrated outstanding performance in semantic image segmentation tasks, large annotation datasets are needed to create accurate models. Annotation of histology images is challenging due to the effort and experience required to carefully delineate tissue structures, and difficulties related to sharing and markup of whole-slide images.

RESULTS

We recruited 25 participants, ranging in experience from senior pathologists to medical students, to delineate tissue regions in 151 breast cancer slides using the Digital Slide Archive. Inter-participant discordance was systematically evaluated, revealing low discordance for tumor and stroma, and higher discordance for more subjectively defined or rare tissue classes. Feedback provided by senior participants enabled the generation and curation of 20 000+ annotated tissue regions. Fully convolutional networks trained using these annotations were highly accurate (mean AUC=0.945), and the scale of annotation data provided notable improvements in image classification accuracy.

AVAILABILITY AND IMPLEMENTATION

Dataset is freely available at: https://goo.gl/cNM4EL.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Dual-Personality DICOM-TIFF for Whole Slide Images: A Migration Technique for Legacy Software

Despite recently organized Digital Imaging and Communications in Medicine (DICOM) testing and demonstration events involving numerous participating vendors, it is still the case that scanner manufacturers, software developers, and users continue to depend on proprietary file formats rather than adopting the standard DICOM whole slide microscopic image object. Many proprietary formats are Tagged Image File Format (TIFF) based, and existing applications and libraries can read tiled TIFF files. The sluggish adoption of DICOM for whole slide image encoding can be temporarily mitigated by the use of dual-personality DICOM-TIFF files. These are compatible with the installed base of TIFF-based software, as well as newer DICOM-based software. The DICOM file format was deliberately designed to support this dual-personality capability for such transitional situations, although it is rarely used. Furthermore, existing TIFF files can be converted into dual-personality DICOM-TIFF without changing the pixel data. This paper demonstrates the feasibility of extending the dual-personality concept to multiframe-tiled pyramidal whole slide images and explores the issues encountered. Open source code and sample converted images are provided for testing.

Clinical genome sequencing uncovers potentially targetable truncations and fusions of MAP3K8 in spitzoid and other melanomas

Spitzoid melanoma is a specific morphologic variant of melanoma that most commonly affects children and adolescents, and ranges on the spectrum of malignancy from low grade to overtly malignant. These tumors are generally driven by fusions of ALK, RET, NTRK1/3, MET, ROS1 and BRAF^1,2. However, in approximately 50% of cases no genetic driver has been established². Clinical whole-genome and transcriptome sequencing (RNA-Seq) of a spitzoid tumor from an adolescent revealed a novel gene fusion of MAP3K8, encoding a serine-threonine kinase that activates MEK^3,4. The patient, who had exhausted all other therapeutic options, was treated with a MEK inhibitor and underwent a transient clinical response. We subsequently analyzed spitzoid tumors from 49 patients by RNA-Seq and found in-frame fusions or C-terminal truncations of MAP3K8 in 33% of cases. The fusion transcripts and truncated genes all contained MAP3K8 exons 1-8 but lacked the autoinhibitory final exon. Data mining of RNA-Seq from the Cancer Genome Atlas (TCGA) uncovered analogous MAP3K8 rearrangements in 1.5% of adult melanomas. Thus, MAP3K8 rearrangements-uncovered by comprehensive clinical sequencing of a single case-are the most common genetic event in spitzoid melanoma, are present in adult melanomas and could be amenable to MEK inhibition.

Recent advances in functional genome analysis

At the beginning of this century, the Human Genome Project produced the first drafts of the human genome sequence. Following this, large-scale functional genomics studies were initiated to understand the molecular basis underlying the translation of the instructions encoded in the genome into the biological traits of organisms. Instrumental in the ensuing revolution in functional genomics were the rapid advances in massively parallel sequencing technologies as well as the development of a wide diversity of protocols that make use of these technologies to understand cellular behavior at the molecular level. Here, we review recent advances in functional genomic methods, discuss some of their current capabilities and limitations, and briefly sketch future directions within the field.

Risk of Subsequent Cutaneous Melanoma in Moderately Dysplastic Nevi Excisionally Biopsied but With Positive Histologic Margins

Importance

Little evidence exists to guide the management of moderately dysplastic nevi excisionally biopsied without residual clinical pigmentation but with positive histologic margins (hereafter referred to as moderately dysplastic nevi with positive histologic margins).

Objective

To determine outcomes and risk for the development of subsequent cutaneous melanoma (CM) from moderately dysplastic nevi with positive histologic margins observed for 3 years or more.

Design, Setting, and Participants

A multicenter (9 US academic dermatology sites) retrospective cohort study was conducted of patients 18 years or older with moderately dysplastic nevi with positive histologic margins and 3 years or more of follow-up data collected consecutively from January 1, 1990, to August 31, 2014. Records were reviewed for patient demographics, biopsy type, pathologic findings, and development of subsequent CM at the biopsy site or elsewhere on the body. The χ2 test, the Fisher exact test, and analysis of variance were used to assess univariate association for risk of subsequent CMs, in addition to multivariable logistic regression models. To confirm histologic grading, each site submitted 5 random representative slide cases for central dermatopathologic review. Statistical analysis was performed from October 1, 2017, to June 22, 2018.

Main Outcomes and Measures

Development of CM at a biopsy site or elsewhere on the body where there were moderately dysplastic nevi with positive histologic margins.

Results

A total of 467 moderately dysplastic nevi with positive histologic margins from 438 patients (193 women and 245 men; mean [SD] age, 46.7 [16.1] years) were evaluated. No cases developed into CM at biopsy sites, with a mean (SD) follow-up time of 6.9 (3.4) years. However, 100 patients (22.8%) developed a CM at a separate site. Results of multivariate analyses revealed that history of CM was significantly associated with the risk of development of subsequent CM at a separate site (odds ratio, 11.74; 95% CI, 5.71-24.15; P < .001), as were prior biopsied dysplastic nevi (odds ratio, 2.55; 95% CI, 1.23-5.28; P = .01). The results of a central dermatopathologic review revealed agreement in 35 of 40 cases (87.5%). Three of 40 cases (7.5%) were upgraded in degree of atypia; of these, 1 was interpreted as melanoma in situ. That patient remains without recurrence or evidence of CM after 5 years of follow-up.

Conclusions and Relevance

This study suggests that close observation with routine skin surveillance is a reasonable management approach for moderately dysplastic nevi with positive histologic margins. However, having 2 or more biopsied dysplastic nevi (with 1 that is a moderately dysplastic nevus) appears to be associated with increased risk for subsequent CM at a separate site.

Twenty Years of Digital Pathology: An Overview of the Road Travelled, What is on the Horizon, and the Emergence of Vendor-Neutral Archives

Almost 20 years have passed since the commercial introduction of whole-slide imaging (WSI) scanners. During this time, the creation of various WSI devices with the ability to digitize an entire glass slide has transformed the field of pathology. Parallel advances in computational technology and storage have permitted rapid processing of large-scale WSI datasets. This article provides an overview of important past and present efforts related to WSI. An account of how the virtual microscope evolved from the need to visualize and manage satellite data for earth science applications is provided. The article also discusses important milestones beginning from the first WSI scanner designed by Bacus to the Food and Drug Administration approval of the first digital pathology system for primary diagnosis in surgical pathology. As pathology laboratories commit to going fully digitalize, the need has emerged to include WSIs into an enterprise-level vendor-neutral archive (VNA). The different types of VNAs available are reviewed as well as how best to implement them and how pathology can benefit from participating in this effort. Differences between traditional image algorithms that extract pixel-, object-, and semantic-level features versus deep learning methods are highlighted. The need for large-scale data management, analysis, and visualization in computational pathology is also addressed.

Calibration Software for Quantitative PET/CT Imaging Using Pocket Phantoms

Multicenter clinical trials that use positron emission tomography (PET) imaging frequently rely on stable bias in imaging biomarkers to assess drug effectiveness. Many well-documented factors cause variability in PET intensity values. Two of the largest scanner-dependent errors are scanner calibration and reconstructed image resolution variations. For clinical trials, an increase in measurement error significantly increases the number of patient scans needed. We aim to provide a robust quality assurance system using portable PET/computed tomography “pocket” phantoms and automated image analysis algorithms with the goal of reducing PET measurement variability. A set of the “pocket” phantoms was scanned with patients, affixed to the underside of a patient bed. Our software analyzed the obtained images and estimated the image parameters. The analysis consisted of 2 steps, automated phantom detection and estimation of PET image resolution and global bias. Performance of the algorithm was tested under variations in image bias, resolution, noise, and errors in the expected sphere size. A web-based application was implemented to deploy the image analysis pipeline in a cloud-based infrastructure to support multicenter data acquisition, under Software-as-a-Service (SaaS) model. The automated detection algorithm localized the phantom reliably. Simulation results showed stable behavior when image properties and input parameters were varied. The PET “pocket” phantom has the potential to reduce and/or check for standardized uptake value measurement errors.

Diagnostic accuracy of whole slide imaging for cutaneous, soft tissue, and melanoma sentinel lymph node biopsies with and without immunohistochemistry

BACKGROUND

Diagnostic accuracy with whole slide imaging (WSI) for complex inpatient and outpatient dermatopathology cases with immunohistochemistry (IHC) is unknown.

METHODS

WSI (Leica Aperio AT2 Digital Pathology scanner, N = 151 cases) was performed for Emory inpatient and outpatient skin (N = 105), soft tissue (N = 30), and melanoma sentinel lymph node biopsies (N = 16) collected between 2000 and 2016. Resultant images were uploaded to an online cloud storage system for review by 2 board-certified dermatopathologists (reviewers 1 and 2) with greater than 5 years of dermatopathology experience and 1 dermatopathology fellow (reviewer 3).

RESULTS

Reviewers 1 (diagnostic accuracy = 97%) and 2 (diagnostic accuracy = 95%) demonstrated high diagnostic accuracy with WSI. Diagnostic accuracy was greater than 90% for inpatient biopsies, melanocytic lesions, melanoma sentinel lymph node biopsies, and cases with immunohistochemistry, but was slightly lower for soft tissue cases (reviewer 1 = 89%; reviewer 2 = 89%). The dermatopathology fellow (reviewer 3) demonstrated lower diagnostic accuracy (84%).

CONCLUSIONS

Diagnostic accuracy with WSI for skin, soft tissue, and melanoma sentinel lymph node biopsies with and without immunohistochemistry was greater than 95% for 2 reviewers with greater than 5 years of dermatopathology experience. Professional experience signing out dermatopathology cases may impact diagnostic accuracy with WSI.

Predicting cancer outcomes from histology and genomics using convolutional networks

Predicting the expected outcome of patients diagnosed with cancer is a critical step in treatment. Advances in genomic and imaging technologies provide physicians with vast amounts of data, yet prognostication remains largely subjective, leading to suboptimal clinical management. We developed a computational approach based on deep learning to predict the overall survival of patients diagnosed with brain tumors from microscopic images of tissue biopsies and genomic biomarkers. This method uses adaptive feedback to simultaneously learn the visual patterns and molecular biomarkers associated with patient outcomes. Our approach surpasses the prognostic accuracy of human experts using the current clinical standard for classifying brain tumors and presents an innovative approach for objective, accurate, and integrated prediction of patient outcomes.

Cancer histology reflects underlying molecular processes and disease progression and contains rich phenotypic information that is predictive of patient outcomes. In this study, we show a computational approach for learning patient outcomes from digital pathology images using deep learning to combine the power of adaptive machine learning algorithms with traditional survival models. We illustrate how these survival convolutional neural networks (SCNNs) can integrate information from both histology images and genomic biomarkers into a single unified framework to predict time-to-event outcomes and show prediction accuracy that surpasses the current clinical paradigm for predicting the overall survival of patients diagnosed with glioma. We use statistical sampling techniques to address challenges in learning survival from histology images, including tumor heterogeneity and the need for large training cohorts. We also provide insights into the prediction mechanisms of SCNNs, using heat map visualization to show that SCNNs recognize important structures, like microvascular proliferation, that are related to prognosis and that are used by pathologists in grading. These results highlight the emerging role of deep learning in precision medicine and suggest an expanding utility for computational analysis of histology in the future practice of pathology.