Assessment of tumour viability in human lung cancer xenografts with texture-based image analysis

Outcome and Biomarker Supervised Deep Learning for Survival Prediction in Two Multicenter Breast Cancer Series

BACKGROUND

Prediction of clinical outcomes for individual cancer patients is an important step in the disease diagnosis and subsequently guides the treatment and patient counseling. In this work, we develop and evaluate a joint outcome and biomarker supervised (estrogen receptor expression and ERBB2 expression and gene amplification) multitask deep learning model for prediction of outcome in breast cancer patients in two nation-wide multicenter studies in Finland (the FinProg and FinHer studies). Our approach combines deep learning with expert knowledge to provide more accurate, robust, and integrated prediction of breast cancer outcomes.

MATERIALS AND METHODS

Using deep learning, we trained convolutional neural networks (CNNs) with digitized tissue microarray (TMA) samples of primary hematoxylin-eosin-stained breast cancer specimens from 693 patients in the FinProg series as input and breast cancer-specific survival as the endpoint. The trained algorithms were tested on 354 TMA patient samples in the same series. An independent set of whole-slide (WS) tumor samples from 674 patients in another multicenter study (FinHer) was used to validate and verify the generalization of the outcome prediction based on CNN models by Cox survival regression and concordance index (c-index). Visual cancer tissue characterization, i.e., number of mitoses, tubules, nuclear pleomorphism, tumor-infiltrating lymphocytes, and necrosis was performed on TMA samples in the FinProg test set by a pathologist and combined with deep learning-based outcome prediction in a multitask algorithm.

RESULTS

The multitask algorithm achieved a hazard ratio (HR) of 2.0 (95% confidence interval [CI] 1.30-3.00), P < 0.001, c-index of 0.59 on the 354 test set of FinProg patients, and an HR of 1.7 (95% CI 1.2-2.6), P = 0.003, c-index 0.57 on the WS tumor samples from 674 patients in the independent FinHer series. The multitask CNN remained a statistically independent predictor of survival in both test sets when adjusted for histological grade, tumor size, and axillary lymph node status in a multivariate Cox analyses. An improved accuracy (c-index 0.66) was achieved when deep learning was combined with the tissue characteristics assessed visually by a pathologist.

CONCLUSIONS

A multitask deep learning algorithm supervised by both patient outcome and biomarker status learned features in basic tissue morphology predictive of survival in a nationwide, multicenter series of patients with breast cancer. The algorithms generalized to another independent multicenter patient series and whole-slide breast cancer samples and provide prognostic information complementary to that of a comprehensive series of established prognostic factors.

Breast cancer outcome prediction with tumour tissue images and machine learning

PURPOSE

Recent advances in machine learning have enabled better understanding of large and complex visual data. Here, we aim to investigate patient outcome prediction with a machine learning method using only an image of tumour sample as an input.

METHODS

Utilising tissue microarray (TMA) samples obtained from the primary tumour of patients (N = 1299) within a nationwide breast cancer series with long-term-follow-up, we train and validate a machine learning method for patient outcome prediction. The prediction is performed by classifying samples into low or high digital risk score (DRS) groups. The outcome classifier is trained using sample images of 868 patients and evaluated and compared with human expert classification in a test set of 431 patients.

RESULTS

In univariate survival analysis, the DRS classification resulted in a hazard ratio of 2.10 (95% CI 1.33-3.32, p = 0.001) for breast cancer-specific survival. The DRS classification remained as an independent predictor of breast cancer-specific survival in a multivariate Cox model with a hazard ratio of 2.04 (95% CI 1.20-3.44, p = 0.007). The accuracy (C-index) of the DRS grouping was 0.60 (95% CI 0.55-0.65), as compared to 0.58 (95% CI 0.53-0.63) for human expert predictions based on the same TMA samples.

CONCLUSIONS

Our findings demonstrate the feasibility of learning prognostic signals in tumour tissue images without domain knowledge. Although further validation is needed, our study suggests that machine learning algorithms can extract prognostically relevant information from tumour histology complementing the currently used prognostic factors in breast cancer.

Clustering approach to identify intratumour heterogeneity combining FDG PET and diffusion-weighted MRI in lung adenocarcinoma

OBJECTIVES

Malignant tumours consist of biologically heterogeneous components; identifying and stratifying those various subregions is an important research topic. We aimed to show the effectiveness of an intratumour partitioning method using clustering to identify highly aggressive tumour subregions, determining prognosis based on pre-treatment PET and DWI in stage IV lung adenocarcinoma.

METHODS

Eighteen patients who underwent both baseline PET and DWI were recruited. Pre-treatment imaging of SUV and ADC values were used to form intensity vectors within manually specified ROIs. We applied k-means clustering to intensity vectors to yield distinct subregions, then chose the subregion that best matched the criteria for high SUV and low ADC to identify tumour subregions with high aggressiveness. We stratified patients into high- and low-risk groups based on subregion volume with high aggressiveness and conducted survival analyses. This approach is referred to as the partitioning approach. For comparison, we computed tumour subregions with high aggressiveness without clustering and repeated the described procedure; this is referred to as the voxel-wise approach.

RESULTS

The partitioning approach led to high-risk (median SUVmax = 14.25 and median ADC = 1.26x10-3 mm2/s) and low-risk (median SUVmax = 14.64 and median ADC = 1.09x10-3 mm2/s) subgroups. Our partitioning approach identified significant differences in survival between high- and low-risk subgroups (hazard ratio, 4.062, 95% confidence interval, 1.21 - 13.58, p-value: 0.035). The voxel-wise approach did not identify significant differences in survival between high- and low-risk subgroups (p-value: 0.325).

CONCLUSION

Our partitioning approach identified intratumour subregions that were predictors of survival.

KEY POINTS

• Multimodal imaging of PET and DWI is useful for assessing intratumour heterogeneity. • Data-driven clustering identified subregions which might be highly aggressive for lung adenocarcinoma. • The data-driven partitioning results might be predictors of survival.

Antibody-supervised deep learning for quantification of tumor-infiltrating immune cells in hematoxylin and eosin stained breast cancer samples

BACKGROUND

Immune cell infiltration in tumor is an emerging prognostic biomarker in breast cancer. The gold standard for quantification of immune cells in tissue sections is visual assessment through a microscope, which is subjective and semi-quantitative. In this study, we propose and evaluate an approach based on antibody-guided annotation and deep learning to quantify immune cell-rich areas in hematoxylin and eosin (H&E) stained samples.

METHODS

Consecutive sections of formalin-fixed parafin-embedded samples obtained from the primary tumor of twenty breast cancer patients were cut and stained with H&E and the pan-leukocyte CD45 antibody. The stained slides were digitally scanned, and a training set of immune cell-rich and cell-poor tissue regions was annotated in H&E whole-slide images using the CD45-expression as a guide. In analysis, the images were divided into small homogenous regions, superpixels, from which features were extracted using a pretrained convolutional neural network (CNN) and classified with a support of vector machine. The CNN approach was compared to texture-based classification and to visual assessments performed by two pathologists.

RESULTS

In a set of 123,442 labeled superpixels, the CNN approach achieved an F-score of 0.94 (range: 0.92-0.94) in discrimination of immune cell-rich and cell-poor regions, as compared to an F-score of 0.88 (range: 0.87-0.89) obtained with the texture-based classification. When compared to visual assessment of 200 images, an agreement of 90% (κ = 0.79) to quantify immune infiltration with the CNN approach was achieved while the inter-observer agreement between pathologists was 90% (κ = 0.78).

CONCLUSIONS

Our findings indicate that deep learning can be applied to quantify immune cell infiltration in breast cancer samples using a basic morphology staining only. A good discrimination of immune cell-rich areas was achieved, well in concordance with both leukocyte antigen expression and pathologists' visual assessment.

Accurate Morphology Preserving Segmentation of Overlapping Cells based on Active Contours

The identification of fluorescently stained cell nuclei is the basis of cell detection, segmentation, and feature extraction in high content microscopy experiments. The nuclear morphology of single cells is also one of the essential indicators of phenotypic variation. However, the cells used in experiments can lose their contact inhibition, and can therefore pile up on top of each other, making the detection of single cells extremely challenging using current segmentation methods. The model we present here can detect cell nuclei and their morphology even in high-confluency cell cultures with many overlapping cell nuclei. We combine the "gas of near circles" active contour model, which favors circular shapes but allows slight variations around them, with a new data model. This captures a common property of many microscopic imaging techniques: the intensities from superposed nuclei are additive, so that two overlapping nuclei, for example, have a total intensity that is approximately double the intensity of a single nucleus. We demonstrate the power of our method on microscopic images of cells, comparing the results with those obtained from a widely used approach, and with manual image segmentations by experts.

Multi-class texture analysis in colorectal cancer histology

Automatic recognition of different tissue types in histological images is an essential part in the digital pathology toolbox. Texture analysis is commonly used to address this problem; mainly in the context of estimating the tumour/stroma ratio on histological samples. However, although histological images typically contain more than two tissue types, only few studies have addressed the multi-class problem. For colorectal cancer, one of the most prevalent tumour types, there are in fact no published results on multiclass texture separation. In this paper we present a new dataset of 5,000 histological images of human colorectal cancer including eight different types of tissue. We used this set to assess the classification performance of a wide range of texture descriptors and classifiers. As a result, we found an optimal classification strategy that markedly outperformed traditional methods, improving the state of the art for tumour-stroma separation from 96.9% to 98.6% accuracy and setting a new standard for multiclass tissue separation (87.4% accuracy for eight classes). We make our dataset of histological images publicly available under a Creative Commons license and encourage other researchers to use it as a benchmark for their studies.