1
|
Balkenhol MC, Ciompi F, Świderska-Chadaj Ż, van de Loo R, Intezar M, Otte-Höller I, Geijs D, Lotz J, Weiss N, de Bel T, Litjens G, Bult P, van der Laak JA. Optimized tumour infiltrating lymphocyte assessment for triple negative breast cancer prognostics. Breast 2021; 56:78-87. [PMID: 33640523 PMCID: PMC7933536 DOI: 10.1016/j.breast.2021.02.007] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Revised: 02/05/2021] [Accepted: 02/08/2021] [Indexed: 12/29/2022] Open
Abstract
The tumour microenvironment has been shown to be a valuable source of prognostic information for different cancer types. This holds in particular for triple negative breast cancer (TNBC), a breast cancer subtype for which currently no prognostic biomarkers are established. Although different methods to assess tumour infiltrating lymphocytes (TILs) have been published, it remains unclear which method (marker, region) yields the most optimal prognostic information. In addition, to date, no objective TILs assessment methods are available. For this proof of concept study, a subset of our previously described TNBC cohort (n = 94) was stained for CD3, CD8 and FOXP3 using multiplex immunohistochemistry and subsequently imaged by a multispectral imaging system. Advanced whole-slide image analysis algorithms, including convolutional neural networks (CNN) were used to register unmixed multispectral images and corresponding H&E sections, to segment the different tissue compartments (tumour, stroma) and to detect all individual positive lymphocytes. Densities of positive lymphocytes were analysed in different regions within the tumour and its neighbouring environment and correlated to relapse free survival (RFS) and overall survival (OS). We found that for all TILs markers the presence of a high density of positive cells correlated with an improved survival. None of the TILs markers was superior to the others. The results of TILs assessment in the various regions did not show marked differences between each other. The negative correlation between TILs and survival in our cohort are in line with previous studies. Our results provide directions for optimizing TILs assessment methodology.
Collapse
Affiliation(s)
- Maschenka Ca Balkenhol
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands.
| | - Francesco Ciompi
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Żaneta Świderska-Chadaj
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands; Warsaw University of Technology, Faculty of Electrical Engineering, Warsaw, Poland
| | - Rob van de Loo
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Milad Intezar
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Irene Otte-Höller
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Daan Geijs
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Johannes Lotz
- Fraunhofer Institute for Image Computing MEVIS, Lübeck, Germany
| | - Nick Weiss
- Fraunhofer Institute for Image Computing MEVIS, Lübeck, Germany
| | - Thomas de Bel
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Geert Litjens
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Peter Bult
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands
| | - Jeroen Awm van der Laak
- Radboud University Medical Center, Radboud Institute for Health Sciences, Department of Pathology, Nijmegen, the Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| |
Collapse
|
2
|
Bulten W, Bándi P, Hoven J, Loo RVD, Lotz J, Weiss N, Laak JVD, Ginneken BV, Hulsbergen-van de Kaa C, Litjens G. Epithelium segmentation using deep learning in H&E-stained prostate specimens with immunohistochemistry as reference standard. Sci Rep 2019; 9:864. [PMID: 30696866 PMCID: PMC6351532 DOI: 10.1038/s41598-018-37257-4] [Citation(s) in RCA: 75] [Impact Index Per Article: 15.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2018] [Accepted: 12/03/2018] [Indexed: 12/24/2022] Open
Abstract
Given the importance of gland morphology in grading prostate cancer (PCa), automatically differentiating between epithelium and other tissues is an important prerequisite for the development of automated methods for detecting PCa. We propose a new deep learning method to segment epithelial tissue in digitised hematoxylin and eosin (H&E) stained prostatectomy slides using immunohistochemistry (IHC) as reference standard. We used IHC to create a precise and objective ground truth compared to manual outlining on H&E slides, especially in areas with high-grade PCa. 102 tissue sections were stained with H&E and subsequently restained with P63 and CK8/18 IHC markers to highlight epithelial structures. Afterwards each pair was co-registered. First, we trained a U-Net to segment epithelial structures in IHC using a subset of the IHC slides that were preprocessed with color deconvolution. Second, this network was applied to the remaining slides to create the reference standard used to train a second U-Net on H&E. Our system accurately segmented both intact glands and individual tumour epithelial cells. The generalisation capacity of our system is shown using an independent external dataset from a different centre. We envision this segmentation as the first part of a fully automated prostate cancer grading pipeline.
Collapse
Affiliation(s)
- Wouter Bulten
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands.
| | - Péter Bándi
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Jeffrey Hoven
- Radboud University Medical Center, Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Rob van de Loo
- Radboud University Medical Center, Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | | | - Nick Weiss
- Fraunhofer MEVIS, 23562, Lübeck, Germany
| | - Jeroen van der Laak
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Radiology and Nuclear Medicine, 6500HB, Nijmegen, The Netherlands
| | | | - Geert Litjens
- Radboud University Medical Center, Diagnostic Image Analysis Group and the Department of Pathology, 6500HB, Nijmegen, The Netherlands
| |
Collapse
|
3
|
Litjens G, Bandi P, Ehteshami Bejnordi B, Geessink O, Balkenhol M, Bult P, Halilovic A, Hermsen M, van de Loo R, Vogels R, Manson QF, Stathonikos N, Baidoshvili A, van Diest P, Wauters C, van Dijk M, van der Laak J. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. Gigascience 2018; 7:5026175. [PMID: 29860392 PMCID: PMC6007545 DOI: 10.1093/gigascience/giy065] [Citation(s) in RCA: 130] [Impact Index Per Article: 21.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2017] [Accepted: 05/22/2018] [Indexed: 12/27/2022] Open
Abstract
Background The presence of lymph node metastases is one of the most important factors in breast cancer prognosis. The most common way to assess regional lymph node status is the sentinel lymph node procedure. The sentinel lymph node is the most likely lymph node to contain metastasized cancer cells and is excised, histopathologically processed, and examined by a pathologist. This tedious examination process is time-consuming and can lead to small metastases being missed. However, recent advances in whole-slide imaging and machine learning have opened an avenue for analysis of digitized lymph node sections with computer algorithms. For example, convolutional neural networks, a type of machine-learning algorithm, can be used to automatically detect cancer metastases in lymph nodes with high accuracy. To train machine-learning models, large, well-curated datasets are needed. Results We released a dataset of 1,399 annotated whole-slide images (WSIs) of lymph nodes, both with and without metastases, in 3 terabytes of data in the context of the CAMELYON16 and CAMELYON17 Grand Challenges. Slides were collected from five medical centers to cover a broad range of image appearance and staining variations. Each WSI has a slide-level label indicating whether it contains no metastases, macro-metastases, micro-metastases, or isolated tumor cells. Furthermore, for 209 WSIs, detailed hand-drawn contours for all metastases are provided. Last, open-source software tools to visualize and interact with the data have been made available. Conclusions A unique dataset of annotated, whole-slide digital histopathology images has been provided with high potential for re-use.
Collapse
Affiliation(s)
- Geert Litjens
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Peter Bandi
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Babak Ehteshami Bejnordi
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Oscar Geessink
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Maschenka Balkenhol
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Peter Bult
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Altuna Halilovic
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Meyke Hermsen
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Rob van de Loo
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Rob Vogels
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| | - Quirine F Manson
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Nikolas Stathonikos
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Alexi Baidoshvili
- Laboratory for Pathology East Netherlands (LabPON), Postbus 516, 7550AM Hengelo, The Netherlands
| | - Paul van Diest
- Department of Pathology, University Medical Center Huispost H04.312, Heidelberglaan 100, 3584CX, Utrecht, The Netherlands
| | - Carla Wauters
- Department of Pathology, Canisius-Wilhelmina Hospital, Postbus 9015, 6500GS Nijmegen, The Netherlands
| | - Marcory van Dijk
- Department of Pathology, Rijnstate Hospital, Pathology-DNA, Postbus 9555, 6800TA Arnhem, The Netherlands
| | - Jeroen van der Laak
- Diagnostic Image Analysis Group, Department of Pathology, Radboud University Medical Center, Huispost 824, Geert Grootteplein-Zuid 10, 6525GA Nijmegen, The Netherlands
| |
Collapse
|
4
|
Tellez D, Balkenhol M, Otte-Holler I, van de Loo R, Vogels R, Bult P, Wauters C, Vreuls W, Mol S, Karssemeijer N, Litjens G, van der Laak J, Ciompi F. Whole-Slide Mitosis Detection in H&E Breast Histology Using PHH3 as a Reference to Train Distilled Stain-Invariant Convolutional Networks. IEEE Trans Med Imaging 2018; 37:2126-2136. [PMID: 29994086 DOI: 10.1109/tmi.2018.2820199] [Citation(s) in RCA: 117] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Manual counting of mitotic tumor cells in tissue sections constitutes one of the strongest prognostic markers for breast cancer. This procedure, however, is time-consuming and error-prone. We developed a method to automatically detect mitotic figures in breast cancer tissue sections based on convolutional neural networks (CNNs). Application of CNNs to hematoxylin and eosin (H&E) stained histological tissue sections is hampered by noisy and expensive reference standards established by pathologists, lack of generalization due to staining variation across laboratories, and high computational requirements needed to process gigapixel whole-slide images (WSIs). In this paper, we present a method to train and evaluate CNNs to specifically solve these issues in the context of mitosis detection in breast cancer WSIs. First, by combining image analysis of mitotic activity in phosphohistone-H3 restained slides and registration, we built a reference standard for mitosis detection in entire H&E WSIs requiring minimal manual annotation effort. Second, we designed a data augmentation strategy that creates diverse and realistic H&E stain variations by modifying H&E color channels directly. Using it during training combined with network ensembling resulted in a stain invariant mitosis detector. Third, we applied knowledge distillation to reduce the computational requirements of the mitosis detection ensemble with a negligible loss of performance. The system was trained in a single-center cohort and evaluated in an independent multicenter cohort from the cancer genome atlas on the three tasks of the tumor proliferation assessment challenge. We obtained a performance within the top three best methods for most of the tasks of the challenge.
Collapse
|