1
|
Cserni B, Kilmartin D, O’Loughlin M, Andreu X, Bagó-Horváth Z, Bianchi S, Chmielik E, Figueiredo P, Floris G, Foschini MP, Kovács A, Heikkilä P, Kulka J, Laenkholm AV, Liepniece-Karele I, Marchiò C, Provenzano E, Regitnig P, Reiner A, Ryška A, Sapino A, Stovgaard ES, Quinn C, Zolota V, Webber M, Glynn SA, Bori R, Csörgő E, Oláh-Németh O, Pancsa T, Sejben A, Sejben I, Vörös A, Zombori T, Nyári T, Callagy G, Cserni G. ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations. Cancers (Basel) 2023; 15:cancers15041199. [PMID: 36831541 PMCID: PMC9954449 DOI: 10.3390/cancers15041199] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 02/04/2023] [Accepted: 02/09/2023] [Indexed: 02/16/2023] Open
Abstract
Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2-11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.
Collapse
Affiliation(s)
- Bálint Cserni
- TNG Technology Consulting GmbH, Király u. 26., 1061 Budapest, Hungary
| | - Darren Kilmartin
- Discipline of Pathology, Lambe Institute for Translational Research, School of Medicine, University of Galway, H91 TK33 Galway, Ireland
| | - Mark O’Loughlin
- Discipline of Pathology, Lambe Institute for Translational Research, School of Medicine, University of Galway, H91 TK33 Galway, Ireland
| | - Xavier Andreu
- Pathology Department, Atryshealth Co., Ltd., 08039 Barcelona, Spain
| | - Zsuzsanna Bagó-Horváth
- Department of Pathology, Medical University of Vienna, Währinger Gürtel 18-20, 1090 Vienna, Austria
| | - Simonetta Bianchi
- Division of Pathological Anatomy, Department of Health Sciences, University of Florence, 50134 Florence, Italy
| | - Ewa Chmielik
- Tumor Pathology Department, Maria Sklodowska-Curie National Research Institute of Oncology, Gliwice Branch, 44-102 Gliwice, Poland
| | - Paulo Figueiredo
- Laboratório de Anatomia Patológica, IPO Coimbra, 3000-075 Coimbra, Portugal
| | - Giuseppe Floris
- Laboratory of Translational Cell & Tissue Research and KU Leuven, Department of Imaging and Pathology, Department of Pathology, University Hospitals Leuven, University of Leuven, Oude Market 13, 3000 Leuven, Belgium
| | - Maria Pia Foschini
- Unit of Anatomic Pathology, Department of Biomedical and Neuromotor Sciences, University of Bologna, Bellaria Hospital, 40139 Bologna, Italy
| | - Anikó Kovács
- Department of Clinical Pathology, Sahlgrenska University Hospital, 41345 Gothenburg, Sweden
| | - Päivi Heikkilä
- Department of Pathology, Helsinki University Central Hospital, 00029 Helsinki, Finland
| | - Janina Kulka
- Department of Pathology, Forensic and Insurance Medicine, Semmelweis University Budapest, Üllői út 93, 1091 Budapest, Hungary
| | - Anne-Vibeke Laenkholm
- Department of Surgical Pathology, Zealand University Hospital, 4000 Roskilde, Denmark
| | - Inta Liepniece-Karele
- Department of Pathology, Riga Stradins University, Riga East Clinical University Hospital, LV-1038 Riga, Latvia
| | - Caterina Marchiò
- Unit of Pathology, Candiolo Cancer Institute FPO-IRCCS, 10060 Candiolo, Italy
- Department of Medical Sciences, University of Turin, 10126 Turin, Italy
| | - Elena Provenzano
- Department of Histopathology, Cambridge University Hospitals National Health Service (NHS) Foundation Trust, Cambridge CB2 0QQ, UK
- National Institute for Health Research Cambridge Biomedical Research Centre, Cambridge CB2 0QQ, UK
| | - Peter Regitnig
- Diagnostic and Research Institute of Pathology, Medical University of Graz, 8010 Graz, Austria
| | - Angelika Reiner
- Department of Pathology, Klinikum Donaustadt, 1090 Vienna, Austria
| | - Aleš Ryška
- The Fingerland Department of Pathology, Charles University Medical Faculty and University Hospital, 50003 Hradec Kralove, Czech Republic
| | - Anna Sapino
- Unit of Pathology, Candiolo Cancer Institute FPO-IRCCS, 10060 Candiolo, Italy
- Department of Medical Sciences, University of Turin, 10126 Turin, Italy
| | | | - Cecily Quinn
- Department of Histopathology, Irish National Breast Screening Programme, BreastCheck, St. Vincent’s University Hospital and School of Medicine, University College Dublin, D04 T6F4 Dublin, Ireland
- School of Medicine, University College Dublin, D04 V1W8 Dublin, Ireland
| | - Vasiliki Zolota
- Department of Pathology, School of Medicine, University of Patras, 26504 Rion, Greece
| | - Mark Webber
- Discipline of Pathology, Lambe Institute for Translational Research, School of Medicine, University of Galway, H91 TK33 Galway, Ireland
| | - Sharon A. Glynn
- Discipline of Pathology, Lambe Institute for Translational Research, School of Medicine, University of Galway, H91 TK33 Galway, Ireland
| | - Rita Bori
- Department of Pathology, Bács-Kiskun County Teaching Hospital, 6000 Kecskemét, Hungary
| | - Erika Csörgő
- Department of Pathology, Bács-Kiskun County Teaching Hospital, 6000 Kecskemét, Hungary
| | | | - Tamás Pancsa
- Department of Pathology, University of Szeged, 6720 Szeged, Hungary
| | - Anita Sejben
- Department of Pathology, University of Szeged, 6720 Szeged, Hungary
| | - István Sejben
- Department of Pathology, Bács-Kiskun County Teaching Hospital, 6000 Kecskemét, Hungary
| | - András Vörös
- Department of Pathology, University of Szeged, 6720 Szeged, Hungary
| | - Tamás Zombori
- Department of Pathology, University of Szeged, 6720 Szeged, Hungary
| | - Tibor Nyári
- Department of Medical Physics and Informatics, University of Szeged, 6720 Szeged, Hungary
| | - Grace Callagy
- Discipline of Pathology, Lambe Institute for Translational Research, School of Medicine, University of Galway, H91 TK33 Galway, Ireland
| | - Gábor Cserni
- Department of Pathology, Bács-Kiskun County Teaching Hospital, 6000 Kecskemét, Hungary
- Department of Pathology, University of Szeged, 6720 Szeged, Hungary
- Correspondence:
| |
Collapse
|
2
|
Cserni B, Bori R, Csörgő E, Oláh-Németh O, Pancsa T, Sejben A, Sejben I, Vörös A, Zombori T, Nyári T, Cserni G. The additional value of ONEST (Observers Needed to Evaluate Subjective Tests) in assessing reproducibility of oestrogen receptor, progesterone receptor, and Ki67 classification in breast cancer. Virchows Arch 2021; 479:1101-1109. [PMID: 34415429 PMCID: PMC8724065 DOI: 10.1007/s00428-021-03172-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2021] [Revised: 07/06/2021] [Accepted: 07/25/2021] [Indexed: 11/28/2022]
Abstract
The reproducibility of assessing potential biomarkers is crucial for their implementation. ONEST (Observers Needed to Evaluate Subjective Tests) has been recently introduced as a new additive evaluation method for the assessment of reliability, by demonstrating how the number of observers impact on interobserver agreement. Oestrogen receptor (ER), progesterone receptor (PR), and Ki67 proliferation marker immunohistochemical stainings were assessed on 50 core needle biopsy and 50 excision samples from breast cancers by 9 pathologists according to daily practice. ER and PR statuses based on the percentages of stained nuclei were the most consistently assessed parameters (intraclass correlation coefficients, ICC 0.918-0.996), whereas Ki67 with 5 different theoretical or St Gallen Consensus Conference-proposed cut-off values demonstrated moderate to good reproducibility (ICC: 0.625-0.760). ONEST highlighted that consistent tests like ER and PR assessment needed only 2 or 3 observers for optimal evaluation of reproducibility, and the width between plots of the best and worst overall percent agreement values for 100 randomly selected permutations of observers was narrow. In contrast, with less consistently evaluated tests of Ki67 categorization, ONEST suggested at least 5 observers required for more trustful assessment of reliability, and the bandwidth of the best and worst plots was wider (up to 34% difference between two observers). ONEST has additional value to traditional calculations of the interobserver agreement by not only highlighting the number of observers needed to trustfully evaluate reproducibility but also by highlighting the rate of agreement with an increasing number of observers and disagreement between the better and worse ratings.
Collapse
Affiliation(s)
| | - Rita Bori
- Department of Pathology, Bács-Kiskun County Teaching Hospital, Kecskemét, Hungary
| | - Erika Csörgő
- Department of Pathology, Bács-Kiskun County Teaching Hospital, Kecskemét, Hungary
| | | | - Tamás Pancsa
- Department of Pathology, University of Szeged, Szeged, Hungary
| | - Anita Sejben
- Department of Pathology, University of Szeged, Szeged, Hungary
| | - István Sejben
- Department of Pathology, Bács-Kiskun County Teaching Hospital, Kecskemét, Hungary
| | - András Vörös
- Department of Pathology, University of Szeged, Szeged, Hungary
| | - Tamás Zombori
- Department of Pathology, University of Szeged, Szeged, Hungary
| | - Tibor Nyári
- Department of Medical Physics and Informatics, University of Szeged, Szeged, Hungary
| | - Gábor Cserni
- Department of Pathology, Bács-Kiskun County Teaching Hospital, Kecskemét, Hungary. .,Department of Pathology, University of Szeged, Szeged, Hungary.
| |
Collapse
|