1
|
Apornvirat S, Namboonlue C, Laohawetwanit T. Comparative analysis of ChatGPT and Bard in answering pathology examination questions requiring image interpretation. Am J Clin Pathol 2024:aqae036. [PMID: 38619043 DOI: 10.1093/ajcp/aqae036] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2023] [Accepted: 03/08/2024] [Indexed: 04/16/2024] Open
Abstract
OBJECTIVES To evaluate the accuracy of ChatGPT and Bard in answering pathology examination questions requiring image interpretation. METHODS The study evaluated ChatGPT-4 and Bard's performance using 86 multiple-choice questions, with 17 (19.8%) focusing on general pathology and 69 (80.2%) on systemic pathology. Of these, 62 (72.1%) included microscopic images, and 57 (66.3%) were first-order questions focusing on diagnosing the disease. The authors presented these artificial intelligence (AI) tools with questions, both with and without clinical contexts, and assessed their answers against a reference standard set by pathologists. RESULTS ChatGPT-4 achieved a 100% (n = 86) accuracy rate in questions with clinical context, surpassing Bard's 87.2% (n = 75). Without context, the accuracy of both AI tools declined significantly, with ChatGPT-4 at 52.3% (n = 45) and Bard at 38.4% (n = 33). ChatGPT-4 consistently outperformed Bard across various categories, particularly in systemic pathology and first-order questions. A notable issue identified was Bard's tendency to "hallucinate" or provide plausible but incorrect answers, especially without clinical context. CONCLUSIONS This study demonstrated the potential of ChatGPT and Bard in pathology education, stressing the importance of clinical context for accurate AI interpretations of pathology images. It underlined the need for careful AI integration in medical education.
Collapse
Affiliation(s)
- Sompon Apornvirat
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| | | | - Thiyaphat Laohawetwanit
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| |
Collapse
|
2
|
Laohawetwanit T, Wanpiyarat N, Lerttanatum N, Apornvirat S, Kantasiripitak C, Atiroj N, Pisutpunya A, Phairintr P, Suttichan K, Poungmeechai N, Tassanawarawat T, Chumponpanich N, Khueankaeo C, Chaijitrawan P, Sooksaen P, Stithsuksanoh C, Thinpanja W, Kaewnopparat W. Histopathologic evaluation of gastric intestinal metaplasia in non-neoplastic biopsy specimens: Accuracy and interobserver reliability among general pathologists and pathology residents. Ann Diagn Pathol 2024; 70:152284. [PMID: 38422806 DOI: 10.1016/j.anndiagpath.2024.152284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2024] [Revised: 02/21/2024] [Accepted: 02/21/2024] [Indexed: 03/02/2024]
Abstract
OBJECTIVES This study aimed to evaluate the accuracy and interobserver reliability of diagnosing and subtyping gastric intestinal metaplasia (IM) among general pathologists and pathology residents at a university hospital in Thailand, focusing on the challenges in the histopathologic evaluation of gastric IM for less experienced practitioners. METHODS The study analyzed 44 non-neoplastic gastric biopsies, using a consensus diagnosis of gastrointestinal pathologists as the reference standard. Participants included 6 general pathologists and 9 pathology residents who assessed gastric IM and categorized its subtype (complete, incomplete, or mixed) on digital slides. After initial evaluations and receiving feedback, participants reviewed specific images of gastric IM, as agreed by experts. Following a one-month washout period, a reevaluation of the slides was conducted. RESULTS Diagnostic accuracy, interobserver reliability, and time taken for diagnosis improved following training, with general pathologists showing higher accuracies than residents (median accuracy of gastric IM detection: 100 % vs. 97.7 %). Increased years of experience were associated with more IM detection accuracy (p-value<0.05). However, the overall median accuracy for diagnosing incomplete IM remained lower than for complete IM (86.4 % vs. 97.7 %). After training, diagnostic errors occurred in 6 out of 44 specimens (13.6 %), reported by over 40 % of participants. Errors involved omitting 5 slides with incomplete IM and 1 with complete IM, all showing a subtle presence of IM. CONCLUSIONS The study highlights the diagnostic challenges in identifying incomplete gastric IM, showing notable discrepancies in accuracy and interobserver agreement. It underscores the need for better diagnostic protocols and training to enhance detection and management outcomes.
Collapse
Affiliation(s)
- Thiyaphat Laohawetwanit
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand; Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand.
| | - Natcha Wanpiyarat
- Department of Pathology, King Chulalongkorn Memorial Hospital, Bangkok, Thailand
| | | | - Sompon Apornvirat
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand; Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand.
| | - Charinee Kantasiripitak
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand; Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand.
| | - Nawaluk Atiroj
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Department of Pathology, Faculty of Medicine, Thammasat University, Pathum Thani, Thailand.
| | - Adiluck Pisutpunya
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Department of Pathology, Faculty of Medicine, Thammasat University, Pathum Thani, Thailand
| | - Putch Phairintr
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Department of Pathology, Faculty of Medicine, Thammasat University, Pathum Thani, Thailand
| | - Komkrit Suttichan
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Department of Pathology, Faculty of Medicine, Thammasat University, Pathum Thani, Thailand
| | - Natcha Poungmeechai
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| | | | | | | | | | - Pornchai Sooksaen
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| | | | - Warut Thinpanja
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| | | |
Collapse
|
3
|
Laohawetwanit T, Namboonlue C, Apornvirat S. Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomas. J Clin Pathol 2024:jcp-2023-209304. [PMID: 38199797 DOI: 10.1136/jcp-2023-209304] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2023] [Accepted: 01/03/2024] [Indexed: 01/12/2024]
Abstract
AIMS To evaluate the accuracy of Chat Generative Pre-trained Transformer (ChatGPT) powered by GPT-4 in histopathological image detection and classification of colorectal adenomas using the diagnostic consensus provided by pathologists as a reference standard. METHODS A study was conducted with 100 colorectal polyp photomicrographs, comprising an equal number of adenomas and non-adenomas, classified by two pathologists. These images were analysed by classic GPT-4 for 1 time in October 2023 and custom GPT-4 for 20 times in December 2023. GPT-4's responses were compared against the reference standard through statistical measures to evaluate its proficiency in histopathological diagnosis, with the pathologists further assessing the model's descriptive accuracy. RESULTS GPT-4 demonstrated a median sensitivity of 74% and specificity of 36% for adenoma detection. The median accuracy of polyp classification varied, ranging from 16% for non-specific changes to 36% for tubular adenomas. Its diagnostic consistency, indicated by low kappa values ranging from 0.06 to 0.11, suggested only poor to slight agreement. All of the microscopic descriptions corresponded with their diagnoses. GPT-4 also commented about the limitations in its diagnoses (eg, slide diagnosis best done by pathologists, the inadequacy of single-image diagnostic conclusions, the need for clinical data and a higher magnification view). CONCLUSIONS GPT-4 showed high sensitivity but low specificity in detecting adenomas and varied accuracy for polyp classification. However, its diagnostic consistency was low. This artificial intelligence tool acknowledged its diagnostic limitations, emphasising the need for a pathologist's expertise and additional clinical context.
Collapse
Affiliation(s)
- Thiyaphat Laohawetwanit
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| | | | - Sompon Apornvirat
- Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand
| |
Collapse
|
4
|
Kantasiripitak C, Laohawetwanit T, Apornvirat S, Niemnapa K. Validation of whole slide imaging for frozen section diagnosis of lymph node metastasis: A retrospective study from a tertiary care hospital in Thailand. Ann Diagn Pathol 2022; 60:151987. [PMID: 35700561 DOI: 10.1016/j.anndiagpath.2022.151987] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Revised: 05/23/2022] [Accepted: 06/03/2022] [Indexed: 11/01/2022]
Abstract
BACKGROUND The use of whole slide imaging (WSI) for frozen section (FS) diagnosis is helpful, particularly in the context of pathologist shortages. However, there is minimal data on such usage in resource-limited settings. This study aims to validate the use of WSI for FS diagnosis of lymph node metastasis using a low-cost virtual microscope scanner with consumer-grade laptops at a tertiary care hospital in Thailand. METHODS FS slides were retrieved for which the clinical query was to evaluate lymph node metastasis. They were digitized by a virtual microscope scanner (MoticEasyScan, Hong Kong) using up to 40× optical magnification. Three observers with different pathology experience levels diagnosed each slide, reviewing glass slides (GS) followed by digital slides (DS) after two weeks of a wash out period. WSI and GS diagnoses were compared. The time used for scanning and diagnosis of each slide was recorded. RESULTS 295 FS slides were retrieved and digitized. The first-time successful scanning rate was 93.6 %. The mean scanning time was 2 min per slide. Both intraobserver agreement and interobserver agreement of WSI and GS diagnoses were high (Cohen's K; kappa value >0.84). The time used for DS diagnosis decreased as the observer's experience with WSI increased. CONCLUSIONS Despite varying pathological experiences, observers using WSI provided accurate FS diagnoses of lymph node metastasis. The time required for DS diagnoses decreased with additional observer's experience with WSI. Therefore, a WSI system containing low-cost scanners and consumer-grade laptops could be used for FS services in hospital laboratories lacking pathologists.
Collapse
Affiliation(s)
| | - Thiyaphat Laohawetwanit
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand.
| | - Sompon Apornvirat
- Division of Pathology, Thammasat University Hospital, Pathum Thani, Thailand; Division of Pathology, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
| | - Kongkot Niemnapa
- Advanced Digital Simulation Center, Chulabhorn International College of Medicine, Thammasat University, Pathum Thani, Thailand
| |
Collapse
|