Interpretation of Clinical Retinal Images Using an Artificial Intelligence Chatbot

Large Language Models in Ophthalmology: A Review of Publications from Top Ophthalmology Journals

Purpose

To review and evaluate the current literature on the application and impact of large language models (LLMs) in the field of ophthalmology, focusing on studies published in high-ranking ophthalmology journals.

Design

This is a retrospective review of published articles.

Participants

This study did not involve human participation.

Methods

Articles published in the first quartile (Q1) of ophthalmology journals on Scimago Journal & Country Rank discussing different LLMs up to June 7, 2024, were reviewed, parsed, and analyzed.

Main Outcome Measures

All available articles were parsed and analyzed, which included the article and author characteristics and data regarding the LLM used and its applications, focusing on its use in medical education, clinical assistance, research, and patient education.

Results

There were 35 Q1-ranked journals identified, 19 of which contained articles discussing LLMs, with 101 articles eligible for review. One-third were original investigations (32%; 32/101), with an average of 5.3 authors per article. The United States (50.4%; 51/101) was the most represented country, followed by the United Kingdom (25.7%; 26/101) and Canada (16.8%; 17/101). ChatGPT was the most used LLM among the studies, with different versions discussed and compared. Large language model applications were discussed relevant to their implications in medical education, clinical assistance, research, and patient education.

Conclusions

The numerous publications on the use of LLM in ophthalmology can provide valuable insights for stakeholders and consumers of these applications. Large language models present significant opportunities for advancement in ophthalmology, particularly in team science, education, clinical assistance, and research. Although LLMs show promise, they also show challenges such as performance inconsistencies, bias, and ethical concerns. The study emphasizes the need for ongoing artificial intelligence improvement, ethical guidelines, and multidisciplinary collaboration.

Financial Disclosures

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Multimodal Performance of GPT-4 in Complex Ophthalmology Cases

Objectives: The integration of multimodal capabilities into GPT-4 represents a transformative leap for artificial intelligence in ophthalmology, yet its utility in scenarios requiring advanced reasoning remains underexplored. This study evaluates GPT-4's multimodal performance on open-ended diagnostic and next-step reasoning tasks in complex ophthalmology cases, comparing it against human expertise. Methods: GPT-4 was assessed across three study arms: (1) text-based case details with figure descriptions, (2) cases with text and accompanying ophthalmic figures, and (3) cases with figures only (no figure descriptions). We compared GPT-4's diagnostic and next-step accuracy across arms and benchmarked its performance against three board-certified ophthalmologists. Results: GPT-4 achieved 38.4% (95% CI [33.9%, 43.1%]) diagnostic accuracy and 57.8% (95% CI [52.8%, 62.2%]) next-step accuracy when prompted with figures without descriptions. Diagnostic accuracy declined significantly compared to text-only prompts (p = 0.007), though the next-step performance was similar (p = 0.140). Adding figure descriptions restored diagnostic accuracy (49.3%) to near parity with text-only prompts (p = 0.684). Using figures without descriptions, GPT-4's diagnostic accuracy was comparable to two ophthalmologists (p = 0.30, p = 0.41) but fell short of the highest-performing ophthalmologist (p = 0.0004). For next-step accuracy, GPT-4 was similar to one ophthalmologist (p = 0.22) but underperformed relative to the other two (p = 0.0015, p = 0.0017). Conclusions: GPT-4's diagnostic performance diminishes when relying solely on ophthalmic images without textual context, highlighting limitations in its current multimodal capabilities. Despite this, GPT-4 demonstrated comparable performance to at least one ophthalmologist on both diagnostic and next-step reasoning tasks, emphasizing its potential as an assistive tool. Future research should refine multimodal prompts and explore iterative or sequential prompting strategies to optimize AI-driven interpretation of complex ophthalmic datasets.

A systematic review and meta-analysis of diagnostic performance comparison between generative AI and physicians

While generative artificial intelligence (AI) has shown potential in medical diagnostics, comprehensive evaluation of its diagnostic performance and comparison with physicians has not been extensively explored. We conducted a systematic review and meta-analysis of studies validating generative AI models for diagnostic tasks published between June 2018 and June 2024. Analysis of 83 studies revealed an overall diagnostic accuracy of 52.1%. No significant performance difference was found between AI models and physicians overall (p = 0.10) or non-expert physicians (p = 0.93). However, AI models performed significantly worse than expert physicians (p = 0.007). Several models demonstrated slightly higher performance compared to non-experts, although the differences were not significant. Generative AI demonstrates promising diagnostic capabilities with accuracy varying by model. Although it has not yet achieved expert-level reliability, these findings suggest potential for enhancing healthcare delivery and medical education when implemented with appropriate understanding of its limitations.

Performance of ChatGPT in French language analysis of multimodal retinal cases

PURPOSE

Prior literature has suggested a reduced performance of large language models (LLMs) in non-English analyses, including Arabic and French. However, there are no current studies testing the multimodal performance of ChatGPT in French ophthalmology cases, and comparing this to the results observed in the English literature. We compared the performance of ChatGPT-4 in French and English on open-ended prompts using multimodal input data from retinal cases.

METHODS

GPT-4 was prompted in English and French using a public dataset containing 67 retinal cases from the ophthalmology education website OCTCases.com. The clinical case and accompanying ophthalmic images comprised the prompt, along with the open-ended question: "What is the most likely diagnosis?" Systematic prompting was used to identify and compare relevant factor(s) contributing to correct and incorrect responses. Diagnostic accuracy was the primary outcome, defined as the proportion of correctly diagnosed cases in French and English. Diagnoses were compared with the answer key on OCTCases to confirm correct or incorrect responses. Clinically relevant factors reported by the LLM as contributory to its decision-making were secondary endpoints.

RESULTS

The diagnostic accuracies of GPT-4 in English and French were 35.8% and 28.4%, respectively (χ2, P=0.36). Imaging findings were reported as most influential for correct diagnosis in English (37.5%) and French (42.1%) (P=0.76). In incorrectly diagnosed cases, imaging findings were primarily implicated in English (35.6%) and French (33.3%) (P=0.81). In incorrectly diagnosed cases, the differential diagnosis list contained the correct diagnosis in 39.5% of English cases and 41.7% of French cases (P=0.83).

CONCLUSION

Our results suggest that GPT-4 performed similarly in English and French on all quantitative performance metrics measured. Ophthalmic images were identified in both languages as critical for correct diagnosis. Future research should assess LLM comprehension through the clarity, grammatical, cultural, and idiomatic accuracy of its responses.

Opportunities and Challenges of Chatbots in Ophthalmology: A Narrative Review

Artificial intelligence (AI) is becoming increasingly influential in ophthalmology, particularly through advancements in machine learning, deep learning, robotics, neural networks, and natural language processing (NLP). Among these, NLP-based chatbots are the most readily accessible and are driven by AI-based large language models (LLMs). These chatbots have facilitated new research avenues and have gained traction in both clinical and surgical applications in ophthalmology. They are also increasingly being utilized in studies on ophthalmology-related exams, particularly those containing multiple-choice questions (MCQs). This narrative review evaluates both the opportunities and the challenges of integrating chatbots into ophthalmology research, with separate assessments of studies involving open- and close-ended questions. While chatbots have demonstrated sufficient accuracy in handling MCQ-based studies, supporting their use in education, additional exam security measures are necessary. The research on open-ended question responses suggests that AI-based LLM chatbots could be applied across nearly all areas of ophthalmology. They have shown promise for addressing patient inquiries, offering medical advice, patient education, supporting triage, facilitating diagnosis and differential diagnosis, and aiding in surgical planning. However, the ethical implications, confidentiality concerns, physician liability, and issues surrounding patient privacy remain pressing challenges. Although AI has demonstrated significant promise in clinical patient care, it is currently most effective as a supportive tool rather than as a replacement for human physicians.

Utilizing Large Language Models in Ophthalmology: The Current Landscape and Challenges

A large language model (LLM) is an artificial intelligence (AI) model that uses natural language processing (NLP) to understand, interpret, and generate human-like language responses from unstructured text input. Its real-time response capabilities and eloquent dialogue enhance the interactive user experience in human-AI communication like never before. By gathering several sources on the internet, LLM chatbots can interact and respond to a wide range of queries, including problem solving, text summarization, and creating informative notes. Since ophthalmology is one of the medical fields integrating image analysis, telemedicine, AI, and other technologies, LLMs are likely to play an important role in eye care in the near future. This review summarizes the performance and potential applicability of LLMs in ophthalmology according to currently available publications.