Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study

Introduction

Artificial intelligence (AI) technology has made rapid progress for disease diagnosis and triage. In the field of ophthalmic diseases, image-based diagnosis has achieved high accuracy but still encounters limitations due to the lack of medical history. The emergence of ChatGPT enables human-computer interaction, allowing for the development of a multimodal AI system that integrates interactive text and image information.

Objective

To develop a multimodal AI system using ChatGPT and anterior segment images for diagnosing and triaging ophthalmic diseases. To assess the AI system's performance through a two-stage cross-sectional study, starting with silent evaluation and followed by early clinical evaluation in outpatient clinics.

Methods and analysis

Our study will be conducted across three distinct centers in Shanghai, Nanjing, and Suqian. The development of the smartphone-based multimodal AI system will take place in Shanghai with the goal of achieving ≥90% sensitivity and ≥95% specificity for diagnosing and triaging ophthalmic diseases. The first stage of the cross-sectional study will explore the system's performance in Shanghai's outpatient clinics. Medical histories will be collected without patient interaction, and anterior segment images will be captured using slit lamp equipment. This stage aims for ≥85% sensitivity and ≥95% specificity with a sample size of 100 patients. The second stage will take place at three locations, with Shanghai serving as the internal validation dataset, and Nanjing and Suqian as the external validation dataset. Medical history will be collected through patient interviews, and anterior segment images will be captured via smartphone devices. An expert panel will establish reference standards and assess AI accuracy for diagnosis and triage throughout all stages. A one-vs.-rest strategy will be used for data analysis, and a post-hoc power calculation will be performed to evaluate the impact of disease types on AI performance.

Discussion

Our study may provide a user-friendly smartphone-based multimodal AI system for diagnosis and triage of ophthalmic diseases. This innovative system may support early detection of ocular abnormalities, facilitate establishment of a tiered healthcare system, and reduce the burdens on tertiary facilities.

Trial registration

The study was registered in ClinicalTrials.gov on June 25th, 2023 (NCT05930444).

Comparisons of artificial intelligence algorithms in automatic segmentation for fungal keratitis diagnosis by anterior segment images

Purpose

This study combines automatic segmentation and manual fine-tuning with an early fusion method to provide efficient clinical auxiliary diagnostic efficiency for fungal keratitis.

Methods

First, 423 high-quality anterior segment images of keratitis were collected in the Department of Ophthalmology of the Jiangxi Provincial People's Hospital (China). The images were divided into fungal keratitis and non-fungal keratitis by a senior ophthalmologist, and all images were divided randomly into training and testing sets at a ratio of 8:2. Then, two deep learning models were constructed for diagnosing fungal keratitis. Model 1 included a deep learning model composed of the DenseNet 121, mobienet_v2, and squeezentet1_0 models, the least absolute shrinkage and selection operator (LASSO) model, and the multi-layer perception (MLP) classifier. Model 2 included an automatic segmentation program and the deep learning model already described. Finally, the performance of Model 1 and Model 2 was compared.

Results

In the testing set, the accuracy, sensitivity, specificity, F1-score, and the area under the receiver operating characteristic (ROC) curve (AUC) of Model 1 reached 77.65, 86.05, 76.19, 81.42%, and 0.839, respectively. For Model 2, accuracy improved by 6.87%, sensitivity by 4.43%, specificity by 9.52%, F1-score by 7.38%, and AUC by 0.086, respectively.

Conclusion

The models in our study could provide efficient clinical auxiliary diagnostic efficiency for fungal keratitis.

Application of artificial intelligence for automatic cataract staging based on anterior segment images: comparing automatic segmentation approaches to manual segmentation

Purpose

Cataract is one of the leading causes of blindness worldwide, accounting for >50% of cases of blindness in low- and middle-income countries. In this study, two artificial intelligence (AI) diagnosis platforms are proposed for cortical cataract staging to achieve a precise diagnosis.

Methods

A total of 647 high quality anterior segment images, which included the four stages of cataracts, were collected into the dataset. They were divided randomly into a training set and a test set using a stratified random-allocation technique at a ratio of 8:2. Then, after automatic or manual segmentation of the lens area of the cataract, the deep transform-learning (DTL) features extraction, PCA dimensionality reduction, multi-features fusion, fusion features selection, and classification models establishment, the automatic and manual segmentation DTL platforms were developed. Finally, the accuracy, confusion matrix, and area under the receiver operating characteristic (ROC) curve (AUC) were used to evaluate the performance of the two platforms.

Results

In the automatic segmentation DTL platform, the accuracy of the model in the training and test sets was 94.59 and 84.50%, respectively. In the manual segmentation DTL platform, the accuracy of the model in the training and test sets was 97.48 and 90.00%, respectively. In the test set, the micro and macro average AUCs of the two platforms reached >95% and the AUC for each classification was >90%. The results of a confusion matrix showed that all stages, except for mature, had a high recognition rate.

Conclusion

Two AI diagnosis platforms were proposed for cortical cataract staging. The resulting automatic segmentation platform can stage cataracts more quickly, whereas the resulting manual segmentation platform can stage cataracts more accurately.

Application of artificial intelligence models for detecting the pterygium that requires surgical treatment based on anterior segment images

Background and aim

A pterygium is a common ocular surface disease, which not only affects facial appearance but can also grow into the tissue layer, causing astigmatism and vision loss. In this study, an artificial intelligence model was developed for detecting the pterygium that requires surgical treatment. The model was designed using ensemble deep learning (DL).

Methods

A total of 172 anterior segment images of pterygia were obtained from the Jiangxi Provincial People's Hospital (China) between 2017 and 2022. They were divided by a senior ophthalmologist into the non-surgery group and the surgery group. An artificial intelligence model was then developed based on ensemble DL, which was integrated with four benchmark models: the Resnet18, Alexnet, Googlenet, and Vgg11 model, for detecting the pterygium that requires surgical treatment, and Grad-CAM was used to visualize the DL process. Finally, the performance of the ensemble DL model was compared with the classical Resnet18 model, Alexnet model, Googlenet model, and Vgg11 model.

Results

The accuracy and area under the curve (AUC) of the ensemble DL model was higher than all of the other models. In the training set, the accuracy and AUC of the ensemble model was 94.20% and 0.978, respectively. In the testing set, the accuracy and AUC of the ensemble model was 94.12% and 0.980, respectively.

Conclusion

This study indicates that this ensemble DL model, coupled with the anterior segment images in our study, might be an automated and cost-saving alternative for detection of the pterygia that require surgery.