ChatGPT4's Proficiency in Addressing Patients' Questions on Systemic Lupus Erythematosus: A Blinded Comparative Study with Specialists

OBJECTIVES

The efficacy of artificial intelligence (AI)-driven chatbots like ChatGPT4 in specialized medical consultations, particularly in rheumatology, remains underexplored. This study compares the proficiency of ChatGPT4' responses with practicing rheumatologists to inquiries from patients with systemic lupus erythematosus (SLE).

METHODS

In this cross-sectional study, we curated 95 frequently asked questions (FAQs), including 55 in Chinese and 40 in English. Responses for FAQs from ChatGPT4 and 5 rheumatologists were scored separately by a panel of rheumatologists and a group of patients with SLE across 6 domains (scientific validity, logical consistency, comprehensibility, completeness, satisfaction level, and empathy) on a 0-10 scale (a score of 0 indicates entirely incorrect responses, while 10 indicates accurate and comprehensive answers).

RESULTS

Rheumatologists' scoring revealed that ChatGPT4-generated responses outperformed those from rheumatologists in satisfaction level and empathy, with mean differences of 0.537 (95% CI, 0.252-0.823; p < 0.01) and 0.460 (95% CI, 0.227-0.693 p < 0.01), respectively. From the SLE patients' perspective, ChatGPT4-generated responses were comparable to the rheumatologist-provided answers in all 6 domains. Subgroup analysis revealed ChatGPT4 responses were more logically consistent and complete regardless of language, and exhibited greater comprehensibility, satisfaction, and empathy in Chinese. However, ChatGPT4 responses were inferior in comprehensibility for English FAQs.

CONCLUSION

ChatGPT4 demonstrated comparable, possibly better in certain domains, to address FAQs from patients with SLE, when compared with the answers provided by specialists. This study showed the potential of applying ChatGPT4 to improve consultation in SLE patients.