Assessing the medical reasoning skills of GPT-4 in complex ophthalmology cases

BACKGROUND/AIMS

This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases.

METHODS

We tested GPT-4 on 422 Journal of the American Medical Association Ophthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort.

RESULTS

Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020).

CONCLUSION

Improved prompting enhances GPT-4's performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis.

Revisiting annual screening for latent tuberculosis infection in healthcare workers: a cost-effectiveness analysis

BACKGROUND

In North America, tuberculosis incidence is now very low and risk to healthcare workers has fallen. Indeed, recent cohort data question routine annual tuberculosis screening in this context. We compared the cost-effectiveness of three potential strategies for ongoing screening of North American healthcare workers at risk of exposure. The analysis did not evaluate the cost-effectiveness of screening at hiring, and considered only workers with negative baseline tests.

METHODS

A decision analysis model simulated a hypothetical cohort of 1000 workers following negative baseline tests, considering duties, tuberculosis exposure, testing and treatment. Two tests were modelled, the tuberculin skin test (TST) and QuantiFERON®-TB-Gold In-Tube (QFT). Three screening strategies were compared: (1) annual screening, where workers were tested yearly; (2) targeted screening, where workers with high-risk duties (e.g. respiratory therapy) were tested yearly and other workers only after recognised exposure; and (3) post exposure-only screening, where all workers were tested only after recognised exposure. Workers with high-risk duties had 1% annual risk of infection, while workers with standard patient care duties had 0.3%. In an alternate higher-risk scenario, the corresponding annual risks of infection were 3% and 1%, respectively. We projected costs, morbidity, quality-adjusted survival and mortality over 20 years after hiring. The analysis used the healthcare system perspective and a 3% annual discount rate.

RESULTS

Over 20 years, annual screening with TST yielded an expected 2.68 active tuberculosis cases/1000 workers, versus 2.83 for targeted screening and 3.03 for post-exposure screening only. In all cases, annual screening was associated with poorer quality-adjusted survival, i.e. lost quality-adjusted life years, compared to targeted or post-exposure screening only. The annual TST screening strategy yielded an incremental cost estimate of $1,717,539 per additional case prevented versus targeted TST screening, which in turn cost an incremental $426,678 per additional case prevented versus post-exposure TST screening only. With the alternate "higher-risk" scenario, the annual TST strategy cost an estimated $426,678 per additional case prevented versus the targeted TST strategy, which cost an estimated $52,552 per additional case prevented versus post-exposure TST screening only. In all cases, QFT was more expensive than TST, with no or limited added benefit. Sensitivity analysis suggested that, even with limited exposure recognition, annual screening was poorly cost-effective.

CONCLUSIONS

For most North American healthcare workers, annual tuberculosis screening appears poorly cost-effective. Reconsideration of screening practices is warranted.