Can ChatGPT pass the MRCP (UK) written examinations? Analysis of performance and errors using a clinical decision-reasoning framework

OBJECTIVE

Large language models (LLMs) such as ChatGPT are being developed for use in research, medical education and clinical decision systems. However, as their usage increases, LLMs face ongoing regulatory concerns. This study aims to analyse ChatGPT's performance on a postgraduate examination to identify areas of strength and weakness, which may provide further insight into their role in healthcare.

DESIGN

We evaluated the performance of ChatGPT 4 (24 May 2023 version) on official MRCP (Membership of the Royal College of Physicians) parts 1 and 2 written examination practice questions. Statistical analysis was performed using Python. Spearman rank correlation assessed the relationship between the probability of correctly answering a question and two variables: question difficulty and question length. Incorrectly answered questions were analysed further using a clinical reasoning framework to assess the errors made.

SETTING

Online using ChatGPT web interface.

PRIMARY AND SECONDARY OUTCOME MEASURES

Primary outcome was the score (percentage questions correct) in the MRCP postgraduate written examinations. Secondary outcomes were qualitative categorisation of errors using a clinical decision-making framework.

RESULTS

ChatGPT achieved accuracy rates of 86.3% (part 1) and 70.3% (part 2). Weak but significant correlations were found between ChatGPT's accuracy and both just-passing rates in part 2 (r=0.34, p=0.0001) and question length in part 1 (r=-0.19, p=0.008). Eight types of error were identified, with the most frequent being factual errors, context errors and omission errors.

CONCLUSION

ChatGPT performance greatly exceeded the passing mark for both exams. Multiple choice examinations provide a benchmark for LLM performance which is comparable to human demonstrations of knowledge, while also highlighting the errors LLMs make. Understanding the reasons behind ChatGPT's errors allows us to develop strategies to prevent them in medical devices that incorporate LLM technology.

Late-onset cluster seizures and intellectual disability associated with a novel truncation variant in SMC1A

•

Epilepsy due to truncating SMC1A variants can present with onset in later childhood.

•

People with to truncating SMC1A variants can have normal development prior to presentation.

•

Seizures occur periodically in clusters and are poorly responsive to antiseizure medications.

SMC1A variants are known to cause Cornelia de Lange Syndrome (CdLS) which encompasses a clinical spectrum of intellectual disability, dysmorphic features (long or thick eyebrows, a hypomorphic philtrum and small nose) and, in some cases, epilepsy. More recently, SMC1A truncating variants have been described as the cause of a neurodevelopmental disorder with early-childhood onset drug-resistant epilepsy with seizures that occur in clusters, similar to that seen in PCDH19-related epilepsy, but without the classical features of CdLS. Here, we report the case of a 28-year-old woman with a de novo heterozygous truncating variant in SMC1A who unusually presented with seizures at the late age of 12 years and had normal development into adulthood.

Carotid intima–media thickness and the prediction of vascular events

Carotid intima–media thickness (cIMT) has received interest as a predictor of cardiovascular events in recent years. Use of cIMT in a clinical setting is limited by the variability in measurement and the lack of evidence for its use in clinical risk prediction. This review examines the major studies that have assessed the relationship between cIMT and cardiovascular event risk and discusses the current role of IMT in cardiovascular risk prediction.

Ca2+ entry in gonadotrophs and alpha T3-1 cells: does store-dependent Ca2+ influx mediate gonadotrophin-releasing hormone action?

In pituitary gonadotrophs GnRH causes biphasic (spike and plateau) increases in cytosolic Ca2+ ([Ca2+]i) and gonadotrophin release. The spike phases reflect mobilization of stored Ca2+ and the plateau responses are attributed, in part, to Ca2+ influx via voltage-sensitive Ca2+ channels. In recent years, store-dependent Ca2+ influx (SDCI), in which depletion of the intracellular inositol 1,4,5-trisphosphate-mobilizable pool stimulates Ca2+ influx, has emerged as a major form of Ca2+ entry activated by phosphoinositidase C-coupled receptors in non-excitable cells. More recent evidence also indicates a role for SDCI in excitable cells. We have used dynamic video imaging of [Ca2+]i in alpha T3-1 cells (a gonadotroph-derived cell line) and manipulation of the filling state of the GnRH-mobilizable Ca2+ pool to test the possible role of SDCI in GnRH action. In Ca(2+)-containing medium, GnRH caused a biphasic increase in [Ca2+]i whereas in Ca(2+)-free medium only a transient increase occurred. The response to a second stimulation with GnRH in Ca(2+)-free medium was reduced by > 95% (demonstrating that Ca2+ pool depletion had occurred) and was recovered after brief exposure to Ca(2+)-containing medium (which enables refilling of the pool). Ionomycin (a Ca2+ ionophore) and thapsigargin (which inhibits the Ca(2+)-sequestering ATPase of the endoplasmic reticulum) also transiently increased [Ca2+]i in Ca(2+)-free medium and depleted the GnRH-mobilizable pool as indicated by greatly reduced subsequent responses to GnRH. Pool depletion also occurs on stimulation with GnRH in Ca(2+)-containing medium because addition of ionomycin and Ca(2+)-free medium during the plateau phase of the GnRH response caused only a reduction in [Ca2+]i rather than the transient increase seen without GnRH. To deplete intracellular Ca2+ pools, cells were pretreated in Ca(2+)-free medium with thapsigargin or GnRH and then, after extensive washing, returned to Ca(2+)-containing medium. Pretreatment with thapsigargin augmented the increase in [Ca2+]i seen on return to Ca(2+)-containing medium (to two- to threefold higher than that seen in control cells) indicating the activation of SDCI, whereas pool depletion by GnRH pretreatment had no such effect. To ensure maintained pool depletion after Ca2+ re-addition, similar studies were performed in which the thapsigargin and GnRH treatments were not washed off, but were retained through the period of return to Ca(2+)-containing medium. Return of GnRH-treated cells to Ca(2+)-containing medium caused an increase in [Ca2+]i which was inhibited by nicardipine, whereas the increase seen on return of thapsigargin-treated cells to Ca(2+)-containing medium was not reduced by nicardipine. The quench of fura-2 fluorescence by MnCl2 (used as a reporter of Ca2+ influx) was increased by GnRH and thapsigargin, indicating that both stimulate Ca2+ influx via Mn2+ permeant channels. The GnRH effect was abolished by nicardipine whereas that of thapsigargin was not. Finally, depletion of intracellular Ca2+ pools by pretreatment of superfused rat pituitary cells with GnRH or thapsigargin in Ca(2+)-free medium did not enhance LH release on return to Ca(2+)-containing medium. The results indicate that (a) thapsigargin stimulates SDCI in alpha T3-1 cells via nicardipine-insensitive Ca2+ channels, (b) in spite of the fact that GnRH depletes the hormone-mobilizable Ca2+ pool, it fails to stimulate SDCI, (c) GnRH stimulates Ca2+ entry predominantly via nicardipine-sensitive channels, a route not activated by SDCI and (d) in rat gonadotrophs, GnRH-stimulated LH release is not mediated by SDCI.