Large language models for generating medical examinations: systematic review

BACKGROUND

Writing multiple choice questions (MCQs) for the purpose of medical exams is challenging. It requires extensive medical knowledge, time and effort from medical educators. This systematic review focuses on the application of large language models (LLMs) in generating medical MCQs.

METHODS

The authors searched for studies published up to November 2023. Search terms focused on LLMs generated MCQs for medical examinations. Non-English, out of year range and studies not focusing on AI generated multiple-choice questions were excluded. MEDLINE was used as a search database. Risk of bias was evaluated using a tailored QUADAS-2 tool.

RESULTS

Overall, eight studies published between April 2023 and October 2023 were included. Six studies used Chat-GPT 3.5, while two employed GPT 4. Five studies showed that LLMs can produce competent questions valid for medical exams. Three studies used LLMs to write medical questions but did not evaluate the validity of the questions. One study conducted a comparative analysis of different models. One other study compared LLM-generated questions with those written by humans. All studies presented faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify.

CONCLUSIONS

LLMs can be used to write MCQs for medical examinations. However, their limitations cannot be ignored. Further study in this field is essential and more conclusive evidence is needed. Until then, LLMs may serve as a supplementary tool for writing medical examinations. 2 studies were at high risk of bias. The study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

Association between long working hours and engagement in preventive healthcare services in Korean workers: Findings from the Korean National Health and Nutrition Examination Survey

BACKGROUND

Engagement in preventive healthcare services is crucial for preventing diseases. We explored how working hours are associated with engagement in preventive healthcare services, with a focus on gender differences.

METHODS

This cross-sectional study used data from the 2007-2012 Korean National Health and Nutrition Examination Survey. The dependent variable was engagement in each of the five preventive healthcare services (health check-ups, influenza vaccination, and stomach, breast, and cervical cancer screenings). We estimated the prevalence ratios (PRs) and 95% confidence intervals (CIs) using robust Poisson regression.

RESULTS

The study analyzed 19,819 workers (9119 women). The adjusted PRs (95% CI) of the association between working ≥55 h per week and engagement in preventive healthcare services among men were 0.95 (0.90-1.00) for health check-ups, 0.86 (0.77-0.96) for influenza vaccination, and 0.95 (0.87-1.03) for stomach cancer screening compared to working 35-40 h per week. Among women, the adjusted PRs (95% CI) of the association between working ≥55 h per week and engagement in preventive healthcare services were 0.84 (0.78-0.91) for health check-ups, 0.82 (0.73-0.92) for influenza vaccination, and 0.88 (0.80-0.97) for stomach, 0.85 (0.78-0.94) for breast, and 0.82 (0.74-0.91) for cervical cancer screenings.

CONCLUSION

Long working hours were negatively associated with engagement in preventive healthcare services, and the association was pronounced among female workers. Efforts to promote preventive healthcare participation among individuals with long working hours are necessary, and it is essential to consider the unique vulnerabilities of women when developing such policies.

Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses

BACKGROUND

Large language models like ChatGPT have revolutionized the field of natural language processing with their capability to comprehend and generate textual content, showing great potential to play a role in medical education. This study aimed to quantitatively evaluate and comprehensively analysis the performance of ChatGPT on three types of national medical examinations in China, including National Medical Licensing Examination (NMLE), National Pharmacist Licensing Examination (NPLE), and National Nurse Licensing Examination (NNLE).

METHODS

We collected questions from Chinese NMLE, NPLE and NNLE from year 2017 to 2021. In NMLE and NPLE, each exam consists of 4 units, while in NNLE, each exam consists of 2 units. The questions with figures, tables or chemical structure were manually identified and excluded by clinician. We applied direct instruction strategy via multiple prompts to force ChatGPT to generate the clear answer with the capability to distinguish between single-choice and multiple-choice questions.

RESULTS

ChatGPT failed to pass the accuracy threshold of 0.6 in any of the three types of examinations over the five years. Specifically, in the NMLE, the highest recorded accuracy was 0.5467, which was attained in both 2018 and 2021. In the NPLE, the highest accuracy was 0.5599 in 2017. In the NNLE, the most impressive result was shown in 2017, with an accuracy of 0.5897, which is also the highest accuracy in our entire evaluation. ChatGPT's performance showed no significant difference in different units, but significant difference in different question types. ChatGPT performed well in a range of subject areas, including clinical epidemiology, human parasitology, and dermatology, as well as in various medical topics such as molecules, health management and prevention, diagnosis and screening.

CONCLUSIONS

These results indicate ChatGPT failed the NMLE, NPLE and NNLE in China, spanning from year 2017 to 2021. but show great potential of large language models in medical education. In the future high-quality medical data will be required to improve the performance.

Success of ChatGPT, an AI language model, in taking the French language version of the European Board of Ophthalmology examination: A novel approach to medical knowledge assessment

PURPOSE

The purpose of this study was to evaluate the performance of ChatGPT, a cutting-edge artificial intelligence (AI) language model developed by OpenAI, in successfully completing the French language version of the European Board of Ophthalmology (EBO) examination and to assess its potential role in medical education and knowledge assessment.

METHODS

ChatGPT, based on the GPT-4 architecture, was exposed to a series of EBO examination questions in French, covering various aspects of ophthalmology. The AI's performance was evaluated by comparing its responses with the correct answers provided by ophthalmology experts. Additionally, the study assessed the time taken by ChatGPT to answer each question as a measure of efficiency.

RESULTS

ChatGPT achieved a 91% success rate on the EBO examination, demonstrating a high level of competency in ophthalmology knowledge and application. The AI provided correct answers across all question categories, indicating a strong understanding of basic sciences, clinical knowledge, and clinical management. The AI model also answered the questions rapidly, taking only a fraction of the time needed by human test-takers.

CONCLUSION

ChatGPT's performance on the French language version of the EBO examination demonstrates its potential to be a valuable tool in medical education and knowledge assessment. Further research is needed to explore optimal ways to implement AI language models in medical education and to address the associated ethical and practical concerns.

Early diagnosis of T2DM using high sensitive tests in the mandatory medical examinations for fishers, seafarers and other transport workers

Transport workers like seafarers, truck-, bus-, train- and taxi drivers and fishers have a known great inequity in health at work including high risk of developing type 2 diabetes. Their routine mandatory medical examinations use urine glucose for diabetes check with more than 50% false negatives, which should be replaced by high sensitive tests for diabetes-2, like A1C, Fasting Glucose (FPG) or Oral Glucose Tolerance Test (OGTT).

[The usefulness of a questionnaire during medical examinations for older subjects in evaluating frailty: Utilization in clinical practice]

AIM

The late-stage medical care system for older people provides medical examinations, including questionnaires for frailty. We examined whether or not this approach is useful in clinical practice.

METHODS

We used this questionnaire for the screening of frailty as follows: according to the manual prepared by the Japan Geriatrics Society, each question was classified as concerning oral (Q4, 5), physical (Q6-9), cognitive (Q10, 11), or social (Q13-15) frailty. Each frailty was defined if there was at least one negative answer in each question. The grip power and skeletal muscle index (SMI) according to a bioelectrical impedance analysis were also evaluated. Subjects who showed a reduced grip strength and SMI were defined as having sarcopenia, and those who showed only a reduced grip strength were defined as having possible sarcopenia.

RESULTS

One hundred and seventy-one subjects aged 81.0±4.2 years old (63.1% female) were enrolled. A total of 12.3% of subjects showed sarcopenia, and 17.5% showed possible sarcopenia. The prevalence of physical, cognitive, and social frailties was associated with sarcopenia. Oral frailty, defined as having decreased swallowing and mastication functions (Q4 and 5), was significantly related to sarcopenia. Physical frailty was associated with age. In patients with hypertension, a low incidence of cognitive frailty was found. Social frailty was related to a decreased body weight.

CONCLUSIONS

A questionnaire during medical examinations for older subjects may be useful for screening various frailties and may lead to promotion of the preventive care activities in the community. Further studies are needed to elucidate the relationship between each type of frailty and background characteristics.

Utility of Virtual Platform for Conducting Practical Examination for Medical Students During Covid Times: A Prospective Study from Gynaecology Department

Background

As the novel coronavirus 2019 (COVID-19) continues its pandemic surge globally along with its social distancing norms, the physical conduction of practical examinations for medical graduates and postgraduates has become difficult. Software-based systems and social media platforms could provide alternatives for ensuring regular medical education and exam-oriented assessments. In this context, we evaluated our own experience with virtual conduction of semester practical exams for medical graduates.

Material and Methods

This prospective study was conducted in Gynaecology and Obstetrics department. We employed live streaming educational video conferencing software for virtual consultation between medical students, patients (case presentations), internal and external examiners. The outcomes were evaluated in terms of conduction of various components of practical examination-Viva, case presentations, instruments, slides, specimen examination. Statistical analysis was performed by descriptive statistics through Microsoft Excel sheet.

Results

Virtual conduction examination/evaluation was performed on 150 medical students by examiners from a distant location. No problems occurred except few short duration (less than 5 min) interruptions due to internet connectivity issues. 125/150 (83.5%) of medical students and all examiners (2 internal and 2 external) expressed satisfaction with virtual medical evaluation.

Conclusions

83.5% of medical students and all examiners expressed satisfaction with virtual medical evaluation during this COVID pandemic. Our findings suggest that virtual conduction of practical annual medical exams through virtual video conferencing platform appears to be an optimal alternative during COVID pandemic.

Conducting a medical examination in the COVID-19 era: an Egyptian experience

Background

On 12 December 2019, a severe respiratory disease was recently reported in Wuhan, known as COVID-19 caused by severe acute respiratory syndrome coronavirus 2. The coronavirus crisis has unequivocally had a marked influence on medical education, particularly in terms of the delivery of assessment. Institutions were forced to implement several changes to medical exam due to COVID-19 pandemic. Written exams were carried out online. Clinical exams were repeatedly canceled or postponed this year. Some institutions carried out clinical exams using remote OSCE stations or without patients. We present our experience in conducting the medical exam in the COVID-19 era.

Main body

Medical exam of the Egyptian Fellowship was canceled in 2020 due to COVID-19 pandemic. The candidates were disappointed and increasingly anxious. After 1 year delay, the internal medicine board decided to carry out the exam after implementing several changes. Changes included the written and clinical exams. The medical exam was successfully conduced in the COVID-19 era.

Conclusion

Conducting a medical exam in the COVID-19 era carries a great challenge for the institutions. Institutions should allow for some degree of flexibility when carrying out exams to prevent suffering of the candidates from the difficult circumstances.

Extra Oral Examination of the Dental Patient

The general assessment and extra oral examination of the dental patient can identify undiagnosed or undisclosed medical conditions, such as malignancies, endocrine conditions and signs indicating raised cardiovascular or cerebral vascular risk. This knowledge may influence dental treatment, and could impact the morbidity and mortality of the patient. This article gives an overview of the general and extra oral examination of the dental patient.

Adolescent arrestees detained in police cells: an observational study in the Paris, France, area

OBJECTIVE

Custody conditions in police cells are often demeaning and considered inappropriate for human beings. The detention of young adolescents in police custody has received little attention. Our study aimed to describe the characteristics of adolescents under 18 detained in custody.

METHODS

We studied all arrestees aged 13-17 examined in 1 year (January 01-December 31, 2014) in a suburban district near Paris. We evaluated the proportion of adolescents under 18 among all arrestees detained in custody and their medical history, addictive behaviors, perceived health status, and opinion on custody.

RESULTS

Arrestees aged 13-17 accounted for 1859 individuals. They were predominantly males (94%) and accounted for 19% of all examinations in custody. Nearly half of the arrestees aged 13-15 (42%), and two thirds of those aged 16-17 (65%) had been previously detained in police cells. Somatic and psychiatric disorders were reported by 7% and 4%, respectively, of arrestees aged 13-17. Alcohol, tobacco, and cannabis consumption were reported by 5%, 24%, and 12%, respectively, of arrestees aged 13-15. These proportions were lower than the 16%, 50%, and 35%, respectively, reported by arrestees aged 16-17 (p < 0.0001). Assaults were reported by 18% of arrestees aged 13-17. They had a fair, bad, or very bad opinion on custody in 43% of cases.

CONCLUSION

The detention of adolescents in police stations is commonly associated with assaults at the time of arrest. High proportions of adolescent arrestees smoke tobacco or cannabis. We suggest that the medical examination in custody could be an opportunity for adolescents to initiate access to health care.

The health of female arrestees in police cells: A descriptive study

INTRODUCTION

Little information is available regarding the medical status and health care needs of female arrestees. Our objective was to evaluate the perceived health and somatic or psychiatric disorders reported by female arrestees in police cells.

MATERIAL AND METHODS

We conducted an observational study in a regional reference department of forensic medicine in France. We studied female arrestees examined in police cells (01/01/2013-06/30/2013). Data were collected regarding individuals' medical characteristics, addictive behaviours, and perceived health status, as well as reported assaults or recent traumatic injuries. We recorded medical decisions regarding fitness for detention in police cells.

RESULTS

A total of 438 women (median age, 29; range, 13-67) accounted for 5% of the 7408 examined arrestees. Females considered their overall health as good or very good in 314/395 cases (70%). Women reported chronic somatic or psychiatric disorders more frequently than men (89/379, 23% vs. 757/6,135, 12%, p < 0.001 and 59/379, 15% vs. 392/6319, 6%, p < 0.001, respectively). Daily tobacco consumption and cannabis use were reported by 255/403 (63%) and 98/438 female arrestees (22%), respectively. Physical assaults were reported in 113/415 cases (27%). Female arrestees were considered fit for detention in 92% of cases. Among 24 pregnant arrestees, 6 (25%) were unfit for detention, 2 (8%) were fit for custody during daytime only and 16 (67%) were fit for detention if certain conditions were met.

CONCLUSIONS

Detention in police custody involves a minority of females. Females are older and report somatic or psychiatric disorders more frequently than males.

The trend and features of physician workforce supply in China: after national medical licensing system reform

BACKGROUND

The annual number of newly licensed physicians is an important indicator of medical workforce supply, which can accurately reflect an inflow into the health care market over a period. In order to both regulate medical professions and improve the quality of health care services, China established its medical licensing system from the point of the implementation of 'Law on Practising Doctors' in 1999. The objective of this study is to depict the trend and structure of newly licensed physicians thereafter.

METHODS

This study analyses a unique census data set that provides the headcount of newly licensed physicians from 2005 to 2015 in China. We also review a short history of medical licensing system reform in China since the 1990s.

RESULTS

The annual number of first-time licensed physicians in China increased from 159 489 in 2005 to 221 639 in 2015. Up to 2015, over half of newly licensed physicians had not received a medical education equivalent to a bachelor degree or higher. Around 51% of China's newly licensed physicians were female in 2005, while the same ratio for females in 2015 was 56%.

CONCLUSION

This article first provides an exploratory analysis of physician inflow into health care market in China using physician licensing data. The medical licensing system in China allows entering physicians with a broad range of educational levels. Moreover, the feminisation of the physician supply in China has become increasingly apparent and its impacts on health care provision still require more rigorous examination.

Social vulnerabilities and health conditions of arrestees in the Greater Paris area, France, in 2013: a multicentre cross-sectional study

OBJECTIVE

The aim of this study was to describe the health and social conditions of arrestees, as compared to the general population.

METHODS

We studied a sample of 600 adult arrestees in three locations in the Greater Paris area, prospectively included (February-May 2013). A descriptive analysis has been performed, then prevalence was estimated using an indirect standardisation according to age, based on data from a population-based, representative survey in the same area.

RESULTS

Arrestees had a median age of 31 years; 92% were males. As compared to the general population, arrestees had a lower level of education (8.6 vs. 7.6%, p < 0.001), were more frequently unemployed (42.0 vs. 11.9%, p < 0.001) and in a difficult financial situation (43 vs. 11%, p < 0.001) and had less significant social support (48.1 vs. 87.9%, p < 0.001). They reported also more frequently a chronic health condition (54 vs. 36%, p < 0.001) and a limited health insurance coverage (36 vs. 15%, p < 0.001).

CONCLUSION

Comparative analysis of male arrestees and males from the general population showed that the former had worse social and health conditions. These results argue for widespread medical interventions on all arrestees. Medical examination during detention could act as a gateway to health care and social support.

Ethics In Health Care Settings: Practices Of Healthcare Professionals And Perceptions Of Patients Regarding Informed Consent, Confidentiality And Privacy At Two Tertiary Care Hospitals Of Islamabad, Pakistan

BACKGROUND

Medical profession works within thin lines of professionalism and trust. Faith of the patients often breached but less reported among the Low and Middle-Income Countries (LMICs). In Pakistan, though the Pakistan Medical and Dental Council (PMDC) have its own code of medical ethics but there isn't much evidence on regulating ethical misconducts. Apart from the gross violations of the code of conduct, the "insensible" misuse of informed consent, confidentiality and privacy is very common. This study is an effort to explore practices of informed consent, confidentiality and privacy among health care providers along with assessment of perceptions of patients about ethical practices in two tertiary care hospitals of Islamabad, Pakistan.

METHODS

We conducted eight homogenous Focus Group Discussions (FGDs), four each in both the public and private sector hospitals till the saturation was achieved.

RESULTS

Informed consent, in clinical practice, was found not being practiced. Confidentiality was not being uniformly applied in practices. Patients perceived the practices being contrary to the ethics. According to patients, ethical measures were found satisfactory in private hospitals. However, patients were not gratified fully with both the systems of healthcare delivery.

CONCLUSIONS

There is insufficient adherence to the ethical principles in clinical practice, in both public and private sector hospitals in Islamabad, Pakistan. Informed consent, privacy and confidentiality are time and time again unheeded due to lack of robust system of monitoring and penalties by the responsible authorities.

Senior driving under the influence: A five-year retrospective study of alcoholized road-users aged 70 and over

The demographic development in Germany shows a steady increase to senior citizens. The driving suitability of older road-users is of large social and political concern, because awareness and reactivity can be influenced by age-related diseases and potential medication, particularly in combination with the consumption of alcohol. This study provides an overview of senior road-users under the influence of alcohol. Therefore, 404 cases of drunken-driving by road-users aged 70 and over within the purview of the Institute of Legal Medicine at Justus-Liebig-University, Giessen, from the years 2009-2013 were evaluated in retrospect. The vast majority of the drivers were male (88.4% of the cases). Distribution of the blood alcohol concentrations were almost equally in male and female drivers with 62.8% of all cases showing a blood alcohol concentration (BAC) of more than 1.1‰ and 10.9% a BAC of more than 2.0‰. In 48.5% of the cases, drunk driving was associated with an accident. 54.5% of the drivers stated suffering from at least one disease and 60% admitted a regular medication or an intake prior to the incident. Moreover, the collected data includes aspects such as the kind of traffic participation as well as neurological and physiological deficits of the road-users. If accidents were caused by drunk driving, the external circumstances and consequences of these accidents were analyzed, too. The evaluation revealed, that the standard medical examination protocol proved to be improbable to cover polypharmacy and multimorbidity of older alcoholized drivers. So, an evaluation and adaptation of the common medical examination protocol must be considered.

Navigation-based tibial rotation at 90° of flexion is associated with better range of motion in navigated total knee arthroplasty

PURPOSE

In clinical practice, people with better femorotibial rotation in the flexed position often achieve a favourable postoperative maximum flexion angle (MFA). However, no objective data have been reported to support this clinical observation. In the present study, we aimed to investigate the correlation between the amount of intraoperative rotation and the pre- and postoperative flexion angles.

METHODS

Fifty-five patients with varus osteoarthritis undergoing computer-assisted posterior-stabilized total knee arthroplasty (TKA) were enrolled. After registration, rotational stress was applied towards the knee joint, and the rotational angles were recorded by using a navigation system at maximum extension and 90° of flexion. After implantation, rotational stress was applied for a second time, and the angles were recorded once more. The MFA was measured before surgery and 1 month after surgery, and the correlation between the amount of femorotibial rotation during surgery and the MFA was statistically evaluated.

RESULTS

Although the amount of tibial rotation at maximum extension was not correlated with the MFA, the amount of tibial rotation at 90° of flexion after registration was positively correlated with the pre- and postoperative MFA (both p < 0.005). However, no significant relationship was observed between the amount of tibial rotation after implantation and the postoperative MFA (n.s.).

CONCLUSION

The results showed that better femorotibial rotation at 90° of flexion is associated with a favourable postoperative MFA, suggesting that the flexibility of the surrounding soft tissues is an important factor for obtaining a better MFA, which has important clinical relevance. Hence, further evaluation of navigation-based kinematics during TKA may provide useful information on MFA.

LEVEL OF EVIDENCE

Diagnostic studies, development of diagnostic criteria in a consecutive series of patients, and a universally applied "gold" standard, Level II.

Self-reported study habits for enhancing medical students' performance in the National Medical Unified Examination

Background:

The National Medical Unified Examination (NMUE) is currently required for graduation, joining postgraduate medical training, and practicing medicine in Syria.

Objective:

To investigate self-reported study habits that correlate with high performance on the NMUE.

Methods:

First through 3^rd year residents at the three main hospitals in Damascus, Syria, were asked to complete a retrospective cross-sectional survey investigating their study habits and previous scores.

Results:

Significantly higher score was associated with >15 study h/day and allocating 1–40% of study time for practicing questions. Mean NMUE score was not significantly different in relation to preparation months for examination or for those who reported spending all their time studying alone compared with spending any amount of time in a group setting. Scores of 231–240 on the Syrian scientific high school exam correlated with significantly higher NMUE performance compared with fewer scores, except scores of 221–230. For every 10 point increase in medical school cumulative grades, the NMUE score increased 3.6 (95% confidence interval 2.5–4.8).

Conclusion:

The NMUE score was significantly affected by hours spent studying per day, number of practice questions completed, percentage of study time allocated for doing questions, Syrian scientific high school exam scores, and the cumulative medical school class grades. It was not significantly affected by preparation months or studying in a group setting. More studies are needed to further describe and investigate the factors that might affect performance in the NMUE.

Detainees arrested for the first time in French police stations

The psychological impact of incarceration vary from individual to individual and most people first entering prison experience severe stress. Our objective was to describe the characteristics of detainees arrested for the first time and who had medical examination during custody. The study sample included 200 detainees arrested for the first time in 2012 and examined for the second time before May 31, 2013, 200 detainees arrested for the first time in 2012 and who had not a second examination by May 31, 2013, and a control group of 200 individuals who had been arrested before. Collected data related to persons' characteristics, the course of detention, alleged assaults and traumatic injuries. In our sample, victimization was the most frequent motivation for requesting a medical examination and affected 31-46% of patients who requested the examination. The medical examination was less frequently requested by the detainee at the first detention than at subsequent detentions (35% and 31% vs. 51%, P < 0.001). Unremarkable psychic states were found in most cases in all groups. Detainees expressed a good or very good opinion on custody in 40-51% of cases. In 75-89% of cases, detainees were considered to be unconditionally fit for detention. The present findings suggest only minor differences between clinical features of individuals arrested for the first time and their clinical status when they were arrested for the second time. The systematic collection of more detailed description of the detainees' psychic state could be relevant at the time of medical examinations in police cells.