ChatGPT: The transformative influence of generative AI on science and healthcare

Evaluating the performance of GPT-3.5, GPT-4, and GPT-4o in the Chinese National Medical Licensing Examination

This study aims to compare and evaluate the performance of GPT-3.5, GPT-4, and GPT-4o in the 2020 and 2021 Chinese National Medical Licensing Examination (NMLE), exploring their potential value in medical education and clinical applications. Six hundred original test questions from the 2020 and 2021 NMLE (covering five types of questions) were selected and input into GPT-3.5, GPT-4, and GPT-4o for response. The accuracy of the models across different question types and units was recorded and analyzed. Statistical methods were employed to compare the performance differences among the three models. GPT-4o demonstrated significantly higher overall accuracy than GPT-4 and GPT-3.5 (P < 0.001). In the 2020 and 2021 exams, GPT-4o achieved accuracy rates of 84.2% and 88.2%, respectively, with the highest accuracy observed in questions related to the digestive system (Unit 3), reaching 94.75%. GPT-4 showed moderate performance, while GPT - 3.5 had the lowest accuracy. Additionally, GPT-4o exhibited a clear advantage in complex question formats, such as case analysis questions (A3/A4 type) and standard matching questions (B1 type). GPT-4o outperformed its predecessors in the NMLE, demonstrating exceptional comprehension and problem-solving abilities in non-English medical examinations. This study provides important insights into the application and promotion of generative AI in medical education and clinical practice.

Revolutionizing Breast Cancer Care: AI-Enhanced Diagnosis and Patient History

Breast cancer poses a significant global health challenge, demanding enhanced diagnostic accuracy and streamlined medical history documentation. This study presents a holistic approach that harnesses the power of artificial intelligence (AI) and machine learning (ML) to address these pressing needs. This study presents a comprehensive methodology for breast cancer diagnosis and medical history generation, integrating data collection, feature extraction, machine learning, and AI-driven history-taking. The research employs a systematic approach to ensure accurate diagnosis and efficient history collection. Data preprocessing merges similar attributes to streamline analysis. Three key algorithms, Support Vector Machine (SVM), K-Nearest Neighbours (KNN), and Fuzzy Logic, are applied. Fuzzy Logic shows exceptional accuracy in handling uncertain data. Deep learning models enhance predictive accuracy, emphasizing the synergy between traditional and deep learning approaches. The AI-driven history collection simplifies the patient history-taking process, adapting questions dynamically based on patient responses. Comprehensive medical history reports summarize patient data, facilitating informed healthcare decisions. The research prioritizes ethical compliance and data privacy. OpenAI has integrated GPT-3.5 to generate automated patient reports, offering structured overviews of patient health history. The study's results indicate the potential for enhanced disease prediction accuracy and streamlined medical history collection, contributing to more reliable healthcare assessments and patient care. Machine learning, deep learning, and AI-driven approaches hold promise for a wide range of applications, particularly in healthcare and beyond.

Application of ChatGPT-based blended medical teaching in clinical education of hepatobiliary surgery

OBJECTIVE

This study evaluates the effectiveness of incorporating the Chat Generative Pre-trained Transformer (ChatGPT) into the clinical teaching of hepatobiliary surgery for undergraduate medical students.

MATERIALS AND METHODS

A group of 61 medical undergraduates from the Affiliated Hospital of Guizhou Medical University, undergoing hepatobiliary surgery training, were randomly assigned to either an experimental group (31 students) using ChatGPT-based blended teaching or a control group (30 students) with traditional teaching methods. The evaluation metrics included final exam scores, teaching satisfaction, and teaching effectiveness ratings, analyzed using SPSS 26.0 (SPSS Inc., Chicago, IL) with t-tests and χ2 tests.

RESULTS

The experimental group significantly outperformed the control group in final exam theoretical scores (86.44 ± 5.59 vs. 77.86 ± 4.16, p < .001) and clinical skills scores (83.84 ± 6.13 vs. 79.12 ± 4.27, p = .001). Additionally, the experimental group reported higher teaching satisfaction (17.23 ± 1.33) and self-evaluation of teaching effectiveness (9.14 ± 0.54) compared to the control group (15.38 ± 1.5 and 8.46 ± 0.70, respectively, p < .001).

CONCLUSIONS

The integration of ChatGPT into hepatobiliary surgery education significantly enhances theoretical knowledge, clinical skills, and overall satisfaction among medical undergraduates, suggesting a beneficial impact on their educational development.

Diagnostic Accuracy and Clinical Value of a Domain-specific Multimodal Generative AI Model for Chest Radiograph Report Generation

Background Generative artificial intelligence (AI) is anticipated to alter radiology workflows, requiring a clinical value assessment for frequent examinations like chest radiograph interpretation. Purpose To develop and evaluate the diagnostic accuracy and clinical value of a domain-specific multimodal generative AI model for providing preliminary interpretations of chest radiographs. Materials and Methods For training, consecutive radiograph-report pairs from frontal chest radiography were retrospectively collected from 42 hospitals (2005-2023). The trained domain-specific AI model generated radiology reports for the radiographs. The test set included public datasets (PadChest, Open-i, VinDr-CXR, and MIMIC-CXR-JPG) and radiographs excluded from training. The sensitivity and specificity of the model-generated reports for 13 radiographic findings, compared with radiologist annotations (reference standard), were calculated (with 95% CIs). Four radiologists evaluated the subjective quality of the reports in terms of acceptability, agreement score, quality score, and comparative ranking of reports from (a) the domain-specific AI model, (b) radiologists, and (c) a general-purpose large language model (GPT-4Vision). Acceptability was defined as whether the radiologist would endorse the report as their own without changes. Agreement scores from 1 (clinically significant discrepancy) to 5 (complete agreement) were assigned using RADPEER; quality scores were on a 5-point Likert scale from 1 (very poor) to 5 (excellent). Results A total of 8 838 719 radiograph-report pairs (training) and 2145 radiographs (testing) were included (anonymized with respect to sex and gender). Reports generated by the domain-specific AI model demonstrated high sensitivity for detecting two critical radiographic findings: 95.3% (181 of 190) for pneumothorax and 92.6% (138 of 149) for subcutaneous emphysema. Acceptance rate, evaluated by four radiologists, was 70.5% (6047 of 8680), 73.3% (6288 of 8580), and 29.6% (2536 of 8580) for model-generated, radiologist, and GPT-4Vision reports, respectively. Agreement scores were highest for the model-generated reports (median = 4 [IQR, 3-5]) and lowest for GPT-4Vision reports (median = 1 [IQR, 1-3]; P < .001). Quality scores were also highest for the model-generated reports (median = 4 [IQR, 3-5]) and lowest for the GPT-4Vision reports (median = 2 [IQR, 1-3]; P < .001). From the ranking analysis, model-generated reports were most frequently ranked the highest (60.0%; 5146 of 8580), and GPT-4Vision reports were most frequently ranked the lowest (73.6%; 6312 of 8580). Conclusion A domain-specific multimodal generative AI model demonstrated potential for high diagnostic accuracy and clinical value in providing preliminary interpretations of chest radiographs for radiologists. © RSNA, 2025 Supplemental material is available for this article. See also the editorial by Little in this issue.

Revolutionizing MASLD: How Artificial Intelligence Is Shaping the Future of Liver Care

Metabolic dysfunction-associated steatotic liver disease (MASLD) is emerging as a leading cause of chronic liver disease. In recent years, artificial intelligence (AI) has attracted significant attention in healthcare, particularly in diagnostics, patient management, and drug development, demonstrating immense potential for application and implementation. In the field of MASLD, substantial research has explored the application of AI in various areas, including patient counseling, improved patient stratification, enhanced diagnostic accuracy, drug development, and prognosis prediction. However, the integration of AI in hepatology is not without challenges. Key issues include data management and privacy, algorithmic bias, and the risk of AI-generated inaccuracies, commonly referred to as "hallucinations". This review aims to provide a comprehensive overview of the applications of AI in hepatology, with a focus on MASLD, highlighting both its transformative potential and its inherent limitations.

Molecular mechanisms in liver repair and regeneration: from physiology to therapeutics

Liver repair and regeneration are crucial physiological responses to hepatic injury and are orchestrated through intricate cellular and molecular networks. This review systematically delineates advancements in the field, emphasizing the essential roles played by diverse liver cell types. Their coordinated actions, supported by complex crosstalk within the liver microenvironment, are pivotal to enhancing regenerative outcomes. Recent molecular investigations have elucidated key signaling pathways involved in liver injury and regeneration. Viewed through the lens of metabolic reprogramming, these pathways highlight how shifts in glucose, lipid, and amino acid metabolism support the cellular functions essential for liver repair and regeneration. An analysis of regenerative variability across pathological states reveals how disease conditions influence these dynamics, guiding the development of novel therapeutic strategies and advanced techniques to enhance liver repair and regeneration. Bridging laboratory findings with practical applications, recent clinical trials highlight the potential of optimizing liver regeneration strategies. These trials offer valuable insights into the effectiveness of novel therapies and underscore significant progress in translational research. In conclusion, this review intricately links molecular insights to therapeutic frontiers, systematically charting the trajectory from fundamental physiological mechanisms to innovative clinical applications in liver repair and regeneration.

Application of large language models in disease diagnosis and treatment

ABSTRACT

Large language models (LLMs) such as ChatGPT, Claude, Llama, and Qwen are emerging as transformative technologies for the diagnosis and treatment of various diseases. With their exceptional long-context reasoning capabilities, LLMs are proficient in clinically relevant tasks, particularly in medical text analysis and interactive dialogue. They can enhance diagnostic accuracy by processing vast amounts of patient data and medical literature and have demonstrated their utility in diagnosing common diseases and facilitating the identification of rare diseases by recognizing subtle patterns in symptoms and test results. Building on their image-recognition abilities, multimodal LLMs (MLLMs) show promising potential for diagnosis based on radiography, chest computed tomography (CT), electrocardiography (ECG), and common pathological images. These models can also assist in treatment planning by suggesting evidence-based interventions and improving clinical decision support systems through integrated analysis of patient records. Despite these promising developments, significant challenges persist regarding the use of LLMs in medicine, including concerns regarding algorithmic bias, the potential for hallucinations, and the need for rigorous clinical validation. Ethical considerations also underscore the importance of maintaining the function of supervision in clinical practice. This paper highlights the rapid advancements in research on the diagnostic and therapeutic applications of LLMs across different medical disciplines and emphasizes the importance of policymaking, ethical supervision, and multidisciplinary collaboration in promoting more effective and safer clinical applications of LLMs. Future directions include the integration of proprietary clinical knowledge, the investigation of open-source and customized models, and the evaluation of real-time effects in clinical diagnosis and treatment practices.

The performance of large language model-powered chatbots compared to oncology physicians on colorectal cancer queries

BACKGROUND

Large language model (LLM)-powered chatbots have become increasingly prevalent in healthcare, while their capacity in oncology remains largely unknown. To evaluate the performance of LLM-powered chatbots compared to oncology physicians in addressing colorectal cancer queries.

METHODS

This study was conducted between August 13, 2023, and January 5, 2024. A total of 150 questions were designed, and each question was submitted three times to eight chatbots: ChatGPT-3.5, ChatGPT-4, ChatGPT-4 Turbo, Doctor GPT, Llama-2-70B, Mixtral-8x7B, Bard, and Claude 2.1. No feedback was provided to these chatbots. The questions were also answered by nine oncology physicians, including three residents, three fellows, and three attendings. Each answer was scored based on its consistency with guidelines, with a score of 1 for consistent answers and 0 for inconsistent answers. The total score for each question was based on the number of corrected answers, ranging from 0 to 3. The accuracy and scores of the chatbots were compared to those of the physicians.

RESULTS

Claude 2.1 demonstrated the highest accuracy, with an average accuracy of 82.67%, followed by Doctor GPT at 80.45%, ChatGPT-4 Turbo at 78.44%, ChatGPT-4 at 78%, Mixtral-8x7B at 73.33%, Bard at 70%, ChatGPT-3.5 at 64.89%, and Llama-2-70B at 61.78%. Claude 2.1 outperformed residents, fellows, and attendings. Doctor GPT outperformed residents and fellows. Additionally, Mixtral-8x7B outperformed residents. In terms of scores, Claude 2.1 outperformed residents and fellows. Doctor GPT, ChatGPT-4 Turbo, and ChatGPT-4 outperformed residents.

CONCLUSIONS

This study shows that LLM-powered chatbots can provide more accurate medical information compared to oncology physicians.

Communicative competence of generative artificial intelligence in responding to patient queries about colorectal cancer surgery

PURPOSE

To examine the ability of generative artificial intelligence (GAI) to answer patients' questions regarding colorectal cancer (CRC).

METHODS

Ten clinically relevant questions about CRC were selected from top-rated hospitals' websites and patient surveys and presented to three GAI tools (Chatbot Generative Pre-Trained Transformer [GPT-4], Google Bard, and CLOVA X). Their responses were compared with answers from the CRC information book. Response evaluation was performed by two groups, each consisting of five healthcare professionals (HCP) and patients. Each question was scored on a 1-5 Likert scale based on four evaluation criteria (maximum score, 20 points/question).

RESULTS

In an analysis including only HCPs, the information book scored 11.8 ± 1.2, GPT-4 scored 13.5 ± 1.1, Google Bard scored 11.5 ± 0.7, and CLOVA X scored 12.2 ± 1.4 (P = 0.001). The score of GPT-4 was significantly higher than those of the information book (P = 0.020) and Google Bard (P = 0.001). In an analysis including only patients, the information book scored 14.1 ± 1.4, GPT-4 scored 15.2 ± 1.8, Google Bard scored 15.5 ± 1.8, and CLOVA X scored 14.4 ± 1.8, without significant differences (P = 0.234). When both groups of evaluators were included, the information book scored 13.0 ± 0.9, GPT-4 scored 14.4 ± 1.2, Google Bard scored 13.5 ± 1.0, and CLOVA X scored 13.3 ± 1.5 (P = 0.070).

CONCLUSION

The three GAIs demonstrated similar or better communicative competence than the information book regarding questions related to CRC surgery in Korean. If high-quality medical information provided by GAI is supervised properly by HCPs and published as an information book, it could be helpful for patients to obtain accurate information and make informed decisions.

AI as a user of AI: Towards responsible autonomy

Recent advancements in Artificial Intelligence (AI), particularly in generative language models and algorithms, have led to significant impacts across diverse domains. AI capabilities to address prompts are growing beyond human capability but we expect AI to perform well also as a prompt engineer. Additionally, AI can serve as a guardian for ethical, security, and other predefined issues related to generated content. We postulate that enforcing dialogues among AI-as-prompt-engineer, AI-as-prompt-responder, and AI-as-Compliance-Guardian can lead to high-quality and responsible solutions. This paper introduces a novel AI collaboration paradigm emphasizing responsible autonomy, with implications for addressing real-world challenges. The paradigm of responsible AI-AI conversation establishes structured interaction patterns, guaranteeing decision-making autonomy. Key implications include enhanced understanding of AI dialogue flow, compliance with rules and regulations, and decision-making scenarios exemplifying responsible autonomy. Real-world applications envision AI systems autonomously addressing complex challenges. We have made preliminary testing of such a paradigm involving instances of ChatGPT autonomously playing various roles in a set of experimental AI-AI conversations and observed evident added value of such a framework.

Evaluation of ChatGPT as a Counselling Tool for Italian-Speaking MASLD Patients: Assessment of Accuracy, Completeness and Comprehensibility

BACKGROUND

Artificial intelligence (AI)-based chatbots have shown promise in providing counseling to patients with metabolic dysfunction-associated steatotic liver disease (MASLD). While ChatGPT3.5 has demonstrated the ability to comprehensively answer MASLD-related questions in English, its accuracy remains suboptimal. Whether language influences these results is unclear. This study aims to assess ChatGPT's performance as a counseling tool for Italian MASLD patients.

METHODS

Thirteen Italian experts rated the accuracy, completeness and comprehensibility of ChatGPT3.5 in answering 15 MASLD-related questions in Italian using a six-point accuracy, three-point completeness and three-point comprehensibility Likert's scale.

RESULTS

Mean scores for accuracy, completeness and comprehensibility were 4.57 ± 0.42, 2.14 ± 0.31 and 2.91 ± 0.07, respectively. The physical activity domain achieved the highest mean scores for accuracy and completeness, whereas the specialist referral domain achieved the lowest. Overall, Fleiss's coefficient of concordance for accuracy, completeness and comprehensibility across all 15 questions was 0.016, 0.075 and -0.010, respectively. Age and academic role of the evaluators did not influence the scores. The results were not significantly different from our previous study focusing on English.

CONCLUSION

Language does not appear to affect ChatGPT's ability to provide comprehensible and complete counseling to MASLD patients, but accuracy remains suboptimal in certain domains.

Systematic analysis of ChatGPT, Google search and Llama 2 for clinical decision support tasks

It is likely that individuals are turning to Large Language Models (LLMs) to seek health advice, much like searching for diagnoses on Google. We evaluate clinical accuracy of GPT-3·5 and GPT-4 for suggesting initial diagnosis, examination steps and treatment of 110 medical cases across diverse clinical disciplines. Moreover, two model configurations of the Llama 2 open source LLMs are assessed in a sub-study. For benchmarking the diagnostic task, we conduct a naïve Google search for comparison. Overall, GPT-4 performed best with superior performances over GPT-3·5 considering diagnosis and examination and superior performance over Google for diagnosis. Except for treatment, better performance on frequent vs rare diseases is evident for all three approaches. The sub-study indicates slightly lower performances for Llama models. In conclusion, the commercial LLMs show growing potential for medical question answering in two successive major releases. However, some weaknesses underscore the need for robust and regulated AI models in health care. Open source LLMs can be a viable option to address specific needs regarding data privacy and transparency of training.