Using ChatGPT for language editing in scientific articles

De novo generation of colorectal patient educational materials using large language models: Prompt engineering key to improved readability

BACKGROUND

Improving patient education has been shown to improve clinical outcomes and reduce disparities, though such efforts can be labor intensive. Large language models may serve as an accessible method to improve patient educational material. The aim of this study was to compare readability between existing educational materials and those generated by large language models.

METHODS

Baseline colorectal surgery educational materials were gathered from a large academic institution (n = 52). Three prompts were entered into Perplexity and ChatGPT 3.5 for each topic: a Basic prompt that simply requested patient educational information the topic, an Iterative prompt that repeated instruction asking for the information to be more health literate, and a Metric-based prompt that requested a sixth-grade reading level, short sentences, and short words. Flesch-Kincaid Grade Level or Grade Level, Flesch-Kincaid Reading Ease or Ease, and Modified Grade Level scores were calculated for all materials, and unpaired t tests were used to compare mean scores between baseline and documents generated by artificial intelligence platforms.

RESULTS

Overall existing materials were longer than materials generated by the large language models across categories and prompts: 863-956 words vs 170-265 (ChatGPT) and 220-313 (Perplexity), all P < .01. Baseline materials did not meet sixth-grade readability guidelines based on grade level (Grade Level 7.0-9.8 and Modified Grade Level 9.6-11.5) or ease of readability (Ease 53.1-65.0). Readability of materials generated by a large language model varied by prompt and platform. Overall, ChatGPT materials were more readable than baseline materials with the Metric-based prompt: Grade Level 5.2 vs 8.1, Modified Grade Level 7.3 vs 10.3, and Ease 70.5 vs 60.4, all P < .01. In contrast, Perplexity-generated materials were significantly less readable except for those generated with the Metric-based prompt, which did not statistically differ.

CONCLUSION

Both existing materials and the majority of educational materials created by large language models did not meet readability recommendations. The exception to this was with ChatGPT materials generated with a Metric-based prompt that consistently improved readability scores from baseline and met recommendations in terms of the average Grade Level score. The variability in performance highlights the importance of the prompt used with large language models.

The undeclared use of third-party service providers in academic publishing is unethical: an epistemic reflection and scoping review

There is a substantial body of scientific literature on the use of third-party services (TPS) by academics to assist as "publication consultants" in scholarly publishing. TPS provide a wide range of scholarly services to research teams that lack the equipment, skills, motivation, or time to produce a paper without external assistance. While services such as language editing, statistical support, or graphic design are common and often legitimate, some TPS also provide illegitimate services and send unsolicited e-mails (spam) to academics offering these services. Such illegitimate types of TPS have the potential to threaten the integrity of the peer-reviewed scientific literature. In extreme cases, for-profit agencies known as "paper mills" even offer fake scientific publications or authorship slots for sale. The use of such illegitimate services as well as the failure to acknowledge their use is an ethical violation in academic publishing, while the failure to declare support for a TPS can be considered a form of contract fraud. We discuss some literature on TPS, highlight services currently offered by ten of the largest commercial publishers and expect authors to be transparent about the use of these services in their publications. From an ethical/moral (i.e., non-commercial) point of view, it is the responsibility of editors, journals, and publishers, and it should be in their best interest to ensure that illegitimate TPS are identified and prohibited, while publisher-employed TPS should be properly disclosed in their publications.

Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles

OBJECTIVE

 To evaluate the reliability of three artificial intelligence (AI) chatbots (ChatGPT, Google Bard, and Chatsonic) in generating accurate references from existing obstetric literature.

STUDY DESIGN

 Between mid-March and late April 2023, ChatGPT, Google Bard, and Chatsonic were prompted to provide references for specific obstetrical randomized controlled trials (RCTs) published in 2020. RCTs were considered for inclusion if they were mentioned in a previous article that primarily evaluated RCTs published by the top medical and obstetrics and gynecology journals with the highest impact factors in 2020 as well as RCTs published in a new journal focused on publishing obstetric RCTs. The selection of the three AI models was based on their popularity, performance in natural language processing, and public availability. Data collection involved prompting the AI chatbots to provide references according to a standardized protocol. The primary evaluation metric was the accuracy of each AI model in correctly citing references, including authors, publication title, journal name, and digital object identifier (DOI). Statistical analysis was performed using a permutation test to compare the performance of the AI models.

RESULTS

 Among the 44 RCTs analyzed, Google Bard demonstrated the highest accuracy, correctly citing 13.6% of the requested RCTs, whereas ChatGPT and Chatsonic exhibited lower accuracy rates of 2.4 and 0%, respectively. Google Bard often substantially outperformed Chatsonic and ChatGPT in correctly citing the studied reference components. The majority of references from all AI models studied were noted to provide DOIs for unrelated studies or DOIs that do not exist.

CONCLUSION

 To ensure the reliability of scientific information being disseminated, authors must exercise caution when utilizing AI for scientific writing and literature search. However, despite their limitations, collaborative partnerships between AI systems and researchers have the potential to drive synergistic advancements, leading to improved patient care and outcomes.

KEY POINTS

· AI chatbots often cite scientific articles incorrectly.. · AI chatbots can create false references.. · Responsible AI use in research is vital..

ChatGPT in medical writing: A game-changer or a gimmick?

OpenAI's ChatGPT (Generative Pre-trained Transformer) is a chatbot that answers questions and performs writing tasks in a conversational tone. Within months of release, multiple sectors are contemplating the varied applications of this chatbot, including medicine, education, and research, all of which are involved in medical communication and scientific publishing. Medical writers and academics use several artificial intelligence (AI) tools and software for research, literature survey, data analyses, referencing, and writing. There are benefits of using different AI tools in medical writing. However, using chatbots for medical communications pose some major concerns such as potential inaccuracies, data bias, security, and ethical issues. Perceived incorrect notions also limit their use. Moreover, ChatGPT can also be challenging if used incorrectly and for irrelevant tasks. If used appropriately, ChatGPT will not only upgrade the knowledge of the medical writer but also save time and energy that could be directed toward more creative and analytical areas requiring expert skill sets. This review introduces chatbots, outlines the progress in ChatGPT research, elaborates the potential uses of ChatGPT in medical communications along with its challenges and limitations, and proposes future research perspectives. It aims to provide guidance for doctors, researchers, and medical writers on the uses of ChatGPT in medical communications.

ChatGPT revisited: Using ChatGPT-4 for finding references and editing language in medical scientific articles

The attainment of academic superiority relies heavily upon the accessibility of scholarly resources and the expression of research findings through faultless language usage. Although modern tools, such as the Publish or Perish software program, are proficient in sourcing academic papers based on specific keywords, they often fall short of extracting comprehensive content, including crucial references. The challenge of linguistic precision remains a prominent issue, particularly for research papers composed by non-native English speakers who may encounter word usage errors. This manuscript serves a twofold purpose: firstly, it reassesses the effectiveness of ChatGPT-4 in the context of retrieving pertinent references tailored to specific research topics. Secondly, it introduces a suite of language editing services that are skilled in rectifying word usage errors, ensuring the refined presentation of research outcomes. The article also provides practical guidelines for formulating precise queries to mitigate the risks of erroneous language usage and the inclusion of spurious references. In the ever-evolving realm of academic discourse, leveraging the potential of advanced AI, such as ChatGPT-4, can significantly enhance the quality and impact of scientific publications.

Human-Comparable Sensitivity of Large Language Models in Identifying Eligible Studies Through Title and Abstract Screening: 3-Layer Strategy Using GPT-3.5 and GPT-4 for Systematic Reviews

BACKGROUND

The screening process for systematic reviews is resource-intensive. Although previous machine learning solutions have reported reductions in workload, they risked excluding relevant papers.

OBJECTIVE

We evaluated the performance of a 3-layer screening method using GPT-3.5 and GPT-4 to streamline the title and abstract-screening process for systematic reviews. Our goal is to develop a screening method that maximizes sensitivity for identifying relevant records.

METHODS

We conducted screenings on 2 of our previous systematic reviews related to the treatment of bipolar disorder, with 1381 records from the first review and 3146 from the second. Screenings were conducted using GPT-3.5 (gpt-3.5-turbo-0125) and GPT-4 (gpt-4-0125-preview) across three layers: (1) research design, (2) target patients, and (3) interventions and controls. The 3-layer screening was conducted using prompts tailored to each study. During this process, information extraction according to each study's inclusion criteria and optimization for screening were carried out using a GPT-4-based flow without manual adjustments. Records were evaluated at each layer, and those meeting the inclusion criteria at all layers were subsequently judged as included.

RESULTS

On each layer, both GPT-3.5 and GPT-4 were able to process about 110 records per minute, and the total time required for screening the first and second studies was approximately 1 hour and 2 hours, respectively. In the first study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.900/0.709 and 0.806/0.996, respectively. Both screenings by GPT-3.5 and GPT-4 judged all 6 records used for the meta-analysis as included. In the second study, the sensitivities/specificities of the GPT-3.5 and GPT-4 were 0.958/0.116 and 0.875/0.855, respectively. The sensitivities for the relevant records align with those of human evaluators: 0.867-1.000 for the first study and 0.776-0.979 for the second study. Both screenings by GPT-3.5 and GPT-4 judged all 9 records used for the meta-analysis as included. After accounting for justifiably excluded records by GPT-4, the sensitivities/specificities of the GPT-4 screening were 0.962/0.996 in the first study and 0.943/0.855 in the second study. Further investigation indicated that the cases incorrectly excluded by GPT-3.5 were due to a lack of domain knowledge, while the cases incorrectly excluded by GPT-4 were due to misinterpretations of the inclusion criteria.

CONCLUSIONS

Our 3-layer screening method with GPT-4 demonstrated acceptable level of sensitivity and specificity that supports its practical application in systematic review screenings. Future research should aim to generalize this approach and explore its effectiveness in diverse settings, both medical and nonmedical, to fully establish its use and operational feasibility.

ChatGPT in medicine: A cross-disciplinary systematic review of ChatGPT's (artificial intelligence) role in research, clinical practice, education, and patient interaction

BACKGROUND

ChatGPT, a powerful AI language model, has gained increasing prominence in medicine, offering potential applications in healthcare, clinical decision support, patient communication, and medical research. This systematic review aims to comprehensively assess the applications of ChatGPT in healthcare education, research, writing, patient communication, and practice while also delineating potential limitations and areas for improvement.

METHOD

Our comprehensive database search retrieved relevant papers from PubMed, Medline and Scopus. After the screening process, 83 studies met the inclusion criteria. This review includes original studies comprising case reports, analytical studies, and editorials with original findings.

RESULT

ChatGPT is useful for scientific research and academic writing, and assists with grammar, clarity, and coherence. This helps non-English speakers and improves accessibility by breaking down linguistic barriers. However, its limitations include probable inaccuracy and ethical issues, such as bias and plagiarism. ChatGPT streamlines workflows and offers diagnostic and educational potential in healthcare but exhibits biases and lacks emotional sensitivity. It is useful in inpatient communication, but requires up-to-date data and faces concerns about the accuracy of information and hallucinatory responses.

CONCLUSION

Given the potential for ChatGPT to transform healthcare education, research, and practice, it is essential to approach its adoption in these areas with caution due to its inherent limitations.

Comparison of Medical Research Abstracts Written by Surgical Trainees and Senior Surgeons or Generated by Large Language Models

Importance

Artificial intelligence (AI) has permeated academia, especially OpenAI Chat Generative Pretrained Transformer (ChatGPT), a large language model. However, little has been reported on its use in medical research.

Objective

To assess a chatbot's capability to generate and grade medical research abstracts.

Design, Setting, and Participants

In this cross-sectional study, ChatGPT versions 3.5 and 4.0 (referred to as chatbot 1 and chatbot 2) were coached to generate 10 abstracts by providing background literature, prompts, analyzed data for each topic, and 10 previously presented, unassociated abstracts to serve as models. The study was conducted between August 2023 and February 2024 (including data analysis).

Exposure

Abstract versions utilizing the same topic and data were written by a surgical trainee or a senior physician or generated by chatbot 1 and chatbot 2 for comparison. The 10 training abstracts were written by 8 surgical residents or fellows, edited by the same senior surgeon, at a high-volume hospital in the Southeastern US with an emphasis on outcomes-based research. Abstract comparison was then based on 10 abstracts written by 5 surgical trainees within the first 6 months of their research year, edited by the same senior author.

Main Outcomes and Measures

The primary outcome measurements were the abstract grades using 10- and 20-point scales and ranks (first to fourth). Abstract versions by chatbot 1, chatbot 2, junior residents, and the senior author were compared and judged by blinded surgeon-reviewers as well as both chatbot models. Five academic attending surgeons from Denmark, the UK, and the US, with extensive experience in surgical organizations, research, and abstract evaluation served as reviewers.

Results

Surgeon-reviewers were unable to differentiate between abstract versions. Each reviewer ranked an AI-generated version first at least once. Abstracts demonstrated no difference in their median (IQR) 10-point scores (resident, 7.0 [6.0-8.0]; senior author, 7.0 [6.0-8.0]; chatbot 1, 7.0 [6.0-8.0]; chatbot 2, 7.0 [6.0-8.0]; P = .61), 20-point scores (resident, 14.0 [12.0-7.0]; senior author, 15.0 [13.0-17.0]; chatbot 1, 14.0 [12.0-16.0]; chatbot 2, 14.0 [13.0-16.0]; P = .50), or rank (resident, 3.0 [1.0-4.0]; senior author, 2.0 [1.0-4.0]; chatbot 1, 3.0 [2.0-4.0]; chatbot 2, 2.0 [1.0-3.0]; P = .14). The abstract grades given by chatbot 1 were comparable to the surgeon-reviewers' grades. However, chatbot 2 graded more favorably than the surgeon-reviewers and chatbot 1. Median (IQR) chatbot 2-reviewer grades were higher than surgeon-reviewer grades of all 4 abstract versions (resident, 14.0 [12.0-17.0] vs 16.9 [16.0-17.5]; P = .02; senior author, 15.0 [13.0-17.0] vs 17.0 [16.5-18.0]; P = .03; chatbot 1, 14.0 [12.0-16.0] vs 17.8 [17.5-18.5]; P = .002; chatbot 2, 14.0 [13.0-16.0] vs 16.8 [14.5-18.0]; P = .04). When comparing the grades of the 2 chatbots, chatbot 2 gave higher median (IQR) grades for abstracts than chatbot 1 (resident, 14.0 [13.0-15.0] vs 16.9 [16.0-17.5]; P = .003; senior author, 13.5 [13.0-15.5] vs 17.0 [16.5-18.0]; P = .004; chatbot 1, 14.5 [13.0-15.0] vs 17.8 [17.5-18.5]; P = .003; chatbot 2, 14.0 [13.0-15.0] vs 16.8 [14.5-18.0]; P = .01).

Conclusions and Relevance

In this cross-sectional study, trained chatbots generated convincing medical abstracts, undifferentiable from resident or senior author drafts. Chatbot 1 graded abstracts similarly to surgeon-reviewers, while chatbot 2 was less stringent. These findings may assist surgeon-scientists in successfully implementing AI in medical research.

Utility and Comparative Performance of Current Artificial Intelligence Large Language Models as Postoperative Medical Support Chatbots in Aesthetic Surgery

BACKGROUND

Large language models (LLMs) have revolutionized the way plastic surgeons and their patients can access and leverage artificial intelligence (AI).

OBJECTIVES

The present study aims to compare the performance of 2 current publicly available and patient-accessible LLMs in the potential application of AI as postoperative medical support chatbots in an aesthetic surgeon's practice.

METHODS

Twenty-two simulated postoperative patient presentations following aesthetic breast plastic surgery were devised and expert-validated. Complications varied in their latency within the postoperative period, as well as urgency of required medical attention. In response to each patient-reported presentation, Open AI's ChatGPT and Google's Bard, in their unmodified and freely available versions, were objectively assessed for their comparative accuracy in generating an appropriate differential diagnosis, most-likely diagnosis, suggested medical disposition, treatments or interventions to begin from home, and/or red flag signs/symptoms indicating deterioration.

RESULTS

ChatGPT cumulatively and significantly outperformed Bard across all objective assessment metrics examined (66% vs 55%, respectively; P < .05). Accuracy in generating an appropriate differential diagnosis was 61% for ChatGPT vs 57% for Bard (P = .45). ChatGPT asked an average of 9.2 questions on history vs Bard's 6.8 questions (P < .001), with accuracies of 91% vs 68% reporting the most-likely diagnosis, respectively (P < .01). Appropriate medical dispositions were suggested with accuracies of 50% by ChatGPT vs 41% by Bard (P = .40); appropriate home interventions/treatments with accuracies of 59% vs 55% (P = .94), and red flag signs/symptoms with accuracies of 79% vs 54% (P < .01), respectively. Detailed and comparative performance breakdowns according to complication latency and urgency are presented.

CONCLUSIONS

ChatGPT represents the superior LLM for the potential application of AI technology in postoperative medical support chatbots. Imperfect performance and limitations discussed may guide the necessary refinement to facilitate adoption.

Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses

Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.

ChatGPT: What Every Pediatric Surgeon Should Know About Its Potential Uses and Pitfalls

ChatGPT - currently the most popular generative artificial intelligence system - has been revolutionizing the world and healthcare since its release in November 2022. ChatGPT is a conversational chatbot that uses machine learning algorithms to enhance its replies based on user interactions and is a part of a broader effort to develop natural language processing that can assist people in their daily lives by understanding and responding to human language in a useful and engaging way. Thus far, many potential applications within healthcare have been described, despite its relatively recent release. This manuscript offers the pediatric surgical community a primer on this new technology and discusses some initial observations about its potential uses and pitfalls. Moreover, it introduces the perspectives of medical journals and surgical societies regarding the use of this artificial intelligence chatbot. As ChatGPT and other large language models continue to evolve, it is the responsibility of the pediatric surgery community to stay abreast of these changes and play an active role in safely incorporating them into our field for the benefit of our patients. LEVEL OF EVIDENCE: V.

Evaluating insomnia queries from an artificial intelligence chatbot for patient education

STUDY OBJECTIVES

We evaluated the accuracy of ChatGPT in addressing insomnia-related queries for patient education and assessed ChatGPT's ability to provide varied responses based on differing prompting scenarios.

METHODS

Four identical sets of 20 insomnia-related queries were posed to ChatGPT. Each set differed by the context in which ChatGPT was prompted: no prompt, patient-centered, physician-centered, and with references and statistics. Responses were reviewed by 2 academic sleep surgeons, 1 academic sleep medicine physician, and 2 sleep medicine fellows across 4 domains: clinical accuracy, prompt adherence, referencing, and statistical precision, using a binary grading system. Flesch-Kincaid grade-level scores were calculated to estimate the grade level of the responses, with statistical differences between prompts analyzed via analysis of variance and Tukey's test. Interrater reliability was calculated using Fleiss's kappa.

RESULTS

The study revealed significant variations in the Flesch-Kincaid grade-level scores across 4 prompts: unprompted (13.2 ± 2.2), patient-centered (8.1 ± 1.9), physician-centered (15.4 ± 2.8), and with references and statistics (17.3 ± 2.3, P < .001). Despite poor Fleiss kappa scores, indicating low interrater reliability for clinical accuracy and relevance, all evaluators agreed that the majority of ChatGPT's responses were clinically accurate, with the highest variability on Form 4. The responses were also uniformly relevant to the given prompts (100% agreement). Eighty percent of the references ChatGPT cited were verified as both real and relevant, and only 25% of cited statistics were corroborated within referenced articles.

CONCLUSIONS

ChatGPT can be used to generate clinically accurate responses to insomnia-related inquiries.

CITATION

Alapati R, Campbell D, Molin N, et al. Evaluating insomnia queries from an artificial intelligence chatbot for patient education. J Clin Sleep Med. 2024;20(4):583-594.

Large Language Models and Artificial Intelligence: A Primer for Plastic Surgeons on the Demonstrated and Potential Applications, Promises, and Limitations of ChatGPT

BACKGROUND

The rapidly evolving field of artificial intelligence (AI) holds great potential for plastic surgeons. ChatGPT, a recently released AI large language model (LLM), promises applications across many disciplines, including healthcare.

OBJECTIVES

The aim of this article was to provide a primer for plastic surgeons on AI, LLM, and ChatGPT, including an analysis of current demonstrated and proposed clinical applications.

METHODS

A systematic review was performed identifying medical and surgical literature on ChatGPT's proposed clinical applications. Variables assessed included applications investigated, command tasks provided, user input information, AI-emulated human skills, output validation, and reported limitations.

RESULTS

The analysis included 175 articles reporting on 13 plastic surgery applications and 116 additional clinical applications, categorized by field and purpose. Thirty-four applications within plastic surgery are thus proposed, with relevance to different target audiences, including attending plastic surgeons (n = 17, 50%), trainees/educators (n = 8, 24.0%), researchers/scholars (n = 7, 21%), and patients (n = 2, 6%). The 15 identified limitations of ChatGPT were categorized by training data, algorithm, and ethical considerations.

CONCLUSIONS

Widespread use of ChatGPT in plastic surgery will depend on rigorous research of proposed applications to validate performance and address limitations. This systemic review aims to guide research, development, and regulation to safely adopt AI in plastic surgery.

Artificial Intelligence-Generated Scientific Literature: A Critical Appraisal

BACKGROUND

Review articles play a critical role in informing medical decisions and identifying avenues for future research. With the introduction of artificial intelligence (AI), there has been a growing interest in the potential of this technology to transform the synthesis of medical literature. Open AI's Generative Pre-trained Transformer (GPT-4) (Open AI Inc, San Francisco, CA) tool provides access to advanced AI that is able to quickly produce medical literature following only simple prompts. The accuracy of the generated articles requires review, especially in subspecialty fields like Allergy/Immunology.

OBJECTIVE

To critically appraise AI-synthesized allergy-focused minireviews.

METHODS

We tasked the GPT-4 Chatbot with generating 2 1,000-word reviews on the topics of hereditary angioedema and eosinophilic esophagitis. Authors critically appraised these articles using the Joanna Briggs Institute (JBI) tool for text and opinion and additionally evaluated domains of interest such as language, reference quality, and accuracy of the content.

RESULTS

The language of the AI-generated minireviews was carefully articulated and logically focused on the topic of interest; however, reviewers of the AI-generated articles indicated that the AI-generated content lacked depth, did not appear to be the result of an analytical process, missed critical information, and contained inaccurate information. Despite being provided instruction to utilize scientific references, the AI chatbot relied mainly on freely available resources, and the AI chatbot fabricated references.

CONCLUSIONS

The AI holds the potential to change the landscape of synthesizing medical literature; however, apparent inaccurate and fabricated information calls for rigorous evaluation and validation of AI tools in generating medical literature, especially on subjects associated with limited resources.

Can ChatGPT-3.5 Pass a Medical Exam? A Systematic Review of ChatGPT's Performance in Academic Testing

OBJECTIVE

We, therefore, aim to conduct a systematic review to assess the academic potential of ChatGPT-3.5, along with its strengths and limitations when giving medical exams.

METHOD

Following PRISMA guidelines, a systemic search of the literature was performed using electronic databases PUBMED/MEDLINE, Google Scholar, and Cochrane. Articles from their inception till April 4, 2023, were queried. A formal narrative analysis was conducted by systematically arranging similarities and differences between individual findings together.

RESULTS

After rigorous screening, 12 articles underwent this review. All the selected papers assessed the academic performance of ChatGPT-3.5. One study compared the performance of ChatGPT-3.5 with the performance of ChatGPT-4 when giving a medical exam. Overall, ChatGPT performed well in 4 tests, averaged in 4 tests, and performed badly in 4 tests. ChatGPT's performance was directly proportional to the level of the questions' difficulty but was unremarkable on whether the questions were binary, descriptive, or MCQ-based. ChatGPT's explanation, reasoning, memory, and accuracy were remarkably good, whereas it failed to understand image-based questions, and lacked insight and critical thinking.

CONCLUSION

ChatGPT-3.5 performed satisfactorily in the exams it took as an examinee. However, there is a need for future related studies to fully explore the potential of ChatGPT in medical education.

The impact and opportunities of large language models like ChatGPT in oral and maxillofacial surgery: a narrative review

Since its release at the end of 2022, the social response to ChatGPT, a large language model (LLM), has been huge, as it has revolutionized the way we communicate with computers. This review was performed to describe the technical background of LLMs and to provide a review of the current literature on LLMs in the field of oral and maxillofacial surgery (OMS). The PubMed, Scopus, and Web of Science databases were searched for LLMs and OMS. Adjacent surgical disciplines were included to cover the entire literature, and records from Google Scholar and medRxiv were added. Out of the 57 records identified, 37 were included; 31 (84%) were related to GPT-3.5, four (11%) to GPT-4, and two (5%) to both. Current research on LLMs is mainly limited to research and scientific writing, patient information/communication, and medical education. Classic OMS diseases are underrepresented. The current literature related to LLMs in OMS has a limited evidence level. There is a need to investigate the use of LLMs scientifically and systematically in the core areas of OMS. Although LLMs are likely to add value outside the operating room, the use of LLMs raises ethical and medical regulatory issues that must first be addressed.

Medical Student Experiences and Perceptions of ChatGPT and Artificial Intelligence: Cross-Sectional Study

BACKGROUND

Artificial intelligence (AI) has the potential to revolutionize the way medicine is learned, taught, and practiced, and medical education must prepare learners for these inevitable changes. Academic medicine has, however, been slow to embrace recent AI advances. Since its launch in November 2022, ChatGPT has emerged as a fast and user-friendly large language model that can assist health care professionals, medical educators, students, trainees, and patients. While many studies focus on the technology's capabilities, potential, and risks, there is a gap in studying the perspective of end users.

OBJECTIVE

The aim of this study was to gauge the experiences and perspectives of graduating medical students on ChatGPT and AI in their training and future careers.

METHODS

A cross-sectional web-based survey of recently graduated medical students was conducted in an international academic medical center between May 5, 2023, and June 13, 2023. Descriptive statistics were used to tabulate variable frequencies.

RESULTS

Of 325 applicants to the residency programs, 265 completed the survey (an 81.5% response rate). The vast majority of respondents denied using ChatGPT in medical school, with 20.4% (n=54) using it to help complete written assessments and only 9.4% using the technology in their clinical work (n=25). More students planned to use it during residency, primarily for exploring new medical topics and research (n=168, 63.4%) and exam preparation (n=151, 57%). Male students were significantly more likely to believe that AI will improve diagnostic accuracy (n=47, 51.7% vs n=69, 39.7%; P=.001), reduce medical error (n=53, 58.2% vs n=71, 40.8%; P=.002), and improve patient care (n=60, 65.9% vs n=95, 54.6%; P=.007). Previous experience with AI was significantly associated with positive AI perception in terms of improving patient care, decreasing medical errors and misdiagnoses, and increasing the accuracy of diagnoses (P=.001, P<.001, P=.008, respectively).

CONCLUSIONS

The surveyed medical students had minimal formal and informal experience with AI tools and limited perceptions of the potential uses of AI in health care but had overall positive views of ChatGPT and AI and were optimistic about the future of AI in medical education and health care. Structured curricula and formal policies and guidelines are needed to adequately prepare medical learners for the forthcoming integration of AI in medicine.

Will ChatGPT's Free Language Editing Service Level the Playing Field in Science Communication?: Insights from a Collaborative Project with Non-native English Scholars

ChatGPT has been widely heralded as a way to level the playing field in scientific communication through its free language editing service. However, such claims lack systematic evidence. A writing scholar (LL) and six non-native English scholars researching health professions education collaborated on this Writer's Craft to fill this gap. Our overarching aim was to provide experiential evidence about ChatGPT's performance as a language editor and writing coach. We implemented three cycles of a systematic procedure, describing how we developed our prompts, selected text for editing, incrementally prompted to refine ChatGPT's responses, and analyzed the quality of its language edits and explanations. From this experience, we offer five insights, and we conclude that the optimism about ChatGPT's capacity to level the playing field for non-native English writers should be tempered. In the writer's craft section we offer simple tips to improve your writing in one of three areas: Energy, Clarity and Persuasiveness. Each entry focuses on a key writing feature or strategy, illustrates how it commonly goes wrong, teaches the grammatical underpinnings necessary to understand it and offers suggestions to wield it effectively. We encourage readers to share comments on or suggestions for this section on Twitter, using the hashtag: #how'syourwriting?

A Cross Sectional Study of ChatGPT in Translation: Magnitude of Use, Attitudes, and Uncertainties

This preliminary cross-sectional study, focusing on Artificial Intelligence (AI), aimed to assess the impact of ChatGPT on translation within an Arab context. It primarily explored the attitudes of a sample of translation teachers and students through semi-structured interviews and projective techniques. Data collection included gathering information about the advantages and challenges that ChatGPT, in comparison to Google Translate, had introduced to the field of translation and translation teaching. The results indicated that nearly all the participants were satisfied with ChatGPT. The results also revealed that most students preferred ChatGPT over Google Translate, while most teachers favored Google Translate. The study also found that the participants recognized both positive and negative aspects of using ChatGPT in translation. Findings also indicated that ChatGPT, as a recent AI-based translation-related technology, is more valuable for mechanical processes of writing and editing translated texts than for tasks requiring judgment, such as fine-tuning and double-checking. While it offers various advantages, AI also presents new challenges that educators and stakeholders need to address accordingly.

ChatGPT: a tool for scientific writing or a threat to integrity?

The use of ChatGPT as a tool for writing and knowledge integration raises concerns about the potential for its use to replace critical thinking and academic writing skills. While ChatGPT can assist in generating text and suggesting appropriate language, it should not replace the human responsibility for creating innovative knowledge through experiential learning. The accuracy and quality of information provided by ChatGPT also require caution, as previous studies have reported inaccuracies in references used by chatbots. ChatGPT acknowledges certain limitations, including the potential for generating erroneous or biased content, and it is essential to exercise caution in interpreting its responses and recognize the indispensable role of human experience in the processes of information retrieval and knowledge creation. Furthermore, the challenge of distinguishing between papers written by humans or AI highlights the need for thorough review processes to prevent the spread of articles that could lead to the loss of confidence in the accuracy and integrity of scientific research. Overall, while the use of ChatGPT can be helpful, it is crucial to raise awareness of the potential issues associated with the use of ChatGPT, as well as to discuss boundaries so that AI can be used without compromising the quality of scientific articles and the integrity of evidence-based knowledge.

Accelerating medical education with ChatGPT: an implementation guide

Chatbots powered by artificial intelligence have revolutionized many industries and fields of study, including medical education. Medical educators are increasingly asked to perform more administrative, written, and assessment functions with less time and resources. Safe use of chatbots, like ChatGPT, can help medical educators efficiently perform these functions. In this article, we provide medical educators with tips for the implementation of ChatGPT in medical education. Through creativity and careful construction of prompts, medical educators can use these and other implementations of chatbots, like ChatGPT, in their practice.

Artificial intelligence in language instruction: impact on English learning achievement, L2 motivation, and self-regulated learning

Introduction

This mixed methods study examines the effects of AI-mediated language instruction on English learning achievement, L2 motivation, and self-regulated learning among English as a Foreign Language (EFL) learners. It addresses the increasing interest in AI-driven educational technologies and their potential to revolutionize language instruction.

Methods

Two intact classes, consisting of a total of 60 university students, participated in this study. The experimental group received AI-mediated instruction, while the control group received traditional language instruction. Pre-tests and post-tests were administered to evaluate English learning achievement across various domains, including grammar, vocabulary, reading comprehension, and writing skills. Additionally, self-report questionnaires were employed to assess L2 motivation and self-regulated learning.

Results

Quantitative analysis revealed that the experimental group achieved significantly higher English learning outcomes in all assessed areas compared to the control group. Furthermore, they exhibited greater L2 motivation and more extensive utilization of self-regulated learning strategies. These results suggest that AI-mediated instruction positively impacts English learning achievement, L2 motivation, and self-regulated learning.

Discussion

Qualitative analysis of semi-structured interviews with 14 students from the experimental group shed light on the transformative effects of the AI platform. It was found to enhance engagement and offer personalized learning experiences, ultimately boosting motivation and fostering self-regulated learning. These findings emphasize the potential of AI-mediated language instruction to improve language learning outcomes, motivate learners, and promote autonomy.

Conclusion

This study contributes to evidence-based language pedagogy, offering valuable insights to educators and researchers interested in incorporating AI-powered platforms into language classrooms. The results support the notion that AI-mediated language instruction holds promise in revolutionizing language learning, and it highlights the positive impact of AI-driven educational technologies in the realm of language education.

A Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis of ChatGPT Integration in Nursing Education: A Narrative Review

Amidst evolving healthcare demands, nursing education plays a pivotal role in preparing future nurses for complex challenges. Traditional approaches, however, must be revised to meet modern healthcare needs. The ChatGPT, an AI-based chatbot, has garnered significant attention due to its ability to personalize learning experiences, enhance virtual clinical simulations, and foster collaborative learning in nursing education. This review aims to thoroughly assess the potential impact of integrating ChatGPT into nursing education. The hypothesis is that valuable insights can be provided for stakeholders through a comprehensive SWOT analysis examining the strengths, weaknesses, opportunities, and threats associated with ChatGPT. This will enable informed decisions about its integration, prioritizing improved learning outcomes. A thorough narrative literature review was undertaken to provide a solid foundation for the SWOT analysis. The materials included scholarly articles and reports, which ensure the study's credibility and allow for a holistic and unbiased assessment. The analysis identified accessibility, consistency, adaptability, cost-effectiveness, and staying up-to-date as crucial factors influencing the strengths, weaknesses, opportunities, and threats associated with ChatGPT integration in nursing education. These themes provided a framework to understand the potential risks and benefits of integrating ChatGPT into nursing education. This review highlights the importance of responsible and effective use of ChatGPT in nursing education and the need for collaboration among educators, policymakers, and AI developers. Addressing the identified challenges and leveraging the strengths of ChatGPT can lead to improved learning outcomes and enriched educational experiences for students. The findings emphasize the importance of responsibly integrating ChatGPT in nursing education, balancing technological advancement with careful consideration of associated risks, to achieve optimal outcomes.

Comprehensive Review of Natural Language Processing (NLP) in Vascular Surgery

Objective

The use of Natural Language Processing (NLP) has attracted increased interest in healthcare with various potential applications including identification and extraction of health information, development of chatbots and virtual assistants. The aim of this comprehensive literature review was to provide an overview of NLP applications in vascular surgery, identify current limitations, and discuss future perspectives in the field.

Data sources

The MEDLINE database was searched on April 2023.

Review methods

The database was searched using a combination of keywords to identify studies reporting the use of NLP and chatbots in three main vascular diseases. Keywords used included Natural Language Processing, chatbot, chatGPT, aortic disease, carotid, peripheral artery disease, vascular, and vascular surgery.

Results

Given the heterogeneity of study design, techniques, and aims, a comprehensive literature review was performed to provide an overview of NLP applications in vascular surgery. By enabling identification and extraction of information on patients with vascular diseases, such technology could help to analyse data from healthcare information systems to provide feedback on current practice and help in optimising patient care. In addition, chatbots and NLP driven techniques have the potential to be used as virtual assistants for both health professionals and patients.

Conclusion

While Artificial Intelligence and NLP technology could be used to enhance care for patients with vascular diseases, many challenges remain including the need to define guidelines and clear consensus on how to evaluate and validate these innovations before their implementation into clinical practice.

Exploring the role of ChatGPT in patient care (diagnosis and treatment) and medical research: A systematic review

BACKGROUND

ChatGPT is an artificial intelligence based tool developed by OpenAI (California, USA). This systematic review examines the potential of ChatGPT in patient care and its role in medical research.

METHODS

The systematic review was done according to the PRISMA guidelines. Embase, Scopus, PubMed and Google Scholar data bases were searched. We also searched preprint data bases. Our search was aimed to identify all kinds of publications, without any restrictions, on ChatGPT and its application in medical research, medical publishing and patient care. We used search term "ChatGPT". We reviewed all kinds of publications including original articles, reviews, editorial/ commentaries, and even letter to the editor. Each selected records were analysed using ChatGPT and responses generated were compiled in a table. The word table was transformed in to a PDF and was further analysed using ChatPDF.

RESULTS

We reviewed full texts of 118 articles. ChatGPT can assist with patient enquiries, note writing, decision-making, trial enrolment, data management, decision support, research support, and patient education. But the solutions it offers are usually insufficient and contradictory, raising questions about their originality, privacy, correctness, bias, and legality. Due to its lack of human-like qualities, ChatGPT's legitimacy as an author is questioned when used for academic writing. ChatGPT generated contents have concerns with bias and possible plagiarism.

CONCLUSION

Although it can help with patient treatment and research, there are issues with accuracy, authorship, and bias. ChatGPT can serve as a "clinical assistant" and be a help in research and scholarly writing.

Fabrication and errors in the bibliographic citations generated by ChatGPT

Although chatbots such as ChatGPT can facilitate cost-effective text generation and editing, factually incorrect responses (hallucinations) limit their utility. This study evaluates one particular type of hallucination: fabricated bibliographic citations that do not represent actual scholarly works. We used ChatGPT-3.5 and ChatGPT-4 to produce short literature reviews on 42 multidisciplinary topics, compiling data on the 636 bibliographic citations (references) found in the 84 papers. We then searched multiple databases and websites to determine the prevalence of fabricated citations, to identify errors in the citations to non-fabricated papers, and to evaluate adherence to APA citation format. Within this set of documents, 55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated. Likewise, 43% of the real (non-fabricated) GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors. Although GPT-4 is a major improvement over GPT-3.5, problems remain.

Expanding horizons and navigating challenges for enhanced clinical workflows: ChatGPT in urology

Purpose of review

ChatGPT has emerged as a potential tool for facilitating doctors' workflows. However, when it comes to applying these findings within a urological context, there have not been many studies. Thus, our objective was rooted in analyzing the pros and cons of ChatGPT use and how it can be exploited and used by urologists.

Recent findings

ChatGPT can facilitate clinical documentation and note-taking, patient communication and support, medical education, and research. In urology, it was proven that ChatGPT has the potential as a virtual healthcare aide for benign prostatic hyperplasia, an educational and prevention tool on prostate cancer, educational support for urological residents, and as an assistant in writing urological papers and academic work. However, several concerns about its exploitation are presented, such as lack of web crawling, risk of accidental plagiarism, and concerns about patients-data privacy.

Summary

The existing limitations mediate the need for further improvement of ChatGPT, such as ensuring the privacy of patient data and expanding the learning dataset to include medical databases, and developing guidance on its appropriate use. Urologists can also help by conducting studies to determine the effectiveness of ChatGPT in urology in clinical scenarios and nosologies other than those previously listed.

Universal skepticism of ChatGPT: a review of early literature on chat generative pre-trained transformer

ChatGPT, a new language model developed by OpenAI, has garnered significant attention in various fields since its release. This literature review provides an overview of early ChatGPT literature across multiple disciplines, exploring its applications, limitations, and ethical considerations. The review encompasses Scopus-indexed publications from November 2022 to April 2023 and includes 156 articles related to ChatGPT. The findings reveal a predominance of negative sentiment across disciplines, though subject-specific attitudes must be considered. The review highlights the implications of ChatGPT in many fields including healthcare, raising concerns about employment opportunities and ethical considerations. While ChatGPT holds promise for improved communication, further research is needed to address its capabilities and limitations. This literature review provides insights into early research on ChatGPT, informing future investigations and practical applications of chatbot technology, as well as development and usage of generative AI.

Examining Real-World Medication Consultations and Drug-Herb Interactions: ChatGPT Performance Evaluation

BACKGROUND

Since OpenAI released ChatGPT, with its strong capability in handling natural tasks and its user-friendly interface, it has garnered significant attention.

OBJECTIVE

A prospective analysis is required to evaluate the accuracy and appropriateness of medication consultation responses generated by ChatGPT.

METHODS

A prospective cross-sectional study was conducted by the pharmacy department of a medical center in Taiwan. The test data set comprised retrospective medication consultation questions collected from February 1, 2023, to February 28, 2023, along with common questions about drug-herb interactions. Two distinct sets of questions were tested: real-world medication consultation questions and common questions about interactions between traditional Chinese and Western medicines. We used the conventional double-review mechanism. The appropriateness of each response from ChatGPT was assessed by 2 experienced pharmacists. In the event of a discrepancy between the assessments, a third pharmacist stepped in to make the final decision.

RESULTS

Of 293 real-world medication consultation questions, a random selection of 80 was used to evaluate ChatGPT's performance. ChatGPT exhibited a higher appropriateness rate in responding to public medication consultation questions compared to those asked by health care providers in a hospital setting (31/51, 61% vs 20/51, 39%; P=.01).

CONCLUSIONS

The findings from this study suggest that ChatGPT could potentially be used for answering basic medication consultation questions. Our analysis of the erroneous information allowed us to identify potential medical risks associated with certain questions; this problem deserves our close attention.

ChatGPT in Plastic and Reconstructive Surgery

Background  Chat Generative Pre-Trained Transformer (ChatGPT) is a versatile large language model-based generative artificial intelligence. It is proficient in a variety of tasks from drafting emails to coding to composing music to passing medical licensing exams. While the potential role of ChatGPT in plastic surgery is promising, evidence-based research is needed to guide its implementation in practice. Methods  This review aims to summarize the literature surrounding ChatGPT's use in plastic surgery. Results  A literature search revealed several applications for ChatGPT in the field of plastic surgery, including the ability to create academic literature and to aid the production of research. However, the ethical implications of using such chatbots in scientific writing requires careful consideration. ChatGPT can also generate high-quality patient discharge summaries and operation notes within seconds, freeing up busy junior doctors to complete other tasks. However, currently clinical information must still be manually inputted, and clinicians must consider data privacy implications. Its use in aiding patient communication and education and training is also widely documented in the literature. However, questions have been raised over the accuracy of answers generated given that current versions of ChatGPT cannot access the most up-to-date sources. Conclusions  While one must be aware of its shortcomings, ChatGPT is a useful tool for plastic surgeons to improve productivity for a range of tasks from manuscript preparation to healthcare communication generation to drafting teaching sessions to studying and learning. As access improves and technology becomes more refined, surely more uses for ChatGPT in plastic surgery will become apparent.

The Potential Usefulness of ChatGPT in Oral and Maxillofacial Radiology

Aim This study aimed to evaluate the potential usefulness of Chat Generated Pre-Trained Transformer-3 (ChatGPT-3) in oral and maxillofacial radiology for report writing by identifying radiographic anatomical landmarks and learning about oral and maxillofacial pathologies and their radiographic features. The study also aimed to evaluate the performance of ChatGPT-3 and its usage in oral and maxillofacial radiology training. Materials and methods A questionnaire consisting of 80 questions was queried on the OpenAI app ChatGPT-3. The questions were stratified based on three categories. The categorization was based on random anatomical landmarks, oral and maxillofacial pathologies, and the radiographic features of some of these pathologies. One oral and maxillofacial radiologist evaluated queries that were answered by the ChatGPT-3 model and rated them on a 4-point, modified Likert scale. The post-survey analysis for the performance of ChatGPT-3 was based on the Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis, its application in oral and maxillofacial radiology training, and its recommended use. Results In order of efficiency, Chat GPT-3 gave 100% accuracy in describing radiographic landmarks. However, the content of the oral and maxillofacial pathologies was limited to major or characteristic radiographic features. The mean scores for the queries related to the anatomic landmarks, oral and maxillofacial pathologies, and radiographic features of the oral and maxillofacial pathologies were 3.94, 3.85, and 3.96, respectively. However, the median and mode scores were 4 and were similar to all categories. The data for the oral and maxillofacial pathologies when the questions were not specifically included in the format of the introduction of the pathology, causes, symptoms, and treatment. Out of two abbreviations, one was not answered correctly. Conclusion The study showed that ChatGPT-3 is efficient in describing the pathology, characteristic radiographic features, and describing anatomical landmarks. ChatGPT-3 can be used as an adjunct when an oral radiologist needs additional information on any pathology, however, it cannot be the mainstay for reference. ChatGPT-3 is less detail-oriented, and the data has a risk of infodemics and the possibility of medical errors. However, Chat GPT-3 can be an excellent tool in helping the community in increasing the knowledge and awareness of various pathologies and decreasing the anxiety of the patients while dental healthcare professionals formulate an appropriate treatment plan.

Revolutionizing Medical Education: Can ChatGPT Boost Subjective Learning and Expression?

ChatGPT is an AI tool that can be used to enhance medical education by helping students develop subjective learning and expression skills. These skills are critical in clinical practice, but the current medical education system is heavily focused on objective assessments, such as Multiple Choice Questions (MCQs). Students from non-English speaking backgrounds can particularly struggle with expressing themselves in English, which is the primary language of instruction in many medical schools worldwide. ChatGPT can provide additional language support for these students to help them develop their language skills and communicate effectively. ChatGPT can be used in small group assessments to serve as a benchmark for students to strive for in their medical education. By comparing their answers to ChatGPT's responses, students can identify gaps in their knowledge and work to fill them. ChatGPT can also provide students with feedback on their writing style and language usage, helping them to improve their subjective expression of medical knowledge. Furthermore, ChatGPT can be used to simulate patient encounters for medical students. By interacting with ChatGPT, students can practice taking medical histories and documenting symptoms accurately. In continuing medical education (CME) programs, physicians can also benefit from ChatGPT's capabilities. By using ChatGPT to search for the latest research articles, clinical trials, and treatment guidelines, physicians can stay informed and provide the best care possible to their patients. Overall, ChatGPT has the potential to be a valuable tool in medical education by helping students and physicians develop the essential skills required for clinical practice, such as communication, problem-solving, and critical thinking.

ChatGPT and the Future of Health Policy Analysis: Potential and Pitfalls of Using ChatGPT in Policymaking

Scholars increasingly rely on new artificial intelligence models for convenience and simple access to necessities due to the rapid evolution of scientific literature and technology. The invention of ChatGPT by OpenAI stands out as a key example of how significant advances in large language model technology have recently changed the field of artificial intelligence (AI). Since ChatGPT's development, it has been tested by multiple sectors on various topics to see how well it functions in a natural and conversational mode. The crucial question is how much ChatGPT can influence global health policy analysis. In this article, the researcher briefly explains ChatGPT's potential and the difficulties that users, such as researchers or policymakers, may continue to face.

ChatGPT in Dentistry: A Comprehensive Review

Chat generative pre-trained transformer (ChatGPT) is an artificial intelligence chatbot that uses natural language processing that can respond to human input in a conversational manner. ChatGPT has numerous applications in the health care system including dentistry; it is used in diagnoses and for assessing disease risk and scheduling appointments. It also has a role in scientific research. In the dental field, it has provided many benefits such as detecting dental and maxillofacial abnormalities on panoramic radiographs and identifying different dental restorations. Therefore, it helps in decreasing the workload. But even with these benefits, one should take into consideration the risks and limitations of this chatbot. Few articles mentioned the use of ChatGPT in dentistry. This comprehensive review represents data collected from 66 relevant articles using PubMed and Google Scholar as databases. This review aims to discuss all relevant published articles on the use of ChatGPT in dentistry.