1
|
Zhong J, Xing Y, Hu Y, Lu J, Yang J, Zhang G, Mao S, Chen H, Yin Q, Cen Q, Jiang R, Chu J, Song Y, Lu M, Ding D, Ge X, Zhang H, Yao W. The policies on the use of large language models in radiological journals are lacking: a meta-research study. Insights Imaging 2024; 15:186. [PMID: 39090273 PMCID: PMC11294318 DOI: 10.1186/s13244-024-01769-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2024] [Accepted: 07/07/2024] [Indexed: 08/04/2024] Open
Abstract
OBJECTIVE To evaluate whether and how the radiological journals present their policies on the use of large language models (LLMs), and identify the journal characteristic variables that are associated with the presence. METHODS In this meta-research study, we screened Journals from the Radiology, Nuclear Medicine and Medical Imaging Category, 2022 Journal Citation Reports, excluding journals in non-English languages and relevant documents unavailable. We assessed their LLM use policies: (1) whether the policy is present; (2) whether the policy for the authors, the reviewers, and the editors is present; and (3) whether the policy asks the author to report the usage of LLMs, the name of LLMs, the section that used LLMs, the role of LLMs, the verification of LLMs, and the potential influence of LLMs. The association between the presence of policies and journal characteristic variables was evaluated. RESULTS The LLM use policies were presented in 43.9% (83/189) of journals, and those for the authors, the reviewers, and the editor were presented in 43.4% (82/189), 29.6% (56/189) and 25.9% (49/189) of journals, respectively. Many journals mentioned the aspects of the usage (43.4%, 82/189), the name (34.9%, 66/189), the verification (33.3%, 63/189), and the role (31.7%, 60/189) of LLMs, while the potential influence of LLMs (4.2%, 8/189), and the section that used LLMs (1.6%, 3/189) were seldomly touched. The publisher is related to the presence of LLM use policies (p < 0.001). CONCLUSION The presence of LLM use policies is suboptimal in radiological journals. A reporting guideline is encouraged to facilitate reporting quality and transparency. CRITICAL RELEVANCE STATEMENT It may facilitate the quality and transparency of the use of LLMs in scientific writing if a shared complete reporting guideline is developed by stakeholders and then endorsed by journals. KEY POINTS The policies on LLM use in radiological journals are unexplored. Some of the radiological journals presented policies on LLM use. A shared complete reporting guideline for LLM use is desired.
Collapse
Affiliation(s)
- Jingyu Zhong
- Laboratory of Key Technology and Materials in Minimally Invasive Spine Surgery, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Center for Spinal Minimally Invasive Research, Shanghai Jiao Tong University, Shanghai, China.
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
| | - Yue Xing
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yangfan Hu
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Junjie Lu
- Department of Epidemiology and Population Health, Stanford University School of Medicine, Stanford, CA, USA
| | - Jiarui Yang
- Department of Biomedical Engineering, Boston University, Boston, MA, USA
| | - Guangcheng Zhang
- Department of Orthopedics, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Shiqi Mao
- Department of Medical Oncology, Shanghai Pulmonary Hospital, Tongji University School of Medicine, Shanghai, China
| | - Haoda Chen
- Department of General Surgery, Pancreatic Disease Center, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Qian Yin
- Department of Pathology, Shanghai Sixth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Qingqing Cen
- Department of Dermatology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Run Jiang
- Department of Pharmacovigilance, Shanghai Hansoh BioMedical Co., Ltd., Shanghai, China
| | - Jingshen Chu
- Editorial Office of Journal of Diagnostics Concepts & Practice, Department of Science and Technology Development, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Yang Song
- MR Scientific Marketing, Siemens Healthineers Ltd., Shanghai, China
| | - Minda Lu
- MR Application, Siemens Healthineers Ltd., Shanghai, China
| | - Defang Ding
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xiang Ge
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Huan Zhang
- Department of Radiology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
| | - Weiwu Yao
- Laboratory of Key Technology and Materials in Minimally Invasive Spine Surgery, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Center for Spinal Minimally Invasive Research, Shanghai Jiao Tong University, Shanghai, China.
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
| |
Collapse
|
2
|
Luo X, Chen F, Zhu D, Wang L, Wang Z, Liu H, Lyu M, Wang Y, Wang Q, Chen Y. Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses. J Med Internet Res 2024; 26:e56780. [PMID: 38819655 PMCID: PMC11234072 DOI: 10.2196/56780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Revised: 05/21/2024] [Accepted: 05/29/2024] [Indexed: 06/01/2024] Open
Abstract
Large language models (LLMs) such as ChatGPT have become widely applied in the field of medical research. In the process of conducting systematic reviews, similar tools can be used to expedite various steps, including defining clinical questions, performing the literature search, document screening, information extraction, and language refinement, thereby conserving resources and enhancing efficiency. However, when using LLMs, attention should be paid to transparent reporting, distinguishing between genuine and false content, and avoiding academic misconduct. In this viewpoint, we highlight the potential roles of LLMs in the creation of systematic reviews and meta-analyses, elucidating their advantages, limitations, and future research directions, aiming to provide insights and guidance for authors planning systematic reviews and meta-analyses.
Collapse
Affiliation(s)
- Xufei Luo
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Fengxian Chen
- School of Information Science & Engineering, Lanzhou University, Lanzhou, China
| | - Di Zhu
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Ling Wang
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Zijun Wang
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Hui Liu
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| | - Meng Lyu
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Ye Wang
- School of Public Health, Lanzhou University, Lanzhou, China
| | - Qi Wang
- Department of Health Research Methods, Evidence and Impact, Faculty of Health Sciences, McMaster University, Hamilton, ON, Canada
- McMaster Health Forum, McMaster University, Hamilton, ON, Canada
| | - Yaolong Chen
- Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
- World Health Organization Collaboration Center for Guideline Implementation and Knowledge Translation, Lanzhou, China
- Institute of Health Data Science, Lanzhou University, Lanzhou, China
- Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou, China
- Research Unit of Evidence-Based Evaluation and Guidelines, Chinese Academy of Medical Sciences (2021RU017), School of Basic Medical Sciences, Lanzhou University, Lanzhou, China
| |
Collapse
|
3
|
Kenig N, Monton Echeverria J, Rubi C. Ethics for AI in Plastic Surgery: Guidelines and Review. Aesthetic Plast Surg 2024; 48:2204-2209. [PMID: 38456892 DOI: 10.1007/s00266-024-03932-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 02/09/2024] [Indexed: 03/09/2024]
Abstract
INTRODUCTION Artificial intelligence (AI) holds the potential to revolutionize medicine, offering vast improvements for plastic surgery. While human physicians are limited to one lifetime of experience, AI is poised to soon surpass human capabilities, as it draws on limitless information and continuous learning abilities. Nevertheless, as AI becomes increasingly prevalent in this domain, it gives rise to critical ethical considerations that must be addressed by professionals. MATERIALS AND METHODS This work reviews the literature referring to the ethical challenges brought on by the ever-expanding use of AI in plastic surgery and offers guidelines for its application. RESULTS Ethical challenges include the disclosure of use of AI by caregivers, validation of decision-making, data privacy, informed consent and autonomy, potential biases in AI systems, the opaque nature of AI models, questions of liability, and the need for regulations. CONCLUSIONS There is a lack of consensus for the ethical use of AI in plastic surgery. Guidelines, such as those presented in this work, are needed within each discipline of medicine to respond to important ethical considerations for the safe use of AI. LEVEL OF EVIDENCE V This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .
Collapse
Affiliation(s)
- Nitzan Kenig
- Instituto Rubi, Cami dels Reis, 308, 07010, Palma de Mallorca, Spain.
| | | | - Carlos Rubi
- Instituto Rubi, Cami dels Reis, 308, 07010, Palma de Mallorca, Spain
| |
Collapse
|
4
|
Patino GA, Amiel JM, Brown M, Lypson ML, Chan TM. The Promise and Perils of Artificial Intelligence in Health Professions Education Practice and Scholarship. ACADEMIC MEDICINE : JOURNAL OF THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES 2024; 99:477-481. [PMID: 38266214 DOI: 10.1097/acm.0000000000005636] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/26/2024]
Abstract
ABSTRACT Artificial intelligence (AI) methods, especially machine learning and natural language processing, are increasingly affecting health professions education (HPE), including the medical school application and selection processes, assessment, and scholarship production. The rise of large language models over the past 18 months, such as ChatGPT, has raised questions about how best to incorporate these methods into HPE. The lack of training in AI among most HPE faculty and scholars poses an important challenge in facilitating such discussions. In this commentary, the authors provide a primer on the AI methods most often used in the practice and scholarship of HPE, discuss the most pressing challenges and opportunities these tools afford, and underscore that these methods should be understood as part of the larger set of statistical tools available.Despite their ability to process huge amounts of data and their high performance completing some tasks, AI methods are only as good as the data on which they are trained. Of particular importance is that these models can perpetuate the biases that are present in those training datasets, and they can be applied in a biased manner by human users. A minimum set of expectations for the application of AI methods in HPE practice and scholarship is discussed in this commentary, including the interpretability of the models developed and the transparency needed into the use and characteristics of such methods.The rise of AI methods is affecting multiple aspects of HPE including raising questions about how best to incorporate these models into HPE practice and scholarship. In this commentary, we provide a primer on the AI methods most often used in HPE and discuss the most pressing challenges and opportunities these tools afford.
Collapse
|
5
|
Carobene A, Padoan A, Cabitza F, Banfi G, Plebani M. Rising adoption of artificial intelligence in scientific publishing: evaluating the role, risks, and ethical implications in paper drafting and review process. Clin Chem Lab Med 2024; 62:835-843. [PMID: 38019961 DOI: 10.1515/cclm-2023-1136] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 11/13/2023] [Indexed: 12/01/2023]
Abstract
BACKGROUND In the rapid evolving landscape of artificial intelligence (AI), scientific publishing is experiencing significant transformations. AI tools, while offering unparalleled efficiencies in paper drafting and peer review, also introduce notable ethical concerns. CONTENT This study delineates AI's dual role in scientific publishing: as a co-creator in the writing and review of scientific papers and as an ethical challenge. We first explore the potential of AI as an enhancer of efficiency, efficacy, and quality in creating scientific papers. A critical assessment follows, evaluating the risks vs. rewards for researchers, especially those early in their careers, emphasizing the need to maintain a balance between AI's capabilities and fostering independent reasoning and creativity. Subsequently, we delve into the ethical dilemmas of AI's involvement, particularly concerning originality, plagiarism, and preserving the genuine essence of scientific discourse. The evolving dynamics further highlight an overlooked aspect: the inadequate recognition of human reviewers in the academic community. With the increasing volume of scientific literature, tangible metrics and incentives for reviewers are proposed as essential to ensure a balanced academic environment. SUMMARY AI's incorporation in scientific publishing is promising yet comes with significant ethical and operational challenges. The role of human reviewers is accentuated, ensuring authenticity in an AI-influenced environment. OUTLOOK As the scientific community treads the path of AI integration, a balanced symbiosis between AI's efficiency and human discernment is pivotal. Emphasizing human expertise, while exploit artificial intelligence responsibly, will determine the trajectory of an ethically sound and efficient AI-augmented future in scientific publishing.
Collapse
Affiliation(s)
- Anna Carobene
- Laboratory Medicine, IRCCS San Raffaele Scientific Institute, Milan, Italy
| | - Andrea Padoan
- Department of Medicine-DIMED, University of Padova, Padova, Italy
- Laboratory Medicine Unit, University Hospital of Padova, Padova, Italy
| | - Federico Cabitza
- DISCo, Università Degli Studi di Milano-Bicocca, Milan, Italy
- IRCCS Ospedale Galeazzi - Sant'Ambrogio, Milan, Italy
| | - Giuseppe Banfi
- IRCCS Ospedale Galeazzi - Sant'Ambrogio, Milan, Italy
- University Vita-Salute San Raffaele, Milan, Italy
| | - Mario Plebani
- Laboratory Medicine Unit, University Hospital of Padova, Padova, Italy
- University of Padova, Padova, Italy
| |
Collapse
|
6
|
Seckel E, Stephens BY, Rodriguez F. Ten simple rules to leverage large language models for getting grants. PLoS Comput Biol 2024; 20:e1011863. [PMID: 38427611 PMCID: PMC10906892 DOI: 10.1371/journal.pcbi.1011863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/03/2024] Open
Affiliation(s)
- Elizabeth Seckel
- Division of Cardiovascular Medicine, Stanford University School of Medicine, Stanford, California, United States of America
| | - Brandi Y. Stephens
- Division of Cardiovascular Medicine, Stanford University School of Medicine, Stanford, California, United States of America
| | - Fatima Rodriguez
- Division of Cardiovascular Medicine, Stanford University School of Medicine, Stanford, California, United States of America
| |
Collapse
|
7
|
Inam M, Sheikh S, Minhas AMK, Vaughan EM, Krittanawong C, Samad Z, Lavie CJ, Khoja A, D'Cruze M, Slipczuk L, Alarakhiya F, Naseem A, Haider AH, Virani SS. A review of top cardiology and cardiovascular medicine journal guidelines regarding the use of generative artificial intelligence tools in scientific writing. Curr Probl Cardiol 2024; 49:102387. [PMID: 38185435 DOI: 10.1016/j.cpcardiol.2024.102387] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Accepted: 01/04/2024] [Indexed: 01/09/2024]
Abstract
BACKGROUND Generative Artificial Intelligence (AI) tools have experienced rapid development over the last decade and are gaining increasing popularity as assistive models in academic writing. However, the ability of AI to generate reliable and accurate research articles is a topic of debate. Major scientific journals have issued policies regarding the contribution of AI tools in scientific writing. METHODS We conducted a review of the author and peer reviewer guidelines of the top 25 Cardiology and Cardiovascular Medicine journals as per the 2023 SCImago rankings. Data were obtained though reviewing journal websites and directly emailing the editorial office. Descriptive data regarding journal characteristics were coded on SPSS. Subgroup analyses of the journal guidelines were conducted based on the publishing company policies. RESULTS Our analysis revealed that all scientific journals in our study permitted the documented use of AI in scientific writing with certain limitations as per ICMJE recommendations. We found that AI tools cannot be included in the authorship or be used for image generation, and that all authors are required to assume full responsibility of their submitted and published work. The use of generative AI tools in the peer review process is strictly prohibited. CONCLUSION Guidelines regarding the use of generative AI in scientific writing are standardized, detailed, and unanimously followed by all journals in our study according to the recommendations set forth by international forums. It is imperative to ensure that these policies are carefully followed and updated to maintain scientific integrity.
Collapse
Affiliation(s)
- Maha Inam
- Office of the Vice Provost, Research, Aga Khan University, Karachi, Pakistan
| | - Sana Sheikh
- Department of Medicine, Aga Khan University, Karachi, Pakistan
| | - Abdul Mannan Khan Minhas
- Section of Cardiovascular Research, Department of Medicine, Baylor College of Medicine, Houston, TX, United States
| | - Elizabeth M Vaughan
- Section of Cardiovascular Research, Department of Medicine, Baylor College of Medicine, Houston, TX, United States; Department of Internal Medicine, UTMB, Galveston, TX, United States
| | - Chayakrit Krittanawong
- Leon H. Charney Division of Cardiology, New York University Langone Health, New York, NY, United States
| | - Zainab Samad
- Section of Cardiology, Department of Medicine, Aga Khan University Hospital, Karachi, Pakistan
| | - Carl J Lavie
- Department of Cardiovascular Diseases, John Ochsner Heart and Vascular Institute, Ochsner Clinical School, The University of Queensland School of Medicine, New Orleans, LA, United States
| | - Adeel Khoja
- Department of Medicine, Aga Khan University, Karachi, Pakistan; Adelaide Medical School, Faculty of Health and Medical Sciences, The University of Adelaide, Adelaide, South Australia, Australia
| | - Melaine D'Cruze
- Institute for Educational Development, Aga Khan University Hospital, Karachi, Pakistan
| | - Leandro Slipczuk
- Cardiology Division, Montefiore Medical Center, Bronx, NY, United States; Albert Einstein College of Medicine, Bronx, NY, United States
| | | | - Azra Naseem
- Institute for Educational Development, Aga Khan University Hospital, Karachi, Pakistan
| | - Adil H Haider
- Dean's Office, Medical College, Aga Khan University Hospital, Karachi, Pakistan
| | - Salim S Virani
- Office of the Vice Provost, Research, Aga Khan University, Karachi, Pakistan; Section of Cardiovascular Research, Department of Medicine, Baylor College of Medicine, Houston, TX, United States; Section of Cardiology, Department of Medicine, Aga Khan University Hospital, Karachi, Pakistan; The Texas Heart Institute, Houston, TX, United States.
| |
Collapse
|
8
|
Hakam HT, Prill R, Korte L, Lovreković B, Ostojić M, Ramadanov N, Muehlensiepen F. Human-Written vs AI-Generated Texts in Orthopedic Academic Literature: Comparative Qualitative Analysis. JMIR Form Res 2024; 8:e52164. [PMID: 38363631 PMCID: PMC10907945 DOI: 10.2196/52164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 11/09/2023] [Accepted: 12/13/2023] [Indexed: 02/17/2024] Open
Abstract
BACKGROUND As large language models (LLMs) are becoming increasingly integrated into different aspects of health care, questions about the implications for medical academic literature have begun to emerge. Key aspects such as authenticity in academic writing are at stake with artificial intelligence (AI) generating highly linguistically accurate and grammatically sound texts. OBJECTIVE The objective of this study is to compare human-written with AI-generated scientific literature in orthopedics and sports medicine. METHODS Five original abstracts were selected from the PubMed database. These abstracts were subsequently rewritten with the assistance of 2 LLMs with different degrees of proficiency. Subsequently, researchers with varying degrees of expertise and with different areas of specialization were asked to rank the abstracts according to linguistic and methodological parameters. Finally, researchers had to classify the articles as AI generated or human written. RESULTS Neither the researchers nor the AI-detection software could successfully identify the AI-generated texts. Furthermore, the criteria previously suggested in the literature did not correlate with whether the researchers deemed a text to be AI generated or whether they judged the article correctly based on these parameters. CONCLUSIONS The primary finding of this study was that researchers were unable to distinguish between LLM-generated and human-written texts. However, due to the small sample size, it is not possible to generalize the results of this study. As is the case with any tool used in academic research, the potential to cause harm can be mitigated by relying on the transparency and integrity of the researchers. With scientific integrity at stake, further research with a similar study design should be conducted to determine the magnitude of this issue.
Collapse
Affiliation(s)
- Hassan Tarek Hakam
- Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany
- Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany
- Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany
| | - Robert Prill
- Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany
- Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany
| | - Lisa Korte
- Center of Health Services Research, Faculty of Health Sciences, University Clinic of Brandenburg, Rüdersdorf bei Berlin, Germany
| | - Bruno Lovreković
- Faculty of Orthopaedics, University Hospital Merkur, Zagreb, Croatia
| | - Marko Ostojić
- Departement of Orthopaedics, University Hospital Mostar, Mostar, Bosnia and Herzegovina
| | - Nikolai Ramadanov
- Center of Orthopaedics and Trauma Surgery, University Clinic of Brandenburg, Brandenburg Medical School, Brandenburg an der Havel, Germany
- Faculty of Health Sciences, University Clinic of Brandenburg, Brandenburg an der Havel, Germany
| | - Felix Muehlensiepen
- Center of Evidence Based Practice in Brandenburg, a JBI Affiliated Group, Brandenburg an der Havel, Germany
- Center of Health Services Research, Faculty of Health Sciences, University Clinic of Brandenburg, Rüdersdorf bei Berlin, Germany
| |
Collapse
|
9
|
Bisi T, Risser A, Clavert P, Migaud H, Dartus J. What is the rate of text generated by artificial intelligence over a year of publication in Orthopedics & Traumatology: Surgery & Research? Analysis of 425 articles before versus after the launch of ChatGPT in November 2022. Orthop Traumatol Surg Res 2023; 109:103694. [PMID: 37776949 DOI: 10.1016/j.otsr.2023.103694] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/17/2023] [Revised: 07/10/2023] [Accepted: 08/24/2023] [Indexed: 10/02/2023]
Abstract
BACKGROUND The use of artificial intelligence (AI) is soaring, and the launch of ChatGPT in November 2022 has accelerated this trend. This "chatbot" can generate complete scientific articles, with risk of plagiarism by mining existing data or downright fraud by fabricating studies with no real data at all. There are tools that detect AI in publications, but to our knowledge they have not been systematically assessed for publication in scientific journals. We therefore conducted a retrospective study on articles published in Orthopaedics & Traumatology: Surgery & Research (OTSR): firstly, to screen for AI-generated content before and after the publicized launch of ChatGPT; secondly, to assess whether AI was more often used in some countries than others to generate content; thirdly, to determine whether plagiarism rate correlated with AI-generation, and lastly, to determine whether elements other than text generation, and notably the translation procedure, could raise suspicion of AI use. HYPOTHESIS The rate of AI use increased after the publicized launch of ChatGPT v3.5 in November 2022. MATERIAL AND METHODS In all, 425 articles published between February 2022 and September 2023 (221 before and 204 after November 1, 2022) underwent ZeroGPT assessment of the level of AI generation in the final English-language version (abstract and body of the article). Two scores were obtained: probability of AI generation, in six grades from Human to AI; and percentage AI generation. Plagiarism was assessed on the Ithenticate application at submission. Articles in French were assessed in their English-language version as translated by a human translator, with comparison to automatic translation by Google Translate and DeepL. RESULTS AI-generated text was detected mainly in Abstracts, with a 10.1% rate of AI or considerable AI generation, compared to only 1.9% for the body of the article and 5.6% for the total body+abstract. Analysis for before and after November 2022 found an increase in AI generation in body+abstract, from 10.30±15.95% (range, 0-100%) to 15.64±19.8% (range, 0-99.93) (p < 0.04; NS for abstracts alone). AI scores differed between types of article: 14.9% for original articles and 9.8% for reviews (p<0.01). The highest rates of probable AI generation were in articles from Japan, China, South America and English-speaking countries (p<0.0001). Plagiarism rates did not increase between the two study periods, and were unrelated to AI rates. On the other hand, when articles were classified as "suspected" of AI generation (plagiarism rate ≥ 20%) or "non-suspected" (rate<20%), the "similarity" score was higher in suspect articles: 25.7±13.23% (range, 10-69%) versus 16.28±10% (range, 0-79%) (p < 0.001). In the body of the article, use of translation software was associated with higher AI rates than with a human translator: 3.5±5% for human translators, versus 18±10% and 21.9±11% respectively for Google Translate and DeepL (p < 0.001). DISCUSSION The present study revealed an increasing rate of AI use in articles published in OTSR. AI grades differed according to type of article and country of origin. Use of translation software increased the AI grade. In the long run, use of ChatGPT incurs a risk of plagiarism and scientific misconduct, and needs to be detected and signaled by a digital tag on any robot-generated text. LEVEL OF EVIDENCE III; case-control study.
Collapse
Affiliation(s)
- Théophile Bisi
- Département universitaire de chirurgie orthopédique, université de Lille, CHU de Lille, 59000 Lille, France; Service de chirurgie orthopédique, centre hospitalier universitaire (CHU) de Lille, hôpital Roger-Salengro, place de Verdun, 59000 Lille, France.
| | - Anthony Risser
- Service de chirurgie du membre supérieur, Hautepierre 2, CHRU Strasbourg, 1, avenue Molière, 67200 Strasbourg, France
| | - Philippe Clavert
- Service de chirurgie du membre supérieur, Hautepierre 2, CHRU Strasbourg, 1, avenue Molière, 67200 Strasbourg, France; Faculté de médecine, institut d'anatomie normale, 4, rue Kirschleger, 67085 Strasbourg, France
| | - Henri Migaud
- Département universitaire de chirurgie orthopédique, université de Lille, CHU de Lille, 59000 Lille, France; Service de chirurgie orthopédique, centre hospitalier universitaire (CHU) de Lille, hôpital Roger-Salengro, place de Verdun, 59000 Lille, France
| | - Julien Dartus
- Département universitaire de chirurgie orthopédique, université de Lille, CHU de Lille, 59000 Lille, France; Service de chirurgie orthopédique, centre hospitalier universitaire (CHU) de Lille, hôpital Roger-Salengro, place de Verdun, 59000 Lille, France
| |
Collapse
|
10
|
Shoja MM, Van de Ridder JMM, Rajput V. The Emerging Role of Generative Artificial Intelligence in Medical Education, Research, and Practice. Cureus 2023; 15:e40883. [PMID: 37492829 PMCID: PMC10363933 DOI: 10.7759/cureus.40883] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Accepted: 06/24/2023] [Indexed: 07/27/2023] Open
Abstract
Recent breakthroughs in generative artificial intelligence (GAI) and the emergence of transformer-based large language models such as Chat Generative Pre-trained Transformer (ChatGPT) have the potential to transform healthcare education, research, and clinical practice. This article examines the current trends in using GAI models in medicine, outlining their strengths and limitations. It is imperative to develop further consensus-based guidelines to govern the appropriate use of GAI, not only in medical education but also in research, scholarship, and clinical practice.
Collapse
Affiliation(s)
| | | | - Vijay Rajput
- Medical Education, Dr. Kiran C. Patel College of Allopathic Medicine, Nova Southeastern University, Fort Lauderdale, USA
| |
Collapse
|
11
|
Using ChatGPT for language editing in scientific articles. Maxillofac Plast Reconstr Surg 2023; 45:13. [PMID: 36882591 PMCID: PMC9992464 DOI: 10.1186/s40902-023-00381-x] [Citation(s) in RCA: 36] [Impact Index Per Article: 36.0] [Reference Citation Analysis] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/09/2023] Open
|