Abstracts written by ChatGPT fool scientists

The Role of Artificial Intelligence Large Language Models in Literature Search Assistance to Evaluate Inguinal Hernia Repair Approaches

Aim: This study assesses the reliability of artificial intelligence (AI) large language models (LLMs) in identifying relevant literature comparing inguinal hernia repair techniques. Material and Methods: We used LLM chatbots (Bing Chat AI, ChatGPT versions 3.5 and 4.0, and Gemini) to find comparative studies and randomized controlled trials on inguinal hernia repair techniques. The results were then compared with existing systematic reviews (SRs) and meta-analyses and checked for the authenticity of listed articles. Results: LLMs screened 22 studies from 2006 to 2023 across eight journals, while the SRs encompassed a total of 42 studies. Through thorough external validation, 63.6% of the studies (14 out of 22), including 10 identified through Chat GPT 4.0 and 6 via Bing AI (with an overlap of 2 studies between them), were confirmed to be authentic. Conversely, 36.3% (8 out of 22) were revealed as fabrications by Google Gemini (Bard), with two (25.0%) of these fabrications mistakenly linked to valid DOIs. Four (25.6%) of the 14 real studies were acknowledged in the SRs, which represents 18.1% of all LLM-generated studies. LLMs missed a total of 38 (90.5%) of the studies included in the previous SRs, while 10 real studies were found by the LLMs but were not included in the previous SRs. Between those 10 studies, 6 were reviews, and 1 was published after the SRs, leaving a total of three comparative studies missed by the reviews. Conclusions: This study reveals the mixed reliability of AI language models in scientific searches. Emphasizing a cautious application of AI in academia and the importance of continuous evaluation of AI tools in scientific investigations.

Identification of dental related ChatGPT generated abstracts by senior and young academicians versus artificial intelligence detectors and a similarity detector

Several researchers have investigated the consequences of using ChatGPT in the education industry. Their findings raised doubts regarding the probable effects that ChatGPT may have on the academia. As such, the present study aimed to assess the ability of three methods, namely: (1) academicians (senior and young), (2) three AI detectors (GPT-2 output detector, Writefull GPT detector, and GPTZero) and (3) one plagiarism detector, to differentiate between human- and ChatGPT-written abstracts. A total of 160 abstracts were assessed by those three methods. Two senior and two young academicians used a newly developed rubric to assess the type and quality of 80 human-written and 80 ChatGPT-written abstracts. The results were statistically analysed using crosstabulation and chi-square analysis. Bivariate correlation and accuracy of the methods were assessed. The findings demonstrated that all the three methods made a different variety of incorrect assumptions. The level of the academician experience may play a role in the detection ability with senior academician 1 demonstrating superior accuracy. GPTZero AI and similarity detectors were very good at accurately identifying the abstracts origin. In terms of abstract type, every variable positively correlated, except in the case of similarity detectors (p < 0.05). Human-AI collaborations may significantly benefit the identification of the abstract origins.

Accuracy of a Large Language Model as a new tool for optometry education

CLINICAL RELEVANCE

The unsupervised introduction of certain Artificial Intelligence tools in optometry education may challenge the proper acquisition of accurate clinical knowledge and skills proficiency.

BACKGROUND

Large Language Models like ChatGPT (Generative Pretrained Transformer) are increasingly being used by researchers and students for work and academic assignments. The authoritative and conversationally correct language provided by these tools may mask their inherent limitations when presented with specific scientific and clinical queries.

METHODS

Three sets of 10 queries related to contact lenses & anterior eye, low vision and binocular vision & vision therapy were presented to ChatGPT, with instructions to provide five relevant references to support each response. Three experts and 53 undergraduate and post-graduate students graded from 0 to 10 the accuracy of the responses, and the references were evaluated for precision and relevance. Students graded from 0 to 10 the potential usefulness of ChatGPT for their academic coursework.

RESULTS

Median scores were 7, 8 and 6 (experts) and 8, 9 and 7.5 (students) for the contact lenses & anterior eye, low vision and binocular vision & vision therapy categories, respectively. Responses to more specific queries were awarded lower scores by both experts (ρ = -0.612; P < 0.001) and students (ρ = -0.578; P = 0.001). Of 150 references, 24% were accurate and 19.3% relevant. Students graded the usefulness of ChatGPT with 7.5 (2 to 9), 7 (3 to 9) and 8.5 (3 to 10) for contact lenses & anterior eye, low vision and binocular vision & vision therapy, respectively.

CONCLUSION

Careful expert appraisal of the responses and, particularly, of the references provided by ChatGPT is required in research and academic settings. As the use of these tools becomes widespread, it is essential to take proactive steps to address their limitations and ensure their responsible use.

Ethical Application of Generative Artificial Intelligence in Medicine

Generative artificial intelligence (AI) may revolutionize health care, providing solutions that range from enhancing diagnostic accuracy to personalizing treatment plans. However, its rapid and largely unregulated integration into medicine raises ethical concerns related to data integrity, patient safety, and appropriate oversight. One of the primary ethical challenges lies in generative AI's potential to produce misleading or fabricated information, posing risks of misdiagnosis or inappropriate treatment recommendations, which underscore the necessity for robust physician oversight. Transparency also remains a critical concern, as the closed-source nature of many large-language models prevents both patients and health care providers from understanding the reasoning behind AI-generated outputs, potentially eroding trust. The lack of regulatory approval for AI as a medical device, combined with concerns around the security of patient-derived data and AI-generated synthetic data, further complicates its safe integration into clinical workflows. Furthermore, synthetic datasets generated by AI, although valuable for augmenting research in areas with scarce data, complicate questions of data ownership, patient consent, and scientific validity. In addition, generative AI's ability to streamline administrative tasks risks depersonalizing care, further distancing providers from patients. These challenges compound the deeper issues plaguing the health care system, including the emphasis of volume and speed over value and expertise. The use of generative AI in medicine brings about mass scaling of synthetic information, thereby necessitating careful adoption to protect patient care and medical advancement. Given these considerations, generative AI applications warrant regulatory and critical scrutiny. Key starting points include establishing strict standards for data security and transparency, implementing oversight akin to institutional review boards to govern data usage, and developing interdisciplinary guidelines that involve developers, clinicians, and ethicists. By addressing these concerns, we can better align generative AI adoption with the core foundations of humanistic health care, preserving patient safety, autonomy, and trust while harnessing AI's transformative potential. LEVEL OF EVIDENCE: Level V, expert opinion.

Evaluating the Efficacy of Perplexity Scores in Distinguishing AI-Generated and Human-Written Abstracts

RATIONALE AND OBJECTIVES

We aimed to evaluate the efficacy of perplexity scores in distinguishing between human-written and AI-generated radiology abstracts and to assess the relative performance of available AI detection tools in detecting AI-generated content.

METHODS

Academic articles were curated from PubMed using the keywords "neuroimaging" and "angiography." Filters included English-language, open-access articles with abstracts without subheadings, published before 2021, and within Chatbot processing word limits. The first 50 qualifying articles were selected, and their full texts were used to create AI-generated abstracts. Perplexity scores, which estimate sentence predictability, were calculated for both AI-generated and human-written abstracts. The performance of three AI tools in discriminating human-written from AI-generated abstracts was assessed.

RESULTS

The selected 50 articles consist of 22 review articles (44%), 12 case or technical reports (24%), 15 research articles (30%), and one editorial (2%). The perplexity scores for human-written abstracts (median; 35.9 IQR; 25.11-51.8) were higher than those for AI-generated abstracts (median; 21.2 IQR; 16.87-28.38), (p=0.057) with an AUC=0.7794. One AI tool performed less than chance in identifying human-written from AI-generated abstracts with an accuracy of 36% (p>0.05) while another tool yielded an accuracy of 95% with an AUC=0.8688.

CONCLUSION

This study underscores the potential of perplexity scores in detecting AI-generated and potentially fraudulent abstracts. However, more research is needed to further explore these findings and their implications for the use of AI in academic writing. Future studies could also investigate other metrics or methods for distinguishing between human-written and AI-generated texts.

Performance of Artificial Intelligence Chatbots in Answering Clinical Questions on Japanese Practical Guidelines for Implant-based Breast Reconstruction

BACKGROUND

Artificial intelligence (AI) chatbots, including ChatGPT-4 (GPT-4) and Grok-1 (Grok), have been shown to be potentially useful in several medical fields, but have not been examined in plastic and aesthetic surgery. The aim of this study is to evaluate the responses of these AI chatbots for clinical questions (CQs) related to the guidelines for implant-based breast reconstruction (IBBR) published by the Japan Society of Plastic and Reconstructive Surgery (JSPRS) in 2021.

METHODS

CQs in the JSPRS guidelines were used as question sources. Responses from two AI chatbots, GPT-4 and Grok, were evaluated for accuracy, informativeness, and readability by five Japanese Board-certified breast reconstruction specialists and five Japanese clinical fellows of plastic surgery.

RESULTS

GPT-4 outperformed Grok significantly in terms of accuracy (p < 0.001), informativeness (p < 0.001), and readability (p < 0.001) when evaluated by plastic surgery fellows. Compared to the original guidelines, Grok scored significantly lower in all three areas (all p < 0.001). The accuracy of GPT-4 was rated to be significantly higher based on scores given by plastic surgery fellows compared to those of breast reconstruction specialists (p = 0.012), whereas there was no significant difference between these scores for Grok.

CONCLUSIONS

The study suggests that GPT-4 has the potential to assist in interpreting and applying clinical guidelines for IBBR but importantly there is still a risk that AI chatbots can misinform. Further studies are needed to understand the broader role of current and future AI chatbots in breast reconstruction surgery.

LEVEL OF EVIDENCE IV

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine Ratings, please refer to Table of Contents or online Instructions to Authors www.springer.com/00266 .

Generative AI lacks the human creativity to achieve scientific discovery from scratch

Scientists are interested in whether generative artificial intelligence (GenAI) can make scientific discoveries similar to those of humans. However, the results are mixed. Here, we examine whether, how and what scientific discovery GenAI can make in terms of the origin of hypotheses and experimental design through the interpretation of results. With the help of a computer-supported molecular genetic laboratory, GenAI assumes the role of a scientist tasked with investigating a Nobel-worthy scientific discovery in the molecular genetics field. We find that current GenAI can make only incremental discoveries but cannot achieve fundamental discoveries from scratch as humans can. Regarding the origin of the hypothesis, it is unable to generate truly original hypotheses and is incapable of having an epiphany to detect anomalies in experimental results. Therefore, current GenAI is good only at discovery tasks involving either a known representation of the domain knowledge or access to the human scientists' knowledge space. Furthermore, it has the illusion of making a completely successful discovery with overconfidence. We discuss approaches to address the limitations of current GenAI and its ethical concerns and biases in scientific discovery. This research provides insight into the role of GenAI in scientific discovery and general scientific innovation.

Artificial Intelligence-Generated Editorials in Radiology: Can Expert Editors Detect Them?

BACKGROUND AND PURPOSE

Artificial intelligence is capable of generating complex texts that may be indistinguishable from those written by humans. We aimed to evaluate the ability of GPT-4 to write radiology editorials and to compare these with human-written counterparts, thereby determining their real-world applicability for scientific writing.

MATERIALS AND METHODS

Sixteen editorials from 8 journals were included. To generate the artificial intelligence (AI)-written editorials, the summary of 16 human-written editorials was fed into GPT-4. Six experienced editors reviewed the articles. First, an unpaired approach was used. The raters were asked to evaluate the content of each article by using a 1-5 Likert scale across specified metrics. Then, they determined whether the editorials were written by humans or AI. The articles were then evaluated in pairs to determine which article was generated by AI and which should be published. Finally, the articles were analyzed with an AI detector and for plagiarism.

RESULTS

The human-written articles had a median AI probability score of 2.0%, whereas the AI-written articles had 58%. The median similarity score among AI-written articles was 3%. Fifty-eight percent of unpaired articles were correctly classified regarding authorship. Rating accuracy was increased to 70% in the paired setting. AI-written articles received slightly higher scores in most metrics. When stratified by perception, human-written perceived articles were rated higher in most categories. In the paired setting, raters strongly preferred publishing the article they perceived as human-written (82%).

CONCLUSIONS

GPT-4 can write high-quality articles that iThenticate does not flag as plagiarized, which may go undetected by editors, and that detection tools can detect to a limited extent. Editors showed a positive bias toward human-written articles.

Citation manipulation through citation mills and pre-print servers

Citations are widely considered in scientists' evaluation. As such, scientists may be incentivized to inflate their citation counts. While previous literature has examined self-citations and citation cartels, it remains unclear whether scientists can purchase citations. Here, we compile a dataset of ~1.6 million profiles on Google Scholar to examine instances of citation fraud on the platform. We survey faculty at highly-ranked universities, and confirm that Google Scholar is widely used when evaluating scientists. We then engage with a citation-boosting service, and manage to purchase 50 citations while assuming the identity of a fictional author. Taken as a whole, our findings bring to light new forms of citation manipulation, and emphasize the need to look beyond citation counts.

Higher education students' perceptions of ChatGPT: A global study of early reactions

The paper presents the most comprehensive and large-scale global study to date on how higher education students perceived the use of ChatGPT in early 2024. With a sample of 23,218 students from 109 countries and territories, the study reveals that students primarily used ChatGPT for brainstorming, summarizing texts, and finding research articles, with a few using it for professional and creative writing. They found it useful for simplifying complex information and summarizing content, but less reliable for providing information and supporting classroom learning, though some considered its information clearer than that from peers and teachers. Moreover, students agreed on the need for AI regulations at all levels due to concerns about ChatGPT promoting cheating, plagiarism, and social isolation. However, they believed ChatGPT could potentially enhance their access to knowledge and improve their learning experience, study efficiency, and chances of achieving good grades. While ChatGPT was perceived as effective in potentially improving AI literacy, digital communication, and content creation skills, it was less useful for interpersonal communication, decision-making, numeracy, native language proficiency, and the development of critical thinking skills. Students also felt that ChatGPT would boost demand for AI-related skills and facilitate remote work without significantly impacting unemployment. Emotionally, students mostly felt positive using ChatGPT, with curiosity and calmness being the most common emotions. Further examinations reveal variations in students' perceptions across different socio-demographic and geographic factors, with key factors influencing students' use of ChatGPT also being identified. Higher education institutions' managers and teachers may benefit from these findings while formulating the curricula and instructions/regulations for ChatGPT use, as well as when designing the teaching methods and assessment tools. Moreover, policymakers may also consider the findings when formulating strategies for secondary and higher education system development, especially in light of changing labor market needs and related digital skills development.

Advancing Medical Research Through Artificial Intelligence: Progressive and Transformative Strategies: A Literature Review

Background and Aims

Artificial intelligence (AI) has become integral to medical research, impacting various aspects such as data analysis, writing assistance, and publishing. This paper explores the multifaceted influence of AI on the process of writing medical research papers, encompassing data analysis, ethical considerations, writing assistance, and publishing efficiency.

Methods

The review was conducted following the PRISMA guidelines; a comprehensive search was performed in Scopus, PubMed, EMBASE, and MEDLINE databases for research publications on artificial intelligence in medical research published up to October 2023.

Results

AI facilitates the writing process by generating drafts, offering grammar and style suggestions, and enhancing manuscript quality through advanced models like ChatGPT. Ethical concerns regarding content ownership and potential biases in AI-generated content underscore the need for collaborative efforts among researchers, publishers, and AI creators to establish ethical standards. Moreover, AI significantly influences data analysis in healthcare, optimizing outcomes and patient care, particularly in fields such as obstetrics and gynecology and pharmaceutical research. The application of AI in publishing, ranging from peer review to manuscript quality control and journal matching, underscores its potential to streamline and enhance the entire research and publication process. Overall, while AI presents substantial benefits, ongoing research, and ethical guidelines are essential for its responsible integration into the evolving landscape of medical research and publishing.

Conclusion

The integration of AI in medical research has revolutionized efficiency and innovation, impacting data analysis, writing assistance, publishing, and others. While AI tools offer significant benefits, ethical considerations such as biases and content ownership must be addressed. Ongoing research and collaborative efforts are crucial to ensure responsible and transparent AI implementation in the dynamic landscape of medical research and publishing.

Performance of a Generative Pre-Trained Transformer in Generating Scientific Abstracts in Dentistry: A Comparative Observational Study

OBJECTIVES

To evaluate the performance of a Generative Pre-trained Transformer (GPT) in generating scientific abstracts in dentistry.

METHODS

Ten scientific articles in dental radiology had their original abstracts collected, while another 10 articles had their methodology and results added to a ChatGPT prompt to generate an abstract. All abstracts were randomised and compiled into a single file for subsequent assessment. Five evaluators classified whether the abstract was generated by a human using a 5-point scale and provided justifications within seven aspects: formatting, information accuracy, orthography, punctuation, terminology, text fluency, and writing style. Furthermore, an online GPT detector provided "Human Score" values, and a plagiarism detector assessed similarity with existing literature.

RESULTS

Sensitivity values for detecting human writing ranged from 0.20 to 0.70, with a mean of 0.58; specificity values ranged from 0.40 to 0.90, with a mean of 0.62; and accuracy values ranged from 0.50 to 0.80, with a mean of 0.60. Orthography and Punctuation were the most indicated aspects for the abstract generated by ChatGPT. The GPT detector revealed confidence levels for a "Human Score" of 16.9% for the AI-generated texts and plagiarism levels averaging 35%.

CONCLUSION

The GPT exhibited commendable performance in generating scientific abstracts when evaluated by humans, as the generated abstracts were indistinguishable from those generated by humans. When evaluated by an online GPT detector, the use of GPT became apparent.

Evaluating human ability to distinguish between ChatGPT-generated and original scientific abstracts

This study aims to analyze the accuracy of human reviewers in identifying scientific abstracts generated by ChatGPT compared to the original abstracts. Participants completed an online survey presenting two research abstracts: one generated by ChatGPT and one original abstract. They had to identify which abstract was generated by AI and provide feedback on their preference and perceptions of AI technology in academic writing. This observational cross-sectional study involved surgical trainees and faculty at the University of British Columbia. The survey was distributed to all surgeons and trainees affiliated with the University of British Columbia, which includes general surgery, orthopedic surgery, thoracic surgery, plastic surgery, cardiovascular surgery, vascular surgery, neurosurgery, urology, otolaryngology, pediatric surgery, and obstetrics and gynecology. A total of 41 participants completed the survey. 41 participants responded, comprising 10 (23.3%) surgeons. Eighteen (40.0%) participants correctly identified the original abstract. Twenty-six (63.4%) participants preferred the ChatGPT abstract (p = 0.0001). On multivariate analysis, preferring the original abstract was associated with correct identification of the original abstract [OR 7.46, 95% CI (1.78, 31.4), p = 0.006]. Results suggest that human reviewers cannot accurately distinguish between human and AI-generated abstracts, and overall, there was a trend toward a preference for AI-generated abstracts. The findings contributed to understanding the implications of AI in manuscript production, including its benefits and ethical considerations.

Health profession students' perceptions of ChatGPT in healthcare and education: insights from a mixed-methods study

OBJECTIVE

The aim of this study was to investigate the perceptions of health profession students regarding ChatGPT use and the potential impact of integrating ChatGPT in healthcare and education.

BACKGROUND

Artificial Intelligence is increasingly utilized in medical education and clinical profession training. However, since its introduction, ChatGPT remains relatively unexplored in terms of health profession students' acceptance of its use in education and practice.

DESIGN

This study employed a mixed-methods approach, using a web-based survey.

METHODS

The study involved a convenience sample recruited through various methods, including Faculty of Medicine announcements, social media, and snowball sampling, during the second semester (March to June 2023). Data were collected using a structured questionnaire with closed-ended questions and three open-ended questions. The final sample comprised 217 undergraduate health profession students, including 73 (33.6%) nursing students, 65 (30.0%) medical students, and 79 (36.4%) occupational therapy, physiotherapy, and speech therapy students.

RESULTS

Among the surveyed students, 86.2% were familiar with ChatGPT, with generally positive perceptions as reflected by a mean score of 4.04 (SD = 0.62) on a scale of 1 to 5. Positive feedback was particularly noted with respect to ChatGPT's role in information retrieval and summarization. The qualitative data revealed three main themes: experiences with ChatGPT, its impact on the quality of healthcare, and its integration into the curriculum. The findings highlight benefits such as serving as a convenient tool for accessing information, reducing human errors, and fostering innovative learning approaches. However, they also underscore areas of concern, including ethical considerations, challenges in fostering critical thinking, and issues related to verification. The absence of significant differences between the different fields of study indicates consistent perceptions across nursing, medicine, and other health profession students.

CONCLUSIONS

Our findings underscore the necessity for continuous refinement to enhance ChatGPT's accuracy, reliability, and alignment with the diverse educational needs of health professions. These insights not only deepen our understanding of student perceptions of ChatGPT in healthcare education but also have significant implications for the future integration of AI in health profession practice. The study emphasizes the importance of a careful balance between leveraging the benefits of AI tools and addressing ethical and pedagogical concerns.

Assessing the Current Limitations of Large Language Models in Advancing Health Care Education

Unlabelled

The integration of large language models (LLMs), as seen with the generative pretrained transformers series, into health care education and clinical management represents a transformative potential. The practical use of current LLMs in health care sparks great anticipation for new avenues, yet its embracement also elicits considerable concerns that necessitate careful deliberation. This study aims to evaluate the application of state-of-the-art LLMs in health care education, highlighting the following shortcomings as areas requiring significant and urgent improvements: (1) threats to academic integrity, (2) dissemination of misinformation and risks of automation bias, (3) challenges with information completeness and consistency, (4) inequity of access, (5) risks of algorithmic bias, (6) exhibition of moral instability, (7) technological limitations in plugin tools, and (8) lack of regulatory oversight in addressing legal and ethical challenges. Future research should focus on strategically addressing the persistent challenges of LLMs highlighted in this paper, opening the door for effective measures that can improve their application in health care education.

ChatGPT is an Unreliable Source of Peer-Reviewed Information for Common Total Knee and Hip Arthroplasty Patient Questions

Background: Advances in artificial intelligence (AI), machine learning, and publicly accessible language model tools such as ChatGPT-3.5 continue to shape the landscape of modern medicine and patient education. ChatGPT's open access (OA), instant, human-sounding interface capable of carrying discussion on myriad topics makes it a potentially useful resource for patients seeking medical advice. As it pertains to orthopedic surgery, ChatGPT may become a source to answer common preoperative questions regarding total knee arthroplasty (TKA) and total hip arthroplasty (THA). Since ChatGPT can utilize the peer-reviewed literature to source its responses, this study seeks to characterize the validity of its responses to common TKA and THA questions and characterize the peer-reviewed literature that it uses to formulate its responses. Methods: Preoperative TKA and THA questions were formulated by fellowship-trained adult reconstruction surgeons based on common questions posed by patients in the clinical setting. Questions were inputted into ChatGPT with the initial request of using solely the peer-reviewed literature to generate its responses. The validity of each response was rated on a Likert scale by the fellowship-trained surgeons, and the sources utilized were characterized in terms of accuracy of comparison to existing publications, publication date, study design, level of evidence, journal of publication, journal impact factor based on the clarivate analytics factor tool, journal OA status, and whether the journal is based in the United States. Results: A total of 109 sources were cited by ChatGPT in its answers to 17 questions regarding TKA procedures and 16 THA procedures. Thirty-nine sources (36%) were deemed accurate or able to be directly traced to an existing publication. Of these, seven (18%) were identified as duplicates, yielding a total of 32 unique sources that were identified as accurate and further characterized. The most common characteristics of these sources included dates of publication between 2011 and 2015 (10), publication in The Journal of Bone and Joint Surgery (13), journal impact factors between 5.1 and 10.0 (17), internationally based journals (17), and journals that are not OA (28). The most common study designs were retrospective cohort studies and case series (seven each). The level of evidence was broadly distributed between Levels I, III, and IV (seven each). The averages for the Likert scales for medical accuracy and completeness were 4.4/6 and 1.92/3, respectively. Conclusions: Investigation into ChatGPT's response quality and use of peer-reviewed sources when prompted with archetypal pre-TKA and pre-THA questions found ChatGPT to provide mostly reliable responses based on fellowship-trained orthopedic surgeon review of 4.4/6 for accuracy and 1.92/3 for completeness despite a 64.22% rate of citing inaccurate references. This study suggests that until ChatGPT is proven to be a reliable source of valid information and references, patients must exercise extreme caution in directing their pre-TKA and THA questions to this medium.

Quantifying the Scope of Artificial Intelligence-Assisted Writing in Orthopaedic Medical Literature: An Analysis of Prevalence and Validation of AI-Detection Software

INTRODUCTION

The popularization of generative artificial intelligence (AI), including Chat Generative Pre-trained Transformer (ChatGPT), has raised concerns for the integrity of academic literature. This study asked the following questions: (1) Has the popularization of publicly available generative AI, such as ChatGPT, increased the prevalence of AI-generated orthopaedic literature? (2) Can AI detectors accurately identify ChatGPT-generated text? (3) Are there associations between article characteristics and the likelihood that it was AI generated?

METHODS

PubMed was searched across six major orthopaedic journals to identify articles received for publication after January 1, 2023. Two hundred and forty articles were randomly selected and entered into three popular AI detectors. Twenty articles published by each journal before the release of ChatGPT were randomly selected as negative control articles. 36 positive control articles (6 per journal) were created by altering 25%, 50%, and 100% of text from negative control articles using ChatGPT and were then used to validate each detector. The mean percentage of text detected as written by AI per detector was compared between pre-ChatGPT and post-ChatGPT release articles using independent t -test. Multivariate regression analysis was conducted using percentage AI-generated text per journal, article type (ie, cohort, clinical trial, review), and month of submission.

RESULTS

One AI detector consistently and accurately identified AI-generated text in positive control articles, whereas two others showed poor sensitivity and specificity. The most accurate detector showed a modest increase in the percentage AI detected for the articles received post release of ChatGPT (+1.8%, P = 0.01). Regression analysis showed no consistent associations between likelihood of AI-generated text per journal, article type, or month of submission.

CONCLUSIONS

As this study found an early, albeit modest, effect of generative AI on the orthopaedic literature, proper oversight will play a critical role in maintaining research integrity and accuracy. AI detectors may play a critical role in regulatory efforts, although they will require further development and standardization to the interpretation of their results.

The Year in Graduate Medical Education: Selected Highlights from 2023

This special article is the third in an annual series of the Journal of Cardiothoracic and Vascular Anesthesia that highlights significant literature from the world of graduate medical education published over the past year. Major themes addressed in this review include the potential uses and pitfalls of artificial intelligence in graduate medical education, trainee well-being and the rise of unionized house staff, the effect of gender and race/ethnicity on residency application and attrition rates, and the adoption of novel technologies in medical simulation and education. The authors thank the editorial board for again allowing us to draw attention to some of the more interesting work published in the field of graduate medical education during 2023. We hope that the readers find these highlights thought-provoking and informative as we all strive to successfully educate the next generation of anesthesiologists.

The impact of using ChatGPT on academic writing among medical undergraduates

BACKGROUND

ChatGPT is widely used for writing tasks, yet its effects on medical students' academic writing remain underexplored. This study aims to elucidate ChatGPT's impact on academic writing efficiency and quality among medical students, while also evaluating students' attitudes towards its use in academic writing.

METHODS

We collected systematic reviews from 130 third-year medical students and administered a questionnaire to assess ChatGPT usage and student attitudes. Three independent reviewers graded the papers using EASE guidelines, and statistical analysis compared articles generated with or without ChatGPT assistance across various parameters, with rigorous quality control ensuring survey reliability and validity.

RESULTS

In this study, 33 students (25.8%) utilized ChatGPT for writing (ChatGPT group) and 95 (74.2%) did not (Control group). The ChatGPT group exhibited significantly higher daily technology use and prior experience with ChatGPT (p < 0.05). Writing time was significantly reduced in the ChatGPT group (p = 0.04), with 69.7% completing tasks within 2-3 days compared to 48.4% in the control group. They also achieved higher article quality scores (p < 0.0001) with improvements in completeness, credibility, and scientific content. Self-assessment indicated enhanced writing skills (p < 0.01), confidence (p < 0.001), satisfaction (p < 0.001) and a positive attitude toward its future use in the ChatGPT group.

CONCLUSIONS

Integrating ChatGPT in medical academic writing, with proper guidance, improves efficiency and quality, illustrating artificial intelligence's potential in shaping medical education methodologies.

Does using artificial intelligence take the person out of personal statements? We can't tell

BACKGROUND

Use of artificial intelligence to generate personal statements for residency is currently not permitted but is difficult to monitor. This study sought to evaluate the ability of surgical residency application reviewers to identify artificial intelligence-generated personal statements and to understand perceptions of this practice.

METHODS

Three personal statements were generated using ChatGPT, and 3 were written by medical students who previously matched into surgery residency. Blinded participants at a single institution were instructed to read all personal statements and identify which were generated by artificial intelligence; they then completed a survey exploring their opinions regarding artificial intelligence use.

RESULTS

Of the 30 participants, 50% were faculty (n = 15) and 50% were residents (n = 15). Overall, experience ranged from 0 to 20 years (median, 2 years; interquartile range, 1-6.25 years). Artificial intelligence-derived personal statements were identified correctly only 59% of the time, with 3 (10%) participants identifying all the artificial intelligence-derived personal statements correctly. Artificial intelligence-generated personal statements were labeled as the best 60% of the time and the worst 43.3% of the time. When asked whether artificial intelligence use should be allowed in personal statements writing, 66.7% (n = 20) said no and 30% (n = 9) said yes. When asked if the use of artificial intelligence would impact their opinion of an applicant, 80% (n = 24) said yes, and 20% (n = 6) said no. When survey questions and ability to identify artificial intelligence-generated personal statements were evaluated by faculty/resident status and experience, no differences were noted (P > .05).

CONCLUSION

This study shows that surgical faculty and residents cannot reliably identify artificial intelligence-generated personal statements and that concerns exist regarding the impact of artificial intelligence on the application process.

Use of generative artificial intelligence (AI) in psychiatry and mental health care: a systematic review

OBJECTIVES

Tools based on generative artificial intelligence (AI) such as ChatGPT have the potential to transform modern society, including the field of medicine. Due to the prominent role of language in psychiatry, e.g., for diagnostic assessment and psychotherapy, these tools may be particularly useful within this medical field. Therefore, the aim of this study was to systematically review the literature on generative AI applications in psychiatry and mental health.

METHODS

We conducted a systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The search was conducted across three databases, and the resulting articles were screened independently by two researchers. The content, themes, and findings of the articles were qualitatively assessed.

RESULTS

The search and screening process resulted in the inclusion of 40 studies. The median year of publication was 2023. The themes covered in the articles were mainly mental health and well-being in general - with less emphasis on specific mental disorders (substance use disorder being the most prevalent). The majority of studies were conducted as prompt experiments, with the remaining studies comprising surveys, pilot studies, and case reports. Most studies focused on models that generate language, ChatGPT in particular.

CONCLUSIONS

Generative AI in psychiatry and mental health is a nascent but quickly expanding field. The literature mainly focuses on applications of ChatGPT, and finds that generative AI performs well, but notes that it is limited by significant safety and ethical concerns. Future research should strive to enhance transparency of methods, use experimental designs, ensure clinical relevance, and involve users/patients in the design phase.

Generative Artificial Intelligence: Applications in Scientific Writing and Data Analysis in Wound Healing Research

ABSTRACT

Generative artificial intelligence (AI) models are a new technological development with vast research use cases among medical subspecialties. These powerful large language models offer a wide range of possibilities in wound care, from personalized patient support to optimized treatment plans and improved scientific writing. They can also assist in efficiently navigating the literature and selecting and summarizing articles, enabling researchers to focus on impactful studies relevant to wound care management and enhancing response quality through prompt-learning iterations. For nonnative English-speaking medical practitioners and authors, generative AI may aid in grammar and vocabulary selection. Although reports have suggested limitations of the conversational agent on medical translation pertaining to the precise interpretation of medical context, when used with verified resources, this language model can breach language barriers and promote practice-changing advancements in global wound care. Further, AI-powered chatbots can enable continuous monitoring of wound healing progress and real-time insights into treatment responses through frequent, readily available remote patient follow-ups.However, implementing AI in wound care research requires careful consideration of potential limitations, especially in accurately translating complex medical terms and workflows. Ethical considerations are vital to ensure reliable and credible wound care research when using AI technologies. Although ChatGPT shows promise for transforming wound care management, the authors warn against overreliance on the technology. Considering the potential limitations and risks, proper validation and oversight are essential to unlock its true potential while ensuring patient safety and the effectiveness of wound care treatments.

Generative AI and large language models in nuclear medicine: current status and future prospects

This review explores the potential applications of Large Language Models (LLMs) in nuclear medicine, especially nuclear medicine examinations such as PET and SPECT, reviewing recent advancements in both fields. Despite the rapid adoption of LLMs in various medical specialties, their integration into nuclear medicine has not yet been sufficiently explored. We first discuss the latest developments in nuclear medicine, including new radiopharmaceuticals, imaging techniques, and clinical applications. We then analyze how LLMs are being utilized in radiology, particularly in report generation, image interpretation, and medical education. We highlight the potential of LLMs to enhance nuclear medicine practices, such as improving report structuring, assisting in diagnosis, and facilitating research. However, challenges remain, including the need for improved reliability, explainability, and bias reduction in LLMs. The review also addresses the ethical considerations and potential limitations of AI in healthcare. In conclusion, LLMs have significant potential to transform existing frameworks in nuclear medicine, making it a critical area for future research and development.

Assessing the Readability, Reliability, and Quality of AI-Modified and Generated Patient Education Materials for Endoscopic Skull Base Surgery

BACKGROUND

Despite National Institutes of Health and American Medical Association recommendations to publish online patient education materials at or below sixth-grade literacy, those pertaining to endoscopic skull base surgery (ESBS) have lacked readability and quality. ChatGPT is an artificial intelligence (AI) system capable of synthesizing vast internet data to generate responses to user queries but its utility in improving patient education materials has not been explored.

OBJECTIVE

To examine the current state of readability and quality of online patient education materials and determined the utility of ChatGPT for improving articles and generating patient education materials.

METHODS

An article search was performed utilizing 10 different search terms related to ESBS. The ten least readable existing patient-facing articles were modified with ChatGPT and iterative queries were used to generate an article de novo. The Flesch Reading Ease (FRE) and related metrics measured overall readability and content literacy level, while DISCERN assessed article reliability and quality.

RESULTS

Sixty-six articles were located. ChatGPT improved FRE readability of the 10 least readable online articles (19.7 ± 4.4 vs. 56.9 ± 5.9, p < 0.001), from university to 10th grade level. The generated article was more readable than 48.5% of articles (38.9 vs. 39.4 ± 12.4) and higher quality than 94% (51.0 vs. 37.6 ± 6.1). 56.7% of the online articles had "poor" quality.

CONCLUSIONS

ChatGPT improves the readability of articles, though most still remain above the recommended literacy level for patient education materials. With iterative queries, ChatGPT can generate more reliable and higher quality patient education materials compared to most existing online articles and can be tailored to match readability of average online articles.

Harnessing the Power of ChatGPT in Cardiovascular Medicine: Innovations, Challenges, and Future Directions

Cardiovascular diseases remain the leading cause of morbidity and mortality globally, posing significant challenges to public health. The rapid evolution of artificial intelligence (AI), particularly with large language models such as ChatGPT, has introduced transformative possibilities in cardiovascular medicine. This review examines ChatGPT's broad applications in enhancing clinical decision-making-covering symptom analysis, risk assessment, and differential diagnosis; advancing medical education for both healthcare professionals and patients; and supporting research and academic communication. Key challenges associated with ChatGPT, including potential inaccuracies, ethical considerations, data privacy concerns, and inherent biases, are discussed. Future directions emphasize improving training data quality, developing specialized models, refining AI technology, and establishing regulatory frameworks to enhance ChatGPT's clinical utility and mitigate associated risks. As cardiovascular medicine embraces AI, ChatGPT stands out as a powerful tool with substantial potential to improve therapeutic outcomes, elevate care quality, and advance research innovation. Fully understanding and harnessing this potential is essential for the future of cardiovascular health.

ChatGPT M.D.: Is there any room for generative AI in neurology?

ChatGPT, a general artificial intelligence, has been recognized as a powerful tool in scientific writing and programming but its use as a medical tool is largely overlooked. The general accessibility, rapid response time and comprehensive training database might enable ChatGPT to serve as a diagnostic augmentation tool in certain clinical settings. The diagnostic process in neurology is often challenging and complex. In certain time-sensitive scenarios, rapid evaluation and diagnostic decisions are needed, while in other cases clinicians are faced with rare disorders and atypical disease manifestations. Due to these factors, the diagnostic accuracy in neurology is often suboptimal. Here we evaluated whether ChatGPT can be utilized as a valuable and innovative diagnostic augmentation tool in various neurological settings. We used synthetic data generated by neurological experts to represent descriptive anamneses of patients with known neurology-related diseases, then the probability for an appropriate diagnosis made by ChatGPT was measured. To give clarity to the accuracy of the AI-determined diagnosis, all cases have been cross-validated by other experts and general medical doctors as well. We found that ChatGPT-determined diagnostic accuracy (ranging from 68.5% ± 3.28% to 83.83% ± 2.73%) can reach the accuracy of other experts (81.66% ± 2.02%), furthermore, it surpasses the probability of an appropriate diagnosis if the examiner is a general medical doctor (57.15% ± 2.64%). Our results showcase the efficacy of general artificial intelligence like ChatGPT as a diagnostic augmentation tool in medicine. In the future, AI-based supporting tools might be useful amendments in medical practice and help to improve the diagnostic process in neurology.

Artificial intelligence chatbots for the nutrition management of diabetes and the metabolic syndrome

BACKGROUND

Recently, there has been a growing interest in exploring AI-driven chatbots, such as ChatGPT, as a resource for disease management and education.

OBJECTIVE

The study aims to evaluate ChatGPT's accuracy and quality/clarity in providing nutritional management for Type 2 Diabetes (T2DM), the Metabolic syndrome (MetS) and its components, in accordance with the Academy of Nutrition and Dietetics' guidelines.

METHODS

Three nutrition management-related domains were considered: (1) Dietary management, (2) Nutrition care process (NCP) and (3) Menu planning (1500 kcal). A total of 63 prompts were used. Two experienced dietitians evaluated the chatbot output's concordance with the guidelines.

RESULTS

Both dietitians provided similar assessments for most conditions examined in the study. Gaps in the ChatGPT-derived outputs were identified and included weight loss recommendations, energy deficit, anthropometric assessment, specific nutrients of concern and the adoption of specific dietary interventions. Gaps in physical activity recommendations were also observed, highlighting ChatGPT's limitations in providing holistic lifestyle interventions. Within the NCP, the generated output provided incomplete examples of diagnostic documentation statements and had significant gaps in the monitoring and evaluation step. In the 1500 kcal one-day menus, the amounts of carbohydrates, fat, vitamin D and calcium were discordant with dietary recommendations. Regarding clarity, dietitians rated the output as either good or excellent.

CONCLUSION

Although ChatGPT is an increasingly available resource for practitioners, users are encouraged to consider the gaps identified in this study in the dietary management of T2DM and the MetS.

Precision of Chatbot Generative Pretrained Transformer Version 4-Generated References for Colon and Rectal Surgical Literature

INTRODUCTION

The objective is to assess the precision of references generated by Chatbot Generative Pretrained Transformer version 4 (ChatGPT-4) in scientific literature pertaining to colon and rectal surgery.

METHODS

Ten frequently studied keywords pertaining to colon and rectal surgery were chosen: colon cancer, rectal cancer, anal cancer, total neoadjuvant therapy, diverticulitis, low anterior resection, transanal minimally invasive surgery, ileal pouch anal anastomosis, abdominoperineal resection, and hemorrhoidectomy. ChatGPT-4 was prompted to search for the most representative citations for all keywords. After this, two separate evaluators meticulously examined the outcomes each key element, awarding full accuracy to generated citations in which there was no discrepancies in any of the fields when cross-referenced with the Scopus, Google, and PubMed databases. References from ChatGPT-4 underwent a thorough review process, which involved careful examination of key elements such as the article title, authors, journal name, publication year, and Digital Object Identifier (DOI).

RESULTS

Forty-one of the 100 references generated by were fully accurate; however, but none included a DOI. Partial accuracy was observed in 67 of the references, which were identifiable by title and journal. Performance varied across specific keywords; for example, references for colon and rectal cancer were 100% identifiable by title and journal, but no term had 100% accuracy across all categories. Notably, none of the generated references correctly listed all authors. Conducted within a short timeframe during which ChatGPT4 is rapidly evolving and updating its knowledge base.

CONCLUSIONS

While ChatGPT-4 offers improvements over its predecessors and shows potential for use in academic literature, its inconsistent performance across categories, lack of DOIs, and irregularities in authorship listings raise concerns about its readiness for application in the field of colon and rectal surgery research.

Large language models and artificial intelligence chatbots in vascular surgery

Natural language processing is a subfield of artificial intelligence that aims to analyze human oral or written language. The development of large language models has brought innovative perspectives in medicine, including the potential use of chatbots and virtual assistants. Nevertheless, the benefits and pitfalls of such technology need to be carefully evaluated before their use in health care. The aim of this narrative review was to provide an overview of potential applications of large language models and artificial intelligence chatbots in the field of vascular surgery, including clinical practice, research, and education. In light of the results, we discuss current limits and future directions.

The role of artificial intelligence in cosmetic and functional gynecology: Stepping into the third millennium

Cosmetic and functional gynecology have gained popularity among patients, but the scientific literature in this field, particularly regarding the cosmetic aspect, is lacking. The use of evidence-based medicine is crucial to validate diagnostic tools and treatment protocols. However, the advent of artificial intelligence (AI) offers a promising solution to address this issue. ChatGPT, a sophisticated language model, can revolutionize AI in medicine, enabling accurate diagnosis, personalized treatment plans, and expedited research analysis. Cosmetic and functional gynecology can leverage AI to develop the field and improve evidence gathering. AI can aid in precise and personalized diagnosis, implement standardized assessment tools, simulate treatment outcomes, and assess under-skin anatomy through virtual reality. AI tools can assist clinicians in diagnosing and comparing difficult cases, calculate treatment risks, and contribute to standardization by collecting global evidence and generating guidelines. The use of AI in cosmetic and functional gynecology holds significant potential to advance the field and improve patient outcomes. This novel combination of AI and gynecology represents a groundbreaking development in medicine, emphasizing the importance of appropriate and correct AI usage.

Assessing ChatGPT's ability to emulate human reviewers in scientific research: A descriptive and qualitative approach

BACKGROUND

ChatGPT is an AI platform whose relevance in the peer review of scientific articles is steadily growing. Nonetheless, it has sparked debates over its potential biases and inaccuracies. This study aims to assess ChatGPT's ability to qualitatively emulate human reviewers in scientific research.

METHODS

We included the first submitted version of the latest twenty original research articles published by the 3rd of July 2023, in a high-profile medical journal. Each article underwent evaluation by a minimum of three human reviewers during the initial review stage. Subsequently, three researchers with medical backgrounds and expertise in manuscript revision, independently and qualitatively assessed the agreement between the peer reviews generated by ChatGPT version GPT-4 and the comments provided by human reviewers for these articles. The level of agreement was categorized into complete, partial, none, or contradictory.

RESULTS

720 human reviewers' comments were assessed. There was a good agreement between the three assessors (Overall kappa >0.6). ChatGPT's comments demonstrated complete agreement in terms of quality and substance with 48 (6.7 %) human reviewers' comments, partially agreed with 92 (12.8 %), identifying issues necessitating further elaboration or recommending supplementary steps to address concerns, had no agreement with a significant 565 (78.5 %), and contradicted 15 (2.1 %). ChatGPT comments on methods had the lowest proportion of complete agreement (13 comments, 3.6 %), while general comments on the manuscript displayed the highest proportion of complete agreement (17 comments, 22.1 %).

CONCLUSION

ChatGPT version GPT-4 has a limited ability to emulate human reviewers within the peer review process of scientific research.

A publishing infrastructure for Artificial Intelligence (AI)-assisted academic authoring

OBJECTIVE

Investigate the use of advanced natural language processing models to streamline the time-consuming process of writing and revising scholarly manuscripts.

MATERIALS AND METHODS

For this purpose, we integrate large language models into the Manubot publishing ecosystem to suggest revisions for scholarly texts. Our AI-based revision workflow employs a prompt generator that incorporates manuscript metadata into templates, generating section-specific instructions for the language model. The model then generates revised versions of each paragraph for human authors to review. We evaluated this methodology through 5 case studies of existing manuscripts, including the revision of this manuscript.

RESULTS

Our results indicate that these models, despite some limitations, can grasp complex academic concepts and enhance text quality. All changes to the manuscript are tracked using a version control system, ensuring transparency in distinguishing between human- and machine-generated text.

CONCLUSIONS

Given the significant time researchers invest in crafting prose, incorporating large language models into the scholarly writing process can significantly improve the type of knowledge work performed by academics. Our approach also enables scholars to concentrate on critical aspects of their work, such as the novelty of their ideas, while automating tedious tasks like adhering to specific writing styles. Although the use of AI-assisted tools in scientific authoring is controversial, our approach, which focuses on revising human-written text and provides change-tracking transparency, can mitigate concerns regarding AI's role in scientific writing.

Transforming Health Care Through Chatbots for Medical History-Taking and Future Directions: Comprehensive Systematic Review

BACKGROUND

The integration of artificial intelligence and chatbot technology in health care has attracted significant attention due to its potential to improve patient care and streamline history-taking. As artificial intelligence-driven conversational agents, chatbots offer the opportunity to revolutionize history-taking, necessitating a comprehensive examination of their impact on medical practice.

OBJECTIVE

This systematic review aims to assess the role, effectiveness, usability, and patient acceptance of chatbots in medical history-taking. It also examines potential challenges and future opportunities for integration into clinical practice.

METHODS

A systematic search included PubMed, Embase, MEDLINE (via Ovid), CENTRAL, Scopus, and Open Science and covered studies through July 2024. The inclusion and exclusion criteria for the studies reviewed were based on the PICOS (participants, interventions, comparators, outcomes, and study design) framework. The population included individuals using health care chatbots for medical history-taking. Interventions focused on chatbots designed to facilitate medical history-taking. The outcomes of interest were the feasibility, acceptance, and usability of chatbot-based medical history-taking. Studies not reporting on these outcomes were excluded. All study designs except conference papers were eligible for inclusion. Only English-language studies were considered. There were no specific restrictions on study duration. Key search terms included "chatbot*," "conversational agent*," "virtual assistant," "artificial intelligence chatbot," "medical history," and "history-taking." The quality of observational studies was classified using the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) criteria (eg, sample size, design, data collection, and follow-up). The RoB 2 (Risk of Bias) tool assessed areas and the levels of bias in randomized controlled trials (RCTs).

RESULTS

The review included 15 observational studies and 3 RCTs and synthesized evidence from different medical fields and populations. Chatbots systematically collect information through targeted queries and data retrieval, improving patient engagement and satisfaction. The results show that chatbots have great potential for history-taking and that the efficiency and accessibility of the health care system can be improved by 24/7 automated data collection. Bias assessments revealed that of the 15 observational studies, 5 (33%) studies were of high quality, 5 (33%) studies were of moderate quality, and 5 (33%) studies were of low quality. Of the RCTs, 2 had a low risk of bias, while 1 had a high risk.

CONCLUSIONS

This systematic review provides critical insights into the potential benefits and challenges of using chatbots for medical history-taking. The included studies showed that chatbots can increase patient engagement, streamline data collection, and improve health care decision-making. For effective integration into clinical practice, it is crucial to design user-friendly interfaces, ensure robust data security, and maintain empathetic patient-physician interactions. Future research should focus on refining chatbot algorithms, improving their emotional intelligence, and extending their application to different health care settings to realize their full potential in modern medicine.

TRIAL REGISTRATION

PROSPERO CRD42023410312; www.crd.york.ac.uk/prospero.

Effectiveness of ChatGPT 4.0 in Telemedicine-Based Management of Metastatic Prostate Carcinoma

The recent rise in telemedicine, notably during the COVID-19 pandemic, highlights the potential of integrating artificial intelligence tools in healthcare. This study assessed the effectiveness of ChatGPT versus medical oncologists in the telemedicine-based management of metastatic prostate cancer. In this retrospective study, 102 patients who met inclusion criteria were analyzed to compare the competencies of ChatGPT and oncologists in telemedicine consultations. ChatGPT's role in pre-charting and determining the need for in-person consultations was evaluated. The primary outcome was the concordance between ChatGPT and oncologists in treatment decisions. Results showed a moderate concordance (Cohen's Kappa = 0.43, p < 0.001). The number of diagnoses made by both parties was not significantly different (median number of diagnoses: 5 vs. 5, p = 0.12). In conclusion, ChatGPT exhibited moderate agreement with oncologists in management via telemedicine, indicating the need for further research to explore its healthcare applications.

Harnessing Large Language Models for Structured Reporting in Breast Ultrasound: A Comparative Study of Open AI (GPT-4.0) and Microsoft Bing (GPT-4)

OBJECTIVES

To assess the capabilities of large language models (LLMs), including Open AI (GPT-4.0) and Microsoft Bing (GPT-4), in generating structured reports, the Breast Imaging Reporting and Data System (BI-RADS) categories, and management recommendations from free-text breast ultrasound reports.

MATERIALS AND METHODS

In this retrospective study, 100 free-text breast ultrasound reports from patients who underwent surgery between January and May 2023 were gathered. The capabilities of Open AI (GPT-4.0) and Microsoft Bing (GPT-4) to convert these unstructured reports into structured ultrasound reports were studied. The quality of structured reports, BI-RADS categories, and management recommendations generated by GPT-4.0 and Bing were evaluated by senior radiologists based on the guidelines.

RESULTS

Open AI (GPT-4.0) was better than Microsoft Bing (GPT-4) in terms of performance in generating structured reports (88% vs. 55%; p < 0.001), giving correct BI-RADS categories (54% vs. 47%; p = 0.013) and providing reasonable management recommendations (81% vs. 63%; p < 0.001). As the ability to predict benign and malignant characteristics, GPT-4.0 performed significantly better than Bing (AUC, 0.9317 vs. 0.8177; p < 0.001), while both performed significantly inferior to senior radiologists (AUC, 0.9763; both p < 0.001).

CONCLUSION

This study highlights the potential of LLMs, specifically Open AI (GPT-4.0), in converting unstructured breast ultrasound reports into structured ones, offering accurate diagnoses and providing reasonable recommendations.

A qualitative analysis of Chinese higher education students' intentions and influencing factors in using ChatGPT: a grounded theory approach

The emergence of ChatGPT has significantly impacted the field of education. While much of the existing research has predominantly examined the theoretical implications of ChatGPT, there is a notable absence of empirical studies substantiating these claims. As pivotal stakeholders in education and primary users of ChatGPT, exploring the willingness and influencing factors of higher education students to use ChatGPT can offer valuable insights into the real-world needs of student users. This, in turn, can serve as a foundation for empowering education with intelligent technologies in the future. This study focuses specifically on the demographic of students in Chinese higher education who have utilized ChatGPT. Using semi-structured interviews and grounded theory methodology, we aim to comprehensively understand the extent to which students embrace new technologies. Our objective is to elucidate the behavioral inclinations and influencing factors of student users. The findings of this study will contribute practical insights for refining policy frameworks, expanding the dissemination of quality resources, optimizing and upgrading products for an enhanced user experience, and fostering higher-order thinking skills to adeptly navigate evolving technological landscapes. In conclusion, this research endeavors to bridge the gap between theoretical discussions and practical applications.

Ethical Considerations and Fundamental Principles of Large Language Models in Medical Education: Viewpoint

This viewpoint article first explores the ethical challenges associated with the future application of large language models (LLMs) in the context of medical education. These challenges include not only ethical concerns related to the development of LLMs, such as artificial intelligence (AI) hallucinations, information bias, privacy and data risks, and deficiencies in terms of transparency and interpretability but also issues concerning the application of LLMs, including deficiencies in emotional intelligence, educational inequities, problems with academic integrity, and questions of responsibility and copyright ownership. This paper then analyzes existing AI-related legal and ethical frameworks and highlights their limitations with regard to the application of LLMs in the context of medical education. To ensure that LLMs are integrated in a responsible and safe manner, the authors recommend the development of a unified ethical framework that is specifically tailored for LLMs in this field. This framework should be based on 8 fundamental principles: quality control and supervision mechanisms; privacy and data protection; transparency and interpretability; fairness and equal treatment; academic integrity and moral norms; accountability and traceability; protection and respect for intellectual property; and the promotion of educational research and innovation. The authors further discuss specific measures that can be taken to implement these principles, thereby laying a solid foundation for the development of a comprehensive and actionable ethical framework. Such a unified ethical framework based on these 8 fundamental principles can provide clear guidance and support for the application of LLMs in the context of medical education. This approach can help establish a balance between technological advancement and ethical safeguards, thereby ensuring that medical education can progress without compromising the principles of fairness, justice, or patient safety and establishing a more equitable, safer, and more efficient environment for medical education.

An Artificial Intelligence Chatbot is an Accurate and Useful Online Patient Resource Prior to Total Knee Arthroplasty

BACKGROUND

Online information is a useful resource for patients seeking advice on their orthopaedic care. While traditional websites provide responses to specific frequently asked questions (FAQs), sophisticated artificial intelligence tools may be able to provide the same information to patients in a more accessible manner. Chat Generative Pretrained Transformer (ChatGPT) is a powerful artificial intelligence chatbot that has been shown to effectively draw on its large reserves of information in a conversational context with a user. The purpose of this study was to assess the accuracy and reliability of ChatGPT-generated responses to FAQs regarding total knee arthroplasty.

METHODS

We distributed a survey that challenged arthroplasty surgeons to identify which of the 2 responses to FAQs on our institution's website was human-written and which was generated by ChatGPT. All questions were total knee arthroplasty-related. The second portion of the survey investigated the potential to further leverage ChatGPT to assist with translation and accessibility as a means to better meet the needs of our diverse patient population.

RESULTS

Surgeons correctly identified the ChatGPT-generated responses 4 out of 10 times on average (range: 0 to 7). No consensus was reached on any of the responses to the FAQs. Additionally, over 90% of our surgeons strongly encouraged the use of ChatGPT to more effectively accommodate the diverse patient populations that seek information from our hospital's online resources.

CONCLUSIONS

ChatGPT provided accurate, reliable answers to our website's FAQs. Surgeons also agreed that ChatGPT's ability to provide targeted, language-specific responses to FAQs would be of benefit to our diverse patient population.

Based on Medicine, The Now and Future of Large Language Models

OBJECTIVES

This review explores the potential applications of large language models (LLMs) such as ChatGPT, GPT-3.5, and GPT-4 in the medical field, aiming to encourage their prudent use, provide professional support, and develop accessible medical AI tools that adhere to healthcare standards.

METHODS

This paper examines the impact of technologies such as OpenAI's Generative Pre-trained Transformers (GPT) series, including GPT-3.5 and GPT-4, and other large language models (LLMs) in medical education, scientific research, clinical practice, and nursing. Specifically, it includes supporting curriculum design, acting as personalized learning assistants, creating standardized simulated patient scenarios in education; assisting with writing papers, data analysis, and optimizing experimental designs in scientific research; aiding in medical imaging analysis, decision-making, patient education, and communication in clinical practice; and reducing repetitive tasks, promoting personalized care and self-care, providing psychological support, and enhancing management efficiency in nursing.

RESULTS

LLMs, including ChatGPT, have demonstrated significant potential and effectiveness in the aforementioned areas, yet their deployment in healthcare settings is fraught with ethical complexities, potential lack of empathy, and risks of biased responses.

CONCLUSION

Despite these challenges, significant medical advancements can be expected through the proper use of LLMs and appropriate policy guidance. Future research should focus on overcoming these barriers to ensure the effective and ethical application of LLMs in the medical field.

Impact of Large Language Models on Medical Education and Teaching Adaptations

Unlabelled

This viewpoint article explores the transformative role of large language models (LLMs) in the field of medical education, highlighting their potential to enhance teaching quality, promote personalized learning paths, strengthen clinical skills training, optimize teaching assessment processes, boost the efficiency of medical research, and support continuing medical education. However, the use of LLMs entails certain challenges, such as questions regarding the accuracy of information, the risk of overreliance on technology, a lack of emotional recognition capabilities, and concerns related to ethics, privacy, and data security. This article emphasizes that to maximize the potential of LLMs and overcome these challenges, educators must exhibit leadership in medical education, adjust their teaching strategies flexibly, cultivate students' critical thinking, and emphasize the importance of practical experience, thus ensuring that students can use LLMs correctly and effectively. By adopting such a comprehensive and balanced approach, educators can train health care professionals who are proficient in the use of advanced technologies and who exhibit solid professional ethics and practical skills, thus laying a strong foundation for these professionals to overcome future challenges in the health care sector.

Can ChatGPT Fool the Match? Artificial Intelligence Personal Statements for Plastic Surgery Residency Applications: A Comparative Study

Introduction: Personal statements can be decisive in Canadian residency applications. With the rise in AI technology, ethical concerns regarding authenticity and originality become more pressing. This study explores the capability of ChatGPT in producing personal statements for plastic surgery residency that match the quality of statements written by successful applicants. Methods: ChatGPT was utilized to generate a cohort of personal statements for CaRMS (Canadian Residency Matching Service) to compare with previously successful Plastic Surgery applications. Each AI-generated and human-written statement was randomized and anonymized prior to assessment. Two retired members of the plastic surgery residency selection committee from the University of British Columbia, evaluated these on a 0 to 10 scale and provided a binary response judging whether each statement was AI or human written. Statistical analysis included Welch 2-sample t tests and Cohen's Kappa for agreement. Results: Twenty-two personal statements (11 AI-generated by ChatGPT and 11 human-written) were evaluated. The overall mean scores were 7.48 (SD 0.932) and 7.68 (SD 0.716), respectively, with no significant difference between AI and human groups (P = .4129). The average accuracy in distinguishing between human and AI letters was 65.9%. The Cohen's Kappa value was 0.374. Conclusions: ChatGPT can generate personal statements for plastic surgery residency applications with quality indistinguishable from human-written counterparts, as evidenced by the lack of significant scoring difference and moderate accuracy in discrimination by experienced surgeons. These findings highlight the evolving role of AI and the need for updated evaluative criteria or guidelines in the residency application process.

The Ability of ChatGPT in Paraphrasing Texts and Reducing Plagiarism: A Descriptive Analysis

Background

The introduction of ChatGPT by OpenAI has garnered significant attention. Among its capabilities, paraphrasing stands out.

Objective

This study aims to investigate the satisfactory levels of plagiarism in the paraphrased text produced by this chatbot.

Methods

Three texts of varying lengths were presented to ChatGPT. ChatGPT was then instructed to paraphrase the provided texts using five different prompts. In the subsequent stage of the study, the texts were divided into separate paragraphs, and ChatGPT was requested to paraphrase each paragraph individually. Lastly, in the third stage, ChatGPT was asked to paraphrase the texts it had previously generated.

Results

The average plagiarism rate in the texts generated by ChatGPT was 45% (SD 10%). ChatGPT exhibited a substantial reduction in plagiarism for the provided texts (mean difference -0.51, 95% CI -0.54 to -0.48; P<.001). Furthermore, when comparing the second attempt with the initial attempt, a significant decrease in the plagiarism rate was observed (mean difference -0.06, 95% CI -0.08 to -0.03; P<.001). The number of paragraphs in the texts demonstrated a noteworthy association with the percentage of plagiarism, with texts consisting of a single paragraph exhibiting the lowest plagiarism rate (P<.001).

Conclusions

Although ChatGPT demonstrates a notable reduction of plagiarism within texts, the existing levels of plagiarism remain relatively high. This underscores a crucial caution for researchers when incorporating this chatbot into their work.

The impact of large language models on radiology: a guide for radiologists on the latest innovations in AI

The advent of Deep Learning (DL) has significantly propelled the field of diagnostic radiology forward by enhancing image analysis and interpretation. The introduction of the Transformer architecture, followed by the development of Large Language Models (LLMs), has further revolutionized this domain. LLMs now possess the potential to automate and refine the radiology workflow, extending from report generation to assistance in diagnostics and patient care. The integration of multimodal technology with LLMs could potentially leapfrog these applications to unprecedented levels.However, LLMs come with unresolved challenges such as information hallucinations and biases, which can affect clinical reliability. Despite these issues, the legislative and guideline frameworks have yet to catch up with technological advancements. Radiologists must acquire a thorough understanding of these technologies to leverage LLMs' potential to the fullest while maintaining medical safety and ethics. This review aims to aid in that endeavor.