Exploring the potential of ChatGPT in the peer review process: An observational study

The role of large language models in the peer-review process: opportunities and challenges for medical journal reviewers and editors

The peer review process ensures the integrity of scientific research. This is particularly important in the medical field, where research findings directly impact patient care. However, the rapid growth of publications has strained reviewers, causing delays and potential declines in quality. Generative artificial intelligence, especially large language models (LLMs) such as ChatGPT, may assist researchers with efficient, high-quality reviews. This review explores the integration of LLMs into peer review, highlighting their strengths in linguistic tasks and challenges in assessing scientific validity, particularly in clinical medicine. Key points for integration include initial screening, reviewer matching, feedback support, and language review. However, implementing LLMs for these purposes will necessitate addressing biases, privacy concerns, and data confidentiality. We recommend using LLMs as complementary tools under clear guidelines to support, not replace, human expertise in maintaining rigorous peer review standards.

Assessing the Impact of ChatGPT in Dermatology: A Comprehensive Rapid Review

Background/Objectives: The use of artificial intelligence (AI) in dermatology is expanding rapidly, with ChatGPT, a large language model (LLM) from OpenAI, showing promise in patient education, clinical decision-making, and teledermatology. Despite its potential, the ethical, clinical, and practical implications of its application remain insufficiently explored. This study aims to evaluate the effectiveness, challenges, and future prospects of ChatGPT in dermatology, focusing on clinical applications, patient interactions, and medical writing. ChatGPT was selected due to its broad adoption, extensive validation, and strong performance in dermatology-related tasks. Methods: A thorough literature review was conducted, focusing on publications related to ChatGPT and dermatology. The search included articles in English from November 2022 to August 2024, as this period captures the most recent developments following the launch of ChatGPT in November 2022, ensuring that the review includes the latest advancements and discussions on its role in dermatology. Studies were chosen based on their relevance to clinical applications, patient interactions, and ethical issues. Descriptive metrics, such as average accuracy scores and reliability percentages, were used to summarize study characteristics, and key findings were analyzed. Results: ChatGPT has shown significant potential in passing dermatology specialty exams and providing reliable responses to patient queries, especially for common dermatological conditions. However, it faces limitations in diagnosing complex cases like cutaneous neoplasms, and concerns about the accuracy and completeness of its information persist. Ethical issues, including data privacy, algorithmic bias, and the need for transparent guidelines, were identified as critical challenges. Conclusions: While ChatGPT has the potential to significantly enhance dermatological practice, particularly in patient education and teledermatology, its integration must be cautious, addressing ethical concerns and complementing, rather than replacing, dermatologist expertise. Future research should refine ChatGPT's diagnostic capabilities, mitigate biases, and develop comprehensive clinical guidelines.

Status of peer review guidelines in international surgical journals: A cross‐sectional survey

Aim

To gain insight into the current status of peer review guidelines in international surgical journals and to offer guidance for the development of peer review guidelines for surgical journals.

Methods

We selected the top 100 journals in the category of ‘Surgery’ according to the Journal Citation Report 2021. We conducted a search of the websites of these journals, and Web of Science, PubMed, other databases, in order to gather the peer review guidelines published by these top 100 journals up until June 30, 2022. Additionally, we analysed the contents of these peer review guidelines.

Results

Only 52% (52/100) of journals provided guidelines for reviewers. Sixteen peer review guidelines which were published by these 52 surgical journals were included in this study. The contents of these peer review guidelines were classified into 33 items. The most common item was research methodology, which was mentioned by 13 journals (25%, 13/52). Other important items include statistical methodology, mentioned by 11 journals (21.2%), the rationality of figures, tables, and data, mentioned by 11 journals (21.2%), innovation of research, mentioned by nine journals (17.3%), and language expression, readability of papers, ethical review, references, and so forth, mentioned by eight journals (15.4%). Two journals described items for quality assessment of peer review. Forty‐three journals offered a checklist to guide reviewers on how to write a review report. Some surgical journals developed peer review guidelines for reviewers with different academic levels, such as professional reviewers and patient/public reviewers. Additionally, some surgical journals provided specific items for different types of papers, such as original articles, reviews, surgical videos, surgical database research, surgery‐related outcome measurements, and case reports in their peer review guidelines.

Conclusions

Key contents of peer review guidelines for the reviewers of surgical journals not only include items relating to reviewing research methodology, statistical methods, figures, tables and data, research innovation, ethical review, but also cover items concerning reviewing surgical videos, surgical database research, surgery‐related outcome measurements, instructions on how to write a review report, and guidelines on how to assess quality of peer review.

The transformative impact of large language models on medical writing and publishing: current applications, challenges and future directions

Large language models (LLMs) are rapidly transforming medical writing and publishing. This review article focuses on experimental evidence to provide a comprehensive overview of the current applications, challenges, and future implications of LLMs in various stages of academic research and publishing process. Global surveys reveal a high prevalence of LLM usage in scientific writing, with both potential benefits and challenges associated with its adoption. LLMs have been successfully applied in literature search, research design, writing assistance, quality assessment, citation generation, and data analysis. LLMs have also been used in peer review and publication processes, including manuscript screening, generating review comments, and identifying potential biases. To ensure the integrity and quality of scholarly work in the era of LLM-assisted research, responsible artificial intelligence (AI) use is crucial. Researchers should prioritize verifying the accuracy and reliability of AI-generated content, maintain transparency in the use of LLMs, and develop collaborative human-AI workflows. Reviewers should focus on higher-order reviewing skills and be aware of the potential use of LLMs in manuscripts. Editorial offices should develop clear policies and guidelines on AI use and foster open dialogue within the academic community. Future directions include addressing the limitations and biases of current LLMs, exploring innovative applications, and continuously updating policies and practices in response to technological advancements. Collaborative efforts among stakeholders are necessary to harness the transformative potential of LLMs while maintaining the integrity of medical writing and publishing.

Effectiveness of AI-powered Chatbots in responding to orthopaedic postgraduate exam questions-an observational study

PURPOSE

This study analyses the performance and proficiency of the three Artificial Intelligence (AI) generative chatbots (ChatGPT-3.5, ChatGPT-4.0, Bard Google AI®) and in answering the Multiple Choice Questions (MCQs) of postgraduate (PG) level orthopaedic qualifying examinations.

METHODS

A series of 120 mock Single Best Answer' (SBA) MCQs with four possible options named A, B, C and D as answers on various musculoskeletal (MSK) conditions covering Trauma and Orthopaedic curricula were compiled. A standardised text prompt was used to generate and feed ChatGPT (both 3.5 and 4.0 versions) and Google Bard programs, which were then statistically analysed.

RESULTS

Significant differences were found between responses from Chat GPT 3.5 with Chat GPT 4.0 (Chi square = 27.2, P < 0.001) and on comparing both Chat GPT 3.5 (Chi square = 63.852, P < 0.001) with Chat GPT 4.0 (Chi square = 44.246, P < 0.001) with. Bard Google AI® had 100% efficiency and was significantly more efficient than both Chat GPT 3.5 with Chat GPT 4.0 (p < 0.0001).

CONCLUSION

The results demonstrate the variable potential of the different AI generative chatbots (Chat GPT 3.5, Chat GPT 4.0 and Bard Google) in their ability to answer the MCQ of PG-level orthopaedic qualifying examinations. Bard Google AI® has shown superior performance than both ChatGPT versions, underlining the potential of such large language processing models in processing and applying orthopaedic subspecialty knowledge at a PG level.