Speeding up to keep up: exploring the use of AI in the research process

How much can we rely on artificial intelligence chatbots such as the ChatGPT software program to assist with scientific writing?

STATEMENT OF PROBLEM: Use of the ChatGPT software program by authors raises many questions, primarily regarding egregious issues such as plagiarism. Nevertheless, little is known about the extent to which artificial intelligence (AI) models can produce high-quality research publications and advance and shape the direction of a research topic.

PURPOSE

The purpose of this study was to determine how well the ChatGPT software program, a writing tool powered by AI, could respond to questions about scientific or research writing and generate accurate references with academic examples.

MATERIAL AND METHODS

Questions were made for the ChatGPT software program to locate an abstract containing a particular keyword in the Journal of Prosthetic Dentistry (JPD). Then, whether the resulting articles existed or were published was determined. Questions were made for the algorithm 5 times to locate 5 JPD articles containing 2 specific keywords, bringing the total number of articles to 25. The process was repeated twice, each time with a different set of keywords, and the ChatGPT software program provided a total of 75 articles. The search was conducted at various times between April 1 and 4, 2023. Finally, 2 authors independently searched the JPD website and Google Scholar to determine whether the articles provided by the ChatGPT software program existed.

RESULTS

When the author tested the ChatGPT software program's ability to locate articles in the JPD and Google Scholar using a set of keywords, the results did not match the papers that the ChatGPT software program had generated with the help of the AI tool. Consequently, all 75 articles provided by the ChatGPT software program were not accurately located in the JPD or Google Scholar databases and had to be added manually to ensure the accuracy of the relevant references.

CONCLUSIONS

Researchers and academic scholars must be cautious when using the ChatGPT software program because AI-generated content cannot provide or analyze the same information as an author or researcher. In addition, the results indicated that writing credit or references to such content or references in prestigious academic journals is not yet appropriate. At this time, scientific writing is only valid when performed manually by researchers.

Artificial intelligence application in counselling practices. A multigroup analysis of acceptance and awareness using gender and professional rank

Introduction

Artificial intelligence (AI) has emerged as a transformative tool in various professional domains, including counselling, where it offers innovative ways to enhance service delivery and client outcomes. Despite its potential, research on AI in counselling practices often focuses on its technical applications, with limited attention to the interplay between awareness, acceptance, and application. This study analyses how professional counsellors apply artificial intelligence in counselling practices using the nexus between awareness and application through acceptance of AI with gender and professional rank as group.

Method

A total of 5,432 professional counsellors were selected for the study. Data collection was conducted online to ensure a wide reach. The research instruments underwent validity checks, demonstrating high content and factorial validity. Convergent and discriminant validity were confirmed using the Average Variance Extracted (AVE) and Fornel-Larcker criterion.

Results

The findings revealed that professional counsellors exhibited high levels of awareness, acceptability, and application of AI in their counselling practices. Acceptance played a positive mediating role in the relationship between awareness and application. However, male practitioners and professors displayed stronger awareness, acceptance, and application of AI tools compared to their counterparts.

Conclusion

The study highlights the significant role of acceptance in bridging awareness and application of AI in counselling practices. It underscores the importance of addressing gender and professional rank disparities to ensure equitable adoption and utilization of AI tools. The findings offer valuable insights for policymakers in promoting the integration of AI in counselling to enhance professional practices.

Applications of Large Language Models in the Field of Suicide Prevention: Scoping Review

BACKGROUND

Prevention of suicide is a global health priority. Approximately 800,000 individuals die by suicide yearly, and for every suicide death, there are another 20 estimated suicide attempts. Large language models (LLMs) hold the potential to enhance scalable, accessible, and affordable digital services for suicide prevention and self-harm interventions. However, their use also raises clinical and ethical questions that require careful consideration.

OBJECTIVE

This scoping review aims to identify emergent trends in LLM applications in the field of suicide prevention and self-harm research. In addition, it summarizes key clinical and ethical considerations relevant to this nascent area of research.

METHODS

Searches were conducted in 4 databases (PsycINFO, Embase, PubMed, and IEEE Xplore) in February 2024. Eligible studies described the application of LLMs for suicide or self-harm prevention, detection, or management. English-language peer-reviewed articles and conference proceedings were included, without date restrictions. Narrative synthesis was used to synthesize study characteristics, objectives, models, data sources, proposed clinical applications, and ethical considerations. This review adhered to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) standards.

RESULTS

Of the 533 studies identified, 36 (6.8%) met the inclusion criteria. An additional 7 studies were identified through citation chaining, resulting in 43 studies for review. The studies showed a bifurcation of publication fields, with varying publication norms between computer science and mental health. While most of the studies (33/43, 77%) focused on identifying suicide risk, newer applications leveraging generative functions (eg, support, education, and training) are emerging. Social media was the most common source of LLM training data. Bidirectional Encoder Representations from Transformers (BERT) was the predominant model used, although generative pretrained transformers (GPTs) featured prominently in generative applications. Clinical LLM applications were reported in 60% (26/43) of the studies, often for suicide risk detection or as clinical assistance tools. Ethical considerations were reported in 33% (14/43) of the studies, with privacy, confidentiality, and consent strongly represented.

CONCLUSIONS

This evolving research area, bridging computer science and mental health, demands a multidisciplinary approach. While open access models and datasets will likely shape the field of suicide prevention, documenting their limitations and potential biases is crucial. High-quality training data are essential for refining these models and mitigating unwanted biases. Policies that address ethical concerns-particularly those related to privacy and security when using social media data-are imperative. Limitations include high variability across disciplines in how LLMs and study methodology are reported. The emergence of generative artificial intelligence signals a shift in approach, particularly in applications related to care, support, and education, such as improved crisis care and gatekeeper training methods, clinician copilot models, and improved educational practices. Ongoing human oversight-through human-in-the-loop testing or expert external validation-is essential for responsible development and use.

TRIAL REGISTRATION

OSF Registries osf.io/nckq7; https://osf.io/nckq7.

A scoping review on what constitutes a good research culture

Background

The crisis in research culture is well documented, covering issues such as a tendency for quantity over quality, unhealthy competitive environments, and assessment based on publications, journal prestige and funding. In response, research institutions need to assess their own practices to promote and advocate for change in the current research ecosystem.

Aims

The purpose of the scoping review was to explore ' What does the evidence say about the 'problem' with 'poor' research culture, what are the benefits of 'good' research culture, and what does 'good' look like?'

Methods

A scoping review was undertaken. Six databases were searched along with grey literature. Eligible literature had relevance to academic research institutions, addressed research culture, and were published between January 2017 to May 2022. Evidence was mapped and themed to specific categories. The search strategy, screening and analysis took place between April-May 2022.

Results

1666 titles and abstracts, and 924 full text articles were assessed for eligibility. Of these, 253 articles met the eligibility criteria for inclusion. A purposive sampling of relevant websites was drawn from to complement the review, resulting in 102 records included in the review. Key areas for consideration were identified across the four themes of job security, wellbeing and equality of opportunity, teamwork and interdisciplinary, and research quality and accountability.

Conclusions

There are opportunities for research institutions to improve their own practice, however institutional solutions cannot act in isolation. Research institutions and research funders need to work together to build a more sustainable and inclusive research culture that is diverse in nature and supports individuals' well-being, career progression and performance.

A Critical Appraisal of Reporting in Randomized Controlled Trials Investigating Osteopathic Manipulative Treatment: A Meta-Research Study

Background/Objectives: In osteopathy, it becomes necessary to produce high-quality evidence to demonstrate its effectiveness. The aim of this meta-research study is to assess the reporting quality of RCTs published in the osteopathic field. Methods: The protocol was preliminarily registered on the "Open Science Framework (OSF)" website. For reporting, we considered the PRISMA 2020 checklist. We included all the RCTs, published between 2011 and 2023, investigating the effectiveness of Osteopathic Manipulative Treatment (OMT) in any possible condition. The search process was conducted on four major biomedical databases including PubMed, Central, Scopus and Embase. A data extraction form was implemented to collect all relevant information. The completeness of reporting was calculated as the percentage of adherence to the CONSORT checklist; the Cochrane ROB 2 tool was considered to assess the risk of bias (RoB) in the following five major domains: randomization (D1), interventions (D2), missing data (D3), outcome measurement (D4), selective reporting (D5). Results: A total of 131 studies were included and the overall adherence was 57%, with the worst section being "other information" (42%). Studies with a lower RoB showed higher adherence to the CONSORT. The "results" section presented the highest differences as follows: D1 (-36.7%), D2 (-27.2%), D3 (-21.5%) and D5 (-25.5%). Significant correlations were also found between the preliminary protocol registration, higher journal quartile, publication in hybrid journals and the completeness of reporting (β: 19.22, CI: 14.45-24.00, p < 0.001; β: 5.41; CI: 2.80-8.02, p ≤ 0.001; β: 5.64, CI: 1.06-10.23, p = 0.016, respectively). Conclusions: The adherence to the CONSORT checklist in osteopathic RCTs is lacking. An association was found between a lower completeness of reporting and a higher RoB, a good journal ranking, publication in hybrid journals and a prospective protocol registration. Journals and authors should adopt all the strategies to adhere to reporting guidelines to guarantee generalization of the results arising from RCTs.

ChatGPT for digital pathology research

The rapid evolution of generative artificial intelligence (AI) models including OpenAI's ChatGPT signals a promising era for medical research. In this Viewpoint, we explore the integration and challenges of large language models (LLMs) in digital pathology, a rapidly evolving domain demanding intricate contextual understanding. The restricted domain-specific efficiency of LLMs necessitates the advent of tailored AI tools, as illustrated by advancements seen in the last few years including FrugalGPT and BioBERT. Our initiative in digital pathology emphasises the potential of domain-specific AI tools, where a curated literature database coupled with a user-interactive web application facilitates precise, referenced information retrieval. Motivated by the success of this initiative, we discuss how domain-specific approaches substantially minimise the risk of inaccurate responses, enhancing the reliability and accuracy of information extraction. We also highlight the broader implications of such tools, particularly in streamlining access to scientific research and democratising access to computational pathology techniques for scientists with little coding experience. This Viewpoint calls for an enhanced integration of domain-specific text-generation AI tools in academic settings to facilitate continuous learning and adaptation to the dynamically evolving landscape of medical research.

Can ChatGPT Fool the Match? Artificial Intelligence Personal Statements for Plastic Surgery Residency Applications: A Comparative Study

Introduction: Personal statements can be decisive in Canadian residency applications. With the rise in AI technology, ethical concerns regarding authenticity and originality become more pressing. This study explores the capability of ChatGPT in producing personal statements for plastic surgery residency that match the quality of statements written by successful applicants. Methods: ChatGPT was utilized to generate a cohort of personal statements for CaRMS (Canadian Residency Matching Service) to compare with previously successful Plastic Surgery applications. Each AI-generated and human-written statement was randomized and anonymized prior to assessment. Two retired members of the plastic surgery residency selection committee from the University of British Columbia, evaluated these on a 0 to 10 scale and provided a binary response judging whether each statement was AI or human written. Statistical analysis included Welch 2-sample t tests and Cohen's Kappa for agreement. Results: Twenty-two personal statements (11 AI-generated by ChatGPT and 11 human-written) were evaluated. The overall mean scores were 7.48 (SD 0.932) and 7.68 (SD 0.716), respectively, with no significant difference between AI and human groups (P = .4129). The average accuracy in distinguishing between human and AI letters was 65.9%. The Cohen's Kappa value was 0.374. Conclusions: ChatGPT can generate personal statements for plastic surgery residency applications with quality indistinguishable from human-written counterparts, as evidenced by the lack of significant scoring difference and moderate accuracy in discrimination by experienced surgeons. These findings highlight the evolving role of AI and the need for updated evaluative criteria or guidelines in the residency application process.

Factors Influencing Interest and Engagement in Biomedical Research Among Community Medicine Residents in India: A Descriptive Cross-Sectional Study

Introduction Medical science must be based on sound and scientific evidence and requires continuous research. Engaging in research allows students and faculty to explore new frontiers, question existing paradigms, and discover innovative solutions to medical challenges. As a specialty, community medicine plays a pivotal role in addressing public health issues. However, the engagement of community medicine residents in biomedical research remains suboptimal, which may impede the generation of evidence-based practices tailored to the Indian context. This study was conducted to find the interest and engagement of community medicine residents, and factors influencing their interest in biomedical research. Methods An online survey was conducted among community medicine residents of Uttar Pradesh, from February to April 2024, using Google Forms having a semi-structured, pretested questionnaire. Results One hundred and ninety-six residents participated in the study, where females (52.6%; 103/196) outnumbered males (47.4%; 93/196). The majority of participants were third-year residents (40.8%). Most participants seemed interested in biomedical research (83.2%) and thought that Basic Course in Biomedical Research (BCBR) helps conduct research projects (75%). Around half had previous experience in research projects, with cross-sectional studies being the most common (75.9%) study design. Enhancing research skills and a desire to contribute to medical knowledge emerged as primary motivators. On the other hand, the lack of time due to being overburdened with academic and educational activities was seen as the most common barrier to conducting research. Conclusions The majority of participants were found interested in research activities. The opportunity to improve research skills, desire to serve the medical fraternity, and a positive impact on resumes were the leading motivating factors for conducting research. Difficulty in sparing time, little knowledge, and poor support from mentors were found as important barriers.

Towards evidence-based practice 2.0: leveraging artificial intelligence in healthcare

Background

Evidence-based practice (EBP) involves making clinical decisions based on three sources of information: evidence, clinical experience and patient preferences. Despite popularization of EBP, research has shown that there are many barriers to achieving the goals of the EBP model. The use of artificial intelligence (AI) in healthcare has been proposed as a means to improve clinical decision-making. The aim of this paper was to pinpoint key challenges pertaining to the three pillars of EBP and to investigate the potential of AI in surmounting these challenges and contributing to a more evidence-based healthcare practice. We conducted a selective review of the literature on EBP and the integration of AI in healthcare to achieve this.

Challenges with the three components of EBP

Clinical decision-making in line with the EBP model presents several challenges. The availability and existence of robust evidence sometimes pose limitations due to slow generation and dissemination processes, as well as the scarcity of high-quality evidence. Direct application of evidence is not always viable because studies often involve patient groups distinct from those encountered in routine healthcare. Clinicians need to rely on their clinical experience to interpret the relevance of evidence and contextualize it within the unique needs of their patients. Moreover, clinical decision-making might be influenced by cognitive and implicit biases. Achieving patient involvement and shared decision-making between clinicians and patients remains challenging in routine healthcare practice due to factors such as low levels of health literacy among patients and their reluctance to actively participate, barriers rooted in clinicians' attitudes, scepticism towards patient knowledge and ineffective communication strategies, busy healthcare environments and limited resources.

AI assistance for the three components of EBP

AI presents a promising solution to address several challenges inherent in the research process, from conducting studies, generating evidence, synthesizing findings, and disseminating crucial information to clinicians to implementing these findings into routine practice. AI systems have a distinct advantage over human clinicians in processing specific types of data and information. The use of AI has shown great promise in areas such as image analysis. AI presents promising avenues to enhance patient engagement by saving time for clinicians and has the potential to increase patient autonomy although there is a lack of research on this issue.

Conclusion

This review underscores AI's potential to augment evidence-based healthcare practices, potentially marking the emergence of EBP 2.0. However, there are also uncertainties regarding how AI will contribute to a more evidence-based healthcare. Hence, empirical research is essential to validate and substantiate various aspects of AI use in healthcare.

The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research

Artificial Intelligence (AI) has revolutionized various domains, including education and research. Natural language processing (NLP) techniques and large language models (LLMs) such as GPT-4 and BARD have significantly advanced our comprehension and application of AI in these fields. This paper provides an in-depth introduction to AI, NLP, and LLMs, discussing their potential impact on education and research. By exploring the advantages, challenges, and innovative applications of these technologies, this review gives educators, researchers, students, and readers a comprehensive view of how AI could shape educational and research practices in the future, ultimately leading to improved outcomes. Key applications discussed in the field of research include text generation, data analysis and interpretation, literature review, formatting and editing, and peer review. AI applications in academics and education include educational support and constructive feedback, assessment, grading, tailored curricula, personalized career guidance, and mental health support. Addressing the challenges associated with these technologies, such as ethical concerns and algorithmic biases, is essential for maximizing their potential to improve education and research outcomes. Ultimately, the paper aims to contribute to the ongoing discussion about the role of AI in education and research and highlight its potential to lead to better outcomes for students, educators, and researchers.

Artificial Intelligence and Public Health: An Exploratory Study

Artificial intelligence (AI) has the potential to revolutionize research by automating data analysis, generating new insights, and supporting the discovery of new knowledge. The top 10 contribution areas of AI towards public health were gathered in this exploratory study. We utilized the "text-davinci-003" model of GPT-3, using OpenAI playground default parameters. The model was trained with the largest training dataset any AI had, limited to a cut-off date in 2021. This study aimed to test the ability of GPT-3 to advance public health and to explore the feasibility of using AI as a scientific co-author. We asked the AI asked for structured input, including scientific quotations, and reviewed responses for plausibility. We found that GPT-3 was able to assemble, summarize, and generate plausible text blocks relevant for public health concerns, elucidating valuable areas of application for itself. However, most quotations were purely invented by GPT-3 and thus invalid. Our research showed that AI can contribute to public health research as a team member. According to authorship guidelines, the AI was ultimately not listed as a co-author, as it would be done with a human researcher. We conclude that good scientific practice also needs to be followed for AI contributions, and a broad scientific discourse on AI contributions is needed.

Expert views about missing AI narratives: is there an AI story crisis?

Stories are an important indicator of our vision of the future. In the case of artificial intelligence (AI), dominant stories are polarized between notions of threat and myopic solutionism. The central storytellers-big tech, popular media, and authors of science fiction-represent particular demographics and motivations. Many stories, and storytellers, are missing. This paper details the accounts of missing AI narratives by leading scholars from a range of disciplines interested in AI Futures. Participants focused on the gaps between dominant narratives and the untold stories of the capabilities, issues, and everyday realities of the technology. One participant proposed a "story crisis" in which these narratives compete to shape the public discourse on AI. Our findings indicate that dominant narratives distract and mislead public understandings and conceptions of AI. This suggests a need to pay closer attention to missing AI narratives. It is not simply about telling new stories, it is about listening to existing stories and asking what is wanted from AI. We call for realistic, nuanced, and inclusive stories, working with and for diverse voices, which consider (1) story-teller; (2) genre, and (3) communicative purpose. Such stories can then inspire the next generation of thinkers, technologists, and storytellers.

Narratives of epistemic agency in citizen science classification projects: ideals of science and roles of citizens

Citizen science (CS) projects have started to utilize Machine Learning (ML) to sort through large datasets generated in fields like astronomy, ecology and biodiversity, biology, and neuroimaging. Human–machine systems have been created to take advantage of the complementary strengths of humans and machines and have been optimized for efficiency and speed. We conducted qualitative content analysis on meta-summaries of documents reporting the results of 12 citizen science projects that used machine learning to optimize classification tasks. We examined the distribution of tasks between citizen scientists, experts, and algorithms, and how epistemic agency was enacted in terms of whose knowledge shapes the distribution of tasks, who decides what knowledge is relevant to the classification, and who validates it. In our descriptive results, we found that experts, who include professional scientists and algorithm developers, are involved in every aspect of a project, from annotating or labelling data to giving data to algorithms to train them to make decisions from predictions. Experts also test and validate models to improve their accuracy by scoring their outputs when algorithms fail to make correct decisions. Experts are mostly the humans involved in a loop, but when algorithms encounter problems, citizens are also involved at several stages. In this paper, we present three main examples of citizens-in-the-loop: (a) when algorithms provide incorrect suggestions; (b) when algorithms fail to know how to perform classification; and (c) when algorithms pose queries. We consider the implications of the emphasis on optimization on the ideal of science and the role of citizen scientists from a perspective informed by Science and Technology Studies (STS) and Information Systems (IS). Based on our findings, we conclude that ML in CS classification projects, far from being deterministic in its nature and effects, may be open to question. There is no guarantee that these technologies can replace citizen scientists, nor any guarantee that they can provide citizens with opportunities for more interesting tasks.