Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study

BACKGROUND

Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022.

OBJECTIVE

The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities.

METHODS

Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines.

RESULTS

Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy.

CONCLUSIONS

ChatGPT's performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy.

Dynamics and landscape of academic discourse on environmental attitudes and behaviors since the 1970s

Introduction

This study addresses the lack of systematic review and analysis of the academic discourse on environmental attitudes and behaviors. Despite the wealth of knowledge published in academic journals, there is a need to understand the order and content of this discourse, including the employed theoretical approaches and empirical evidence.

Methods

A combination of systematic literature review and quantitative content analysis methods was employed. Articles for analysis were identified through Web of Science and SCOPUS, followed by a detailed analysis of 200 papers from the journal Environment and Behavior. The study aimed to explore the historical stages, theoretical diversity, and the empirical evidence brought forward in the academic discourse on environmental attitudes and behaviors.

Results

The findings reveal distinct historical stages within the academic field of environmental attitudes and behaviors. There is a notable growth in theoretical diversity and intensity of the discourse, particularly after 2000. The dominance of socio-psychological explanatory models is evident. Furthermore, the empirical evidence base is geographically limited, mostly coming from the US.

Discussion

The study discusses the limitations of the academic discourse on environmental attitudes and behaviors and provides guidelines for future research. It emphasizes the need to address the identified shortcomings, such as expanding theoretical perspectives and increasing the geographical diversity of empirical evidence. The study's findings contribute to understanding the development and characteristics of the academic field, while also identifying avenues for further exploration.