Methodological and statistical characteristics of meta-analyses on spinal cord stimulation for chronic pain: a systematic review

Human versus artificial intelligence: evaluating ChatGPT's performance in conducting published systematic reviews with meta-analysis in chronic pain research

INTRODUCTION

Artificial intelligence (AI), particularly large-language models like Chat Generative Pre-Trained Transformer (ChatGPT), has demonstrated potential in streamlining research methodologies. Systematic reviews and meta-analyses, often considered the pinnacle of evidence-based medicine, are inherently time-intensive and demand meticulous planning, rigorous data extraction, thorough analysis, and careful synthesis. Despite promising applications of AI, its utility in conducting systematic reviews with meta-analysis remains unclear. This study evaluated ChatGPT's accuracy in conducting key tasks of a systematic review with meta-analysis.

METHODS

This validation study used data from a published meta-analysis on emotional functioning after spinal cord stimulation. ChatGPT-4o performed title/abstract screening, full-text study selection, and data pooling for this systematic review with meta-analysis. Comparisons were made against human-executed steps, which were considered the gold standard. Outcomes of interest included accuracy, sensitivity, specificity, positive predictive value, and negative predictive value for screening and full-text review tasks. We also assessed for discrepancies in pooled effect estimates and forest plot generation.

RESULTS

For title and abstract screening, ChatGPT achieved an accuracy of 70.4%, sensitivity of 54.9%, and specificity of 80.1%. In the full-text screening phase, accuracy was 68.4%, sensitivity 75.6%, and specificity 66.8%. ChatGPT successfully pooled data for five forest plots, achieving 100% accuracy in calculating pooled mean differences, 95% CIs, and heterogeneity estimates (I2 score and tau-squared values) for most outcomes, with minor discrepancies in tau-squared values (range 0.01-0.05). Forest plots showed no significant discrepancies.

CONCLUSION

ChatGPT demonstrates modest to moderate accuracy in screening and study selection tasks, but performs well in data pooling and meta-analytic calculations. These findings underscore the potential of AI to augment systematic review methodologies, while also emphasizing the need for human oversight to ensure accuracy and integrity in research workflows.

Hidden influence? Unmasking conflicts of interest from randomized clinical trials on spinal cord stimulation for chronic pain

BACKGROUND

Randomized clinical trials (RCTs) are considered the gold standard for evaluating the efficacy of healthcare interventions. However, conflicts of interest (COIs) can compromise the scientific integrity in these trials. This study characterized COIs in RCTs on spinal cord stimulation for chronic pain, focusing on the prevalence, disclosure, and monetary value of COIs.

METHODS

This cross-sectional study analyzed RCTs published from January 1, 2013 to July 27, 2023. Primary outcomes included the presence, disclosure, and monetary value of COIs, while secondary outcomes assessed the presence of direct/indirect COIs, sponsor access to data, and associations between COIs and select variables, including journal impact factor, publication year, and study outcomes.

RESULTS

Of 38 RCTs, 30 (78.9%) reported COIs. On average, 35.6% of authors per RCT had at least one COI, with a mean of 0.7 COIs per author. The mean annual monetary value of COIs was US$41,157.83 per author per RCT. 29 RCTs (76.3%) had undisclosed COIs, with an average of 24.2% of authors per RCT having undisclosed COIs. Sponsor access to data was reported in 67.6% of RCTs. No associations were observed between the mean percentage of authors with COIs and the monetary value of COIs and select dependent variables (impact factor, publication year, and study outcomes).

CONCLUSIONS

A substantial majority of RCTs reported COIs with many authors having undisclosed conflicts, highlighting the need for stringent COI disclosure guidelines to maintain research integrity. Expanding COI registry systems globally and increasing non-industry funding are crucial steps toward enhancing transparency and reducing biases in medical research.