Registered report: Survey on attitudes and experiences regarding preregistration in psychological research

Methodological quality in randomised clinical trials of mental health apps: systematic review and longitudinal analysis

QUESTION

This study investigated the methodological rigour of randomised controlled trials (RCTs) of mental health apps for depression and anxiety, and whether quality has improved over time.

STUDY SELECTION AND ANALYSIS

RCTs were drawn from the most recent meta-analysis of mental health apps for depression and anxiety symptoms. 20 indicators of study quality were coded, encompassing risk of bias, participant diversity, study design features and app accessibility measures. Regression models tested associations between year of publication and each quality indicator.

FINDINGS

176 RCTs conducted between 2011 and 2023 were included. Methodological concerns were common for several quality indicators (eg, <20% were replication trials, <35% of trials reported adverse events). Regression models revealed only three significant changes over time: an increase in preregistration (OR=1.27; 95% CI 1.10, 1.46) and reporting of adverse events (OR=1.32; 95% CI 1.11, 1.56), and a decrease in apps reported to be compatible with iOS and/or Android (OR=0.78; 95% CI 0.64, 0.96). Results were unchanged when excluding outliers. Results were similar when excluding three high-quality studies published between 2011 and 2013, with additional evidence for an increase in modern missing data methods (OR=1.22; 95% CI 1.04, 1.42) and studies reporting intention-to-treat analysis (OR=1.20; 95% CI 1.03, 1.39).

CONCLUSIONS

Findings provide minimal evidence of improvements in the quality of clinical trials of mental health apps, highlighting the need for higher methodological standards in future research to ensure the reliability and generalisability of evidence for these digital tools.

How Transparent and Reproducible Are Studies That Use Animal Models of Opioid Addiction?

The reproducibility crisis in psychology has caused various fields to consider the reliability of their own findings. Many of the unfortunate aspects of research design that undermine reproducibility also threaten translation potential. In preclinical addiction research, the rates of translation have been disappointing. We tallied indices of transparency and accurate and thorough reporting in animal models of opioid addiction from 2019 to 2023. By examining the prevalence of these practices, we aimed to understand whether efforts to improve reproducibility are relevant to this field. For 255 articles, we report the prevalence of transparency measures such as preregistration, registered reports, open data and open code, as well as compliance to the Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines. We also report rates of bias minimization practices (randomization, masking and data exclusion), sample size calculations and multiple corrections adjustments. Lastly, we estimated the accuracy of test statistic reporting using a version of StatCheck. All the transparency measures and the ARRIVE guideline items had low prevalence, including no cases of study preregistration and no cases where authors shared their analysis code. Similarly, the levels of bias minimization practices and sample size calculations were unsatisfactory. In contrast, adjustments for multiple comparisons were implemented in most articles (76.5%). Lastly, p-value inconsistencies with test statistics were detected in about half of papers, and 11% contained statistical significance errors. We recommend that researchers, journal editors and others take steps to improve study reporting and to facilitate both replication and translation.

Transparency in the secondary use of health data: assessing the status quo of guidance and best practices

We evaluated what guidance exists in the literature to improve the transparency of studies that make secondary use of health data. To find peer-reviewed papers, we searched PubMed and Google Scholar. To find institutional documents, we used our personal expertise to draft a list of health organizations and searched their websites. We quantitatively and qualitatively coded different types of research transparency: registration, methods reporting, results reporting, data sharing and code sharing. We found 56 documents that provide recommendations to improve the transparency of studies making secondary use of health data, mainly in relation to study registration (n = 27) and/or methods reporting (n = 39). Only three documents made recommendations on data sharing or code sharing. Recommendations for study registration and methods reporting mainly came in the form of structured documents like registration templates and reporting guidelines. Aside from the recommendations aimed directly at researchers, we also found recommendations aimed at the wider research community, typically on how to improve research infrastructure. Limitations or challenges of improving transparency were rarely mentioned, highlighting the need for more nuance in providing transparency guidance for studies that make secondary use of health data.

Reporting and Methodological Quality of Systematic Reviews Underpinning Clinical Practice Guidelines for Vascular Surgery: A Systematic Review

OBJECTIVE

Clinical practice guideline recommendations are often informed by systematic reviews. This review aimed to appraise the reporting and methodological quality of systematic reviews informing clinical practice recommendations relevant to vascular surgery.

DATA SOURCES

MEDLINE and Embase.

METHODS

MEDLINE and Embase were searched from 1 January 2021 to 5 May 2023 for clinical practice guidelines relevant to vascular surgery. Guidelines were then screened for systematic reviews informing recommendations. The reporting and methodological quality of these systematic reviews were assessed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement and Assessment of Multiple Systematic Reviews 2 (AMSTAR 2) 2017 tool. Pearson correlation and multiple regression analyses were performed to determine associations between these scores and extracted study characteristics.

RESULTS

Eleven clinical practice guidelines were obtained, containing 1 783 references informing guideline recommendations. From these, 215 systematic reviews were included for synthesis. PRISMA item completeness ranged 14 - 100%, with a mean of 63% across reviews. AMSTAR 2 item completeness ranged 2 - 95%, with a mean of 50%. Pearson correlation highlighted a statistically significant association between a review's PRISMA and AMSTAR 2 score (r = 0.85, p < .001). A more recent publication year was associated with a statistically significant increase in both scores (PRISMA coefficient 1.28, p < .001; and AMSTAR 2 coefficient 1.31, p < .001). Similarly, the presence of funding in a systematic review was shown to be statistically significantly associated with an increase in both PRISMA and AMSTAR 2 scores (coefficient 4.93, p = .024; and coefficient 6.07, p = .019, respectively).

CONCLUSION

Systematic reviews informing clinical practice guidelines relevant to vascular surgery were of moderate quality at best. Organisations producing clinical practice guidelines should consider funding systematic reviews to improve the quality of their recommendations.

Open science practices among authors published in complementary, alternative, and integrative medicine journals: An international, cross-sectional survey

Open science practices aim to increase transparency in research and increase research availability through open data, open access platforms, and public access. Due to the increasing popularity of complementary, alternative, and integrative medicine (CAIM) research, our study aims to explore current open science practices and perceived barriers among CAIM researchers in their own respective research articles. We conducted an international cross-sectional online survey that was sent to authors that published articles in MEDLINE-indexed journals categorized under the broad subject of "Complementary Therapies" or articles indexed under the MeSH term "Complementary Therapies." Articles were extracted to obtain the names and emails of all corresponding authors. Eight thousand seven hundred eighty-six researchers were emailed our survey, which included questions regarding participants' familiarity with open science practices, their open science practices, and perceived barriers to open science in CAIM with respect to participants' most recently published article. Basic descriptive statistics was generated based on the quantitative data. The survey was completed by 292 participants (3.32% response rate). Results indicate that the majority of participants were "very familiar" (n = 83, 31.68%) or "moderately familiar" (n = 83, 31.68%) with the concept of open science practices while creating their study. Open access publishing was the most familiar to participants, with 51.96% (n = 136) of survey respondents publishing with open access. Despite participants being familiar with other open science practices, the actual implementation of these practices was low. Common barriers participants experienced in implementing open science practices include not knowing where to share the study materials, where to share the data, or not knowing how to make a preprint. Although participants responded that they were familiar with the concept of open science practices, the actual implementation and uses of these practices were low. Barriers included a lack of overall knowledge about open science, and an overall lack of funding or institutional support. Future efforts should aim to explore how to implement methods to improve open science training for CAIM researchers.

Supporting study registration to reduce research waste

An estimated 82-89% of ecological research and 85% of medical research has limited or no value to the end user because of various inefficiencies. We argue that registration and registered reports can enhance the quality and impact of ecological research. Drawing on evidence from other fields, chiefly medicine, we support our claim that registration can reduce research waste. However, increasing registration rates, quality and impact will be very slow without coordinated effort of funders, publishers and research institutions. We therefore call on them to facilitate the adoption of registration by providing adequate support. We outline several aspects to be considered when designing a registration system that would best serve the field of ecology. To further inform the development of such a system, we call for more research to identify the causes of low registration rates in ecology. We suggest short- and long-term actions to bolster registration and reduce research waste.

Looking for a broader mindset in psychometrics: the case for more participatory measurement practices

Psychometrics and the consequences of its use as the method of quantitative empirical psychology has been continuously criticized by both psychologists and psychometrists. However, the scope of the possible solutions to these issues has been mostly focused on the establishment of methodological-statistical best practices for researchers, without any regard to the pitfalls of previous stages of measurement as well as theory development of the targeted phenomenon. Conversely, other researchers advance the idea that, since psychometrics is riddled with many issues, the best way forward is a complete rework of the discipline even if it leaves psychologists and other practitioners without any way to measure quantitatively for a long period of time. Given these tensions, we therefore advocate for an alternative path to consider while we work on making substantive change in measurement. We propose a set of research practices focusing on the inclusion and active participation of groups involved in measurement activities, such as psychometrists, researchers but most importantly practitioners and potential participants. Involving a wider community while measuring in psychology could tackle some key issues that would take us closer to a more authentic approach to our phenomenon of interest.

Registered Reports in Child Development: Introduction to the Special Section

Registered Reports (RRs) are an emerging format for publishing empirical journal articles in which the decision to publish an article is based on sound conceptualization, methods, and planned analyses rather than the specific nature of the results. This article introduces the Special Section on Registered Reports in Child Development by describing what RRs are and why they are necessary, outlining the thought process that guided the Special Section, describing key thematic insights across the eight articles included in the collection, and providing recommendations for developmental researchers interested in publishing via the RR format. This article also serves as a formal announcement that RRs will be a standard publishing option at Child Development, effective immediately.