Prevention of false positive findings in observational studies: registration will not work but replication might

The majority of observational studies in leading peer-reviewed medicine journals are not registered and do not have a publicly accessible protocol: a scoping review

OBJECTIVES

Observational studies are not subject to the same requirements as randomized controlled trials, such as registration or publishing a protocol. The aim of this scoping review was to estimate the registration rate of observational studies in leading peer-reviewed medicine journals and to evaluate whether protocols were available in the public domain.

STUDY DESIGN AND SETTING

In March 2023, we searched OVID Medline for observational studies published in 2022 in the top five general medicine journals according to impact factor (The Lancet, The British Medical Journal (BMJ), The Journal of the American Medical Association, The New England Journal of Medicine, and Annals of Internal Medicine). We defined an observational study as a cohort study, a case-control study, a cross-sectional study, or a case series. Information on i) the proportion of observational studies that have been registered and ii) the proportion of observational studies that have a protocol available in the public domain was extracted from a random sample of studies.

RESULTS

Our search identified 699 studies; 290 studies were selected as full text, and a random sample of 200 studies was included. For half of the studies, the first author worked at a US institution. Most studies were cohort studies (n = 126, 63.0%) and used administrative healthcare records, electronic healthcare records, and registries. Of the 200 observational studies, 20 (10.0%) were registered. Among those, 14 were prospectively registered. Twenty-four studies (12.0%) had a protocol available in the public domain. Studies that were registered or had a protocol, were more frequently published in the BMJ (n = 12/28, 42.9%), had a first author working in the UK (n = 10/28, 35.7%) and used electronic health care records (n = 13/28, 46.4%) compared to studies with no registration and no protocol.

CONCLUSION

The rate of prospectively registered observational studies is worryingly low. Prospective registration of observational studies should be encouraged and standardized to ensure transparency in clinical research and reduce research waste.

Updated analysis of pediatric clinical studies registered in ClinicalTrials.gov, 2008-2019

Background

Since the national clinical trials registry (ClinicalTrials.gov) launched in February 2000, more than 360,000 research studies in the United States and over 200 countries have registered. As the characteristics of pediatric clinical studies keep changing over time and the results-reporting mechanism is under evolving, to know about the relevant updates of data elements and the effect of policies on the quality of reporting results is significant.

Methods

In this research, 53,060 clinical studies related to children registered from January 2008 to December 2019 were downloaded from ClinicalTrials.gov on August 1st, 2020. Different types of studies and critical categorical variables were identified, based on which, Cochran-Armitage test was performed to explore temporal trend of study characteristics and common pediatric clinical conditions in four time subsets. Further, to examine heterogeneity among subgroups (funding sources, funding sites, pediatric clinical conditions,etc), chi-squared test was applied.

Results

A total of 36,136 clinical trials and 16,692 observational studies were identified during the study period. The pediatric clinical trials increased from 7,029 (January 2008–December 2010) to 11,738 (January 2017–December 2019). The number of missing data has declined, with the maximum extent decline from 3.7 to 0.0% (Z = − 15.90, p <  0.001). Drug trials decreased from 48.8 to 28.9% (Z = − 24.68, p <  0.001). Behavioral trials, on the other hand, increased from 12.6 to 20.4% (Z = 12.28, p <  0.001). Most pediatric clinical trials were small-scale (58.9% enrolling 1–100 participants), single-site (61.4%) and funded neither by industry nor by the NIH (59.2%). The proportion of reporting study results varied by study type (χ² = 1,256.8, p <  0.001), lead sponsor (χ² = 4,545.6, p <  0.001), enrollment (χ² = 29.4, p <  0.001) and trial phase (χ² = 218.8, p <  0.001).

Conclusion

Pediatric clinical studies registered in ClinicalTrials.gov were dominated by small-scale interventional trials, containing significant heterogeneity in funding sources, funding sites, pediatric clinical conditions and study characteristics. Although the results database has evolved in the past decade, efforts to strengthen the practice of systematic reporting must be continued.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12887-021-02658-4.

Intrapartum uterine activity and neonatal outcomes: a systematic review

BACKGROUND

Increased uterine activity (UA) may not allow adequate recovery time for foetal oxygenation.

METHODS

The aim of the study was to determine if increased UA during labour is associated with an increased risk of either short- or long-term neurological injury in term neonates, or with neonatal proxy measures of intrapartum hypoxia-ischemia. MEDLINE, CINAHL, and ClinicalTrials.gov were searched using the following terms: uterine activity, excessive uterine activity, XSUA, uterine hyperstimulation, and tachysystole. Any study that analysed the relationship between UA during term labour and neurological outcomes/selected proxy neurological outcomes was eligible for inclusion. Outcomes from individual studies were reported in tables and presented descriptively with odds ratios (OR) and 95% confidence intervals (CI) for dichotomous outcomes and means with standard deviations for continuous outcomes. Where group numbers were provided, ORs and their CIs were calculated according to Altman.

MAIN RESULTS

Twelve studies met the inclusion criteria. Seven studies featured umbilical artery pH as an individual outcome. Umbilical artery base excess and Apgar scores were both reported as individual outcomes in four studies. No study examined long term neurodevelopmental outcomes and only one study reported on encephalopathy as an outcome. The evidence for a relationship between UA and adverse infant outcomes was inconsistent. The reported estimated effect size varied from non-existent to clinically significant.

CONCLUSIONS

There is some evidence that increased UA may be a non-specific predictor of depressed neurological function in the newborn, but it is inconsistent and insufficient to support the conclusion that an association generally exists.

Standards for design and measurement would make clinical research reproducible and usable

We find standards useful in everyday life and in science, although we do not always follow them. Adopting new standards can be expensive, so there may be a strong incentive to maintain the status quo rather than adopt new standards. The scientific community has many standards encompassing both doing clinical research and reporting it, including standards for design and measurement. Although existing research standards have improved both research and its reporting, we need to unify existing standards and to fill the gaps between steps throughout the research process. Existing gaps include implementation of standards and links between standards for study registration (to know about all studies undertaken), study protocols (to identify the preplanned study design and methods), data collection (to assess outcomes that are important and comparable across studies), dissemination of findings (to know the results of previous studies), data sharing (to make best use of existing data), and evidence synthesis (to draw appropriate conclusions from the body of evidence). The scientific community must work together to harmonize existing standards, to ensure that standards are kept up to date, to check that standards are followed, and to develop standards where they are still needed. A unified system of standards will make our work more reproducible.

Citation bias and selective focus on positive findings in the literature on the serotonin transporter gene (5-HTTLPR), life stress and depression

BACKGROUND

Caspi et al.'s 2003 report that 5-HTTLPR genotype moderates the influence of life stress on depression has been highly influential but remains contentious. We examined whether the evidence base for the 5-HTTLPR-stress interaction has been distorted by citation bias and a selective focus on positive findings.

METHOD

A total of 73 primary studies were coded for study outcomes and focus on positive findings in the abstract. Citation rates were compared between studies with positive and negative results, both within this network of primary studies and in Web of Science. In addition, the impact of focus on citation rates was examined.

RESULTS

In all, 24 (33%) studies were coded as positive, but these received 48% of within-network and 68% of Web of Science citations. The 38 (52%) negative studies received 42 and 23% of citations, respectively, while the 11 (15%) unclear studies received 10 and 9%. Of the negative studies, the 16 studies without a positive focus (42%) received 47% of within-network citations and 32% of Web of Science citations, while the 13 (34%) studies with a positive focus received 39 and 51%, respectively, and the nine (24%) studies with a partially positive focus received 14 and 17%.

CONCLUSIONS

Negative studies received fewer citations than positive studies. Furthermore, over half of the negative studies had a (partially) positive focus, and Web of Science citation rates were higher for these studies. Thus, discussion of the 5-HTTLPR-stress interaction is more positive than warranted. This study exemplifies how evidence-base-distorting mechanisms undermine the authenticity of research findings.

Challenges in reproducing results from publicly available data: an example of sexual orientation and cardiovascular disease risk

BACKGROUND

Replication is a vital part of the research process and has recently received considerable attention. Analyses using publicly available data should, if adequately described, be reproducible without assistance from the original investigators. Using data from the US National Health and Nutrition Examination Survey (NHANES), a recent study reported a statistically significant difference in cardiovascular disease risk comparing subgroups of sexual minority men. We attempted to reproduce these findings and assessed whether the results were robust to alternative analytic strategies and assumptions.

METHODS

We used the exclusion criteria and coding strategy described in the original paper to construct our analytical data set. Sampling weights were constructed in accordance with NHANES analytical guidelines. We estimated crude and covariate-adjusted associations between sexual orientation and vascular age using the regression models specified in the original report. We also conducted a series of sensitivity analyses to improve on the original findings.

RESULTS

Our replication attempt was partially successful: we replicated the general trends reported in the original analysis, but not identical effect estimates. Importantly, we identified a potential misapplication of the Framingham Risk Score; correcting for this increased the probability that the reported null hypothesis test was a type I error.

CONCLUSIONS

This paper supports the recent calls for greater transparency and improved reporting in research. Even with a publicly available and well-documented data source, we were unable to exactly replicate another study's original findings. Our sensitivity analyses revealed key issues in the original analysis and demonstrate the scientific importance of research replication.

Registration practices for observational studies on ClinicalTrials.gov indicated low adherence

OBJECTIVE

The study aims to assess the status of registration of observational studies.

STUDY DESIGN AND SETTING

We identified studies on cancer research with prospective recruitment of participants that were registered from February 2000 to December 2011 in ClinicalTrials.gov. We recorded the dates of registration and start of recruitment, outcomes, and description of statistical method. We searched for publications corresponding to the registered studies through May 31, 2014.

RESULTS

One thousand one hundred nine registered studies were eligible. Primary and secondary outcomes were reported in 809 (73.0%) and 464 (41.8%) of them. The date of registration preceded the month of the study start in 145 (13.8%) and coincided in 205 (19.5%). A total of 151 publications from 120 (10.8%) registered studies were identified. In 2 (33.3%) of the 6 publications where ClinicalTrials.gov reported that the study started recruitment after registration, and in 9 (50.0%) of 18 publications where ClinicalTrials.gov reported the same date for registration and start of recruitment, the articles showed that the study had actually started recruiting before registration.

CONCLUSION

During the period reviewed, few observational studies have been registered. Registration usually occurred after the study started, and prespecification of outcomes and statistical analysis rarely occurred.

Making valid causal inferences from observational data

The ability to make strong causal inferences, based on data derived from outside of the laboratory, is largely restricted to data arising from well-designed randomized control trials. Nonetheless, a number of methods have been developed to improve our ability to make valid causal inferences from data arising from observational studies. In this paper, I review concepts of causation as a background to counterfactual causal ideas; the latter ideas are central to much of current causal theory. Confounding greatly constrains causal inferences in all observational studies. Confounding is a biased measure of effect that results when one or more variables, that are both antecedent to the exposure and associated with the outcome, are differentially distributed between the exposed and non-exposed groups. Historically, the most common approach to control confounding has been multivariable modeling; however, the limitations of this approach are discussed. My suggestions for improving causal inferences include asking better questions (relates to counterfactual ideas and "thought" trials); improving study design through the use of forward projection; and using propensity scores to identify potential confounders and enhance exchangeability, prior to seeing the outcome data. If time-dependent confounders are present (as they are in many longitudinal studies), more-advanced methods such as marginal structural models need to be implemented. Tutorials and examples are cited where possible.