Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Perspectives on the role of industry in informatics research and authorship

OBJECTIVES

Advances in informatics research come from academic, nonprofit, and for-profit industry organizations, and from academic-industry partnerships. While scientific studies of commercial products may offer critical lessons for the field, manuscripts authored by industry scientists are sometimes categorically rejected. We review historical context, community perceptions, and guidelines on informatics authorship.

PROCESS

We convened an expert panel at the American Medical Informatics Association 2022 Annual Symposium to explore the role of industry in informatics research and authorship with community input. The panel summarized session themes and prepared recommendations.

CONCLUSIONS

Authorship for informatics research, regardless of affiliation, should be determined by International Committee of Medical Journal Editors uniform requirements for authorship. All authors meeting criteria should be included, and categorical rejection based on author affiliation is unethical. Informatics research should be evaluated based on its scientific rigor; all sources of bias and conflicts of interest should be addressed through disclosure and, when possible, methodological mitigation.

Unpublished trials of alprazolam XR and their influence on its apparent efficacy for panic disorder

OBJECTIVE

To test for publication bias with alprazolam, the most widely prescribed benzodiazepine, by comparing its efficacy for panic disorder using trial results from (1) the published literature and (2) the US Food and Drug Administration (FDA).

METHODS

From FDA reviews, we included data from all phase 2/3 efficacy trials of alprazolam extended-release (Xanax XR) for the treatment of panic disorder. A search for matching publications was performed using PubMed and Google Scholar. Publication bias was examined by comparing: (1) overall trial results (positive or not) according to the FDA v. corresponding publications; (2) effect size (Hedges's g) based on FDA data v. published data.

RESULTS

The FDA review showed that five trials were conducted, only one of which (20%) was positive. Of the four not-positive trials, two were published conveying a positive outcome; the other two were not published. Thus, according to the published literature, three trials were conducted and all (100%) were positive. Alprazolam's effect size calculated using FDA data was 0.33 (CI95% 0.07-0.60) v. 0.47 (CI95% 0.30-0.65) using published data, an increase of 0.14, or 42%.

CONCLUSIONS

Publication bias substantially inflates the apparent efficacy of alprazolam XR.

LFK index does not reliably detect small-study effects in meta-analysis: A simulation study

The LFK index has been promoted as an improved method to detect bias in meta-analysis. Putatively, its performance does not depend on the number of studies in the meta-analysis. We conducted a simulation study, comparing the LFK index test to three standard tests for funnel plot asymmetry in settings with smaller or larger group sample sizes. In general, false positive rates of the LFK index test markedly depended on the number and size of studies as well as the between-study heterogeneity with values between 0% and almost 30%. Egger's test adhered well to the pre-specified significance level of 5% under homogeneity, but was too liberal (smaller groups) or conservative (larger groups) under heterogeneity. The rank test was too conservative for most simulation scenarios. The Thompson-Sharp test was too conservative under homogeneity, but adhered well to the significance level in case of heterogeneity. The true positive rate of the LFK index test was only larger compared with classic tests if the false positive rate was inflated. The power of classic tests was similar or larger than the LFK index test if the false positive rate of the LFK index test was used as significance level for the classic tests. Under ideal conditions, the false positive rate of the LFK index test markedly and unpredictably depends on the number and sample size of studies as well as the extent of between-study heterogeneity. The LFK index test in its current implementation should not be used to assess funnel plot asymmetry in meta-analysis.

Positive Outcome Bias in the Influential Otolaryngology Clinical Trial Literature

OBJECTIVE

To assess for the presence of positive outcome bias in the otolaryngology clinical trial literature. Specifically, we investigate the prevalence of clinical trials with positive findings (CTP) and clinical trials with negative findings (CTN), as well as their quality of evidence and subsequent impact.

STUDY DESIGN

Retrospective analysis.

SETTING

Clinical Trials in the Influential Otolaryngology Literature.

METHODS

We reviewed all clinical trials published in 4 major otolaryngology journals between 2000 and 2020. We constructed several multivariable regression models to investigate the relationship of finding direction with randomization status and citation count. Subsequently, we incorporated an interaction term between year and the primary covariate of each model to assess the temporal trajectory of these relationships. All models accounted for sample size, journal, subspecialty, and the affiliated program prestige.

RESULTS

Of the 1367 trials analyzed, 1143 (84%) were CTPs, a rate that persisted throughout the study period (aOR 1.00, 95% CI 0.98-1.03). CTPs were significantly less likely to be randomized compared to CTNs (aOR 0.25, 0.17-0.37), a relationship that persisted over time (aOR 1.05, 0.99-1.03). CTPs received significantly more citations compared to CTNs (aIRR 1.41, 1.25-1.60), a disparity that also persisted over time (aIRR 0.99, 0.97-1.01).

CONCLUSION

The otolaryngology clinical trial literature has been heavily dominated by positive findings. CTPs were more frequently cited and published even with a lower level of evidence compared to CTNs. This bias may influence the objectivity of evidence used to guide clinical practice and warrants attention when reviewing findings and changing practices.

Postpartum pain and the risk of postpartum depression: A meta-analysis of observational studies

OBJECTIVE

This meta-analysis of observational studies aimed to derive a more precise estimation of the relationship between postpartum pain and postpartum depression (PPD).

METHODS

A systematic literature search was completed in the following databases from inception to September 26, 2022: PubMed, Embase, and Web of Science. Quality evaluation of each study was achieved through Newcastle-Ottawa scale (NOS) assessment. Heterogeneity across studies was evaluated by Cochran's Q test and I2 test. Pooled estimates of odds ratios (ORs) and corresponding 95% confidence intervals (CIs) were analyzed using fixed-effects model or random-effects model, according to heterogeneity. Subgroup analysis, sensitivity analysis, and Egger's test were also performed.

RESULTS

From the identified 1884 articles, a total of 8 studies involving 3973 participants were included in the final meta-analysis. Seven of the 8 studies were evaluated as high-quality, with NOS scores ≥7. A significant heterogeneity was observed (I2  = 66.5%, p = 0.004) among eight studies. Therefore, the performed random-effect model suggested a significant association between postpartum pain and PPD risk (OR 1.29, 95% CI 1.10-1.52, p = 0.002). However, the subgroup analyses did not define the source of heterogeneity. Moreover, the sensitivity analysis showed the stability of the pooled results, but the significant publication bias was identified (p = 0.009). The trim and fill method was performed and resulted in an OR of 1.14 (95% CI 0.95-1.37, p = 0.162).

CONCLUSIONS

This meta-analysis found a potential association between postpartum pain and PPD. Further researches are needed to provide more robust evidences.

Association of TIMP2 418 G/C and MMP Gene Polymorphism with Risk of Urinary Cancers: Systematic Review and Meta-analysis

Aim: The matrix metalloproteinases (MMPs) inhibit tissue inhibitors of metalloproteinases (TIMPs), playing a notable role in various biological processes, and mutations in TIMP2 genes impact a variety of urinary cancers. In this study, we analyze and evaluate the potential involvement of the TIMP2 418 G/C and MMP gene polymorphism in the etiology of urinary cancer. Methodology: For suitable case-control studies, a literature search was undertaken from various database sources such as PubMed, EMBASE, and Google Scholar. Incorporated into the analysis were case-control or cohort studies that documented the correlation between TIMP2 418 G/C and urological cancers. MetaGenyo served as the tool for conducting the meta-analysis, employing a fixed-effects model. The collective odds ratios, along with their corresponding 95% confidence intervals, were calculated and presented to assess the robustness of the observed associations. Results: A total of seven studies involving controls and cases out of recorded 1265 controls and 1154 cases were analyzed to ascertain the significant association of the TIMP2 gene with urologic cancer. No statistically significant correlation was observed between allelic, recessive, dominant, and overdominant models for the genetic variant under investigation. A 95% confidence interval (CI) and odds ratio (OR) were computed for each model, considering p-values <0.05. The OR and 95% CI for the allelic model were 0.99 and 0.77-1.27, respectively, whereas the respective values were 1.00 and 0.76-1.32 for the recessive model. In the dominant contrast model, OR and 95% CI were 1.09 and 0.62-1.90, while the same were 0.93 and 0.77-1.12 for the overdominant model. A funnel plot was used to reanalyze and detect the results as statically satisfactory. Conclusions: As a result of the data obtained, the TIMP2 gene polymorphism does not correlate statistically with cancer risk. The significance of this finding can only be confirmed using a large population, extensive epidemiological research, a comprehensive survey, and a better understanding of the molecular pathways associated.

Estimating the extent of selective reporting: An application to economics

Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as non-significant. We estimate the extent of selectively reported p-values to range between 57.7% and 71.9% of the significant p-values. The counterfactual p-value distribution also allows us to assess shifts of p-values along the entire distribution of published p-values, revealing that particularly very small p-values (p < 0.001) are unexpectedly abundant in the published literature. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies.

Primary Author Characteristics Associated With Publication in the Journal of Pain and Symptom Management

CONTEXT

Scientific journals are the primary source for dissemination of research findings, and this process relies on rigorous editorial and peer-review. As part of continuing efforts by the Journal of Pain and Symptom Management (JPSM) to advance equity, diversity, and inclusion, JPSM's leadership requested an external evaluation of their publication decisions.

OBJECTIVES

1) Describe primary author characteristics associated with final decisions to accept or reject manuscripts submitted for publication; 2) Report on whether there are potential publication biases in the JPSM editorial or peer-review processes.

METHODS

Data consisted of self-reported primary author demographic characteristics associated with manuscript submissions between June 18, 2020, and December 31, 2022. Characteristics included region of residence, race, gender, and ethnicity. A multiple logistic regression model was used to estimate adjusted odds of rejection for each author characteristic.

RESULTS

A total of 1940 submissions were evaluated. Compared to authors residing in North America, authors residing in Asia had six-fold greater odds of rejection, authors residing in Europe had four-fold greater odds of rejection, and authors residing in other regions had two-fold greater odds of rejection. Female authors submitted 1.7 times more papers than males, but there was no difference in acceptance rates of their papers in adjusted analysis.

CONCLUSION

In this analysis of publication decisions by the JPSM, there were differences in acceptance rates by region of residence, ethnicity, and race but not by gender. Asian authors and authors residing in regions outside of North America had greater odds of rejection compared to White or North American authors.

Is biomedical research self-correcting? Modelling insights on the persistence of spurious science

The reality that volumes of published biomedical research are not reproducible is an increasingly recognized problem. Spurious results reduce trustworthiness of reported science, increasing research waste. While science should be self-correcting from a philosophical perspective, that in insolation yields no information on efforts required to nullify suspect findings or factors shaping how quickly science may be corrected. There is also a paucity of information on how perverse incentives in the publishing ecosystem favouring novel positive findings over null results shape the ability of published science to self-correct. Knowledge of factors shaping self-correction of science remain obscure, limiting our ability to mitigate harms. This modelling study introduces a simple model to capture dynamics of the publication ecosystem, exploring factors influencing research waste, trustworthiness, corrective effort and time to correction. Results from this work indicate that research waste and corrective effort are highly dependent on field-specific false positive rates and time delays to corrective results to spurious findings are propagated. The model also suggests conditions under which biomedical science is self-correcting and those under which publication of correctives alone cannot stem propagation of untrustworthy results. Finally, this work models a variety of potential mitigation strategies, including researcher- and publisher-driven interventions.

Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies

When comparing the performance of two or more competing tests, simulation studies commonly focus on statistical power. However, if the size of the tests being compared are either different from one another or from the nominal size, comparing tests based on power alone may be misleading. By analogy with diagnostic accuracy studies, we introduce relative positive and negative likelihood ratios to factor in both power and size in the comparison of multiple tests. We derive sample size formulas for a comparative simulation study. As an example, we compared the performance of six statistical tests for small-study effects in meta-analyses of randomized controlled trials: Begg's rank correlation, Egger's regression, Schwarzer's method for sparse data, the trim-and-fill method, the arcsine-Thompson test, and Lin and Chu's combined test. We illustrate that comparing power alone, or power adjusted or penalized for size, can be misleading, and how the proposed likelihood ratio approach enables accurate comparison of the trade-off between power and size between competing tests.

The prognostic value of Eastern Cooperative Oncology Group performance status on overall survival among patients with metastatic prostate cancer: a systematic review and meta-analysis

Background

There is heterogeneity in the literature regarding the strength of association between Eastern Cooperative Oncology Group performance status (ECOG PS) and mortality. We conducted a systematic review and meta-analysis of studies reporting the prognostic value of ECOG PS on overall survival (OS) in metastatic prostate cancer (mPC).

Methods

PubMed was searched from inception to March 21, 2022. A meta-analysis pooling the effect of ECOG PS categories (≥2 vs. <2, 2 vs. <2, and ≥1 vs. <1) on OS was performed separately for studies including patients with metastatic castration-resistant prostate cancer (mCRPC) and metastatic castration-sensitive prostate cancer (mCSPC) using a random-effects model. Analyses were stratified by prior chemotherapy and study type.

Results

Overall, 75 studies, comprising 32,298 patients, were included. Most studies (72/75) included patients with mCRPC. Higher ECOG PS was associated with a significant increase in mortality risk, with the highest estimate observed among patients with mCRPC with an ECOG PS of ≥2 versus <2 (hazard ratio [HR]: 2.10, 95% confidence interval [CI]: 1.87-2.37). When stratifying by study type, there was a higher risk estimate of mortality among patients with mCRPC with an ECOG PS of ≥1 versus <1 in real-world data studies (HR: 1.98, 95% CI: 1.72-2.26) compared with clinical trials (HR: 1.32, 95% CI: 1.13-1.54; p < 0.001). There were no significant differences in the HR of OS stratified by previous chemotherapy.

Conclusion

ECOG PS was a significant predictor of OS regardless of category, previous chemotherapy, and mPC population. Additional studies are needed to better characterize the effect of ECOG PS on OS in mCSPC.

Assessment of consistency between peer-reviewed publications and clinical trial registrations in nursing journals

BACKGROUND

The inconsistencies between randomized clinical trials (RCTs) registrations and peer-reviewed publications may distort trial results and threaten the validity of evidence-based medicine. Previous studies have found many inconsistencies between RCTs registrations and peer-reviewed publications, and outcome reporting bias is prevalent.

AIMS

The aims of this review were to assess whether the primary outcomes and other data reported in publications and registered records in RCTs of nursing journals were consistent and whether discrepancies in the reporting of primary outcomes favored statistically significant results. Moreover, we reviewed the proportion of RCTs for prospective registration.

METHODS

We systematically searched PubMed for RCTs published in the top 10 nursing journals between March 5, 2020, and March 5, 2022. Registration numbers were extracted from the publications, and registered records were identified from the registration platforms. The publications and registered records were compared to identify consistency. Inconsistencies were subdivided into discrepancies and omissions.

RESULTS

A total of 70 RCTs published in seven journals were included. The inconsistencies involved sample size estimation (71.4%), random sequence generation (75.7%), allocation concealment (97.1%), blinding (82.9%), primary outcomes (60.0%) and secondary outcomes (84.3%). Among the inconsistencies in the primary outcomes, 21.4% were due to discrepancies and 38.6% resulted from omissions. Fifty-three percent (8/15) presented discrepancies in the primary outcomes that favored statistically significant results. Additionally, although only 40.0% of the studies were prospective registrations, the number of prospectively registered trials has trended upward over time.

LINKING EVIDENCE TO ACTION

While not including all RCTs in the nursing field, our sample reflected a general trend: inconsistencies between publications and trial registrations were prevalent in the included nursing journals. Our research helps to provide a way to improve the transparency of research reports. Ensuring that clinical practice has access to transparent and reliable research results are essential to achieve the best possible evidence-based medicine.

Statistical significance and publication reporting bias in abstracts of reproductive medicine studies

STUDY QUESTION

What were the frequency and temporal trends of reporting P-values and effect measures in the abstracts of reproductive medicine studies in 1990-2022, how were reported P-values distributed, and what proportion of articles that present with statistical inference reported statistically significant results, i.e. 'positive' results?

SUMMARY ANSWER

Around one in six abstracts reported P-values alone without effect measures, while the prevalence of effect measures, whether reported alone or accompanied by P-values, has been increasing, especially in meta-analyses and randomized controlled trials (RCTs); the reported P-values were frequently observed around certain cut-off values, notably at 0.001, 0.01, or 0.05, and among abstracts present with statistical inference (i.e. P-value, CIs, or significant terms), a large majority (77%) reported at least one statistically significant finding.

WHAT IS KNOWN ALREADY

Publishing or reporting only results that show a 'positive' finding causes bias in evaluating interventions and risk factors and may incur adverse health outcomes for patients.

Despite efforts to minimize publication reporting bias in medical research, it remains unclear whether the magnitude and patterns of the bias have changed over time.

STUDY DESIGN, SIZE, DURATION

We studied abstracts of reproductive medicine studies from 1990 to 2022. The reproductive medicine studies were published in 23 first-quartile journals under the category of Obstetrics and Gynaecology and Reproductive Biology in Journal Citation Reports and 5 high-impact general medical journals (The Journal of the American Medical Association, The Lancet, The BMJ, The New England Journal of Medicine, and PLoS Medicine). Articles without abstracts, animal studies, and non-research articles, such as case reports or guidelines, were excluded.

PARTICIPANTS/MATERIALS, SETTING, METHODS

Automated text-mining was used to extract three types of statistical significance reporting, including P-values, CIs, and text description. Meanwhile, abstracts were text-mined for the presence of effect size metrics and Bayes factors. Five hundred abstracts were randomly selected and manually checked for the accuracy of automatic text extraction. The extracted statistical significance information was then analysed for temporal trends and distribution in general as well as in subgroups of study designs and journals.

MAIN RESULTS AND THE ROLE OF CHANCE

A total of 24 907 eligible reproductive medicine articles were identified from 170 739 screened articles published in 28 journals. The proportion of abstracts not reporting any statistical significance inference halved from 81% (95% CI, 76-84%) in 1990 to 40% (95% CI, 38-44%) in 2021, while reporting P-values alone remained relatively stable, at 15% (95% CI, 12-18%) in 1990 and 19% (95% CI, 16-22%) in 2021. By contrast, the proportion of abstracts reporting effect measures alone increased considerably from 4.1% (95% CI, 2.6-6.3%) in 1990 to 26% (95% CI, 23-29%) in 2021. Similarly, the proportion of abstracts reporting effect measures together with P-values showed substantial growth from 0.8% (95% CI, 0.3-2.2%) to 14% (95% CI, 12-17%) during the same timeframe. Of 30 182 statistical significance inferences, 56% (n = 17 077) conveyed statistical inferences via P-values alone, 30% (n = 8945) via text description alone such as significant or non-significant, 9.3% (n = 2820) via CIs alone, and 4.7% (n = 1340) via both CI and P-values. The reported P-values (n = 18 417), including both a continuum of P-values and dichotomized P-values, were frequently observed around common cut-off values such as 0.001 (20%), 0.05 (16%), and 0.01 (10%). Of the 13 200 reproductive medicine abstracts containing at least one statistical inference, 77% of abstracts made at least one statistically significant statement. Among articles that reported statistical inference, a decline in the proportion of making at least one statistically significant inference was only seen in RCTs, dropping from 71% (95% CI, 48-88%) in 1990 to 59% (95% CI, 42-73%) in 2021, whereas the proportion in the rest of study types remained almost constant over the years. Of abstracts that reported P-value, 87% (95% CI, 86-88%) reported at least one statistically significant P-value; it was 92% (95% CI, 82-97%) in 1990 and reached its peak at 97% (95% CI, 93-99%) in 2001 before declining to 81% (95% CI, 76-85%) in 2021.

LIMITATIONS, REASONS FOR CAUTION

First, our analysis focused solely on reporting patterns in abstracts but not full-text papers; however, in principle, abstracts should include condensed impartial information and avoid selective reporting. Second, while we attempted to identify all types of statistical significance reporting, our text mining was not flawless. However, the manual assessment showed that inaccuracies were not frequent.

WIDER IMPLICATIONS OF THE FINDINGS

There is a welcome trend that effect measures are increasingly reported in the abstracts of reproductive medicine studies, specifically in RCTs and meta-analyses. Publication reporting bias remains a major concern. Inflated estimates of interventions and risk factors could harm decisions built upon biased evidence, including clinical recommendations and planning of future research.

STUDY FUNDING/COMPETING INTEREST(S)

No funding was received for this study. B.W.M. is supported by an NHMRC Investigator grant (GNT1176437); B.W.M. reports research grants and travel support from Merck and consultancy from Merch and ObsEva. W.L. is supported by an NHMRC Investigator Grant (GNT2016729). Q.F. reports receiving a PhD scholarship from Merck. The other author has no conflict of interest to declare.

TRIAL REGISTRATION NUMBER

N/A.

DTAmetasa: An R shiny application for meta-analysis of diagnostic test accuracy and sensitivity analysis of publication bias

Meta-analysis of diagnostic test accuracy (DTA) is a powerful statistical method for synthesizing and evaluating the diagnostic capacity of medical tests and has been extensively used by clinical physicians and healthcare decision-makers. However, publication bias (PB) threatens the validity of meta-analysis of DTA. Some statistical methods have been developed to deal with PB in meta-analysis of DTA, but implementing these methods requires high-level statistical knowledge and programming skill. To assist non-technical users in running most routines in meta-analysis of DTA and handling with PB, we developed an interactive application, DTAmetasa. DTAmetasa is developed as a web-based graphical user interface based on the R shiny framework. It allows users to upload data and conduct meta-analysis of DTA by "point and click" operations. Moreover, DTAmetasa provides the sensitivity analysis of PB and presents the graphical results to evaluate the magnitude of the PB under various publication mechanisms. In this study, we introduce the functionalities of DTAmetasa and use the real-world meta-analysis to show its capacity for dealing with PB.

A Classification Bias and an Exclusion Bias Jointly Overinflated the Estimation of Publication Biases in Bilingualism Research

A publication bias has been argued to affect the fate of results in bilingualism research. It was repeatedly suggested that studies presenting evidence for bilingual advantages are more likely to be published compared to studies that do not report results in favor of the bilingual advantage hypothesis. This work goes back to the original claim and re-examines both the dataset and the classification of the studies that were employed. We find that the exclusion of published works such as doctoral dissertations, book chapters, and conference proceedings from the original dataset significantly inflated the presumed publication bias. Moreover, the estimation of the publication bias was affected by a classification bias that uses a mega-category that consists of both null and negative outcomes. Yet finding evidence for a bilingual disadvantage is not synonymous with obtaining a result indistinguishable from zero. Consequently, grouping together null and negative findings in a mega-category has various ramifications, not only for the estimation of the presumed publication bias but also for the field's ability to appreciate the insofar hidden correlations between bilingual advantages and disadvantages. Tracking biases that inflate scientific results is important, but it is not enough. The next step is recognizing the nested Matryoshka doll effect of bias-within-bias, and this entails raising awareness for one's own bias blind spots in science.

Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability

Two factors that decrease the replicability of studies in the scientific literature are publication bias and studies with underpowered desgins. One way to ensure that studies have adequate statistical power to detect the effect size of interest is by conducting a-priori power analyses. Yet, a previous editorial published in the Journal of Sports Sciences reported a median sample size of 19 and the scarce usage of a-priori power analyses. We meta-analysed 89 studies from the same journal to assess the presence and extent of publication bias, as well as the average statistical power, by conducting a z-curve analysis. In a larger sample of 174 studies, we also examined a) the usage, reporting practices and reproducibility of a-priori power analyses; and b) the prevalence of reporting practices of t-statistic or F-ratio, degrees of freedom, exact p-values, effect sizes and confidence intervals. Our results indicate that there was some indication of publication bias and the average observed power was low (53% for significant and non-significant findings and 61% for only significant findings). Finally, the usage and reporting practices of a-priori power analyses as well as statistical results including test statistics, effect sizes and confidence intervals were suboptimal.

Unpublished completed obstetric randomized clinical trials registered at ClinicalTrials.gov: how big is this issue?

BACKGROUND

A portion of obstetrical randomized clinical trials registered on ClinicalTrials.gov are not published in peer-reviewed journals.

OBJECTIVE

This study aimed to compare the characteristics of completed published vs unpublished randomized clinical trials in obstetrics registered on ClinicalTrials.gov and to identify barriers to publication.

STUDY DESIGN

This cross-sectional study queried ClinicalTrials.gov for all completed obstetrical randomized clinical trials registered between January 1, 2009, and December 31, 2018. For each completed obstetrical randomized clinical trial, we abstracted the following registration fields from ClinicalTrials.gov: ClinicalTrials.gov identifier, recruitment status, trial start and completion dates, study results, type of intervention, study phase, enrollment size, funder type, location, and facilities. Calculated variables included time to completion. In May 2021, we used PubMed and Google Scholar to identify the publication status of completed trials, and we compared the characteristics of published vs unpublished randomized clinical trials. The corresponding authors' e-mail addresses for the unpublished studies were collected from ClinicalTrials.gov and departmental websites. Between September 2021 and March 2022, the authors of these completed but unpublished obstetrical randomized clinical trials were contacted and invited to respond to a survey examining perceptions of barriers to publication, responses of which were collected and presented as counts and percentages.

RESULTS

Of the 647 obstetrical randomized clinical trials marked as completed on ClinicalTrials.gov, 378 (58%) were published, and 269 (42%) were unpublished. Unpublished trials were more likely to have an enrollment size of <50 participants (14.5% published vs 25.3% unpublished; P<.001) and less likely to be conducted at multiple sites (25.4% published vs 17.5% unpublished; P<.02). The main barriers to publication reported in the survey by authors whose trials were not published included lack of time (30%), change in employment or completion of training (25%), and results that were not of statistical significance (15%).

CONCLUSION

Among the obstetrical randomized clinical trials registered and marked as completed on ClinicalTrials.gov, more than 40% were unpublished. Unpublished trials were more likely to be smaller studies, conducted by researchers who reported experiencing a lack of time as the most common barrier to study publication.

Are European clinical trial funders policies on clinical trial registration and reporting improving? A cross-sectional study

Objectives

Assess the extent to which the clinical trial registration and reporting policies of 25 of the world's largest public and philanthropic medical research funders meet best practice benchmarks as stipulated by the 2017 WHO Joint Statement, and document changes in the policies and monitoring systems of 19 European funders over the past year.

Design Setting Participants

Cross-sectional study, based on assessments of each funder's publicly available documentation plus validation of results by funders. Our cohort includes 25 of the largest medical research funders in Europe, Oceania, South Asia, and Canada.

Interventions

Scoring all 25 funders using an 11-item assessment tool based on WHO best practice benchmarks, grouped into three primary categories: trial registries, academic publication, and monitoring, plus validation of results by funders.

Main outcome measures

How many of the 11 WHO best practice items each of the 25 funders has put into place, and changes in the performance of 19 previously assessed funders over the preceding year.

Results

The 25 funders we assessed had put into place an average of 5/11 (49%) WHO best practices. Only 6/25 funders (24%) took the PI's past reporting record into account during grant application reviews. Funders' performance varied widely from 0/11 to 11/11 WHO best practices adopted. Of the 19 funders for which 2021(2) baseline data was available, 10/19 (53%) had strengthened their policies over the preceding year.

Conclusions

Most medical research funders need to do more to curb research waste and publication bias by strengthening their clinical trial policies.

IFN-gene polymorphisms as a risk factor for tuberculosis infection in Asian populations: A meta-analysis

Background

The purpose of this meta-analysis is to verify that rs1861494 and rs2069718, two polymorphisms in the IFN-gene, are associated with tuberculosis (TB) infection in Asian populations.

Methods

To find appropriate case-control studies, a search was done from the databases, including Google Scholar, Science Direct, Embase, and PubMed. With the aid of MetaGenyo software, statistical analyses were performed. Case and control studies from the available database were used to investigate the relationship between IFN-γ gene polymorphisms and TB infection risk. The protocol for the present meta-analysis was registered using PROSPERO (ID Number: 443605).

Results

Information obtained through examining two different variants of the IFN-γ gene showed associations with recessive, allelic, overdominant, and dominant models. This indicates that the statistical value obtained was found to be statistically significant at P = 0.05. The findings of the IFN-γ rs1861494 gene polymorphisms for allelic, dominant, and overdominant models were statistically significant with P > 0.05, whereas the recessive model exhibited a statistically insignificant value (P = 0.25). The IFN-γ rs2069718 gene polymorphism demonstrated statistically significant value for overdominant, recessive, and allelic models (P > 0.05). However, the dominant model shows a statistically insignificant value P = 0.11.

Conclusion

The two genetic variations of the IFN-γ gene polymorphisms (rs1861494 and rs2069718) and their association with TB were confirmed by the meta-analysis conducted. More in-depth research into the molecular basis of the association is necessary, and larger-scale epidemiological studies are needed to confirm these findings.

Meta-analyses in psychology often overestimate evidence for and size of effects

Adjusting for publication bias is essential when drawing meta-analytic inferences. However, most methods that adjust for publication bias do not perform well across a range of research conditions, such as the degree of heterogeneity in effect sizes across studies. Sladekova et al. 2022 (Estimating the change in meta-analytic effect size estimates after the application of publication bias adjustment methods. Psychol. Methods) tried to circumvent this complication by selecting the methods that are most appropriate for a given set of conditions, and concluded that publication bias on average causes only minimal over-estimation of effect sizes in psychology. However, this approach suffers from a 'Catch-22' problem-to know the underlying research conditions, one needs to have adjusted for publication bias correctly, but to correctly adjust for publication bias, one needs to know the underlying research conditions. To alleviate this problem, we conduct an alternative analysis, robust Bayesian meta-analysis (RoBMA), which is not based on model-selection but on model-averaging. In RoBMA, models that predict the observed results better are given correspondingly larger weights. A RoBMA reanalysis of Sladekova et al.'s dataset reveals that more than 60% of meta-analyses in psychology notably overestimate the evidence for the presence of the meta-analytic effect and more than 50% overestimate its magnitude.

Erector spinae plane block: the ultimate 'plan A' block?

The erector spinae plane block (ESPB) is one of seven 'Plan A' blocks proposed by Regional Anaesthesia UK, covering the key areas of commonly encountered surgeries and acute pain. Unlike the other six blocks, the ESPB can be performed at all levels of the spine and provides analgesia to most regions of the body, leading to the argument that the ESPB is the ultimate Plan A block. Current studies show a high level of evidence supporting use in thoracoabdominal surgery but a lack of benefit in upper and lower limb surgery compared with local infiltration and other Plan A blocks. Thus, there is insufficient evidence to support the claim that the erector spinae plane block is the ultimate Plan A block.

Healthcare Workers' Attitudes towards Mandatory COVID-19 Vaccination: A Systematic Review and Meta-Analysis

BACKGROUND

COVID-19 vaccine mandates are considered a controversial public health policy both in public debate and among healthcare workers (HCWs). Thus, the objective of this systematic review is to give a deep insight into HCWs' views and attitudes towards COVID-19 vaccination mandates amid the ongoing COVID-19 pandemic.

METHODS

A systematic literature search of five databases (PubMed, Scopus, Embase, CINAHL, and Web of Science) was conducted between July 2022 and November 2022. Original quantitative studies that addressed the attitudes of HCWs regarding COVID-19 vaccine mandates were considered eligible for this systematic review. All the included studies (n = 57) were critically appraised and assessed for risk of systematic bias. Meta-analyses were performed, providing a pooled estimate of HCWs' acceptance towards COVID-19 vaccine mandates for: 1. HCWs and 2. the general population.

RESULTS

In total, 64% (95% CI: 55%, 72%) of HCWs favored COVID-19 vaccine mandates for HCWs, while 50% (95% CI: 38%, 61%) supported mandating COVID-19 vaccines for the general population.

CONCLUSIONS

Our findings indicate that mandatory vaccination against COVID-19 is a highly controversial issue among HCWs. The present study provides stakeholders and policy makers with useful evidence related to the compulsory or non-compulsory nature of COVID-19 vaccinations for HCWs and the general population. Other: The protocol used in this review is registered on PROSPERO with the ID number: CRD42022350275.

Toward Open and Reproducible Epidemiology

Starting in the 2010s, researchers in the experimental social sciences rapidly began to adopt increasingly open and reproducible scientific practices. These practices include publicly sharing deidentified data when possible, sharing analytical code, and preregistering study protocols. Empirical evidence from the social sciences suggests such practices are feasible, can improve analytical reproducibility, and can reduce selective reporting. In academic epidemiology, adoption of open-science practices has been slower than in the social sciences (with some notable exceptions, such as registering clinical trials). Epidemiologic studies are often large, complex, conceived after data have already been collected, and difficult to replicate directly by collecting new data. These characteristics make it especially important to ensure their integrity and analytical reproducibility. Open-science practices can also pay immediate dividends to researchers' own work by clarifying scientific reasoning and encouraging well-documented, organized workflows. We consider how established epidemiologists and early-career researchers alike can help midwife a culture of open science in epidemiology through their research practices, mentorship, and editorial activities.

Nonpublication and Selective Outcome Reporting of Oral Health Trials

Less than one-quarter of oral health trials are registered in a public registry. However, no study has assessed the extent of study publication and selective outcome reporting bias in the field of oral health. We identified oral health trials registered between 2006 and 2016 in ClinicalTrials.gov. We assessed whether results of early discontinued trials, trials having an unknown status, and completed trials had been published and, among published trials, whether outcomes differed between the registered record and the corresponding publication. We included 1,399 trials, of which 81 (5.8%) were discontinued, 247 (17.7%) had an unknown status, and 1,071 (76.6%) were completed. The registration was prospective for 719 (51.9%) trials. Over half the registered trials were unpublished (n = 793, 56.7%). To explore the association between trials publication and characteristics of trials, we performed a multivariate logistic regression analysis. Trials conducted in the United States (P = 0.003) or Brazil (P < 0.001) were associated with increased odds of publication, whereas trials registered prospectively (P = 0.001) and industry-sponsored trials (P = 0.02) were associated with decreased odds. Among the 479 published trials with completed status, the primary outcomes of 215 (44.9%) articles differed from that registered. Major discrepancies consisted of the introduction of a new primary outcome in the published article (196 [91.2%]) and the transformation of a registered secondary outcome into a primary outcome (112 [52.1%]). In the remaining 264 (55.1%) trials, primary outcomes did not differ from that registered, but 141 (53.4%) had been registered retrospectively. Our study highlights the high rate of nonpublication and selective outcome reporting in the field of oral health. These results could alert sponsors, funders, authors of systematic reviews, and the oral health research community at large to combat the nondisclosure of trial results.

Transcriptional Responses as Biomarkers of General Toxicity: A Systematic Review and Meta-analysis on Metal-Exposed Bivalves

Through a systematic review and a series of meta-analyses, we evaluated the general responsiveness of putative transcriptional biomarkers of general toxicity and chemical stress. We targeted metal exposures performed on bivalves under controlled laboratory conditions and selected six transcripts associated with general toxicity for evaluation: catalase, glutathione-S-transferase, heat shock proteins 70 and 90, metallothionein, and superoxide dismutase. Transcriptional responses (n = 396) were extracted from published scientific articles (k = 22) and converted to log response ratios (lnRRs). By estimating toxic units, we normalized different metal exposures to a common scale, as a proxy of concentration. Using Bayesian hierarchical random effect models, we then tested the effects of metal exposure on lnRR, both for metal exposure in general and in meta-regressions using toxic unit and exposure time as independent variables. Corresponding analyses were also repeated with transcript and tissue as additional moderators. Observed patterns were similar for general and for transcript- and tissue-specific responses. The expected overall response to arbitrary metal exposure was an lnRR of 0.50, corresponding to a 65% increase relative to a nonexposed control. However, when accounting for publication bias, the estimated "true" response showed no such effect. Furthermore, expected response magnitude increased slightly with exposure time, but there was little support for general monotonic concentration dependence with regard to toxic unit. Altogether, the present study reveals potential limitations that need consideration prior to applying the selected transcripts as biomarkers in environmental risk assessment. Environ Toxicol Chem 2023;42:628-641. © 2022 The Authors. Environmental Toxicology and Chemistry published by Wiley Periodicals LLC on behalf of SETAC.

We don't know what we're missing: Evidence of a vastly undersampled invasive plant pool

Invasive plants are a prominent threat to ecosystems and economies worldwide. Knowing the identity of invasive plants is critical for preventing their introduction and spread. Yet several lines of evidence, including spatial and taxonomic biases in reporting and the ongoing emergence of new invasives, suggest that we are missing basic information about the identity of invasive plants. Using a database of invasive plants reported in the peer-reviewed literature between 1959 and 2020, we examined trends in the accumulation of new invasive plants over time and estimated the size of the current pool of invasive plants both continentally and globally. The number of new invasive plants continues to increase exponentially over time, showing no sign of saturation, even in the best studied regions. Moreover, a sample-size based rarefaction-extrapolation curve of reported taxa suggests that what is documented in the current literature (3008 taxa) only captures 64% of the likely number of invasive plants globally (4721 taxa ± 132 SE). These estimates varied continentally; less than half of invasive plant taxa have likely been identified in Oceania and Central and South Americas. Studies that included multiple invasive plants (e.g., floristic studies) were much more efficient at adding new taxa to our global understanding of what is invasive (identifying 4.2 times more new taxa than single-taxon studies). With more potential invaders arriving every day, this analysis highlights a critical gap in our knowledge of the current invasive plant pool. Expanding invasion science to better encompass understudied geographic areas and increasing the numbers of floristic surveys would greatly improve our ability to accurately and efficiently identify what taxa are invasive. Preventing invasive plant introductions is incumbent upon knowing the identity of invasive plants. Thus, large knowledge gaps remain in invasion ecology that hinder efforts to proactively prevent and manage invasive plants.

Factors relating to nonpublication and publication bias in clinical trials in Canada: A qualitative interview study

AIMS

This study aims to understand factors contributing to nonpublication and publication bias in clinical trials in Canada.

METHODS

Qualitative interviews were conducted between March 2019 and April 2021 with 34 participants from the Canadian provinces of Alberta, British Columbia and Ontario, including 17 clinical trial investigators, 1 clinical research coordinator, 3 research administrators, 3 research ethics board members and 10 clinical trial participants. We conducted a thematic analysis involving coding of interview transcripts and memo-writing to identify key themes.

RESULTS

Several factors contribute to nonpublication and publication bias in clinical trial research. A core theme was that reporting practices are shaped by incentives within the research system taht favour publication of positive over negative trials. Investigators are discouraged from reporting by experiences or perceptions of difficulty in publishing negative findings but rewarded for publishing positive findings in various ways. Trial investigators more strongly associated positive clinical trials than negative trials with opportunities for industry and nonindustry funding and with academic promotion, bonuses and recognition. Research institutions and ethics boards tended to lack well-resourced, proactive policies and practices to ensure trial findings are reported in registries or journals.

CONCLUSION

Clinical trial reporting practices in Canada are shaped by incentives favouring reporting of positive over negative trials, such as funding opportunities and academic promotion, bonuses and recognition. Research institutions could help change incentives by adopting performance metrics that emphasize full reporting of results in journals or registries.

The efficacy and safety of patient-specific instrumentation in primary total knee replacement: a systematic review and meta-analysis

INTRODUCTION

Patient-specific instrumentation (PSI) for primary total knee arthroplasty (TKA) surgery has been shown to increase accuracy of component positioning. However, it is unclear whether this also translates to actual benefits for patients in terms of better outcomes (efficacy) or less complications such as revisions (safety). We therefore systematically reviewed the literature to determine the efficacy and safety of PSI in primary TKA.

METHODS

Randomized controlled trials comparing PSI to non-PSI in primary TKA were included. A random effects model was used with meta-regression in case of heterogeneity.

RESULTS

Forty-three studies were included with a total of 1816 TKA in the PSI group and 1887 TKA in the control group. There were no clinically relevant differences between the PSI-group and non-PSI group regarding all outcomes. There was considerable heterogeneity: meta-regression analyses showed that the year the study was published was an important effect modifier. Early publications tended to show a positive effect for PSI compared to non-PSI TKA, whereas later studies found the opposite.

CONCLUSION

Based on evidence of moderate certainty, our study suggested that there were no clinically relevant differences in efficacy and safety between patients treated with PSI TKA and patients treated with non-PSI TKA.

Publication of studies registered in Clinical Trials Registry of India: An audit of mood disorder research protocols from 2009-2019

BACKGROUND

Selective publication of studies has important scientific, ethical, and public health implications.

AIM

We studied selective publication among mood disorder research protocols registered in the Clinical Trials Registry of India (CTRI) database. We also examined the frequency and nature of protocol deviations among the published articles.

METHODS

Using a systematic search strategy, we examined the publication status of all mood disorder-related research protocols registered in the CTRI database from inception till December 31, 2019. Logistic regression analysis was used to identify variables associated with selective publication.

RESULTS

Of 129 eligible protocols identified, only a third (n = 43, 33.3%) were published in literature; among those published, only 28 (21.7%) were placed in MEDLINE indexed journals. Protocol deviations were observed in more than half of the published papers (n = 25, 58.1%); many of these (41.9%) were related to sample size deviations, though, importantly, deviations in primary and secondary outcomes were also noted (16.2%). Retrospective registration of trials (odds ratio, 2.98, 95% confidence interval, 1.32-6.71) was significantly associated with publication; other variables, such as funding status or multicentric sampling, were not associated with eventual publication.

CONCLUSIONS

Two out of three mood disorder research protocols registered in India do not translate into published research. These findings from a low- and middle-income country with limited spending on health care research and development represent wastage of resources and raise scientific and ethical concerns about unpublished data and futile patient participation in research.

Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods

Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.

Has the degree of outcome reporting bias in surgical randomized trials changed? A meta-regression analysis

BACKGROUND

Outcome reporting bias in individual trials can compromise the validity of pooled estimates within systematic reviews. Recent strategies have attempted to address outcome reporting bias, which favours the full reporting of statistically significant outcomes over non-significant outcomes. We examined whether the association between full outcome reporting and statistical significance in surgical trials has changed from 2009 to 2019.

METHODS

We systematically searched for 350 surgical randomized controlled trials (RCTs) from 2009 and 350 surgical RCTs from 2019. Outcomes were classified as fully reported, partially reported, qualitatively reported or unreported. For each outcome, a contingency table was populated with full outcome reporting (yes/no) and statistical significance (yes/no). We combined odds ratios in random effects meta-analysis to estimate the association between full outcome reporting and statistical significance in 2009 compared with 2019.

RESULTS

Twenty-eight percent of outcomes in 2009 were incompletely reported, compared with 30% in 2019. In 2009, significant outcomes were more likely to be fully reported than non-significant outcomes (OR = 2.4, 95% CI 1.7-3.4, I²  = 35%), but the opposite association was seen in 2019 (OR = 0.51, 95% CI 0.34-0.77, I²  = 43%). RCTs from 2019 were less likely to demonstrate outcome reporting bias favouring significant outcomes (OR = 0.21, 95% CI 0.12-0.35, P < 0.001).

CONCLUSION

Outcome reporting bias favouring the full reporting of significant over non-significant outcomes was demonstrated in 2009, but the opposite association was seen in 2019. There remains a high prevalence of incomplete outcome reporting. We recommend ongoing adherence to trial protocol guidelines to improve outcome reporting transparency and completeness.

Impact of searching clinical trials registers in systematic reviews of pharmaceutical and non-pharmaceutical interventions: Reanalysis of meta-analyses

Systematic reviewers are advised to search trials registers to minimise risk of reporting biases. However, there has been little research on the impact of searching trials registers on the results of meta-analyses. We aimed to evaluate the impact of searching clinical trials registers for systematic reviews of pharmaceutical or non-pharmaceutical interventions. We searched PubMed, Scopus, Science Citation Index and Social Sciences Citation Index, and Education Collection for systematic reviews with meta-analyses indexed from 2 November to 2 December 2020. A random sample of systematic reviews was initially drawn, and for reviews which considered randomised trials eligible for inclusion, which had not searched a trials register, we searched ClinicalTrials.gov, EudraCT, ANZCTR, and the WHO ICTRP search portal for eligible trials. We compared meta-analytic effect estimates before and after including results from additional trials identified. We found additional trials for 63% (63/101) of eligible reviews; however, trials with results that could contribute to a meta-analysis were identified for only 20% (20/101) of the reviews. On average, there was no difference in the meta-analytic effect estimates before versus after adding the new trials. In summary, searching clinical trial registers led to identification of additional trials for many reviews; however, very few trials had results available for inclusion in meta-analyses. Including results from the new trials led to no change in the meta-analytic estimates, on average. Trials registers would be even more valuable to systematic reviewers if more trialists made use of them (i.e., registered their trials and posted results in a timely manner).

A likelihood-based sensitivity analysis for publication bias on the summary receiver operating characteristic in meta-analysis of diagnostic test accuracy

In meta-analysis of diagnostic test accuracy, the summary receiver operating characteristic (SROC) curve is a recommended method to summarize the diagnostic capacity of a medical test in the presence of study-specific cutoff values. The SROC curve can be estimated by bivariate modeling of pairs of sensitivity and specificity across multiple diagnostic studies, and the area under the SROC curve (SAUC) gives the aggregate estimate of diagnostic test accuracy. However, publication bias is a major threat to the validity of the estimates. To make inference of the impact of publication bias on the SROC curve or the SAUC, we propose a sensitivity analysis method by extending the likelihood-based sensitivity analysis of Copas. In the proposed method, the SROC curve or the SAUC are estimated by maximizing the likelihood constrained by different values of the marginal probability of selective publication under different mechanisms of selective publication. A cutoff-dependent selection function is developed to model the selective publication mechanism via thet $$ t $$ -type statistics orP $$ P $$ -value of the linear combination of the logit-transformed sensitivity and specificity from the published studies. It allows us to model selective publication suggested by the funnel plots of sensitivity, specificity, or diagnostic odds ratio, which are often observed in practice. A real meta-analysis of diagnostic test accuracy is re-analyzed to illustrate the proposed method, and simulation studies are conducted to evaluate its performance.

Digital Biomarker-Based Interventions: Systematic Review of Systematic Reviews

BACKGROUND

The introduction of new medical technologies such as sensors has accelerated the process of collecting patient data for relevant clinical decisions, which has led to the introduction of a new technology known as digital biomarkers.

OBJECTIVE

This study aims to assess the methodological quality and quality of evidence from meta-analyses of digital biomarker-based interventions.

METHODS

This study follows the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guideline for reporting systematic reviews, including original English publications of systematic reviews reporting meta-analyses of clinical outcomes (efficacy and safety endpoints) of digital biomarker-based interventions compared with alternative interventions without digital biomarkers. Imaging or other technologies that do not measure objective physiological or behavioral data were excluded from this study. A literature search of PubMed and the Cochrane Library was conducted, limited to 2019-2020. The quality of the methodology and evidence synthesis of the meta-analyses were assessed using AMSTAR-2 (A Measurement Tool to Assess Systematic Reviews 2) and GRADE (Grading of Recommendations, Assessment, Development, and Evaluations), respectively. This study was funded by the National Research, Development and Innovation Fund of Hungary.

RESULTS

A total of 25 studies with 91 reported outcomes were included in the final analysis; 1 (4%), 1 (4%), and 23 (92%) studies had high, low, and critically low methodologic quality, respectively. As many as 6 clinical outcomes (7%) had high-quality evidence and 80 outcomes (88%) had moderate-quality evidence; 5 outcomes (5%) were rated with a low level of certainty, mainly due to risk of bias (85/91, 93%), inconsistency (27/91, 30%), and imprecision (27/91, 30%). There is high-quality evidence of improvements in mortality, transplant risk, cardiac arrhythmia detection, and stroke incidence with cardiac devices, albeit with low reporting quality. High-quality reviews of pedometers reported moderate-quality evidence, including effects on physical activity and BMI. No reports with high-quality evidence and high methodological quality were found.

CONCLUSIONS

Researchers in this field should consider the AMSTAR-2 criteria and GRADE to produce high-quality studies in the future. In addition, patients, clinicians, and policymakers are advised to consider the results of this study before making clinical decisions regarding digital biomarkers to be informed of the degree of certainty of the various interventions investigated in this study. The results of this study should be considered with its limitations, such as the narrow time frame.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/28204.

A Meta-Epidemiological Study of Positive Results in Clinical Nutrition Research: The Good, the Bad and the Ugly of Statistically Significant Findings

Positive (statistically significant) findings are easily produced in nutrition research when specific aspects of the research design and analysis are not accounted for. To address this issue, recently, a pledge was made to reform nutrition research and improve scientific trust on the science, encompass research transparency and achieve reproducibility. The aim of the present meta-epidemiological study was to evaluate the statistical significance status of research items published in three academic journals, all with a focus on clinical nutrition science and assessing certain methodological/transparency issues. All research items were published between the years 2015 and 2019. Study design, primary and secondary findings, sample size and age group, funding sources, positivist findings, the existence of a published research protocol and the adjustment of nutrients/dietary indexes to the energy intake (EI) of participants, were extracted for each study. Out of 2127 studies in total, those with positive findings consisted of the majority, in all three journals. Most studies had a published research protocol, however, this was mainly due to the randomized controlled trials and not to the evidence-synthesis studies. No differences were found in the distribution of positive findings according to the existence/inexistence of a published research protocol. In the pooled sample of studies, positive findings differed according to study design and more significant findings were reported by researchers failing to report any funding source. The majority of items published in the three journals (65.9%) failed to account for the EI of participants. The present results indicate that there is still room for the improvement of nutrition research in terms of design, analyses and reporting.

Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field

Known methodological issues such as publication bias, questionable research practices and studies with underpowered designs are known to decrease the replicability of study findings. The presence of such issues has been widely established across different research fields, especially in psychology. Their presence raised the first concerns that the replicability of study findings could be low and led researchers to conduct large replication projects. These replication projects revealed that a significant portion of original study findings could not be replicated, giving rise to the conceptualization of the replication crisis. Although previous research in the field of sports and exercise science has identified the first warning signs, such as an overwhelming proportion of significant findings, small sample sizes and lack of data availability, their possible consequences for the replicability of our field have been overlooked. We discuss the consequences of the above issues on the replicability of our field and offer potential solutions to improve replicability.

A p-curve analysis of the emotional Stroop effect among women with eating disorders

OBJECTIVE

A recent meta-review of attentional bias research in eating disorders suggests that meta-analyses and systematic reviews include many low-quality and underpowered studies (Stott et al., 2021). As such, we examined whether published research examining the link between attentional bias, using the emotional Stroop task, and eating disorders among women with eating disorders has evidential value (ruling out selective reporting of a statistically significant effect) using a p-curve analysis. A p-curve analysis plots statisticall significant p-values onto a curve from .01 to .05 to examine its distribution. We hypothesized that the p-curve would be flat, indicating no true effect.

METHOD

The hypothesis, database search strategy, and data analytic approach were pre-registered. The inclusion criteria were reports that compared control and eating disorder groups, reported inferential statistics, and that used body shape/weight or general threat target words.

RESULTS

Fifty published reports were included in the p-curve analyses. Unexpectedly, the half and full p-curves were significantly right-skewed, indicating evidential value. However, the results were not robust to the exclusion of the seven lowest p-values and on average, reports were underpowered. There were also 18 reports with null results (they had a p-value greater than .05), which precluded their inclusion in the p-curve analyses.

DISCUSSION

The findings suggest that most of the evidence from research examining attentional biases using the emotional Stroop task among women with an eating disorder or with elevated eating disorder symptoms is underpowered and so should be interpreted with considerable caution.

PUBLIC SIGNIFICANCE

Concerns have been raised about the low quality of research examining attentional biases among women with eating disorders using the emotional Stroop task. In the current research, we observed that the evidential value of primary research reporting differences between women with and without eating disorders was equivocal and had low statistical power. These results can guide researchers towards conducting more rigorous research on attentional biases among people with eating disorders.

Residual Disease After Primary Surgical Treatment for Advanced Epithelial Ovarian Cancer, Part 2: Network Meta-analysis Incorporating Expert Elicitation to Adjust for Publication Bias

BACKGROUND

Previous work has identified a strong association between the achievements of macroscopic cytoreduction and improved overall survival (OS) after primary surgical treatment of advanced epithelial ovarian cancer. Despite the use of contemporary methodology, resulting in the most comprehensive currently available evidence to date in this area, opponents remain skeptical.

AREAS OF UNCERTAINTY

We aimed to conduct sensitivity analyses to adjust for potential publication bias, to confirm or refute existing conclusions and recommendations, leveraging elicitation to incorporate expert opinion. We recommend our approach as an exemplar that should be adopted in other areas of research.

DATA SOURCES

We conducted random-effects network meta-analyses in frequentist and Bayesian (using Markov Chain Montel Carlo simulation) frameworks comparing OS across residual disease thresholds in women with advanced epithelial ovarian cancer after primary cytoreductive surgery. Elicitation methods among experts in gynecology were used to derive priors for an extension to a previously reported Copas selection model and a novel approach using effect estimates calculated from the elicitation exercise, to attempt to adjust for publication bias and increase confidence in the certainty of the evidence.

THERAPEUTIC ADVANCES

Analyses using data from 25 studies (n = 20,927 women) all showed the prognostic importance of complete cytoreduction (0 cm) in both frameworks. Experts accepted publication bias was likely, but after adjustment for their opinions, published results overpowered the informative priors incorporated into the Bayesian sensitivity analyses. Effect estimates were attenuated but conclusions were robust in all analyses.

CONCLUSIONS

There remains a strong association between the achievement of complete cytoreduction and improved OS even after adjustment for publication bias using strong informative priors formed from an expert elicitation exercise. The concepts of the elicitation survey should be strongly considered for utilization in other meta-analyses.

Do Prospective Intent and Established Metrics Correlate with Journal Impact Factor in Musculoskeletal Physical Therapy Trials?: A Secondary Analysis of A Methodological Review

OBJECTIVES

To determine if there are any statistically significant associations between: 1) randomized clinical trials (RCTs) investigating physical therapy musculoskeletal interventions, 2) journal impact factor (JIF), 3) frequency of RCT citation, 4) whether prospective intent was identifiable, and 5) the Physiotherapy Evidence Database (PEDro) scores.

METHODS

MEDLINE indexed RCTs addressing musculoskeletal interventions published between January 2016 and July 2020 in physical therapy journals were included. Two blinded reviewers identified the RCTs and extracted the variables of interest.

RESULTS

With a familywise alpha adjustment, there was no statistically significant correlation between JIF and number of citations (rho = 0.187; p = 0.0280). Statistically significant weak positive correlations were identified between the JIF and prospectively registered RCTs (rho = 0.240; p = 0.0046), JIF and PEDro scores (rho = 0.250; p = 0.0031), and PEDro scores and prospectively registered RCTs (rho = 0.335; p < 0.0001).

CONCLUSION

The findings of this study suggest that JIF and PEDro scores may not be accurate measures of RCT quality. Failing to ensure that published RCTs followed their prospective intent and using bibliometrics that fail to accurately measure what they propose appears to create untrustworthy preprocessed resources for practicing physical therapists during the evidence-based practice process.

LEVEL OF EVIDENCE

1a.

Clinical Trial Quality Assessment in Adult Spinal Surgery: What Do Publication Status, Funding Source, and Result Reporting Tell Us?

STUDY DESIGN

Narrative Review.

OBJECTIVES

The objective of this study was to compare publication status of clinical trials in adult spine surgery registered on ClinicalTrials.gov by funding source as well as to identify other trends in clinical trials in adult spine surgery.

METHODS

All prospective, comparative, therapeutic (intervention-based) trials of adult spinal disease that were registered on ClinicalTrials.gov with a start date of January 1, 2000 and completion date before December 17, 2018 were included. Primary outcome was publication status of published or unpublished. A bivariate analysis was used to compare publication status to funding source of industry vs non-industry.

RESULTS

Our search identified 107 clinical trials. The most common source of funding was industry (62 trials, 57.9% of total), followed by University funding (26 trials, 24.3%). The results of 76 trials (71.0%) were published, with industry-funded trials less likely to be published compared to non-industry-funded trials (62.9% compared to 82.2%, P = .03). Of the 31 unpublished studies, 13 did not report any results on ClinicalTrials.gov, and of those with reported results, none was a positive trial.

CONCLUSIONS

Clinician researchers in adult spine surgery should be aware that industry-funded trials are less likely to go on to publication compared to non-industry-funded trials, and that negative trials are frequently not published. Future opportunities include improvement in result reporting and in publishing negative studies.

Do Salient Social Norms Moderate Mortality Salience Effects? A (Challenging) Meta-Analysis of Terror Management Studies

Terror management theory postulates that mortality salience (MS) increases the motivation to defend one's cultural worldviews. How that motivation is expressed may depend on the social norm that is momentarily salient. Meta-analyses were conducted on studies that manipulated MS and social norm salience. Results based on 64 effect sizes for the hypothesized interaction between MS and norm salience revealed a small-to-medium effect of g = 0.34, 95% confidence interval [0.26, 0.41]. Bias-adjustment techniques suggested the presence of publication bias and/or the exploitation of researcher degrees of freedom and arrived at smaller effect size estimates for the hypothesized interaction, in several cases reducing the effect to nonsignificance (range g_corrected = -0.36 to 0.15). To increase confidence in the idea that MS and norm salience interact to influence behavior, preregistered, high-powered experiments using validated norm salience manipulations are necessary. Concomitantly, more specific theorizing is needed to identify reliable boundary conditions of the effect.

Growth of non-English-language literature on biodiversity conservation

English is widely recognized as the language of science, and English-language publications (ELPs) are rapidly increasing. It is often assumed that the number of non-ELPs is decreasing. This assumption contributes to the underuse of non-ELPs in conservation science, practice, and policy, especially at the international level. However, the number of conservation articles published in different languages is poorly documented. Using local and international search systems, we searched for scientific articles on biodiversity conservation published from 1980 to 2018 in English and 15 non-English languages. We compared the growth rate in publications across languages. In 12 of the 15 non-English languages, published conservation articles significantly increased every year over the past 39 years, at a rate similar to English-language articles. The other three languages showed contrasting results, depending on the search system. Since the 1990s, conservation science articles in most languages increased exponentially. The variation in the number of non-English-language articles identified among the search systems differed markedly (e.g., for simplified Chinese, 11,148 articles returned with local search system and 803 with Scopus). Google Scholar and local literature search systems returned the most articles for 11 and 4 non-English languages, respectively. However, the proportion of peer-reviewed conservation articles published in non-English languages was highest in Scopus, followed by Web of Science and local search systems, and lowest in Google Scholar. About 20% of the sampled non-English-language articles provided no title or abstract in English; thus, in theory, they were undiscoverable with English keywords. Possible reasons for this include language barriers and the need to disseminate research in countries where English is not widely spoken. Given the known biases in statistical methods and study characteristics between English- and non-English-language studies, non-English-language articles will continue to play an important role in improving the understanding of biodiversity and its conservation.

Publication rate and research topics of studies in pediatric kidney transplantation

BACKGROUND

The quality of medical care for pediatric kidney transplant recipients depends on sound evidence from published clinical trials.

METHODS

We examined the publication rate, time to publication, and factors associated with publication of studies in pediatric kidney transplantation registered on ClinicalTrials.gov from 1999 to 2020.

RESULTS

We identified 136 studies with an overall enrollment of 36255 study participants, of which only 58.8% have been published yet. Unpublished studies included data from 14 350 participants. The median time to publication was 25 months (range, 0-117) with a significantly shorter time to publication in more recent years. The most frequently investigated research topic was immunosuppressants (49.3%), followed by perioperative management (11.0%) and infectiology (10.3%). The percentage of published studies was highest for the topic steroid withdrawal (87.5%), followed by infectiology (78.6%), and nutrition, sports and quality of life (71.4%). Studies, which were co-funded by industry, showed a significantly higher 5-year publication rate (p = 0.019).

CONCLUSIONS

In conclusion, nearly half of all studies in pediatric kidney transplantation remain unpublished. Non-publication of studies might lead to a publication bias with a negative impact on clinical decision-making.

Decline effects are rare in ecology

The scientific evidence base on any given topic changes over time as more studies are published. Currently, there is widespread concern about non-random, directional changes over time in the scientific evidence base associated with many topics. In particular, if studies finding large effects (e.g., large differences between treatment and control means) tend to get published quickly, while small effects tend to get published slowly, the net result will be a decrease over time in the estimated magnitude of the mean effect size, known as a "decline effect". If decline effects are common, then the published scientific literature will provide a biased and misleading guide to management decisions, and to the allocation of future research effort. We compiled data from 466 meta-analyses in ecology to look for evidence of decline effects. We found that decline effects are rare. Only ~5% of ecological meta-analyses truly exhibit a directional change in mean effect size over time arising for some reason other than random chance, usually but not always in the direction of decline. Most apparent directional changes in mean effect size over time are attributable to regression to the mean, consistent with primary studies being published in random order with respect to the effect sizes they report. Our results are good news: decline effects are the exception to the rule in ecology. Identifying and rectifying rare cases of true decline effects remains an important task, but ecologists should not overgeneralize from anecdotal reports of decline effects.

Improving clinical trial transparency at UK universities: Evaluating 3 years of policies and reporting performance on the European Clinical Trial Register

BACKGROUND

January 2019, the House of Commons' Science and Technology Committee sent letters to UK universities admonishing them to achieve compliance with results reporting requirements for Clinical Trials of Investigative Medicinal Products by summer 2019. This study documents changes in the clinical trial policies and Clinical Trials of Investigative Medicinal Product reporting performance of 20 major UK universities following that intervention.

METHODS

Freedom of Information requests were filed in June 2018 and June 2020 to obtain clinical trial registration and reporting policies covering both Clinical Trials of Investigative Medicinal Products and all other clinical trials. Two independent reviewers assessed policies against transparency benchmarks based on World Health Organization best practices. To evaluate universities' trial reporting performance, we used a public online tracking tool, the European Union Trials Tracker, which assesses universities' compliance with regulatory Clinical Trials of Investigative Medicinal Product disclosure requirements on the European Clinical Trial Register. Specifically, we evaluated whether universities were adhering to the European Union requirement to post summary results on the trial registry within 12 months of completion.

RESULTS

Mean policy strength increased from 2.8 to 4.9 points (out of a maximum of 7 points) between June 2018 and June 2020. In October 2018 the average percentage of due Clinical Trials of Investigative Medicinal Products that had results available on the European trial registry across university sponsors included in the cohort was 29%. By June 2021, this had increased to 91%, with 5 universities achieving a reporting performance of 100%. All 20 universities reported more than 70% of their due trial results on the European trial registry.

INTERPRETATION

Political pressure appears to have a significant positive impact on UK universities' clinical trial reporting policies and performance. Similar approaches could be used to improve reporting performance for other types of sponsors, other types of trials, and in other countries.

Shame and Self-Esteem: A Meta-Analysis

Scholars agree that shame has many effects related to psychological functioning declines, and one among others is the fluctuation of self-esteem. However, the association between shame and self-esteem requires further studies. Heterogeneity studies due to different measurements, various sample characteristics, and potential missing research findings may result in uncertain conclusions. This study aimed to explore the relationship between shame and self-esteem by meta-analysis to come up with evidence of heterogeneity and publication bias of the study. Eighteen studies from the initial 235 articles involving the term shame and self-esteem were studied using the random-effects model. A total of 578 samples were included in the study. The overall effect size estimate between shame and self-esteem (r = −.64) indicates that shame correlates negatively with self-esteem and is large effect size. The result showed that heterogeneity study was found (I² = 95.093%). The Meta-regression showed that age moderated the relationship between shame and self-esteem (p = .002), while clinical sample characteristics (p = .232) and study quality (p = .184) did not affect the overall effect size.

Association of Accuracy, Conclusions, and Reporting Completeness With Acceptance by Radiology Conferences and Journals

BACKGROUND

Preferential publication of studies with positive findings can lead to overestimation of diagnostic test accuracy (i.e. publication bias). Understanding the contribution of the editorial process to publication bias could inform interventions to optimize the evidence guiding clinical decisions.

PURPOSE/HYPOTHESIS

To evaluate whether accuracy estimates, abstract conclusion positivity, and completeness of abstract reporting are associated with acceptance to radiology conferences and journals.

STUDY TYPE

Meta-research.

POPULATION

Abstracts submitted to radiology conferences (European Society of Gastrointestinal and Abdominal Radiology (ESGAR) and International Society for Magnetic Resonance in Medicine (ISMRM)) from 2008 to 2018 and manuscripts submitted to radiology journals (Radiology, Journal of Magnetic Resonance Imaging [JMRI]) from 2017 to 2018. Primary clinical studies evaluating sensitivity and specificity of a diagnostic imaging test in humans with available editorial decisions were included.

ASSESSMENT

Primary variables (Youden's index [YI > 0.8 vs. <0.8], abstract conclusion positivity [positive vs. neutral/negative], number of reported items on the Standards for Reporting of Diagnostic Accuracy Studies [STARD] for Abstract guideline) and confounding variables (prospective vs. retrospective/unreported, sample size, study duration, interobserver agreement assessment, subspecialty, modality) were extracted.

STATISTICAL TESTS

Multivariable logistic regression to obtain adjusted odds ratio (OR) as a measure of the association between the primary variables and acceptance by radiology conferences and journals; 95% confidence intervals (CIs) and P-values were obtained; the threshold for statistical significance was P < 0.05.

RESULTS

A total of 1000 conference abstracts (500 ESGAR and 500 ISMRM) and 1000 journal manuscripts (505 Radiology and 495 JMRI) were included. Conference abstract acceptance was not significantly associated with YI (adjusted OR = 0.97 for YI > 0.8; CI = 0.70-1.35), conclusion positivity (OR = 1.21 for positive conclusions; CI = 0.75-1.90) or STARD for Abstracts adherence (OR = 0.96 per unit increase in reported items; CI = 0.82-1.18). Manuscripts with positive abstract conclusions were less likely to be accepted by radiology journals (OR = 0.45; CI = 0.24-0.86), while YI (OR = 0.85; CI = 0.56-1.29) and STARD for Abstracts adherence (OR = 1.06; CI = 0.87-1.30) showed no significant association. Positive conclusions were present in 86.7% of submitted conference abstracts and 90.2% of journal manuscripts.

DATA CONCLUSION

Diagnostic test accuracy studies with positive findings were not preferentially accepted by the evaluated radiology conferences or journals.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.