Reporting bias in the literature on the associations of health-related behaviors and statins with cardiovascular disease and all-cause mortality

Variation observed in consensus judgments between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research

OBJECTIVES

To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments.

STUDY DESIGN AND SETTING

We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet's Agreement Coefficient (AC) in consensus judgments between pairs.

RESULTS

Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet's AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet's AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as "low risk".

CONCLUSION

Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.

How Often do Medical Specialties Question the Practices that They Perform? An Empirical, Cross-Sectional Analysis of the Published Literature

An aspect of overuse is who decides which practices are evaluated for overuse and which of the studies on overuse are published in the medical literature. We sought to examine the frequency with which studies in medical journals questioned an established practice. As a secondary objective, we sought to determine if there was variance among medical specialties. We conducted a retrospective, cross-sectional review of the published literature in 14 medical specialty journals. We included studies from one issue in three high-impact journals (November/December 2020) for each specialty. We assessed whether the study reported on a medical practice, whether it reported on an existing practice, whether the author expressed uncertainty regarding the practice, whether the study was a randomized design, and if the authors encouraged further testing in randomized studies. For all medical specialties combined, we found that 37% (n = 98) questioned existing practices, and 15% (n = 40) either tested the practice in a randomized trial or encouraged future randomized testing of the practice. The medical specialties that questioned their practices the most were gastroenterology (61%; n = 10/18), obstetrics/gynecology (52%; n = 11/21), and cardiovascular (50%; n = 5/10). These findings indicate that, although research is being conducted to examine current medical practices, few studies advocate for randomized testing of these practices, and even fewer actually test them in a randomized fashion. Additionally, the variation across medical specialties suggests areas in which to look for potential practices that are low-value, duplicative, and/or wasteful.

Annual Research Review: Ecological momentary assessment studies in child psychology and psychiatry

BACKGROUND

Enhancements in mobile phone technology allow the study of children and adolescents' everyday lives like never before. Ecological momentary assessment (EMA) uses these advancements to allow in-depth measurements of links between context, behavior, and physiology in youths' everyday lives.

FINDINGS

A large and diverse literature now exists on using EMA to study mental and behavioral health among youth. Modern EMA methods are built on a rich tradition of idiographic inquiry focused on the intensive study of individuals. Studies of child and adolescent mental and behavioral health have used EMA to characterize lived experience, document naturalistic within-person processes and individual differences in these processes, measure familiar constructs in novel ways, and examine temporal order and dynamics in youths' everyday lives.

CONCLUSIONS

Ecological momentary assessment is feasible and reliable for studying the daily lives of youth. EMA can inform the development and augmentation of traditional and momentary intervention. Continued research and technological development in mobile intervention design and implementation, EMA-sensor integration, and complex real-time data analysis are needed to realize the potential of just-in-time adaptive intervention, which may allow researchers to reach high-risk youth with intervention content when and where it is needed most.

Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol

Background: Dietary guidelines should be informed by systematic reviews (SRs) of the available scientific evidence. However, if the SRs that underpin dietary guidelines are flawed in their design, conduct or reporting, the recommendations contained therein may be misleading or harmful. To date there has been little empirical investigation of bias due to selective inclusion of results, and bias due to missing results, in SRs of food/diet-outcome relationships. Objectives: To explore in SRs with meta-analyses of the association between food/diet and health-related outcomes: (i) whether systematic reviewers selectively included study effect estimates in meta-analyses when multiple effect estimates were available; (ii) what impact selective inclusion of study effect estimates may have on meta-analytic effects, and; (iii) the risk of bias due to missing results (publication bias and selective non-reporting bias) in meta-analyses. Methods: We will systematically search for SRs with meta-analysis of the association between food/diet and health-related outcomes in a generally healthy population, published between January 2018 and June 2019. We will randomly sort titles and abstracts and screen them until we identify 50 eligible SRs. The first reported meta-analysis of a binary or continuous outcome in each SR (the 'index meta-analysis') will be evaluated. We will extract from study reports all study effect estimates that were eligible for inclusion in the index meta-analyses (e.g. from multiple instruments and time points) and will quantify and test for evidence of selective inclusion of results. We will also assess the risk of bias due to missing results in the index meta-analyses using a new tool (ROB-ME). Ethics and dissemination: Ethics approval is not required because information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. We will make all data collected from this study publicly available via the Open Science Framework.

Investigation of Risk Of Bias due to Unreported and SelecTively included results in meta-analyses of nutrition research: the ROBUST study protocol

Background: Dietary guidelines should be informed by systematic reviews (SRs) of the available scientific evidence. However, if the SRs that underpin dietary guidelines are flawed in their design, conduct or reporting, the recommendations contained therein may be misleading or harmful. To date there has been little empirical investigation of bias due to selective inclusion of results, and bias due to missing results, in SRs of food/diet-outcome relationships. Objectives: To explore in SRs with meta-analyses of the association between food/diet and health-related outcomes: (i) whether systematic reviewers selectively included study effect estimates in meta-analyses when multiple effect estimates were available; (ii) what impact selective inclusion of study effect estimates may have on meta-analytic effects, and; (iii) the risk of bias due to missing results (publication bias and selective non-reporting bias) in meta-analyses. Methods: We will systematically search for SRs with meta-analysis of the association between food/diet and health-related outcomes in a generally healthy population, published between January 2018 and June 2019. We will randomly sort titles and abstracts and screen them until we identify 50 eligible SRs. The first reported meta-analysis of a binary or continuous outcome in each SR (the 'index meta-analysis') will be evaluated. We will extract from study reports all study effect estimates that were eligible for inclusion in the index meta-analyses (e.g. from multiple instruments and time points) and will quantify and test for evidence of selective inclusion of results. We will also assess the risk of bias due to missing results in the index meta-analyses using a new tool (ROB-ME). Ethics and dissemination: Ethics approval is not required because information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. We will make all data collected from this study publicly available via the Open Science Framework.

How does exercise treatment compare with antihypertensive medications? A network meta-analysis of 391 randomised controlled trials assessing exercise and medication effects on systolic blood pressure

OBJECTIVE

To compare the effect of exercise regimens and medications on systolic blood pressure (SBP).

DATA SOURCES

Medline (via PubMed) and the Cochrane Library.

ELIGIBILITY CRITERIA

Randomised controlled trials (RCTs) of angiotensin-converting enzyme inhibitors (ACE-I), angiotensin-2 receptor blockers (ARBs), β-blockers, calcium channel blockers (CCBs) and diuretics were identified from existing Cochrane reviews. A previously published meta-analysis of exercise interventions was updated to identify recent RCTs that tested the SBP-lowering effects of endurance, dynamic resistance, isometric resistance, and combined endurance and resistance exercise interventions (up to September 2018).

DESIGN

Random-effects network meta-analysis.

OUTCOME

Difference in mean change from baseline SBP between comparator treatments (change from baseline in one group minus that in the other group) and its 95% credible interval (95% CrI), measured in mmHg.

RESULTS

We included a total of 391 RCTs, 197 of which evaluated exercise interventions (10 461 participants) and 194 evaluated antihypertensive medications (29 281 participants). No RCTs compared directly exercise against medications. While all medication trials included hypertensive populations, only 56 exercise trials included hypertensive participants (≥140 mmHg), corresponding to 3508 individuals. In a 10% random sample, risk of bias was higher in exercise RCTs, primarily due to lack of blinding and incomplete outcome data. In analyses that combined all populations, antihypertensive medications achieved higher reductions in baseline SBP compared with exercise interventions (mean difference -3.96 mmHg, 95% CrI -5.02 to -2.91). Compared with control, all types of exercise (including combination of endurance and resistance) and all classes of antihypertensive medications were effective in lowering baseline SBP. Among hypertensive populations, there were no detectable differences in the SBP-lowering effects of ACE-I, ARB, β-blocker and diuretic medications when compared with endurance or dynamic resistance exercise. There was no detectable inconsistency between direct and indirect comparisons. Although there was evidence of small-study effects, this affected both medication and exercise trials.

CONCLUSIONS

The effect of exercise interventions on SBP remains under-studied, especially among hypertensive populations. Our findings confirm modest but consistent reductions in SBP in many studied exercise interventions across all populations but individuals receiving medications generally achieved greater reductions than those following structured exercise regimens. Assuming equally reliable estimates, the SBP-lowering effect of exercise among hypertensive populations appears similar to that of commonly used antihypertensive medications. Generalisability of these findings to real-world clinical settings should be further evaluated.

Is positive publication bias really a bias, or an intentionally created discrimination toward negative results?

Today in publish or perish era, where manuscripts and research with successfully proven hypothesis or positive results are given more importance by journals, editors, funders, and institutions. The publication of researches with negative or null results is on the verge of extinction, thus creating an intentional bias known as publication bias. This review aims to discuss the consequence of the undermined importance of negative results and problems associated with it and will elaborate the importance of reporting negative results. Under-reporting of negative results not only wastes other researchers time, money, and manpower on which their researchers will be based but also introduces bias in meta-analysis leading to distortion of the scientific literature and misleads researchers, doctors, and policymakers in their decision-making. Many such important studies with negative results remain unpublished and therefore unavailable to the scientific community for understanding their values. A large number of human studies with huge risk to life's are carried out with the assurance that the proposed study will be performed with the aim to benefit, and results will be dissipated to everyone concerned, non-publication of such studies with negative results will not only be morally wrong but will also have ethical obligations to deal with. Therefore, all journals and their editor along with researchers and stakeholders need to be generous in giving importance to disseminating negative and positive findings alike.