Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

New horizons for comparative studies and meta-analyses

Comparative analyses and meta-analyses are key tools to elucidate broad biological principles, yet the two approaches often appear different in purpose. We propose an integrated approach that can generate deeper insights into ecoevolutionary processes. Marrying comparative and meta-analytic approaches will allow for (i) a more accurate investigation of drivers of biological variation, (ii) a greater ability to account for sources of non-independence in experimental data, (iii) more effective control of publication bias, and (iv) improved transparency and reproducibility. Stronger integration of meta-analytic and comparative studies can also broaden the scope from species-centric investigations to community-level responses and function-valued traits (e.g., reaction norms). We illuminate commonalities, differences, and the transformative potential of combining these methodologies for advancing ecology and evolutionary biology.

Estimating the extent of selective reporting: An application to economics

Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as non-significant. We estimate the extent of selectively reported p-values to range between 57.7% and 71.9% of the significant p-values. The counterfactual p-value distribution also allows us to assess shifts of p-values along the entire distribution of published p-values, revealing that particularly very small p-values (p < 0.001) are unexpectedly abundant in the published literature. Subsample analysis suggests that the extent of selective reporting is reduced in research fields that use experimental designs, analyze microeconomics research questions, and have at least some adequately powered studies.

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Phenotypic Diversity of a Leafroller Archips podana (Lepidoptera, Tortricidae) Does Not Change along an Industrial Pollution Gradient

Morphological polymorphism offers rich opportunities for studying the eco-evolutionary mechanisms that drive the adaptations of local populations to heterogeneous and changing environments. In this study, I explore the association between pollution load, abundance of large fruit-tree tortrix Archips podana and its within-species diversity (expressed in the presence of apical and/or lateral prongs on the phallus in male genitalia) across 26 study sites located 0.5 to 31 km from the industrial city of Lipetsk in central Russia. The Shannon diversity index, calculated from the frequencies of four morphs, correlated neither with the distance to the nearest industrial polluter (a proxy of pollution load) nor with the number of moths captured by pheromone traps (a measure of population abundance). The statistical power of the correlation analysis was sufficient (67%) to detect a medium effect (i.e., Pearson correlation coefficient with an absolute value of 0.40), if it existed. I conclude that the four phenotypes of A. podana do not differ in tolerance to industrial pollution and similarly respond to pollution-induced environmental disturbance. This is the first study of industrial pollution impacts on within-species diversity of insects expressed in the discrete traits of their male genitalia.

Effects of parental mental health and family environment on impulsivity in preadolescents: a longitudinal ABCD study®

Introduction

Impulsivity is a known risk factor for the development of substance use disorders and other psychiatric conditions that is influenced by both genetics and environment. Although research has linked parental mental health to children's impulsivity, potential mediators of this relationship remain understudied. The current investigation leverages the large national Adolescent Brain Cognitive Development (ABCD) Study to assess the mediating role of family conflict - an important social context for youth development - in the relationship between parental mental health and youth impulsivity.

Methods

Data were from the first three annual waves of the ABCD study (Baseline N = 11,876 children, Mage = 9.9 years; 48% female; 52% White). Parental mental health conditions were self-reported internalizing, externalizing, and total problems. Youth completed the family conflict scale, and Urgency, Planning (lack of), Perseverance (lack of), Sensation Seeking, and Positive Urgency (UPPS-P) scale to measure impulsivity. To determine if within-family change in conflict from baseline to year 1 explained changes in the strength of relations between baseline parental mental health and year 2 youth impulsivity, longitudinal causal mediation analyses were conducted, controlling for demographic factors (i.e., age, sex, race, household income, parental education, marital status), as well as baseline levels of family conflict and outcomes. Separate mediation models were run for each mental health condition and each UPPS-P subscale.

Results

Above and beyond bivariate relations, longitudinal mediation models, which included covariates, showed family conflict significantly (ps < 0.001) mediated relations between all three parental mental health conditions and all but one (i.e., sensation seeking) UPPS-P subscales. The proportion mediated through family conflict for internalizing problems and total problems on facets of impulsivity (except sensation seeking) ranged from 9% (for lack of perseverance) to 17% (for lack of planning). Proportion mediated via family conflict for externalizing problems on youth's impulsivity (except sensation seeking) was slightly higher, ranging between 13% (lack of perseverance) to 21% (lack of planning).

Discussion

Family conflict may be an important intergenerational factor linking parental mental health and youth's impulsivity. Addressing parental mental health and family conflict may help curb increased impulsivity in youth, and in turn reduce adolescent substance use disorders.

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.