Questionable research practices in ecology and evolution

Highs and Lows of Arboreal Life: Space use and Movement Strategies of Lion-Tailed Macaques (Macaca silenus) in the Western Ghats of India

Lion-tailed macaques (Macaca silenus), endemic to the Western Ghats of India, are increasingly threatened by habitat loss and degradation, and are vulnerable to environmental change. Insights into their spatial ecology can reveal the strategies that enable these macaques to navigate and use spatially complex heterogeneous spaces. This is crucial for conservation, given the increasing human disturbance in and around the Western Ghats. This study, conducted in Silent Valley National Park between January and June of 2019, 2022, and 2023, focused on two macaque troops occupying habitats with varying human presence-in the park's core (Sairandhri) and in the buffer (Keeripara). The observed group sizes of the Sairandhri and Keeripara troops were 51 and 21, respectively. We examined differences in the spatial ecology of these troops by assessing their vertical space use, two-dimensional (2D) and three-dimensional (3D) home ranges, and using an integrated step-selection function to understand movement-based habitat selection. We observed similarities in movement patterns across the two troops, with notable differences mainly in their vertical use of the forest. The Keeripara troop, resident in a more disturbed area, utilized a smaller home range (1.80 km²). They spent 94.2% of their time in mid-level canopy (6-20 m), descending rarely (1.1%) into lower strata (≤ 5 m), likely due to perceived high risk from predators and humans, and proximity to roads. The Sairandhri troop, resident in the less disturbed area, had a larger home range (3.05 km²) and exhibited greater terrestriality (11.3%), with 84.2% observed time in mid-canopies. Both troops preferred areas with canopy cover and fruiting trees, displaying similar slow movements (indicated through small step lengths and large turning angles) near these resources. Our findings suggest that lion-tailed macaques may be less strictly arboreal than previously believed, displaying more terrestriality in less disturbed habitats. Strategic use of 3D space based on local ecological and anthropogenic conditions highlights the need for conservation efforts that maintain canopy connectivity to support the ecological flexibility of this vulnerable species.

Testing the reproducibility of ecological studies on insect behavior in a multi-laboratory setting identifies opportunities for improving experimental rigor

The reproducibility of studies involving insect species is an underexplored area in the broader discussion about poor reproducibility in science. Our study addresses this gap by conducting a systematic multi-laboratory investigation into the reproducibility of ecological studies on insect behavior. We implemented a 3 × 3 experimental design, incorporating three study sites, and three independent experiments on three insect species from different orders: the turnip sawfly (Athalia rosae, Hymenoptera), the meadow grasshopper (Pseudochorthippus parallelus, Orthoptera), and the red flour beetle (Tribolium castaneum, Coleoptera). Using random-effect meta-analysis, we compared the consistency and accuracy of treatment effects on insect behavioral traits across replicate experiments. We successfully reproduced the overall statistical treatment effect in 83% of the replicate experiments, but overall effect size replication was achieved in only 66% of the replicates. Thus, though demonstrating sufficient reproducibility in some measures, this study also provides the first experimental evidence for cases of poor reproducibility in insect experiments. Our findings further show that reasons causing poor reproducibility established in rodent research also hold for other study organisms and research questions. We believe that a rethinking of current best practices is required to face reproducibility issues in insect studies but also across disciplines. Specifically, we advocate for adopting open research practices and the implementation of methodological strategies that reduce bias and problems arising from over-standardization. With respect to the latter, the introduction of systematic variation through multi-laboratory or heterogenized designs may contribute to improved reproducibility in studies involving any living organisms.

The Pragmatic American Revisited: A Direct Replication of Pickett and Baker (2014)

In 2014, Pickett and Baker cast doubt on the scholarly consensus that Americans are pragmatic about criminal justice. Previous research suggested this pragmaticism was evidenced by either null or positive relationships between seemingly opposite items (i.e., between dispositional and situational crime attributions and between punitiveness and rehabilitative policy support). Pickett and Baker argued that because these studies worded survey items in the same positive direction, respondents' susceptibility to acquiescence bias led to artificially inflated positive correlations. Using a simple split-ballot experiment, they manipulated the direction of survey items and demonstrated bidirectional survey items resulted in negative relationships between attributions and between support for punitive and rehabilitative policies. We replicated Pickett and Baker's methodology with a nationally representative sample of American respondents supplemented by a diverse student sample. Our results were generally consistent, and, in many cases, effect sizes were stronger than those observed in the original study. Americans appear much less pragmatic when survey items are bidirectional. Yet, we suggest the use of bidirectional over unidirectional survey items trades one set of problems for another. Instead, to reduce acquiescence bias and improve overall data quality, we encourage researchers to adopt item-specific questioning.

Differential effects of multiplex and uniplex affiliative relationships on biomarkers of inflammation

Social relationships profoundly impact health in social species. Much of what we know regarding the impact of affiliative social relationships on health in nonhuman primates (NHPs) has focused on the structure of connections or the quality of relationships. These relationships are often quantified by comparing different types of affiliative behaviors (e.g., contact sitting, grooming, proximity) or pooling affiliative behaviors into an overall measure of affiliation. However, it is unclear how the breadth of affiliative behaviors (e.g., how many different types or which ones) a dyad engages in impact health and fitness outcomes. We used a novel social network approach to quantify the breadth of affiliative relationships based on two behaviors: grooming and sitting in contact. Dyadic relationships were filtered into separate networks depending on whether the pair engaged in multiple affiliative behaviors (multiplex networks) or just one (uniplex networks). Typically, in social network analysis, the edges in the network represent the presence of a single behavior (e.g., grooming) regardless of the presence or absence of other behaviors (e.g., contact sitting, proximity). Therefore, to validate this method, we first compared the overall structure of the standard network for each affiliative behavior: all grooming interactions regardless of contact sitting, and all contact sitting interactions regardless of grooming. We then similarly compared the structure of our filtered multiplex vs. uniplex networks. Results indicated that multiplex networks were more modular, reciprocal, and kin-based while connections in uniplex networks were more strongly associated with social status. These differences were not replicated when comparing networks based on a single behavior alone (i.e., all grooming networks vs. all contact sitting networks). Next, we evaluated whether individual network position in multiplex vs. uniplex (novel approach) or grooming vs. contact sitting (traditional approach) networks differentially impact inflammatory biomarkers in a commonly studied non-human primate model system, the rhesus macaque (Macaca mulatta). Being well connected in multiplex networks (networks where individuals both contact sat and groomed) was associated with lower inflammation (IL-6, TNF-alpha). In contrast, being well connected in uniplex grooming networks (dyad engaged only in grooming and not in contact sitting) was associated with greater inflammation. Altogether, these results suggest that multiplex relationships may function as supportive relationships (e.g., those between kin or strong bonds) that promote health. In contrast, the function of uniplex grooming relationships may be more transactional (e.g., based on social tolerance or social status) and may incur physiological costs. This complexity is important to consider for understanding the mechanisms underlying the association of social relationships on human and animal health.

Environmental stress differentially affects phenotypic modularity and fluctuating asymmetry in generalist and specialist cactophilic Drosophila

Modularity and developmental (in)stability have the potential to influence phenotype production and, consequently, the evolutionary trajectories of species. Depending on the environmental factors involved and the buffering capacity of an organism, different developmental outcomes are expected. Cactophilic Drosophila species provide an established eco-evolutionary model with well-studied ecological conditions, making them ideal for studying these phenomena. Here, we investigated how variations in larval diet and exposure to alkaloids on primary and secondary host plants affect the degree of integration/modularity and fluctuating asymmetry (FA, a proxy for developmental instability) of wing shape in two sibling species with different degrees of specialisation: Drosophila buzzatii (generalist) and Drosophila koepferae (specialist). Additionally, we compared the anterior-posterior modular configuration with a recently proposed proximal-distal modular configuration. Our results revealed greater independence among proximal-distal modules compared to anterior-posterior modules. Moreover, we observed sex-specific responses, with males exhibiting greater susceptibility to stressful environments than females. Each species showed a particular trait pattern across treatments: D. buzzatii showed increased integration and FA when reared in a nutrient-poor, alkaloid-rich secondary host, while D. koepferae displayed similar responses in novel environments characterised by double doses of alkaloids on the secondary host plant. These findings align with the generalist-specialist paradigm, suggesting that specialists may be challenged by novel environments, whereas generalists may be more affected by stressful conditions. Our study highlights the importance of considering each part of the proximal-distal wing axis independently, and the need to consider ecological-evolutionary history when investigating the relationship between complex phenotypic traits and environmental stress.

Same data, different analysts: variation in effect sizes due to analytical decisions in ecology and evolutionary biology

Although variation in effect sizes and predicted values among studies of similar phenomena is inevitable, such variation far exceeds what might be produced by sampling error alone. One possible explanation for variation among results is differences among researchers in the decisions they make regarding statistical analyses. A growing array of studies has explored this analytical variability in different fields and has found substantial variability among results despite analysts having the same data and research question. Many of these studies have been in the social sciences, but one small "many analyst" study found similar variability in ecology. We expanded the scope of this prior work by implementing a large-scale empirical exploration of the variation in effect sizes and model predictions generated by the analytical decisions of different researchers in ecology and evolutionary biology. We used two unpublished datasets, one from evolutionary ecology (blue tit, Cyanistes caeruleus, to compare sibling number and nestling growth) and one from conservation ecology (Eucalyptus, to compare grass cover and tree seedling recruitment). The project leaders recruited 174 analyst teams, comprising 246 analysts, to investigate the answers to prespecified research questions. Analyses conducted by these teams yielded 141 usable effects (compatible with our meta-analyses and with all necessary information provided) for the blue tit dataset, and 85 usable effects for the Eucalyptus dataset. We found substantial heterogeneity among results for both datasets, although the patterns of variation differed between them. For the blue tit analyses, the average effect was convincingly negative, with less growth for nestlings living with more siblings, but there was near continuous variation in effect size from large negative effects to effects near zero, and even effects crossing the traditional threshold of statistical significance in the opposite direction. In contrast, the average relationship between grass cover and Eucalyptus seedling number was only slightly negative and not convincingly different from zero, and most effects ranged from weakly negative to weakly positive, with about a third of effects crossing the traditional threshold of significance in one direction or the other. However, there were also several striking outliers in the Eucalyptus dataset, with effects far from zero. For both datasets, we found substantial variation in the variable selection and random effects structures among analyses, as well as in the ratings of the analytical methods by peer reviewers, but we found no strong relationship between any of these and deviation from the meta-analytic mean. In other words, analyses with results that were far from the mean were no more or less likely to have dissimilar variable sets, use random effects in their models, or receive poor peer reviews than those analyses that found results that were close to the mean. The existence of substantial variability among analysis outcomes raises important questions about how ecologists and evolutionary biologists should interpret published results, and how they should conduct analyses in the future.

Poor hypotheses and research waste in biology: learning from a theory crisis in psychology

While psychologists have extensively discussed the notion of a "theory crisis" arising from vague and incorrect hypotheses, there has been no debate about such a crisis in biology. However, biologists have long discussed communication failures between theoreticians and empiricists. We argue such failure is one aspect of a theory crisis because misapplied and misunderstood theories lead to poor hypotheses and research waste. We review its solutions and compare them with methodology-focused solutions proposed for replication crises. We conclude by discussing how promoting inclusion, diversity, equity, and accessibility (IDEA) in theoretical biology could contribute to ameliorating breakdowns in the theory-empirical cycle.

Substantial heterogeneity in trauma triage tool characteristic operationalization for identification of major trauma: a hybrid systematic review

PURPOSE

Trauma Triage Tools (TTTs) support pre-hospital staff to identify major trauma patients based on prehospital characteristics and bring them to appropriate trauma centres. However, while triaging trauma has been examined extensively, there appears to be little consensus on how variables within TTTs are applied. We therefore aimed to examine the prehospital characteristics and their operationalization applied in the international literature in TTTs.

METHODS

We applied a hybrid systematic review approach. Searches were conducted in multiple databases. We initially searched for systematic reviews that analyse prehospital characteristics applied in TTTs, then supplemented this with an updated search of original TTT papers from November 2019.

RESULTS

We identified 92 papers which identified 52 adult general population TTTs. Results indicate considerable heterogeneity in prehospital characteristics included in TTTs internationally. There was similarity in the higher-level categories included in the tools: tools often included measurements of a patient's physiological characteristics, injury characteristics, mechanism of injury and any modifiers for high-risk groups. However, the prehospital characteristics that made up those groups, how they were applied and interpreted were found to vary considerably.

CONCLUSION

While there is agreement in the higher-level categories used in TTTs, the thresholds adopted in specific variables vary widely, which may reflect statistical rather than clinical considerations. This may contribute to considerable variation in standards of major trauma triaging internationally. An agreed taxonomy of operationalization of prehospital characteristics used in TTTs is required to prevent sub-optimal clinical decision-making in major trauma triaging.

REGISTRATION

PROSPERO CRD42023393094.

A novel framework for increasing research transparency: Exploring the connection between diversity and innovation

A split sample/dual method research protocol is demonstrated to increase transparency while reducing the probability of false discovery. We apply the protocol to examine whether diversity in ownership teams increases or decreases the likelihood of a firm reporting a novel innovation using data from the 2018 United States Census Bureau's Annual Business Survey. Transparency is increased in three ways: 1) all specification testing and identifying potentially productive models is done in an exploratory subsample that 2) preserves the validity of hypothesis test statistics from de novo estimation in the holdout confirmatory sample with 3) all findings publicly documented in an earlier registered report and in this journal publication. Bayesian estimation procedures that leverage information from the exploratory stage included in the confirmatory stage estimation replace traditional frequentist null hypothesis significance testing. In addition to increasing statistical power by using information from the full sample, Bayesian methods directly estimate a probability distribution for the magnitude of an effect, allowing much richer inference. Estimated magnitudes of diversity along academic discipline, race, ethnicity, and foreign-born status dimensions are positively associated with innovation. A maximally diverse ownership team on these dimensions would be roughly six times more likely to report new-to-market innovation than a homophilic team.

A Systematic Review on the Evolution of Power Analysis Practices in Psychological Research

Performing hypothesis tests with adequate statistical power is indispensable for psychological research. In response to several large-scale replication projects following the replication crisis, concerns about the root causes of this crisis - such as questionable research practices (QRPs) - have grown. While initial efforts primarily addressed the inflation of the type I error rate of research due to QRPs, recent attention has shifted to the adverse consequences of low statistical power. In this paper we first argue how underpowered studies, in combination with publication bias, contribute to a literature rife with false positive results and overestimated effect sizes. We then examine whether the prevalence of power analyses in psychological research has effectively increased over time in response to the increased awareness regarding these phenomena. To address this, we conducted a systematic review of 903 published empirical articles across four APA-disciplines, comparing 453 papers published in 2015-2016, with 450 papers from 2020-2021. Although the prevalence of power analysis across different domains in psychology has increased over time (from 9.5% to 30%), it remains insufficient overall. We conclude by discussing the implications of these findings and elaborating on some alternative methods to a priori power analysis that can help ensure sufficient statistical power.

"1-800-Help-Me-With-Open-Science-Stuff": A Qualitative Examination of Open Science Practices in Communication Sciences and Disorders

PURPOSE

The purpose of this qualitative study was to examine the perceptions of communication sciences and disorders (CSD) assistant professors in the United States related to barriers and facilitators to engaging in open science practices and identify opportunities for improving open science training and support in the field.

METHOD

Thirty-five assistant professors (16 from very high research activity [R1] institutions, 19 from institutions with other Carnegie classifications) participated in one 1-hr virtual focus group conducted via Zoom recording technology. The researchers used a conventional content analysis approach to analyze the focus group data and develop categories from the discussions.

RESULTS

Five categories were developed from the focus groups: (a) a desire to learn about open science through opportunities for independent learning and learning with peers; (b) perceived benefits of engaging in open science on assistant professors' careers, the broader scientific community, and the quality of research in the field of CSD; (c) personal factors that act as barriers and/or facilitators to engaging in open science practices; (d) systemic factors that act as barriers and/or facilitators to engaging in open science practices; and (e) differences in perceptions of R1 and non-R1 assistant professors.

CONCLUSIONS

Assistant professors in CSD perceive benefits of open science for their careers, the scientific community, and the field. However, they face many barriers (e.g., time, lack of knowledge and training), which impede their engagement in open science practices. Preliminary recommendations for CSD assistant professors, academic institutions, publishers, and funding agencies are provided to reduce barriers to engagement in open science practices.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.27996839.

Reporting guidelines for terrestrial respirometry: Building openness, transparency of metabolic rate and evaporative water loss data

Respirometry is an important tool for understanding whole-animal energy and water balance in relation to the environment. Consequently, the growing number of studies using respirometry over the last decade warrants reliable reporting and data sharing for effective dissemination and research synthesis. We provide a checklist guideline on five key sections to facilitate the transparency, reproducibility, and replicability of respirometry studies: 1) materials, set up, plumbing, 2) subject conditions/maintenance, 3) measurement conditions, 4) data processing, and 5) data reporting and statistics, each with explanations and example studies. Transparency in reporting and data availability has benefits on multiple fronts. Authors can use this checklist to design and report on their study, and reviewers and editors can use the checklist to assess the reporting quality of the manuscripts they review. Improved standards for reporting will enhance the value of primary studies and will greatly facilitate the ability to carry out higher quality research syntheses to address ecological and evolutionary theories.

Advances and challenges in ecological connectivity science

Maintaining and restoring ecological connectivity will be key in helping to prevent and reverse the loss of biodiversity. Fortunately, a growing body of research conducted over the last few decades has advanced our understanding of connectivity science, which will help inform evidence-based connectivity conservation actions. Increases in data availability and computing capacity have helped to dramatically increase our ability to model functional connectivity using more sophisticated models. Keeping track of these advances can be difficult, even for connectivity scientists and practitioners. In this article, we highlight some key advances from the past decade and outline many of the remaining challenges. We describe the efforts to increase the biological realism of connectivity models by, for example, isolating movement behaviors, population parameters, directional movements, and the effects of climate change. We also discuss considerations of when to model connectivity for focal or multiple species. Finally, we reflect on how to account for uncertainty and increase the transparency and reproducibility of connectivity research and discuss situations where decisions may require forgoing sophistication for more simple approaches.

Is something rotten in the state of Denmark? Cross-national evidence for widespread involvement but not systematic use of questionable research practices across all fields of research

Questionable research practices (QRP) are believed to be widespread, but empirical assessments are generally restricted to a few types of practices. Furthermore, conceptual confusion is rife with use and prevalence of QRPs often being confused as the same quantity. We present the hitherto most comprehensive study examining QRPs across scholarly fields and knowledge production modes. We survey perception, use, prevalence and predictors of QRPs among 3,402 researchers in Denmark and 1,307 in the UK, USA, Croatia and Austria. Results reveal remarkably similar response patterns among Danish and international respondents (τ = 0.85). Self-reported use indicates whether respondents have used a QRP in recent publications. 9 out of 10 respondents admitted using at least one QRP. Median use is three out of nine QRP items. Self-reported prevalence reflects the frequency of use. On average, prevalence rates were roughly three times lower compared to self-reported use. Findings indicated that the perceived social acceptability of QRPs influenced self-report patterns. Results suggest that most researchers use different types of QRPs within a restricted time period. The prevalence estimates, however, do not suggest outright systematic use of specific QRPs. Perceived pressure was the strongest systemic predictor for prevalence. Conversely, more local attention to research cultures and academic age was negatively related to prevalence. Finally, the personality traits conscientiousness and, to a lesser degree, agreeableness were also inversely associated with self-reported prevalence. Findings suggest that explanations for engagement with QRPs are not only attributable to systemic factors, as hitherto suggested, but a complicated mixture of experience, systemic and individual factors, and motivated reasoning.

Supporting study registration to reduce research waste

An estimated 82-89% of ecological research and 85% of medical research has limited or no value to the end user because of various inefficiencies. We argue that registration and registered reports can enhance the quality and impact of ecological research. Drawing on evidence from other fields, chiefly medicine, we support our claim that registration can reduce research waste. However, increasing registration rates, quality and impact will be very slow without coordinated effort of funders, publishers and research institutions. We therefore call on them to facilitate the adoption of registration by providing adequate support. We outline several aspects to be considered when designing a registration system that would best serve the field of ecology. To further inform the development of such a system, we call for more research to identify the causes of low registration rates in ecology. We suggest short- and long-term actions to bolster registration and reduce research waste.

Finding the right power balance: Better study design and collaboration can reduce dependence on statistical power

Power analysis currently dominates sample size determination for experiments, particularly in grant and ethics applications. Yet, this focus could paradoxically result in suboptimal study design because publication biases towards studies with the largest effects can lead to the overestimation of effect sizes. In this Essay, we propose a paradigm shift towards better study designs that focus less on statistical power. We also advocate for (pre)registration and obligatory reporting of all results (regardless of statistical significance), better facilitation of team science and multi-institutional collaboration that incorporates heterogenization, and the use of prospective and living meta-analyses to generate generalizable results. Such changes could make science more effective and, potentially, more equitable, helping to cultivate better collaborations.

Is N-Hacking Ever OK? The consequences of collecting more data in pursuit of statistical significance

Upon completion of an experiment, if a trend is observed that is "not quite significant," it can be tempting to collect more data in an effort to achieve statistical significance. Such sample augmentation or "N-hacking" is condemned because it can lead to an excess of false positives, which can reduce the reproducibility of results. However, the scenarios used to prove this rule tend to be unrealistic, assuming the addition of unlimited extra samples to achieve statistical significance, or doing so when results are not even close to significant; an unlikely situation for most experiments involving patient samples, cultured cells, or live animals. If we were to examine some more realistic scenarios, could there be any situations where N-hacking might be an acceptable practice? This Essay aims to address this question, using simulations to demonstrate how N-hacking causes false positives and to investigate whether this increase is still relevant when using parameters based on real-life experimental settings.

Evidence of questionable research practices in clinical prediction models

BACKGROUND

Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.

METHODS

We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.

RESULTS

The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.

CONCLUSIONS

The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.

Plant volatiles and priority effects interactively determined initial community assembly of arthropods on multiple willow species

Plant traits, which are often species specific, can serve as environmental filtering for community assembly on plants. At the same time, the species identity of the initially colonizing arthropods would vary between plant individuals, which would subsequently influence colonizing arthropods and community development in the later stages. However, it remains unclear whether interindividual divergence due to priority effects is equally important as plant trait-specific environmental filtering in the initial stages. In this study, we propose that plant volatile organic compounds (PVOCs) may play a crucial role as an environmental filter in the initial stages of community assembly, which can prevent the community assembly process from being purely stochastic. To test this hypothesis, we conducted short term but highly frequent monitoring (19 observations over 9 days) of arthropod community assembly on intact individuals of six willow species in a common garden. PVOC compositions were analyzed before starting the experiment and compared with arthropod compositions occurring on Days 1-2 of the experiment (earliest colonizer community) and those occurring on Days 8-9 of the experiment (subsequent colonizer community). Unintentionally, deer herbivory also occurred at night of Day 2. Distance-based statistics demonstrated that PVOC compositions were plant species specific, but neither the earliest colonizer nor the subsequent colonizer community composition could be explained by plant species identity. Rather, Procrustes analysis showed that both the PVOC composition and that of the earliest colonizer community can be used to explain the subsequent colonizer community. In addition, the linkage between PVOCs and the subsequent colonizer community was stronger on individuals with deer herbivory. These findings indicate that PVOCs have widespread effects on initial community assembly, as well as priority effects brought on by stochastic immigration, and that plant species identity only has weak and indirect effects on the actual composition of the community.

Open Science Practices in Communication Sciences and Disorders: A Survey

PURPOSE

Open science is a collection of practices that seek to improve the accessibility, transparency, and replicability of science. Although these practices have garnered interest in related fields, it remains unclear whether open science practices have been adopted in the field of communication sciences and disorders (CSD). This study aimed to survey the knowledge, implementation, and perceived benefits and barriers of open science practices in CSD.

METHOD

An online survey was disseminated to researchers in the United States actively engaged in CSD research. Four-core open science practices were examined: preregistration, self-archiving, gold open access, and open data. Data were analyzed using descriptive statistics and regression models.

RESULTS

Two hundred twenty-two participants met the inclusion criteria. Most participants were doctoral students (38%) or assistant professors (24%) at R1 institutions (58%). Participants reported low knowledge of preregistration and gold open access. There was, however, a high level of desire to learn more for all practices. Implementation of open science practices was also low, most notably for preregistration, gold open access, and open data (< 25%). Predictors of knowledge and participation, as well as perceived barriers to implementation, are discussed.

CONCLUSION

Although participation in open science appears low in the field of CSD, participants expressed a strong desire to learn more in order to engage in these practices in the future. Supplemental Material and Open Science Form: https://doi.org/10.23641/asha.21569040.

Next steps after airing disagreement on a scientific issue with policy implications: a meta-analysis, multi-lab replication and adversarial collaboration

Canadian policymakers are interested in determining whether farmed Atlantic salmon, frequently infected with Piscine orthoreovirus (PRV), may threaten wild salmon populations in the Pacific Northwest. A relevant work has been published in BMC Biology by Polinksi and colleagues, but their conclusion that PRV has a negligible impact on the energy expenditure and respiratory performance of sockeye salmon is disputed by Mordecai and colleagues, whose re-analysis is presented in a correspondence article. So, what is the true effect and what should follow this unresolved dispute? We suggest a 'registered multi-lab replication with adversaries'.

Registered Reports: A new chapter at Ecology & Evolution

Ecology & Evolution has published its first Registered Report and offers the perspective of the editor, author, and student on the publication process.

Tempest in a teacup: An analysis of p-Hacking in organizational research

We extend questionable research practices (QRPs) research by conducting a robust, large-scale analysis of p-hacking in organizational research. We leverage a manually curated database of more than 1,000,000 correlation coefficients and sample sizes, with which we calculate exact p-values. We test for the prevalence and magnitude of p-hacking across the complete database as well as various subsets of the database according to common bivariate relation types in the organizational literature (e.g., attitudes-behaviors). Results from two analytical approaches (i.e., z-curve, critical bin comparisons) were consistent in both direction and significance in nine of 18 datasets. Critical bin comparisons indicated p-hacking in 12 of 18 subsets, three of which reached statistical significance. Z-curve analyses indicated p-hacking in 11 of 18 subsets, two of which reached statistical significance. Generally, results indicated that p-hacking is detectable but small in magnitude. We also tested for three predictors of p-hacking: Publication year, journal prestige, and authorship team size. Across two analytic approaches, we observed a relatively consistent positive relation between p-hacking and journal prestige, and no relationship between p-hacking and authorship team size. Results were mixed regarding the temporal trends (i.e., evidence for p-hacking over time). In sum, the present study of p-hacking in organizational research indicates that the prevalence of p-hacking is smaller and less concerning than earlier research has suggested.

Are weak dispersers more vulnerable than strong dispersers to land use intensification?

Ecologists often state that weak dispersers are particularly at risk from land use intensification, and that they therefore should be prioritized for conservation. We reviewed the empirical evidence, to evaluate whether this idea should be used as a general rule in conservation. While 89% of authors predicted that weak dispersers are more vulnerable to land use intensification (80 out of 90 papers), only 56% of reported tests (235 out of 422) were consistent with this prediction. Thirty per cent of tests (128 out of 422) were consistent with the opposite prediction, that strong dispersers are more vulnerable to intensification, and 60% of articles (45 out of 75) had at least one test where strong dispersers were most vulnerable. The likelihood of finding that weak dispersers are more vulnerable to intensification than strong dispersers varied with latitude, taxonomic group and type of land use intensification. Notably, the odds of finding that weak dispersers are more vulnerable to intensification than strong dispersers was higher if the study was nearer to the equator. Taken together, our results show that the prediction that weak dispersers are more vulnerable than strong dispersers to intensification is not sufficiently supported to justify using weak dispersal as a general indicator of species risk in human-modified landscapes.

Reducing bias, increasing transparency and calibrating confidence with preregistration

Flexibility in the design, analysis and interpretation of scientific studies creates a multiplicity of possible research outcomes. Scientists are granted considerable latitude to selectively use and report the hypotheses, variables and analyses that create the most positive, coherent and attractive story while suppressing those that are negative or inconvenient. This creates a risk of bias that can lead to scientists fooling themselves and fooling others. Preregistration involves declaring a research plan (for example, hypotheses, design and statistical analyses) in a public registry before the research outcomes are known. Preregistration (1) reduces the risk of bias by encouraging outcome-independent decision-making and (2) increases transparency, enabling others to assess the risk of bias and calibrate their confidence in research outcomes. In this Perspective, we briefly review the historical evolution of preregistration in medicine, psychology and other domains, clarify its pragmatic functions, discuss relevant meta-research, and provide recommendations for scientists and journal editors.

Special Issue Introduction: The GLES Open Science Challenge 2021: A Pilot Project on the Applicability of Registered Reports in Quantitative Political Science

The GLES Open Science Challenge 2021 was a pioneering initiative in quantitative political science. Aimed at increasing the adoption of replicable and transparent research practices, it led to this special issue. The project combined the rigor of registered reports-a new publication format in which studies are evaluated prior to data collection/access and analysis-with quantitative political science research in the context of the 2021 German federal election. This special issue, which features the registered reports that resulted from the project, shows that transparent research following open science principles benefits our discipline and substantially contributes to quantitative political science. In this introduction to the special issue, we first elaborate on why more transparent research practices are necessary to guarantee the cumulative progress of scientific knowledge. We then show how registered reports can contribute to increasing the transparency of scientific practices. Next, we discuss the application of open science practices in quantitative political science to date. And finally, we present the process and schedule of the GLES Open Science Challenge and give an overview of the contributions included in this special issue.

A Blockchain-Based Architecture for Trust in Collaborative Scientific Experimentation

In scientific collaboration, data sharing, the exchange of ideas and results are essential to knowledge construction and the development of science. Hence, we must guarantee interoperability, privacy, traceability (reinforcing transparency), and trust. Provenance has been widely recognized for providing a history of the steps taken in scientific experiments. Consequently, we must support traceability, assisting in scientific results' reproducibility. One of the technologies that can enhance trust in collaborative scientific experimentation is blockchain. This work proposes an architecture, named BlockFlow, based on blockchain, provenance, and cloud infrastructure to bring trust and traceability in the execution of collaborative scientific experiments. The proposed architecture is implemented on Hyperledger, and a scenario about the genomic sequencing of the SARS-CoV-2 coronavirus is used to evaluate the architecture, discussing the benefits of providing traceability and trust in collaborative scientific experimentation. Furthermore, the architecture addresses the heterogeneity of shared data, facilitating interpretation by geographically distributed researchers and analysis of such data. Through a blockchain-based architecture that provides support on provenance and blockchain, we can enhance data sharing, traceability, and trust in collaborative scientific experiments.

Incidence and prevalence of coma in the UK and the USA

The epidemiology of coma is unknown because case ascertainment with traditional methods is difficult. Here, we used crowdsourcing methodology to estimate the incidence and prevalence of coma in the UK and the USA. We recruited UK and US laypeople (aged ≥18 years) who were nationally representative (i.e. matched for age, gender and ethnicity according to census data) of the UK and the USA, respectively, utilizing a crowdsourcing platform. We provided a description of coma and asked survey participants if they-'right now' or 'within the last year'-had a family member in coma. These participants (UK n = 994, USA n = 977) provided data on 30 387 family members (UK n = 14 124, USA n = 16 263). We found more coma cases in the USA (n = 47) than in the UK (n = 20; P = 0.009). We identified one coma case in the UK (0.007%, 95% confidence interval 0.00-0.04%) on the day of the survey and 19 new coma cases (0.13%, 95% confidence interval 0.08-0.21%) within the preceding year, resulting in an annual incidence of 135/100 000 (95% confidence interval 81-210) and a point prevalence of 7 cases per 100 000 population (95% confidence interval 0.18-39.44) in the UK. We identified five cases in the USA (0.031%, 95% confidence interval 0.01-0.07%) on the day of the survey and 42 new cases (0.26%, 95% confidence interval 0.19-0.35%) within the preceding year, resulting in an annual incidence of 258/100 000 (95% confidence interval 186-349) and a point prevalence of 31 cases per 100 000 population (95% confidence interval 9.98-71.73) in the USA. The five most common causes were stroke, medically induced coma, COVID-19, traumatic brain injury and cardiac arrest. To summarize, for the first time, we report incidence and prevalence estimates for coma across diagnosis types and settings in the UK and the USA using crowdsourcing methods. Coma may be more prevalent in the USA than in the UK, which requires further investigation. These data are urgently needed to expand the public health perspective on coma and disorders of consciousness.

A How-To Guide for Open-Science Practices in Special Education Research

Open-science reforms, which aim to increase credibility and access of research, have the potential to benefit the research base in special education, as well as practice and policy informed by that research base. Awareness of open science is increasing among special education researchers. However, relatively few researchers in the field have experience using multiple open-science practices, and few practical guidelines or resources have been tailored to special education researchers to support their exploration and adoption of open science. In this paper, we described and provided guidelines and resources for applying five core open-science practices-preregistration, registered reports, data sharing, materials sharing, and open-access publishing-in special education research.

A mixed methodology for evaluating use of evidence in conservation planning

Conservation practitioners widely recognize the importance of making decisions based on the best available evidence. However, the effectiveness of evidence use in conservation planning is rarely assessed, which limits opportunities to improve evidence-based practice. We devised a mixed methodology for empirically evaluating use of evidence that applies social science tools to systematically appraise what kinds of evidence are used in conservation planning, to what effect, and under what limitations. We applied our approach in a case study of the Nature Conservancy of Canada (NCC), a leading land conservation organization. We conducted qualitative and quantitative analyses of 65 NCC planning documents (n = 13 in-depth) to identify patterns in evidence use, and surveyed 35 conservation planners to examine experiences of and barriers to using evidence. Although claims in plans contained a wide range of evidence types, 26% of claims were not referenced or associated with an identifiable source. Lack of evidence use was particularly apparent in claims associated with direct threats, particularly those identified as low (71% coded as insufficient or lacking evidence) or medium (45%) threats. Survey participants described relying heavily on practitioner experience and highlighted capacity limitations and disciplinary gaps in expertise among planning teams as barriers to using evidence effectively. We found that although time-intensive, this approach yielded actionable recommendations for improving evidence use in NCC conservation plans. Similar mixed-method assessments may streamline the process by including interviews and refining the document analysis frames to target issues or sections of concern. We suggest our method provides an accessible and robust point of departure for conservation practitioners to evaluate whether the use of conservation planning reflects in-house standards and more broadly recognized best practices.

Quantifying research waste in ecology

Research inefficiencies can generate huge waste: evidence from biomedical research has shown that most research is avoidably wasted and steps have been taken to tackle this costly problem. Although other scientific fields could also benefit from identifying and quantifying waste and acting to reduce it, no other estimates of research waste are available. Given that ecological issues interweave most of the United Nations Sustainable Development Goals, we argue that tackling research waste in ecology should be prioritized. Our study leads the way. We estimate components of waste in ecological research based on a literature review and a meta-analysis. Shockingly, our results suggest only 11-18% of conducted ecological research reaches its full informative value. All actors within the research system-including academic institutions, policymakers, funders and publishers-have a duty towards science, the environment, study organisms and the public, to urgently act and reduce this considerable yet preventable loss. We discuss potential ways forward and call for two major actions: (1) further research into waste in ecology (and beyond); (2) focused development and implementation of solutions to reduce unused potential of ecological research.

Reply to: "Research on agroforestry systems and biodiversity conservation: what can we conclude so far and what should we improve?" by Boinot et al. 2022

In our article ‘European agroforestry has no unequivocal effect on biodiversity: a time-cumulative meta-analysis’ (BMC Ecology and Evolution, 2021) we synthesize the effect of agroforestry on biodiversity. Boinot et al. (BMC Ecology and Evolution, 2022) criticise our approach arguing that our definitions of agroforestry and biodiversity are too narrow; that we use inappropriate control sites for primary studies lacking distance to the treatment sites; that there are too few studies for a meta-analysis in silvoarable systems; and that local practice should be emphasized. We agree on the importance of local practices and that more studies would improve the robustness of our meta-analytical results. However, our conclusions are robust to removing studies criticised for inappropriate controls. We also recognize the problem of different definitions of agroforestry and using species richness as sole proxy for biodiversity. We appreciate being given the opportunity to clarify our results and to trigger future discussions about definitions and the interpretation of results from meta-analysis.

Why and how we should join the shift from significance testing to estimation

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, p-values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, 'statistically significant' results have overestimated effect sizes, a bias declining with increasing statistical power. Third, 'statistically non-significant' results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre-specified alternative hypothesis, pre-study power calculation and the probability of 'false negatives' (beta error rate). The results sections of the papers presented 49 significance tests on average (median 23, range 0-390). Of 41 studies that contained verbal descriptions of a 'statistically non-significant' result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to 'test' specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility ('confidence') intervals covering these hypotheses.

Failures to disagree are essential for environmental science to effectively influence policy development

While environmental science, and ecology in particular, is working to provide better understanding to base sustainable decisions on, the way scientific understanding is developed can at times be detrimental to this cause. Locked-in debates are often unnecessarily polarised and can compromise any common goals of the opposing camps. The present paper is inspired by a resolved debate from an unrelated field of psychology where Nobel laureate David Kahneman and Garry Klein turned what seemed to be a locked-in debate into a constructive process for their fields. The present paper is also motivated by previous discourses regarding the role of thresholds in natural systems for management and governance, but its scope of analysis targets the scientific process within complex social-ecological systems in general. We identified four features of environmental science that appear to predispose for locked-in debates: (1) The strongly context-dependent behaviour of ecological systems. (2) The dominant role of single hypothesis testing. (3) The high prominence given to theory demonstration compared investigation. (4) The effect of urgent demands to inform and steer policy. This fertile ground is further cultivated by human psychological aspects as well as the structure of funding and publication systems.

The Academic, Societal and Animal Welfare Benefits of Open Science for Animal Science

Animal science researchers have the obligation to reduce, refine, and replace the usage of animals in research (3R principles). Adherence to these principles can be improved by transparently publishing research findings, data and protocols. Open Science (OS) can help to increase the transparency of many parts of the research process, and its implementation should thus be considered by animal science researchers as a valuable opportunity that can contribute to the adherence to these 3R-principles. With this article, we want to encourage animal science researchers to implement a diverse set of OS practices, such as Open Access publishing, preprinting, and the pre-registration of test protocols, in their workflows.

Perspective: Scientific rigor or ritual? Statistical significance in pest management science

P values, binary hypothesis tests, and statistical significance are too often overused or used incorrectly in pest management reports. These statistical results are not the point of an analysis, they are just a by-product that can sometimes be informative. The point of the analysis is always a biological interpretation of the data focusing on whether something new has been learned and then quantifying what was learned in terms of biological effects or supported hypotheses. The term 'statistical significance' should be retired if only because of the common misunderstanding that it implies biological relevance; these terms are not equivalent. © 2021 Society of Chemical Industry.

Heterogeneity estimates in a biased world

Meta-analyses typically quantify heterogeneity of results, thus providing information about the consistency of the investigated effect across studies. Numerous heterogeneity estimators have been devised. Past evaluations of their performance typically presumed lack of bias in the set of studies being meta-analysed, which is often unrealistic. The present study used computer simulations to evaluate five heterogeneity estimators under a range of research conditions broadly representative of meta-analyses in psychology, with the aim to assess the impact of biases in sets of primary studies on estimates of both mean effect size and heterogeneity in meta-analyses of continuous outcome measures. To this end, six orthogonal design factors were manipulated: Strength of publication bias; 1-tailed vs. 2-tailed publication bias; prevalence of p-hacking; true heterogeneity of the effect studied; true average size of the studied effect; and number of studies per meta-analysis. Our results showed that biases in sets of primary studies caused much greater problems for the estimation of effect size than for the estimation of heterogeneity. For the latter, estimation bias remained small or moderate under most circumstances. Effect size estimations remained virtually unaffected by the choice of heterogeneity estimator. For heterogeneity estimates, however, relevant differences emerged. For unbiased primary studies, the REML estimator and (to a lesser extent) the Paule-Mandel performed well in terms of bias and variance. In biased sets of primary studies however, the Paule-Mandel estimator performed poorly, whereas the DerSimonian-Laird estimator and (to a slightly lesser extent) the REML estimator performed well. The complexity of results notwithstanding, we suggest that the REML estimator remains a good choice for meta-analyses of continuous outcome measures across varied circumstances.

Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.

Rigour and reproducibility in Canadian research: call for a coordinated approach

Shortcomings in the rigour and reproducibility of research have become well-known issues and persist despite repeated calls for improvement. A coordinated effort among researchers, institutions, funders, publishers, learned societies, and regulators may be the most effective way of tackling these issues. The UK Reproducibility Network (UKRN) has fostered collaboration across various stakeholders in research and are creating the infrastructure necessary to advance rigorous and reproducible research practices across the United Kingdom. Other Reproducibility Networks, modelled on UKRN, are now emerging in other countries. Canada could benefit from a comparable network to unify the voices around research quality and maximize the value of Canadian research.

Sampling bias exaggerates a textbook example of a trophic cascade

Understanding trophic cascades in terrestrial wildlife communities is a major challenge because these systems are difficult to sample properly. We show how a tradition of non-random sampling has confounded this understanding in a textbook system (Yellowstone National Park) where carnivore [Canis lupus (wolf)] recovery is associated with a trophic cascade involving changes in herbivore [Cervus canadensis (elk)] behaviour and density that promote plant regeneration. Long-term data indicate a practice of sampling only the tallest young plants overestimated regeneration of overstory aspen (Populus tremuloides) by a factor of 4-7 compared to random sampling because it favoured plants taller than the preferred browsing height of elk and overlooked non-regenerating aspen stands. Random sampling described a trophic cascade, but it was weaker than the one that non-random sampling described. Our findings highlight the critical importance of basic sampling principles (e.g. randomisation) for achieving an accurate understanding of trophic cascades in terrestrial wildlife systems.

Rewriting results sections in the language of evidence

Despite much criticism, black-or-white null-hypothesis significance testing with an arbitrary P-value cutoff still is the standard way to report scientific findings. One obstacle to progress is likely a lack of knowledge about suitable alternatives. Here, we suggest language of evidence that allows for a more nuanced approach to communicate scientific findings as a simple and intuitive alternative to statistical significance testing. We provide examples for rewriting results sections in research papers accordingly. Language of evidence has previously been suggested in medical statistics, and it is consistent with reporting approaches of international research networks, like the Intergovernmental Panel on Climate Change, for example. Instead of re-inventing the wheel, ecology and evolution might benefit from adopting some of the 'good practices' that exist in other fields.