The role of replication studies in ecology

Predictability and conceptual repeatability of the predator-associated burst speed ecophenotype in western mosquitofish (Gambusia affinis)

Predation exerts strong selection pressure on morphological traits and is often studied in freshwater fishes. A common morphological ecophenotype driven by predation from pursuit predators is the predator-associated burst speed (PABS) ecophenotype. This ecophenotype, characterized by a smaller head, smaller body, and larger caudal region, is commonly found in western mosquitofish (Gambusia affinis) residing in environments with sunfish (family Centrarchidae) predators. However, the repeatability and transferability of the PABS ecophenotype across populations have not been tested. The four objectives of this study were to (1) assess the repeatability of methods to confirm the presence of the PABS ecophenotype in independent populations, (2) test whether morphology is correlated with predator abundance, (3) assess the repeatability of utilizing a discriminant function analysis (DFA) to accurately classify fish to predator status, and (4) use a DFA to conduct a cross-validation test of the PABS model using previously studied populations of G. affinis to predict predator status of newly studied populations. There was consistency in the occurrence of the PABS ecophenotype among populations with predators. Permuted MANOVA (PMANOVA) models fit to each dataset revealed a significant effect of collection site nested within predator status (i.e., presence or absence of predator), suggesting location-specific variations of the PABS ecophenotype. In addition, a PMANOVA model fit to body shape as a function of predator abundance (i.e., 0, 0.1-1.0, or >1.0 predators per minute of electrofishing) revealed a significant effect of predation level in the newly studied populations, suggesting morphology differs among populations with varying predator abundances. Cross-study validation of the DFA revealed low between-study accuracy compared to within-study accuracy, but elongation of the caudal region in the presence of predators was consistent across studies. Our findings ultimately suggest that although the PABS ecophenotype at studied locations provides partial predictive capacity at unstudied locations, the nature and magnitude of the PABS ecophenotype depend on environmental settings, encounter histories with predators, level of abundance of predators, or other spatially structured mechanisms.

Female behavior drives the formation of distinct social structures in C57BL/6J versus wild-derived outbred mice in field enclosures

BACKGROUND

Social behavior and social organization have major influences on individual health and fitness. Yet, biomedical research focuses on studying a few genotypes under impoverished social conditions. Understanding how lab conditions have modified social organizations of model organisms, such as lab mice, relative to natural populations is a missing link between socioecology and biomedical science.

RESULTS

Using a common garden design, we describe the formation of social structure in the well-studied laboratory mouse strain, C57BL/6J, in replicated mixed-sex populations over 10-day trials compared to control trials with wild-derived outbred house mice in outdoor field enclosures. We focus on three key features of mouse social systems: (i) territory establishment in males, (ii) female social relationships, and (iii) the social networks formed by the populations. Male territorial behaviors were similar but muted in C57 compared to wild-derived mice. Female C57 sharply differed from wild-derived females, showing little social bias toward cage mates and exploring substantially more of the enclosures compared to all other groups. Female behavior consistently generated denser social networks in C57 than in wild-derived mice.

CONCLUSIONS

C57 and wild-derived mice individually vary in their social and spatial behaviors which scale to shape overall social organization. The repeatable societies formed under field conditions highlights opportunities to experimentally study the interplay between society and individual biology using model organisms.

Compromise or choose: shared movement decisions in wild vulturine guineafowl

Shared-decision making is beneficial for the maintenance of group-living. However, little is known about whether consensus decision-making follows similar processes across different species. Addressing this question requires robust quantification of how individuals move relative to each other. Here we use high-resolution GPS-tracking of two vulturine guineafowl (Acryllium vulturinum) groups to test the predictions from a classic theoretical model of collective motion. We show that, in both groups, all individuals can successfully initiate directional movements, although males are more likely to be followed than females. When multiple group members initiate simultaneously, follower decisions depend on directional agreement, with followers compromising directions if the difference between them is small or choosing the majority direction if the difference is large. By aligning with model predictions and replicating the findings of a previous field study on olive baboons (Papio anubis), our results suggest that a common process governs collective decision-making in moving animal groups.

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

In many scientific disciplines, common research practices have led to unreliable and exaggerated evidence about scientific phenomena. Here we describe some of these practices and quantify their pervasiveness in recent ecology publications in five popular journals. In an analysis of over 350 studies published between 2018 and 2020, we detect empirical evidence of exaggeration bias and selective reporting of statistically significant results. This evidence implies that the published effect sizes in ecology journals exaggerate the importance of the ecological relationships that they aim to quantify. An exaggerated evidence base hinders the ability of empirical ecology to reliably contribute to science, policy, and management. To increase the credibility of ecology research, we describe a set of actions that ecologists should take, including changes to scientific norms about what high-quality ecology looks like and expectations about what high-quality studies can deliver.

Impact of dominance rank specification in dyadic interaction models

Dominance rank is a vital descriptor of social dynamics in animal societies and regularly used in studies to explain observed interaction patterns. However, researchers can choose between different indices and standardizations, and can specify dyadic rank relations differently when studying interaction distributions. These researcher degrees of freedom potentially introduce biases into studies and reduce replicability. Here, I demonstrate the impact of researcher choices by comparing the performance of different combinations of rank index, standardization, and model specification when explaining dyadic interaction patterns in sooty mangabeys (Cercocebus atys atys). I show that while no combination consistently performed best across interaction types (aggression, grooming, proximity, supplants), model specifications allowing for nonlinear patterns performed better than other models on average. Choices made in pre-processing and model building impacted model performance and subsequent interpretation of results. Researchers could end up describing social systems differently based on the same data. These results highlight the impact of researcher choices in the processing of behavioural data and potential limitations when using indirect species comparisons in animal behaviour research. To increase repeatability, researchers could make the impact of their processing choices more transparent and report results using a variety of indices and model specifications.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Exploring enablers and barriers to implementing the Transparency and Openness Promotion Guidelines: a theory-based survey of journal editors

The Transparency and Openness Promotion (TOP) Guidelines provide a framework to help journals develop open science policies. Theories of behaviour change can guide understanding of why journals do (not) implement open science policies and the development of interventions to improve these policies. In this study, we used the Theoretical Domains Framework to survey 88 journal editors on their capability, opportunity and motivation to implement TOP. Likert-scale questions assessed editor support for TOP, and enablers and barriers to implementing TOP. A qualitative question asked editors to provide reflections on their ratings. Most participating editors supported adopting TOP at their journal (71%) and perceived other editors in their discipline to support adopting TOP (57%). Most editors (93%) agreed their roles include maintaining policies that reflect current best practices. However, most editors (74%) did not see implementing TOP as a high priority compared with other editorial responsibilities. Qualitative responses expressed structural barriers to implementing TOP (e.g. lack of time, resources and authority to implement changes) and varying support for TOP depending on study type, open science standard, and level of implementation. We discuss how these findings could inform the development of theoretically guided interventions to increase open science policies, procedures and practices.

Female preferences for the spectral content of advertisement calls in Cope's gray treefrog (Hyla chrysoscelis)

Amphibians have inner ears with two sensory papillae tuned to different frequency ranges of airborne sounds. In frogs, male advertisement calls possess distinct spectral components that match the tuning of one or both sensory papillae. Female preferences for the spectral content of advertisement calls can depend on signal amplitude and can vary among closely related lineages. In this study of Cope's gray tree frog (Hyla chrysoscelis), we investigated the amplitude dependence of female preferences for the spectral content of male advertisement calls, which have a "bimodal" spectrum with separate low-frequency (1.25 kHz) and high-frequency (2.5 kHz) components. In two-alternative choice tests, females generally preferred synthetic calls with bimodal spectra over "unimodal" calls having only one of the two spectral components. They also preferred unimodal calls with a high-frequency component over one with the low-frequency component. With few exceptions, preferences were largely independent of amplitude across both a 30 dB range of overall signal amplitude and an 11 dB range in the relative amplitudes of the two spectral components. We discuss these results in the context of evolutionary lability in female preferences for the spectral content of advertisement calls in North American tree frogs in the genus Hyla.

Chiral NMR analysis reveals the environmental dependence of areolal scalemization in Piptothrix areolare

The occurrence of racemic and enantiomerically enriched (scalemic) mixtures of secondary metabolites in their natural sources is a rare phenomenon. The unprecedent case of enantiomeric variations from levorotatory to dextrorotatory, and back to levorotatory, passing through an almost racemic mixture, was recently documented for areolal, the major epoxythymol of Piptothrix areolare. In an attempt to shed some light to understand the reasons for such an unusual behavior, herein, we evaluated this phenomenon by correlating the areolal enantiomeric purity with several environmental variables, including temperature, humidity, rain precipitation, wind speed, and radiation during over 1 year of the plant life cycle. The specific rotation and enantiomeric excess determined by ¹ H-NMR-BINOL measurements provided the scalemic variations of areolal samples isolated from the roots collected from the same location along a 427-day period. The ¹ H-NMR-BINOL methodology provided better sensitivity to enantiomeric variations than specific rotation measurements. Statistical data, including matrix correlation analysis, exploratory analysis by heatmap plotting, and the principal component analysis (PCA), suggested direct correlation of the scalemic variation with humidity, rain precipitation, and radiation variables with the best PCA explanation (78.4%) and noncritical or poor correlations in PCA explained in 60.2% and 48.4%, respectively. When variations in the optical activity parameter of any metabolite are observed, the search for scalemic mixtures along their host plant life cycle should be undertaken. Herein, this phenomenon could be associated with interactions with soil microorganisms and with evolutionary aspects of Piptothrix areolare which belongs to Asteraceae, one of the most successfully adaptable plant families.

Paths towards greater consensus building in experimental biology

In a recent editorial, the Editors-in-Chief of Journal of Experimental Biology argued that consensus building, data sharing, and better integration across disciplines are needed to address the urgent scientific challenges posed by climate change. We agree and expand on the importance of cross-disciplinary integration and transparency to improve consensus building and advance climate change research in experimental biology. We investigated reproducible research practices in experimental biology through a review of open data and analysis code associated with empirical studies on three debated paradigms and for unrelated studies published in leading journals in comparative physiology and behavioural ecology over the last 10 years. Nineteen per cent of studies on the three paradigms had open data, and 3.2% had open code. Similarly, 12.1% of studies in the journals we examined had open data, and 3.1% had open code. Previous research indicates that only 50% of shared datasets are complete and re-usable, suggesting that fewer than 10% of studies in experimental biology have usable open data. Encouragingly, our results indicate that reproducible research practices are increasing over time, with data sharing rates in some journals reaching 75% in recent years. Rigorous empirical research in experimental biology is key to understanding the mechanisms by which climate change affects organisms, and ultimately promotes evidence-based conservation policy and practice. We argue that a greater adoption of open science practices, with a particular focus on FAIR (Findable, Accessible, Interoperable, Re-usable) data and code, represents a much-needed paradigm shift towards improved transparency, cross-disciplinary integration, and consensus building to maximize the contributions of experimental biologists in addressing the impacts of environmental change on living organisms.

Low statistical power and overestimated anthropogenic impacts, exacerbated by publication bias, dominate field studies in global change biology

Field studies are essential to reliably quantify ecological responses to global change because they are exposed to realistic climate manipulations. Yet such studies are limited in replicates, resulting in less power and, therefore, potentially unreliable effect estimates. Furthermore, while manipulative field experiments are assumed to be more powerful than non-manipulative observations, it has rarely been scrutinized using extensive data. Here, using 3847 field experiments that were designed to estimate the effect of environmental stressors on ecosystems, we systematically quantified their statistical power and magnitude (Type M) and sign (Type S) errors. Our investigations focused upon the reliability of field experiments to assess the effect of stressors on both ecosystem's response magnitude and variability. When controlling for publication bias, single experiments were underpowered to detect response magnitude (median power: 18%-38% depending on effect sizes). Single experiments also had much lower power to detect response variability (6%-12% depending on effect sizes) than response magnitude. Such underpowered studies could exaggerate estimates of response magnitude by 2-3 times (Type M errors) and variability by 4-10 times. Type S errors were comparatively rare. These observations indicate that low power, coupled with publication bias, inflates the estimates of anthropogenic impacts. Importantly, we found that meta-analyses largely mitigated the issues of low power and exaggerated effect size estimates. Rather surprisingly, manipulative experiments and non-manipulative observations had very similar results in terms of their power, Type M and S errors. Therefore, the previous assumption about the superiority of manipulative experiments in terms of power is overstated. These results call for highly powered field studies to reliably inform theory building and policymaking, via more collaboration and team science, and large-scale ecosystem facilities. Future studies also require transparent reporting and open science practices to approach reproducible and reliable empirical work and evidence synthesis.

Personality, density and habitat drive the dispersal of invasive crayfish

There is increasing evidence that personality traits may drive dispersal patterns of animals, including invasive species. We investigated, using the widespread signal crayfish Pacifastacus leniusculus as a model invasive species, whether effects of personality traits on dispersal were independent of, or affected by, other factors including population density, habitat, crayfish size, sex and limb loss, along an invasion gradient. Behavioural traits (boldness, activity, exploration, willingness to climb) of 310 individually marked signal crayfish were measured at fully-established, newly-established and invasion front sites of two upland streams. After a period at liberty, recaptured crayfish were reassessed for behavioural traits (newly-established, invasion front). Dispersal distance and direction of crayfish movement, local population density, fine-scale habitat characteristics and crayfish size, sex and limb loss were also measured. Individual crayfish exhibited consistency in behavioural traits over time which formed a behavioural syndrome. Dispersal was both positively and negatively affected by personality traits, positively by local population density and negatively by refuge availability. No effect of size, sex and limb loss was recorded. Personality played a role in promoting dispersal but population density and local habitat complexity were also important determinants. Predicting biological invasion in animals is likely to require better integration of these processes.

Crowdsourcing Global Perspectives in Ecology Using Social Media

Transparent, open, and reproducible research is still far from routine, and the full potential of open science has not yet been realized. Crowdsourcing–defined as the usage of a flexible open call to a heterogeneous group of individuals to recruit volunteers for a task –is an emerging scientific model that encourages larger and more outwardly transparent collaborations. While crowdsourcing, particularly through citizen- or community-based science, has been increasing over the last decade in ecological research, it remains infrequently used as a means of generating scientific knowledge in comparison to more traditional approaches. We explored a new implementation of crowdsourcing by using an open call on social media to assess its utility to address fundamental ecological questions. We specifically focused on pervasive challenges in predicting, mitigating, and understanding the consequences of disturbances. In this paper, we briefly review open science concepts and their benefits, and then focus on the new methods we used to generate a scientific publication. We share our approach, lessons learned, and potential pathways forward for expanding open science. Our model is based on the beliefs that social media can be a powerful tool for idea generation and that open collaborative writing processes can enhance scientific outcomes. We structured the project in five phases: (1) draft idea generation, (2) leadership team recruitment and project development, (3) open collaborator recruitment via social media, (4) iterative paper development, and (5) final editing, authorship assignment, and submission by the leadership team. We observed benefits including: facilitating connections between unusual networks of scientists, providing opportunities for early career and underrepresented groups of scientists, and rapid knowledge exchange that generated multidisciplinary ideas. We also identified areas for improvement, highlighting biases in the individuals that self-selected participation and acknowledging remaining barriers to contributing new or incompletely formed ideas into a public document. While shifting scientific paradigms to completely open science is a long-term process, our hope in publishing this work is to encourage others to build upon and improve our efforts in new and creative ways.

Addressing context dependence in ecology

Context dependence is widely invoked to explain disparate results in ecology. It arises when the magnitude or sign of a relationship varies due to the conditions under which it is observed. Such variation, especially when unexplained, can lead to spurious or seemingly contradictory conclusions, which can limit understanding and our ability to transfer findings across studies, space, and time. Using examples from biological invasions, we identify two types of context dependence resulting from four sources: mechanistic context dependence arises from interaction effects; and apparent context dependence can arise from the presence of confounding factors, problems of statistical inference, and methodological differences among studies. Addressing context dependence is a critical challenge in ecology, essential for increased understanding and prediction.

The hidden side of animal cognition research: Scientists' attitudes toward bias, replicability and scientific practice

Animal cognition research aims to understand animal minds by using a diverse range of methods across an equally diverse range of species. Throughout its history, the field has sought to mitigate various biases that occur when studying animal minds, from experimenter effects to anthropomorphism. Recently, there has also been a focus on how common scientific practices might affect the reliability and validity of published research. Usually, these issues are discussed in the literature by a small group of scholars with a specific interest in the topics. This study aimed to survey a wider range of animal cognition researchers to ask about their attitudes towards classic and contemporary issues facing the field. Two-hundred and ten active animal cognition researchers completed our survey, and provided answers on questions relating to bias, replicability, statistics, publication, and belief in animal cognition. Collectively, researchers were wary of bias in the research field, but less so in their own work. Over 70% of researchers endorsed Morgan’s canon as a useful principle but many caveated this in their free-text responses. Researchers self-reported that most of their studies had been published, however they often reported that studies went unpublished because they had negative or inconclusive results, or results that questioned “preferred” theories. Researchers rarely reported having performed questionable research practices themselves—however they thought that other researchers sometimes (52.7% of responses) or often (27.9% of responses) perform them. Researchers near unanimously agreed that replication studies are important but too infrequently performed in animal cognition research, 73.0% of respondents suggested areas of animal cognition research could experience a ‘replication crisis’ if replication studies were performed. Consistently, participants’ free-text responses provided a nuanced picture of the challenges animal cognition research faces, which are available as part of an open dataset. However, many researchers appeared concerned with how to interpret negative results, publication bias, theoretical bias and reliability in areas of animal cognition research. Collectively, these data provide a candid overview of barriers to progress in animal cognition and can inform debates on how individual researchers, as well as organizations and journals, can facilitate robust scientific research in animal cognition.

Australian and Italian Psychologists’ View of Replication

Solutions to the crisis in confidence in the psychological literature have been proposed in many recent articles, including increased publication of replication studies, a solution that requires engagement by the psychology research community. We surveyed Australian and Italian academic research psychologists about the meaning and role of replication in psychology. When asked what they consider to be a replication study, nearly all participants (98% of Australians and 96% of Italians) selected options that correspond to a direct replication. Only 14% of Australians and 8% of Italians selected any options that included changing the experimental method. Majorities of psychologists from both countries agreed that replications are very important, that more replications should be done, that more resources should be allocated to them, and that they should be published more often. Majorities of psychologists from both countries reported that they or their students sometimes or often replicate studies, yet they also reported having no replication studies published in the prior 5 years. When asked to estimate the percentage of published studies in psychology that are replications, both Australians (with a median estimate of 13%) and Italians (with a median estimate of 20%) substantially overestimated the actual rate. When asked what constitute the main obstacles to replications, difficulty publishing replications was the most frequently cited obstacle, coupled with the high value given to innovative or novel research and the low value given to replication studies.

Replications, Comparisons, Sampling and the Problem of Representativeness in Animal Cognition Research

Animal cognition research often involves small and idiosyncratic samples. This can constrain the generalizability and replicability of a study's results and prevent meaningful comparisons between samples. However, there is little consensus about what makes a strong replication or comparison in animal research. We apply a resampling definition of replication to answer these questions in Part 1 of this article, and, in Part 2, we focus on the problem of representativeness in animal research. Through a case study and a simulation study, we highlight how and when representativeness may be an issue in animal behavior and cognition research and show how the representativeness problems can be viewed through the lenses of, i) replicability, ii) generalizability and external validity, iii) pseudoreplication and, iv) theory testing. Next, we discuss when and how researchers can improve their ability to learn from small sample research through, i) increasing heterogeneity in experimental design, ii) increasing homogeneity in experimental design, and, iii) statistically modeling variation. Finally, we describe how the strongest solutions will vary depending on the goals and resources of individual research programs and discuss some barriers towards implementing them.

The REPRISE project: protocol for an evaluation of REProducibility and Replicability In Syntheses of Evidence

BACKGROUND

Investigations of transparency, reproducibility and replicability in science have been directed largely at individual studies. It is just as critical to explore these issues in syntheses of studies, such as systematic reviews, given their influence on decision-making and future research. We aim to explore various aspects relating to the transparency, reproducibility and replicability of several components of systematic reviews with meta-analysis of the effects of health, social, behavioural and educational interventions.

METHODS

The REPRISE (REProducibility and Replicability In Syntheses of Evidence) project consists of four studies. We will evaluate the completeness of reporting and sharing of review data, analytic code and other materials in a random sample of 300 systematic reviews of interventions published in 2020 (Study 1). We will survey authors of systematic reviews to explore their views on sharing review data, analytic code and other materials and their understanding of and opinions about replication of systematic reviews (Study 2). We will then evaluate the extent of variation in results when we (a) independently reproduce meta-analyses using the same computational steps and analytic code (if available) as used in the original review (Study 3), and (b) crowdsource teams of systematic reviewers to independently replicate a subset of methods (searches for studies, selection of studies for inclusion, collection of outcome data, and synthesis of results) in a sample of the original reviews; 30 reviews will be replicated by 1 team each and 2 reviews will be replicated by 15 teams (Study 4).

DISCUSSION

The REPRISE project takes a systematic approach to determine how reliable systematic reviews of interventions are. We anticipate that results of the REPRISE project will inform strategies to improve the conduct and reporting of future systematic reviews.