Assessing the reproducibility of discriminant function analyses

Examining the generalizability of research findings from archival data

The extent to which results from complex datasets generalize across contexts is critically important to numerous scientific fields as well as to practitioners who rely on such analyses to guide important strategic decisions. Our initiative systematically investigated whether findings from the field of strategic management would emerge in new time periods and new geographies. Original findings that were statistically reliable in the first place were typically obtained again in novel tests, suggesting surprisingly little sensitivity to context. For some social scientific areas of inquiry, results from a specific time and place can be a meaningful guide as to what will be observed more generally.

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Estimating the reproducibility of social learning research published between 1955 and 2018

Reproducibility is integral to science, but difficult to achieve. Previous research has quantified low rates of data availability and results reproducibility across the biological and behavioural sciences. Here, we surveyed 560 empirical publications, published between 1955 and 2018 in the social learning literature, a research topic that spans animal behaviour, behavioural ecology, cultural evolution and evolutionary psychology. Data were recoverable online or through direct data requests for 30% of this sample. Data recovery declines exponentially with time since publication, halving every 6 years, and up to every 9 years for human experimental data. When data for a publication can be recovered, we estimate a high probability of subsequent data usability (87%), analytical clarity (97%) and agreement of published results with reproduced findings (96%). This corresponds to an overall rate of recovering data and reproducing results of 23%, largely driven by the unavailability or incompleteness of data. We thus outline clear measures to improve the reproducibility of research on the ecology and evolution of social behaviour.

Exploring farmers' attitudes and determinants of dairy calf welfare in an expanding dairy sector

The 2015 European Union milk quota abolition initiated considerable expansion in the dairy sector in many European Union countries, most significantly in Ireland. However, this major production increase also had wider societal implications, such as negative environmental and animal welfare consequences. In this article, we used survey data of 441 Irish dairy farmers to assess farmers' attitudes toward the welfare of farmed animals and dairy calves, as well as the reputation of the Irish dairy sector. We also explored how expansion, breeding, calf management, and farmer characteristics relate to calf welfare outcomes (i.e., calf mortality, calf export, and premature culling). In relation to attitudes, farmers expressed a general concern toward animal welfare, while views toward dairy calves and industry reputation were mixed. We used Ward's linkage hierarchical cluster analysis to group farmers based on their attitudes. The cluster analysis revealed 3 distinct groups relating to high, medium, and low animal welfare concern. Herd expansion was negatively associated with being in a higher animal welfare concern cluster, whereas beef trait-focused breeding was positively associated with it. In relation to dairy calf welfare outcomes, our econometric analyses based on multiple regression and binary choice models revealed that expansion was positively associated with calf mortality, whereas improved breeding and calf management factors had a negative association. In addition, being in the high animal welfare concern cluster was negatively associated with calf mortality. Furthermore, breeding decisions were significantly associated with whether calves were exported, and being in the high animal welfare concern cluster was negatively associated with the probability that calves were sent for live export. Finally, farmers' breeding and calf management decisions were associated with premature culling of calves. Overall, this article revealed strategies worth promoting to improve dairy calf welfare, such as beef trait-focused breeding leading to greater dairy-beef integration.

Insect morphometry is reproducible under average investigation standards

Morphometric research is being applied to a growing number and variety of organisms. Discoveries achieved via morphometric approaches are often considered highly transferable, in contrast to the tacit and idiosyncratic interpretation of discrete character states. The reliability of morphometric workflows in insect systematics has never been a subject of focused research, but such studies are sorely needed. In this paper, we assess the reproducibility of morphometric studies of ants where the mode of data collection is a shared routine.We compared datasets generated by eleven independent gaugers, that is, collaborators, who measured 21 continuous morphometric traits on the same pool of individuals according to the same protocol. The gaugers possessed a wide range of morphometric skills, had varying expertise among insect groups, and differed in their facility with measuring equipment. We used intraclass correlation coefficients (ICC) to calculate repeatability and reproducibility values (i.e., intra- and intergauger agreements), and we performed a multivariate permutational multivariate analysis of variance (PERMANOVA) using the Morosita index of dissimilarity with 9,999 iterations.The calculated average measure of intraclass correlation coefficients of different gaugers ranged from R = 0.784 to R = 0.9897 and a significant correlation was found between the repeatability and the morphometric skills of gaugers (p = 0.016). There was no significant association with the magnification of the equipment in the case of these rather small ants. The intergauger agreement, that is the reproducibility, varied between R = 0.872 and R = 0.471 (mean R = 0.690), but all gaugers arrived at the same two-species conclusion. A PERMANOVA test revealed no significant gauger effect on species identity (R 2 = 0.69, p = 0.58).Our findings show that morphometric studies are reproducible when observers follow the standard protocol; hence, morphometric findings are widely transferable and will remain a valuable data source for alpha taxonomy.

Successes and Struggles with Computational Reproducibility: Lessons from the Fragile Families Challenge

Reproducibility is fundamental to science, and an important component of reproducibility is computational reproducibility: the ability of a researcher to recreate the results of a published study using the original author's raw data and code. Although most people agree that computational reproducibility is important, it is still difficult to achieve in practice. In this article, the authors describe their approach to enabling computational reproducibility for the 12 articles in this special issue of Socius about the Fragile Families Challenge. The approach draws on two tools commonly used by professional software engineers but not widely used by academic researchers: software containers (e.g., Docker) and cloud computing (e.g., Amazon Web Services). These tools made it possible to standardize the computing environment around each submission, which will ease computational reproducibility both today and in the future. Drawing on their successes and struggles, the authors conclude with recommendations to researchers and journals.

The links between ecosystem multifunctionality and above- and belowground biodiversity are mediated by climate

Plant biodiversity is often correlated with ecosystem functioning in terrestrial ecosystems. However, we know little about the relative and combined effects of above- and belowground biodiversity on multiple ecosystem functions (for example, ecosystem multifunctionality, EMF) or how climate might mediate those relationships. Here we tease apart the effects of biotic and abiotic factors, both above- and belowground, on EMF on the Tibetan Plateau, China. We found that a suite of biotic and abiotic variables account for up to 86% of the variation in EMF, with the combined effects of above- and belowground biodiversity accounting for 45% of the variation in EMF. Our results have two important implications: first, including belowground biodiversity in models can improve the ability to explain and predict EMF. Second, regional-scale variation in climate, and perhaps climate change, can determine, or at least modify, the effects of biodiversity on EMF in natural ecosystems.

Plant biodiversity often has a positive influence on ecosystem functioning. Here Jing et al. show that belowground diversity can also significantly impact ecosystem multifunctionality, but its relative effect varies by regional-scale climate variation.