Reproducibility in the Social Sciences

Reporting Standards for Bayesian Network Modelling

Reproducibility is a key measure of the veracity of a modelling result or finding. In other research areas, notably in medicine, reproducibility is supported by mandating the inclusion of an agreed set of details into every research publication, facilitating systematic reviews, transparency and reproducibility. Governments and international organisations are increasingly turning to modelling approaches in the development and decision-making for policy and have begun asking questions about accountability in model-based decision making. The ethical issues of relying on modelling that is biased, poorly constructed, constrained by heroic assumptions and not reproducible are multiplied when such models are used to underpin decisions impacting human and planetary well-being. Bayesian Network modelling is used in policy development and decision support across a wide range of domains. In light of the recent trend for governments and other organisations to demand accountability and transparency, we have compiled and tested a reporting checklist for Bayesian Network modelling which will bring the desirable level of transparency and reproducibility to enable models to support decision making and allow the robust comparison and combination of models. The use of this checklist would support the ethical use of Bayesian network modelling for impactful decision making and research.

The Replicability Crisis and Human Agency in the Neo-Structured World

The paper analyzes current discussions concerning the so called "replicability crisis" - a notion describing difficulties in attempts to confirm existing research findings by their additional scrutiny or by new empirical studies. We propose interpretation that this "crisis" may be seen as a manifestation of the increasing inconsistency between, on the one hand, the outdated views on a human being and social structures dominating in the academic mainstream across various disciplines, including psychology and sociology, and, on the other hand, the reality of the emerging new stage of societal evolution, neo-structuration, which brings to the forefront individual agency. Our analysis suggests the possibilities for the future inter-disciplinary paradigmatic shift, which implies putting in the center of research not the idea of a constant or predictably developing individual in the context of solid external structures operating in line with a presumably sustainable "progress". Instead, under increasing neo-structuration, individual agency becomes, simultaneously, a manifestation of the essence of human nature (as cultural psychology argues) and the driving force for societal transformations, including solving most acute social problems, in the concrete historical period. It means a fundamentally new task for social sciences and humanities: to elaborate methodological solutions and theoretical frameworks to systematically comprehend the contextually conditioned human ability to create and transform - and not only to reproduce. Addressing more attention to agency manifestations in digital environment and, in particular, to those congruent to social activism or volunteering, seems especially fruitful for comprehending human activity in the neo-structurated world.

Reproducibility and replicability in health professions education research

In this editorial the editor considers the twin issues of replicability and reproducibility in health professions education research, and notes challenges and opportunities that scholars in the field face in attending to the replicability and reproducibility of the work they produce.

Quality of systematic reviews with meta-analyses of resveratrol: A methodological systematic review

Recently, several meta-analyses (MAs) have focused on the health effects of resveratrol. However, the methodological and reporting quality of these MAs has not yet been fully evaluated so far. Therefore, the present study evaluated the quality of these MAs through a methodological systematic review. Systematic searches were conducted in PubMed, Embase, Web of Science, and Cochrane Library from inception until May 20, 2022, and PubMed was used to update the search until September 6, 2023. The methodological and reporting quality of the selected MAs was evaluated using AMSTAR-2 and PRISMA 2009. Fifty-one MAs published during 2013-2023 were included. In each review, the number of primary studies ranged from 3 to 37, and the number of participants ranged from 50 to 2114. Among the first-listed primary outcomes, only 23 (45.10%) were "positive." As for the methodological quality, most MAs (44, 86.27%) on resveratrol were rated critically low. Inadequate reporting of the included MAs mainly involved items 2 ("Structured summary"), 5 ("Protocol and registration"), 8 ("Search"), 9 ("Study selection"), 10 ("Data collection process"), 12 ("Risk of bias in individual studies"), and 24 ("Summary of evidence") based on the PRISMA 2009. Additionally, journal's impact factor, number of authors, and funding support were positively associated with the overall methodological quality but were not statistically significant (p > 0.05). Future MAs on resveratrol require better design, implementation, and reporting by following the Cochrane Handbook, AMSTAR-2, and PRISMA.

Bridging the Built and Social Environments: A Systematic Review of Studies Investigating Influences on Physical Activity

BACKGROUND

The purpose of this review was to examine articles assessing aspects of the built and social environment simultaneously, and how these environments influence physical activity (PA). A thorough review of studies is needed to identify patterns across studies and gaps for future research and practice.

METHODS

To be included, articles needed to contain: (1) self-report or objective measure of PA; (2) a measure of the built environment; (3) a measure of the social environment; and (4) an analysis between built environment, social environment, and PA. A systematic literature search of 4358 articles resulted in 87 articles.

RESULTS

Several populations were present within the sample including various age groups and different countries. As previously established, the built environment and social environment were consistently associated with PA; however, mediating factors between these 2 layers were less clear. Further, there was a lack of longitudinal and experimental study designs.

CONCLUSIONS

Results suggest a need for longitudinal and experimental designs with validated and granular measures. As communities recover from the COVID-19 pandemic, a thorough understanding of how built environment factors enhance or detract from social connectedness and how this reciprocal relationship impacts PA behavior is needed for future policy, environment, and systematic change.

Quantifying the robustness of causal inferences: Sensitivity analysis for pragmatic social science

Social scientists seeking to inform policy or public action must carefully consider how to identify effects and express inferences because actions based on invalid inferences may not yield the intended results. Recognizing the complexities and uncertainties of social science, we seek to inform inevitable debates about causal inferences by quantifying the conditions necessary to change an inference. Specifically, we review existing sensitivity analyses within the omitted variables and potential outcomes frameworks. We then present the Impact Threshold for a Confounding Variable (ITCV) based on omitted variables in the linear model and the Robustness of Inference to Replacement (RIR) based on the potential outcomes framework. We extend each approach to include benchmarks and to fully account for sampling variability represented by standard errors as well as bias. We exhort social scientists wishing to inform policy and practice to quantify the robustness of their inferences after utilizing the best available data and methods to draw an initial causal inference.