Saving Science Through Replication Studies

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

Suspicion About Suspicion Probes: Ways Forward

Suspicion probes are the traditional tool employed to assess the extent to which participants suspect intentional misdirection or deception within the research context. A primary reason psychologists use deception in research settings is to prevent participants from altering their behavior in light of knowing what is being studied, which could undermine internal validity as well as threaten the generalizability of findings to the real world (i.e., external validity). The present article elucidates a number of challenges with suspicion probes. A definition and framework for conceptualizing the construct of suspicion in research settings are proposed. Following a literature review, an analysis of existing evidence, and new data on the prevalence of using and reporting suspicion probes, we conclude that suspicion is a likely problem in research practice. We provide a decision guide to help researchers navigate the numerous choices involved in addressing potential suspicion and call for a combination of (a) renewed research leading to empirically supported tools and best practices and (b) systemic changes to editorial policies, funding practices, professional standards, and research training that would increase rigor and focus on this aspect of research methodology.

Unconscious integration: Current evidence for integrative processing under subliminal conditions

Integrative processing is traditionally believed to be dependent on consciousness. While earlier studies within the last decade reported many types of integration under subliminal conditions (i.e. without perceptual awareness), these findings are widely challenged recently. This review evaluates the current evidence for 10 types of subliminal integration that are widely studied: arithmetic processing, object-context integration, multi-word processing, same-different processing, multisensory integration and 5 different types of associative learning. Potential methodological issues concerning awareness measures are also taken into account. It is concluded that while there is currently no reliable evidence for subliminal integration, this does not necessarily refute 'unconscious' integration defined through non-subliminal (e.g. implicit) approaches.

Ecological validity of don't remember and don't know for distinguishing accessibility- versus availability-based retrieval failures in older and younger adults: knowledge for news events

With pursuit of incremental progress and generalizability of findings in mind, we examined a possible boundary for older and younger adults' metacognitive distinction between what is not stored in memory versus merely inaccessible with materials that are not process pure to knowledge or events: information regarding news events. Participants were asked questions about public events such as celebrity news, tragedies, and political events that were widely experienced in the previous 10-12 years, responding "I don't know" (DK) or "I don't remember" (DR) when retrieval failed. Memories of these events are relatively recently acquired in rich, naturalistic contexts and are likely not fully separated from episodic details. When retrieval failed, DR items were recognized with higher accuracy than DK items, both immediately and 2 years later, confirming that self-reported not remembering reflects failures of accessibility, whereas not knowing better captures a lack of availability. In fact, older adults distinguished between the causes of retrieval failures more precisely than younger adults. Together, these findings advance the reliability, validity, and generalizability of using DR and DK as a metacognitive tool to address the phenomenological experience and behavioral consequences of retrieval failures of information that contains both semantic and episodic features. Implications for metacognition in aging and related constructs like familiarity, remembering, and knowing are discussed.