The replication crisis in epidemiology: snowball, snow job, or winter solstice?

Cardiovascular Outcomes of α-Blockers vs 5-α Reductase Inhibitors for Benign Prostatic Hyperplasia

Importance

The most prescribed class of medications for benign prostatic hyperplasia (BPH) is α-blockers (ABs). However, the cardiovascular safety profile of these medications among patients with BPH is not well understood.

Objective

To compare the safety of ABs vs 5-α reductase inhibitors (5-ARIs) for risk of adverse cardiovascular outcomes.

Design, Setting, and Participants

This active comparator, new-user cohort study was conducted using insurance claims data from a 20% random sample of Medicare beneficiaries from 2007 to 2019 to evaluate the 1-year risk of adverse cardiovascular outcomes. Males aged 66 to 90 years were indexed into the cohort at new use of an AB or 5-ARI. Twelve months of continuous enrollment and at least 1 diagnosis code for BPH within 12 months prior to initiation were required. Data were analyzed from January 2007 through December 2019.

Exposures

Exposure was defined by a qualifying prescription fill for an AB or 5-ARI after at least 12 months without a prescription for these drug classes.

Main Outcomes and Measures

Follow-up began at a qualified refill for the study drug. Primary study outcomes were hospitalization for heart failure (HF), composite major adverse cardiovascular events (MACE; hospitalization for stroke, myocardial infarction, or death), composite MACE or hospitalization for HF, and death. Inverse probability of treatment and censoring-weighted 1-year risks, risk ratios (RRs), and risk differences (RDs) were estimated for each outcome.

Results

Among 189 868 older adult males, there were 163 829 patients initiating ABs (mean [SD] age, 74.6 [6.2] years; 579 American Indian or Alaska Native [0.4%], 5890 Asian or Pacific Islander [3.6%], 9179 Black [5.6%], 10 610 Hispanic [6.5%], and 133 510 non-Hispanic White [81.5%]) and 26 039 patients initiating 5-ARIs (mean [SD] age, 75.3 [6.4] years; 76 American Indian or Alaska Native [0.3%], 827 Asian or Pacific Islander [3.2%], 1339 Black [5.1%], 1656 Hispanic [6.4%], and 21 605 non-Hispanic White [83.0%]). ABs compared with 5-ARIs were associated with an increased 1-year risk of MACE (8.95% [95% CI, 8.81%-9.09%] vs 8.32% [95% CI, 7.92%-8.72%]; RR = 1.08 [95% CI, 1.02-1.13]; RD per 1000 individuals = 6.26 [95% CI, 2.15-10.37]), composite MACE and HF (RR = 1.07; [95% CI, 1.03-1.12]; RD per 1000 individuals = 7.40 [95% CI, 2.88-11.93 ]), and death (RR = 1.07; [95% CI, 1.01-1.14]; RD per 1000 individuals = 3.85 [95% CI, 0.40-7.29]). There was no difference in risk for HF hospitalization alone.

Conclusions and Relevance

These results suggest that ABs may be associated with an increased risk of adverse cardiovascular outcomes compared with 5-ARIs. If replicated with more detailed confounder data, these results may have important public health implications given these medications' widespread use.

Promoting Reproducibility and Integrity in Observational Research: One Approach of an Epidemiology Research Community

To increase research reproducibility, sharing of study data, analysis code, and use of standardized reporting are increasingly advocated. However, beyond reproducibility, few initiatives have addressed the integrity of how research is conducted before manuscripts are submitted. We describe a decades-long experience with a comprehensive approach based in an academic research community around prospective cohort studies that is aimed at promoting a culture of integrity in observational research. The approach includes prespecifying hypotheses and analysis plans, which are discussed in the research community and posted; presentation and discussion of analysis results; mandatory analysis code review by a programmer; review of concordance between analysis output and manuscripts by a technical reviewer; and checks of adherence to the process, including compliance with institutional review board requirements and reporting stipulations by the National Institutes of Health. The technical core is based in shared computing and analytic environments with long-term archiving. More than simply a list of rules, our approach promotes research integrity through integrated educational elements, making it part of the "hidden curriculum," by fostering a sense of belonging, and by providing efficiency gains to the research community. Unlike reproducibility checklists, such long-term investments into research integrity require substantial and sustained funding for research personnel and computing infrastructure. Our experiences suggest avenues for how institutions, research communities, and funders involved in observational research can strengthen integrity within the research process.

Toward Open and Reproducible Epidemiology

Starting in the 2010s, researchers in the experimental social sciences rapidly began to adopt increasingly open and reproducible scientific practices. These practices include publicly sharing deidentified data when possible, sharing analytical code, and preregistering study protocols. Empirical evidence from the social sciences suggests such practices are feasible, can improve analytical reproducibility, and can reduce selective reporting. In academic epidemiology, adoption of open-science practices has been slower than in the social sciences (with some notable exceptions, such as registering clinical trials). Epidemiologic studies are often large, complex, conceived after data have already been collected, and difficult to replicate directly by collecting new data. These characteristics make it especially important to ensure their integrity and analytical reproducibility. Open-science practices can also pay immediate dividends to researchers' own work by clarifying scientific reasoning and encouraging well-documented, organized workflows. We consider how established epidemiologists and early-career researchers alike can help midwife a culture of open science in epidemiology through their research practices, mentorship, and editorial activities.

Prevention of Neisseria gonorrhoeae With Meningococcal B Vaccine: A Matched Cohort Study in Southern California

BACKGROUND

Neisseria gonorrhoeae is acquiring increasing resistance to available oral antibiotics, and current screening and treatment approaches have not decreased gonorrhea incidence. Although a gonorrhea-specific vaccine does not exist, N. gonorrhoeae shares much of its genome with Neisseria meningitidis, notably critical antigenic determinants including outer membrane vesicles (OMV). Prior observational studies have suggested that OMV-based meningococcal serogroup B vaccines confer protection against gonorrhea.

METHODS

We conducted a matched cohort study from 2016 to 2020 to examine the association of OMV-containing recombinant meningococcal serogroup B vaccine (4CMenB) with gonorrhea infection among teens and young adults at Kaiser Permanente Southern California. Recipients of 4CMenB were matched in a ratio of 1:4 to recipients of non-OMV-containing polysaccharide-conjugate vaccine targeting serotypes A, C, W, and Y (MenACWY) who had not received 4CMenB and were followed for incident gonorrhea. We used Cox proportional hazards regression to compare gonorrhea rates among recipients of 4CMenB vs MenACWY, adjusting for potential confounders. We conducted the same analysis with chlamydia as a negative control outcome.

RESULTS

The study included 6641 recipients of 4CMenB matched to 26 471 recipients of MenACWY. During follow-up, gonorrhea incidence rates per 1000 person-years (95% confidence intervals [CIs]) were 2.0 (1.3-2.8) for recipients of 4CMenB and 5.2 (4.6-5.8) for recipients of MenACWY. In adjusted analyses, gonorrhea rates were 46% lower among recipients of 4CMenB vs MenACWY (hazard ratio [HR], 0.54; 95% CI, .34-.86), but chlamydia rates were similar between vaccine groups (HR, 0.98; 95% CI, .82-1.17).

CONCLUSIONS

These results suggest cross-protection of 4CMenB against gonorrhea, supporting the potential for vaccination strategies to prevent gonorrhea.

Understanding Suicide over the Life Course Using Data Science Tools within a Triangulation Framework

Suicide and suicidal behaviors are important global health concerns. Preventing suicide requires a nuanced understanding of the nature of suicide risk, both acutely during periods of crisis and broader variation over the lifespan. However, current knowledge of the sources of variation in suicide risk is limited due to methodological and conceptual challenges. New methodological approaches are needed to close the gap between research and clinical practice. This review describes the life course framework as a conceptual model for organizing the scientific study of suicide risk across in four major domains: social relationships, health, housing, and employment. In addition, this review discusses the utility of data science tools as a means of identifying novel, modifiable risk factors for suicide, and triangulation as an overarching approach to ensuring rigor in suicide research as means of addressing existing knowledge gaps and strengthening future research.

Open science, the replication crisis, and environmental public health

Concerns about a crisis of mass irreplicability across scientific fields ("the replication crisis") have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code. Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health. The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis. There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health - or not - seem premature. By distinguishing three concepts - reproducibility, replicability, and robustness - it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability. I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

Assessing the Robustness of Mediation Analysis Results Using Multiverse Analysis

There is an increasing awareness that replication should become common practice in empirical studies. However, study results might fail to replicate for various reasons. The robustness of published study results can be assessed using the relatively new multiverse-analysis methodology, in which the robustness of the effect estimates against data analytical decisions is assessed. However, the uptake of multiverse analysis in empirical studies remains low, which might be due to the scarcity of guidance available on performing multiverse analysis. Researchers might experience difficulties in identifying data analytical decisions and in summarizing the large number of effect estimates yielded by a multiverse analysis. These difficulties are amplified when applying multiverse analysis to assess the robustness of the effect estimates from a mediation analysis, as a mediation analysis involves more data analytical decisions than a bivariate analysis. The aim of this paper is to provide an overview and worked example of the use of multiverse analysis to assess the robustness of the effect estimates from a mediation analysis. We showed that the number of data analytical decisions in a mediation analysis is larger than in a bivariate analysis. By using a real-life data example from the Longitudinal Aging Study Amsterdam, we demonstrated the application of multiverse analysis to a mediation analysis. This included the use of specification curves to determine the impact of data analytical decisions on the magnitude and statistical significance of the direct, indirect, and total effect estimates. Although the multiverse analysis methodology is still relatively new and future research is needed to further advance this methodology, this paper shows that multiverse analysis is a useful method for the assessment of the robustness of the direct, indirect, and total effect estimates in a mediation analysis and thereby to inform replication studies.

Social Epidemiology: Past, Present, and Future

In a context where epidemiologic research has been heavily influenced by a biomedical and individualistic approach, the naming of “social epidemiology” allowed explicit emphasis on the social production of disease as a powerful explanatory paradigm and as critically important for interventions to improve population health. This review briefly highlights key substantive areas of focus in social epidemiology over the past 30 years, reflects on major advances and insights, and identifies challenges and possible future directions. Future opportunities for social epidemiology include grounding research in theoretically based and systemic conceptual models of the fundamental social drivers of health; implementing a scientifically rigorous yet realistic approach to drawing conclusions about social causes; using complementary methods to generate valid explanations and identify effective actions; leveraging the power of harmonization, replication, and big data; extending interdisciplinarity and diversity; advancing emerging critical approaches to understanding the health impacts of systemic racism and its policy implications; going global; and embracing a broad approach to generating socially useful research.

Expected final online publication date for the Annual Review of Public Health, Volume 43 is April 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Code Review as a Simple Trick to Enhance Reproducibility, Accelerate Learning, and Improve the Quality of Your Team's Research

Programming for data wrangling and statistical analysis is an essential technical tool of modern epidemiology, yet many epidemiologists receive limited formal training in strategies to optimize the quality of our code. In complex projects, coding mistakes are easy to make, even for skilled practitioners. Such mistakes can lead to invalid research claims that reduce the credibility of the field. Code review is a straightforward technique used by the software industry to reduce the likelihood of coding bugs. The systematic implementation of code review in epidemiologic research projects could not only improve science but also decrease stress, accelerate learning, contribute to team building, and codify best practices. In the present article, we argue for the importance of code review and provide some recommendations for successful implementation for 1) the research laboratory, 2) the code author (the initial programmer), and 3) the code reviewer. We outline a feasible strategy for implementation of code review, though other successful implementation processes are possible to accommodate the resources and workflows of different research groups, including other practices to improve code quality. Code review isn't always glamorous, but it is critically important for science and reproducibility. Humans are fallible; that's why we need code review.

Powering population health research: Considerations for plausible and actionable effect sizes

Evidence for Action (E4A), a signature program of the Robert Wood Johnson Foundation, funds investigator-initiated research on the impacts of social programs and policies on population health and health inequities. Across thousands of letters of intent and full proposals E4A has received since 2015, one of the most common methodological challenges faced by applicants is selecting realistic effect sizes to inform calculations of power, sample size, and minimum detectable effect (MDE). E4A prioritizes health studies that are both (1) adequately powered to detect effect sizes that may reasonably be expected for the given intervention and (2) likely to achieve intervention effects sizes that, if demonstrated, correspond to actionable evidence for population health stakeholders. However, little guidance exists to inform the selection of effect sizes for population health research proposals. We draw on examples of five rigorously evaluated population health interventions. These examples illustrate considerations for selecting realistic and actionable effect sizes as inputs to calculations of power, sample size and MDE for research proposals to study population health interventions. We show that plausible effects sizes for population health interventions may be smaller than commonly cited guidelines suggest. Effect sizes achieved with population health interventions depend on the characteristics of the intervention, the target population, and the outcomes studied. Population health impact depends on the proportion of the population receiving the intervention. When adequately powered, even studies of interventions with small effect sizes can offer valuable evidence to inform population health if such interventions can be implemented broadly. Demonstrating the effectiveness of such interventions, however, requires large sample sizes.

Language-Related Disparities in Pain Management in the Post-Anesthesia Care Unit for Children Undergoing Laparoscopic Appendectomy

Race and ethnicity are associated with disparities in pain management in children. While low English language proficiency is correlated with minority race/ethnicity in the United States, it is less frequently explored in the study of health disparities. We therefore investigated whether English language proficiency influenced pain management in the post-anesthesia care unit (PACU) in a cohort of children who underwent laparoscopic appendectomy at our pediatric hospital in San Francisco. Our primary exposure was English language proficiency, and our primary outcome was administration of any opioid medication in the PACU. Secondary outcomes included the amount of opioid administered in the PACU and whether any pain score was recorded during the patient's recovery period. Statistical analysis included adjusting for demographic covariates including race in estimating the effect of language proficiency on these outcomes. In our cohort of 257 pediatric patients, 57 (22.2%) had low English proficiency (LEP). While LEP and English proficient (EP) patients received the same amount of opioid medication intraoperatively, in multivariable analysis, LEP patients had more than double the odds of receiving any opioid in the PACU (OR 2.45, 95% CI 1.22-4.92). LEP patients received more oral morphine equivalents (OME) than EP patients (1.64 OME/kg, CI 0.67-3.84), and they also had almost double the odds of having no pain score recorded during their PACU recovery period (OR 1.93, CI 0.79-4.73), although the precision of these estimates was limited by small sample size. Subgroup analysis showed that children over the age of 5 years, who were presumably more verbal and would therefore undergo verbal pain assessments, had over triple the odds of having no recorded pain score (OR 3.23, CI 1.48-7.06). In summary, English language proficiency may affect the management of children's pain in the perioperative setting. The etiology of this language-related disparity is likely multifactorial and should be investigated further.

Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise

BACKGROUND

Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions.

METHODS

We use the Shannon transform of the P-value p, also known as the binary surprisal or S-value s = -log2(p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study.

CONCLUSIONS

In line with other recent recommendations, we advise that teaching materials and research reports discuss P-values as measures of compatibility rather than significance, compute P-values for alternative hypotheses whenever they are computed for null hypotheses, and interpret interval estimates as showing values of high compatibility with data, rather than regions of confidence. Our recommendations emphasize cognitive devices for displaying the compatibility of the observed data with various hypotheses of interest, rather than focusing on single hypothesis tests or interval estimates. We believe these simple reforms are well worth the minor effort they require.

Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing

The increasing pursuit of replicable research and actual replication of research is a political project that articulates a very specific technology of accountability for science. This project was initiated in response to concerns about the openness and trustworthiness of science. Though applicable and valuable in many fields, here we argue that this value cannot be extended everywhere, since the epistemic content of fields, as well as their accountability infrastructures, differ. Furthermore, we argue that there are limits to replicability across all fields; but in some fields, including parts of the humanities, these limits severely undermine the value of replication to account for the value of research.

A Future for Observational Epidemiology: Clarity, Credibility, Transparency

Observational studies are ambiguous, difficult, and necessary for epidemiology. Presently, there are concerns that the evidence produced by most observational studies in epidemiology is not credible and contributes to research waste. I argue that observational epidemiology could be improved by focusing greater attention on 1) defining questions that make clear whether the inferential goal is descriptive or causal; 2) greater utilization of quantitative bias analysis and alternative research designs that aim to decrease the strength of assumptions needed to estimate causal effects; and 3) promoting, experimenting with, and perhaps institutionalizing both reproducible research standards and replication studies to evaluate the fragility of study findings in epidemiology. Greater clarity, credibility, and transparency in observational epidemiology will help to provide reliable evidence that can serve as a basis for making decisions about clinical or population-health interventions.

Psychology's Replication Crisis and Clinical Psychological Science

Despite psychological scientists' increasing interest in replicability, open science, research transparency, and the improvement of methods and practices, the clinical psychology community has been slow to engage. This has been shifting more recently, and with this review, we hope to facilitate this emerging dialogue. We begin by examining some potential areas of weakness in clinical psychology in terms of methods, practices, and evidentiary base. We then discuss a select overview of solutions, tools, and current concerns of the reform movement from a clinical psychological science perspective. We examine areas of clinical science expertise (e.g., implementation science) that should be leveraged to inform open science and reform efforts. Finally, we reiterate the call to clinical psychologists to increase their efforts toward reform that can further improve the credibility of clinical psychological science.