The Importance of Making Assumptions in Bias Analysis

Use of sensitivity analyses to assess uncontrolled confounding from unmeasured variables in observational, active comparator pharmacoepidemiologic studies: a systematic review

Understanding the potential for, and direction and magnitude of uncontrolled confounding is critical for generating informative real-world evidence. Many sensitivity analyses are available to assess robustness of study results to residual confounding, but it is unclear how researchers are using these methods. We conducted a systematic review of published active-comparator cohort studies of drugs or biologics to summarize use of sensitivity analyses aimed at assessing uncontrolled confounding from an unmeasured variable. We reviewed articles in 5 medical and 7 epidemiologic journals published between January 1, 2017, and June 30, 2022. We identified 158 active-comparator cohort studies: 76 from medical and 82 from epidemiologic journals. Residual, unmeasured, or uncontrolled confounding was noted as a potential concern in 93% of studies, but only 84 (53%) implemented at least 1 sensitivity analysis to assess uncontrolled confounding from an unmeasured variable. The most common analyses were E-values among medical journal articles (21%) and restriction on measured variables among epidemiologic journal articles (22%). Researchers must rigorously consider the role of residual confounding in their analyses and the best sensitivity analyses for assessing this potential bias. This article is part of a Special Collection on Pharmacoepidemiology.

Vaccination against measles-mumps-rubella and rates of non-targeted infectious disease hospitalisations: Nationwide register-based cohort studies in Denmark, Finland, Norway, and Sweden

OBJECTIVES

To investigate if receipt of measles-mumps-rubella (MMR) vaccine following the third dose of diphtheria-tetanus-acellular pertussis (DTaP3) is associated with reduced rates of non-targeted infectious disease hospitalisations.

METHODS

Register based cohort study following 1,397,027 children born in Denmark, Finland, Norway, and Sweden until 2 years of age. Rates of infectious disease hospitalisations with minimum one overnight stay according to time-varying vaccination status were compared using Cox proportional hazards regression analysis with age as the underlying timescale and including multiple covariates. Summary estimates were calculated using random-effects meta-analysis.

RESULTS

Compared with DTaP3 and no MMR vaccine, MMR after DTaP3 was associated with reduced rates of infectious disease hospitalisations: aHR was 0.86 (0.83-0.89) in Denmark, 0.70 (0.64-0.75) in Finland, 0.71 (0.68-0.74) in Norway, and 0.71 (0.65-0.77) in Sweden: summary estimate was 0.75 (0.65 to 0.84). A beneficial association was also seen in a negative control exposure analysis (3 vs. 2 DTaP doses): summary estimate aHR was 0.81 (0.75-0.87).

CONCLUSIONS

Having MMR as the most recent vaccine was consistently associated with reduced rates of infectious disease hospitalisation. However, bias may account for at least some of the observed association. Randomised controlled trials are warranted to inform the optimal timing of MMR for both its specific and potential non-specific effects.

Quantitative bias analysis methods for summary-level epidemiologic data in the peer-reviewed literature: a systematic review

OBJECTIVES

Quantitative bias analysis (QBA) methods evaluate the impact of biases arising from systematic errors on observational study results. This systematic review aimed to summarize the range and characteristics of QBA methods for summary-level data published in the peer-reviewed literature.

STUDY DESIGN AND SETTING

We searched MEDLINE, Embase, Scopus, and Web of Science for English-language articles describing QBA methods. For each QBA method, we recorded key characteristics, including applicable study designs, bias(es) addressed; bias parameters, and publicly available software. The study protocol was preregistered on the Open Science Framework (https://osf.io/ue6vm/).

RESULTS

Our search identified 10,249 records, of which 53 were articles describing 57 QBA methods for summary-level data. Of the 57 QBA methods, 53 (93%) were explicitly designed for observational studies, and 4 (7%) for meta-analyses. There were 29 (51%) QBA methods that addressed unmeasured confounding, 19 (33%) misclassification bias, 6 (11%) selection bias, and 3 (5%) multiple biases. Thirty-eight (67%) QBA methods were designed to generate bias-adjusted effect estimates and 18 (32%) were designed to describe how bias could explain away observed findings. Twenty-two (39%) articles provided code or online tools to implement the QBA methods.

CONCLUSION

In this systematic review, we identified a total of 57 QBA methods for summary-level epidemiologic data published in the peer-reviewed literature. Future investigators can use this systematic review to identify different QBA methods for summary-level epidemiologic data.

Evaluation of quantitative bias analysis in epidemiological research: A systematic review from 2010 to mid-2023

OBJECTIVE

We aimed to demonstrate the use of quantitative bias analysis (QBA), which reveals the effects of systematic error, including confounding, misclassification and selection bias, on study results in epidemiological studies published in the period from 2010 to mid-23.

METHOD

The articles identified through a keyword search using Pubmed and Scopus were included in the study. The articles obtained from this search were eliminated according to the exclusion criteria, and the articles in which QBA analysis was applied were included in the detailed evaluation.

RESULTS

It can be said that the application of QBA analysis has gradually increased over the 13-year period. Accordingly, the number of articles in which simple is used as a method in QBA analysis is 9 (9.89%), the number of articles in which the multidimensional approach is used is 10 (10.99%), the number of articles in which the probabilistic approach is used is 60 (65.93%) and the number of articles in which the method is not specified is 12 (13.19%). The number of articles with misclassification bias model is 44 (48.35%), the number of articles with uncontrolled confounder(s) bias model is 32 (35.16%), the number of articles with selection bias model is 7 (7.69%) and the number of articles using more than one bias model is 8 (8.79%). Of the 49 (53.85%) articles in which the bias parameter source was specified, 19 (38.78%) used internal validation, 26 (53.06%) used external validation and 4 (8.16%) used educated guess, data constraints and hypothetical data. Probabilistic approach was used as a bias method in 60 (65.93%) of the articles, and mostly beta (8 [13.33%)], normal (9 [15.00%]) and uniform (8 [13.33%]) distributions were selected.

CONCLUSION

The application of QBA is rare in the literature but is increasing over time. Future researchers should include detailed analyzes such as QBA analysis to obtain inferences with higher evidence value, taking into account systematic errors.

Causal Inference About the Effects of Interventions From Observational Studies in Medical Journals

Importance

Many medical journals, including JAMA, restrict the use of causal language to the reporting of randomized clinical trials. Although well-conducted randomized clinical trials remain the preferred approach for answering causal questions, methods for observational studies have advanced such that causal interpretations of the results of well-conducted observational studies may be possible when strong assumptions hold. Furthermore, observational studies may be the only practical source of information for answering some questions about the causal effects of medical or policy interventions, can support the study of interventions in populations and settings that reflect practice, and can help identify interventions for further experimental investigation. Identifying opportunities for the appropriate use of causal language when describing observational studies is important for communication in medical journals.

Observations

A structured approach to whether and how causal language may be used when describing observational studies would enhance the communication of research goals, support the assessment of assumptions and design and analytic choices, and allow for more clear and accurate interpretation of results. Building on the extensive literature on causal inference across diverse disciplines, we suggest a framework for observational studies that aim to provide evidence about the causal effects of interventions based on 6 core questions: what is the causal question; what quantity would, if known, answer the causal question; what is the study design; what causal assumptions are being made; how can the observed data be used to answer the causal question in principle and in practice; and is a causal interpretation of the analyses tenable?

Conclusions and Relevance

Adoption of the proposed framework to identify when causal interpretation is appropriate in observational studies promises to facilitate better communication between authors, reviewers, editors, and readers. Practical implementation will require cooperation between editors, authors, and reviewers to operationalize the framework and evaluate its effect on the reporting of empirical research.

Is vaccination against measles, mumps, and rubella associated with reduced rates of antibiotic treatments among children below the age of 2 years? Nationwide register-based study from Denmark, Finland, Norway, and Sweden

OBJECTIVES

Previous studies have shown that vaccination against measles, mumps, and rubella (MMR) may have beneficial non-specific effects, reducing the risk of infections not targeted by the vaccine. We investigated if MMR vaccine given after the third dose of diphtheria-tetanus-acellular pertussis vaccine (DTaP3), was associated with reduced rates of antibiotic treatments.

METHODS

Register-based cohort study following children from the age of recommended MMR vaccination until age 2 years. We included 831,287 children born in Denmark, Finland, Norway, and Sweden who had received DTaP3 but not yet MMR vaccine. Cox proportional hazards regression with age as the underlying timescale and vaccination status as a time-varying exposure was used to estimate covariate-adjusted Hazard Ratios (aHRs) and inverse probability of treatment weighted (IPTW) HRs of antibiotic treatments. Summary estimates were calculated using random-effects meta-analysis.

RESULTS

Compared with only having received DTaP3, receipt of MMR vaccine after DTaP3 was associated with reduced rates of antibiotic treatments in all countries: the aHR was 0.92 (0.91-0.93) in Denmark, 0.92 (0.90-0.94) in Finland, 0.84 (0.82-0.85) in Norway, and 0.87 (0.85-0.90) in Sweden, yielding a summary estimate of 0.89 (0.85-0.93). A stronger beneficial association was seen in a negative control exposure analysis comparing children vaccinated with DTaP3 vs two doses of DTaP.

CONCLUSIONS

Across the Nordic countries, receipt of MMR vaccine after DTaP3 was associated with an 11% lower rate of antibiotic treatments. The negative control analysis suggests that the findings are affected by residual confounding. Findings suggest that potential non-specific effects of MMR vaccine are of limited clinical and public health importance for the milder infections treated out-of-hospital in the Nordic setting.

Association between trajectories of prescription opioid use and risk of opioid use disorder and overdose among US nonmetastatic breast cancer survivors

PURPOSE

To examine the association between prescription opioid use trajectories and risk of opioid use disorder (OUD) or overdose among nonmetastatic breast cancer survivors by treatment type.

METHODS

This retrospective cohort study included female nonmetastatic breast cancer survivors with at least 1 opioid prescription fill in 2010-2019 Surveillance, Epidemiology and End Results linked Medicare data. Opioid mean daily morphine milligram equivalents (MME) calculated within 1.5 years after initiating active breast cancer therapy. Group-based trajectory models identified distinct opioid use trajectory patterns. Risk of time to first OUD/overdose event within 1 year after the trajectory period was calculated for distinct trajectory groups using Cox proportional hazards models. Analyses were stratified by treatment type.

RESULTS

Four opioid use trajectories were identified for each treatment group. For 38,030 survivors with systemic endocrine therapy, 3 trajectories were associated with increased OUD/overdose risk compared with early discontinuation: minimal dose (< 5 MME; adjusted hazard ratio [aHR] = 1.73 [95% CI  1.43-2.09]), very low dose (5-25 MME; 2.67 [2.05-3.48]), and moderate dose (51-90 MME; 6.20 [4.69-8.19]). For 9477 survivors with adjuvant chemotherapy, low-dose opioid use was associated with higher OUD/overdose risk (aHR = 7.33 [95% CI 2.52-21.31]) compared with early discontinuation. For 3513 survivors with neoadjuvant chemotherapy, the differences in OUD/OD risks across the 4 trajectories were not significant.

CONCLUSIONS

Among Medicare nonmetastatic breast cancer survivors receiving systemic endocrine therapy or adjuvant chemotherapy, compared with early discontinuation, low-dose or moderate-dose opioid use were associated with six- to sevenfold higher OUD/overdose risk. Breast cancer survivors at high-risk of OUD/overdose may benefit from targeted interventions (e.g., pain clinic referral).

Sensitivity analysis for causality in observational studies for regulatory science

Objective

The United States Congress passed the 21st Century Cures Act mandating the development of Food and Drug Administration guidance on regulatory use of real-world evidence. The Forum on the Integration of Observational and Randomized Data conducted a meeting with various stakeholder groups to build consensus around best practices for the use of real-world data (RWD) to support regulatory science. Our companion paper describes in detail the context and discussion of the meeting, which includes a recommendation to use a causal roadmap for study designs using RWD. This article discusses one step of the roadmap: the specification of a sensitivity analysis for testing robustness to violations of causal model assumptions.

Methods

We present an example of a sensitivity analysis from a RWD study on the effectiveness of Nifurtimox in treating Chagas disease, and an overview of various methods, emphasizing practical considerations on their use for regulatory purposes.

Results

Sensitivity analyses must be accompanied by careful design of other aspects of the causal roadmap. Their prespecification is crucial to avoid wrong conclusions due to researcher degrees of freedom. Sensitivity analysis methods require auxiliary information to produce meaningful conclusions; it is important that they have at least two properties: the validity of the conclusions does not rely on unverifiable assumptions, and the auxiliary information required by the method is learnable from the corpus of current scientific knowledge.

Conclusions

Prespecified and assumption-lean sensitivity analyses are a crucial tool that can strengthen the validity and trustworthiness of effectiveness conclusions for regulatory science.

Guidelines for Reporting Observational Research in Urology: The Importance of Clear Reference to Causality

Observational studies often dance around the issue of causality. We propose guidelines to ensure that papers refer to whether or not the study aim is to investigate causality, and suggest language to use and language to avoid.

Use of antihypertensive drugs and risk of cutaneous melanoma: a nationwide nested case-control study

BACKGROUND

Most antihypertensives can induce dermal photosensitivity, which may increase melanoma risk. However, corroborating evidence is limited. We examined the associations between use of antihypertensives and melanoma risk.

METHODS

A nationwide nested case-control study was conducted using data from the Cancer Registry of Norway, the National Registry and the Norwegian Prescription Database in 2004-15. Ten controls were randomly selected for each melanoma case, matched on sex and birth year. The study included 12 048 cases and 117 895 controls. We estimated rate ratios (RRs) with 95% confidence intervals (CIs). All analyses were adjusted for ambient ultraviolet radiation (UVR). We additionally performed active comparator analyses, and sensitivity analyses by only including new users, distinguishing between exclusive and mixed users, allowing for different latency periods, and subgroup analyses by melanoma subtype and clinical stage.

RESULTS

Compared with non-use, we observed a slightly increased melanoma risk in users of diuretics (RR 1.08, CI 1.01-1.15), calcium-channel blockers (RR 1.10, CI 1.04-1.18) and drugs affecting the renin-angiotensin system (RR 1.10, CI 1.04-1.16), but not for beta blockers (RR 0.97, CI 0.92-1.03). We found no heterogeneity of associations by melanoma subtype or clinical stage and no dose-response relationship between the cumulative defined daily doses (DDDs) and melanoma. No interaction was found between cumulative DDDs and ambient UVR.

CONCLUSIONS

Weak associations, with lack of a dose-response relationship and lack of interactions with ambient UVR, in the DDD analysis in this nationwide study do not support a causal relationship between antihypertensives and melanoma risk.

Quantitative bias analysis in practice: review of software for regression with unmeasured confounding

BACKGROUND

Failure to appropriately account for unmeasured confounding may lead to erroneous conclusions. Quantitative bias analysis (QBA) can be used to quantify the potential impact of unmeasured confounding or how much unmeasured confounding would be needed to change a study's conclusions. Currently, QBA methods are not routinely implemented, partly due to a lack of knowledge about accessible software. Also, comparisons of QBA methods have focused on analyses with a binary outcome.

METHODS

We conducted a systematic review of the latest developments in QBA software published between 2011 and 2021. Our inclusion criteria were software that did not require adaption (i.e., code changes) before application, was still available in 2022, and accompanied by documentation. Key properties of each software tool were identified. We provide a detailed description of programs applicable for a linear regression analysis, illustrate their application using two data examples and provide code to assist researchers in future use of these programs.

RESULTS

Our review identified 21 programs with [Formula: see text] created post 2016. All are implementations of a deterministic QBA with [Formula: see text] available in the free software R. There are programs applicable when the analysis of interest is a regression of binary, continuous or survival outcomes, and for matched and mediation analyses. We identified five programs implementing differing QBAs for a continuous outcome: treatSens, causalsens, sensemakr, EValue, and konfound. When applied to one of our illustrative examples, causalsens incorrectly indicated sensitivity to unmeasured confounding whereas the other four programs indicated robustness. sensemakr performs the most detailed QBA and includes a benchmarking feature for multiple unmeasured confounders.

CONCLUSIONS

Software is now available to implement a QBA for a range of different analyses. However, the diversity of methods, even for the same analysis of interest, presents challenges to their widespread uptake. Provision of detailed QBA guidelines would be highly beneficial.

Sensitivity Analyses for Unmeasured Confounders

Purpose of Review

This review expands on sensitivity analyses for unmeasured confounding techniques, demonstrating state-of-the-art methods as well as specifying which should be used under various scenarios, depending on the information about a potential unmeasured confounder available to the researcher.

Recent Findings

Methods to assess how sensitive an observed estimate is to unmeasured confounding have been developed for decades. Recent advancements have allowed for the incorporation of measured confounders in these assessments, updating the methods used to quantify the impact of an unmeasured confounder, whether specified in terms of the magnitude of the effect from a regression standpoint, for example, as a risk ratio, or with respect to the percent of variation in the outcome or exposure explained by the unmeasured confounder. Additionally, single number summaries, such as the E-value or robustness value, have been proposed to allow for ease of computation when less is known about a specific potential unmeasured confounder.

Summary

This paper aimed to provide methods and tools to implement sensitivity to unmeasured confounder analyses appropriate for various research settings depending on what is known or assumed about a potential unmeasured confounder. We have provided mathematical justification, recommendations, as well as R code to ease the implementation of these methods.

Serious adverse events of special interest following mRNA COVID-19 vaccination in randomized trials in adults

INTRODUCTION

In 2020, prior to COVID-19 vaccine rollout, the Brighton Collaboration created a priority list, endorsed by the World Health Organization, of potential adverse events relevant to COVID-19 vaccines. We adapted the Brighton Collaboration list to evaluate serious adverse events of special interest observed in mRNA COVID-19 vaccine trials.

METHODS

Secondary analysis of serious adverse events reported in the placebo-controlled, phase III randomized clinical trials of Pfizer and Moderna mRNA COVID-19 vaccines in adults (NCT04368728 and NCT04470427), focusing analysis on Brighton Collaboration adverse events of special interest.

RESULTS

Pfizer and Moderna mRNA COVID-19 vaccines were associated with an excess risk of serious adverse events of special interest of 10.1 and 15.1 per 10,000 vaccinated over placebo baselines of 17.6 and 42.2 (95 % CI -0.4 to 20.6 and -3.6 to 33.8), respectively. Combined, the mRNA vaccines were associated with an excess risk of serious adverse events of special interest of 12.5 per 10,000 vaccinated (95 % CI 2.1 to 22.9); risk ratio 1.43 (95 % CI 1.07 to 1.92). The Pfizer trial exhibited a 36 % higher risk of serious adverse events in the vaccine group; risk difference 18.0 per 10,000 vaccinated (95 % CI 1.2 to 34.9); risk ratio 1.36 (95 % CI 1.02 to 1.83). The Moderna trial exhibited a 6 % higher risk of serious adverse events in the vaccine group: risk difference 7.1 per 10,000 (95 % CI -23.2 to 37.4); risk ratio 1.06 (95 % CI 0.84 to 1.33). Combined, there was a 16 % higher risk of serious adverse events in mRNA vaccine recipients: risk difference 13.2 (95 % CI -3.2 to 29.6); risk ratio 1.16 (95 % CI 0.97 to 1.39).

DISCUSSION

The excess risk of serious adverse events found in our study points to the need for formal harm-benefit analyses, particularly those that are stratified according to risk of serious COVID-19 outcomes. These analyses will require public release of participant level datasets.

Smoking During Pregnancy and Risk of Attention-deficit/Hyperactivity Disorder in the Third Generation

BACKGROUND

Animal experiments indicate that environmental factors, such as cigarette smoke, can have multigenerational effects through the germline. However, there are little data on multigenerational effects of smoking in humans. We examined the associations between grandmothers' smoking while pregnant and risk of attention-deficit/hyperactivity disorder (ADHD) in her grandchildren.

METHODS

Our study population included 53,653 Nurses' Health Study II (NHS-II) participants (generation 1 [G1]), their mothers (generation 0 [G0]), and their 120,467 live-born children (generation 2 [G2]). In secondary analyses, we used data from 23,844 mothers of the nurses who were participants in the Nurses' Mothers' Cohort Study (NMCS), a substudy of NHS-II.

RESULTS

The prevalence of G0 smoking during the pregnancy with the G1 nurse was 25%. ADHD was diagnosed in 9,049 (7.5%) of the grandchildren (G2). Grand-maternal smoking during pregnancy was associated with increased odds of ADHD among the grandchildren (adjusted odds ratio [aOR] = 1.2; 95% confidence interval [CI] = 1.1, 1.2), independent of G1 smoking during pregnancy. In the Nurses' Mothers' Cohort Study, odds of ADHD increased with increasing cigarettes smoked per day by the grandmother (1-14 cigarettes: aOR = 1.1; 95% CI = 1.0, 1.2; 15+: aOR = 1.2; 95% CI = 1.0, 1.3), compared with nonsmoking grandmothers.

CONCLUSIONS

Grandmother smoking during pregnancy is associated with an increased risk of ADHD among the grandchildren.

Receipt of Early Intervention Services Before Age 3 Years and Performance on Third-Grade Standardized Tests Among Children Exposed to Lead

IMPORTANCE

Research has shown that early intervention programs can improve academic outcomes of children with developmental delays. It has been suggested that similar programs may combat the deleterious effects of lead on children's neurodevelopment. However, to our knowledge, there are no published studies examining this possibility.

OBJECTIVE

The objective of this study was to estimate the association between receipt of early intervention services and third-grade standardized test scores among children exposed to lead before age 3 years.

DESIGN, SETTING, AND PARTICIPANTS

Cohort study including children born in New York City, New York, from 1994 to 1998 within an administrative data linkage of birth, lead monitoring, early intervention, and education data systems. Participants had a blood lead level of 4 μg/dL or greater at any point before age 3 years and later attended public school in New York City.

EXPOSURES

Any use of early intervention services from birth through age 3 years.

MAIN OUTCOMES AND MEASURES

Children who did or did not receive early intervention services were matched using propensity scores. Linear and log-binomial regression were used to estimate the association between receipt of early intervention services before age 3 years and standardized test scores in math and English-language arts in third grade.

RESULTS

There were 2986 children exposed to lead who received early intervention services before age 36 months. Of these children, 2757 were propensity score-matched to 8160 children who did not receive services. Children who received early intervention services did 7% (95% CI, 3%-12%) of an SD better on math and 10% (95% CI, 5%-14%) of an SD better on English-language arts tests than children who did not receive services. In addition, children who received services were 14% (95% CI, 9%-19%) and 16% (95% CI, 9%-23%) more likely to meet test-based standards in math and English-language arts, respectively, than children who did not receive services. These associations became larger in magnitude when analyses were restricted to children with higher blood lead levels.

CONCLUSIONS AND RELEVANCE

By leveraging existing public health data, this study found evidence that receipt of early intervention services may benefit the academic performance of children exposed to lead early in life.