Generalizability of findings from randomized controlled trials: application to the National Institute of Drug Abuse Clinical Trials Network

Systematic review of applied transportability and generalizability analyses: A landscape analysis

Transportability and generalizability analysis are novel causal inference methods that quantitatively assess external validity. Currently, it is unclear how these analyses are applied in practice. To characterize applications and methods, we conducted a landscape analysis of applied transportability and generalizability analyses using a systematic literature search of PubMed, CINAHL and Embase supplemented with hand-searches. We identified 68 publications describing transportability and generalizability analyses conducted with 83 unique source-target dataset pairs and reporting 99 distinct analyses. The majority of source and target datasets were collected in the US (n = 63/83, 75.9 %; and n = 59/83, 71.1 %, respectively). These methods were most often applied to transport RCT findings to observational studies (n = 38/83; 45.8 %), or to another RCT (n = 20/83; 24.1 %). Several studies used transportability analysis outside the standard application, for example to identify effect modifiers or calibrate measurements within an RCT. Methods that used weights and individual-level patient data were most common (n = 56/99, 56.5 %; n = 80/83, 96.4 %, respectively). Reporting quality varied across studies. Transportability analysis has a wide range of applications including supporting decision-making by improving evidence relevance and improving trial design by identifying contextual effect modifiers and calibrating outcome measurements. Efforts are needed to standardize analysis and reporting of these methods to improve transparency and uptake.

Emulation of ARISTOTLE and ROCKET AF trials in real-world atrial fibrillation patients results in similar efficacy and safety as original landmark trials: insights from the GARFIELD-AF registry

AIMS

This study aimed to determine the robustness, reproducibility and representativeness of the landmark Apixaban for Reduction in Stroke and Other Thromboembolic Events in Atrial Fibrillation (AF) (ARISTOTLE) and Rivaroxaban Once Daily Oral Direct Factor Xa Inhibition Compared with Vitamin K Antagonism for Prevention of Stroke and Embolism Trial in AF (ROCKET AF) randomised trials through replication in an observational AF patient registry.

METHODS AND RESULTS

Patients from the Global Anticoagulant Registry in the FIELD (GARFIELD)-AF registry treated with apixaban, rivaroxaban or vitamin K antagonist (VKA) were assessed for eligibility for the ARISTOTLE and ROCKET AF trials. HRs of apixaban and rivaroxaban versus comparator for stroke/systemic embolism, major bleeding and all-cause mortality within 2 years follow-up were calculated using propensity score overlap-weighted Cox models. Among GARFIELD-AF patients on apixaban, 2570/3615 (71%) would have been eligible for ARISTOTLE. Among patients using rivaroxaban, 2005/4914 (41%) would have been eligible for ROCKET AF. Eligibility rates were steady over time, with minor differences across medical specialties. Real-world AF patients selected according to trial criteria had lower cardiovascular burden than the original trial participants, especially compared with ROCKET AF. HRs (95% CI) for apixaban versus VKA among ARISTOTLE-eligible users were 0.57 (0.34 to 0.94) for stroke/systemic embolism, 0.76 (0.48 to 1.20) for major bleeding and 0.89 (0.70 to 1.12) for all-cause mortality. Among ROCKET AF-eligible rivaroxaban users, HRs for rivaroxaban versus VKA were 0.90 (0.57 to 1.43), 0.92 (0.59 to 1.43) and 0.86 (0.69 to 1.08), respectively. All safety and efficacy estimates were similar to those in the original trials.

CONCLUSION

Real-world representativeness of the selection criteria was greater for ARISTOTLE than ROCKET AF. The pivotal randomised trials of apixaban and rivaroxaban versus warfarin can be successfully emulated in real-world AF patients by applying trial-specific selection criteria and appropriate methodology for non-randomised treatment allocation.

TRIAL REGISTRATION NUMBER

NCT01090362.

Equitable Representation of Pregnant and Lactating Women in Clinical Research: A Historical Review and Critical Analysis of Proposed Legislation

A long history of policymaking and regulation constructed for the purpose of ensuring adequate fetal and infant protections has inadvertently sanctioned the widespread exclusion of pregnant and lactating patients from biomedical research, leaving a paucity of high quality data necessary for clinical decision-making. Although well-intended, the regulatory classification of pregnant women as "vulnerable," in conjunction with burdensome enrollment criteria and other factors weighing against broad inclusion, have ultimately placed the health and safety of these women and their babies in jeopardy. Robust measures are urgently needed to overcome patient and physician reluctance, address substantial evidence gaps, and rectify long-standing disparities which precipitate disproportionately poor health outcomes among this population. In February 2023, the Advancing Safe Medications for Moms and Babies Act of 2023 (the Act) was introduced in the United States House of Representatives with the overarching goal of enabling pregnant and lactating women to achieve equitable participation in clinical research and contribute to developing important biomedical knowledge to guide and improve health care delivered to these patients. This review discusses the historical influence of federal human subject protection regulations on the health and clinical treatment of pregnant and lactating women, outlines and critically analyzes the provisions incorporated into the Act, and reflects on the potential long-term impact the Act would have should it be successful in becoming law. KEY POINTS: · Pregnant and lactating patients are widely excluded from clinical research.. · Evidence guiding the treatment of these patients is limited and of unacceptably low quality.. · Proposed legislation seeks to rectify disparities and empower these patients through improved representation in research..

Predictors of retention in heroin-assisted treatment in Denmark 2010-2018 - A record-linkage study

INTRODUCTION

While randomized-controlled trials have shown that heroin-assisted treatment (HAT) is superior to methadone maintenance alone in treatment of refractory clients, little is known about client factors associated with retention in HAT in routine care.

METHODS

This retrospective cohort study assessed predictors of retention in first treatment episode among a consecutive cohort of clients admitted to HAT in Denmark from 2010 to 2018, who could be matched to the Danish population register and for whom a Short Form Health Survey (SF-36) was available at admission (N = 432). The study derived predictors from client self-reports at intake and administrative data available in national registers. Cox proportional hazards regression modelled retention in treatment.

RESULTS

The one-year retention rate was 69.63 % (95 % CI 65.06 %-73.74 %), and the median time in treatment was 2.45 years (95 % CI, 1.83-3.12). Bivariate analyses showed that retention was lower for clients who had recent cocaine or benzodiazepine use and among those who had experienced an overdose in the year prior to enrollment in HAT. Age below 40, recent illegal activity, poorer emotional wellbeing, previous residential treatment experience, and previous intensive outpatient treatment were also predictors of dropout from HAT.

CONCLUSIONS

This observational study found that retention in HAT in routine care was similar to rates observed in randomized-controlled trials conducted in other countries. The results suggest that addressing polysubstance use as part of the HAT program may promote long-term retention, as may directing resources to certain subgroups identified at intake, including clients under 40 years and those who report recent criminal activity, emotional problems, or overdoses. The findings that previous residential treatment and intensive outpatient treatment were associated with dropout were unexpected.

The Effect of Tirzepatide on Body Composition in People with Overweight and Obesity: A Systematic Review of Randomized, Controlled Studies

Tirzepatide (TZP) is a new anti-obesity drug, and little is currently known about its effect on body composition (BC) in people with overweight or obesity. The aim of this study is to conduct a systematic review on the impact of TZP on BC compartments in this population during weight loss programs. Literature searches, study selection, method development, and quality appraisal were performed. The data were synthesized using a narrative approach, in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Of the 1379 papers retrieved, 6 randomized controlled trials met the inclusion criteria and were reviewed, revealing the following findings. Firstly, TZP was shown to result in a significant reduction in total fat mass (FM), visceral adipose tissue (VAT) and waist circumference (WC) between baseline and short as well as intermediate follow-ups. Compared to other anti-obesity medications (e.g., dulaglutide and semaglutide) taken over the same duration, TZP showed a superior decrease in body fat compartments (i.e., total FM, VAT and WC). Finally, the effect of TZP on fat-free mass (FFM) is still uncertain because the findings remain inconclusive. In conclusion, TZP appears to be an effective strategy for achieving significant improvements in body fat and its distribution, but additional investigations are still needed to determine the impact of TZP on lean mass in this population.

Feasibility of using real-world data to emulate substance use disorder clinical trials: a cross-sectional study

INTRODUCTION

Real-world evidence is receiving considerable attention as a way to evaluate the efficacy and safety of medical products for substance use disorders (SUDs). However, the feasibility of using real-world data (RWD) to emulate clinical trials evaluating treatments for SUDs is uncertain. The aim of this study is to identify the number of clinical trials evaluating treatments for SUDs with reported results that could be feasibly emulated using observational data from contemporary insurance claims and/or electronic health record (EHR) data.

METHODS

In this cross-sectional study, all phase 2-4 trials evaluating treatments for SUDs registered on ClinicalTrials.gov with reported results were identified. Each trial was evaluated to determine if the indications, interventions, at least 80% of eligibility criteria, comparators, and primary end points could be ascertained using contemporarily available administrative claims and/or structured EHR data.

RESULTS

There were 272 SUD trials on ClinicalTrials.gov with reported results. Of these, when examining feasibility using contemporarily available administrative claims and/or structured EHR data, 262 (96.3%) had indications that were ascertainable; 194 (71.3%) had interventions that were ascertainable; 21 (7.7%) had at least 80% of eligibility criteria that were ascertainable; 17 (6.3%) had active comparators that were ascertainable; and 61 (22.4%) had primary end points that were ascertainable. In total, there were no trials for which all 5 characteristics were ascertainable using contemporarily available administrative claims and/or structured EHR data. When considering placebo comparators as ascertainable, there were 6 (2.2%) trials that had all 5 key characteristics classified as ascertainable from contemporarily available administrative claims and/or structured EHR data.

CONCLUSIONS

No trials evaluating treatments for SUDs could be feasibly emulated using contemporarily available RWD, demonstrating a need for an increase in the resolution of data capture within a public health system to facilitate trial emulation.

Eligibility for Substance Use Clinical Trials Among Emergency Psychiatry Patients: The Impact of Exclusion Criteria

Objective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems.

Designing and testing treatments for alcohol use disorder

This chapter provides a succinct overview of several recommendations for the design and analysis of treatments for AUD with a specific focus on increasing rigor and generalizability of treatment studies in order to increase the reach of AUD treatment. We recommend that researchers always register their trials in a clinical trial registry and make the protocol accessible so that the trial can be replicated in future work, follow CONSORT reporting guidelines when reporting the results of the trial, carefully describe all inclusion and exclusion criteria as well as the randomization scheme, and always use an intent to treat design with attention to analysis of missing data. In addition, we recommend that researchers pay closer attention to recruitment and engagement strategies that increase enrollment and retention of historically marginalized and understudied populations, and we end with a plea for more consideration of implementation science approaches to increase the dissemination and implementation of AUD treatment in real-world settings.

Assessing the representativeness of cluster randomized trials: Evidence from two large pragmatic trials in United States nursing homes

BACKGROUND/AIMS

When the randomized clusters in a cluster randomized trial are selected based on characteristics that influence treatment effectiveness, results from the trial may not be directly applicable to the target population. We used data from two large nursing home-based pragmatic cluster randomized trials to compare nursing home and resident characteristics in randomized facilities to eligible non-randomized and ineligible facilities.

METHODS

We linked data from the high-dose influenza vaccine trial and the Music & Memory Pragmatic TRIal for Nursing Home Residents with ALzheimer's Disease (METRICaL) to nursing home assessments and Medicare fee-for-service claims. The target population for the high-dose trial comprised Medicare-certified nursing homes; the target population for the METRICaL trial comprised nursing homes in one of four US-based nursing home chains. We used standardized mean differences to compare facility and individual characteristics across the three groups and logistic regression to model the probability of nursing home trial participation.

RESULTS

In the high-dose trial, 4476 (29%) of the 15,502 nursing homes in the target population were eligible for the trial, of which 818 (18%) were randomized. Of the 1,361,122 residents, 91,179 (6.7%) were residents of randomized facilities, 463,703 (34.0%) of eligible non-randomized facilities, and 806,205 (59.3%) of ineligible facilities. In the METRICaL trial, 160 (59%) of the 270 nursing homes in the target population were eligible for the trial, of which 80 (50%) were randomized. Of the 20,262 residents, 973 (34.4%) were residents of randomized facilities, 7431 (36.7%) of eligible non-randomized facilities, and 5858 (28.9%) of ineligible facilities. In the high-dose trial, randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (132.5 vs 145.9 and 91.9, respectively), for-profit status (91.8% vs 66.8% and 68.8%), belonging to a nursing home chain (85.8% vs 49.9% and 54.7%), and presence of a special care unit (19.8% vs 25.9% and 14.4%). In the METRICaL trial randomized facilities differed from eligible non-randomized and ineligible facilities by the number of beds (103.7 vs 110.5 and 67.0), resource-poor status (4.6% vs 10.0% and 18.8%), and presence of a special care unit (26.3% vs 33.8% and 10.9%). In both trials, the characteristics of residents in randomized facilities were similar across the three groups.

CONCLUSION

In both trials, facility-level characteristics of randomized nursing homes differed considerably from those of eligible non-randomized and ineligible facilities, while there was little difference in resident-level characteristics across the three groups. Investigators should assess the characteristics of clusters that participate in cluster randomized trials, not just the individuals within the clusters, when examining the applicability of trial results beyond participating clusters.

Addressing Systematic Missing Data in the Context of Causally Interpretable Meta-analysis

Evidence synthesis involves drawing conclusions from trial samples that may differ from the target population of interest, and there is often heterogeneity among trials in sample characteristics, treatment implementation, study design, and assessment of covariates. Stitching together this patchwork of evidence requires subject-matter knowledge, a clearly defined target population, and guidance on how to weigh evidence from different trials. Transportability analysis has provided formal identifiability conditions required to make unbiased causal inference in the target population. In this manuscript, we review these conditions along with an additional assumption required to address systematic missing data. The identifiability conditions highlight the importance of accounting for differences in treatment effect modifiers between the populations underlying the trials and the target population. We perform simulations to evaluate the bias of conventional random effect models and multiply imputed estimates using the pooled trials sample and describe causal estimators that explicitly address trial-to-target differences in key covariates in the context of systematic missing data. Results indicate that the causal transportability estimators are unbiased when treatment effect modifiers are accounted for in the analyses. Results also highlight the importance of carefully evaluating identifiability conditions for each trial to reduce bias due to differences in participant characteristics between trials and the target population. Bias can be limited by adjusting for covariates that are strongly correlated with missing treatment effect modifiers, including data from trials that do not differ from the target on treatment modifiers, and removing trials that do differ from the target and did not assess a modifier.

Calibrated meta-analysis to estimate the efficacy of mental health treatments in target populations: an application to paliperidone trials for treatment of schizophrenia

BACKGROUNDS

Meta-analyses can be a powerful tool but need to calibrate potential unrepresentativeness of the included trials to a target population. Estimating target population average treatment effects (TATE) in meta-analyses is important to understand how treatments perform in well-defined target populations. This study estimated TATE of paliperidone palmitate in patients with schizophrenia using meta-analysis with individual patient trial data and target population data.

METHODS

We conducted a meta-analysis with data from four randomized clinical trials and target population data from the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) study. Efficacy was measured using the Positive and Negative Syndrome Scale (PANSS). Weights to equate the trial participants and target population were calculated by comparing baseline characteristics between the trials and CATIE. A calibrated weighted meta-analysis with random effects was performed to estimate the TATE of paliperidone compared to placebo.

RESULTS

A total of 1,738 patients were included in the meta-analysis along with 1,458 patients in CATIE. After weighting, the covariate distributions of the trial participants and target population were similar. Compared to placebo, paliperidone palmitate was associated with a significant reduction of the PANSS total score under both unweighted (mean difference 9.07 [4.43, 13.71]) and calibrated weighted (mean difference 6.15 [2.22, 10.08]) meta-analysis.

CONCLUSIONS

The effect of paliperidone palmitate compared with placebo is slightly smaller in the target population than that estimated directly from the unweighted meta-analysis. Representativeness of samples of trials included in a meta-analysis to a target population should be assessed and incorporated properly to obtain the most reliable evidence of treatment effects in target populations.

Transporting monovalent rotavirus vaccine efficacy estimates to an external target population: a secondary analysis of data from a randomised controlled trial in Malawi

Oral rotavirus vaccine efficacy estimates from randomised controlled trials are highly variable across settings. Although the randomised study design increases the likelihood of internal validity of findings, results from trials may not always apply outside the context of the study due to differences between trial participants and the target population. Here, we used a weight-based method to transport results from a monovalent rotavirus vaccine clinical trial conducted in Malawi between 2005 and 2008 to a target population of all trial-eligible children in Malawi, represented by data from the 2015-2016 Malawi Demographic and Health Survey (DHS). We reweighted trial participants to reflect the population characteristics described by the Malawi DHS. Vaccine efficacy was estimated for 1008 trial participants after applying these weights such that they represented trial-eligible children in Malawi. We also conducted subgroup analyses to examine the heterogeneous treatment effects by stunting and tuberculosis vaccination status at enrolment. In the original trial, the estimates of one-year vaccine efficacy against severe rotavirus gastroenteritis and any-severity rotavirus gastroenteritis in Malawi were 49.2% (95% CI 15.6%-70.3%) and 32.1% (95% CI 2.5%-53.1%), respectively. After weighting trial participants to represent all trial-eligible children in Malawi, vaccine efficacy increased to 62.2% (95% CI 35.5%-79.0%) against severe rotavirus gastroenteritis and 38.9% (95% CI 11.4%-58.5%) against any-severity rotavirus gastroenteritis. Rotavirus vaccine efficacy may differ between trial participants and target populations when these two populations differ. Differences in tuberculosis vaccination status between the trial sample and DHS population contributed to varying trial and target population vaccine efficacy estimates.

Non-linear Relationship of Maternal Age With Risk of Spontaneous Abortion: A Case-Control Study in the China Birth Cohort

Background

Spontaneous abortion is one of the prevalent adverse reproductive outcomes, which seriously threatens maternal health around the world.

Objective

The current study is aimed to evaluate the association between maternal age and risk for spontaneous abortion among pregnant women in China.

Methods

This was a case-control study based on the China Birth Cohort, we compared 338 cases ending in spontaneous abortion with 1,352 controls resulting in normal live births. The main exposure indicator and outcome indicator were maternal age and spontaneous abortion, respectively. We used both a generalized additive model and a two-piece-wise linear model to determine the association. We further performed stratified analyses to test the robustness of the association between maternal age and spontaneous abortion in different subgroups.

Results

We observed a J-shaped relationship between maternal age and spontaneous abortion risk, after adjusting for multiple covariates. Further, we found that the optimal threshold age was 29.68 years old. The adjusted odds ratio (95% confidence interval) of spontaneous abortion per 1 year increase in maternal age were 0.97 (0.90–1.06) on the left side of the turning point and 1.25 (1.28–1.31) on the right side. Additionally, none of the covariates studied modified the association between maternal age and spontaneous abortion (P > 0.05).

Conclusions

Advanced maternal age (>30 years old) was significantly associated with increased prevalence of spontaneous abortion, supporting a J-shaped association between maternal age and spontaneous abortion.

Identification of tools used to assess the external validity of randomized controlled trials in reviews: a systematic review of measurement properties

Background

Internal and external validity are the most relevant components when critically appraising randomized controlled trials (RCTs) for systematic reviews. However, there is no gold standard to assess external validity. This might be related to the heterogeneity of the terminology as well as to unclear evidence of the measurement properties of available tools. The aim of this review was to identify tools to assess the external validity of RCTs. It was further, to evaluate the quality of identified tools and to recommend the use of individual tools to assess the external validity of RCTs in future systematic reviews.

Methods

A two-phase systematic literature search was performed in four databases: PubMed, Scopus, PsycINFO via OVID, and CINAHL via EBSCO. First, tools to assess the external validity of RCTs were identified. Second, studies investigating the measurement properties of these tools were selected. The measurement properties of each included tool were appraised using an adapted version of the COnsensus based Standards for the selection of health Measurement INstruments (COSMIN) guidelines.

Results

38 publications reporting on the development or validation of 28 included tools were included. For 61% (17/28) of the included tools, there was no evidence for measurement properties. For the remaining tools, reliability was the most frequently assessed property. Reliability was judged as “sufficient” for three tools (very low certainty of evidence). Content validity was rated as “sufficient” for one tool (moderate certainty of evidence).

Conclusions

Based on these results, no available tool can be fully recommended to assess the external validity of RCTs in systematic reviews. Several steps are required to overcome the identified difficulties to either adapt and validate available tools or to develop a better suitable tool.

Trial registration

Prospective registration at Open Science Framework (OSF): 10.17605/OSF.IO/PTG4D.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12874-022-01561-5.

Efficacy of vonoprazan against bleeding from endoscopic submucosal dissection-induced gastric ulcers under antithrombotic medication: A cross-design synthesis of randomized and observational studies

Vonoprazan, a potassium-competitive acid blocker, is expected to be superior to proton pump inhibitors (PPIs) in preventing post-endoscopic submucosal dissection (ESD)-induced gastric bleeding. However, the results of randomized controlled trials (RCTs) and observational studies on the efficacy of vonoprazan have been inconsistent. This study aimed to evaluate the effectiveness of vonoprazan in antithrombotic drug users, a population that has been excluded from RCTs. Treatment effects were assessed using cross-design synthesis, which can be adjusted for differences in study design and patient characteristics. We used data from an RCT in Japan (70 patients in the vonoprazan group and 69 in the PPI group) and an observational study (408 patients in the vonoprazan group and 870 in the PPI group). After matching, among the antithrombotic drug users in the observational study, post-ESD bleeding was noted in 8 out of 86 patients in the vonoprazan group and 18 out of 86 patients in the PPI group. After pooling the data from the RCT and observational study, the risk difference for antithrombotic drug users was -14.6% (95% CI: -22.0 to -7.2). CDS analysis suggested that vonoprazan is more effective than PPIs in preventing post-ESD bleeding among patients administered antithrombotic medications.

Practical Research Ethics in Psychiatric Clinical Trials: A Guide for Investigators

The conduct of clinical psychiatric research is critical to advance the science and efficacy of treatment while also safeguarding the interests of participants. This article emerges from the authors' experience, providing practical guidance to colleagues seeking input on how to design and implement clinical research protocols in accordance with key ethical considerations. Thus, the intent of this article is to provide (1) an overview of common ethical considerations when conducting psychiatric clinical research along with (2) practical advice for preparing Institutional Review Board applications and associated materials in the ethical conduct of psychiatric clinical research.

Adjunct interventions to standard medical management of buprenorphine in outpatient settings: A systematic review of the evidence

BACKGROUND

A growing body of research has examined adjunctive interventions supportive of engagement and retention in treatment among patients receiving buprenorphine for opioid use disorder (OUD). We conducted a systematic review of the literature addressing the effect on key outcomes of adjunctive interventions provided alongside standard medical management of buprenorphine in outpatient settings.

METHODS

We included prospective studies examining adults receiving buprenorphine paired with an adjunctive intervention for the treatment of OUD in an outpatient setting. Data sources included Medline, Cochrane Central Register of Controlled Trials, CINAHL and PsycINFO from inception through January 2020. Two raters independently reviewed full-text articles, abstracted data and appraised risk of bias. Outcomes examined included abstinence, retention in treatment and non-addiction-related health outcomes.

RESULTS

The final review includes 20 manuscripts, 11 randomized control trials (RCTs), three secondary analyses of RCTs and six observational studies. Most studies examined psychosocial interventions (n = 14). Few examined complementary therapies (e.g., yoga; n = 2) or technological interventions (e.g., electronic pill dispensation; n = 3); one study examined an intervention addressing structural barriers to care (patient navigators; n = 1). Low risk of bias RCTs found no evidence that adding psychosocial interventions to buprenorphine treatment improves substance use outcomes.

CONCLUSIONS

Research is needed to identify adjunctive interventions with potential to support medication adherence and addiction-related outcomes for patients engaged in buprenorphine treatment. Data from clinical trials suggest that lack of ready access to psychosocial treatments should not discourage clinicians from prescribing buprenorphine.

Development and implementation of standardized study performance metrics for a VA healthcare system clinical research consortium

The cost of conducting clinical trials is continuously increasing and is driven in large part by the time and resources required to activate trials and reach accrual targets. The impact of low enrollment in a clinical trial can negatively affect the validity of study results and delay its generalizability to the broader population. Quality is a multidimensional concept which could relate to the design, conduct, and analysis of a trial, its clinical relevance, protection/safety of study participants, or quality of reporting. Furthermore, the quality of controlled trials is of obvious relevance to systematic reviews and if the "raw material" or "data" is flawed then the conclusions of systematic reviews cannot be trusted. To date, the literature surrounding the establishment of standardized study enrollment and quality metrics to assess site performance in clinical trial consortiums is scarce. The lack of these metrics presents challenges to study site teams, sponsors, and other clinical research enterprise key stakeholders for adequately monitoring and evaluating study site performance as it relates to fulfilling trial enrollment and quality goals. The Department of Veterans Affairs (VA) Cooperative Studies Program (CSP) Network of Dedicated Enrollment Sites (NODES) undertook an effort to determine the feasibility of establishing and implementing standardized study enrollment and quality metrics for a clinical research consortium (NODES) as a tool to evaluate its performance. In this manuscript, we describe the development and implementation of standardized study enrollment and quality metrics to assess site performance across studies in our clinical research consortium.

Exploring the feasibility of using real-world data from a large clinical data research network to simulate clinical trials of Alzheimer's disease

In this study, we explored the feasibility of using real-world data (RWD) from a large clinical research network to simulate real-world clinical trials of Alzheimer’s disease (AD). The target trial (i.e., NCT00478205) is a Phase III double-blind, parallel-group trial that compared the 23 mg donepezil sustained release with the 10 mg donepezil immediate release formulation in patients with moderate to severe AD. We followed the target trial’s study protocol to identify the study population, treatment regimen assignments and outcome assessments, and to set up a number of different simulation scenarios and parameters. We considered two main scenarios: (1) a one-arm simulation: simulating a standard-of-care (SOC) arm that can serve as an external control arm; and (2) a two-arm simulation: simulating both intervention and control arms with proper patient matching algorithms for comparative effectiveness analysis. In the two-arm simulation scenario, we used propensity score matching controlling for baseline characteristics to simulate the randomization process. In the two-arm simulation, higher serious adverse event (SAE) rates were observed in the simulated trials than the rates reported in original trial, and a higher SAE rate was observed in the 23 mg arm than in the 10 mg SOC arm. In the one-arm simulation scenario, similar estimates of SAE rates were observed when proportional sampling was used to control demographic variables. In conclusion, trial simulation using RWD is feasible in this example of AD trial in terms of safety evaluation. Trial simulation using RWD could be a valuable tool for post-market comparative effectiveness studies and for informing future trials’ design. Nevertheless, such an approach may be limited, for example, by the availability of RWD that matches the target trials of interest, and further investigations are warranted.

Generalizability of Subgroup Effects

Generalizability methods are increasingly used to make inferences about the effect of interventions in target populations using a study sample. Most existing methods to generalize effects from sample to population rely on the assumption that subgroup-specific effects generalize directly. However, researchers may be concerned that in fact subgroup-specific effects differ between sample and population. In this brief report, we explore the generalizability of subgroup effects. First, we derive the bias in the sample average treatment effect estimator as an estimate of the population average treatment effect when subgroup effects in the sample do not directly generalize. Next, we present a Monte Carlo simulation to explore bias due to unmeasured heterogeneity of subgroup effects across sample and population. Finally, we examine the potential for bias in an illustrative data example. Understanding the generalizability of subgroup effects may lead to increased use of these methods for making externally valid inferences of treatment effects using a study sample.

Target validity: Bringing treatment of external validity in line with internal validity

Purpose of Review

"Target bias" is the difference between an estimate of association from a study sample and the causal effect in the target population of interest. It is the sum of internal and external bias. Given the extensive literature on internal validity, here, we review threats and methods to improve external validity.

Recent findings

External bias may arise when the distribution of modifiers of the effect of treatment differs between the study sample and the target population. Methods including those based on modeling the outcome, modeling sample membership, and doubly robust methods are available, assuming data on the target population is available.

Summary

The relevance of information for making policy decisions is dependent on both the actions that were studied and the sample in which they were evaluated. Combining methods for addressing internal and external validity can improve the policy relevance of study results.

A pragmatic methodology for the evaluation of digital care management in the context of multimorbidity

Multimorbidity is a defining challenge for health systems and requires coordination of care delivery and care management. Care management is a clinical service designed to remotely engage patients between visits and after discharge in order to support self-management of chronic and emergent conditions, encourage increased use of scheduled care and address the use of unscheduled care. Care management can be provided using digital technology - digital care management. A robust methodology to assess digital care management, or any traditional or digital primary care intervention aimed at longitudinal management of multimorbidity, does not exist outside of randomized controlled trials (RCTs). RCTs are not always generalizable and are also not feasible for most healthcare organizations. We describe here a novel and pragmatic methodology for the evaluation of digital care management that is generalizable to any longitudinal intervention for multimorbidity irrespective of its mode of delivery. This methodology implements propensity matching with bootstrapping to address some of the major challenges in evaluation including identification of robust outcome measures, selection of an appropriate control population, small sample sizes with class imbalances, and limitations of RCTs. We apply this methodology to the evaluation of digital care management at a U.S. payor and demonstrate a 9% reduction in ER utilization, a 17% reduction in inpatient admissions, and a 29% increase in the utilization of preventive medicine services. From these utilization outcomes, we drive forward an estimated cost saving that is specific to a single payor's payment structure for the study time period of $641 per-member-per-month at 3 months. We compare these results to those derived from existing observational approaches, 1:1 and 1:n propensity matching, and discuss the circumstances in which our methodology has advantages over existing techniques. Whilst our methodology focuses on cost and utilization and is applied in the U.S. context, it is applicable to other outcomes such as Patient Reported Outcome Measures (PROMS) or clinical biometrics and can be used in other health system contexts where the challenge of multimorbidity is prevalent.

Data management in substance use disorder treatment research: Implications from data harmonization of National Institute on Drug Abuse-funded randomized controlled trials

Background:

Secondary analysis of data from completed randomized controlled trials is a critical and efficient way to maximize the potential benefits from past research. De-identified primary data from completed randomized controlled trials have been increasingly available in recent years; however, the lack of standardized data products is a major barrier to further use of these valuable data. Pre-statistical harmonization of data structure, variables, and codebooks across randomized controlled trials would facilitate secondary data analysis, including meta-analyses and comparative effectiveness studies. We describe a pre-statistical data harmonization initiative to standardize de-identified primary data from substance use disorder treatment randomized controlled trials funded by the National Institute on Drug Abuse available on the National Institute on Drug Abuse Data Share website.

Methods:

Standardized datasets and codebooks with consistent data structures, variable names, labels, and definitions were developed for 36 completed randomized controlled trials. Common data domains were identified to bundle data files from individual randomized controlled trials according to relevant concepts. Variables were harmonized if at least two randomized controlled trials used the same instruments. The structures of the harmonized data were determined based on the feedback from clinical trialists and substance use disorder research experts.

Results:

We have created a harmonized database of variables across 36 randomized controlled trials with a build-in label and a brief definition for each variable. Data files from the randomized controlled trials have been consistently categorized into eight domains (enrollment, demographics, adherence, adverse events, physical health measures, mental-behavioral-cognitive health measures, self-reported substance use measures, and biologic substance use measures). Standardized codebooks and concordance tables have also been developed to help identify instruments and variables of interest more easily.

Conclusion:

The harmonized data of randomized controlled trials of substance use disorder treatments can potentially promote future secondary data analysis of completed randomized controlled trials, allowing combining data from multiple randomized controlled trials and provide guidance for future randomized controlled trials in substance use disorder treatment research.

Usability and Acceptability of a Mobile App to Help Emerging Adults Address their Friends' Substance Use (Harbor): Quantitative Study

BACKGROUND

Technology-assisted intervention and prevention strategies present opportunities for substance use-related research with emerging adults (EAs) and their peers. Emerging adulthood is a developmentally distinct period in which individuals between the ages of 18 and 29 years undergo unique emotional, cultural, developmental, and biological changes as they transition into adulthood. Crowdsourcing, or gathering feedback from a large group within web-based communities, offers researchers a unique and cost-effective way to obtain large amounts of information in a short period.

OBJECTIVE

This paper presents market feedback obtained via Amazon's Mechanical Turk from EAs (N=458) on the acceptability and utility of brief intervention scripts for a smartphone app currently under development. The mobile app, Harbor, teaches friends of EAs with substance use problems effective and supportive strategies for helping their friend make changes in their substance use behavior.

METHODS

We examined feedback on the wording of the intervention scripts and estimated the market size of EAs who may use this app. Furthermore, we calculated correlations between script ratings and measures of personal risky drinking (ie, Alcohol Use Disorder Identification Test) and the participants' use of confrontational, enabling, or supportive behaviors with an existing friend.

RESULTS

Approximately half of our sample (208/458, 45.4%) indicated that they had a close friend for whom they had concerns about their substance use, suggesting a potentially high demand for an app such as Harbor. Initial findings suggest that peers who engage in less enabling behaviors with friends who have a substance use problem exhibited lower risky drinking behaviors overall (r₂₀₆=-0.501; P<.001). Concerning acceptability, 98.0% (449/458) of the sample rated the scripts' dialogue as either somewhat, moderately, or extremely realistic (mean 3.92, SD 0.48) on 5-point Likert scale items. Finally, 95.4% (437/454) of respondents indicated that the scripts would be at least slightly helpful for training peers to help their friends with substance use issues. Finally, individuals who were better able to identify enabling language in enabling scripts self-reported fewer enabling behaviors toward their friend's substance use (r₂₀₆=-0.236; P=.001).

CONCLUSIONS

There exists a demonstrated level of desirability and acceptability among EAs for a mobile app such as Harbor. EAs who wish to engage in more supportive behaviors with their friends who engage in substance use and who are amenable to assisting their friends with sobriety likely would use and benefit from this app.

Opioid agonist treatment and fatal overdose risk in a state-wide US population receiving opioid use disorder services

BACKGROUND AND AIMS

Evidence from randomized controlled trials establishes that medication treatment with methadone and buprenorphine reduces opioid use and improves treatment retention. However, little is known about the role of such medications compared with non-medication treatments in mitigating overdose risk among US patient populations receiving treatment in usual care settings. This study compared overdose mortality among those in medication versus non-medication treatments in specialty care settings.

DESIGN

Retrospective cohort study using state-wide treatment data linked to death records. Survival analysis was used to analyze data in a time-to-event framework.

SETTING

Services delivered by 757 providers in publicly funded out-patient specialty treatment programs in Maryland, USA between 1 January 2015 and 31 December 2016.

PARTICIPANTS

A total of 48 274 adults admitted to out-patient specialty treatment programs in 2015-16 for primary diagnosis of opioid use disorder.

MEASUREMENTS

Main exposure was time in medication treatment (methadone/buprenorphine), time following medication treatment, time exposed to non-medication treatments and time following non-medication treatment. Main outcome was opioid overdose death during and after treatment. Hazard ratios were calculated using Cox proportional hazard regression. Propensity score weights were adjusted for patient information on sex, age, race, region of residence, marital and veteran status, employment, homelessness, primary opioid, mental health treatment, arrests and criminal justice referral.

FINDINGS

The study population experienced 371 opioid overdose deaths. Periods in medication treatment were associated with substantially reduced hazard of opioid overdose death compared with periods in non-medication treatment [adjusted hazard ratio (aHR) = 0.18, 95% confidence interval (CI) = 0.08-0.40]. Periods after discharge from non-medication treatment (aHR = 5.45, 95% CI = 2.80-9.53) and medication treatment (aHR = 5.85, 95% CI = 3.10-11.02) had similar and substantially elevated risks compared with periods in non-medication treatments.

CONCLUSIONS

Among Maryland patients in specialty opioid treatment, periods in treatment are protective against overdose compared with periods out of care. Methadone and buprenorphine are associated with significantly lower overdose death compared with non-medication treatments during care but not after treatment is discontinued.

Reweighting Oranges to Apples: Transported RE-LY Trial Versus Nonexperimental Effect Estimates of Anticoagulation in Atrial Fibrillation

BACKGROUND

Results from trials and nonexperimental studies are often directly compared, with little attention paid to differences between study populations. When target and trial population data are available, accounting for these differences through transporting trial results to target populations of interest provides useful perspective. We aimed to compare two-year risk differences (RDs) for ischemic stroke, mortality, and gastrointestinal bleeding in older adults with atrial fibrillation initiating dabigatran and warfarin when using trial transport methods versus nonexperimental methods.

METHODS

We identified Medicare beneficiaries who initiated warfarin or dabigatran from a 20% nationwide sample. To transport treatment effects observed in the randomized evaluation of long-term anticoagulation trial, we applied inverse odds weights to standardize estimates to two Medicare target populations of interest, initiators of: (1) dabigatran and (2) warfarin. Separately, we conducted a nonexperimental study in the Medicare populations using standardized morbidity ratio weighting to control measured confounding.

RESULTS

Comparing dabigatran to warfarin, estimated two-year RDs for ischemic stroke were similar with trial transport and nonexperimental methods. However, two-year mortality RDs were closer to the null when using trial transport versus nonexperimental methods for the dabigatran target population (transported RD: -0.57%; nonexperimental RD: -1.9%). Estimated gastrointestinal bleeding RDs from trial transport (dabigatran initiator RD: 1.8%; warfarin initiator RD: 1.9%) appeared more harmful than nonexperimental results (dabigatran initiator RD: 0.14%; warfarin initiator RD: 0.57%).

CONCLUSIONS

Differences in study populations can and should be considered quantitatively to ensure results are relevant to populations of interest, particularly when comparing trial with nonexperimental findings. See video abstract: http://links.lww.com/EDE/B703.

Comparing the performance of statistical methods that generalize effect estimates from randomized controlled trials to much larger target populations

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods - including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation - that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations.

Benchmarking Treatment Effectiveness of Community-Delivered Trauma-Focused Cognitive Behavioral Therapy

It is critical for urban youth with post-traumatic stress disorder (PTSD) living in poverty to have access to evidence-based interventions for their traumatic stress. However, there is limited research on the effectiveness of these interventions when provided in urban, community settings. The objectives of the current study are to (a) evaluate the effectiveness of trauma-focused cognitive behavioral therapy delivered from 2013 to 2016 in 15 behavioral health agencies on youth (N = 114) PTSD as well as  general mental health symptoms and  functioning, and (b) benchmark these clinical outcomes against other published efficacy and effectiveness trials. Effectiveness data are from the Philadelphia County Community Behavioral Health System, a system that has invested significantly in the training and ongoing support of clinicians providing high-quality trauma services to youth since 2012. From baseline to last assessment, youth PTSD symptom severity (d = 0.34), PTSD functional impairment (d = 0.38), and overall mental health problem severity (d = 0.29) improved. The effect sizes of  improvements were smaller than effect sizes observed in efficacy and effectiveness studies. This study is the first benchmarking study of TF-CBT and provides preliminary findings with regard to the effectiveness, and transportability, of TF-CBT to urban community settings that serve youth in poverty.

Effect heterogeneity and variable selection for standardizing causal effects to a target population

The participants in randomized trials and other studies used for causal inference are often not representative of the populations seen by clinical decision-makers. To account for differences between populations, researchers may consider standardizing results to a target population. We discuss several different types of homogeneity conditions that are relevant for standardization: Homogeneity of effect measures, homogeneity of counterfactual outcome state transition parameters, and homogeneity of counterfactual distributions. Each of these conditions can be used to show that a particular standardization procedure will result in an unbiased estimate of the effect in the target population, given assumptions about the relevant scientific context. We compare and contrast the homogeneity conditions, in particular their implications for selection of covariates for standardization and their implications for how to compute the standardized causal effect in the target population. While some of the recently developed counterfactual approaches to generalizability rely upon homogeneity conditions that avoid many of the problems associated with traditional approaches, they often require adjustment for a large (and possibly unfeasible) set of covariates.

Escitalopram for agitation in Alzheimer's disease (S-CitAD): Methods and design of an investigator-initiated, randomized, controlled, multicenter clinical trial

INTRODUCTION

Alzheimer's disease (AD) is a disabling, common cause of dementia, and agitation is one of the most common and distressing symptoms for patients with AD. Escitalopram for agitation in Alzheimer's disease (S-CitAD) tests a novel, clinically derived therapeutic approach to treat agitation in patients with AD.

METHODS

S-CitAD is a NIH-funded, investigator-initiated, randomized, multicenter clinical trial. Participants receive a structured psychosocial intervention (PSI) as standard of care. Participants without sufficient response to PSI are randomized to receive 15 mg escitalopram/day or a matching placebo in addition to PSI. Primary outcome is the Modified Alzheimer's Disease Cooperative Study - Clinical Global Impression of Change (mADCS-CGIC).

DISCUSSION

S-CitAD will provide information about a practical, immediately available approach to treating agitation in patients with AD. S-CitAD may become a model of how to evaluate and predict treatment response in patients with AD and agitation as a neuropsychiatric symptom (ClinicalTrials.gov Identifier: NCT03108846).

Representativeness of Randomized Clinical Trial Cohorts in End-stage Kidney Disease: A Meta-analysis

IMPORTANCE

Systematic differences between patients included in randomized clinical trials (RCTs) and the general patient population may influence the generalizability of RCT findings. Comprehensive national registries of patients with end-stage kidney disease who are undergoing dialysis provide a unique opportunity to compare trial and real-world patient cohorts.

OBJECTIVE

To determine if participants in large, multicenter dialysis trials were similar to the general population undergoing dialysis in terms of age, comorbidities, and mortality rate.

DATA SOURCES

MEDLINE, PubMed, and the Cochrane Central Register of Controlled Trials were systematically searched on January 6, 2017, for studies published from January 1, 2007, to December 31, 2016. Data sources were published manuscripts, supplementary material, and trial registration information. Data on the general population undergoing dialysis were derived from the US Renal Data System (USRDS). Data were analyzed from March 17 to July 22, 2018.

STUDY SELECTION

Randomized clinical trials enrolling only participants undergoing dialysis for end-stage kidney disease with 100 or more adult participants from 2 or more sites.

DATA EXTRACTION AND SYNTHESIS

Abstract screening and data extraction were performed independently by 2 researchers. Data were pooled using a random-effects model.

MAIN OUTCOMES AND MEASURES

The primary outcome was difference in mean age between the RCT and USRDS populations. Secondary outcomes included differences in mortality rate and comorbidities.

RESULTS

The search identified 186 RCTs, enrolling 79 104 participants. Compared with the 2011 USRDS population, RCT participants were younger (mean age, 58.9 years; 95% CI, 58.3-59.5 years vs 61.2 years; P < .001), more likely to be male (58.9%; 95% CI, 57.6%-60.1% vs 55.7%; P < .001), and have coronary artery disease (26.9%; 95% CI, 22.2%-31.7% vs 17.7%; P < .001) and less likely to have diabetes (40.2%; 95% CI, 36.7%-43.6% vs 44.2%; P = .03) or heart failure (19.6%; 95% CI, 15.1%-24.0% vs 29.8%; P < .001). The mortality rate per 100 patient-years during trial participation was less than half that of the USRDS population (8.9; 95% CI, 7.8-10.0 vs 18.6; P < .001). The differences in age, mortality, and coronary artery disease remained when studies recruiting only from the United States were considered. Diabetes was more common in RCT participants from the United States than in the registry population.

CONCLUSIONS AND RELEVANCE

Participants in large, multicenter RCTs of patients with end-stage kidney disease undergoing dialysis are younger, have a different pattern of comorbidities, and have a lower mortality rate than the general population of patients undergoing dialysis. This finding has implications for the generalization of trial results to the broader patient population and for future trial design.

Implementing statistical methods for generalizing randomized trial findings to a target population

Randomized trials are considered the gold standard for assessing the causal effects of a drug or intervention in a study population, and their results are often utilized in the formulation of health policy. However, there is growing concern that results from trials do not necessarily generalize well to their respective target populations, in which policies are enacted, due to substantial demographic differences between study and target populations. In trials related to substance use disorders (SUDs), especially, strict exclusion criteria make it challenging to obtain study samples that are fully "representative" of the populations that policymakers may wish to generalize their results to. In this paper, we provide an overview of post-trial statistical methods for assessing and improving upon the generalizability of a randomized trial to a well-defined target population. We then illustrate the different methods using a randomized trial related to methamphetamine dependence and a target population of substance abuse treatment seekers, and provide software to implement the methods in R using the "generalize" package. We discuss several practical considerations for researchers who wish to utilize these tools, such as the importance of acquiring population-level data to represent the target population of interest, and the challenges of data harmonization.

Visual Acuity Outcomes after Anti-Vascular Endothelial Growth Factor Treatment for Neovascular Age-Related Macular Degeneration: Age-Related Eye Disease Study 2 Report Number 19

PURPOSE

To analyze best-corrected visual acuity (BCVA) outcomes after anti-vascular endothelial growth factor (VEGF) treatment for neovascular age-related macular degeneration (AMD).

DESIGN

Prospective cohort study of participants enrolled in a clinical trial of oral supplements and receiving anti-VEGF therapy in routine clinical practice.

PARTICIPANTS

Age-Related Eye Disease Study 2 (AREDS2) participants (50-85 years of age) whose eyes met AREDS2 inclusion criteria at baseline (no late AMD, BCVA ≥20/100, no previous anti-VEGF injections) but received at least 1 anti-VEGF injection for incident neovascular AMD during follow-up.

METHODS

Participants underwent refracted BCVA testing, ophthalmoscopic examination, and stereoscopic color fundus photography at baseline and annual study visits over 5 years. Self-reports of anti-VEGF injections (numbers, dates, and names of drug) were collected at baseline and annual study visits and during telephone calls every 6 months.

MAIN OUTCOME MEASURES

Primary outcome measures were mean refracted BCVA and proportions of eyes with BCVA of 20/40 or better and 20/200 or worse. An exploratory outcome measure was the mean number of self-reported anti-VEGF injections.

RESULTS

One thousand one hundred five eyes of 986 AREDS2 participants met the inclusion criteria; of these, 977 participants (99.1%) underwent at least 1 posttreatment visit. At the first and subsequent annual examinations after the first injection, mean refracted BCVAs were 68.0 letters (Snellen equivalent, 20/40), 66.1 letters, 64.7 letters, 63.2 letters, and 61.5 letters (Snellen equivalent, 20/60). Proportions of eyes with BCVA of 20/40 or better were 59.3%, 55.1%, 53.5%, 50.6%, and 49.7%, and those with BCVA of 20/200 or worse were 5.5%, 8.6%, 9.4%, 12.4%, and 14.4%. Mean annual numbers of self-reported anti-VEGF injections per eye were 2.9, 3.9, 3.3, 3.1, and 3.0.

CONCLUSIONS

Refracted BCVA data were obtained in a clinical trial environment but were related to anti-VEGF treatment administered in normal clinical practice. Visual outcomes declined slowly with increased follow-up time: mean BCVA decreased by approximately 1.5 to 2 letters per year. At 5 years, half of eyes achieved BCVA of 20/40 or better, but approximately one sixth showed BCVA of 20/200 or worse. These data may be useful in assessing the long-term effects of anti-VEGF therapy.

Diagnostic Assessment of Assumptions for External Validity: An Example Using Data in Metastatic Colorectal Cancer

BACKGROUND

Methods developed to estimate intervention effects in external target populations assume that all important effect measure modifiers have been identified and appropriately modeled. Propensity score-based diagnostics can be used to assess the plausibility of these assumptions for weighting methods.

METHODS

We demonstrate the use of these diagnostics when assessing the transportability of treatment effects from the standard of care for metastatic colorectal cancer control arm in a phase III trial (HORIZON III) to a target population of 1,942 Medicare beneficiaries age 65+ years.

RESULTS

In an unadjusted comparison, control arm participants had lower mortality compared with target population patients treated with the standard of care therapy (trial vs. target hazard ratio [HR] = 0.72, 95% confidence interval [CI], 0.58, 0.89). Applying inverse odds of sampling weights attenuated the trial versus target HR (weighted HR = 0.96, 95% CI = 0.73, 1.26). However, whether unadjusted or weighted, hazards did not appear proportional. At 6 months of follow-up, mortality was lower in the weighted trial population than the target population (weighted trial vs. target risk difference [RD] = -0.07, 95% CI = -0.13, -0.01), but not at 12 months (weighted RD = 0.00, 95% CI = -0.09, 0.09).

CONCLUSION

These diagnostics suggest that direct transport of treatment effects from HORIZON III to the Medicare population is not valid. However, the proposed sampling model might allow valid transport of the treatment effects on longer-term mortality from HORIZON III to the Medicare population treated in clinical practice. See video abstract at, http://links.lww.com/EDE/B435.

Comparing pharmacological treatments for cocaine dependence: Incorporation of methods for enhancing generalizability in meta-analytic studies

OBJECTIVES

Few head-to-head comparisons of cocaine dependence medications exist, and combining data from different randomized controlled trials (RCTs) is fraught with methodological challenges including limited generalizability of the RCT findings. This study applied a novel meta-analytic approach to data of cocaine dependence medications.

METHODS

Data from 4 placebo-controlled RCTs (Reserpine, Modafinil, Buspirone, and Ondansetron) were obtained from the National Institute of Drug Abuse Clinical Trials Network (n = 456). The RCT samples were weighted to resemble treatment-seeking patients (Treatment Episodes Data Set-Admissions) and individuals with cocaine dependence in general population (National Survey on Drug Use and Health). We synthesized the generalized outcomes with pairwise meta-analysis using individual-level data and compared the generalized outcomes across the 4 RCTs with network meta-analysis using study-level data.

RESULTS

Weighting the data by the National Survey on Drug Use and Health generalizability weight made the overall population effect on retention significantly larger than the RCT sample effect. However, there was no significant difference between the population effect and the RCT sample effect on abstinence. Weighting changed the ranking of the effectiveness across treatments.

CONCLUSIONS

Applying generalizability weights to meta-analytic studies is feasible and potentially provides a useful tool in assessing comparative effectiveness of treatments for substance use disorders in target populations.

Generalizability of randomized trial results to target populations: Design and analysis possibilities

Randomized trials play an important role in estimating the effect of a policy or social work program in a given population. While most trial designs benefit from strong internal validity, they often lack external validity, or generalizability, to the target population of interest. In other words, one can obtain an unbiased estimate of the study sample average treatment effect (SATE) from a randomized trial; however, this estimate may not equal the target population average treatment effect (TATE) if the study sample is not fully representative of the target population. This paper provides an overview of existing strategies to assess and improve upon the generalizability of randomized trials, both through statistical methods and study design, as well as recommendations on how to implement these ideas in social work research.

The effectiveness and safety of nonsurgical integrative interventions for symptomatic lumbar spinal spondylolisthesis: A randomized controlled multinational, multicenter trial protocol

BACKGROUND

Surgery is generally accepted as the main therapeutic option for symptomatic lumbar spondylolisthesis. However, new nonsurgical therapeutic options need to be explored for this population.

OBJECTIVES

The objective of this study is to assess the effectiveness and safety of a 5-week Mokhuri treatment program compared with conventional nonsurgical treatments for symptomatic lumbar spondylolisthesis.

METHODS

This is a study protocol for a multinational, multicenter clinical randomized controlled trial comparing the effectiveness and safety of 5 weeks of nonsurgical integrative treatments (a Mokhuri treatment program consisting of Chuna, acupuncture, and patient education) with nonsurgical conventional treatments (drugs for pain relief, epidural steroid injections, and physical therapy). Clinical outcomes including visual analogue scale (VAS) scores ranging from 0 to 100 for low back pain and leg pain, EQ-5D scores, Oswestry disability index (ODI) scores, Roland-Morris Disability Questionnaire (RMDQ) scores, Zurich Claudication Questionnaire (ZCQ) scores, walking duration and distance without leg pain, and a 5-minute treadmill test, and the ratio between the actual duration of participation and the originally scheduled duration in each group, the presence of any additional spondylolisthesis treatments, the types of concomitant treatments during the follow-up period, and adverse events (AEs) will be assessed at 7 weeks, 18 weeks, 30 weeks, 54 weeks, and 102 weeks after the end of the treatments.

CONCLUSION AND DISCUSSION

The results of this study will provide clinical evidence on nonsurgical integrative interventions for patients with symptomatic lumbar spondylolisthesis. CLINICAL TRIAL REGISTRY:: clinicaltrials.gov (NCT03107468).

Generalizability of the findings from a randomized controlled trial of a web-based substance use disorder intervention

BACKGROUND AND OBJECTIVES

There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population.

METHODS

The sample of the web-based SUD intervention (Therapeutic Education System vs. Treatment-as-usual; n = 507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, we computed weighted treatment effects on retention and abstinence.

RESULTS

Substantial differences between the RCT sample and the target population was demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p < .001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight.

DISCUSSIONS AND CONCLUSIONS

Generalizability of the findings from the RCT could be limited when the RCT sample does not well represent the target population.

SCIENTIFIC SIGNIFICANCE

Application of generalizability weights can be a potentially useful tool to improve generalizability of RCT findings. (Am J Addict 2018;27:231-237).

Recruitment practices for U.S. minority and underserved populations in NRG oncology: Results of an online survey

Introduction

Cancer clinical trials (CCT) provide much of the evidence for clinical guidelines and standards of care. But low levels of CCT participation are well documented, especially for minorities.

Methods and materials

We conducted an online survey of 556 recruitment practices across the NRG Oncology network. Survey aims were 1) to learn how sites recruit minority/underserved populations; 2) to better understand the catchment areas of the NRG institutions; and 3) to aid in planning education programs for accrual of minority/underserved populations.

Results

The survey response rate was 34.9%. The most effective methods reported for recruiting minority/underserved participants were patient navigators (44.4%) and translators (38.9%). All institutions reported using a mechanism for eligibility screening and 71% of institutions reported using a screening/enrollment tracking system. CCT training was required at 78.1% and cultural competency training was required at 47.5% of responding institutions. Only 19.9% of sites used community partners to assist with minority recruitment and just 37.1% of respondents reported a defined catchment area. Sites reported very little race and ethnicity data.

Conclusion

This NRG Oncology online survey provides useful data for improvements in trial enrollment and training to recruit minority/underserved populations to CCT. Areas for further investigation include web-based methods for recruitment and tracking, cultural competency training, definition of catchment areas, use of patient navigators, and community partnerships. The survey results will guide recruitment training programs.