Central Adjudication Identified Additional and Prognostically Important Myocardial Infarctions in Patients Undergoing Percutaneous Coronary Intervention

Accuracy of Event Rate and Effect Size Estimation in Major Cardiovascular Trials: A Systematic Review

Importance

For the design of a randomized clinical trial (RCT), estimation of the expected event rate and effect size of an intervention is needed to calculate the sample size. Overestimation may lead to an underpowered trial.

Objective

To evaluate the accuracy of published estimates of event rate and effect size in contemporary cardiovascular RCTs.

Evidence Review

A systematic search was conducted in MEDLINE for multicenter cardiovascular RCTs associated with MeSH (Medical Subject Headings) terms for cardiovascular diseases published in the New England Journal of Medicine, JAMA, or the Lancet between January 1, 2010, and December 31, 2019. Identified trials underwent abstract review; eligible trials then underwent full review, and those with insufficiently reported data were excluded. Data were extracted from the original publication or the study protocol, and a random-effects model was used for data pooling. This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting guideline. The primary outcome was the accuracy of event rate and effect size estimation. Accuracy was determined by comparing the observed event rate in the control group and the effect size with their hypothesized values. Linear regression was used to determine the association between estimation accuracy and trial characteristics.

Findings

Of the 873 RCTs identified, 374 underwent full review and 30 were subsequently excluded, resulting in 344 trials for analysis. The median observed event rate was 9.0% (IQR, 4.3% to 21.4%), which was significantly lower than the estimated event rate of 11.0% (IQR, 6.0% to 25.0%) with a median deviation of -12.3% (95% CI, -16.4% to -5.6%; P < .001). More than half of the trials (196 [61.1%]) overestimated the expected event rate. Accuracy of event rate estimation was associated with a higher likelihood of refuting the null hypothesis (0.13 [95% CI, 0.01 to 0.25]; P = .03). The median observed effect size in superiority trials was 0.91 (IQR, 0.74 to 0.99), which was significantly lower than the estimated effect size of 0.72 (IQR, 0.60 to 0.80), indicating a median overestimation of 23.1% (95% CI, 17.9% to 28.3%). A total of 216 trials (82.1%) overestimated the effect size.

Conclusions and Relevance

In this systematic review of contemporary cardiovascular RCTs, event rates of the primary end point and effect sizes of an intervention were frequently overestimated. This overestimation may have contributed to the inability to adequately test the trial hypothesis.

Clinical Adjudication of Hemodialysis Catheter-Related Bloodstream Infections: Findings from the REDUCCTION Trial

Key Points

The inter-rater reliability of reporting hemodialysis catheter-related infectious events between site investigators and trial adjudicators in Australia and New Zealand was substantial. The high concordance level in reporting catheter infections improves confidence in using site-level bacteremia rates as a clinical metric for quality benchmarking and future pragmatic clinical trials. A rigorous adjudication protocol may not be needed if clearly defined criteria to ascertain catheter-associated bacteremia are used.

Background

Hemodialysis catheter-related bloodstream infection (HD-CRBSI) are a significant source of morbidity and mortality among dialysis patients, but benchmarking remains difficult because of varying definitions of HD-CRBSI. This study explored the effect of clinical adjudication process on HD-CRBSI reporting.

Methods

The REDUcing the burden of Catheter ComplicaTIOns: a National approach trial implemented an evidence-based intervention bundle using a stepped-wedge design to reduce HD-CRBSI rates in 37 Australian kidney services. Six New Zealand services participated in an observational capacity. Adult patients with a new hemodialysis catheter between December 2016 and March 2020 were included. HD-CRBSI events reported were compared with the adjudicated outcomes using the end point definition and adjudication processes of the REDUcing the burden of Catheter ComplicaTIOns: a National approach trial. The concordance level was estimated using Gwet agreement coefficient (AC1) adjusted for service-level effects and implementation tranches (Australia only), with the primary outcome being the concordance of confirmed HD-CRBSI.

Results

A total of 744 hemodialysis catheter-related infectious events were reported among 7258 patients, 12,630 catheters, and 1.3 million catheter-exposure days. The majority were confirmed HD-CRBSI, with 77.9% agreement and substantial concordance (AC1=0.77; 95% confidence interval [CI], 0.73 to 0.81). Exit site infections have the highest concordance (AC1=0.85; 95% CI, 0.78 to 0.91); the greatest discordance was in events classified as other (AC1=0.33; 95% CI, 0.16 to 0.49). The concordance of all hemodialysis catheter infectious events remained substantial (AC1=0.80; 95% CI, 0.76 to 0.83) even after adjusting for the intervention tranches in Australia and overall service-level clustering.

Conclusions

There was a substantial level of concordance in overall and service-level reporting of confirmed HD-CRBSI. A standardized end point definition of HD-CRBSI resulted in comparable hemodialysis catheter infection rates in Australian and New Zealand kidney services. Consistent end point definition could enable reliable benchmarking outside clinical trials without the need for independent clinical adjudication.

Evaluating registry-based trial economics: Results from the STRESS clinical trial

Background

Registry-based trials have the potential to reduce randomized clinical trial (RCT) costs. However, observed cost differences also may be achieved through pragmatic trial designs. A systematic comparison of trial costs across different designs has not been previously performed.

Methods

We conducted a study to compare the current Steroids to Reduce Systemic inflammation after infant heart surgery (STRESS) registry-based RCT vs. two established designs: pragmatic RCT and explanatory RCT. The primary outcome was total RCT design costs. Secondary outcomes included: RCT duration and personnel hours. Costs were estimated using the Duke Clinical Research Institute's pricing model.

Results

The Registry-Based RCT estimated duration was 31.9 weeks greater than the other designs (259.5 vs. 227.6 weeks). This delay was caused by the Registry-Based design's periodic data harvesting that delayed site closing and statistical reporting. Total personnel hours were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design (52,488 vs 29,763 vs. 24,480 h, respectively). Total costs were greatest for the Explanatory design followed by the Pragmatic design and the Registry-Based design ($10,140,263 vs. $4,164,863 vs. $3,268,504, respectively). Thus, Registry-Based total costs were 32 % of the Explanatory and 78 % of the Pragmatic design.

Conclusion

Total costs for the STRESS RCT with a registry-based design were less than those for a pragmatic design and much less than an explanatory design. Cost savings reflect design elements and leveraging of registry resources to improve cost efficiency, but delays to trial completion should be considered.

Endpoint adjudication in cardiovascular clinical trials

Endpoint adjudication (EA) is a common feature of contemporary randomized controlled trials (RCTs) in cardiovascular medicine. Endpoint adjudication refers to a process wherein a group of expert reviewers, known as the clinical endpoint committee (CEC), verify potential endpoints identified by site investigators. Events that are determined by the CEC to meet pre-specified trial definitions are then utilized for analysis. The rationale behind the use of EA is that it may lessen the potential misclassification of clinical events, thereby reducing statistical noise and bias. However, it has been questioned whether this is universally true, especially given that EA significantly increases the time, effort, and resources required to conduct a trial. Herein, we compare the summary estimates obtained using adjudicated vs. non-adjudicated site designated endpoints in major cardiovascular RCTs in which both were reported. Based on these data, we lay out a framework to determine which trials may warrant EA and where it may be redundant. The value of EA is likely greater when cardiovascular trials have nuanced primary endpoints, endpoint definitions that align poorly with practice, sub-optimal data completeness, greater operator variability, and lack of blinding. EA may not be needed if the primary endpoint is all-cause death or all-cause hospitalization. In contrast, EA is likely merited for more nuanced endpoints such as myocardial infarction, bleeding, worsening heart failure as an outpatient, unstable angina, or transient ischaemic attack. A risk-based approach to adjudication can potentially allow compromise between costs and accuracy. This would involve adjudication of a small proportion of events, with further adjudication done if inconsistencies are detected.

Impact of design characteristics among studies comparing coronary computed tomography angiography to noninvasive functional testing in chronic coronary syndromes

BACKGROUND

Coronary computed tomography angiography (CCTA) is widely adopted to detect obstructive coronary artery disease (CAD) in patients with chronic coronary syndromes (CCS). However, it is unknown to which extent study-specific characteristics yield different conclusions.

METHODS

We summarized non-randomized and randomized studies comparing CCTA and noninvasive functional testing for CCS with information on the outcome of myocardial infarction (MI). We evaluated the differential effect according to study characteristics using random-effect meta-analysis with Hartung-Knapp-Sidik-Jonkman adjustments.

RESULTS

Fifteen studies (8 non-randomized, 7 randomized) were included. CCTA was associated with decrease in relative (odds ratio (OR) 0.54, 95%CI 0.47 to 0.62, P < .001) and absolute MI risk (risk difference (RD) -0.4%, 95%CI -0.6 to -0.1, P = .005). The results remained consistent among the non-randomized (RD -0.4%, 95%CI -0.7 to -0.1, P=.029), but not among the randomized trials where there was no difference in the observed risk (RD 0.2%, 95%CI -0.6 to 0.1, P = .158). CCTA was not associated with MI reduction in studies with clinical outcome definition (OR 0.77, 95%CI 0.41 to 1.44, P = .212), research driven follow-up (OR 0.54, 95%CI 0.24 to 1.21, P = .090), central event assessment (OR 0.63, 95%CI 0.21 to 1.86, P = .207), outcome adjudication (OR 0.74, 95%CI 0.24 to 2.23, P = .178), or at low-risk of bias (OR 0.74, 95%CI 0.24 to 2.23, P = .178).

CONCLUSIONS

Among studies of any design, CCTA was associated with lower risk of MI in CCS compared to noninvasive functional testing. This benefit was diminished among studies with clinical outcome definition, central outcome assessment/adjudication or at low-risk of bias.

Ascertaining Nonfatal Endpoints in Clinical Trials: Central Adjudication Versus Patient Insurance Claims

BACKGROUND

The 21st Century Cures Act allows the US Food and Drug Administration (FDA) to utilize real-world data (RWD) to create real-world evidence (RWE) for new indications or post approval study requirements. We compared central adjudication with two insurance claims data sources to understand how endpoint accuracy differences impact RWE results.

METHODS

We developed a decision analytic model to compare differences in efficacy (all-cause death, stroke and myocardial infarction) and safety (bleeding requiring transfusion) results for a simulated acute coronary syndrome antiplatelet therapy clinical trial. Endpoint accuracy metrics were derived from previous studies that compared centrally-adjudicated and insurance claims-based clinical trial endpoints.

RESULTS

Efficacy endpoint results per 100 patients were similar for the central adjudication model (intervention event rate, 11.3; control, 13.7; difference, 2.4) and the prospective claims data collection model (intervention event rate, 11.2; control 13.6; difference, 2.3). However, the retrospective claims linking model's efficacy results were larger (intervention event rate, 14.6; control, 18.0; difference, 3.4). True positive event rate results (intervention, control and difference) for both insurance claims-based models were less than the central adjudication model due to false negative events. Differences in false positive event rates were responsible for differences in efficacy results for the two insurance claims-based models.

CONCLUSION

Efficacy endpoint results differed by data source. Investigators need guidance to determine which data sources produce regulatory-grade RWE.

Importance of Internal Variability in Clinical Trials of Cardiovascular Disease

A well conducted randomised controlled trial (RCT) is extremely important in the field of cardiovascular medicine. At the same time, it is equally important to understand the strengths and limitations of any RCT, and internal variability is a concept in clinical trials that is poorly understood. Variability in a clinical trial may be introduced at an individual level or during measurement, sampling, or conduct of the trial. It is not the same as internal validity, which is a broader concept of accuracy; to be valid, a study should minimise variability and have sound methodology. There are various steps that may be followed to minimise the internal variability in a clinical trial. One aspect of great importance is the adjudication process, which should be done meticulously and is often a step that is overlooked. It is important to standardise each step as much as possible, to ensure consistency and reduce noise at all levels. The concepts discussed in this review may serve as a roadmap to limit the influence of internal variability and maximise internal validity of RCT results.

Prognosis of Claims- Versus Trial-Based Ischemic and Bleeding Events Beyond 1 Year After Coronary Stenting

Background It is unknown whether clinical events identified with administrative claims have similar prognosis compared with trial-adjudicated events in cardiovascular clinical trials. We compared the prognostic significance of claims-based end points in context of trial-adjudicated end points in the DAPT (Dual Antiplatelet Therapy) study. Methods and Results We matched 1336 patients aged ≥65 years who received percutaneous coronary intervention in the DAPT study with the CathPCI registry linked to Medicare claims. We compared death at 21 months post-randomization using Cox proportional hazards models among patients with ischemic events (myocardial infarction or stroke) and bleeding events identified by: (1) both trial adjudication and claims; (2) trial adjudication only; and (3) claims only. A total of 47 patients (3.5%) had ischemic events identified by both trial adjudication and claims, 24 (1.8%) in trial adjudication only, 15 (1.1%) in claims only, and 1250 (93.6%) had no ischemic events, with annualized unadjusted mortality rates of 12.8, 5.5, 14.9, and 1.26 per 100 person-years, respectively. A total of 44 patients (3.3%) had bleeding events identified with both trial adjudication and claims, 13 (1.0%) in trial adjudication only, 65 (4.9%) in claims only, and 1214 (90.9%) had no bleeding events, with annualized unadjusted mortality rates of 11.0, 16.8, 10.7, and 0.95 per 100 person-years, respectively. Among patients with no trial-adjudicated events, patients with events in claims only had a high subsequent adjusted mortality risk (hazard ratio (HR) ischemic events: 31.5; 95% CI, 8.9‒111.9; HR bleeding events 23.9; 95% CI, 10.7‒53.2). Conclusions In addition to trial-adjudicated events, claims identified additional clinically meaningful ischemic and bleeding events that were prognostically significant for death.

Validation of Administrative Claims to Ascertain Outcomes in Pivotal Trials of Transcatheter Aortic Valve Replacement

OBJECTIVES

The aim of this study was to evaluate the performance of administrative claims in ascertaining trial clinical events committee-adjudicated outcomes in the U.S. CoreValve studies.

BACKGROUND

Real-world data offer tremendous opportunity to improve outcome ascertainment in clinical trials. However, little is known about the validity of outcomes ascertained using real-world data to capture trial endpoints.

METHODS

Patients enrolled in 3 pivotal trials and 2 pre-market continued-access studies evaluating transcatheter aortic valve replacement were linked to Medicare fee-for-service inpatient claims. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and kappa agreement statistic of claims to detect clinical endpoints and procedural complications in trial patients were calculated.

RESULTS

Claims accurately identified trial-adjudicated deaths (sensitivity, specificity, PPV, and NPV all >99.6%; kappa 1.00). Claims had good performance in identifying trial-adjudicated permanent pacemaker implantation (sensitivity 92.2%, specificity 99.1%, PPV 96.1%, NPV 98.2%, kappa 0.93) and aortic valve reintervention (sensitivity 84.4%, specificity 99.6%, PPV 69.1%, NPV 99.8%, kappa 0.76). Claims had more modest performance in ascertaining trial-adjudicated myocardial infarction (sensitivity 63.6%, specificity 97.2%, PPV 29.9%, NPV 99.3%, kappa 0.39) and acute kidney injury (sensitivity 70.2%, specificity 85.4%, PPV 38.2%, NPV 95.7%, kappa 0.41) and the poorest performance for identifying trial-adjudicated bleeding events (sensitivity 86.4%, specificity 36.8%, PPV 35.0%, NPV 86.3%, kappa 0.16).

CONCLUSIONS

Compared with trial-adjudicated outcomes, claims data performed well in ascertaining death and outcomes with procedural billing codes and more modestly in identifying other outcomes. Claims may be cautiously and selectively used to augment data collection in future cardiovascular device trials.