Added Value of a Blinded Outcome Adjudication Committee in an Open-Label Randomized Stroke Trial

Clinical Adjudication of Hemodialysis Catheter-Related Bloodstream Infections: Findings from the REDUCCTION Trial

Key Points

The inter-rater reliability of reporting hemodialysis catheter-related infectious events between site investigators and trial adjudicators in Australia and New Zealand was substantial. The high concordance level in reporting catheter infections improves confidence in using site-level bacteremia rates as a clinical metric for quality benchmarking and future pragmatic clinical trials. A rigorous adjudication protocol may not be needed if clearly defined criteria to ascertain catheter-associated bacteremia are used.

Background

Hemodialysis catheter-related bloodstream infection (HD-CRBSI) are a significant source of morbidity and mortality among dialysis patients, but benchmarking remains difficult because of varying definitions of HD-CRBSI. This study explored the effect of clinical adjudication process on HD-CRBSI reporting.

Methods

The REDUcing the burden of Catheter ComplicaTIOns: a National approach trial implemented an evidence-based intervention bundle using a stepped-wedge design to reduce HD-CRBSI rates in 37 Australian kidney services. Six New Zealand services participated in an observational capacity. Adult patients with a new hemodialysis catheter between December 2016 and March 2020 were included. HD-CRBSI events reported were compared with the adjudicated outcomes using the end point definition and adjudication processes of the REDUcing the burden of Catheter ComplicaTIOns: a National approach trial. The concordance level was estimated using Gwet agreement coefficient (AC1) adjusted for service-level effects and implementation tranches (Australia only), with the primary outcome being the concordance of confirmed HD-CRBSI.

Results

A total of 744 hemodialysis catheter-related infectious events were reported among 7258 patients, 12,630 catheters, and 1.3 million catheter-exposure days. The majority were confirmed HD-CRBSI, with 77.9% agreement and substantial concordance (AC1=0.77; 95% confidence interval [CI], 0.73 to 0.81). Exit site infections have the highest concordance (AC1=0.85; 95% CI, 0.78 to 0.91); the greatest discordance was in events classified as other (AC1=0.33; 95% CI, 0.16 to 0.49). The concordance of all hemodialysis catheter infectious events remained substantial (AC1=0.80; 95% CI, 0.76 to 0.83) even after adjusting for the intervention tranches in Australia and overall service-level clustering.

Conclusions

There was a substantial level of concordance in overall and service-level reporting of confirmed HD-CRBSI. A standardized end point definition of HD-CRBSI resulted in comparable hemodialysis catheter infection rates in Australian and New Zealand kidney services. Consistent end point definition could enable reliable benchmarking outside clinical trials without the need for independent clinical adjudication.

Accuracy and interrater agreement of death event adjudications by physician trainees: validation of the ISTH definition of pulmonary embolism-related death in an autopsy cohort

BACKGROUND

We previously determined good agreement and high specificity of the International Society on Thrombosis and Haemostasis (ISTH) definition of pulmonary embolism (PE)-related death among an expert central adjudication committee (CAC). CACs are often composed of experts in the corresponding research field. Involving physician trainees in CACs would allow investigators to divide the workload and foster trainees' research experience.

OBJECTIVE

To evaluate the accuracy of the ISTH definition of PE-related death for PE- versus non-PE-related deaths as confirmed by autopsy and its interrater agreement among physician trainees.

METHODS

This retrospective autopsy cohort included all patients with PE-related deaths between January 2010 and July 2019 as well as patients who died in 2018 from a cause other than PE at the New York-Presbyterian Hospital. Based on premortem clinical summaries, two physician trainees independently determined the cause of death using the ISTH definition of PE-related death. We calculated the sensitivity and specificity of the ISTH definition to identify autopsy-confirmed PE-related death and its interrater agreement.

RESULTS

Overall, 126 death events were adjudicated (median age, 68 years; 60 [48%] women), of which 29 (23%) were due to PE, as confirmed by autopsy. Sensitivity and specificity of the ISTH definition for autopsy-confirmed PE-related death was 48% (95% CI, 29-67) and 100% (95% CI, 96-100), respectively. Interrater reliability for PE-related death was good (percentage agreement, 93%; 95% CI, 87-96, Cohen's Kappa, 0.67; 95% CI, 44-85).

CONCLUSION

Our findings are consistent with our previous validation study. They further support the use of the ISTH definition of PE-related death and revealed high agreement between adjudicators with varied experience.

Should we adjudicate outcomes in stroke trials? A systematic review

BACKGROUND

Central adjudication of outcomes is common in randomized clinical trials in stroke. The rationale for adjudication is clear; centrally adjudicated outcomes should have less random and systematic errors than outcomes assessed locally by site investigators. However, adjudication brings added complexities to a clinical trial and can be costly.

AIM

To assess the evidence for outcome adjudication in stroke trials.

SUMMARY OF REVIEW

We identified 12 studies evaluating central adjudication in stroke trials. The majority of these were secondary analyses of trials, and the results of all of these would have remained unchanged had central adjudication not taken place, even for trials without sufficient blinding. The largest differences between site-assessed and adjudicator-assessed outcomes were between the most subjective outcomes, such as causality of serious adverse events. We found that the cost of adjudication could be upward of £100,000 for medium to large prevention trials. These findings suggest that the cost of central adjudication may outweigh the advantages it brings in many cases. However, through simulation, we found that only a small amount of bias is required in site investigators' outcome assessments before adjudication becomes important.

CONCLUSION

Central adjudication may not be necessary in stroke trials with blinded outcome assessment. However, for open-label studies, central adjudication may be more important.

Central vs site outcome adjudication in the IRIS trial

BACKGROUND

Central adjudication of outcome events is the standard in clinical trial research. We examine the benefit of central adjudication in the Insulin Resistance Intervention after Stroke (IRIS) trial and show how the benefit is influenced by outcome definition and features of the adjudicated events.

METHODS

IRIS tested pioglitazone for prevention of stroke and myocardial infarction in patients with a recent transient ischemic attack or ischemic stroke. We compared the hazard ratios for study outcomes classified by site and central adjudication. We repeated the analysis for an updated stroke definition.

RESULTS

The hazard ratios for the primary outcome were identical (0.76) and statistically significant regardless of adjudicator. The hazard ratios for stroke alone were nearly identical with site and central adjudication, but only reached significance with site adjudication (HR, 0.80; p = 0.049 vs. HR, 0.82; p = 0.09). Using the updated stroke definition, all hazard ratios were lower than with the original IRIS definition and statistically significant regardless of adjudication method. Agreement was higher when stroke type was certain, subtype was large vessel or cardioembolic, signs or symptoms lasted > 24 h, imaging showed a stroke, and when NIHSS was ≥3.

DISCUSSION

Central stroke adjudication caused the hazard ratio for a main secondary outcome in IRIS (stroke alone) to be higher and lose statistical significant compared with site review. The estimate of treatment effects were larger with the updated stroke definition. There may be benefit of central adjudication for events with specific features, such as shorter symptom duration or normal brain imaging.