Outcomes and statistical power in adult critical care randomized trials

Unexplained mortality differences between septic shock trials: a systematic analysis of population characteristics and control-group mortality rates

Purpose

Although the definition of septic shock has been standardized, some variation in mortality rates among clinical trials is expected. Insights into the sources of heterogeneity may influence the design and interpretation of septic shock studies. We set out to identify inclusion criteria and baseline characteristics associated with between-trial differences in control group mortality rates.

Methods

We conducted a systematic review of RCTs published between 2006 and 2018 that included patients with septic shock. The percentage of variance in control-group mortality attributable to study heterogeneity rather than chance was measured by I². The association between control-group mortality and population characteristics was estimated using linear mixed models and a recursive partitioning algorithm.

Results

Sixty-five septic shock RCTs were included. Overall control-group mortality was 38.6%, with significant heterogeneity (I² = 93%, P < 0.0001) and a 95% prediction interval of 13.5–71.7%. The mean mortality rate did not differ between trials with different definitions of hypotension, infection or vasopressor or mechanical ventilation inclusion criteria. Population characteristics univariately associated with mortality rates were mean Sequential Organ Failure Assessment score (standardized regression coefficient (β) = 0.57, P = 0.007), mean serum creatinine (β = 0.48, P = 0.007), the proportion of patients on mechanical ventilation (β = 0.61, P < 0.001), and the proportion with vasopressors (β = 0.57, P = 0.002). Combinations of population characteristics selected with a linear model and recursive partitioning explained 41 and 42%, respectively, of the heterogeneity in mortality rates.

Conclusions

Among 65 septic shock trials, there was a clinically relevant amount of heterogeneity in control group mortality rates which was explained only partly by differences in inclusion criteria and reported baseline characteristics.

Electronic supplementary material

The online version of this article (10.1007/s00134-018-5134-8) contains supplementary material, which is available to authorized users.

Reproducibility of clinical research in critical care: a scoping review

BACKGROUND

The ability to reproduce experiments is a defining principle of science. Reproducibility of clinical research has received relatively little scientific attention. However, it is important as it may inform clinical practice, research agendas, and the design of future studies.

METHODS

We used scoping review methods to examine reproducibility within a cohort of randomized trials examining clinical critical care research and published in the top general medical and critical care journals. To identify relevant clinical practices, we searched the New England Journal of Medicine, The Lancet, and JAMA for randomized trials published up to April 2016. To identify a comprehensive set of studies for these practices, included articles informed secondary searches within other high-impact medical and specialty journals. We included late-phase randomized controlled trials examining therapeutic clinical practices in adults admitted to general medical-surgical or specialty intensive care units (ICUs). Included articles were classified using a reproducibility framework. An original study was the first to evaluate a clinical practice. A reproduction attempt re-evaluated that practice in a new set of participants.

RESULTS

Overall, 158 practices were examined in 275 included articles. A reproduction attempt was identified for 66 practices (42%, 95% CI 33-50%). Original studies reported larger effects than reproduction attempts (primary endpoint, risk difference 16.0%, 95% CI 11.6-20.5% vs. 8.4%, 95% CI 6.0-10.8%, P = 0.003). More than half of clinical practices with a reproduction attempt demonstrated effects that were inconsistent with the original study (56%, 95% CI 42-68%), among which a large number were reported to be efficacious in the original study and to lack efficacy in the reproduction attempt (34%, 95% CI 19-52%). Two practices reported to be efficacious in the original study were found to be harmful in the reproduction attempt.

CONCLUSIONS

A minority of critical care practices with research published in high-profile journals were evaluated for reproducibility; less than half had reproducible effects.

Outcomes of Surgical Sepsis

BACKGROUND

Discussion of outcomes of surgical sepsis is no longer straightforward. Definitions of sepsis have changed recently and updated data are scant. Surgical patient populations are often heterogeneous; the patient population being considered must be described with precision. Traditional 30-d operative mortality may not be the most relevant outcome to consider. What should change or be the emphasis going forward?

METHODS

Review and synthesis of pertinent English-language literature.

RESULTS

Epidemiologic data are abundant for short-term outcomes of sepsis in general, but despite the fact that approximately 30% of patients with sepsis are surgical patients, sepsis outcome data for surgical patients are scant, especially for durations longer than 30 d, and essentially non-existent for patients defined under the new Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Interpretability of extant data is hampered by non-standard and changing definitions.

CONCLUSIONS

Sepsis and organ dysfunction may be decreasing in prevalence and magnitude among surgical patients, but terminology must be standardized to enhance the interpretability of data generated in the future. It behooves journal editors, reviewers, and authors to insist upon standardized definitions and rigorous study design and data interpretation. Longer term data (e.g., 90-d mortality as opposed to in-hospital or traditional 30-d mortality) will be needed to justify to payers the complex, expensive care that these patients require. There is an urgent need to redefine the research agenda for surgical infections.

Assessing Noninferiority in Treatment Trials for Severe Infectious Diseases: an Extension to the Entire Follow-Up Period Using a Cure-Death Multistate Model

In current and former clinical trials for the development of antibacterial drugs, various primary endpoints have been used, and treatment effects are evaluated mostly in noninferiority analyses at the end of follow-up, which varies between studies. A more convincing and highly patient-relevant statement would be a noninferiority assessment over the entire follow-up period with cure and death as coprimary endpoints, while preserving the desired alpha level for statistical testing. To account for the time-dynamic pattern of cure and death, we apply a cure-death multistate model. The endpoint of interest is "get cured and stay alive over time." Noninferiority between treatments over the entire follow-up period is studied by means of one-sided confidence bands provided by a flexible resampling technique. We illustrate the technique by applying it to a recently published study and establish noninferiority in being cured and alive over a time frame of interest for the entire population, patients with hospital-acquired pneumonia, but not for the subset of patients with ventilator-associated pneumonia. Our analysis improves the original results in the sense that our endpoint is more patient benefiting, a stronger noninferiority statement is demonstrated, and the time dependency of cure and death, competing events, and different follow-up times is captured. Multistate methodology combined with confidence bands adds a valuable statistical tool for clinical trials in the context of infection control. The framework is not restricted to the cure-death model but can be adapted to more complex multistate endpoints and equivalence or superiority analyses.

Stakeholder Engagement in Trial Design: Survey of Visitors to Critically Ill Patients Regarding Preferences for Outcomes and Treatment Options during Weaning from Mechanical Ventilation

RATIONALE

Stakeholder engagement in research is expected to provide unique insights, make research investments more accountable and transparent, and ensure that future research is applicable to patients and family members.

OBJECTIVES

To inform the design of a trial of strategies for weaning from mechanical ventilation, we sought to identify preferences of patient visitors regarding outcome and treatment measures.

METHODS

We conducted an interviewer-administered questionnaire of visitors of critically ill patients in two family waiting rooms serving three intensive care units (ICUs) in Toronto, Canada. Respondents rated the importance of general and ventilation-related outcomes in two hypothetical scenarios (before a first spontaneous breathing trial, and after a failed spontaneous breathing trial) and selected a preferred technique for the breathing trials. With regard to the patient they were visiting, respondents identified the most important outcome to them at ICU admission, during the ICU stay, and at ICU discharge.

MEASUREMENTS AND MAIN RESULTS

We analyzed 322 questionnaires (95.5% response rate). All outcomes were highly rated (average range: 7.82-9.74). Across scenarios, outcomes rated as most important were ICU and hospital survival (9.72, 9.70), avoiding complications (9.45), quality of life (9.394), patient comfort (9.393), and returning to previous living arrangements (9.31). Overall, the most important ventilation-related outcomes were being ventilator-free (8.95), avoiding reintubation (8.905), and passing a spontaneous breathing trial (8.903). Passing a spontaneous breathing trial assumed greater importance after an initial failed attempt. "Time to event" outcomes were less important to visitors. We did not identify a preferred spontaneous breathing trial technique. Although ICU survival was the most important outcome at ICU admission and during the ICU stay, visitors rated quality of life higher than hospital survival at ICU discharge.

CONCLUSIONS

Visitors to critically ill patients prioritized two general outcomes (ICU and hospital survival) and three ventilation-related outcomes (being ventilator free, avoiding reintubation, passing a spontaneous breathing trial), and valued avoiding complications, maintaining quality of life, comfort, and returning to previous living arrangements. The outcomes preferences of the survey respondents evolved temporally during the ICU stay.

Measurement Error Due to Patient Flow in Estimates of Intensive Care Unit Length of Stay

Clinical endpoints measured in terms of duration, such as intensive care unit (ICU) length of stay (LOS), are widely used in randomized clinical trials (RCTs) and observational research. In analyses of patient-level data from a recent RCT, in which ICU LOS was the primary endpoint, and in administrative data, we showed that additional ICU time is often accrued by patients after they are deemed ready for discharge. This "immutable" time (which cannot plausibly be altered by interventions under study) varies by day, week, and year, adding on average one-third of a day to total LOS. We then used statistical simulations, informed by the administrative data and RCT, to assess the impact of immutable time on the measurement and statistical comparison of patients' ICU LOS. These simulations demonstrated that immutable time combines with clinically necessary ICU time (neither of which is likely to be normally distributed) to produce overall LOS distributions that might either mask true treatment effects or suggest false treatment effects relative to analyses of time to discharge readiness. The extent and direction of bias were complex functions of the statistical method used, mortality rates and distributions, and the magnitude of immutable time relative to intervention-associated reductions in LOS.

Prehospital cooling to improve successful targeted temperature management after cardiac arrest: A randomized controlled trial

RATIONALE

Targeted temperature management (TTM) improves survival with good neurological outcome after out-of-hospital cardiac arrest (OHCA), but is delivered inconsistently and often with delay.

OBJECTIVE

To determine if prehospital cooling by paramedics leads to higher rates of 'successful TTM', defined as achieving a target temperature of 32-34°C within 6h of hospital arrival.

METHODS

Pragmatic RCT comparing prehospital cooling (surface ice packs, cold saline infusion, wristband reminders) initiated 5min after return of spontaneous circulation (ROSC) versus usual resuscitation and transport. The primary outcome was rate of 'successful TTM'; secondary outcomes were rates of applying TTM in hospital, survival with good neurological outcome, pulmonary edema in emergency department, and re-arrest during transport.

RESULTS

585 patients were randomized to receive prehospital cooling (n=279) or control (n=306). Prehospital cooling did not increase rates of 'successful TTM' (30% vs 25%; RR, 1.17; 95% confidence interval [CI] 0.91-1.52; p=0.22), but increased rates of applying TTM in hospital (68% vs 56%; RR, 1.21; 95%CI 1.07-1.37; p=0.003). Survival with good neurological outcome (29% vs 26%; RR, 1.13, 95%CI 0.87-1.47; p=0.37) was similar. Prehospital cooling was not associated with re-arrest during transport (7.5% vs 8.2%; RR, 0.94; 95%CI 0.54-1.63; p=0.83) but was associated with decreased incidence of pulmonary edema in emergency department (12% vs 18%; RR, 0.66; 95%CI 0.44-0.99; p=0.04).

CONCLUSIONS

Prehospital cooling initiated 5min after ROSC did not increase rates of achieving a target temperature of 32-34°C within 6h of hospital arrival but was safe and increased application of TTM in hospital.

Clinical research in critical care. Difficulties and perspectives

In the field of Intensive Care Medicine, improved survival has resulted from better patient care, the early detection of clinical deterioration, and the prevention of iatrogenic complications, while research on new treatments has been followed by an overwhelming number of disappointments. The origins of these fiascos must be sought in the conjunction of methodological problems - common to other disciplines - and the particularities of critically ill patients. The present article discusses both aspects and suggests some options for progress.

Applying Precision Medicine to Trial Design Using Physiology. Extracorporeal CO2 Removal for Acute Respiratory Distress Syndrome

In clinical trials of therapies for acute respiratory distress syndrome (ARDS), the average treatment effect in the study population may be attenuated because individual patient responses vary widely. This inflates sample size requirements and increases the cost and difficulty of conducting successful clinical trials. One solution is to enrich the study population with patients most likely to benefit, based on predicted patient response to treatment (predictive enrichment). In this perspective, we apply the precision medicine paradigm to the emerging use of extracorporeal CO₂ removal (ECCO₂R) for ultraprotective ventilation in ARDS. ECCO₂R enables reductions in tidal volume and driving pressure, key determinants of ventilator-induced lung injury. Using basic physiological concepts, we demonstrate that dead space and static compliance determine the effect of ECCO₂R on driving pressure and mechanical power. This framework might enable prediction of individual treatment responses to ECCO₂R. Enriching clinical trials by selectively enrolling patients with a significant predicted treatment response can increase treatment effect size and statistical power more efficiently than conventional enrichment strategies that restrict enrollment according to the baseline risk of death. To support this claim, we simulated the predicted effect of ECCO₂R on driving pressure and mortality in a preexisting cohort of patients with ARDS. Our computations suggest that restricting enrollment to patients in whom ECCO₂R allows driving pressure to be decreased by 5 cm H₂O or more can reduce sample size requirement by more than 50% without increasing the total number of patients to be screened. We discuss potential implications for trial design based on this framework.

Appropriate endpoints for evaluation of new antibiotic therapies for severe infections: a perspective from COMBACTE's STAT-Net

PURPOSE

In this era of rising antimicrobial resistance, slowly refilling antibiotic development pipelines, and an aging population, we need to ensure that randomized clinical trials (RCTs) determine the added benefit of new antibiotic agents effectively and in a valid way, especially for severely ill patients. Unfortunately, universally accepted endpoints for the evaluation of new drugs in severe infections are lacking.

METHODS

We review and discuss the current practices and challenges regarding endpoints in RCTs in this field and propose novel approaches.

RESULTS

Usual endpoints actually recommended for drug development suffer from important flaws. Mortality requires large sample size and only partly related to the infectious process. Clinical cure rate is highly subjective in critically ill patients where symptoms may be related to other intercurrent events. Currently, composite endpoints, hierarchical nested designs, and competing risks analysis seem to be the most promising new tools for designing and analyzing clinical trials in this area, although they require further validation.

CONCLUSION

Regulatory authorities, pharmaceutical companies, and clinicians need to agree on the most appropriate clinical endpoints for severe infections to ensure efficient approval of new, effective antibiotic agents.

The time-dependent "cure-death" model investigating two equally important endpoints simultaneously in trials treating high-risk patients with resistant pathogens

A variety of primary endpoints are used in clinical trials treating patients with severe infectious diseases, and existing guidelines do not provide a consistent recommendation. We propose to study simultaneously two primary endpoints, cure and death, in a comprehensive multistate cure-death model as starting point for a treatment comparison. This technique enables us to study the temporal dynamic of the patient-relevant probability to be cured and alive. We describe and compare traditional and innovative methods suitable for a treatment comparison based on this model. Traditional analyses using risk differences focus on one prespecified timepoint only. A restricted logrank-based test of treatment effect is sensitive to ordered categories of responses and integrates information on duration of response. The pseudo-value regression provides a direct regression model for examination of treatment effect via difference in transition probabilities. Applied to a topical real data example and simulation scenarios, we demonstrate advantages and limitations and provide an insight into how these methods can handle different kinds of treatment imbalances. The cure-death model provides a suitable framework to gain a better understanding of how a new treatment influences the time-dynamic cure and death process. This might help the future planning of randomised clinical trials, sample size calculations, and data analyses.

Effect sizes in ongoing randomized controlled critical care trials

Background

An important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians’ views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality.

Methods

We conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations.

Results

Estimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10–20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5–5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6–10%) (P = 0.27).

Conclusions

For some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-017-1726-x) contains supplementary material, which is available to authorized users.

An Exploratory Study of Long-Term Outcome Measures in Critical Illness Survivors: Construct Validity of Physical Activity, Frailty, and Health-Related Quality of Life Measures

OBJECTIVE

Functional capacity is commonly impaired after critical illness. We sought to clarify the relationship between objective measures of physical activity, self-reported measures of health-related quality of life, and clinician reported global functioning capacity (frailty) in such patients, as well as the impact of prior chronic disease status on these functional outcomes.

DESIGN

Prospective outcome study of critical illness survivors.

SETTING

Community-based follow-up.

PATIENTS

Participants of the Musculoskeletal Ultrasound Study in Critical Care: Longitudinal Evaluation Study (NCT01106300), invasively ventilated for more than 48 hours and on the ICU greater than 7 days.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Physical activity levels (health-related quality of life [36-item short-form health survey] and daily step counts [accelerometry]) were compared to norm-based or healthy control scores, respectively. Controls for frailty (Clinical Frailty Score) were non-morbid, age- and gender-matched to survivors. Ninety-one patients were recruited on ICU admission: 41 were contacted for post-discharge assessment, and data were collected from 30 (14 female; mean age, 55.3 yr [95% CI, 48.3-62.3]; mean post-discharge, 576 d [95% CI, 539-614]). Patients' mean daily step count (5,803; 95% CI, 4,792-6,813) was lower than that in controls (11,735; 95% CI, 10,928-12,542; p < 0.001), and lower in those with preexisting chronic disease than without (2,989 [95% CI, 776-5,201] vs 7,737 [95% CI, 4,907-10,567]; p = 0.013). Physical activity measures (accelerometry, health-related quality of life, and frailty) demonstrated good construct validity across all three tools. Step variability (from SD) was highly correlated with daily steps (r = 0.67; p < 0.01) demonstrating a potential boundary constraint.

CONCLUSIONS

Subjective and objective measures of physical activity are all informative in ICU survivors. They are all reduced 18 months post-discharge in ICU survivors, and worse in those with pre-admission chronic disease states. Investigating interventions to improve functional capacity in ICU survivors will require stratification based on the presence of premorbidity.

Primary Outcome Measures in Pediatric Septic Shock Trials: A Systematic Review

OBJECTIVE

To evaluate all published pediatric randomized controlled trials of patients with septic shock from any cause to examine the outcome measures used, the strengths and limitations of these measurements and whether the trial outcomes met feasibility criteria.

DATA SOURCES

We used a previously published database of pediatric critical care randomized controlled trials (PICUtrials.net) derived from searches of MEDLINE, EMBASE, LILACS, and CENTRAL.

STUDY SELECTION

We included randomized controlled trials of interventions to children admitted to a PICU with septic or dengue hemorrhagic shock which were published in English.

DATA EXTRACTION

Study characteristics and outcomes were retrieved by two independent reviewers with disagreement being resolved by a third reviewer. We defined feasibility as 1) recruitment of at least 90% of the targeted sample size and agreement of the observed outcome rate in the control group with the rate used for the sample size calculation to within 10% or 2) finding of a statistically significant difference in an interim or final analysis.

DATA SYNTHESIS

Nineteen of 321 identified articles were selected for review. Fourteen of 19 studies (74%) provided an a priori definition of their primary outcome measure in their "Methods section." Mortality rate was the most commonly reported primary outcome (8/14; 57%), followed by duration of shock (4/14; 29%) followed by organ failure (1/14; 7%). Only three of 19 included trials met feasibility criteria.

CONCLUSIONS

Our review found that use of mortality alone as a primary outcome in pediatric septic shock trials was associated with significant limitations and that long-term patient-centered outcomes were not used in this setting. Composite outcomes incorporating mortality and long-term outcomes should be explored for use in future pediatric septic shock trials.

A "placement of death" approach for studies of treatment effects on ICU length of stay

Length of stay in the intensive care unit (ICU) is a common outcome measure in randomized trials of ICU interventions. Because many patients die in the ICU, it is difficult to disentangle treatment effects on length of stay from effects on mortality; conventional analyses depend on assumptions that are often unstated and hard to interpret or check. We adapt a proposal from Rosenbaum that addresses concerns about selection bias and makes its assumptions explicit. A composite outcome is constructed that equals ICU length of stay if the patient was discharged alive and indicates death otherwise. Given any preference ordering that compares death with possible lengths of stay, we can estimate the intervention's effects on the composite outcome distribution. Sensitivity analyses can show results for different preference orderings. We discuss methods for constructing approximate confidence intervals for treatment effects on quantiles of the outcome distribution or on proportions of patients with outcomes preferable to various cutoffs. Strengths and weaknesses of possible primary significance tests (including the Wilcoxon-Mann-Whitney rank sum test and a heteroskedasticity-robust variant due to Brunner and Munzel) are reviewed. An illustrative example reanalyzes a randomized trial of an ICU staffing intervention.

Different research designs and their characteristics in intensive care

Different research designs have various advantages and limitations inherent to their main characteristics. Knowledge of the proper use of each design is of great importance to understanding the applicability of research findings to clinical epidemiology. In intensive care, a hierarchical classification of designs can often be misleading if the characteristics of the design in this context are not understood. One must therefore be alert to common problems in randomized clinical trials and systematic reviews/meta-analyses that address clinical issues related to the care of the critically ill patient.

Variation in the Contents of Sepsis Bundles and Quality Measures. A Systematic Review

RATIONALE

Sepsis contributes to one in every two to three inpatient hospital deaths. Early recognition and treatment are instrumental in reducing mortality, yet there are substantial quality gaps. Sepsis bundles containing quality metrics are often used in efforts to improve outcomes. Several prominent organizations have published their own bundles, but there are few head-to-head comparisons of content.

OBJECTIVES

We sought to determine the degree of agreement on component elements of sepsis bundles and the associated timing goals for completion of each element. We additionally sought to evaluate the amount of variation between metrics associated with bundles.

METHODS

We reviewed the components of and level of agreement among several sepsis resuscitation and management bundles. We compared the individual bundle elements, together with their associated goals and metrics. We performed a systematic review (PubMed 2008-2015) and searched publically available online content, supplemented by interviews with key informants, to identify eight distinct bundles. Bundles are presented as current as of April 2015.

MEASUREMENTS AND MAIN RESULTS

Broadly, elements of care covered early resuscitation and short-term management. Bundles varied from 6 to 10 elements, and there were 12 distinct elements listed across all bundles. Only lactate collection and broad-spectrum antibiotics were common to all eight bundles, although there were seven elements included in at least 75% of the bundles. Timing goals for the collection of lactate and antibiotic administration varied among bundles from within 1 to 6 hours of diagnosis or admission. Notably, no bundle included metrics evaluating timeliness or completeness of sepsis recognition.

CONCLUSIONS

There is a lack of consensus on component elements and timing goals across highly recognized sepsis bundles. These differences highlight an urgent need for comparative effectiveness research to guide future implementation and for metrics to evaluate progress. None of the widely instituted bundles include metrics to evaluate sepsis recognition or diagnostic accuracy.

Individualizing endpoints in randomized clinical trials to better inform individual patient care: the TARGET proposal

In practice, critical care practitioners individualize treatments and goals of care for each patient in light of that patient’s acute and chronic pathophysiology, as well as their beliefs and values. Yet critical care researchers routinely measure one endpoint for all patients during randomized clinical trials (RCTs), eschewing any such individualization. More recent methodology work has explored the possibility that enrollment criteria in RCTs can be individualized, as can data analysis plans. Here we propose that the specific endpoints of a RCT can be individualized—that is, different patients within a single RCT might have different secondary endpoints measured. If done rigorously and objectively, based on pre-randomization data, such individualization of endpoints may improve the bedside usefulness of information obtained during a RCT, while perhaps also improving the power and efficiency of any RCT. We discuss the theoretical underpinnings of this proposal in light of related innovations in RCT design such as sliding dichotomies. We discuss what a full elaboration of such individualization would require, and outline a pragmatic initial step towards the use of “individualized secondary endpoints” in a large RCT evaluating optimal enteral nutrition targets in the critically ill.

Randomized controlled trials - a matter of design

Randomized controlled trials (RCTs) are the hallmark of evidence-based medicine and form the basis for translating research data into clinical practice. This review summarizes commonly applied designs and quality indicators of RCTs to provide guidance in interpreting and critically evaluating clinical research data. It further reflects on the principle of equipoise and its practical applicability to clinical science with an emphasis on critical care and neurological research. We performed a review of educational material, review articles, methodological studies, and published clinical trials using the databases MEDLINE, PubMed, and ClinicalTrials.gov. The most relevant recommendations regarding design, conduction, and reporting of RCTs may include the following: 1) clinically relevant end points should be defined a priori, and an unbiased analysis and report of the study results should be warranted, 2) both significant and nonsignificant results should be objectively reported and published, 3) structured study design and performance as indicated in the Consolidated Standards of Reporting Trials statement should be employed as well as registration in a public trial database, 4) potential conflicts of interest and funding sources should be disclaimed in study report or publication, and 5) in the comparison of experimental treatment with standard care, preplanned interim analyses during an ongoing RCT can aid in maintaining clinical equipoise by assessing benefit, harm, or futility, thus allowing decision on continuation or termination of the trial.

Implications of Heterogeneity of Treatment Effect for Reporting and Analysis of Randomized Trials in Critical Care

Randomized clinical trials (RCTs) are conducted to guide clinicians' selection of therapies for individual patients. Currently, RCTs in critical care often report an overall mean effect and selected individual subgroups. Yet work in other fields suggests that such reporting practices can be improved. Specifically, this Critical Care Perspective reviews recent work on so-called "heterogeneity of treatment effect" (HTE) by baseline risk and extends that work to examine its applicability to trials of acute respiratory failure and severe sepsis. Because patients in RCTs in critical care medicine-and patients in intensive care units-have wide variability in their risk of death, these patients will have wide variability in the absolute benefit that they can derive from a given therapy. If the side effects of the therapy are not perfectly collinear with the treatment benefits, this will result in HTE, where different patients experience quite different expected benefits of a therapy. We use simulations of RCTs to demonstrate that such HTE could result in apparent paradoxes, including: (1) positive trials of therapies that are beneficial overall but consistently harm or have little benefit to low-risk patients who met enrollment criteria, and (2) overall negative trials of therapies that still consistently benefit high-risk patients. We further show that these results persist even in the presence of causes of death unmodified by the treatment under study. These results have implications for reporting and analyzing RCT data, both to better understand how our therapies work and to improve the bedside applicability of RCTs. We suggest a plan for measurement in future RCTs in the critically ill.

Event-rate and delta inflation when evaluating mortality as a primary outcome from randomized controlled trials of nutritional interventions during critical illness: a systematic review

BACKGROUND

There is a lack of high-quality evidence that proves that nutritional interventions during critical illness reduce mortality.

OBJECTIVES

We evaluated whether power calculations for randomized controlled trials (RCTs) of nutritional interventions that used mortality as the primary outcome were realistic, and whether overestimation was systematic in the studies identified to determine whether this was due to overestimates of event rate or delta.

DESIGN

A systematic review of the literature between 2005 and 2015 was performed to identify RCTs of nutritional interventions administered to critically ill adults that had mortality as the primary outcome. Predicted event rate (predicted mortality during the control), predicted mortality during intervention, predicted delta (predicted difference between mortality during the control and intervention), actual event rate (observed mortality during control), observed mortality during intervention, and actual delta (difference between observed mortality during the control and intervention) were recorded. The event-rate gap (predicted event rate minus observed event rate), the delta gap (predicted delta minus observed delta), and the predicted number needed to treat were calculated. Data are shown as median (range).

RESULTS

Fourteen articles were extracted, with power calculations provided for 10 studies. The predicted event rate was 29.9% (20.0–52.4%), and the predicted delta was 7.9% (3.0–20.0%). If the study hypothesis was proven correct then, on the basis of the power calculations, the number needed to treat would have been 12.7 (5.0–33.3) patients. The actual event rate was 25.3% (6.1–50.0%), the observed mortality during the intervention was 24.4% (6.3–39.7%), and the actual delta was 0.5% (−10.2–10.3%), such that the event-rate gap was 2.6% (−3.9–23.7%) and delta gap was 7.5% (3.2–25.2%).

CONCLUSIONS

Overestimates of delta occur frequently in RCTs of nutritional interventions in the critically ill that are powered to determine a mortality benefit. Delta inflation may explain the number of "negative" studies in this field of research.

Designing phase 3 sepsis trials: application of learned experiences from critical care trials in acute heart failure

Substantial attention and resources have been directed to improving outcomes of patients with critical illnesses, in particular sepsis, but all recent clinical trials testing various interventions or strategies have failed to detect a robust benefit on mortality. Acute heart failure is also a critical illness, and although the underlying etiologies differ, acute heart failure and sepsis are critical care illnesses that have a high mortality in which clinical trials have been difficult to conduct and have not yielded effective treatments. Both conditions represent a syndrome that is often difficult to define with a wide variation in patient characteristics, presentation, and standard management across institutions. Referring to past experiences and lessons learned in acute heart failure may be informative and help frame research in the area of sepsis. Academic heart failure investigators and industry have worked closely with regulators for many years to transition acute heart failure trials away from relying on dyspnea assessments and all-cause mortality as the primary measures of efficacy, and recent trials have been designed to assess novel clinical composite endpoints assessing organ dysfunction and mortality while still assessing all-cause mortality as a separate measure of safety. Applying the lessons learned in acute heart failure trials to severe sepsis and septic shock trials might be useful to advance the field. Novel endpoints beyond all-cause mortality should be considered for future sepsis trials.

Trials in adult critical care that show increased mortality of the new intervention: Inevitable or preventable mishaps?

Several promising therapies assessed in the adult critically ill in large, multicenter randomized controlled trials (RCTs) were associated with significantly increased mortality in the intervention arms. Our hypothesis was that there would be wide ranges in sponsorship (industry or not), type(s) of intervention(s), use of DSMBs, presence of interim analyses and early stopping rules, absolute risk increase (ARI), and whether or not adequate prior proof-of-principle Phase II studies were done of RCTs that found increased mortality rates of the intervention compared to control groups. We reviewed RCTs that showed a statistically significant increased mortality rate in the intervention compared to control group(s). We recorded source of sponsorship, sample sizes, types of interventions, mortality rates, ARI (as well as odds ratios, relative risks and number needed to harm), whether there were pre-specified interim analyses and early stopping rules, and whether or not there were prior proof-of-principle (also known as Phase II) RCTs. Ten RCTs (four industry sponsored) of many interventions (high oxygen delivery, diaspirin cross-linked hemoglobin, growth hormone, methylprednisolone, hetastarch, high-frequency oscillation ventilation, intensive insulin, NOS inhibition, and beta-2 adrenergic agonist, TNF-α receptor) included 19,126 patients and were associated with wide ranges of intervention versus control group mortality rates (25.7–59 %, mean 29.9 vs 17–49 %, mean 25 %, respectively) yielding ARIs of 2.6–29 % (mean 5 %). All but two RCTs had pre-specified interim analyses, and seven RCTs were stopped early. All RCTs were preceded by published proof-of-principle RCT(s), two by the same group. Seven interventions (except diaspirin cross-linked hemoglobin and the NOS inhibitor) were available for use clinically at the time of the pivotal RCT. Common, clinically available interventions used in the critically ill were associated with increased mortality in large, pivotal RCTs even though safety was often addressed by interim analyses and early stopping rules.

Identifying low-value clinical practices in critical care medicine: protocol for a scoping review

INTRODUCTION

Reducing unnecessary, low-value clinical practice (ie, de-adoption) is key to improving value for money in healthcare, especially among patients admitted to intensive care units (ICUs) where resource consumption exceeds other medical and surgical populations. Research suggests that low-value clinical practices are common in medicine, however systematically and objectively identifying them is a widely cited barrier to de-adoption. We will conduct a scoping review to identify low-value clinical practices in adult critical care medicine that are candidates for de-adoption.

METHODS AND ANALYSIS

We will systematically search the literature to identify all randomised controlled trials or systematic reviews that focus on diagnostic or therapeutic interventions in adult patients admitted to medical, surgical or specialty ICUs, and are published in 3 general medical journals with the highest impact factor (New England Journal of Medicine, The Lancet, Journal of the American Medical Association). 2 investigators will independently screen abstracts and full-text articles against inclusion criteria, and extract data from included citations. Included citations will be classified according to whether or not they represent a repeat examination of the given research question (ie, replication research), and whether the results are similar or contradictory to the original study. Studies with contradictory results will determine clinical practices that are candidates for de-adoption.

ETHICS AND DISSEMINATION

Our scoping review will use robust methodology to systematically identify a list of clinical practices in adult critical care medicine with evidence supporting their de-adoption. In addition to adding to advancing the study of de-adoption, this review may also serve as the launching point for clinicians and researchers in critical care to begin reducing the number of low-value clinical practices. Dissemination of these results to relevant stakeholders will include tailored presentations at local, national and international meetings, and publication of a manuscript. Ethical approval is not required for this study.

When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study

Introduction

In critical care observational studies, when clinicians administer different treatments to sicker patients, any treatment comparisons will be confounded by differences in severity of illness between patients. We sought to investigate the extent that observational studies assessing treatments are at risk of incorrectly concluding such treatments are ineffective or even harmful due to inadequate risk adjustment.

Methods

We performed Monte Carlo simulations of observational studies evaluating the effect of a hypothetical treatment on mortality in critically ill patients. We set the treatment to have either no association with mortality or to have a truly beneficial effect, but more often administered to sicker patients. We varied the strength of the treatment’s true effect, strength of confounding, study size, patient population, and accuracy of the severity of illness risk-adjustment (area under the receiver operator characteristics curve, AUROC). We measured rates in which studies made inaccurate conclusions about the treatment’s true effect due to confounding, and the measured odds ratios for mortality for such false associations.

Results

Simulated observational studies employing adequate risk-adjustment were generally able to measure a treatment’s true effect. As risk-adjustment worsened, rates of studies incorrectly concluding the treatment provided no benefit or harm increased, especially when sample size was large (n = 10,000). Even in scenarios of only low confounding, studies using the lower accuracy risk-adjustors (AUROC < 0.66) falsely concluded that a beneficial treatment was harmful. Measured odds ratios for mortality of 1.4 or higher were possible when the treatment’s true beneficial effect was an odds ratio for mortality of 0.6 or 0.8.

Conclusions

Large observational studies confounded by severity of illness have a high likelihood of obtaining incorrect results even after employing conventionally “acceptable” levels of risk-adjustment, with large effect sizes that may be construed as true associations. Reporting the AUROC of the risk-adjustment used in the analysis may facilitate an evaluation of a study’s risk for confounding.

Electronic supplementary material

The online version of this article (doi:10.1186/s13054-015-0923-8) contains supplementary material, which is available to authorized users.