Current Practices in Choosing Estimands and Sensitivity Analyses in Clinical Trials: Results of the ICH E9 Survey

Management of partial-thickness rotator cuff tears with autologous adipose-derived regenerative cells is safe and more effective than injection of corticosteroid

Symptomatic, partial-thickness rotator cuff tears (sPTRCT) are problematic. This study tested the hypothesis that management of sPTRCT with injection of fresh, uncultured, unmodified, autologous, adipose-derived regenerative cells (UA-ADRCs) is safe and more effective than injection of corticosteroid even in the long run. To this end, subjects who had completed a former randomized controlled trial were enrolled in the present study. At baseline these subjects had not responded to physical therapy treatments for at least 6 weeks, and were randomly assigned to receive respectively a single injection of UA-ADRCs (n = 11) or a single injection of methylprednisolone (n = 5). Efficacy was assessed using the ASES Total score, pain visual analogue scale (VAS), RAND Short Form-36 Health Survey and range of motion at 33.2 ± 1.0 (mean ± SD) and 40.6 ± 1.9 months post-treatment. Proton density, fat-saturated, T2-weighted MRI of the index shoulder was performed at both study visits. There were no greater risks connected with injection of UA-ADRCs than those connected with injection of corticosteroid. The subjects in the UA-ADRCs group showed statistically significantly higher mean ASES Total scores than the subjects in the corticosteroid group. The MRI scans at 6 months post-treatment allowed to "watch the UA-ADRCs at work".

Evaluating how clear the questions being investigated in randomised trials are: systematic review of estimands

OBJECTIVES

To evaluate how often the precise research question being addressed about an intervention (the estimand) is stated or can be determined from reported methods, and to identify what types of questions are being investigated in phase 2-4 randomised trials.

DESIGN

Systematic review of the clarity of research questions being investigated in randomised trials in 2020 in six leading general medical journals.

DATA SOURCE

PubMed search in February 2021.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

Phase 2-4 randomised trials, with no restrictions on medical conditions or interventions. Cluster randomised, crossover, non-inferiority, and equivalence trials were excluded.

MAIN OUTCOME MEASURES

Number of trials that stated the precise primary question being addressed about an intervention (ie, the primary estimand), or for which the primary estimand could be determined unambiguously from the reported methods using statistical knowledge. Strategies used to handle post-randomisation events that affect the interpretation or existence of patient outcomes, such as intervention discontinuations or uses of additional drug treatments (known as intercurrent events), and the corresponding types of questions being investigated.

RESULTS

255 eligible randomised trials were identified. No trials clearly stated all the attributes of the estimand. In 117 (46%) of 255 trials, the primary estimand could be determined from the reported methods. Intercurrent events were reported in 242 (95%) of 255 trials; but the handling of these could only be determined in 125 (49%) of 255 trials. Most trials that provided this information considered the occurrence of intercurrent events as irrelevant in the calculation of the treatment effect and assessed the effect of the intervention regardless (96/125, 77%)-that is, they used a treatment policy strategy. Four (4%) of 99 trials with treatment non-adherence owing to adverse events estimated the treatment effect in a hypothetical setting (ie, the effect as if participants continued treatment despite adverse events), and 19 (79%) of 24 trials where some patients died estimated the treatment effect in a hypothetical setting (ie, the effect as if participants did not die).

CONCLUSIONS

The precise research question being investigated in most trials is unclear, mainly because of a lack of clarity on the approach to handling intercurrent events. Clear reporting of estimands is necessary in trial reports so that all stakeholders, including clinicians, patients and policy makers, can make fully informed decisions about medical interventions.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42021238053.

Estimation of treatment effects in short-term depression studies. An evaluation based on the ICH E9(R1) estimands framework

Estimands aim to incorporate intercurrent events in design, data collection and estimation of treatment effects in clinical trials. Our aim was to understand what estimands may correspond to efficacy analyses commonly employed in clinical trials conducted before publication of ICH E9(R1). We re‐analysed six clinical trials evaluating a new anti‐depression treatment. We selected the following analysis methods—ANCOVA on complete cases, following last observation carried forward (LOCF) imputation and following multiple imputation; mixed‐models for repeated measurements without imputation (MMRM), MMRM following LOCF imputation and following jump‐to‐reference imputation; and pattern‐mixture mixed models. We included a principal stratum analysis based on the predicted subset of the study population who would not discontinue due to adverse events or lack of efficacy. We translated each analysis into the implicitly targeted estimand, and formulated corresponding clinical questions. We could map six estimands to analysis methods. The same analysis method could be mapped to more than one estimand. The major difference between estimands was the strategy for intercurrent events, with other attributes mostly the same across mapped estimands. The quantitative differences in MADRS10 population‐level summaries between the estimands were 4–8 points. Not all six estimands had a clinically meaningful interpretation. Only a few analyses would target the same estimand, hence only few could be used as sensitivity analyses. The fact that an analysis could estimate different estimands emphasises the importance of prospectively defining the estimands targeting the primary objective of a trial. The fact that an estimand can be targeted by different analyses emphasises the importance of prespecifying precisely the estimator for the targeted estimand.

A narrative review of estimands in drug development and regulatory evaluation: old wine in new barrels?

Background

An estimand defines the target of estimation for a clinical trial through specification of the treatment, target population, variable, population-level summary and of the strategies for intercurrent events. A carefully defined estimand aligns the clinical trial design and analysis with the scientific question of interest and adequately accounts for so-called intercurrent events. The ICH E9(R1) addendum suggests five estimand strategies. We evaluated to what extent current practice in drug development and regulatory assessment fits in the estimand framework.

Methods

We systematically evaluated what estimands, especially what strategies for intercurrent events are advised in European Medicines Agency disease guidelines, used in sponsors’ trials and additionally requested by the European Medicines Agency in assessment dossiers. We selected four therapeutic areas: nervous system, oncology, cardiovascular diseases and respiratory diseases. For each, we evaluated all disease guidelines with approved drugs, the dossiers of the most recently approved drugs matching the guidelines and corresponding regulatory questions.

Results

Strategies for intercurrent events were present in 18 (53%) of 34 guidelines, in all 34 sponsor documentations and in 15 (44%) of 34 sets of regulatory questions. Treatment policy was advised in 13 (38%) guidelines and was applied in 9 corresponding sponsor documentations. Of these 9, it was the sole strategy in 4 cases and accompanied by another strategy in 5 cases. Hypothetical strategy was not advised in guidelines. However, it was the leading strategy applied in 25 (74%) sponsor documentations. Composite strategy was advised in 3 (9%) guidelines and applied accompanied by another strategy in 2 corresponding sponsor documentations. While on treatment strategy was not advised in guidelines, but was applied in 2 sponsor documentations. Principal stratum strategy was advised in 2 guidelines but not applied in corresponding sponsor documentations. Of the regulatory questions, treatment policy was present in 2 cases (6%), hypothetical in 6 cases (18%), composite in 6 cases (18%) and while on treatment in 1 case (3%).

Conclusions

Estimand attributes are present in guidelines, sponsor documentations and regulatory questions, but not described as estimands. Treatment policy was most often advised in guidelines, but hypothetical was the leading strategy applied in sponsor documentations. Thus, results indicate not a full concordance between the regulatory target of estimation and what is actually estimated. The lack of concordance was mostly due to limitations in collection of intercurrent events data to enable a treatment policy strategy. There is, therefore, a need to better define estimands at the design stage and throughout the applications dossiers and assessment reports.

Empirical evaluation of the implementation of the EMA guideline on missing data in confirmatory clinical trials: Specification of mixed models for longitudinal data in study protocols

In confirmatory clinical trials, the prespecification of the primary analysis model is a universally accepted scientific principle to allow strict control of the type I error. Consequently, both the ICH E9 guideline and the European Medicines Agency (EMA) guideline on missing data in confirmatory clinical trials require that the primary analysis model is defined unambiguously. This requirement applies to mixed models for longitudinal data handling missing data implicitly. To evaluate the compliance with the EMA guideline, we evaluated the model specifications in those clinical study protocols from development phases II and III submitted between 2015 and 2018 to the Ethics Committee at Hannover Medical School under the German Medicinal Products Act, which planned to use a mixed model for longitudinal data in the confirmatory testing strategy. Overall, 39 trials from different types of sponsors and a wide range of therapeutic areas were evaluated. While nearly all protocols specify the fixed and random effects of the analysis model (95%), only 77% give the structure of the covariance matrix used for modeling the repeated measurements. Moreover, the testing method (36%), the estimation method (28%), the computation method (3%), and the fallback strategy (18%) are given by less than half the study protocols. Subgroup analyses indicate that these findings are universal and not specific to clinical trial phases or size of company. Altogether, our results show that guideline compliance is to various degrees poor and consequently, strict type I error rate control at the intended level is not guaranteed.

Strategic Shift of Statistical Review on Data Quality Assessment for New Drug Applications in China

Data quality is critical for clinical trials to obtain robust conclusions about drug safety and efficacy evaluation. Effective data quality evaluation has been one of the major obstacles to new drug approvals in China, which hinders innovation in drug discovery and development ultimately. To improve the data quality submitted for regulatory drug approval, the China Food and Drug Administration (CFDA) has issued serial official announcements and industry guidelines regarding improvement of the clinical trial data integrity and quality since 2015. These announcements and follow-up measures are shaping up the entire pharmaceutical industry in China. While data quality is being strongly emphasized more than ever at the trial conduction phase, it is still an open question about how to assess data quality effectively at the review stage. Thus, this article describes the authors' standpoints to assess the quality and integrity of submitted clinical data via statistical review methods including advanced risk-based approaches, which may bring significant impact to new drug applications and motivate sustainable development of innovative medicines in China.