Systematic review of statistical methods for safety data in malaria chemoprevention in pregnancy trials

Statistical methods leveraging the hierarchical structure of adverse events for signal detection in clinical trials: a scoping review of the methodological literature

BACKGROUND

In randomised controlled trials with efficacy-related primary outcomes, adverse events are collected to monitor potential intervention harms. The analysis of adverse event data is challenging, due to the complex nature of the data and the large number of unprespecified outcomes. This is compounded by a lack of guidance on best analysis approaches, resulting in widespread inadequate practices and the use of overly simplistic methods; leading to sub-optimal exploitation of these rich datasets. To address the complexities of adverse events analysis, statistical methods are proposed that leverage existing structures within the data, for instance by considering groupings of adverse events based on biological or clinical relationships.

METHODS

We conducted a methodological scoping review of the literature to identify all existing methods using structures within the data to detect signals for adverse reactions in a trial. Embase, MEDLINE, Scopus and Web of Science databases were systematically searched. We reviewed the analysis approaches of each method, extracted methodological characteristics and constructed a narrative summary of the findings.

RESULTS

We identified 18 different methods from 14 sources. These were categorised as either Bayesian approaches (n=11), which flagged events based on posterior estimates of treatment effects, or error controlling procedures (n=7), which flagged events based on adjusted p-values while controlling for some type of error rate. We identified 5 defining methodological characteristics: the type of outcomes considered (e.g. binary outcomes), the nature of the data (e.g. summary data), the timing of the analysis (e.g. final analysis), the restrictions on the events considered (e.g. rare events) and the grouping systems used.

CONCLUSIONS

We found a large number of analysis methods that use the group structures of adverse events. Continuous methodological developments in this area highlight the growing awareness that better practices are needed. The use of more adequate analysis methods could help trialists obtain a better picture of the safety-risk profile of an intervention. The results of this review can be used by statisticians to better understand the current methodological landscape and identify suitable methods for data analysis - although further research is needed to determine which methods are best suited and create adequate recommendations.

Public perspective on potential treatment intervention harm in clinical trials-terminology and communication

BACKGROUND

Randomised controlled trials (RCTs) are typically designed to determine beneficial intervention effects. In addition, an important aspect of every trial is to collect data on any potential harmful effects, with the aim of ensuring that the benefit-risk balance is appropriate. The language used by trialists to describe these potential harmful effects is inconsistent. In pharmacological trials, researchers collect adverse events; when a causal relationship is suspected adverse events are further classified as adverse reactions. Academic researchers have moved to collectively refer to these as harm outcomes; the pharmaceutical industry refer to these events as safety outcomes. In trials of complex interventions, phrases such as unintended consequences or effects are used. With the inconsistent use of terminology by researchers and the potential benefits to be gained from harmonising communications, we sought public opinion on terminology used to describe harmful effects and how these outcomes are communicated in the scientific literature, as well as in public facing material on medications.

METHODS

We held two in-person public involvement meetings with public partners, in London and Aberdeen in 2023. Both meetings followed a pre-specified format. We provided a background to the topic including the information researchers collect on potential harms in clinical trials and shared examples on how this information gets presented in practice. We then discussed public partners' perspectives on terminology used and communication of intervention harm in academic journals and in public facing materials. A summary of these discussions and the main topics raised by public partners are presented.

RESULTS

Public partners endorsed the use of different terms for different situations, preferring the use of 'side-effect' across all contexts and reserving the use of 'harm' to indicate more severe events. Generally, public partners were happy with the type of information presented in public facing materials but discussions revealed that presentation of information on public NHS websites led to misconceptions about harm.

CONCLUSION

This work provides a starting point on preferred terminology by patients and the public to describe potential harmful intervention effects. Whilst researchers have tried to seek agreement, public partners endorsed use of different terms for different situations. We highlight some key areas for improvement in public facing materials that are necessary to avoid miscommunication and incorrect perception of harm.

An Overview of Regression Models for Adverse Events Analysis

Over the last few years, several review articles described the adverse events analysis as sub-optimal in clinical trials. Indeed, the context surrounding adverse events analyses often imply an overwhelming number of events, a lack of power to find associations, but also a lack of specific training regarding those complex data. In randomized controlled trials or in observational studies, comparing the occurrence of adverse events according to a covariable of interest (e.g., treatment) is a recurrent question in the analysis of drug safety data, and adjusting other important factors is often relevant. This article is an overview of the existing regression models that may be considered to compare adverse events and to discuss model choice regarding the characteristics of the adverse events of interest. Many dimensions may be relevant to compare the adverse events between patients, (e.g., timing, recurrence, and severity). Recent efforts have been made to cover all of them. For chronic treatments, the occurrence of intercurrent events during the patient follow-up usually needs the modeling approach to be adapted (at least with regard to their interpretation). Moreover, analysis based on regression models should not be limited to the estimation of relative effects. Indeed, absolute risks stemming from the model should be presented systematically to help the interpretation, to validate the model, and to encourage comparison of studies.

Achieving white blood cell equity: are the safety profiles of biosimilar and reference pegfilgrastims comparable?

Biosimilars can provide choices for patients and may provide cost savings; however, their uptake has been slow in the USA, in part due to limited knowledge. To provide additional confidence in US pegfilgrastim biosimilars, this narrative review compared the safety profiles of biosimilar pegfilgrastims, currently approved or filed for approval in the USA, with the EU- and US-approved reference pegfilgrastims. Headache and bone pain were common to biosimilars and reference products and occurred at a similar incidence. Clinical trial data on the safety profiles of biosimilar pegfilgrastims and reference products have demonstrated similarity and comparability, with no unexpected safety outcomes. Overall, the safety profiles of biosimilar pegfilgrastims and reference pegfilgrastims demonstrated a high degree of similarity and comparability.

Comparison of statistical methods for the analysis of recurrent adverse events in the presence of non-proportional hazards and unobserved heterogeneity: a simulation study

Background

In preventive drug trials such as intermittent preventive treatment for malaria prevention during pregnancy (IPTp), where there is repeated treatment administration, recurrence of adverse events (AEs) is expected. Challenges in modelling the risk of the AEs include accounting for time-to-AE and within-patient-correlation, beyond the conventional methods. The correlation comes from two sources; (a) individual patient unobserved heterogeneity (i.e. frailty) and (b) the dependence between AEs characterised by time-dependent treatment effects. Potential AE-dependence can be modelled via time-dependent treatment effects, event-specific baseline and event-specific random effect, while heterogeneity can be modelled via subject-specific random effect. Methods that can improve the estimation of both the unobserved heterogeneity and treatment effects can be useful in understanding the evolution of risk of AEs, especially in preventive trials where time-dependent treatment effect is expected.

Methods

Using both a simulation study and the Chloroquine for Malaria in Pregnancy (NCT01443130) trial data to demonstrate the application of the models, we investigated whether the lognormal shared frailty models with restricted cubic splines and non-proportional hazards (LSF-NPH) assumption can improve estimates for both frailty variance and treatment effect compared to the conventional inverse Gaussian shared frailty model with proportional hazard (ISF-PH), in the presence of time-dependent treatment effects and unobserved patient heterogeneity. We assessed the bias, precision gain and coverage probability of 95% confidence interval of the frailty variance estimates for the models under varying known unobserved heterogeneity, sample sizes and time-dependent effects.

Results

The ISF-PH model provided a better coverage probability of 95% confidence interval, less bias and less precise frailty variance estimates compared to the LSF-NPH models. The LSF-NPH models yielded unbiased hazard ratio estimates at the expense of imprecision and high mean square error compared to the ISF-PH model.

Conclusion

The choice of the shared frailty model for the recurrent AEs analysis should be driven by the study objective. Using the LSF-NPH models is appropriate if unbiased hazard ratio estimation is of primary interest in the presence of time-dependent treatment effects. However, ISF-PH model is appropriate if unbiased frailty variance estimation is of primary interest.

Trial registration

ClinicalTrials.gov; NCT01443130

Effect of adverse events on non-adherence and study non-completion in malaria chemoprevention during pregnancy trial: A nested case control study

Background

In drug trials, adverse events (AEs) burden can induce treatment non-adherence or discontinuation. The non-adherence and discontinuation induce selection bias, affecting drug safety interpretation. Nested case-control (NCC) study can efficiently quantify the impact of the AEs, although choice of sampling approach is challenging. We investigated whether NCC study with incidence density sampling is more efficient than NCC with path sampling under conditional logistic or weighted Cox models in assessing the effect of AEs on treatment non-adherence and participation in preventive antimalarial drug during pregnancy trial.

Methods

Using data from a trial of medication to prevent malaria in pregnancy that randomized 600 women to receive chloroquine or sulfadoxine-pyrimethamine during pregnancy, we conducted a NCC study assessing the role of prospectively collected AEs, as exposure of interest, on treatment non-adherence and study non-completion. We compared estimates from NCC study with incidence density against those from NCC with path sampling under conditional logistic and weighted Cox models.

Results

Out of 599 women with the outcomes of interest, 474 (79%) experienced at least one AE before delivery. For conditional logistic model, the hazard ratio for the effect of AE occurrence on treatment non-adherence was 0.70 (95% CI: 0.42, 1.17; p = 0.175) under incidence density sampling and 0.68 (95% CI: 0.41, 1.13; p = 0.137) for path sampling. For study non-completion, the hazard ratio was 1.02 (95% CI: 0.56, 1.83; p = 0.955) under incidence density sampling and 0.85 (95% CI: 0.45, 1.60; p = 0.619) under path sampling. We obtained similar hazard ratios and standard errors under incidence density sampling and path sampling whether weighted Cox or conditional logistic models were used.

Conclusion

NCC with incidence density sampling and NCC with path sampling are practically similar in efficiency whether conditional logistic or weighted Cox analytical methods although path sampling uses more unique controls to achieve the similar estimates.

Trial registration

ClinicalTrials.gov: NCT01443130.

Improving the analysis of adverse event data in randomised controlled trials

Analysing treatment harm is vital but problematic with the relatively small sample sizes afforded in randomised controlled trials (RCTs). Good analysis practice for efficacy outcomes are well established but there has been minimal progress for the analysis of adverse events (AEs). In this commentary we examine four key issues for AE analysis. Namely, why harm data in RCTs is undervalued, why AE analysis is difficult, what aspects of current analysis practice are unsatisfactory, and the challenges for selection and interpretation of AEs results in publications. We discuss how the value of harm data from RCTs could be better realised by reframing the research question to one for detecting signals of adverse reactions. We align established good statistical practice to current unsatisfactory practice. We encourage use of Bayesian analyses to enable cumulative assessment of harm across trial research phases and discourage selecting AEs to report based on arbitrary rules. We propose comprehendible summaries to be based on core outcome sets, serious and pre-specified events, and events leading to discontinuation. Analysis of AEs in contemporary clinical trials needs attention to progress. In the following we have outlined immediate, mid and longer-term strategies for trialists to adopt to support a change in practice.

Joint modelling of multivariate longitudinal clinical laboratory safety outcomes, concomitant medication and clinical adverse events: application to artemisinin-based treatment during pregnancy clinical trial

Background

In drug trials, clinical adverse events (AEs), concomitant medication and laboratory safety outcomes are repeatedly collected to support drug safety evidence. Despite the potential correlation of these outcomes, they are typically analysed separately, potentially leading to misinformation and inefficient estimates due to partial assessment of safety data. Using joint modelling, we investigated whether clinical AEs vary by treatment and how laboratory outcomes (alanine amino-transferase, total bilirubin) and concomitant medication are associated with clinical AEs over time following artemisinin-based antimalarial therapy.

Methods

We used data from a trial of artemisinin-based treatments for malaria during pregnancy that randomized 870 women to receive artemether–lumefantrine (AL), amodiaquine–artesunate (ASAQ) and dihydroartemisinin–piperaquine (DHAPQ). We fitted a joint model containing four sub-models from four outcomes: longitudinal sub-model for alanine aminotransferase, longitudinal sub-model for total bilirubin, Poisson sub-model for concomitant medication and Poisson sub-model for clinical AEs. Since the clinical AEs was our primary outcome, the longitudinal sub-models and concomitant medication sub-model were linked to the clinical AEs sub-model via current value and random effects association structures respectively. We fitted a conventional Poisson model for clinical AEs to assess if the effect of treatment on clinical AEs (i.e. incidence rate ratio (IRR)) estimates differed between the conventional Poisson and the joint models, where AL was reference treatment.

Results

Out of the 870 women, 564 (65%) experienced at least one AE. Using joint model, AEs were associated with the concomitant medication (log IRR 1.7487; 95% CI: 1.5471, 1.9503; p < 0.001) but not the total bilirubin (log IRR: -0.0288; 95% CI: − 0.5045, 0.4469; p = 0.906) and alanine aminotransferase (log IRR: 0.1153; 95% CI: − 0.0889, 0.3194; p = 0.269). The Poisson model underestimated the effects of treatment on AE incidence such that log IRR for ASAQ was 0.2118 (95% CI: 0.0082, 0.4154; p = 0.041) for joint model compared to 0.1838 (95% CI: 0.0574, 0.3102; p = 0.004) for Poisson model.

Conclusion

We demonstrated that although the AEs did not vary across the treatments, the joint model yielded efficient AE incidence estimates compared to the Poisson model. The joint model showed a positive relationship between the AEs and concomitant medication but not with laboratory outcomes.

Trial registration

ClinicalTrials.gov: NCT00852423

Improving analysis practice of continuous adverse event outcomes in randomised controlled trials - a distributional approach

BACKGROUND

Randomised controlled trials (RCTs) provide valuable information for developing harm profiles but current analysis practices to detect between-group differences are suboptimal. Drug trials routinely screen continuous clinical and biological data to monitor participant harm. These outcomes are regularly dichotomised into abnormal/normal values for analysis. Despite the simplicity gained for clinical interpretation, it is well established that dichotomising outcomes results in a considerable reduction in information and thus statistical power. We propose an automated procedure for the routine implementation of the distributional method for the dichotomisation of continuous outcomes proposed by Peacock and Sauzet, which retains the precision of the comparison of means.

METHODS

We explored the use of a distributional approach to compare differences in proportions based on the comparison of means which retains the power of the latter. We applied this approach to the screening of clinical and biological data as a means to detect 'signals' for potential adverse drug reactions (ADRs). Signals can then be followed-up in further confirmatory studies. Three distributional methods suitable for different types of distributions are described. We propose the use of an automated approach using the observed data to select the most appropriate distribution as an analysis strategy in a RCT setting for multiple continuous outcomes. We illustrate this approach using data from three RCTs assessing the efficacy of mepolizumab in asthma or COPD. Published reference ranges were used to define the proportions of participants with abnormal values for a subset of 10 blood tests. The between-group distributional and empirical differences in proportions were estimated for each blood test and compared.

RESULTS

Within trials, the distributions varied across the 10 outcomes demonstrating value in a practical approach to selecting the distributional method in the context of multiple adverse event outcomes. Across trials, there were three outcomes where the method chosen by the automated procedure varied for the same outcome. The distributional approach identified three signals (eosinophils, haematocrit, and haemoglobin) compared to only one when using the Fisher's exact test (eosinophils) and two identified by use of the 95% confidence interval for the difference in proportions (eosinophils and potassium).

CONCLUSION

When dichotomisation of continuous adverse event outcomes aids clinical interpretation, we advocate use of a distributional approach to retain statistical power. Methods are now easy to implement. Retaining information is especially valuable in the context of the analysis of adverse events in RCTs. The routine implementation of this automated approach requires further evaluation.

Transition of radical, preventive and presumptive treatment regimens for malaria in China: a systematic review

BACKGROUND

Globally, malaria is still a major public health challenge. Drug-based treatment is the primary intervention in malaria control and elimination. However, optimal use of mass or targeted treatments remains unclear. A variety of radical, preventive and presumptive treatment regimens have been administrated in China and a systematic review was conducted to evaluate effectiveness, and discuss experiences, limitations, and lessons learnt in relation to the use of these regimens.

METHODS

The search for information includes both paper documents, such as books, malaria control annals and guidelines for malaria prevention and treatment, as well as three computer-based databases in Chinese (CNKI, WanFangdata and Xueshu.baidu) and two databases in English (PubMed and Google Scholar), to identify original articles and reports associated with drug administration for malaria in China.

RESULTS

Starting from hyperendemicity to elimination of malaria in China, a large number of radical, preventive and presumptive treatment regimens had been tried. Those effective regimens were scaled up for malaria control and elimination programmes in China. Between 1949 and 1959, presumptive treatment with available anti-malarial drugs was given to people with enlarged spleens and those who had symptoms suggestive of malaria within the last 6 months. Between 1960 and 1999, mass drug administration (MDA) was given for preventive and radical treatment. Between 2000 and 2009, the approach was more targeted, and drugs were administed only to prevent malaria infection in those at high risk of exposure and those who needed radical treatment for suspected malaria. Presumptive therapy was only given to febrile patients. From 2010, the malaria programme changed into elimination phase, radical treatment changed to target individuals with confirmed either Plasmodium vivax or Plasmodium ovale within the last year. Preventive treatment was given to those who will travel to other endemic countries. Presumptive treatment was normally not given during this elimination phase. All cases of suspected were confirmed by either microscopy or rapid diagnosis tests for malaria antigens before drugs were administered. The engagement of the broader community ensured high coverage of these drug-based interventions, and the directly-observed therapy improved patient safety during drug administration.

CONCLUSION

A large number of radical, preventive and presumptive treatment regimens for malaria had been tried in China with reported success, but the impact of drug-based interventions has been difficult to quantify because they are just a part of an integrated malaria control strategy. The historical experiences of China suggest that intervention trials should be done by the local health facilities with community involvement, and a local decision is made according to their own trial results.

Advantages of visualisations to evaluate and communicate adverse event information in randomised controlled trials

BACKGROUND

Randomised controlled trials (RCTs) provide valuable information and inform the development of harm profiles of new treatments. Harms are typically assessed through the collection of adverse events (AEs). Despite AEs being routine outcomes collected in trials, analysis and reporting of AEs in journal articles are continually shown to be suboptimal. One key challenge is the large volume of AEs, which can make evaluation and communication problematic. Prominent practice is to report frequency tables of AEs by arm. Visual displays offer an effective solution to assess and communicate complex information; however, they are rarely used and there is a lack of practical guidance on what and how to visually display complex AE data.

METHODS

In this article, we demonstrate the use of two plots identified to be beneficial for wide use in RCTs, since both can display multiple AEs and are suitable to display point estimates for binary, count, or time-to-event AE data: the volcano and dot plots. We compare and contrast the use of data visualisations against traditional frequency table reporting, using published AE information in two placebo-controlled trials, of remdesivir for COVID-19 and GDNF for Parkinson disease. We introduce statistical programmes for implementation in Stata.

RESULTS/CASE STUDY

Visualisations of AEs in the COVID-19 trial communicated a risk profile for remdesivir which differed from the main message in the published authors' conclusion. In the Parkinson's disease trial of GDNF, the visualisation provided immediate communication of harm signals, which had otherwise been contained within lengthy descriptive text and tables. Asymmetry in the volcano plot helped flag extreme events that were less obvious from review of the frequency table and dot plot. The dot plot allowed a more comprehensive representation by means of a more detailed summary.

CONCLUSIONS

Visualisations can better support investigators to assimilate large volumes of data and enable improved informal between-arm comparisons compared to tables. We endorse increased uptake for use in trial publications. Care in construction of visual displays needs to be taken as there can be potential to overemphasise treatment effects in some circumstances.

Statistical methods for the analysis of adverse event data in randomised controlled trials: a scoping review and taxonomy

BACKGROUND

Statistical methods for the analysis of harm outcomes in randomised controlled trials (RCTs) are rarely used, and there is a reliance on simple approaches to display information such as in frequency tables. We aimed to identify whether any statistical methods had been specifically developed to analyse prespecified secondary harm outcomes and non-specific emerging adverse events (AEs).

METHODS

A scoping review was undertaken to identify articles that proposed original methods or the original application of existing methods for the analysis of AEs that aimed to detect potential adverse drug reactions (ADRs) in phase II-IV parallel controlled group trials. Methods where harm outcomes were the (co)-primary outcome were excluded. Information was extracted on methodological characteristics such as: whether the method required the event to be prespecified or could be used to screen emerging events; and whether it was applied to individual events or the overall AE profile. Each statistical method was appraised and a taxonomy was developed for classification.

RESULTS

Forty-four eligible articles proposing 73 individual methods were included. A taxonomy was developed and articles were categorised as: visual summary methods (8 articles proposing 20 methods); hypothesis testing methods (11 articles proposing 16 methods); estimation methods (15 articles proposing 24 methods); or methods that provide decision-making probabilities (10 articles proposing 13 methods). Methods were further classified according to whether they required a prespecified event (9 articles proposing 12 methods), or could be applied to emerging events (35 articles proposing 61 methods); and if they were (group) sequential methods (10 articles proposing 12 methods) or methods to perform final/one analyses (34 articles proposing 61 methods).

CONCLUSIONS

This review highlighted that a broad range of methods exist for AE analysis. Immediate implementation of some of these could lead to improved inference for AE data in RCTs. For example, a well-designed graphic can be an effective means to communicate complex AE data and methods appropriate for counts, time-to-event data and that avoid dichotomising continuous outcomes can improve efficiencies in analysis. Previous research has shown that adoption of such methods in the scientific press is limited and that strategies to support change are needed.

TRIAL REGISTRATION

PROSPERO registration: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=97442.

Understanding current practice, identifying barriers and exploring priorities for adverse event analysis in randomised controlled trials: an online, cross-sectional survey of statisticians from academia and industry

OBJECTIVES

To gain a better understanding of current adverse event (AE) analysis practices and the reasons for the lack of use of sophisticated statistical methods for AE data analysis in randomised controlled trials (RCTs), with the aim of identifying priorities and solutions to improve practice.

DESIGN

A cross-sectional, online survey of statisticians working in clinical trials, followed up with a workshop of senior statisticians working across the UK.

PARTICIPANTS

We aimed to recruit into the survey a minimum of one statistician from each of the 51 UK Clinical Research Collaboration registered clinical trial units (CTUs) and industry statisticians from both pharmaceuticals and clinical research organisations.

OUTCOMES

To gain a better understanding of current AE analysis practices, measure awareness of specialist methods for AE analysis and explore priorities, concerns and barriers when analysing AEs.

RESULTS

Thirty-eight (38/51; 75%) CTUs, 5 (5/7; 71%) industry and 21 attendees at the 2019 Promoting Statistical Insights Conference participated in the survey. Of the 64 participants that took part, 46 participants were classified as public sector participants and 18 as industry participants. Participants indicated that they predominantly (80%) rely on subjective comparisons when comparing AEs between treatment groups. Thirty-eight per cent were aware of specialist methods for AE analysis, but only 13% had undertaken such analyses. All participants believed guidance on appropriate AE analysis and 97% thought training specifically for AE analysis is needed. These were both endorsed as solutions by workshop participants.

CONCLUSIONS

This research supports our earlier work that identified suboptimal AE analysis practices in RCTs and confirms the underuse of more sophisticated AE analysis approaches. Improvements are needed, and further research in this area is required to identify appropriate statistical methods. This research provides a unanimous call for the development of guidance, as well as training on suitable methods for AE analysis to support change.