Causal estimands and confidence intervals associated with Wilcoxon-Mann-Whitney tests in randomized experiments

Generalized Pairwise Comparisons to Support Shared Decision-Making in the CODA Trial

Importance

Shared decision-making (SDM) can be made difficult by the multifaceted nature of outcome assessment. A rigorous method for analyzing results from multiple outcomes is called generalized pairwise comparisons (GPC), which could assist in SDM.

Objective

To examine whether GPC can be useful in SDM by using individual-patient data from the Comparison of Outcomes of Antibiotic Drugs and Appendectomy (CODA) trial.

Design, Setting, and Participants

This comparative effectiveness study used data from participants in the multicenter US CODA trial (conducted between May 2016 and March 2020). All possible pairs of patients (one from each arm) were formed to analyze each of 7 outcomes of interest sequentially. Data were analyzed between February 2020 and early 2024.

Exposures

Three scenarios of priorities related to a different order of outcomes were considered. The first scenario came from a consensus exercise with patients that favored antibiotics, whereas the other 2 were arbitrarily chosen to illustrate the range of possible outcomes depending on prioritizations. Scenario 2 favored neither treatment, and scenario 3 favored appendectomy.

Main Outcomes and Measures

The primary outcome was the net treatment benefit (NTB), a formal measure of benefit-risk, which is the net probability that a randomly selected patient from the antibiotic-assigned arm would have a more favorable outcome than a randomly selected patient from the appendectomy-assigned arm.

Results

A total of 1552 patients were included in the CODA trial, with 776 (mean [SD] age, 38.3 [13.4] years; 286 [37%] female) in the antibiotic arm and 776 (mean [SD] age, 37.8 [13.7] years; 290 [37%] female) in the appendectomy arm. The NTB of antibiotic treatment was 12.8% (95% CI, 7.1% to 18.3%; P < .001) for the first scenario, 3.2% (95% CI -2.4% to 8.7%; P = .27) for the second, and -14.5% (95% CI. -20.2% to -8.8%; P < .001) for the third. These results respectively favored antibiotics, neither treatment, or appendectomy, thus illustrating that benefit-risk varies considerably according to individual priorities.

Conclusions and Relevance

This comparative effectiveness study of antibiotics and appendectomy illustrates that the GPC method is a flexible yet mathematically rigorous quantitative analysis of benefit-risk balance. This method provides a more exhaustive and nuanced quantitative assessment of the differences between 2 treatment modalities in terms of prioritized outcomes. Furthermore, GPC could support SDM by considering individual prioritizations of the multiple outcomes.

Causal interpretation of the hazard ratio in randomized clinical trials

BACKGROUND

Although the hazard ratio has no straightforward causal interpretation, clinical trialists commonly use it as a measure of treatment effect.

METHODS

We review the definition and examples of causal estimands. We discuss the causal interpretation of the hazard ratio from a two-arm randomized clinical trial, and the implications of proportional hazards assumptions in the context of potential outcomes. We illustrate the application of these concepts in a synthetic model and in a model of the time-varying effects of COVID-19 vaccination.

RESULTS

We define causal estimands as having either an individual-level or population-level interpretation. Difference-in-expectation estimands are both individual-level and population-level estimands, whereas without strong untestable assumptions the causal rate ratio and hazard ratio have only population-level interpretations. We caution users against making an incorrect individual-level interpretation, emphasizing that in general a hazard ratio does not on average change each individual's hazard by a factor. We discuss a potentially valid interpretation of the constant hazard ratio as a population-level causal effect under the proportional hazards assumption.

CONCLUSION

We conclude that the population-level hazard ratio remains a useful estimand, but one must interpret it with appropriate attention to the underlying causal model. This is especially important for interpreting hazard ratios over time.

Defining estimand for the win ratio: Separate the true effect from censoring

The win ratio has been increasingly used in trials with hierarchical composite endpoints. While the outcomes involved and the rule for their comparisons vary with the application, there is invariably little attention to the estimand of the resulting statistic, causing difficulties in interpretation and cross-trial comparison. We make the case for articulating the estimand as a first step to win ratio analysis and establish that the root cause for its elusiveness is its intrinsic dependency on the time frame of comparison, which, if left unspecified, is set haphazardly by trial-specific censoring. From the statistical literature, we summarize two general approaches to overcome this uncertainty-a nonparametric one that pre-specifies the time frame for all comparisons, and a semiparametric one that posits a constant win ratio across all times-each with publicly available software and real examples. Finally, we discuss unsolved challenges, such as estimand construction and inference in the presence of intercurrent events.

Evaluating whether the proportional odds models to analyse ordinal outcomes in COVID-19 clinical trials is providing clinically interpretable treatment effects: A systematic review

BACKGROUND

After an initial recommendation from the World Health Organisation, trials of patients hospitalised with COVID-19 often include an ordinal clinical status outcome, which comprises a series of ordered categorical variables, typically ranging from 'Alive and discharged from hospital' to 'Dead'. These ordinal outcomes are often analysed using a proportional odds model, which provides a common odds ratio as an overall measure of effect, which is generally interpreted as the odds ratio for being in a higher category. The common odds ratio relies on the assumption of proportional odds, which implies an identical odds ratio across all ordinal categories; however, there is generally no statistical or biological basis for which this assumption should hold; and when violated, the common odds ratio may be a biased representation of the odds ratios for particular categories within the ordinal outcome. In this study, we aimed to evaluate to what extent the common odds ratio in published COVID-19 trials differed to simple binary odds ratios for clinically important outcomes.

METHODS

We conducted a systematic review of randomised trials evaluating interventions for patients hospitalised with COVID-19, which used a proportional odds model to analyse an ordinal clinical status outcome, published between January 2020 and May 2021. We assessed agreement between the common odds ratio and the odds ratio from a standard logistic regression model for three clinically important binary outcomes: 'Alive', 'Alive without mechanical ventilation', and 'Alive and discharged from hospital'.

RESULTS

Sixteen randomised clinical trials, comprising 38 individual comparisons, were included in this study; of these, only 6 trials (38%) formally assessed the proportional odds assumption. The common odds ratio differed by more than 25% compared to the binary odds ratios in 55% of comparisons for the outcome 'Alive', 37% for 'Alive without mechanical ventilation', and 24% for 'Alive and discharged from hospital'. In addition, the common odds ratio systematically underestimated the odds ratio for the outcome 'Alive' by -16.8% (95% confidence interval: -28.7% to -2.9%, p = 0.02), though differences for the other outcomes were smaller and not statistically significant (-8.4% for 'Alive without mechanical ventilation' and 3.6% for 'Alive and discharged from hospital'). The common odds ratio was statistically significant for 18% of comparisons, while the binary odds ratio was significant in 5%, 16%, and 3% of comparisons for the outcomes 'Alive', 'Alive without mechanical ventilation', and 'Alive and discharged from hospital', respectively.

CONCLUSION

The common odds ratio from proportional odds models often differs substantially to odds ratios from clinically important binary outcomes, and similar to composite outcomes, a beneficial common OR from a proportional odds model does not necessarily indicate a beneficial effect on the most important categories within the ordinal outcome.

A nonparametric relative treatment effect for direct comparisons of censored paired survival outcomes

A frequently addressed issue in clinical trials is the comparison of censored paired survival outcomes, for example, when individuals were matched based on their characteristics prior to the analysis. In this regard, a proper incorporation of the dependence structure of the paired censored outcomes is required and, up to now, appropriate methods are only rarely available in the literature. Moreover, existing methods are not motivated by the strive for insights by means of an easy-to-interpret parameter. Hence, we seek to develop a new estimand-driven method to compare the effectiveness of two treatments in the context of right-censored survival data with matched pairs. With the help of competing risks techniques, the so-called relative treatment effect is estimated. This estimand describes the probability that individuals under Treatment 1 have a longer lifetime than comparable individuals under Treatment 2. We derive hypothesis tests and confidence intervals based on a studentized version of the estimator, where resampling-based inference is established by means of a randomization method. In a simulation study, we demonstrate for numerous sample sizes and different amounts of censoring that the developed test exhibits a good power. Finally, we apply the methodology to a well-known benchmark data set from a trial with patients suffering from diabetic retinopathy.

Highly robust causal semiparametric U-statistic with applications in biomedical studies

With our increased ability to capture large data, causal inference has received renewed attention and is playing an ever-important role in biomedicine and economics. However, one major methodological hurdle is that existing methods rely on many unverifiable model assumptions. Thus robust modeling is a critically important approach complementary to sensitivity analysis, where it compares results under various model assumptions. The more robust a method is with respect to model assumptions, the more worthy it is. The doubly robust estimator (DRE) is a significant advance in this direction. However, in practice, many outcome measures are functionals of multiple distributions, and so are the associated estimands, which can only be estimated via U-statistics. Thus most existing DREs do not apply. This article proposes a broad class of highly robust U-statistic estimators (HREs), which use semiparametric specifications for both the propensity score and outcome models in constructing the U-statistic. Thus, the HRE is more robust than the existing DREs. We derive comprehensive asymptotic properties of the proposed estimators and perform extensive simulation studies to evaluate their finite sample performance and compare them with the corresponding parametric U-statistics and the naive estimators, which show significant advantages. Then we apply the method to analyze a clinical trial from the AIDS Clinical Trials Group.

Statistical analyses of ordinal outcomes in randomised controlled trials: a scoping review

BACKGROUND

Randomised controlled trials (RCTs) aim to estimate the causal effect of one or more interventions relative to a control. One type of outcome that can be of interest in an RCT is an ordinal outcome, which is useful to answer clinical questions regarding complex and evolving patient states. The target parameter of interest for an ordinal outcome depends on the research question and the assumptions the analyst is willing to make. This review aimed to provide an overview of how ordinal outcomes have been used and analysed in RCTs.

METHODS

The review included RCTs with an ordinal primary or secondary outcome published between 2017 and 2022 in four highly ranked medical journals (the British Medical Journal, New England Journal of Medicine, The Lancet, and the Journal of the American Medical Association) identified through PubMed. Details regarding the study setting, design, the target parameter, and statistical methods used to analyse the ordinal outcome were extracted.

RESULTS

The search identified 309 studies, of which 144 were eligible for inclusion. The most used target parameter was an odds ratio, reported in 78 (54%) studies. The ordinal outcome was dichotomised for analysis in 47 ( 33 % ) studies, and the most common statistical model used to analyse the ordinal outcome on the full ordinal scale was the proportional odds model (64 [ 44 % ] studies). Notably, 86 (60%) studies did not explicitly check or describe the robustness of the assumptions for the statistical method(s) used.

CONCLUSIONS

The results of this review indicate that in RCTs that use an ordinal outcome, there is variation in the target parameter and the analytical approaches used, with many dichotomising the ordinal outcome. Few studies provided assurance regarding the appropriateness of the assumptions and methods used to analyse the ordinal outcome. More guidance is needed to improve the transparent reporting of the analysis of ordinal outcomes in future trials.

Group sequential methods for the Mann-Whitney parameter

Late phase clinical trials are occasionally planned with one or more interim analyses to allow for early termination or adaptation of the study. While extensive theory has been developed for the analysis of ordered categorical data in terms of the Wilcoxon-Mann-Whitney test, there has been comparatively little discussion in the group sequential literature on how to provide repeated confidence intervals and simple power formulas to ease sample size determination. Dealing more broadly with the nonparametric Behrens-Fisher problem, we focus on the comparison of two parallel treatment arms and show that the Wilcoxon-Mann-Whitney test, the Brunner-Munzel test, as well as a test procedure based on the log win odds, a modification of the win ratio, asymptotically follow the canonical joint distribution. In addition to developing power formulas based on these results, simulations confirm the adequacy of the proposed methods for a range of scenarios. Lastly, we apply our methodology to the FREEDOMS clinical trial (ClinicalTrials.gov Identifier: NCT00289978) in patients with relapse-remitting multiple sclerosis.

A nonparametric approach to confidence intervals for concordance index and difference between correlated indices

Concordance refers to the probability that subjects with high values on one variable also have high values on another variable. This index has wide application in practice, as a measure of effect size in group-comparison studies, an index of accuracy in diagnostic studies, and a discrimination index for prediction models. Herein, we provide a unified framework for statistical inference involving concordance indices for standard variables of binary, ordinal, and continuous types. In particular, we develop confidence interval procedures for a single concordance index and differences between two correlated indices. Simulation results show that procedures based on logit-transformation for a single index and Fisher's z-transformation for a difference between indices perform very well in terms of coverage and tail errors even when the sample size is as small as 30, unless the concordance is high and the standard is a binary variable for which at least 50 subjects are needed. We illustrate the procedures for a variety of standard variables with previously published data. Illustrative SAS code is provided.

Estimation and Testing of Wilcoxon–Mann–Whitney Effects in Factorial Clustered Data Designs

Clustered data arise frequently in many practical applications whenever units are repeatedly observed under a certain condition. One typical example for clustered data are animal experiments, where several animals share the same cage and should not be assumed to be completely independent. Standard methods for the analysis of such data are Linear Mixed Models and Generalized Estimating Equations—however, checking their assumptions is not easy, especially in scenarios with small sample sizes, highly skewed, count, and ordinal or binary data. In such situations, Wilcoxon–Mann–Whitney type effects are suitable alternatives to mean-based or other distributional approaches. Hence, no specific data distribution, symmetric or asymmetric, is required. Within this work, we will present different estimation techniques of such effects in clustered factorial designs and discuss quadratic- and multiple contrast type-testing procedures for hypotheses formulated in terms of Wilcoxon–Mann–Whitney effects. Additionally, the framework allows for the occurrence of missing data: estimation and testing hypotheses are based on all-available data instead of complete-cases. An extensive simulation study investigates the precision of the estimators and the behavior of the test procedures in terms of their type-I error control. One real world dataset exemplifies the applicability of the newly proposed procedures.

The asymptotic distribution of the Net Benefit estimator in presence of right-censoring

The benefit-risk balance is a critical information when evaluating a new treatment. The Net Benefit has been proposed as a metric for the benefit-risk assessment, and applied in oncology to simultaneously consider gains in survival and possible side effects of chemotherapies. With complete data, one can construct a U-statistic estimator for the Net Benefit and obtain its asymptotic distribution using standard results of the U-statistic theory. However, real data is often subject to right-censoring, e.g. patient drop-out in clinical trials. It is then possible to estimate the Net Benefit using a modified U-statistic, which involves the survival time. The latter can be seen as a nuisance parameter affecting the asymptotic distribution of the Net Benefit estimator. We present here how existing asymptotic results on U-statistics can be applied to estimate the distribution of the net benefit estimator, and assess their validity in finite samples. The methodology generalizes to other statistics obtained using generalized pairwise comparisons, such as the win ratio. It is implemented in the R package BuyseTest (version 2.3.0 and later) available on Comprehensive R Archive Network.

Semiparametric recurrent event vs time-to-first-event analyses in randomized trials: Estimands and model misspecification

Insights regarding the merits of recurrent event and time-to-first-event analyses are needed to provide guidance on strategies for analyzing intervention effects in randomized trials involving recurrent event responses. Using established asymptotic results we introduce a framework for studying the large sample properties of estimators arising from semiparametric proportional rate function models and Cox regression under model misspecification. The asymptotic biases and power implications are investigated for different data generating models, and we study the impact of dependent censoring on these findings. Illustrative applications are given involving data from a cystic fibrosis trial and a carcinogenicity experiment, following which we summarize findings and discuss implications for clinical trial design.

Win odds: An adaptation of the win ratio to include ties

The win ratio, a recently proposed measure for comparing the benefit of two treatment groups, allows ties in the data but ignores ties in the inference. In this article, we highlight some difficulties that this can lead to, and we propose to focus on the win odds instead, a modification of the win ratio which takes ties into account. We construct hypothesis tests and confidence intervals for the win odds, and we investigate their properties through simulations and in a case study. We conclude that the win odds should be preferred over the win ratio.

Effects of tissue-specific biomolecules on piglets after-weaning period

Background and Aim

Now-a-days antibiotics are the main tool for correcting the pathological conditions of pigs; unfortunately, antibiotics are a potential threat to the environment, as they lead to the spread of antibiotic-resistant infections. This study aimed to study the immunomodulatory encapsulated biomolecules on piglets in the post-weaning period.

Materials and Methods

An immunomodulator based on biomolecules obtained from animal raw materials included in alginate capsules to improve absorption has been developed. The study presents the results of a study on 25 weaned piglets (25-30 days old) which received biomolecules at a dose of 200 mg/piglet for 14 days, followed by 400 mg/piglet from days 15 to 28. Blood was taken from animals for analysis (biochemical, hematological, cytometric, and enzyme immunoassay) and the integral index of blood serum antimicrobial activity was determined.

Results

Experimental animals, whose initial weight was 1.6 times less than that of the control animals, were able to bridge this gap and, on the 28^th day, there were no differences in weight. Stimulation of the production of cytokines interleukin (IL)-2 and IL-4 was observed and the antimicrobial resistance of blood serum to Escherichia coli also increased. A positive effect on the metabolism of piglets was noted, which helped them adapt to a change in diet (from colostrum to solid food).

Conclusion

The results show that the immunomodulation at the dose of 150 mg/kg body weight has a great potential for improving weaned pigs.

Semiparametric linear transformation models for indirectly observed outcomes

We propose a regression framework to analyze outcomes that are indirectly observed via one or multiple proxies. Semiparametric transformation models, including Cox proportional hazards regression, turn out to be well suited to model the association between the covariates and the unobserved outcome. By coupling this regression model to a semiparametric measurement model, we can estimate these associations without requiring calibration data and without imposing strong functional assumptions on the relationship between the unobserved outcome and its proxy. When multiple proxies are available, we propose a data-driven aggregation resulting in an improved proxy. We empirically validate the proposed methodology in a simulation study, revealing good finite sample properties, especially when multiple proxies are aggregated. The methods are demonstrated on two case studies.

Unbiasedness and efficiency of non-parametric and UMVUE estimators of the probabilistic index and related statistics

In reliability theory, diagnostic accuracy, and clinical trials, the quantity P(X>Y)+1/2P(X=Y), also known as the Probabilistic Index (PI), is a common treatment effect measure when comparing two groups of observations. The quantity P(X>Y)-P(Y>X), a linear transformation of PI known as the net benefit, has also been advocated as an intuitively appealing treatment effect measure. Parametric estimation of PI has received a lot of attention in the past 40 years, with the formulation of the Uniformly Minimum-Variance Unbiased Estimator (UMVUE) for many distributions. However, the non-parametric Mann-Whitney estimator of the PI is also known to be UMVUE in some situations. To understand this seeming contradiction, in this paper a systematic comparison is performed between the non-parametric estimator for the PI and parametric UMVUE estimators in various settings. We show that the Mann-Whitney estimator is always an unbiased estimator of the PI with univariate, completely observed data, while the parametric UMVUE is not when the distribution is misspecified. Additionally, the Mann-Whitney estimator is the UMVUE when observations belong to an unrestricted family. When observations come from a more restrictive family of distributions, the loss in efficiency for the non-parametric estimator is limited in realistic clinical scenarios. In conclusion, the Mann-Whitney estimator is simple to use and is a reliable estimator for the PI and net benefit in realistic clinical scenarios.

Sharp bounds on the relative treatment effect for ordinal outcomes

For ordinal outcomes, the average treatment effect is often ill-defined and hard to interpret. Echoing Agresti and Kateri, we argue that the relative treatment effect can be a useful measure, especially for ordinal outcomes, which is defined as γ = pr { Y i ( 1 ) > Y i ( 0 ) } - pr { Y i ( 1 ) < Y i ( 0 ) } , with Y i ( 1 ) and Y i ( 0 ) being the potential outcomes of unit i under treatment and control, respectively. Given the marginal distributions of the potential outcomes, we derive the sharp bounds on γ , which are identifiable parameters based on the observed data. Agresti and Kateri focused on modeling strategies under the assumption of independent potential outcomes, but we allow for arbitrary dependence.

Constructing a confidence interval for the fraction who benefit from treatment, using randomized trial data

The fraction who benefit from treatment is the proportion of patients whose potential outcome under treatment is better than that under control. Inference on this parameter is challenging since it is only partially identifiable, even in our context of a randomized trial. We propose a new method for constructing a confidence interval for the fraction, when the outcome is ordinal or binary. Our confidence interval procedure is pointwise consistent. It does not require any assumptions about the joint distribution of the potential outcomes, although it has the flexibility to incorporate various user-defined assumptions. Our method is based on a stochastic optimization technique involving a second-order, asymptotic approximation that, to the best of our knowledge, has not been applied to biomedical studies. This approximation leads to statistics that are solutions to quadratic programs, which can be computed efficiently using optimization tools. In simulation, our method attains the nominal coverage probability or higher, and can have narrower average width than competitor methods. We apply it to a trial of a new intervention for stroke.

On Causal Inferences for Personalized Medicine: How Hidden Causal Assumptions Led to Erroneous Causal Claims About the D-Value

Personalized medicine asks if a new treatment will help a particular patient, rather than if it improves the average response in a population. Without a causal model to distinguish these questions, interpretational mistakes arise. These mistakes are seen in an article by Demidenko [2016] that recommends the "D-value," which is the probability that a randomly chosen person from the new-treatment group has a higher value for the outcome than a randomly chosen person from the control-treatment group. The abstract states "The D-value has a clear interpretation as the proportion of patients who get worse after the treatment" with similar assertions appearing later. We show these statements are incorrect because they require assumptions about the potential outcomes which are neither testable in randomized experiments nor plausible in general. The D-value will not equal the proportion of patients who get worse after treatment if (as expected) those outcomes are correlated. Independence of potential outcomes is unrealistic and eliminates any personalized treatment effects; with dependence, the D-value can even imply treatment is better than control even though most patients are harmed by the treatment. Thus, D-values are misleading for personalized medicine. To prevent misunderstandings, we advise incorporating causal models into basic statistics education.