A more powerful exact test of noninferiority from binary matched‐pairs data

MOVER tests for non-inferiority of the difference between two binary-outcome treatments in the matched-pairs design

A non-inferiority trial is usually conducted to investigate whether a new drug/treatment is no worse than a reference drug/treatment by a small, pre-specified, non-inferiority margin. This study aimed to assess the non-inferiority of the difference between two binary-outcome treatments in a matched-pairs design based on the method of variance of estimates recovery (MOVER). The processes for estimating the confidence interval of a single proportion included in the MOVER are the Wilson score interval, Agresti - Coull interval, Jeffreys interval, modified Jeffreys interval, score method with continuity correction, and arcsin interval. The performance of the six MOVER tests, the fiducial test, and the restricted maximum likelihood estimation test were evaluated by comparing their type I error rates and power at different pre-assigned levels and with varying combinations of parameters. The evaluation results showed that the modified Jeffreys MOVER test can be a competitive alternative to the other recommended tests. It can control type I errors well, and its power is not inferior to other methods. The proposed tests were illustrated with three real-world examples.

A fiducial approach for testing the non-inferiority of proportion difference in matched-pairs design

The non-inferiority of one treatment/drug to another is a common and important issue in medical and pharmaceutical fields. This study explored a fiducial approach for testing the non-inferiority of proportion difference in matched-pairs design. Approximate tests constructed using fiducial quantities with a combination of different parameters were proposed. Four simulation studies were employed to compare the performance of fiducial tests by comparing their type I errors and powers. The results showed that fiducial quantities with parameter 0.6 ≤ w 1 ≤ 0.8 $$ 0.6\le {w}_1\le 0.8 $$ performed satisfactorily from small to large samples. Therefore, the fiducial tests could be recommended for practical applications. The recommended fiducial tests might be a competitive alternative to other available tests. Three real data sets were analyzed to illustrate the proposed methods were competitive or even better than other tests.

Recommended confidence intervals for the conditional odds ratio in matched-pairs designs

There has been limited research on the confidence intervals of the conditional odds ratio in matched-pairs design. This article investigates the interval estimation of the conditional odds ratio. We described several confidence intervals, which are available in some situations, and they can produce different results. We tried to determine which method(s) should be recommended for different situations. We derived four confidence intervals from the delta test, the score test, the inferential model test, and the fiducial test, and employed four exact calculation studies to compare the performances of the four methods, in order to make recommendations for small and moderate-to-large sample sizes. All of the methods are illustrated using a real example. And we offered the recommendations for different situations.

Confidence intervals for difference in proportions for matched pairs compatible with exact McNemar's or sign tests

For testing with paired data (eg, twins randomized between two treatments), a simple test is the sign test, where we test if the distribution of the sign of the differences in responses between the two treatments within pairs is more often positive (favoring one treatment) or negative (favoring the other). When the responses are binary, this reduces to a McNemar-type test, and the calculations are the same. Although it is easy to calculate an exact P-value by conditioning on the total number of discordant pairs, the accompanying confidence interval on a parameter of interest (proportion positive minus proportion negative) is not straightforward. Effect estimates and confidence intervals are important for interpretation because it is possible that the treatment helps a very small proportion of the population yet gives a highly significant effect. We construct a confidence interval that is compatible with an exact sign test, meaning the 100 ( 1 - α ) % interval excludes the null hypothesis of equality of proportions if and only if the associated exact sign test rejects at level α . We conjecture that the proposed confidence intervals guarantee nominal coverage, and we support that conjecture with extensive numerical calculations, but we have no mathematical proof to show guaranteed coverage. We have written and made available the function mcnemarExactDP in the exact2x2 R package and the function signTest in the asht R package to perform the methods described in this article.

Accurate unconditional p-values for a two-arm study with binary endpoints

Unconditional exact tests are increasingly used in practice for categorical data to increase the power of a study and to make the data analysis approach being consistent with the study design. In a two-arm study with a binary endpoint, p-value based on the exact unconditional Barnard test is computed by maximizing the tail probability over a nuisance parameter with a range from 0 to 1. The traditional grid search method is able to find an approximate maximum with a partition of the parameter space, but it is not accurate and this approach becomes computationally intensive for a study beyond two groups. We propose using a polynomial method to rewrite the tail probability as a polynomial. The solutions from the derivative of the polynomial contain the solution for the global maximum of the tail probability. We use an example from a double-blind randomized Phase II cancer clinical trial to illustrate the application of the proposed polynomial method to achieve an accurate p-value. We also compare the performance of the proposed method and the traditional grid search method under various conditions. We would recommend using this new polynomial method in computing accurate exact unconditional p-values.

An efficient test based on the inferential model for the non-inferiority of odds ratio in matched-pairs design

Non-inferiority of one treatment to another based on odds ratio for the matched-pair design is a common issue in the medical research. Liu et al. derived two asymptotic tests, delta method and score test, which can be applicable for large samples but may tend to be liberal for small sample sizes. Jin et al. proposed an IM-based method that can control the type I risk well but may be somewhat conservative. In this paper, we extend the IM-based method to RIM-based test using the randomized plausibility function. We prove that our new proposed method is not only valid but also efficient. Simulation studies confirm that the RIM-based test is better than other methods. A numerical example illustrates the proposed method.

Sample size calculation for agreement between two raters with binary endpoints using exact tests

In an agreement test between two raters with binary endpoints, existing methods for sample size calculation are always based on asymptotic approaches that use limiting distributions of a test statistic under null and alternative hypotheses. These calculated sample sizes may be not reliable due to the unsatisfactory type I error control of asymptotic approaches. We propose a new sample size calculation based on exact approaches which control for the type I error rate. The two exact approaches are considered: one approach based on maximization and the other based on estimation and maximization. We found that the latter approach is generally more powerful than the one based on maximization. Therefore, we present the sample size calculation based on estimation and maximization. A real example from a clinical trial to diagnose low back pain of patients is used to illustrate the two exact testing procedures and sample size determination.

Efficient Noninferiority Testing Procedures for Simultaneously Assessing Sensitivity and Specificity of Two Diagnostic Tests

Sensitivity and specificity are often used to assess the performance of a diagnostic test with binary outcomes. Wald-type test statistics have been proposed for testing sensitivity and specificity individually. In the presence of a gold standard, simultaneous comparison between two diagnostic tests for noninferiority of sensitivity and specificity based on an asymptotic approach has been studied by Chen et al. (2003). However, the asymptotic approach may suffer from unsatisfactory type I error control as observed from many studies, especially in small to medium sample settings. In this paper, we compare three unconditional approaches for simultaneously testing sensitivity and specificity. They are approaches based on estimation, maximization, and a combination of estimation and maximization. Although the estimation approach does not guarantee type I error, it has satisfactory performance with regard to type I error control. The other two unconditional approaches are exact. The approach based on estimation and maximization is generally more powerful than the approach based on maximization.

Sample size calculation based on efficient unconditional tests for clinical trials with historical controls

In historical clinical trials, the sample size and the number of success in the control group are often considered as given. The traditional method for sample size calculation is based on an asymptotic approach developed by Makuch and Simon (1980). Exact unconditional approaches may be considered as alternative to control for the type I error rate where the asymptotic approach may fail to do so. We provide the sample size calculation using an efficient exact unconditional testing procedure based on estimation and maximization. The sample size using the exact unconditional approach based on estimation and maximization is generally smaller than those based on the other approaches.

Powerful exact unconditional tests for agreement between two raters with binary endpoints

Asymptotic and exact conditional approaches have often been used for testing agreement between two raters with binary outcomes. The exact conditional approach is guaranteed to respect the test size as compared to the traditionally used asymptotic approach based on the standardized Cohen's kappa coefficient. An alternative to the conditional approach is an unconditional strategy which relaxes the restriction of fixed marginal totals as in the conditional approach. Three exact unconditional hypothesis testing procedures are considered in this article: an approach based on maximization, an approach based on the conditional p-value and maximization, and an approach based on estimation and maximization. We compared these testing procedures based on the commonly used Cohen's kappa with regards to test size and power. We recommend the following two exact approaches for use in practice due to power advantages: the approach based on conditional p-value and maximization and the approach based on estimation and maximization.

Unconditional tests for comparing two ordered multinomials

We consider two exact unconditional procedures to test the difference between two multinomials with ordered categorical data. Exact unconditional procedures are compared to other approaches based on the Wilcoxon mid-rank test and the proportional odds model. We use a real example from an arthritis pain study to illustrate the various test procedures and provide an extensive numerical study to compare procedures with regards to type I error rates and power under the unconditional framework. The exact unconditional procedure based on estimation followed by maximization is generally more powerful than other procedures, and is therefore recommended for use in practice.

Evaluation of current Australian health service accreditation processes (ACCREDIT-CAP): protocol for a mixed-method research project

INTRODUCTION

Accreditation programmes aim to improve the quality and safety of health services, and have been widely implemented. However, there is conflicting evidence regarding the outcomes of existing programmes. The Accreditation Collaborative for the Conduct of Research, Evaluation and Designated Investigations through Teamwork-Current Accreditation Processes (ACCREDIT-CAP) project is designed to address key gaps in the literature by evaluating the current processes of three accreditation programmes used across Australian acute, primary and aged care services.

METHODS AND DESIGN

The project comprises three mixed-method studies involving documentary analyses, surveys, focus groups and individual interviews. Study samples will comprise stakeholders from across the Australian healthcare system: accreditation agencies; federal and state government departments; consumer advocates; professional colleges and associations; and staff of acute, primary and aged care services. Sample sizes have been determined to ensure results allow robust conclusions. Qualitative information will be thematically analysed, supported by the use of textual grouping software. Quantitative data will be subjected to a variety of analytical procedures, including descriptive and comparative statistics. The results are designed to inform health system policy and planning decisions in Australia and internationally.

ETHICS AND DISSEMINATION

The project has been approved by the University of New South Wales Human Research Ethics Committee (approval number HREC 10274). Results will be reported to partner organisations, healthcare consumers and other stakeholders via peer-reviewed publications, conference and seminar presentations, and a publicly accessible website.

An empirical test of short notice surveys in two accreditation programmes

OBJECTIVE

To evaluate short notice surveys in accreditation programmes.

DESIGN

Two trials using short notice surveys were conducted independently: a study of 20 healthcare organizations with the Australian Council on Healthcare Standards (ACHS) and a study of 7 general practices with the Australian General Practice Accreditation Limited (AGPAL). Participating organizations volunteered. ACHS and AGPAL selected 17 and 13 surveyors, respectively, and provided training for them on short notice surveys.

METHODS

Each agency's short notice surveys were an abbreviated version of their current advanced notification surveys. Short notice surveys assessed accreditation programme criteria or indicators that corresponded to the Australian Commission on Safety and Quality in Health Care's priority issues. Fifteen (out of 45) ACHS criteria and 48 (out of 174) AGPAL indicators that aligned to the Commission's criteria were evaluated. Participating organizations were given 2 days notice prior to the short notice surveys. Ratings from the short notice surveys were compared with those from the most recent advanced notification surveys, and statistical tests were performed to detect differences and potential confounding factors. Surveyors and organizational staff completed a post-survey feedback questionnaire which was analysed thematically and by inferential statistics.

RESULTS

The short notice survey approach overall produced ratings congruent with the advanced notification survey for both accreditation programmes. However, for both programmes short notice surveys assessed that more organizations would not reach the accreditation threshold as compared with the previous survey. Organizations in both programmes were judged to have achieved less successful performance against clinical standards by the short notice survey than the advanced notification survey. There was support from surveyors and organizational staff for short notice survey to be adopted. However, there were mixed views about the impact of short notice surveys and whether they validated trial participants' continuous improvement efforts.

CONCLUSIONS

The study demonstrated that short notice surveys are more critical in their assessment of clinical than administrative or corporate items. Short notice surveys, while broadly comparable with existing advanced notification survey practice, produced different accreditation outcomes for a significant proportion of the study organizations. The overall value and worth of short notice surveys remains to be proved.

A note on monotonicity assumptions for exact unconditional tests in binary matched-pairs designs

Exact unconditional tests have been widely applied to test the difference between two probabilities for 2 × 2 matched-pairs binary data with small sample size. In this context, Lloyd (2008, Biometrics 64, 716-723) proposed an E + M p-value, that showed better performance than the existing M p-value and C p-value. However, the analytical calculation of the E + M p-value requires that the Barnard convexity condition be satisfied; this can be challenging to prove theoretically. In this article, by a simple reformulation, we show that a weaker condition, conditional monotonicity, is sufficient to calculate all three p-values (M, C, and E + M) and their corresponding exact sizes. Moreover, this conditional monotonicity condition is applicable to noninferiority tests.