Who's afraid of Thomas Bayes?

A conservative approach to leveraging external evidence for effective clinical trial design

Prior probabilities of clinical hypotheses are not systematically used for clinical trial design yet, due to a concern that poor priors may lead to poor decisions. To address this concern, a conservative approach to Bayesian trial design is illustrated here, requiring that the operational characteristics of the primary trial outcome are stronger than the prior. This approach is complementary to current Bayesian design methods, in that it insures against prior-data conflict by defining a sample size commensurate to a discrete design prior. This approach is ethical, in that it requires designs appropriate to achieving pre-specified levels of clinical equipoise imbalance. Practical examples are discussed, illustrating design of trials with binary or time to event endpoints. Moderate increases in phase II study sample size are shown to deliver strong levels of overall evidence for go/no-go clinical development decisions. Levels of negative evidence provided by group sequential confirmatory designs are found negligible, highlighting the importance of complementing efficacy boundaries with non-binding futility criteria.

Decision Making in a Multidisciplinary Cancer Team: Does Team Discussion Result in Better Quality Decisions?

To establish whether treatment recommendations made by clinicians concur with the best outcomes predicted from their prognostic estimates and whether team discussion improves the quality or outcome of their decision making, the authors studied real-time decision making by a lung cancer team. Clinicians completed pre- and postdiscussion questionnaires for 50 newly diagnosed patients. For each patient/doctor pairing, a decision model determined the expected patient outcomes from the clinician’s prognostic estimates. The difference between the expected utility of the recommended treatment and the maximum utility derived from the clinician’s predictions of the outcomes (the net utility loss) following all potential treatment modalities was calculated as an indicator of quality of the decision. The proportion of treatment decisions changed by the multidisciplinary team discussion was also calculated. Insofar as the change in net utility loss brought about by multidisciplinary team discussion was not significantly different from zero, team discussion did not improve the quality of decision making overall. However, given the modest power of the study, these findings must be interpreted with caution. In only 23 of 87 instances (26%) in which an individual specialist’s initial treatment preference differed from the final group judgment did the specialist finally concur with the group treatment choice after discussion. This study does not support the theory that team discussion improves decision making by closing a knowledge gap.

Synthesis of evidence for reimbursement decisions: a Bayesian reanalysis

OBJECTIVES

The aim of this study was to compare Bayesian methods with the standard methods that are used for evidence-based policy making.

METHODS

We performed a Bayesian reanalysis of the data underlying a reimbursement advice by the Dutch National Health Insurance Board (CVZ) regarding the anti-diabetic drug exenatide (an alternative to insulin). We synthesized evidence from various sources that was available when the CVZ advice was drafted: expert opinion (as elicited from internists), experimental data (from direct comparison studies), and observational data. Subsequently, the original frequentist results and the results from the Bayesian reanalysis were compared in terms of outcomes and interpretations. These results were presented in a meeting with staff from CVZ, whose opinions about the usefulness of a Bayesian approach were assessed using a questionnaire.

RESULTS

The Bayesian approach yields outcomes that summarize different pieces of evidence, which would have been difficult to obtain otherwise. Moreover, there are conceptual differences, and the Bayesian approach allows for determining probabilities of clinically relevant differences. The staff at CVZ were fairly positive with respect to the use of Bayesian methods, although practical barriers were also seen as important.

CONCLUSIONS

The Bayesian outcomes are different and could be more suited to the informational needs of policy makers. The response from staff at CVZ provides some support for this statement, but more research at the interface of science and policy is needed.

Recommendations for evaluation of health care improvement initiatives

Intensive efforts are underway across the world to improve the quality of health care. It is important to use evaluation methods to identify improvement efforts that work well before they are replicated across a broad range of contexts. Evaluation methods need to provide an understanding of why an improvement initiative has or has not worked and how it can be improved in the future. However, improvement initiatives are complex, and evaluation is not always well aligned with the intent and maturity of the intervention, thus limiting the applicability of the results. We describe how initiatives can be grouped into 1 of 3 improvement phases-innovation, testing, and scale-up and spread-depending on the degree of belief in the associated interventions. We describe how many evaluation approaches often lead to a finding of no effect, consistent with what has been termed Rossi's Iron Law of Evaluation. Alternatively, we recommend that the guiding question of evaluation in health care improvement be, "How and in what contexts does a new model work or can be amended to work?" To answer this, we argue for the adoption of formative, theory-driven evaluation. Specifically, evaluations start by identifying a program theory that comprises execution and content theories. These theories should be revised as the initiative develops by applying a rapid-cycle evaluation approach, in which evaluation findings are fed back to the initiative leaders on a regular basis. We describe such evaluation strategies, accounting for the phase of improvement as well as the context and setting in which the improvement concept is being deployed. Finally, we challenge the improvement and evaluation communities to come together to refine the specific methods required so as to avoid the trap of Rossi's Iron Law.

Estimating the effectiveness of HPV vaccination in the open population: a Bayesian approach

OBJECTIVES

Estimation of the effectiveness of human papillomavirus (HPV) vaccination in the open population on the basis of published data from various sources.

METHODS

A Bayesian approach was used to reanalyze the data underlying a guidance by the Dutch National Health Insurance Board about the quadrivalent HPV vaccine Gardasil. Several studies document the vaccine's effectiveness in preventing cases in different subpopulations. None of these (sub)populations, however, is representative of the actual target population that the vaccination program will be applied to. We used a Bayesian approach for restructuring the data by means of reweighting the subpopulations by using HPV prevalence data, to estimate the effectiveness that can be expected in the actual target population.

RESULTS

The original data show an effectiveness of 44% in the entire population and an effectiveness of 98% for women who were compliant and were HPV-free at the start of the study. In the study population, the HPV prevalence was below 4%. In the relevant target population, however, the actual prevalence could be very different. In fact, some publications find an HPV prevalence of around 10%. We used Bayesian techniques to estimate the effectiveness in the actual target population. We found a mean effectiveness of 25%, and the probability that the effectiveness in the target population exceeds 50% is virtually zero. The results are very sensitive to the HPV prevalence that is used.

CONCLUSIONS

A supplementary analysis can put together the bits and pieces of information to arrive at more relevant answers. A Bayesian approach allows for integrating all the evidence into one model in a straightforward way and results in very intuitive probability statements.

Bayesian modelling of household solid fuel use: insights towards designing effective interventions to promote fuel switching in Africa

Indoor air pollution from solid fuel use is a significant risk factor for acute lower respiratory infections among children in sub-Saharan Africa. Interventions that promote a switch to modern fuels hold a large health promise, but their effective design and implementation require an understanding of the web of upstream and proximal determinants of household fuel use. Using Demographic and Health Survey data for Benin, Kenya and Ethiopia together with Bayesian hierarchical and spatial modelling, this paper quantifies the impact of household-level factors on cooking fuel choice, assesses variation between communities and districts and discusses the likely nature of contextual effects. Household- and area-level characteristics appear to interact as determinants of cooking fuel choice. In all three countries, wealth and the educational attainment of women and men emerge as important; the nature of area-level factors varies between countries. In Benin, a two-level model with spatial community random effects best explains the data, pointing to an environmental explanation. In Ethiopia and Kenya, a three-level model with unstructured community and district random effects is selected, implying relatively autonomous economic and social areas. Area-level heterogeneity, indicated by large median odds ratios, appears to be responsible for a greater share of variation in the data than household-level factors. This may be an indication that fuel choice is to a considerable extent supply-driven rather than demand-driven. Consequently, interventions to promote fuel switching will carefully need to assess supply-side limitations and devise appropriate policy and programmatic approaches to overcome them. To our knowledge, this paper represents the first attempt to model the determinants of solid fuel use, highlighting socio-economic differences between households and, notably, the dramatic influence of contextual effects. It illustrates the potential that multilevel and spatial modelling approaches hold for understanding determinants of major public health problems in the developing world.

The contrast and convergence of Bayesian and frequentist statistical approaches in pharmacoeconomic analysis

The application of Bayesian statistical analyses has been facilitated in recent years by methodological advances and an increasing complexity necessitated within research. Substantial debate has historically accompanied this analytic approach relative to the frequentist method, which is the predominant statistical ideology employed in clinical studies. While the essence of the debate between the two branches of statistics centres on differences in the use of prior information and the definition of probability, the ramifications involve the breadth of research design, analysis and interpretation. The purpose of this paper is to discuss the application of frequentist and Bayesian statistics in the pharmacoeconomic assessment of healthcare technology. A description of both paradigms is offered in the context of potential advantages and disadvantages, and applications within pharmacoeconomics are briefly addressed. Additional considerations are presented to stimulate further development and to direct appropriate applications of each method such that the integrity and robustness of scientific inference be strengthened.

Survey of Australasian clinicians' prior beliefs concerning lipiodol flushing as a treatment for infertility: A Bayesian study

OBJECTIVE

To evaluate clinicians' beliefs concerning the effectiveness of lipiodol flushing as a treatment for unexplained infertility, and to integrate these prior beliefs with evidence from randomised trials.

DESIGN

Survey.

SETTING

Specialists in Australasian in vitro fertilisation (IVF) clinics in 2001.

METHODS

One of two types of structured survey was used to gather information from fertility specialists in Australasian IVF clinics. Prior beliefs were captured graphically and textually from responses.

RESULTS

Nineteen specialists returned questionnaires. Eighteen of the 19 specialists believed that lipiodol flushing was more likely to be beneficial than harmful. The most widely held prior belief, reflected in both textual and numerical responses, was that lipiodol was likely to produce a small beneficial response. The credible limits of this belief were compatible with a reasonable fertility benefit, as more than 50% believed that a 1.5-fold increase in pregnancy rate was plausible. The two surveys found that a 1.2-fold or 1.4-fold increase in pregnancy rate was the median expected level of benefit at which clinicians would have been inclined to recommend lipiodol flushing to their patients (combined range 1.1- to 2.3-fold) - new evidence suggests that for women with endometriosis but otherwise unexplained infertility, these levels of benefit are exceeded.

CONCLUSIONS

Among Australasian fertility specialists there is variation in prior beliefs concerning the effectiveness of lipiodol flushing as a treatment for unexplained infertility and in the expected level of benefit at which clinicians are inclined to recommend this treatment. Generalisability of these beliefs remains uncertain owing to a low study response rate.

Evidence-based medicine in pathology: an introduction

Evidence-based medicine (EBM) is a school of thought that has spread rapidly through medicine in the past 2 decades and is eliciting an increasing interest in Anatomic Pathology and Laboratory Medicine. It has been defined as "the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients." The environmental factors that created a need for EBM and basic concepts of this discipline are reviewed. Methods for the accrual and critical appraisal of the validity of available evidence and its impact, applicability and usefulness in pathology practice are discussed. Basic concepts of bayesian data analysis with an emphasis on concepts such as prior and posterior probability and the use of "holdout" or "test" data are introduced. The future of EBM in pathology is discussed and potential applications of these concepts to pathology practice and research are proposed.

Bayes’ theorem: A negative example of a RCT on grommets in children with glue ear

Bayesian inference presupposes that practitioners' belief in the effectiveness of medical intervention is the product of prior belief and recent evidence from studies. Although increasingly used, up to now the posterior belief calculated according to the theorem has not been compared with an empirically measured posterior belief. We conducted a RCT, which was preceded by elicitation of prior beliefs among ENT-surgeons, and which was followed by elicitation of posterior beliefs among ENT-surgeons, 1 year after completion of the trial. We compared the posterior beliefs of ENT-surgeons about the effect of grommets in children with glue ears, as predicted by Bayes' theorem with actual measured posterior beliefs. The distribution of the measured posterior beliefs was not in line with the calculated posterior, but almost identical to the distribution of the measured prior beliefs. The results showed that our trial had little or no impact on the beliefs of the ENT-surgeons, i.e. they did not adjust their belief to the extent that was expected according to Bayes' theorem.

The cost effectiveness of two new antiepileptic therapies in the absence of direct comparative data: a first approximation

BACKGROUND

A number of new antiepileptic agents have been introduced within a short period of time. Direct comparisons are not available, and information about the balance between costs and effects for these new therapies is lacking.

OBJECTIVE

To introduce a first approximation of the cost effectiveness of the new therapeutic agents (topiramate and lamotrigine) for epilepsy that have been assessed in clinical trials against placebo.

METHODS

Without head to head comparative data no formal methods are available to assess the relative cost effectiveness of two products; therefore, a Bayesian approach was developed. The approach starts with the 'proportionality assumption' saying that the differences in healthcare expenditure (less the direct cost of therapy) are directly proportional to the differences in effectiveness. Given this assumption, a therapy that is x times as expensive as an alternative therapy has an equivalent cost-effectiveness profile if the acquisition cost is x times as high. Moreover, simple formulas can be derived to calculate the probabilities that a therapy is dominant (more effective and less expensive) and that it is weakly dominant (more effective and a better cost-effectiveness profile). The approach is applied to data from published fixed dosage, parallel-design studies comparing both topiramate and lamotrigine with placebo.

RESULTS

Assuming that the 'proportionality assumption' holds for the medical treatment of epilepsy, and disregarding uncertainties, it is estimated that topiramate may be priced more than 2.2 times its current acquisition cost and still be more cost effective than lamotrigine. Taking uncertainties into account, it is estimated that lamotrigine 500 mg/day is dominated by topiramate 200 mg/day with a probability of 0.875 and by topiramate 400 mg/day with a probability of 0.986.

CONCLUSIONS

A simple method can be applied to assess the relative cost effectiveness of two therapies in the absence of direct comparative data. Applying this method to compare topiramate and lamotrigine leads to a strong preference for topiramate. However, to be able to draw this conclusion, some heroic assumptions need to be made. As such the method as developed here only reflects a first approximation. It needs to be used with care and is not intended to replace good comparative research.

Bayesian methods for design and analysis of cost-effectiveness trials in the evaluation of health care technologies

We review the development of Bayesian statistical methods for the design and analysis of randomized controlled trials in the assessment of the cost-effectiveness of health care technologies. We place particular emphasis on the benefits of the Bayesian approach; the implications of skew cost data; the need to model the data appropriately to generate efficient and robust inferences instead of relying on distribution-free methods; the importance of making full use of quantitative and structural prior information to produce realistic inferences; and issues in the determination of sample size. Several new examples are presented to illustrate the methods. We conclude with a discussion of the key areas for future research.

At what degree of belief in a research hypothesis is a trial in humans justified?

RATIONALE, AIMS AND OBJECTIVES

Randomized controlled trials (RCTs) have emerged as the most reliable method of assessing the effects of health care interventions in clinical medicine. However, RCTs should be undertaken only if there is substantial uncertainty about which of the trial treatments would benefit a patient most. The purpose of this study is to determine the degree of uncertainty in a research hypothesis before it can empirically be tested in an RCT.

METHODS

We integrated arguments from three independent lines of research - on ethics, principles of the design and conduct of clinical trials, and medical decision making - to develop a decision model to help solve the dilemma of under which circumstances innovative treatments should be tested in an RCT.

RESULTS

We showed that RCTs are the preferable option to resolve uncertainties about competing treatment alternatives whenever we desire reliable, undisputed, high-quality evidence with a low likelihood of false-positive or false-negative results.

CONCLUSIONS

When the expected benefit:risk ratio of a new treatment is small, an RCT is justified to resolve uncertainties over a wide range of prior belief (e.g. 10-90) in the accuracy of the research hypothesis. Randomized controlled trials represent the best means for resolving uncertainties about health care interventions.

Prospective application of Bayesian monitoring and analysis in an "open" randomized clinical trial

We describe the prospective application of Bayesian monitoring and analysis in an ongoing large multi-centre, randomized trial in which interim results are released to investigators. Substantial variability in prior opinion led us to reject the use of elicited clinical priors for monitoring, in favour of archetypal prior distributions representing reasonable scepticism and enthusiasm. Likelihoods for odds ratios for different covariate values are derived from a logistic regression model, which allows us to incorporate information from prognostic factors without resorting to specialized software. Priors, likelihoods and posterior distributions are regularly reported to both an independent Data Monitoring Committee and the trial investigators.

[The Bayesian approach: another way of drawing inferences]

The serious objections made with regard to significance tests account for the necessity of employing another inferential procedure. Bayesian methods are free of such objections, and they provide a very attractive alternative. By means of a simple example, this article illustrates how a typical problem of medical research could be solved using these two approaches. Bayesian methods offer more information and are more useful than conventional ones for analyzing experimental outcomes. In addition, a natural interpretation of conclusions are given by Bayesian methods. Finally, modern computational programs allow us to solve their complex calculation.