Large-scale randomized evidence: large, simple trials and overviews of trials

Lessons learned in the conduct of a global, large simple trial of treatments indicated for schizophrenia

Large, "practical" or streamlined trials (LSTs) are used to study the effectiveness and/or safety of medicines in real world settings with minimal study imposed interventions. While LSTs have benefits over traditional randomized clinical trials and observational studies, there are inherent challenges to their conduct. Enrollment and follow-up of a large study sample of patients with mental illness pose a particular difficulty. To assist in overcoming operational barriers in future LSTs in psychiatry, this paper describes the recruitment and observational follow-up strategies used for the ZODIAC study, an international, open-label LST, which followed 18,239 persons randomly assigned to one of two treatments indicated for schizophrenia for 1 year. ZODIAC enrolled patients in 18 countries in North America, South America, Europe, and Asia using broad study entry criteria and required minimal clinical care intervention. Recruitment of adequate numbers and continued engagement of both study centers and subjects were significant challenges. Strategies implemented to mitigate these in ZODIAC include global study expansion, study branding, field coordinator and site relations programs, monthly site newsletters, collection of alternate contact information, conduct of national death index (NDI) searches, and frequent sponsor, contract research organization (CRO) and site interaction to share best practices and address recruitment challenges quickly. We conclude that conduct of large LSTs in psychiatric patient populations is feasible, but importantly, realistic site recruitment goals and maintaining site engagement are key factors that need to be considered in early study planning and conduct.

Final tips in interpreting evidence-based medicine

In this issue, the Southern Medical Journal presents a series of articles to help students of medicine understand the principles of evidence-based medicine. These articles are not meant to be a comprehensive review, but rather an easy-to-read primer. In this final article, the authors offer suggestions to aid the reader in navigating the ever-expanding amount of information. These tips address a number of points that are commonly encountered in the medical literature, but are not all-inclusive.

Pragmatic vs explanatory trials: the pragmascope tool to help measure differences in protocols of mental health randomized controlled trials

In the pragmatic-explanatory continuum, a randomized controlled trial (RCT) can at one extreme investigate whether a treatment could work in ideal circumstances (explanatory), or at the other extreme, whether it would work in everyday practice (pragmatic). How explanatory or pragmatic a study is can have implications for clinicians, policy makers, patients, researchers, funding bodies, and the public. There is an increasing need for studies to be open and pragmatic; however, explanatory trials are also needed. The previously developed Pragmatic-explanatory continuum indicator summary (PRECIS) was adapted into the Pragmascope tool to assist mental health researchers in designing RCTs, taking the pragmatic-explanatory continuum into account. Ten mental health trial protocols were randomly chosen and scored using the tool by three independent raters. Their results were compared for consistency and the tool was found to be reliable and practical. This preliminary work suggests that evaluating different domains of an RCT at the protocol level is useful, and suggests that using the Pragmascope tool presented here might be a practical way of doing this.

A rational approach to red blood cell transfusion in the neonatal ICU

PURPOSE OF REVIEW

There have been several recent randomized controlled trials collectively aimed at either the prevention or the management of anemia of prematurity. We aim to summarize evidence on prevention, management and long-term outcomes.

RECENT FINDINGS

Current guidelines for red blood cell transfusion are based on expert opinion and vary. Conservative transfusion policies can reduce the number of transfusions, but other benefits are more uncertain. Delivery room prevention by using delayed cord clamping or cord milking is promising, but requires long-term outcome assessments in preterms. Some measures of hypoxemia to guide 'need' for transfusion have potential, but are not yet ready for general use. Pragmatic management trials have compared a 'restrictive' with a 'liberal' policy with respect to effects on clinically relevant outcomes by neonatal ICU discharge, but conclusions have differed. Follow-up data to 24 months is available for only one study, which showed no benefit in the primary outcome of death and or neurodisability. However, an a-priori subgroup analysis shows benefit in the cognitive Bayley scores, favoring high hemoglobins.

SUMMARY

This field is plagued by lack of replication, small studies and speculative findings. Hence, the risk-benefit ratio of blood transfusions for preterms still needs adequate definition. Evidence suggests that a restrictive hemoglobin, hematocrit threshold or both for transfusion decreases the number of blood transfusions in preterm infants. However, uncertainty remains on long-term outcomes. Large randomized controlled trials are needed to clarify the safety of a lower threshold or the longer-term benefit of a high threshold.

The child and adolescent psychiatry trials network (CAPTN): infrastructure development and lessons learned

BACKGROUND

In 2003, the National Institute of Mental Health funded the Child and Adolescent Psychiatry Trials Network (CAPTN) under the Advanced Center for Services and Intervention Research (ACSIR) mechanism. At the time, CAPTN was believed to be both a highly innovative undertaking and a highly speculative one. One reviewer even suggested that CAPTN was "unlikely to succeed, but would be a valuable learning experience for the field."

OBJECTIVE

To describe valuable lessons learned in building a clinical research network in pediatric psychiatry, including innovations intended to decrease barriers to research participation.

METHODS

The CAPTN Team has completed construction of the CAPTN network infrastructure, conducted a large, multi-center psychometric study of a novel adverse event reporting tool, and initiated a large antidepressant safety registry and linked pharmacogenomic study focused on severe adverse events. Specific challenges overcome included establishing structures for network organization and governance; recruiting over 150 active CAPTN participants and 15 child psychiatry training programs; developing and implementing procedures for site contracts, regulatory compliance, indemnification and malpractice coverage, human subjects protection training and IRB approval; and constructing an innovative electronic casa report form (eCRF) running on a web-based electronic data capture system; and, finally, establishing procedures for audit trail oversight requirements put forward by, among others, the Food and Drug Administration (FDA).

CONCLUSION

Given stable funding for network construction and maintenance, our experience demonstrates that judicious use of web-based technologies for profiling investigators, investigator training, and capturing clinical trials data, when coupled to innovative approaches to network governance, data management and site management, can reduce the costs and burden and improve the feasibility of incorporating clinical research into routine clinical practice. Having successfully achieved its initial aim of constructing a network infrastructure, CAPTN is now a capable platform for large safety registries, pharmacogenetic studies, and randomized practical clinical trials in pediatric psychiatry.

Subject eligibility criteria can substantially influence the results of alcohol-treatment outcome research

OBJECTIVE

Most alcohol-treatment studies exclude some patients from participation based on particular criteria (e.g., comorbid illegal drug abuse, homelessness). The current study evaluated whether such eligibility criteria can change the outcome results a study obtains.

METHOD

Five widely used treatment research eligibility criteria--(1) psychiatric problems, (2) medical problems, (3) social-residential instability, (4) low motivation/noncompliance, and (5) drug problems--were applied to two samples of real-world alcohol patients whose outcomes were known. Comparing outcomes of the samples with and without the application of eligibility criteria produced estimates of bias in outcome results, as well as an assessment of change in statistical power.

RESULTS

Medical and psychiatric eligibility criteria produced a moderate bias in outcome estimates (e.g., a 10% or less change in outcome results). In contrast, social-residential instability, low motivation/noncompliance, and drug use produced a large (e.g., up to an 18% change) to a very large (e.g., up to a 51% change) bias in outcome estimates. Sensitivity analyses showed that these biases are even larger if eligibility criteria are operationalized in a broad rather than a narrow fashion. Contrary to expectation, eligibility criteria did not produce their theoretically expected benefit of increased statistical power.

CONCLUSIONS

Researchers who use eligibility criteria should do so judiciously and interpret outcome results in light of potential bias introduced by the ineligibility of some patients for study enrollment. Efforts to integrate findings across treatment outcome studies should also consider how conclusions might be affected by the eligibility criteria used in different research areas.

Technology transfer from the science of medicine to the real world: the potential role played by artificial adaptive systems

The author describes a refiguration of medical thought that originates from nonlinear dynamics and chaos theory. The coupling of computer science and these new theoretical bases coming from complex systems mathematics allows the creation of "intelligent" agents capable of adapting themselves dynamically to problems of high complexity: the artificial neural networks (ANNs). ANNs are able to reproduce the dynamic interaction of multiple factors simultaneously, allowing the study of complexity; they can also draw conclusions on an individual basis and not as average trends. These tools can allow a more efficient technology transfer from the science of medicine to the real world, overcoming many obstacles responsible for the present translational failure. They also contribute to a new holistic vision of the human subject person, contrasting the statistical reductionism that tends to squeeze or even delete the single subject, sacrificing him to his group of belongingness. A remarkable contribution to this individual approach comes from fuzzy logic, according to which there are no sharp limits between opposite things, such as wealth and disease. This approach allows one to partially escape from the probability theory trap in situations where it is fundamental to express a judgement based on a single case and favor a novel humanism directed to the management of the patient as an individual subject person.

Artificial intelligence and outcome research

The coupling of computer science and theoretical bases like non‐linear dynamics and chaos, quite new for medicine theory, allows the creation of “intelligent” agents (Artificial Adaptive Systems [AAS]) able to adapt themselves dynamically to problems of high complexity. ASS are able to reproduce the dynamical interaction of multiple factors simultaneously, allowing the study of complexity; they can also draw conclusions on an individual basis and not as average trends. These tools can offer specific advantages within the outcome research arena, helping to answer some open issues like enhancing the internal validity of observational studies, transferring evidence derived from clinical research to a single patient level, and performing “virtual” clinical trials as a guide for more efficient clinical development. A remarkable contribution to this individual approach comes from Fuzzy Logic, according to which there are no sharp limits between opposite things, like wealth and disease. This approach allows for partially escaping from the probability theory trap in situations where it is fundamental to express a judgement based on a single case and favour a novel humanism directed to the management of the patient as an individual subject. Some examples of original applications in the authors' experience are described. Drug Dev. Res. 67:227–244, 2006. © 2006 Wiley‐Liss, Inc.

Issues in clinical trial design: stopping a trial early and the large and simple trial

During the conduct of a clinical trial, a primary function of the Data Safety and Monitoring Committee is to select the trial conduct and the accumulating data to determine whether the trial should continue or be discontinued earlier than planned. Reasons for early discontinuation of a trial include: evidence of benefit, evidence of harm, and evidence of futility. More than 1 of these elements will often be present. These principles will be illustrated with examples from National Institute of Child Health and Human Development-Maternal-Fetal Medicine Units clinical trials. The "large and simple clinical trial" is a study design rarely undertaken in the United States but commonly used elsewhere. The principles of this type of trial will be introduced and contrasted with those of the "conventional clinical trial."

A randomised controlled trial of weaning from mechanical ventilation in paediatric intensive care (PIC). Methodological and practical issues

INTRODUCTION

Most children admitted to the Paediatric Intensive Care Unit (PICU) require assistance with breathing via a mechanical ventilator. Weaning from mechanical ventilation is the transition from ventilatory support to spontaneous breathing. Traditionally weaning has been with the authority of the medical staff. However, current opinion suggests that weaning could be performed by nurses using a standardised protocol [Schultz TR, Lin RJ, Watzman HM, Durning SM, Hales R, Woodson A, et al. Weaning children from mechanical ventilation: A prospective randomised trial of protocol-directed versus physician-directed weaning. Respir Care 2001;46(8):772-82]. The potential advantages of nurse-led (protocol-directed) weaning include: A reduction in weaning time and PICU stay with cost savings. Reduced complications. Improved quality of care. Appropriate use of resources.

METHODS

A Randomised Controlled Trial was performed to test the null hypothesis: there is no difference between the clinical effectiveness of nurse-led versus medical-led weaning of infants from mechanical ventilation. Data was collected for 7 infants and analysed.

RESULTS

Results indicated no significant differences between the two study groups. Unfortunately due to recruitment problems few inferences can be drawn from the data.

CONCLUSION

The trial was unsuccessful due to Recruitment issues. Physical constraints. Impractical entry criteria. Limited randomisation service. Ethical constraints. Barriers to parental participation. The methods, the difficulties encountered and the implications for future research are addressed.

Funding, disease area, and internal validity of hepatobiliary randomized clinical trials

OBJECTIVE

The aim of this study was to assess whether funding and the disease area are related to the internal validity of hepatobiliary randomized clinical trials.

METHODS

We gathered data on funding, disease area, methodological quality (randomization and double blinding), and sample size from 616 hepatobiliary randomized clinical trials published from 1985 to 1996 in 12 MEDLINE indexed journals.

RESULTS

The internal validity (methodological quality and sample size) of trials funded by profit or nonprofit organizations was not significantly different. Compared with these trials, trials without funding were significantly less likely to report adequate generation of the allocation sequence (55% vs 41%, p = 0.001) and to be double blind (42% vs 25%, p < 0.001), but the proportion with adequate allocation concealment and the sample size were not significantly different. The trials covered 12 disease areas. The proportion of funded trials did not differ significantly in different disease areas. The disease area was significantly associated with the proportion of trials with adequate generation of the allocation sequence (p < 0.001), allocation concealment (p = 0.003), and double blinding (p < 0.001) as well as the sample size (p < 0.001). This association was not explained by the proportion of trials with funding.

CONCLUSIONS

External funding was significantly associated with adequate methodological quality, but not with the sample size. Irrespective of funding, the disease area was significantly associated with the methodological quality and sample size. Accordingly, external funding and the disease area are significant predictors of the internal validity of hepatobiliary randomized clinical trials.

Randomized controlled trials in schizophrenia: a critical perspective on the literature

OBJECTIVE

The randomized trial provides an opportunity to minimize the inclusion of biases in the evaluation of interventions in psychiatry. Difficulties arise, however, when applying their results to 'real world' clinical practice and decision-making. We, therefore, examined the real world applicability of schizophrenia trials.

METHOD

A narrative overview of the content and quality of the randomized trials relevant to the care of those with schizophrenia is provided.

RESULTS

Complex, explanatory, under-powered randomized drug trials dominate evaluative research in schizophrenia.

CONCLUSION

Explanatory designs are a necessary but insufficient step in establishing the true worth of interventions in schizophrenia. Research from other spheres of mental health and wider health care suggest that pragmatic trials are feasible. This design allows large scale trials to be conducted which include patients which we would recognize from routine practice and which record outcomes which are of genuine interest to decision-makers.

Best practice in psychogeriatric care

This paper presents a best practice model for psychogeriatric care. Best practice is becoming one of the most common expressions used in the area of health care, and is often referred to in government reports and documents. The definition of 'best practice', however, is still evolving. What then, is best practice? And how can the principles of best practice be applied and integrated into the clinical speciality of psychogeriatrics? The article emphasizes the importance of evidence-based interventions and the need to focus on the pragmatic aspects of providing best practice in the clinical area of psychogeriatrics; that is, what works best in practice? The position taken by the authors of this paper is that the conceptualization of a best practice model in psychogeriatrics is necessary in order to describe and explain the different components involved in the service provided. This conceptualization also communicates and articulates the role of the major stakeholders, and the key players in the achievement of best practice. A psychogeriatric service may become more coherent, more goal-orientated and more efficient if a model is utilized. This paper outlines a proposed model of best practice in psychogeriatrics, and discusses the potential implications for achieving desirable clinical outcomes.

Randomised controlled trial comparing effectiveness of touch screen system with leaflet for providing women with information on prenatal tests

OBJECTIVE

To compare the effectiveness of touch screen system with information leaflet for providing women with information on prenatal tests.

DESIGN

Randomised controlled trial; participants allocated to intervention group (given access to touch screen and leaflet information) or control group (leaflet information only).

SETTING

Antenatal clinic in university teaching hospital.

SUBJECTS

875 women booking antenatal care.

INTERVENTIONS

All participants received a leaflet providing information on prenatal tests. Women in the intervention arm also had access to touch screen information system in antenatal clinic.

MAIN OUTCOME MEASURES

Women's informed decision making on prenatal testing as measured by their uptake of and understanding of the purpose of specific tests; their satisfaction with information provided; and their levels of anxiety.

RESULTS

All women in the trial had a good baseline knowledge of prenatal tests. Women in the intervention group did not show any greater understanding of the purpose of the tests than control women. However, uptake of detailed anomaly scans was significantly higher in intervention group than the control group (94% (351/375) v 87% (310/358), P=0.0014). Levels of anxiety among nulliparous women in intervention group declined significantly over time (P<0.001).

CONCLUSIONS

The touch screen seemed to convey no benefit over well prepared leaflets in improving understanding of prenatal tests among the pregnant women. It did, however, seem to reduce levels of anxiety and may be most effective for providing information to selected women who have a relevant adverse history or abnormal results from tests in their current pregnancy.

Patients' perceptions of physicians communication and outcomes of the accrual to trial process

The purpose of this study was to examine the relations among patients' perceptions of their physicians' communicative behavior during the informed consent interview, the patient's feeling of being confirmed by the physician and satisfied with care delivered by the physician, and the patient's decision to participate in a clinical trial or not. Respondents included 130 cancer patients who were eligible for a clinical trial and who had recently discussed trial participation with their physicians. Results indicated that a linear combination of the variables physician affiliative style, physician dominant or controlling style, patient satisfaction, patient confirmation, patient preference for decision making, patient desire for information, and patient age discriminate between patients who agree to participate in clinical trials and patients who refuse to participate. Physicians' affiliative communicative behaviors and patient satisfaction were clearly important to patients who agreed to participate. Motivations for patients who declined to participate in trials were less clear. Implications for physicians who offer clinical trials to their patients are that specific communication skills may enhance their patients' satisfaction and may help increase enrollment in clinical trials.

Methodological approaches of health technology assessment

In this era of evolving health care systems throughout the world, technology remains the substance of health care. Medical informatics comprises a growing contribution to the technologies used in the delivery and management of health care. Diverse, evolving technologies include artificial neural networks, computer-assisted surgery, computer-based patient records, hospital information systems, and more. Decision-makers increasingly demand well-founded information to determine whether or how to develop these technologies, allow them on the market, acquire them, use them, pay for their use, and more. The development and wider use of health technology assessment (HTA) reflects this demand. While HTA offers systematic, well-founded approaches for determining the value of medical informatics technologies, HTA must continue to adapt and refine its methods in response to these evolving technologies. This paper provides a basic overview of HTA principles and methods.

Lessons from large trials: the MTA study as a model for evaluating the treatment of childhood psychiatric disorder

OBJECTIVE

To review the methodology of the Multimodal Treatment Study of Children With Attention-Deficit Hyperactivity Disorder (ADHD), the MTA study, and its implications for the design of future child mental health treatment studies.

METHOD

The characteristics of large-scale studies envisioned by trialists engaged in cardiovascular and cancer research provide the framework for reviewing key elements of the MTA study--objectives, research questions, measurement, sampling, and exposure to treatment--and discussing important methodological decisions the MTA investigators had to make.

RESULTS

The MTA study is a complex, standardized, carefully implemented, randomized control trial. Review of the MTA study indicated that future studies should be aligned clearly with either effectiveness or efficacy objectives but not both. Questions selected for study should be simple, clear, and important. Measurement, sampling, and data collection must adhere to the principle of simplicity to ensure maximum participation. All methodological decisions should be geared to attaining the research objectives: in effectiveness trials, this means evaluating treatments that have a high potential for dissemination if proven successful and recruiting only new referrals from child mental health settings.

CONCLUSIONS

The MTA study has raised the standard for technical excellence in child treatment research and will draw attention to some fundamental issues in large-scale child treatment studies, including articulating objectives and questions, setting priorities for measurement and sampling, and identifying treatments for evaluation that have a high probability of dissemination if found effective.

Randomized clinical trials in HEPATOLOGY: predictors of quality

Evidence shows that the quality of randomized clinical trials (RCTs) affects estimates of intervention efficacy, which is significantly exaggerated in low-quality trials. The present study examines the quality of all 235 RCTs published in HEPATOLOGY from the initiation in 1981 through August 1998. Quality was assessed by means of a validated 5-point scale and separate quality components associated with empirical evidence of bias. Only 26% of all RCTs reported sample size calculations, 52% adequate generation of the allocation sequence, 34% adequate allocation concealment and 34% double-blinding. The median quality score of all trials was 3 points (range, 1-5 points). Multiple logistic regression analysis explored the association between quality and therapeutic areas, number of centers, external funding, year of publication, and country of origin. High-quality trials were most likely to investigate portal hypertension (odds ratio [OR]: 2.4; 95% CI: 1.1-5.5; P =.03), be multicentered (OR: 3.4; 95% CI: 1.3-8.9; P =.01), sponsored by public organizations (OR: 4.2; 95% CI: 2.1-8.6; P =.0001), or the drug and device industry (OR: 4.7; 95% CI: 2.2-10.2; P =.0001) compared with other therapeutic areas, single-center trials, and trials with no external funding. Quality did not improve with time and was not associated with country of origin. The main conclusions are that the quality of RCTs in HEPATOLOGY needs improvement and that the probability of high quality increased with the number of centers involved and external funding.

Randomised trial of educational visits to enhance use of systematic reviews in 25 obstetric units

OBJECTIVE

To evaluate the effectiveness of an educational visit to help obstetricians and midwives select and use evidence from a Cochrane database containing 600 systematic reviews.

DESIGN

Randomised single blind controlled trial with obstetric units allocated to an educational visit or control group.

SETTING

25 of the 26 district general obstetric units in two former NHS regions.

SUBJECTS

The senior obstetrician and midwife from each intervention unit participated in educational visits. Clinical practices of all staff were assessed in 4508 pregnancies.

INTERVENTION

Single informal educational visit by a respected obstetrician including discussion of evidence based obstetrics, guidance on implementation, and donation of Cochrane database and other materials.

MAIN OUTCOME MEASURES

Rates of perineal suturing with polyglycolic acid, ventouse delivery, prophylactic antibiotics in caesarean section, and steroids in preterm delivery, before and 9 months after visits, and concordance of guidelines with review evidence for same marker practices before and after visits.

RESULTS

Rates varied greatly, but the overall baseline mean of 43% (986/2312) increased to 54% (1189/2196) 9 months later. Rates of ventouse delivery increased significantly in intervention units but not in control units; there was no difference between the two types of units in uptake of other practices. Pooling rates from all 25 units, use of antibiotics in caesarean section and use of polyglycolic acid sutures increased significantly over the period, but use of steroids in preterm delivery was unchanged. Labour ward guidelines seldom agreed with evidence at baseline; this hardly improved after visits. Educational visits cost pound860 each (at 1995 prices).

CONCLUSIONS

There was considerable uptake of evidence into practice in both control and intervention units between 1994 and 1995. Our educational visits added little to this, despite the informal setting, targeting of senior staff from two disciplines, and donation of educational materials. Further work is needed to define cost effective methods to enhance the uptake of evidence from systematic reviews and to clarify leadership and roles of senior obstetric staff in implementing the evidence.

Quantitative evaluation of clinical software, exemplified by decision support systems

To ensure the correctness of publicity material ('truth in labelling') and to inform their licensing decisions, agencies certifying or regulating any clinical computer system will need information about the system's structure, performance and likely impact on users and the environment in which they work. This information must be reliable and complete, so it needs to be collected in a structured, rigorous evaluation programme. Clinical decision support systems are generally more complex and their effects less easy to predict than most other clinical software, so pose the greatest challenge to evaluators. They are therefore the focus of this paper.

The effects of simple interventions on tooth mortality: findings in one trial and implications for future studies

The purpose of this report was to use a particular clinical trial, the Preventive Geriatric Trial (PGT), as a starting point to discuss whether treatment efficacy can be evaluated by means of tooth mortality. In the PGT, 296 subjects were recruited and randomly assigned to five treatment groups: (1) usual procedures (UP); (2) UP + a cognitive-behavioral intervention (CB); (3) UP + CB + weekly chlorhexidine rinse (CHX); (4) UP + CB + CHX + semi-annual fluoride varnish (F); and (5) UP + CB + CHX + F + semi-annual prophylaxis, including scaling (P). Exploratory analyses revealed that tooth mortality after the 1st year was lower in treatment groups 3, 4, and 5 than in groups 1 and 2. A one-year exposure resulted in a 45% reduction in tooth mortality (p < 0.05); a two-year exposure resulted in a 59% reduction (p-value < 0.04). The PGT findings suggested that it is possible to design trials based on clinically relevant endpoints, such as tooth mortality. For the detection of moderate treatment effects, such trials could take the form of Large, Simple Trials (LST), where many subjects are recruited with minimally restrictive entry criteria, and data are collected only on essential baseline characteristics and tooth mortality. LSTs have provided "reliable answers to important clinical questions" for other chronic diseases, and several arguments suggest that they could play a similar critical role in dental research: (1) Periodontitis and caries are among the most common and costly chronic diseases affecting humans, and the identification of even moderately effective treatments by LSTs can have a large socio-economic impact; (2) the identification of low-cost widely practicable treatments that lend themselves to be investigated in LSTs is likely to benefit more people than the identification of high-cost complex treatments; and (3) tooth mortality is simple to assess and more relevant than the unvalidated surrogate endpoints that have largely failed for more than 20 years to provide reliable answers to certain controversial issues regarding treatment efficacy. The cost of not reliably establishing the safety and the efficacy of treatments may be far greater than the cost of conducting LSTs.

Systematic reviews of randomized controlled trials: the need for complete data

If the relative effectiveness of different treatments that might be used in clinical practice is to be evaluated reliably, it is very important that the evaluation is carried out in an appropriate manner. This is especially true where the differences between treatments are expected to be moderate, and so easily obscured by the play of chance or systematic bias. Although such differences are often of considerable clinical importance, they can be difficult to assess and require a large amount of randomized evidence. This evidence can be obtained through prospective randomized controlled trials, meta-analysis of results from past randomized trials, or ideally a combination of the two, with prospective trials contributing to future meta-analyses. Whichever technique is adopted, all possible biases must be minimized through the collection of as much randomized evidence as possible. In meta-analyses, this is best achieved by ensuring that all relevant trials, and all randomized participants in these trials, are included in the analysis. The gold standard for this might be a meta-analysis of individual patient data, in which details for each participant in every trial are collected and analysed centrally. This approach requires considerable time and effort. However, it will add to the analyses that can be performed and will remove many of the problems associated with a reliance on published data alone and some of the problems that can arise from the use of aggregate data. This paper sets out some of the reasons for this and some of the techniques used for individual patient data-based meta-analysis.

Acquisition and use of clinical data for audit and research

Acquisition of adequate patient data for clinical management is hard enough, but higher quality patient data are needed for clinical audit and research. This article discusses some of the problems of using routine clinical data for audit and research, aspects of data quality, sources of audit or research data and their problems, methods for improving data quality, the benefits and problems of computer-based systems and current trends in the capture and processing of clinical data.

Costing neonatal care alongside the Collaborative ECMO trial: how much primary research is required?

Researchers working on economic evaluations alongside trials have to balance minimising data collection with maximising the ability to measure differences in costs. Using existing data sources may keep the costs of research down, but these data may not be entirely appropriate to the evaluation question. When evaluating technologies in intensive care it is particularly important to be able to classify patients correctly by their resource requirements especially when those requirements vary considerably from day to day. This paper describes and justifies methods for costing the care provided for babies in (one arm of) an on-going multi-centre trial, the Collaborative ECMO trial. This trial is evaluating alternative policies of life support for mature (full term) newborn babies with severe respiratory failure. The most reliable cost information on neonatal intensive care is available from a study, conducted independently from the trial, which has used simple cost apportionment on a large sample of units. By drawing on clinical opinion and carrying out a case note exercise we assessed whether this available information was appropriate to estimate 'baseline' costs for the control group during their initial 'acute' phase of illness. We concluded that the available cost estimates would need to be weighted to reflect the additional costs of drugs and investigations for this group of babies during the acute phase. Multidisciplinary collaboration on trials can help economists and other researchers to balance the requirement for simple cost measurements with more detailed primary research.

Cumulative meta-analysis of clinical trials builds evidence for exemplary medical care

Cumulative meta-analysis of clinical trials (a Bayesian interpretation for accumulating evidence) will profoundly affect medical care by summarizing evidence in the assessment of technology innovations. Application of the technique to the randomized control trials (RCTs) of streptokinase treatment of acute myocardial infarction, reduction of peri-operative mortality by antibiotic prophylaxis, and prevention of death from bleeding peptic ulcers has revealed efficacy years before it was suspected by any other means. Arrangement of the trials according to event rate in the controls, effect sizes, quality of the trials or according to covariables of interest has supplied unique information. If carried out prospectively the technique supplies invaluable information regarding indications for another trial, the number of patients necessary to determine the validity of past trends, and the type of patients who might be benefitted. Careful examination in a cumulative manner of the prior trials can reduce the need for future large trials.

The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?

OBJECTIVE

To determine whether inappropriate subgroup analysis together with chance could change the conclusion of a systematic review of several randomised trials of an ineffective treatment.

DESIGN

44 randomised controlled trials of DICE therapy for stroke were performed (simulated by rolling different coloured dice; two trials per investigator). Each roll of the dice yielded the outcome (death or survival) for that "patient." Publication bias was also simulated. The results were combined in a systematic review.

SETTING

Edinburgh.

MAIN OUTCOME MEASURE

Mortality.

RESULTS

The "hypothesis generating" trial suggested that DICE therapy provided complete protection against death from acute stroke. However, analysis of all the trials suggested a reduction of only 11% (SD 11) in the odds of death. A predefined subgroup analysis by colour of dice suggested that red dice therapy increased the odds by 9% (22). If the analysis excluded red dice trials and those of poor methodological quality the odds decreased by 22% (13, 2P = 0.09). Analysis of "published" trials showed a decrease of 23% (13, 2P = 0.07) while analysis of only those in which the trialist had become familiar with the intervention showed a decrease of 39% (17, 2P = 0.02).

CONCLUSION

The early benefits of DICE therapy were not confirmed by subsequent trials. A plausible (but inappropriate) subset analysis of the effects of treatment led to the qualitatively different conclusion that DICE therapy reduced mortality, whereas in truth it was ineffective. Chance influences the outcome of clinical trials and systematic reviews of trials much more than many investigators realise, and its effects may lead to incorrect conclusions about the benefits of treatment.

Obtaining data from randomised controlled trials: how much do we need for reliable and informative meta-analyses?

Many randomised controlled trials compare treatments that will produce only moderate differences in outcome, but these differences can be clinically important. However, they are difficult to assess reliably and require a large amount of randomised evidence. This can be achieved through large prospective randomised trials which will accrue future patients, the meta-analysis of results from randomised trials involving patients from the past, or--ideally--both. The techniques require that all possible biases are minimised, and in meta-analyses this can best be achieved by ensuring that all of the randomised evidence--both trials and participants in those trials--is included. The meta-analysis of individual patient data has been described as the gold standard for this approach. It will remove many of the problems associated with relying solely on published data and some of the problems arising from a reliance on aggregate data, and will also add to the analyses that can be performed. Such projects, however, require considerable time and effort.