Methodological issues associated with using different cut-off points to categorize outcome variables

Knowledge of the factors that contribute to delay in seeking medical treatment for acute myocardial infarction (AMI) provides the basis for interventions that are intended to facilitate prompt care-seeking behaviour. However, operational definitions of delay time vary across research studies. The use of inconsistent cut-off times to distinguish between delayers and non-delayers is likely to compromise comparability and generalizability of the findings across studies. The purpose of this paper is to examine the impact of inconsistent operationalization of delay, in terms of cut-off times, on the validity of research findings pertaining to identifying its predictors. Secondary data analysis was performed using a sample of 73 patients who had recently experienced out-of-hospital AMI and concluded that their symptoms were related to the heart. Several regression models were built to examine the influence of using different cut-off times (1, 2, 3, 6, and 12 hours, median delay) on the number and nature of predictors ofAMI care-seeking delay. The impact of varying cut-off times on the explained variance, sensitivity, specificity, and predictive values associated with each regression model was examined. The use of different cut-off times produced different sets of independent predictors, which varied in number and nature. The variance explained by the different regression models as well as their classification indices varied. Use of different cut-off times for the definition of delay time led to inconsistent results. Thus, it is recommended that criteria be established among clinicians and researchers with regard to operationally defining care-seeking delay for AMI.

Analyzing the "correct" endpoint

The choice of QOL endpoints for a study should be based on which score will most likely change if the treatment is favorable. How the QOL change is calculated should be based on the expected amount of missing data, how many time points data will be collected, and whether extreme outliers in the scores impact results. The study should have sufficient power to detect a meaningful difference between arms (typically 10 points on a 0-100 point scale) in the chosen QOL endpoint. At the conclusion of a study, several secondary endpoints can be analyzed which can provide additional information and confirm primary endpoint results.

[On uniform social medical evaluation of performance capacity in orthopedic rehabilitation]

The need for standardized and unified performance capacity assessment in orthopedic rehabilitation has led to the attempt to use an existing standardized procedure. Graduated disease features have been defined which are relevant for work and their effects on performance capacity been formulated as rules. In this way so-called disease-conditioned performance capacities are obtained which help to identify problems in carrying out various activities. These activities can be directly assessed using tests. Appropriate tests are listed in a table.

Variable Impact of Combining Fatal and Nonfatal End Points in Heart Failure Trials

Randomized clinical trials (RCTs) are a cornerstone of evidence-based medicine. Their design, implementation, and interpretation are sometimes subject to flaws and errors. To achieve statistical significance and economically feasible RCTs, the use of composite end points in heart failure (HF) trials has become more common. We analyzed the incremental value of combining HF hospitalizations with all-cause mortality in trials of chronic HF that enrolled >1000 placebo patients and had a mean follow-up >9 months. We tested the assumption that, compared with mortality, combining HF hospitalization with all-cause death would yield a consistently predictable increase in event rate across HF RCTs. Average placebo arm duration of follow-up was determined, and standardized placebo event rates per 100 patient-years were estimated. Twelve major HF RCTs were included in this analysis. There was a substantial relative increase in the event rate ranging from 64% to 134% when a composite end point was used. This increase was not related to disease severity as described by annual mortality rate. The relative contribution of combining HF hospitalization with all-cause mortality was, however, influenced by the duration of the trial. Combining HF hospitalization with all-cause mortality increases the overall event rate in HF clinical trials; the relative increase varies widely and is unrelated to disease severity. Longer-duration trials have a more predictable increase in events than short RCTs. Trial duration must be considered when composite end points are used during the design and interpretation of HF RCTs.

Clinical trials in pediatric traumatic brain injury: unique challenges and potential responses

In order to optimize pediatric traumatic brain injury translational and clinical research, scientific and ethical challenges need to be recognized and addressed. Having recently conducted a multisite phase II safety/feasibility trial of magnesium sulfate as a neuroprotective agent, we supplement our own experience by a mini review of similar studies, identifying challenges and possible responses from the perspective of families, investigators, funding agencies and society.

Brief report: incidence, etiology, risk factors, and outcome of hospital-acquired fever: a systematic, evidence-based review

OBJECTIVES

Temperature is universally measured in the hospitalized patient, but the literature on hospital-acquired fever has not been systematically reviewed. This systematic review is intended to provide clinicians with an overview of the incidence, etiology, and outcome of hospital-acquired fever.

DATA SOURCES

We searched MEDLINE (1970 to 2005), EMBASE (1988 to 2004), and Web of Knowledge. References of all included articles were reviewed. Articles that focused on children, fever in the developing world, classic fever of unknown origin, or specialized patient populations were excluded.

REVIEW METHODS

Articles were reviewed independently by 2 authors before inclusion; a third author acted as arbiter.

RESULTS

Of over 1,000 studies reviewed, 7 met the criteria for inclusion. The incidence of hospital-acquired fever ranged from 2% to 17%. The etiology of fever was infection in 37% to 74%. Rates of antibiotic use for patients with a noninfectious cause of fever ranged from 29% to 55% for a mean duration of 6.6 to 9.6 days. Studies varied widely in their methodology and the patient population studied.

CONCLUSIONS

Limited information is available to guide an evidence-based approach to hospital-acquired fever. We propose criteria to help standardize future studies of this important clinical situation.

Hypothermia pediatric head injury trial: the value of a pretrial clinical evaluation phase

BACKGROUND

The utility of a pretrial clinical evaluation or run-in phase prior to conducting trials of complex interventions such as hypothermia therapy following severe traumatic brain injury in children and adolescents has not been established.

METHODS

The primary objective of this study was to prospectively evaluate the ability of investigators to adhere to the clinical protocols of care including the cooling and rewarming procedures as well as management guidelines in patients with severe traumatic brain injury (Glasgow Coma Scale<or=8) treated with 24 h of hypothermia therapy. A secondary objective was to evaluate the ability of study research assistants to complete the study case report form using a procedures manual. The study was conducted at 18 sites in Canada, the United Kingdom and France prior to proceeding to a randomized controlled trial (RCT). After 2 patients were enrolled at each center, an independent clinical evaluation committee examined the process of care and the completeness of data collection. Centers were permitted to enroll patients in the RCT once they met pre-established adherence criteria.

RESULTS

Seventeen of the 18 centers completed the pretrial clinical evaluation phase demonstrating compliance with study procedures and proceeded to an RCT of hypothermia therapy. One center enrolled only 1 patient in the pretrial clinical evaluation phase due to small numbers of patients with traumatic brain injury, and therefore, did not proceed to the RCT. Three centers were required to enroll more than 2 patients in the pretrial clinical evaluation phase prior to proceeding to the RCT because of problems with adherence to the clinical protocols at two centers and the training of new study personnel at another center. Of the 39 patients enrolled during the pretrial clinical evaluation phase, 8 (20.5%) died and 22 (62.9%) had a good outcome defined as normal, mild or moderate disability assessed using the Pediatric Cerebral Performance Category score at 6 months following injury.

DISCUSSION

The pretrial clinical evaluation phase was useful to ensure compliance with complex hypothermia therapy and consensus-based clinical management guidelines of care successfully implemented across 17 of 18 centers. This study maneuver allowed us to complete a subsequent RCT in 225 children following severe traumatic brain injury.

Primary care funding, contract status, and outcomes: an observational study

BACKGROUND

The introduction of the Quality and Outcomes Framework (QOF) provides a quantitative way of assessing quality of care in general practice. We explore the achievements of general practice in the first year of the QOF, with specific reference to practice funding and contract status.

AIM

To determine the extent to which differences in funding and contract status affect quality in primary care.

DESIGN OF STUDY

Cross-sectional observational study using practice data obtained under the Freedom of Information Act 2000.

SETTING

One hundred and sixty-four practices from six primary care trusts (PCTs) in England.

METHOD

Practice data for all 164 practices were collated for income and contract status. The outcome measure was QOF score for the year 2004-2005. All data were analysed statistically.

RESULTS

Contract status has an impact on practice funding, with Employed Medical Services (EMS) and Personal Medical Services (PMS) practices receiving higher levels of funding than General Medical Services (GMS) practices (P<0.001). QOF scores also vary according to contract status. Higher funding levels in EMS practices are associated with lower QOF scores (P=0.04); while GMS practices exhibited the opposite trend, with higher-funded practices achieving better quality scores (P<0.001).

CONCLUSION

GMS practices are the most efficient contract status, achieving high quality scores for an average of pound 62.51 per patient per year. By contrast, EMS practices are underperforming, achieving low quality scores for an average of pound 105.37 per patient per year. Funding and contract status are therefore important factors in determining achievement in the QOF.

Use of standardised outcome measures in adult mental health services: randomised controlled trial

BACKGROUND

Routine use of standardised outcome measures is not universal.

AIMS

To evaluate the effectiveness of standardised outcome assessment.

METHOD

A randomised controlled trial, involving 160 representative adult mental health patients and paired staff (ISRCTN16971059). The intervention group (n=101) (a) completed monthly postal questionnaires assessing needs, quality of life, mental health problem severity and therapeutic alliance, and (b) received 3-monthly feedback. The control group (n=59) received treatment as usual.

RESULTS

The intervention did not improve primary outcomes of patient-rated unmet need and of quality of life. Other subjective secondary outcome measures were also not improved. The intervention reduced psychiatric inpatient days (3.5 v.16.4 mean days, bootstrapped 95% CI1.6-25.7), and hence service use costs were 2586 UK pounds (95% CI 102-5391) less for intervention-group patients. Net benefit analysis indicated that the intervention was cost-effective.

CONCLUSIONS

Routine use of outcome measures as implemented in this study did not improve subjective outcomes, but was associated with reduced psychiatric inpatient admissions.

Development of quality indicators: lessons learned in bariatric surgery

Quality indicators will likely be used in comprehensive surgical quality assessment and improvement programs. Quality indicators are the actions equated with good quality of care. As a case example, bariatric surgery quality indicators were developed using evidence in the literature combined with formal expert opinion validation. Qualitative analysis was performed to identify the critical thematic issues surrounding development of these surgical quality indicators. Researchers identified five major thematic categories during the development process. These included feasibility in medical records (availability, ease of abstraction, and cost), the number of indicators developed (optimal number), the lack of evidence in the literature (weight on expert opinion), structural versus process indicators, and linkage to outcomes (need to demonstrate that adherence to indicators is associated with better outcomes). This project, using bariatric surgery as an example, uncovered important issues that need to be addressed when developing quality assessment and quality improvement programs for evaluating surgical quality. As quality indicators will likely be developed and used increasingly, future projects in this regard will benefit from these lessons.

How a well-grounded minimal important difference can enhance transparency of labelling claims and improve interpretation of a patient reported outcome measure

The evaluation and use of patient reported outcome (PRO) measures requires detailed understanding of the meaning of the outcome of interest. The Food and Drug Administration (FDA) recently presented its draft guidance and view on the use of PRO measures as endpoints in clinical trials. One section of the guidance document specifically deals with advice about the use of the minimal important difference (MID) that we redefined as the smallest difference in score in the outcome of interest that informed patients or informed proxies perceive as important. The advice, however, is short, indeed much too short. We believe that expanding the section and making it more specific will benefit all stakeholders: patients, clinicians, other clinical decision makers, those designing trials and making claims, payers and the FDA.

There is no "gold standard" methodology of estimating the MID or achieving the meaningfulness of clinical trial results based on patient reported outcomes. There are many methods of estimating the MID usually grouped into two distinct categories: anchor-based methods, that examine the relationship between scores on the target instrument and some independent measure, and distribution-based methods resorting to the statistical characteristics of the obtained scores.

Estimation of an MID and interpretation of clinical trial results that present patient important outcomes is demanding but vital for informing the decision to recommend approve a given intervention. Investigators are encouraged to use reliable and valid methods to achieve meaningfulness of their results, preferably those that rely on patients to estimate what constitutes a minimal important, small, moderate, or large difference. However, acquiring the meaningfulness of PRO measures transcends beyond a concept of the MID and we advocate that dichotomizing the scores of patient-reported outcome measures facilitate interpretability of clinical trial results for those who need to understand trial results after a labelling claim has been granted. Irrespective of the strategy investigators use to estimate these values, from the individual patient perspective it is much more relevant if investigators report both the estimated thresholds and the proportion of patients achieving that benefit.

[The evaluation of health care outcomes and hospital performance indicators]

INTRODUCTION

The assessment of the performance of health care establishments has undergone a considerable development over the past 15 years in the United States and to a lesser extent in other developed countries.

BACKGROUND

The aim of measurement of performance indicators is to improve the quality of care (outcomes), patient information and the contractual arrangements with purchasers. However, this approach poses numerous methodological problems in the choice of performance indicators as well as the collection and interpretation of data. Specific structural patterns such as social and geographic environment, research and educational assignments, are often inadequately considered. In terms of public health the impact of the publication of these measurements has not been well studied. Based on the data in the literature this revue defines the measures of hospital performance and describes the main studies, their impacts and limitations.

VIEWPOINT

It seems likely that the French public authorities will, in the short term, ask health care establishments to undertake this approach.

CONCLUSIONS

Complimentary studies are needed to clarify the links between performance indicators and health care outcomes.

Audience response system technology improves accuracy and reliability of trauma outcome judgments

BACKGROUND

Peer-review judgments are necessary for effective trauma performance improvement (PI), but may be influenced by peer pressure and the tendency to vote with the majority. Incorporation of Audience Response System (ARS) technology into trauma PI should result in improved outcome assessments.

METHODS

We compared 30 months of nonanonymous trauma care judgments with 30 months of anonymous judgments obtained with the use of a keypad-based ARS. Statistical methods included the chi2 test and the Wilcoxon rank sum test.

RESULTS

Use of the ARS resulted in a 28% reduction in deaths judged nonpreventable and a 24% reduction in trauma care judged to be appropriate (p < 0.0001). Unanimous outcome judgments were also significantly reduced (p < 0.0001).

CONCLUSIONS

Outcome judgments obtained anonymously were significantly more divergent and less positive than those obtained nonanonymously. Anonymously derived outcome judgments may provide a better opportunity to identify adverse outcomes and thereby potentially improve trauma PI and trauma care.

When to use agreement versus reliability measures

BACKGROUND

Reproducibility concerns the degree to which repeated measurements provide similar results. Agreement parameters assess how close the results of the repeated measurements are, by estimating the measurement error in repeated measurements. Reliability parameters assess whether study objects, often persons, can be distinguished from each other, despite measurement errors. In that case, the measurement error is related to the variability between persons. Consequently, reliability parameters are highly dependent on the heterogeneity of the study sample, while the agreement parameters, based on measurement error, are more a pure characteristic of the measurement instrument.

METHODS AND RESULTS

Using an example of an interrater study, in which different physical therapists measure the range of motion of the arm in patients with shoulder complaints, the differences and relationships between reliability and agreement parameters for continuous variables are illustrated.

CONCLUSION

If the research question concerns the distinction of persons, reliability parameters are the most appropriate. But if the aim is to measure change in health status, which is often the case in clinical practice, parameters of agreement are preferred.

Reliability of two goniometric methods of measuring active inversion and eversion range of motion at the ankle

BACKGROUND

Active inversion and eversion ankle range of motion (ROM) is widely used to evaluate treatment effect, however the error associated with the available measurement protocols is unknown. This study aimed to establish the reliability of goniometry as used in clinical practice.

METHODS

30 subjects (60 ankles) with a wide variety of ankle conditions participated in this study. Three observers, with different skill levels, measured active inversion and eversion ankle ROM three times on each of two days. Measurements were performed with subjects positioned (a) sitting and (b) prone. Intra-class correlation coefficients (ICC[2,1]) were calculated to determine intra- and inter-observer reliability.

RESULTS

Within session intra-observer reliability ranged from ICC[2,1] 0.82 to 0.96 and between session intra-observer reliability ranged from ICC[2,1] 0.42 to 0.80. Reliability was similar for the sitting and the prone positions, however, between sessions, inversion measurements were more reliable than eversion measurements. Within session inter-observer measurements in sitting were more reliable than in prone and inversion measurements were more reliable than eversion measurements.

CONCLUSION

Our findings show that ankle inversion and eversion ROM can be measured with high to very high reliability by the same observer within sessions and with low to moderate reliability by different observers within a session. The reliability of measures made by the same observer between sessions varies depending on the direction, being low to moderate for eversion measurements and moderate to high for inversion measurements in both positions.

Practice Parameter: Prediction of outcome in comatose survivors after cardiopulmonary resuscitation (an evidence-based review): Report of the Quality Standards Subcommittee of the American Academy of Neurology

OBJECTIVE

To systematically review outcomes in comatose survivors after cardiac arrest and cardiopulmonary resuscitation (CPR).

METHODS

The authors analyzed studies (1966 to 2006) that explored predictors of death or unconsciousness after 1 month or unconsciousness or severe disability after 6 months.

RESULTS

The authors identified four class I studies, three class II studies, and five class III studies on clinical findings and circumstances. The indicators of poor outcome after CPR are absent pupillary light response or corneal reflexes, and extensor or no motor response to pain after 3 days of observation (level A), and myoclonus status epilepticus (level B). Prognosis cannot be based on circumstances of CPR (level B) or elevated body temperature (level C). The authors identified one class I, one class II, and nine class III studies on electrophysiology. Bilateral absent cortical responses on somatosensory evoked potential studies recorded 3 days after CPR predicted poor outcome (level B). Burst suppression or generalized epileptiform discharges on EEG predicted poor outcomes but with insufficient prognostic accuracy (level C). The authors identified one class I, 11 class III, and three class IV studies on biochemical markers. Serum neuron-specific enolase higher than 33 microg/L predicted poor outcome (level B). Ten class IV studies on brain monitoring and neuroimaging did not provide data to support or refute usefulness in prognostication (level U).

CONCLUSION

Pupillary light response, corneal reflexes, motor responses to pain, myoclonus status epilepticus, serum neuron-specific enolase, and somatosensory evoked potential studies can reliably assist in accurately predicting poor outcome in comatose patients after cardiopulmonary resuscitation for cardiac arrest.

The association between quality of care and the intensity of diabetes disease management programs

BACKGROUND

Although disease management programs are widely implemented, little is known about their effectiveness.

OBJECTIVE

To determine whether disease management by physician groups is associated with diabetes care processes, control of intermediate outcomes, or the amount of medication used when intermediate outcomes are above target levels.

DESIGN

Cross-sectional study.

SETTING

Patients were randomly sampled from 63 physician groups nested in 7 health plans sponsored by Translating Research into Action for Diabetes (87%) and from 4 health plans with individual physician contracts (13%).

PATIENTS

8661 adults with diabetes who completed a survey (2000-2001) and had medical record data.

MEASUREMENTS

Physician group and health plan directors described their organizations' use of physician reminders, performance feedback, and structured care management on a survey; their responses were used to determine measures of intensity of disease management. The current study measured 8 processes of care, including most recent hemoglobin A1c level, systolic blood pressure, serum low-density lipoprotein cholesterol level, and several measures of medication use.

RESULTS

Increased use of any of 3 disease management strategies was significantly associated with higher adjusted rates of retinal screening, nephropathy screening, foot examinations, and measurement of hemoglobin A1c levels. Serum lipid level testing and influenza vaccine administration were associated with greater use of structured care management and performance feedback. Greater use of performance feedback correlated with an increased rate of foot examinations (difference, 5 percentage points [95% CI, 1 to 8 percentage points]), and greater use of physician reminders was associated with an increased rate of nephropathy screening (difference, 15 percentage points [CI, 6 to 23 percentage points]). No strategies were associated with intermediate outcome levels or level of medication management.

LIMITATIONS

Physician groups were not randomly sampled from population-based listings, and disease management strategies were not randomly allocated across groups.

CONCLUSIONS

Disease management strategies were associated with better processes of diabetes care but not with improved intermediate outcomes or level of medication management. A greater focus on direct measurement, feedback, and reporting of intermediate outcome levels or of level of medication management may enhance the effectiveness of these programs.

A Delphi consensus on criteria for contraindications, assessment indicators and expected outcomes related to tibialis posterior transfer surgery

A team of experts in the field of reconstructive surgery for leprosy-affected people was identified. Using the Delphi method, an exercise was undertaken to ascertain whether a consensus on essential criteria and indicators for Tibialis Posterior Transfer (TPT) could be reached among the team. This paper describes the Delphi Exercise, giving results at each stage of consensus development. The final outcome was that essential criteria, including contraindications for surgery, pre- and post- operative assessments and expected outcomes, were agreed. The criteria are presented with recommendations.

[Making repeated clinical trials aiming at finding specific indications is the way for improving clinical efficacy of traditional Chinese medicine]

The modern clinical evaluation methods and index system are the tools for clinical evaluation in traditional Chinese medical trial, and more importantly the prerequisite for approbation of their outcomes. From the view of syndrome differentiation theory of traditional Chinese medicine, the previous therapeutic process is also a diagnostic process for the following therapeutic course, so to make repeated clinical trials aiming at finding specific indications would be one of the important ways to improve the clinical efficacy of traditional Chinese medicine.