The importance of disease prevalence in transporting clinical prediction rules. The case of streptococcal pharyngitis

Why do probabilistic clinical models fail to transport between sites

The rising popularity of artificial intelligence in healthcare is highlighting the problem that a computational model achieving super-human clinical performance at its training sites may perform substantially worse at new sites. In this perspective, we argue that we should typically expect this failure to transport, and we present common sources for it, divided into those under the control of the experimenter and those inherent to the clinical data-generating process. Of the inherent sources we look a little deeper into site-specific clinical practices that can affect the data distribution, and propose a potential solution intended to isolate the imprint of those practices on the data from the patterns of disease cause and effect that are the usual target of probabilistic clinical models.

Diagnostic performance of clinical prediction rules to detect group A beta-haemolytic streptococci in people with acute pharyngitis: a systematic review

OBJECTIVE

To assess and compare the diagnostic performance of Clinical Prediction Rules (CPRs) developed to detect group A Beta-haemolytic streptococci in people with acute pharyngitis (or sore throat).

STUDY DESIGN

A systematic review.

METHODS

We searched PubMed, Embase and Web of Science (inception-September 2022) for studies deriving and/or validating CPRs comprised of ≥2 predictors from an individual's history or physical examination. Two authors independently screened articles, extracted data and assessed risk of bias in included studies. A meta-analysis was not possible due to heterogeneity. Instead we compared the performance of CPRs when they were validated in the same study population (head-to-head comparisons). We used a modified grading of recommendations, assessment, development, and evaluations (GRADE) approach to assess certainty of the evidence.

RESULTS

We included 63 studies, all judged at high risk of bias. Of 24 derived CPRs, 7 were externally validated (in 46 external validations). Five validation studies provided data for head-to-head comparison of four pairs of CPRs. Very low certainty evidence favoured the Centor CPR over the McIsaac (2 studies) and FeverPain CPRs (1 study) and found the Centor CPR was equivalent to the Walsh CPR (1 study). The AbuReesh and Steinhoff 2005 CPRs had a similar poor discriminative ability (1 study). Within and between study comparisons suggested the performance of the Centor CPR may be better in adults (>18 years).

CONCLUSION

Very low certainty evidence suggests a better performance of the Centor CPR. When deciding about antibiotic prescribing for pharyngitis patients, involving patients in a shared decision making discussion about the likely benefits and harms, including antibiotic resistance, is recommended. Further research of higher rigour, which compares CPRs across multiple settings, is needed.

A 10- and 15-year performance analysis of ESC/EAS and ACC/AHA cardiovascular risk scores in a Southern European cohort

Background

A key strategy for the primary prevention of cardiovascular disease (CVD) is the use of risk prediction algorithms. We aimed to investigate the predictive ability of SCORE (Systematic COronary Risk Estimation) and PCE (Pooled Cohort Equations) systems for atherosclerotic CVD (ASCVD) risk in Portugal, a low CVD risk country, at the 10-year landmark and at a longer, 15-year follow-up.

Methods

The SCORE and PCE 10-year risk estimates were calculated for 455 and 448 patients, respectively. Discrimination was assessed by Harrell’s C-statistic. Calibration was analyzed by standardized incidence ratios (SIR).

Results

During the 10-year follow-up, 7 fatal ASCVD events (the SCORE outcome) and 32 any ASCVD events (the PCE outcome) occurred. The SCORE system showed good discrimination (C-statistic 0.83), while the PCE showed poor discrimination (C-statistic 0.62). Calibration was similar for both systems, according to SIR: SCORE, 0.3 (95% CI 0.1–0.7); PCE, 0.5 (95% CI 0.4–0.7). Globally, both 10-year fatal ASCVD risk and any ASCVD risk were overestimated in the overall population and men. However, the risk was underestimated by both systems in women. Despite an overestimation of 15-year fatal ASCVD by SCORE, the 15-year any ASCVD observed incidence was 1.8 times the 10-year incidence among men and 1.4 times among women. This acceleration of CVD risk was more relevant in the lowest classes of ASCVD risk.

Conclusion

In this prospective, contemporary, Portuguese cohort, the SCORE had better discriminatory power and similar calibration compared to PCE. However, both risk scores underestimated 10-year ASCVD risk in women.

Developing a framework for evidence-based grading and assessment of predictive tools for clinical decision support

BACKGROUND

Clinical predictive tools quantify contributions of relevant patient characteristics to derive likelihood of diseases or predict clinical outcomes. When selecting predictive tools for implementation at clinical practice or for recommendation in clinical guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools, most of which have never been implemented or assessed for comparative effectiveness. To overcome this challenge, we have developed a conceptual framework to Grade and Assess Predictive tools (GRASP) that can provide clinicians with a standardised, evidence-based system to support their search for and selection of efficient tools.

METHODS

A focused review of the literature was conducted to extract criteria along which tools should be evaluated. An initial framework was designed and applied to assess and grade five tools: LACE Index, Centor Score, Well's Criteria, Modified Early Warning Score, and Ottawa knee rule. After peer review, by six expert clinicians and healthcare researchers, the framework and the grading of the tools were updated.

RESULTS

GRASP framework grades predictive tools based on published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports a positive conclusion. Ottawa knee rule had the highest grade since it has demonstrated positive post-implementation impact on healthcare. LACE Index had the lowest grade, having demonstrated only pre-implementation positive predictive performance.

CONCLUSION

GRASP framework builds on widely accepted concepts to provide standardised assessment and evidence-based grading of predictive tools. Unlike other methods, GRASP is based on the critical appraisal of published evidence reporting the tools' predictive performance before implementation, potential effect and usability during implementation, and their post-implementation impact. Implementing the GRASP framework as an online platform can enable clinicians and guideline developers to access standardised and structured reported evidence of existing predictive tools. However, keeping GRASP reports up-to-date would require updating tools' assessments and grades when new evidence becomes available, which can only be done efficiently by employing semi-automated methods for searching and processing the incoming information.

Effect of model updating strategies on the performance of prevalent diabetes risk prediction models in a mixed-ancestry population of South Africa

Background

Prediction model updating methods are aimed at improving the prediction performance of a model in a new setting. This study sought to critically assess the impact of updating techniques when applying existent prevalent diabetes prediction models to a population different to the one in which they were developed, evaluating the performance in the mixed-ancestry population of South Africa.

Methods

The study sample consisted of 1256 mixed-ancestry individuals from the Cape Town Bellville-South cohort, of which 173 were excluded due to previously diagnosed diabetes and 162 individuals had undiagnosed diabetes. The primary outcome, undiagnosed diabetes, was based on an oral glucose tolerance test. Model updating techniques and prediction models were identified via recent systematic reviews. Model performance was assessed using the C-statistic and expected/observed (E/O) events rates ratio.

Results

Intercept adjustment and logistic calibration improved calibration across all five models (Cambridge, Kuwaiti, Omani, Rotterdam and Simplified Finnish diabetes risk models). This was improved further by model revision, where likelihood ratio tests showed that the effect of body mass index, waist circumference and family history of diabetes required additional adjustment (Omani, Rotterdam and Finnish models). However, discrimination was poor following internal validation of these models. Re-estimation of the regression coefficients did not increase performance, while the addition of new variables resulted in the highest discriminatory and calibration performance combination for the models it was undertaken in.

Conclusions

While the discriminatory performance of the original existent models during external validation were higher, calibration was poor. The highest performing models, based on discrimination and calibration, were the Omani diabetes model following model revision, and the Cambridge diabetes risk model following the addition of waist circumference as a predictor. However, while more extensive methods incorporating development population information were superior over simpler methods, the increase in model performance was not great enough for recommendation.

Evaluation of the Pooled Cohort Risk Equations for Cardiovascular Risk Prediction in a Multiethnic Cohort From the Women's Health Initiative

IMPORTANCE

Atherosclerotic cardiovascular disease (ASCVD) kills approximately 1 in every 3 US women. Current cholesterol, hypertension, and aspirin guidelines recommend calculating 10-year risk of ASCVD using the 2013 Pooled Cohort Equations (PCE). However, numerous studies have reported apparent overestimation of risk with the PCE, and reasons for overestimation are unclear.

OBJECTIVE

We evaluated the predictive accuracy of the PCE in the Women's Health Initiative (WHI), a multiethnic cohort of contemporary US postmenopausal women. We evaluated the effects of time-varying treatments such as aspirin and statins, and ascertainment of additional ASCVD events by linkage with the Centers for Medicare and Medicaid Services (CMS) claims.

DESIGN, SETTING, AND PARTICIPANTS

The WHI recruited the largest number of US women (n = 161 808) with the racial/ethnic, geographic, and age diversity of the general population (1993-1998). For this study, we included women aged 50 to 79 (n = 19 995) participating in the WHI with data on the risk equation variables at baseline and who met the guideline inclusion and exclusion criteria. Median follow-up was 10 years.

MAIN OUTCOMES AND MEASURES

For this study, ASCVD was defined as myocardial infarction, stroke, or cardiovascular death.

RESULTS

Among the 19 995 women (mean [SD] age, 64 [7.3] years; 8305 [41.5%] white, 7688 [38.5%] black, 3491 [17.5%] Hispanic, 103 [0.5%] American Indian, 321 [1.6%] Asian/Pacific Islander, and 87 [0.4%] other/unknown), a total of 1236 ASCVD events occurred in 10 years and were adjudicated through medical record review by WHI investigators. The WHI-adjudicated observed risks were lower than predicted. The observed (predicted) risks for baseline 10-year risk categories less than 5%, 5% to less than 7.5%, 7.5% to less than 10%, and 10% or more were 1.7 (2.8), 4.4 (6.2), 5.3 (8.7), and 12.4 (18.2), respectively. Small changes were noted after adjusting for time-dependent changes in statin and aspirin use. Among women 65 years or older enrolled in Medicare, WHI-adjudicated risks were also lower than predicted, but observed (predicted) risks became aligned after including events ascertained by linkage with CMS for additional surveillance for events: 3.8 (4.3), 7.1 (6.4), 8.3 (8.7), and 18.9 (18.7), respectively. Similar results were seen across ethnic/racial groups. Overall, the equations discriminated risk well (C statistic, 0.726; 95% CI, 0.714-0.738).

CONCLUSIONS AND RELEVANCE

Without including surveillance for ASCVD events using CMS, observed risks in the WHI were lower than predicted by PCE as noted in several other US cohorts, but risks were better aligned after including CMS events.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT00000611.

External validation study of a clinical decision aid to reduce unnecessary antibiotic prescriptions in women with acute cystitis

Background

Empirical prescribing of antibiotics to women with symptoms of acute cystitis prior to culture results is common, but subsequent culture results are often negative. A clinical decision aid for prescribing decisions in acute cystitis was previously developed that could reduce these unnecessary antibiotic prescriptions but has not been validated. This study sought to validate this decision aid for empirical antibiotic prescribing decisions in a new cohort of women with suspected acute cystitis.

Methods

External validation study of a clinical decision aid in 397 women with symptoms of acute cystitis, involving 230 Canadian family practitioners across Canada between 2009 and 2011. The sensitivity and specificity of the decision aid compared to a gold standard positive urine culture (defined as ≥10² cfu/ml (≥10⁵ CFU/L)) was determined, and compared with physician management, and the earlier development cohort study estimates. Other outcomes assessed were total antibiotic prescriptions, unnecessary antibiotics for negative urine cultures, and recommendations for urine culture testing. Chi-square tests were used for unpaired comparisons, adjusted for physician clustering. McNemar’s test was used for paired comparisons.

Results

There were 245/397 (61.7%) positive urine cultures. The cystitis aid sensitivity was 202/245 (82.5%, 95% Confidence Interval (CI)) = 77.1%, 86.8%), compared to 167/208 (80.3%) in the previous development cohort (p = 0.54), and 239/245 (97.6%) by family physicians in the current study (p < 0.001). Specificity was low for physicians (10/152, 6.6%) compared to the decision aid (54/152, 35.5%; p < 0.001, resulting in more antibiotic prescriptions by physicians (381/397, 96.0%) than would occur with decision aid recommendations (300/397, 75.6%, p < 0.001). Unnecessary antibiotic prescriptions where urine cultures were negative would be reduced an absolute 11.1% with cystitis aid recommendations (98/397, 24.7%) compared to usual physician care (142/397, 35.8%; p = 0.001). Urine cultures would also be reduced (97/397, 24.4% decision aid vs 351/397, 88.4% physicians; p < 0.001).

Conclusions

A 3-item clinical decision aid demonstrated reproducible accuracy in two cohorts of women with acute cystitis symptoms. Clinically important reductions in total and unnecessary antibiotic use, as well as urine culture testing, could result with routine clinical use compared to current empirical physician management practices.

Electronic supplementary material

The online version of this article (10.1186/s12875-017-0660-y) contains supplementary material, which is available to authorized users.

Subarachnoid hemorrhage admissions retrospectively identified using a prediction model

OBJECTIVE

To create an accurate prediction model using variables collected in widely available health administrative data records to identify hospitalizations for primary subarachnoid hemorrhage (SAH).

METHODS

A previously established complete cohort of consecutive primary SAH patients was combined with a random sample of control hospitalizations. Chi-square recursive partitioning was used to derive and internally validate a model to predict the probability that a patient had primary SAH (due to aneurysm or arteriovenous malformation) using health administrative data.

RESULTS

A total of 10,322 hospitalizations with 631 having primary SAH (6.1%) were included in the study (5,122 derivation, 5,200 validation). In the validation patients, our recursive partitioning algorithm had a sensitivity of 96.5% (95% confidence interval [CI] 93.9-98.0), a specificity of 99.8% (95% CI 99.6-99.9), and a positive likelihood ratio of 483 (95% CI 254-879). In this population, patients meeting criteria for the algorithm had a probability of 45% of truly having primary SAH.

CONCLUSIONS

Routinely collected health administrative data can be used to accurately identify hospitalized patients with a high probability of having a primary SAH. This algorithm may allow, upon validation, an easy and accurate method to create validated cohorts of primary SAH from either ruptured aneurysm or arteriovenous malformation.

Advances in Quantitative Techniques for Making Medical Decisions

Articles relating decision theory to medical problems have appeared in the medical literature over the past thirtyyears. Mostof the work important to recent advances has been performed in the past 10 to 15 years. We describe the progress which occurred between 1978 and 1988: an increased acceptance of quantitative approaches, advances in analytical techniques, simplification of older methods to enhance accessibility, and a better understanding of the interface between methods that prescribe how decisions should be made and those that describe how decisions actually are made. We also discuss problems that have not been overcome and those which reflect the "growing pains" of success, including increased scrutiny by non-methodologist medical content experts.

Adult prehospital scoring systems: a critical review

The goal of trauma triage is to get the right patient to the right place at the right time. In an attempt to achieve this objective numerous authors have developed a variety of prehospital triage tools. Comparative evaluation to determine which of the currently developed triage tools has the greatest predictive power has been prevented mainly because of the varying definitions of major trauma. The use of small data sets and the absence of model validation techniques have also prevented the true worth of these triage tools from being accurately determined. The aim of this article is to critically appraise the major prehospital triage tools.

Traumatic transfers: calibration is adversely affected when prediction models are transferred between trauma care contexts in India and the United States

OBJECTIVE

We evaluated the transferability of prediction models between trauma care contexts in India and the United States and explored updating methods to adjust such models for new contexts.

STUDY DESIGN AND SETTINGS

Using a combination of prospective cohort and registry data from 3,728 patients of Towards Improved Trauma Care Outcomes in India (TITCO) and from 18,756 patients of the US National Trauma Data Bank (NTDB), we derived models in one context and validated them in the other, assessing them for discrimination and calibration using systolic blood pressure, heart rate, and Glasgow coma scale as candidate predictors.

RESULTS

Early mortality was 8% in the TITCO and 1-2% in the NTDB samples. Both models discriminated well, but the TITCO model overestimated the risk of mortality in NTDB patients, and the NTDB model underestimated the risk in TITCO patients.

CONCLUSION

Transferability was good in terms of discrimination but poor in terms of calibration. It was possible to improve this miscalibration by updating the models' intercept. This updating method could be used in samples with as few as 25 events.

Canadian C-Spine Rule study for alert and stable trauma patients: II. Study objectives and methodology

Clinical prediction rules are decision-making tools that incorporate three or more variables from the history, physical examination or simple tests. They help clinicians make diagnostic or therapeutic decisions by standardizing the collection and interpretation of clinical data. There is growing interest in the methodological standards for their development and validation. This article describes the methods used to derive the Canadian C-Spine Rule and provides a valuable reference for investigators planning to develop future clinical prediction rules.

A simplified approach to the pooled analysis of calibration of clinical prediction rules for systematic reviews of validation studies

Objective

Estimating calibration performance of clinical prediction rules (CPRs) in systematic reviews of validation studies is not possible when predicted values are neither published nor accessible or sufficient or no individual participant or patient data are available. Our aims were to describe a simplified approach for outcomes prediction and calibration assessment and evaluate its functionality and validity.

Study design and methods:

Methodological study of systematic reviews of validation studies of CPRs: a) ABCD² rule for prediction of 7 day stroke; and b) CRB-65 rule for prediction of 30 day mortality. Predicted outcomes in a sample validation study were computed by CPR distribution patterns (“derivation model”). As confirmation, a logistic regression model (with derivation study coefficients) was applied to CPR-based dummy variables in the validation study. Meta-analysis of validation studies provided pooled estimates of “predicted:observed” risk ratios (RRs), 95% confidence intervals (CIs), and indexes of heterogeneity (I²) on forest plots (fixed and random effects models), with and without adjustment of intercepts. The above approach was also applied to the CRB-65 rule.

Results

Our simplified method, applied to ABCD² rule in three risk strata (low, 0–3; intermediate, 4–5; high, 6–7 points), indicated that predictions are identical to those computed by univariate, CPR-based logistic regression model. Discrimination was good (c-statistics =0.61–0.82), however, calibration in some studies was low. In such cases with miscalibration, the under-prediction (RRs =0.73–0.91, 95% CIs 0.41–1.48) could be further corrected by intercept adjustment to account for incidence differences. An improvement of both heterogeneities and P-values (Hosmer-Lemeshow goodness-of-fit test) was observed. Better calibration and improved pooled RRs (0.90–1.06), with narrower 95% CIs (0.57–1.41) were achieved.

Conclusion

Our results have an immediate clinical implication in situations when predicted outcomes in CPR validation studies are lacking or deficient by describing how such predictions can be obtained by everyone using the derivation study alone, without any need for highly specialized knowledge or sophisticated statistics.

Selective testing strategies for diagnosing group A streptococcal infection in children with pharyngitis: a systematic review and prospective multicentre external validation study

BACKGROUND

Several clinical prediction rules for diagnosing group A streptococcal infection in children with pharyngitis are available. We aimed to compare the diagnostic accuracy of rules-based selective testing strategies in a prospective cohort of children with pharyngitis.

METHODS

We identified clinical prediction rules through a systematic search of MEDLINE and Embase (1975-2014), which we then validated in a prospective cohort involving French children who presented with pharyngitis during a 1-year period (2010-2011). We diagnosed infection with group A streptococcus using two throat swabs: one obtained for a rapid antigen detection test (StreptAtest, Dectrapharm) and one obtained for culture (reference standard). We validated rules-based selective testing strategies as follows: low risk of group A streptococcal infection, no further testing or antibiotic therapy needed; intermediate risk of infection, rapid antigen detection for all patients and antibiotic therapy for those with a positive test result; and high risk of infection, empiric antibiotic treatment.

RESULTS

We identified 8 clinical prediction rules, 6 of which could be prospectively validated. Sensitivity and specificity of rules-based selective testing strategies ranged from 66% (95% confidence interval [CI] 61-72) to 94% (95% CI 92-97) and from 40% (95% CI 35-45) to 88% (95% CI 85-91), respectively. Use of rapid antigen detection testing following the clinical prediction rule ranged from 24% (95% CI 21-27) to 86% (95% CI 84-89). None of the rules-based selective testing strategies achieved our diagnostic accuracy target (sensitivity and specificity>85%).

INTERPRETATION

Rules-based selective testing strategies did not show sufficient diagnostic accuracy in this study population. The relevance of clinical prediction rules for determining which children with pharyngitis should undergo a rapid antigen detection test remains questionable.

PRImary care Streptococcal Management (PRISM) study: in vitro study, diagnostic cohorts and a pragmatic adaptive randomised controlled trial with nested qualitative study and cost-effectiveness study

BACKGROUND

Antibiotics are still prescribed to most patients attending primary care with acute sore throat, despite evidence that there is modest benefit overall from antibiotics. Targeting antibiotics using either clinical scoring methods or rapid antigen detection tests (RADTs) could help. However, there is debate about which groups of streptococci are important (particularly Lancefield groups C and G), and uncertainty about the variables that most clearly predict the presence of streptococci.

OBJECTIVE

This study aimed to compare clinical scores or RADTs with delayed antibiotic prescribing.

DESIGN

The study comprised a RADT in vitro study; two diagnostic cohorts to develop streptococcal scores (score 1; score 2); and, finally, an open pragmatic randomised controlled trial with nested qualitative and cost-effectiveness studies.

SETTING

The setting was UK primary care general practices.

PARTICIPANTS

Participants were patients aged ≥ 3 years with acute sore throat.

INTERVENTIONS

An internet program randomised patients to targeted antibiotic use according to (1) delayed antibiotics (control group), (2) clinical score or (3) RADT used according to clinical score.

MAIN OUTCOME MEASURES

The main outcome measures were self-reported antibiotic use and symptom duration and severity on seven-point Likert scales (primary outcome: mean sore throat/difficulty swallowing score in the first 2-4 days).

RESULTS

The IMI TestPack Plus Strep A (Inverness Medical, Bedford, UK) was sensitive, specific and easy to use. Lancefield group A/C/G streptococci were found in 40% of cohort 2 and 34% of cohort 1. A five-point score predicting the presence of A/C/G streptococci [FeverPAIN: Fever; Purulence; Attend rapidly (≤ 3 days); severe Inflammation; and No cough or coryza] had moderate predictive value (bootstrapped estimates of area under receiver operating characteristic curve: 0.73 cohort 1, 0.71 cohort 2) and identified a substantial number of participants at low risk of streptococcal infection. In total, 38% of cohort 1 and 36% of cohort 2 scored ≤ 1 for FeverPAIN, associated with streptococcal percentages of 13% and 18%, respectively. In an adaptive trial design, the preliminary score (score 1; n = 1129) was replaced by FeverPAIN (n = 631). For score 1, there were no significant differences between groups. For FeverPAIN, symptom severity was documented in 80% of patients, and was lower in the clinical score group than in the delayed prescribing group (-0.33; 95% confidence interval -0.64 to -0.02; p = 0.039; equivalent to one in three rating sore throat a slight rather than moderately bad problem), and a similar reduction was observed for the RADT group (-0.30; -0.61 to 0.00; p = 0.053). Moderately bad or worse symptoms resolved significantly faster (30%) in the clinical score group (hazard ratio 1.30; 1.03 to 1.63) but not the RADT group (1.11; 0.88 to 1.40). In the delayed group, 75/164 (46%) used antibiotics, and 29% fewer used antibiotics in the clinical score group (risk ratio 0.71; 0.50 to 0.95; p = 0.018) and 27% fewer in the RADT group (0.73; 0.52 to 0.98; p = 0.033). No significant differences in complications or reconsultations were found. The clinical score group dominated both other groups for both the cost/quality-adjusted life-years and cost/change in symptom severity analyses, being both less costly and more effective, and cost-effectiveness acceptability curves indicated the clinical score to be the most likely to be cost-effective from an NHS perspective. Patients were positive about RADTs. Health professionals' concerns about test validity, the time the test took and medicalising self-limiting illness lessened after using the tests. For both RADTs and clinical scores, there were tensions with established clinical experience.

CONCLUSIONS

Targeting antibiotics using a clinical score (FeverPAIN) efficiently improves symptoms and reduces antibiotic use. RADTs used in combination with FeverPAIN provide no clear advantages over FeverPAIN alone, and RADTs are unlikely to be incorporated into practice until health professionals' concerns are met and they have experience of using them. Clinical scores also face barriers related to clinicians' perceptions of their utility in the face of experience. This study has demonstrated the limitation of using one data set to develop a clinical score. FeverPAIN, derived from two data sets, appears to be valid and its use improves outcomes, but diagnostic studies to confirm the validity of FeverPAIN in other data sets and settings are needed. Experienced clinicians need to identify barriers to the use of clinical scoring methods. Implementation studies that address perceived barriers in the use of FeverPAIN are needed.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN32027234.

SOURCE OF FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 18, No. 6. See the NIHR Journals Library website for further project information.

Risk stratification of adult emergency department syncope patients to predict short-term serious outcomes after discharge (RiSEDS) study

Background

While Canadian ED physicians discharge most syncope patients with no specific further follow-up, approximately 5% will suffer serious outcomes after ED discharge. The goal of this study is to prospectively identify risk factors and to derive a clinical decision tool to accurately predict those at risk for serious outcomes after ED discharge within 30 days.

Methods/Design

We will conduct a prospective cohort study at 6 Canadian EDs to include adults with syncope and exclude patients with loss of consciousness > 5 minutes, mental status changes from baseline, obvious witnessed seizure, or head trauma prior to syncope. Emergency physicians will collect standardized clinical variables including historical features, physical findings, and results of immediately available tests (blood, ECG, and ED cardiac monitoring) prior to ED discharge/hospital admission. A second emergency physician will evaluate approximately 10% of study patients for interobserver agreement calculation of predictor variables. The primary outcome will be a composite serious outcome occurring within 30 days of ED discharge and includes three distinct categories: serious adverse events (death, arrhythmia); identification of serious underlying disease (structural heart disease, aortic dissection, pulmonary embolism, severe pulmonary hypertension, subarachnoid hemorrhage, significant hemorrhage, myocardial infarction); or procedures to treat the cause of syncope. The secondary outcome will be any of the above serious outcomes either suspected or those occurring in the ED. A blinded Adjudication Committee will confirm all serious outcomes. Univariate analysis will be performed to compare the predictor variables in patients with and without primary outcome. Variables with p-values <0.2 and kappa values ≥0.60 will be selected for stepwise logistic regression to identify the risk factors and to develop the clinical decision tool. We will enroll 5,000 patients (with 125 positive for primary outcome) for robust identification of risk factors and clinical decision tool development.

Discussion

Once successfully developed, this tool will accurately risk-stratify adult syncope patients; however, validation and implementation will still be required. This program of research should lead to standardized care of syncope patients, and improve patient safety.

The predictive ability of bleeding risk stratification models in very old patients on vitamin K antagonist treatment for venous thromboembolism: results of the prospective collaborative EPICA study

BACKGROUND

The optimal duration of anticoagulant treatment after venous thromboembolism (VTE) should be evaluated in relation to bleeding risk. This assessment is particularly difficult with elderly patients, because of their increased risk of both recurrences and hemorrhages. Bleeding risk stratification models have been proposed, but their predictive ability in very elderly patients is unknown. We aimed to assess six bleeding stratification models in this setting, by using information available in our dataset.

PATIENTS AND METHODS

Patients aged ≥ 80 years receiving vitamin K antagonists (VKAs) for the secondary prevention of VTE were eligible for this prospective cohort study. All patients were followed at Italian anticoagulation clinics for monitoring of VKA treatment. Risk factors for bleeding were collected, and major bleeding events and mortality were documented during follow-up. The association of bleeding events with the available risk factors was tested by means of Cox regression analysis; the c-statistic was used to quantify the predictive validity of the classification schemes.

RESULTS

A total of 1078 patients (37.2% males; mean age, 84 years) were enrolled in the study, for a total observation period of 1981 patient-years. The rate of major bleeding was 2.4 per 100 patient-years (47 events; one was fatal). The mortality rate was 5.2 per 100 patient-years. None of the considered risk factors were significantly associated with bleeding events. The predictive validity of the risk stratification models was low, and the most accurate model was not specifically developed for VTE patients (HEMORR2 HAGES, c-statistic 0.60, 95% confidence interval 0.49-0.70).

CONCLUSIONS

Bleeding risk stratification models appear to have little accuracy in very elderly VTE patients.

Evaluating the evidence for the implementation of C-reactive protein measurement in adult patients with suspected lower respiratory tract infection in primary care: a systematic review

BACKGROUND

Excessive prescription of antibiotics in patients with lower respiratory tract infection (LRTI) is common in primary care and might be reduced by rapid point-of-care (POC) C-reactive protein (CRP) testing. However, the exact benefits of this test are unclear.

OBJECTIVE

To review the available evidence for the role of POC CRP measurement in (i) guiding antibiotic prescription, (ii) predicting aetiology, (iii) prognosis and (iv) diagnosis (pneumonia) in LRTI patients.

METHODS

For each research question, studies were retrieved through an electronic literature search in Medline, Embase and the Cochrane Library using synonyms for CRP and LRTI combined with different relevant subheadings. Study quality was assessed using validated instruments and predefined outcome measures were extracted from each study.

RESULTS

The search yielded 13 articles, each answering one or more questions; one was excluded by insufficient internal validity. (i) One of four studies showed a significant reduction in the antibiotic prescriptions when applying POC CRP measurement [relative risk (RR) 0.6, 95% confidence interval (CI) 0.5-0.7]. (ii) Three studies on aetiology demonstrated that an elevated CRP was associated with bacterial [odds ratio (OR) 2.46-4.8] and one with viral (OR 2.7) aetiology. (iii) Results on the prognostic value were contradictory, providing evidence for faster symptom resolution (RR 1.16, 95% CI 1.1-1.3), higher mortality rate (RR 2.5, 95% CI 1.2-5.1) and no difference in outcome in patients with high CRP levels. (iv) Four studies showed that CRP had limited value as a single predictor of pneumonia. When combined with clinical assessment, its value increased according to two of these studies (receiver operating characteristic area from 0.7 to 0.9). However, methodological flaws and/or wide CIs limit the generalizability of findings in all studies.

CONCLUSION

The evidence for the benefits of POC CRP measurement in LRTI patients in primary care is limited, contradictory and does not support its use to guide treatment decisions yet.

Chapter 12: systematic review of prognostic tests

A number of new biological markers are being studied as predictors of disease or adverse medical events among those who already have a disease. Systematic reviews of this growing literature can help determine whether the available evidence supports use of a new biomarker as a prognostic test that can more accurately place patients into different prognostic groups to improve treatment decisions and the accuracy of outcome predictions. Exemplary reviews of prognostic tests are not widely available, and the methods used to review diagnostic tests do not necessarily address the most important questions about prognostic tests that are used to predict the time-dependent likelihood of future patient outcomes. We provide suggestions for those interested in conducting systematic reviews of a prognostic test. The proposed use of the prognostic test should serve as the framework for a systematic review and to help define the key questions. The outcome probabilities or level of risk and other characteristics of prognostic groups are the most salient statistics for review and perhaps meta-analysis. Reclassification tables can help determine how a prognostic test affects the classification of patients into different prognostic groups, hence their treatment. Review of studies of the association between a potential prognostic test and patient outcomes would have little impact other than to determine whether further development as a prognostic test might be warranted.

Predicting streptococcal pharyngitis in adults in primary care: a systematic review of the diagnostic accuracy of symptoms and signs and validation of the Centor score

BACKGROUND

Stratifying patients with a sore throat into the probability of having an underlying bacterial or viral cause may be helpful in targeting antibiotic treatment. We sought to assess the diagnostic accuracy of signs and symptoms and validate a clinical prediction rule (CPR), the Centor score, for predicting group A β-haemolytic streptococcal (GABHS) pharyngitis in adults (> 14 years of age) presenting with sore throat symptoms.

METHODS

A systematic literature search was performed up to July 2010. Studies that assessed the diagnostic accuracy of signs and symptoms and/or validated the Centor score were included. For the analysis of the diagnostic accuracy of signs and symptoms and the Centor score, studies were combined using a bivariate random effects model, while for the calibration analysis of the Centor score, a random effects model was used.

RESULTS

A total of 21 studies incorporating 4,839 patients were included in the meta-analysis on diagnostic accuracy of signs and symptoms. The results were heterogeneous and suggest that individual signs and symptoms generate only small shifts in post-test probability (range positive likelihood ratio (+LR) 1.45-2.33, -LR 0.54-0.72). As a decision rule for considering antibiotic prescribing (score ≥ 3), the Centor score has reasonable specificity (0.82, 95% CI 0.72 to 0.88) and a post-test probability of 12% to 40% based on a prior prevalence of 5% to 20%. Pooled calibration shows no significant difference between the numbers of patients predicted and observed to have GABHS pharyngitis across strata of Centor score (0-1 risk ratio (RR) 0.72, 95% CI 0.49 to 1.06; 2-3 RR 0.93, 95% CI 0.73 to 1.17; 4 RR 1.14, 95% CI 0.95 to 1.37).

CONCLUSIONS

Individual signs and symptoms are not powerful enough to discriminate GABHS pharyngitis from other types of sore throat. The Centor score is a well calibrated CPR for estimating the probability of GABHS pharyngitis. The Centor score can enhance appropriate prescribing of antibiotics, but should be used with caution in low prevalence settings of GABHS pharyngitis such as primary care.

Evaluation of the safety of C-spine clearance by paramedics: design and methodology

BACKGROUND

Canadian Emergency Medical Services annually transport 1.3 million patients with potential neck injuries to local emergency departments. Less than 1% of those patients have a c-spine fracture and even less (0.5%) have a spinal cord injury. Most injuries occur before the arrival of paramedics, not during transport to the hospital, yet most patients are transported in ambulances immobilized. They stay fully immobilized until a bed is available, or until physician assessment and/or X-rays are complete. The prolonged immobilization is often unnecessary and adds to the burden of already overtaxed emergency medical services systems and crowded emergency departments.

METHODS/DESIGN

The goal of this study is to evaluate the safety and potential impact of an active strategy that allows paramedics to assess very low-risk trauma patients using a validated clinical decision rule, the Canadian C-Spine Rule, in order to determine the need for immobilization during transport to the emergency department.This cohort study will be conducted in Ottawa, Canada with one emergency medical service. Paramedics with this service participated in an earlier validation study of the Canadian C-Spine Rule. Three thousand consecutive, alert, stable adult trauma patients with a potential c-spine injury will be enrolled in the study and evaluated using the Canadian C-Spine Rule to determine the need for immobilization. The outcomes that will be assessed include measures of safety (numbers of missed fractures and serious adverse outcomes), measures of clinical impact (proportion of patients transported without immobilization, key time intervals) and performance of the Rule.

DISCUSSION

Approximately 40% of all very low-risk trauma patients could be transported safely, without c-spine immobilization, if paramedics were empowered to make clinical decisions using the Canadian C-Spine Rule. This safety study is an essential step before allowing all paramedics across Canada to selectively immobilize trauma victims before transport. Once safety and potential impact are established, we intend to implement a multi-centre study to study actual impact.

Predicting postresection hydrocephalus in pediatric patients with posterior fossa tumors

OBJECT

Approximately 30% of children with posterior fossa tumors exhibit hydrocephalus after tumor resection. Recent literature has suggested that prophylactic endoscopic third ventriculostomy diminishes the risk of this event. Because the majority of patients will not have postoperative hydrocephalus, a preoperative clinical prediction rule that identifies patients at high or low risk for postresection hydrocephalus would be helpful to optimize the care of these children.

METHODS

The authors evaluated a derivation cohort of 343 consecutive children with posterior fossa tumors who underwent treatment between 1989 and 2003. Multivariate methods were used on these data to generate the Canadian Preoperative Prediction Rule for Hydrocephalus. The rule's estimated risk of postresection hydrocephalus was compared with risk observed in 111 independent patients in the validation cohort.

RESULTS

Variables identified as significant in predicting postresection hydrocephalus were age < 2 years (score of 3), papilledema (score of 1), moderate to severe hydrocephalus (score of 2), cerebral metastases (score of 3), and specific estimated tumor pathologies (score of 1). Patients with scores > or = 5 were deemed as high risk. Predicted probabilities for the high- and low-risk groups were 0.73 and 0.25, respectively, from the derivation cohort, and 0.59 and 0.14 after prevalence adjustment compared with the observed values of 0.42 and 0.17 in the validation cohort.

CONCLUSIONS

A patient's score on the Preoperative Prediction Rule for Hydrocephalus will allow improved patient counseling and surgical planning by identifying patients at high risk of developing postresection hydrocephalus. These patients might selectively be exposed to the risks of preresection CSF diversion to improve outcome.

A simple method to adjust clinical prediction models to local circumstances

Introduction

Clinical prediction models estimate the risk of having or developing a particular outcome or disease. Researchers often develop a new model when a previously developed model is validated and the performance is poor. However, the model can be adjusted (updated) using the new data. The updated model is then based on both the development and validation data. We show how a simple updating method may suffice to update a clinical prediction model.

Methods

A prediction model that preoperatively predicts the risk of severe postoperative pain was developed with multivariable logistic regression from the data of 1944 surgical patients in the Academic Medical Center Amsterdam, the Netherlands. We studied the predictive performance of the model in 1,035 new patients, scheduled for surgery at a later time in the University Medical Center Utrecht, the Netherlands. We assessed the calibration (agreement between predicted risks and the observed frequencies of an outcome) and discrimination (ability of the model to distinguish between patients with and without postoperative pain). When the incidence of the outcome is different, all predicted risks may be systematically over- or underestimated. Hence, the intercept of the model can be adjusted (updating).

Results

The predicted risks were systematically higher than the observed frequencies, corresponding to a difference in the incidence of postoperative pain between the development (62%) and validation set (36%). The updated model resulted in better calibration.

Discussion

When a clinical prediction model in new patients does not show adequate performance, an alternative to developing a new model is to update the prediction model with new data. The updated model will be based on more patient data, and may yield better risk estimates.

Diagnosis, diagnostic criteria, and consensus

Clinicians need accurate, reliable diagnostic criteria for the conditions they evaluate because this is the basis for rational, effective treatment and the estimation of a meaningful prognosis. Standardizing diagnostic approaches wherever this is feasible can reduce variations in care. The development of diagnostic scales has the potential to meet this goal. The approaches to the development of these scales should follow the same measurement principles used in the development of outcome instruments. The concept of consensus is fundamental to understanding how we make diagnoses and should be thought of as a potentially flexible and fluid idea that can accommodate the discovery of new knowledge in our efforts to accurately diagnose medical conditions.

Updating methods improved the performance of a clinical prediction model in new patients

OBJECTIVE

Ideally, clinical prediction models are generalizable to other patient groups. Unfortunately, they perform regularly worse when validated in new patients and are then often redeveloped. While the original prediction model usually has been developed on a large data set, redevelopment then often occurs on the smaller validation set. Recently, methods to update existing prediction models with the data of new patients have been proposed. We used an existing model that preoperatively predicts the risk of severe postoperative pain (SPP) to compare five updating methods.

STUDY DESIGN AND SETTING

The model was tested and updated with a set of 752 new patients (274 [36] with SPP). We studied the discrimination (ability to distinguish between patients with and without SPP) and calibration (agreement between the predicted risks and observed frequencies of SPP) of the five updated models in 283 other patients (100 [35%] with SPP).

RESULTS

Simple recalibration methods improved the calibration to a similar extent as revision methods that made more extensive adjustments to the original model. Discrimination could not be improved by any of the methods.

CONCLUSION

When the performance is poor in new patients, updating methods can be applied to adjust the model, rather than to develop a new model.

A matched-pair cluster design study protocol to evaluate implementation of the Canadian C-spine rule in hospital emergency departments: Phase III

BACKGROUND

Physicians in Canadian emergency departments (EDs) annually treat 185,000 alert and stable trauma victims who are at risk for cervical spine (C-spine) injury. However, only 0.9% of these patients have suffered a cervical spine fracture. Current use of radiography is not efficient. The Canadian C-Spine Rule is designed to allow physicians to be more selective and accurate in ordering C-spine radiography, and to rapidly clear the C-spine without the need for radiography in many patients. The goal of this phase III study is to evaluate the effectiveness of an active strategy to implement the Canadian C-Spine Rule into physician practice. Specific objectives are to: 1) determine clinical impact, 2) determine sustainability, 3) evaluate performance, and 4) conduct an economic evaluation.

METHODS

We propose a matched-pair cluster design study that compares outcomes during three consecutive 12-months "before," "after," and "decay" periods at six pairs of "intervention" and "control" sites. These 12 hospital ED sites will be stratified as "teaching" or "community" hospitals, matched according to baseline C-spine radiography ordering rates, and then allocated within each pair to either intervention or control groups. During the "after" period at the intervention sites, simple and inexpensive strategies will be employed to actively implement the Canadian C-Spine Rule. The following outcomes will be assessed: 1) measures of clinical impact, 2) performance of the Canadian C-Spine Rule, and 3) economic measures. During the 12-month "decay" period, implementation strategies will continue, allowing us to evaluate the sustainability of the effect. We estimate a sample size of 4,800 patients in each period in order to have adequate power to evaluate the main outcomes.

DISCUSSION

Phase I successfully derived the Canadian C-Spine Rule and phase II confirmed the accuracy and safety of the rule, hence, the potential for physicians to improve care. What remains unknown is the actual change in clinical behaviors that can be affected by implementation of the Canadian C-Spine Rule, and whether implementation can be achieved with simple and inexpensive measures. We believe that the Canadian C-Spine Rule has the potential to significantly reduce health care costs and improve the efficiency of patient flow in busy Canadian EDs.

Impact of clinical decision rules on clinical care of traumatic injuries to the foot and ankle, knee, cervical spine, and head

INTRODUCTION

Traumatic injuries to the ankle/foot, knee, cervical spine, and head are very commonly seen in emergency and accident departments around the world. There has been much interest in the development of clinical decision rules to help guide the investigations of these patients in a standardised and cost-effective manner.

METHODS

In this article we reviewed the impact of the Ottawa ankle rules, Ottawa knee rules, Canadian C-spine rule and the Canadian CT head rule.

RESULTS

The studies conducted have confirmed that the use of well developed clinical decision rules results in less radiography, less time spent in the emergency department and does not decrease patient satisfaction or result in misdiagnosis.

CONCLUSIONS

Emergency physicians around the world should adopt the use of clinical decision rules for ankle/foot, knee, cervical spine and minor head injuries. With relatively simple implementation strategies, care can be standardized and costs reduced while providing excellent clinical care.

Assessing the Likelihood of an Important Clinical Outcome: New Insights from a Comparison of Clinical and Actuarial Judgment

Purpose. To assess and rank the performance of different methods of predicting the probability of death following a specified surgical procedure. Method. Actuarial estimates of the probability of early mortality for 40 patients were derived from 2 sources: a large published surgical series and a smaller series from the center where surgery was performed. Surgeons and cardiologists also provided probability estimates for these patients. Results. Estimates derived from the published literature were too optimistic and did not differentiate between patients more, or less, likely to die (i.e., failed to discriminate). Doctors’ judgments were unbiased but failed to discriminate. Local actuarial estimates (influenced by only 1 or 2 variables) were unbiased, did discriminate, but exhibited more random variation. Conclusions. The preferred source of estimates depends upon which aspect of accuracy is of greatest importance. Differences in patient selection and error in the identification of risk factors mean that published results will not always appropriately predict surgical risk at other institutions. Risk stratification may be more robust when based on a small set of cross-validated predictors rather than a larger set of predictors that includes some whose reliability has not been confirmed.

Pharyngitis clinical prediction rules: effect of interobserver agreement: a MetroNet study

OBJECTIVE

Pharyngitis clinical prediction rules improve Group A beta-hemolytic streptococcus (GABHS) diagnosis and decrease unnecessary antibiotic use, yet few studies have addressed clinician variability in assessment of sore throat signs and symptoms.

STUDY DESIGN AND SETTING

We conducted a cross-sectional study in which two clinicians examined each of 200 adult sore throat patients. Each patient had a rapid GABHS antigen test. Clinicians were blinded to each other's assessment and to the rapid antigen result. Interobserver agreement was estimated using a kappa coefficient. Effect of agreement on sensitivity, specificity, and hypothetic rapid antigen testing and antibiotic prescribing was determined for two clinical prediction rules.

RESULTS

We found moderate inter-rater reliability on sore throat history and physical assessments. Clinician agreement was associated with significantly fewer hypothetic rapid antigen tests performed.

CONCLUSION

Interobserver agreement enhances the utility of pharyngitis clinical prediction rules. Medical school and residency training should focus on correct assessment of history and physical examination components used in GABHS clinical prediction rules. Correct assessment will result in less GABHS testing and antibiotic prescriptions for sore throat patients.

The use of CRP tests in patients with respiratory tract infections in primary care in Sweden can be questioned

A diagnosis-antibiotic prescribing study was performed in 5 counties in Sweden during 1 week in November in 2000 and 2002 respectively. As a part of the study, the use and results of C-reactive protein (CRP) tests in relation to duration of symptoms and antibiotic prescribing in 6778 patients assigned a diagnosis of respiratory tract infections were analysed. In almost half (42%) of the patients, a CRP test was performed. The majority of CRP tests (69%) were performed in patients assigned diagnosis upper respiratory tract infection, where the test is not recommended. Overall, there was a minor decrease in antibiotic prescribing when CRP was used (41%), in comparison to 44% of the patients where no CRP was performed (p < 0.01). Patients assigned diagnoses implying a bacterial aetiology were prescribed antibiotics irrespective of result of CRP or length of symptoms before consultation. For patients assigned viral diagnoses, antibiotic prescribing increased with increasing duration of symptoms and increasing value of CRP. The use of CRP decreased antibiotic prescribing in patients assigned to viral diagnoses and with longstanding symptoms (p < 0.001). However, 59% of the patients assigned viral diagnoses with CRP > or = 25 received antibiotics, which seems to indicate a misinterpretation of CRP and a non-optimal use of antibiotics.

Preoperative prediction of severe postoperative pain

We developed and validated a prediction rule for the occurrence of early postoperative severe pain in surgical inpatients, using predictors that can be easily documented in a preoperative setting. A cohort of surgical inpatients (n=1416) undergoing various procedures except cardiac surgery and intracranial neurosurgery in a University Hospital were studied. Preoperatively the following predictors were collected: age, gender, type of scheduled surgery, expected incision size, blood pressure, heart rate, Quetelet index, the presence and severity of preoperative pain, health-related quality of life the (SF-36), Spielberger's State-Trait Anxiety Inventory (STAI) and the Amsterdam Preoperative Anxiety and Information Scale (APAIS). The outcome was the presence of severe postoperative pain (defined as Numeric Rating Scale > or =8) within the first hour postoperatively. Multivariate logistic regression in combination with bootstrapping techniques (as a method for internal validation) was used to derive a stable prediction model. Independent predictors of severe postoperative pain were younger age, female gender, level of preoperative pain, incision size and type of surgery. The area under the receiver operator characteristic (ROC) curve was 0.71 (95% CI: 0.68-0.74). Adding APAIS scores (measures of preoperative anxiety and need for information), but not STAI, provided a slightly better model (ROC area 0.73). The reliability of this extended model was good (Hosmer and Lemeshow test p-value 0.78). We have demonstrated that severe postoperative pain early after awakening from general anesthesia can be predicted with a scoring rule, using a small set of variables that can be easily obtained from all patients at the preoperative visit. Before this internally validated preoperative prediction rule can be applied in clinical practice to support anticipatory pain management, external validation in other clinical settings is necessary.

A neural network aid for the early diagnosis of cardiac ischemia in patients presenting to the emergency department with chest pain

STUDY OBJECTIVE

Chest pain is the second most common chief complaint presented to the emergency department. Although the causes of chest pain span the clinical spectrum from the trivial to the life threatening, it is often difficult to identify which patients have the most common life-threatening cause, cardiac ischemia. Because of the potential for poor outcome if this diagnosis is missed, physicians have had a low threshold for admitting patients with chest pain to the hospital, the vast majority of whom are found not to have cardiac ischemia. In an earlier study with a large chest pain patient registry, an artificial neural network was shown to be able to identify the subset of patients who present to the ED with chest pain who have sustained acute myocardial infarction. The objective of this study was to use the same registry to determine whether a network could be trained accurately to identify the larger subset of patients who have cardiac ischemia.

METHODS

Two thousand two hundred four adult patients presenting to the ED with chest pain who received an ECG were used to train and test an artificial neural network to recognize the presence of cardiac ischemia. Only the data available at the time of initial patient contact were used to replicate the conditions of real-time evaluation. Forty variables from patient history, physical examination, ECG, and the first set of chemical cardiac marker determinations were used to train and subsequently test the network. The network was trained and tested by using the jackknife variance technique to allow for the network to be trained on as many of the features of the small subset of ischemic patients as possible. Network accuracy was compared with 2 existing aids to the diagnosis of cardiac ischemia, as well as a derived regression model.

RESULTS

The network had a sensitivity of 88.1% (95% confidence interval [CI] 84.8% to 91.4%) and a specificity of 86.2% (95% CI 84.6% to 87.7%) for cardiac ischemia despite the fact that a mean of 5% of all required network input data and 41% of cardiac chemical marker data were missing. The network also performed more accurately than the 3 other tested approaches.

CONCLUSION

These data suggest that an artificial neural network might be able to identify which patients who present to the ED with chest pain have cardiac ischemia with useful sensitivities and specificities.

Exercise testing scores as an example of better decisions through science

INTRODUCTION

The application of common statistical techniques to clinical and exercise test data has the potential to become a useful tool for assisting in the diagnosis of coronary artery disease, assessing prognosis, and reducing the cost of evaluating patients with suspected coronary disease. Since general practitioners function as gatekeepers and decide which patients must be referred to the cardiologist, they need to optimally use the basic tools they have available (i.e., history, physical exam, and the exercise test).

METHODS

Review of the literature with a focus on the scientific techniques for aiding the decision-making process.

RESULTS

Scores derived from multivariable statistical techniques considering clinical and exercise data have demonstrated superior discriminating power when compared using receiver-operating-characteristic curves with the ST segment response. In addition, by stratifying patients as to probability of disease and prognosis, they provide a management strategy. While computers as part of information management systems can calculate complicated equations to provide scores, physicians are reluctant to trust them. Thus, these scores have been represented as nomograms or simple additive tables so physicians are comfortable with their application. Scores have also been compared with physician judgment and been found to estimate the presence of coronary disease and prognosis as well as expert cardiologists, and often better than nonspecialists.

CONCLUSION

Multivariate scores can empower the clinician to assure the cardiac patient with access to appropriate and cost-effective cardiological care.

Validation of a clinical prediction rule to reduce preoperative type and screen procedures

BACKGROUND

We have developed a prediction rule for the occurrence of perioperative red blood cell transfusion to help to reduce the number of unnecessary preoperative type and screen procedures. We evaluated the robustness of this prediction rule in patients from another hospital.

METHODS

The rule was retrospectively applied to 1282 consecutive patients ('validation set') who underwent similar surgical procedures to the patients in the derivation study. The outcome was similarly defined as any allogeneic transfusion on the day of surgery or during the first postoperative day. The predictive value of the rule was assessed using a Receiver Operating Characteristic curve (ROC) and compared with the results of the derivation study. Subsequently, the number of correctly predicted transfusions was compared.

RESULTS

The patient characteristics did not differ between the two sets, except for the incidence of transfusion (derivation study: 18%; present study: 8%). In the validation set, the ROC area of the prediction rule was 0.78 (95% confidence intervals [CI]: 0.73-0.82), which was within the CI of the ROC area found in the derivation study (0.75; 95% CI: 0.72-0.79). In total, 35% of the type and screen procedures could be omitted (derivation study: 50%), with 13% missed transfused patients (derivation study: 20%).

CONCLUSIONS

After comparing the results of this validation study with that of the derivation study, the prediction rule was robust and may work in other clinics as well.

Diagnosis of strep throat in adults: are clinical criteria really good enough?

The clinical manifestations of group A streptococcal and nonstreptococcal pharyngitis overlap quite broadly. For this reason, the updated Infectious Diseases Society of America practice guideline for group A streptococcal pharyngitis, published in this issue of Clinical Infectious Diseases, recommends laboratory confirmation of the clinical diagnosis by means of either throat culture or a rapid antigen detection test. However, a recently published guideline, developed by a subcommittee of the American College of Physicians-American Society of Internal Medicine (ACP-ASIM) in collaboration with the Centers for Disease Control and Prevention, advocates use of a clinical algorithm alone, in lieu of microbiologic testing, for confirmation of the diagnosis in adults for whom the suspicion of streptococcal infection is high. In this discussion, we examine the assumptions of the ACP-ASIM guideline, question whether its recommendations will achieve the stated objective of dramatically decreasing excess antibiotic use, and suggest that its recommendations be confirmed by clinical trials before clinicians abandon long-held teachings regarding diagnosis and management of group A streptococcal pharyngitis.

A neural computational aid to the diagnosis of acute myocardial infarction

STUDY OBJECTIVE

Accurate identification of the presence of acute myocardial infarction in adult patients who present to the emergency department with anterior chest pain remains elusive. The artificial neural network is a powerful nonlinear statistical paradigm for the recognition of complex patterns, with the ability to maintain accuracy when some data required for network function are missing. Earlier studies revealed that the artificial neural network is able to accurately identify acute myocardial infarction in patients experiencing chest pain. However, these studies did not measure network performance in real time, when a significant amount of data required for network function may not be available. They also did not use chemical cardiac marker data.

METHODS

Two thousand two hundred four adult patients presenting to the ED with anterior chest pain were used to train an artificial neural network to recognize the presence of acute myocardial infarction. Only data available at the time of initial patient evaluation were used to replicate the conditions of real-time patient evaluation. Forty variables from patient histories, physical examinations, ECG results, and chemical cardiac marker determinations were used to train and then test the network.

RESULTS

The network correctly identified 121 of the 128 patients (sensitivity 94.5%; 95% confidence interval 90.6% to 97.9%) with myocardial infarction at a specificity of 95.9% (95% confidence interval 93.0% to 98.5%), despite the fact that an average of 5% (individual range 0% to 35%) of the input data required by the network were missing on all patients.

CONCLUSION

Network accuracy and the maintenance of that accuracy when some data required for function are unavailable suggest that the artificial neural network may be a potential real time aid to the diagnosis of acute myocardial infarction during initial patient evaluation.

Better decisions through science: exercise testing scores

Statistical tools can be used to create scores for assisting in the diagnosis of coronary artery disease and assessing prognosis. General practitioners and internists frequently function as gatekeepers, deciding which patients must be referred to the cardiologist. Therefore, they need to use the basic tools they have available (ie, history, physical examination and the exercise test) in an optimal fashion. Scores derived from multivariable statistical techniques considering clinical and exercise data have demonstrated superior discriminating power compared with diagnosis only using the ST segment response. In addition, by stratifying patients as to probability of disease and prognosis, they provide a more practical management strategy than a response of normal or abnormal. Although computers, as part of information management systems, can calculate complicated equations and derive these scores, physicians are reluctant to trust them. However, when represented as nomograms or simple additive discrete pieces of information, scores are more readily accepted. The scores have been compared with physician judgment and have been found to estimate the presence of coronary disease and prognosis as well as expert cardiologists and often better than nonspecialists. However, the discriminating power of specific variables from the medical history and exercise test remains unclear because of inadequate study design and differences in study populations. Should expired gases be substituted for estimated METs? Should ST/heart rate index be used instead of putting ST depression and heart rate separately into the models? Should right-sided chest leads and heart rate in recovery be considered? There is a need for further evaluation of these easily obtained variables to improve the accuracy of prediction algorithms, especially in women. The portability and reliability of scores must be ensured because access to specialized care must be safeguarded. Assessment of the clinical and exercise test data and application of the newer scores can empower the clinician to assure the cardiac patient access to appropriate and cost-effective cardiologic care.

Prediction of stroke in the general population in Europe (EUROSTROKE): Is there a role for fibrinogen and electrocardiography?

BACKGROUND

To decide whether a person with certain characteristics should be given any kind of intervention to prevent a cardiovascular event, it would be helpful to classify subjects in low, medium and high risk categories. The study evaluated which well known cerebrovascular and cardiovascular correlates, in particular fibrinogen level and ECG characteristics, are able to predict the occurrence of stroke in men of the general population using data from three European cohorts participating in EUROSTROKE.

METHODS

EUROSTROKE is a collaborative project among ongoing European population based cohort studies and designed as a prospective nested case-control study. For each stroke case two controls were sampled. Strokes were classified according to MONICA criteria or reviewed by a panel of four neurologists. Complete data were available of 698 men (219 stroke events) from cohorts in Cardiff (84 cases/200 controls), Kuopio (74/148) and Rotterdam (61/131). Multivariable logistic regression modeling was used to evaluate which information from history, physical examination (for example, blood pressure), blood lipids, and fibrinogen and ECG measurements independently contributed to the prediction of stroke. The area under receiver operating characteristic curve (ROC area) was used to estimate the predictive ability of models.

RESULTS

Independent predictors from medical history and physical examination were age, stroke history, medically treated hypertension, smoking, diabetes mellitus and diastolic blood pressure. The ROC area of this model was 0.69. After validating and transforming this model to an easy applicable rule, 40% of all future stroke cases could be predicted. Adding pulse rate, body mass index, blood lipids, fibrinogen level and ECG parameters did not improve the classification of subjects in low, medium and high risk. Results were similar when fibrinogen was dichotomised at the upper tertile or quintile.

CONCLUSION

In the general male population the future occurrence of stroke may be predicted using easy obtainable information from medical history and physical examination. Measurement of pulse rate, body mass index, blood lipids, fibrinogen level and ECG characteristics do not contribute to the risk stratification of stroke and have no value in the screening for stroke in the general male population.

Assessing patients with possible heart disease using scores

Multivariable analysis of clinical and exercise test data has the potential to become a useful tool for assisting in the diagnosis of coronary artery disease, assessing prognosis, and reducing the cost of evaluating patients with suspected coronary disease. Since general practitioners are functioning as gatekeepers and decide which patients must be referred to the cardiologist, they need to use the basic tools they have available (i.e. history, physical examination and the exercise test), in an optimal fashion. Scores derived from multivariable statistical techniques considering clinical and exercise data have demonstrated superior discriminating power compared with simple classification of the ST response. In addition, by stratifying patients as to probability of disease and prognosis, they provide a management strategy. While computers, as part of information management systems, can run complicated equations and derive these scores, physicians are reluctant to trust them. Thus, these scores have been represented as nomograms or simple additive tables so physicians are comfortable with their application. Their results have also been compared with physician judgment and found to estimate the presence of coronary disease and prognosis as well as expert cardiologists and often better than nonspecialists. However, the discriminating power of specific variables from the medical history and exercise test remains unclear because of inadequate study design and differences in study populations. Should expired gases be substituted for estimated metabolic equivalents (METs)? Should ST/heart rate (HR) index be used instead of putting these measurements separately into the models? Should right-sided chest leads and HR in recovery be considered? There is a need for further evaluation of these routinely obtained variables to improve the accuracy of prediction algorithms especially in women. The portability and reliability of these equations must be demonstrated since access to specialised care must be safe-guarded. Hopefully, sequential assessment of the clinical and exercise test data and application of the newer generation of multivariable equations can empower the clinician to assure the cardiac patient access to appropriate and cost-effective cardiological care.

Canadian CT head rule study for patients with minor head injury: methodology for phase II (validation and economic analysis)

Prospective validation on a new set of patients is an essential test of a new decision rule. However, many clinical decision rules are not prospectively assessed to determine their accuracy, reliability, clinical sensibility, or potential impact on practice. This validation process is important because many statistically derived rules or guidelines do not perform well when tested in a new population. The methodologic standards for a validation study are similar to those described in the article on phase I for derivation studies in the August 2001 issue of Annals of Emergency Medicine. The goal of phase II is to prospectively assess the accuracy, reliability, and acceptability of the decision rule in a new set of patients with minor head injury. This will determine the clinical utility of the rule and is essential if such a rule is to be widely adopted into clinical practice.

Evaluation of a computerized diagnostic decision support system for patients with pneumonia: study design considerations

Planning the clinical evaluation of a computerized decision support system requires a strategy that encompasses the different aspects of the clinical problem, the technical difficulties of software and hardware integration and implementation, the behavioral aspects of the targeted users, and the discipline of study design. Although clinical information systems are becoming more widely available, only a few decision support systems have been formally evaluated in clinical environments. Published accounts of difficulties associated with the clinical evaluation of decision support systems remain scarce. The authors report on a variety of behavioral, logistical, technical, clinical, cost, and work flow issues that they had to address when choosing a study design for a clinical trial for the evaluation of an integrated, real-time decision support system for the automatic identification of patients likely to have pneumonia in an emergency department. In the absence of a true gold standard, they show how they created a credible, clinically acceptable, and economical reference standard for the diagnosis of pneumonia, to determine the overall accuracy of the system. For the creation of a reference standard, they describe the importance of recognizing verification bias and avoiding it. Finally, advantages and disadvantages of different study designs are explored with respect to the targeted users and the clinical setting.

Need for validation of clinical decision aids: use of the AST/ALT ratio in predicting cirrhosis in chronic hepatitis C

OBJECTIVE

A value of > or = 1 for the ratio of aspartate amino-transferase to alanine aminotransferase (the AST/ALT ratio or AAR) has been shown to have a positive predictive value of 100% for the diagnosis of cirrhosis in patients with chronic hepatitis C. If validated on separate cohorts, an AAR > or = 1 might obviate the need for liver biopsy in some patients with hepatitis C.

METHODS

We attempted to validate the AAR by abstracting demographic and clinical data from a database of consecutive patients with hepatitis C who had a liver biopsy between 1993 and 1998. We used definitions, methods of data collection, and analyses comparable to those of the published study. A hepatopathologist blindly reviewed 49 liver biopsies for histological grade and stage.

RESULTS

The current cohort of 177 patients and the previous cohort of 139 patients were comparable in mean age (42.3 vs 43.8 yr), percentage of men (63 vs 67), percentage with an AAR > or =1 (20 vs 17), and Child-Pugh distribution, but differed in substantial use of ethanol (11% vs 3.6%; p = 0.01) and in the prevalence of cirrhosis (23% vs 34%, p = 0.06). Respective sensitivities of the AAR were 56% and 53%. An AAR > or =1 had a positive predictive value of 64% (95% confidence interval 48-78%) for the current cohort. Thirteen of 36 patients (36%) with an AAR > or =1 were incorrectly identified as having cirrhosis. Of these 13 patients, 6 had a normal AST and ALT, 5 had a minimally elevated AST or ALT, and 1 had advanced fibrosis without cirrhosis.

CONCLUSIONS

These results suggest that an AAR > or =1 may not be as useful for predicting cirrhosis in chronic hepatitis C as previously thought, and emphasizes the need for validation of clinical decision aids on independent patient cohorts.

Results of report cards for patients with congestive heart failure depend on the method used to adjust for severity

BACKGROUND

The validity of outcome report cards may depend on the ways in which they are adjusted for risk.

OBJECTIVES

To compare the predictive ability of generic and disease-specific survival prediction models appropriate for use in patients with heart failure, to simulate outcome report cards by comparing survival across hospitals and adjusting for severity of illness using these models, and to assess the ways in which the results of these comparisons depend on the adjustment method.

DESIGN

Analysis of data from a prospective cohort study.

SETTING

A university hospital, a Veterans Affairs (VA) medical center, and a community hospital.

PATIENTS

Sequential patients presenting in the emergency department with acute congestive heart failure.

MEASUREMENTS

Unadjusted 30-day and 1-year mortality across hospitals and 30-day and 1-year mortality adjusted by using disease-specific survival prediction models (two sickness-at-admission models, the Cleveland Health Quality Choice model, the Congestive Heart Failure Mortality Time-Independent Predictive Instrument) and generic models (Acute Physiology and Chronic Health Evaluation [APACHE] II, APACHE III, the mortality prediction model, and the Chadson comorbidity index).

RESULTS

The community hospital's unadjusted 30-day survival rate (85.0%) and the VA medical center's unadjusted 1-year survival rate (60.9%) were significantly lower than corresponding rates at the university hospital (92.7% and 67.5%, respectively). No severity model had excellent ability to discriminate patients by survival rates (all areas under the receiver-operating characteristic curve < 0.73). Whether the VA medical center, the community hospital, both, or neither had worse survival rates on simulated report cards than the university hospital depended on the prediction model used for adjustment.

CONCLUSIONS

Results of simulated outcome report cards for survival in patients with congestive heart failure depend on the method used to adjust for severity.

Predictive accuracy study: comparing a statistical model to clinicians' estimates of outcomes after coronary bypass surgery

BACKGROUND

The purpose of this study was to compare clinicians' prior probability estimates of operative mortality (OM) and prolonged intensive care unit stay (ICU) length of stay greater than 48 hours after coronary artery bypass graft surgery (CABG) with estimates derived from statistical models alone.

METHODS

Nine clinicians estimated the predicted probability of OM and ICU stay greater than 48 hours from an abstract of information for each of 100 patients selected from the 1996 to 1997 database of 1,904 patients who underwent isolated CABG. Logistic regression models were used to calculate the predicted probability of OM and ICU stay greater than 48 hours for each patient. The study sample was split into two parts; clinicians were randomly given access to a predictive rule to guide their judgements for one part of the study.

RESULTS

Clinicians' estimates were similar with or without access to the rule, and both parts of the study were therefore pooled. Clinicians significantly overestimated the probability of OM (model 6.3% +/- 1%, clinicians 7.6% +/- 3%, p = 0.0001) and ICU stay greater than 48 hours (model 25% +/- 2%, clinicians 28% +/- 1%, p = 0.0012). Clinicians' estimates of OM were not significantly higher than the model's for nonsurvivors (0.8% +/- 0.7%, p = 0.2), but were significantly higher for survivors (1.4% +/- 0.3%, p = 0.039).

CONCLUSIONS

Clinicians trusted their own empiric estimates rather than a predictive rule and overestimated the probability of OM and ICU stay greater than 48 hours.