Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices

ABSTRACT

The area under the ROC curve is frequently used for assessing the predictive efficacy of a model, and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 - specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature as these are indicators of correct classification of already known conditions. They are not indicators of future events and are not valid for predictions. Predictivity intimately depends on the prevalence which may be ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could differ greatly from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cut-off for prediction, in place of the Youden index, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1. To conclude, for correctly assessing adequacy of a prediction models, predictivity-based ROC curves should be used instead of the usual sensitivity-specificity-based ROC curves and the P-index should replace the Youden index.

Optimising plasma clozapine levels to improve treatment response: an individual patient data meta-analysis and receiver operating characteristic curve analysis

BACKGROUND

Although clozapine is the most efficacious medication for treatment-refractory schizophrenia, not all patients will have an adequate response. Optimising clozapine dose using therapeutic drug monitoring could therefore maximise response.

AIMS

Using individual patient data, we undertook a receiver operating characteristic (ROC) curve analysis to determine an optimal therapeutic range for clozapine levels to guide clinical practice.

METHOD

We conducted a systematic review of PubMed, PsycINFO and Embase for studies that provided individual participant level data on clozapine levels and response. These data were analysed using ROC curves to determine the prediction performance of plasma clozapine levels for treatment response.

RESULTS

We included data on 294 individual participants from nine studies. ROC analysis yielded an area under the curve of 0.612. The clozapine level at the point of optimal diagnostic benefit was 372 ng/mL; at this level, the response sensitivity was 57.3%, and specificity 65.7%. The interquartile range for treatment response was 223-558 ng/mL. There was no improvement in ROC performance with mixed models including patient gender, age or length of trial. Clozapine dose and clozapine concentration to dose ratio did not provide significantly meaningful prediction of response to clozapine.

CONCLUSIONS

Clozapine dose should be optimised based on clozapine therapeutic levels. We found that a range between 250 and 550 ng/mL could be recommended, while noting that a level of >350 ng/mL is the most optimal for response. Although some patients may not respond without clozapine levels >550 ng/mL, the benefits should be weighed against the increased risk of adverse drug reactions.

Impact of a multidimensional infection control strategy on central line-associated bloodstream infection rates in pediatric intensive care units of five developing countries: findings of the International Nosocomial Infection Control Consortium (INICC)

PURPOSE

To analyze the impact of the International Nosocomial Infection Control Consortium (INICC) multidimensional infection control strategy including a practice bundle to reduce the rates of central line-associated bloodstream infection (CLAB) in patients hospitalized in pediatric intensive care units (PICUs) of hospitals, which are members of the INICC, from nine cities of five developing countries: Colombia, India, Mexico, Philippines, and Turkey.

METHODS

CLAB rates were determined by means of a prospective surveillance study conducted on 1,986 patients hospitalized in nine PICUs, over a period of 12,774 bed-days. The study was divided into two phases. During Phase 1 (baseline period), active surveillance was performed without the implementation of the multi-faceted approach. CLAB rates obtained in Phase 1 were compared with CLAB rates obtained in Phase 2 (intervention period), after implementation of the INICC multidimensional infection control program.

RESULTS

During Phase 1, 1,029 central line (CL) days were recorded, and during Phase 2, after implementing the CL care bundle and interventions, we recorded 3,861 CL days. The CLAB rate was 10.7 per 1,000 CL days in Phase 1, and in Phase 2, the CLAB rate decreased to 5.2 per 1,000 CL days (relative risk [RR] 0.48, 95% confidence interval [CI] 0.29-0.94, P = 0.02), showing a reduction of 52% in the CLAB rate.

CONCLUSIONS

This study shows that the implementation of a multidimensional infection control strategy was associated with a significant reduction in the CLAB rates in the PICUs of developing countries.

Socioeconomic impact on device-associated infections in limited-resource neonatal intensive care units: findings of the INICC

PURPOSE

To evaluate the impact of country socioeconomic status and hospital type on device-associated healthcare-associated infections (DA-HAIs) in neonatal intensive care units (NICUs).

METHODS

Data were collected on DA-HAIs from September 2003 to February 2010 on 13,251 patients in 30 NICUs in 15 countries. DA-HAIs were defined using criteria formulated by the Centers for Disease Control and Prevention. Country socioeconomic status was defined using World Bank criteria.

RESULTS

Central-line-associated bloodstream infection (CLA-BSI) rates in NICU patients were significantly lower in private than academic hospitals (10.8 vs. 14.3 CLA-BSI per 1,000 catheter-days; p < 0.03), but not different in public and academic hospitals (14.6 vs. 14.3 CLA-BSI per 1,000 catheter-days; p = 0.86). NICU patient CLA-BSI rates were significantly higher in low-income countries than in lower-middle-income countries or upper-middle-income countries [37.0 vs. 11.9 (p < 0.02) vs. 17.6 (p < 0.05) CLA-BSIs per 1,000 catheter-days, respectively]. Ventilator-associated-pneumonia (VAP) rates in NICU patients were significantly higher in academic hospitals than in private or public hospitals [13.2 vs. 2.4 (p < 0.001) vs. 4.9 (p < 0.001) VAPs per 1,000 ventilator days, respectively]. Lower-middle-income countries had significantly higher VAP rates than low-income countries (11.8 vs. 3.8 per 1,000 ventilator-days; p < 0.001), but VAP rates were not different in low-income countries and upper-middle-income countries (3.8 vs. 6.7 per 1,000 ventilator-days; p = 0.57). When examined by hospital type, overall crude mortality for NICU patients without DA-HAIs was significantly higher in academic and public hospitals than in private hospitals (5.8 vs. 12.5%; p < 0.001). In contrast, NICU patient mortality among those with DA-HAIs was not different regardless of hospital type or country socioeconomic level.

CONCLUSIONS

Hospital type and country socioeconomic level influence DA-HAI rates and overall mortality in developing countries.

Device-associated nosocomial infection rates in intensive care units of seven Indian cities. Findings of the International Nosocomial Infection Control Consortium (INICC)

We sought to determine the rate of healthcare-associated infection (HCAI), microbiological profile, bacterial resistance, length of stay (LOS) and excess mortality in 12 ICUs of the seven hospital members of the International Infection Control Consortium (INICC) of seven Indian cities. Prospective surveillance was introduced from July 2004 to March 2007; 10 835 patients hospitalized for 52 518 days acquired 476 HCAIs, an overall rate of 4.4%, and 9.06 HCAIs per 1000 ICU-days. The central venous catheter-related bloodstream infection (CVC-BSI) rate was 7.92 per 1000 catheter-days;the ventilator-associated pneumonia (VAP) rate was 10.46 per 1000 ventilator-days; and the catheter-associated urinary tract infection (CAUTI) rate was 1.41 per 1000 catheter-days. Overall 87.5% of all Staphylococcus aureus HCAIs were caused by meticillin-resistant strains, 71.4% of Enterobacteriaceae were resistant to ceftriaxone and 26.1% to piperacillin-tazobactam; 28.6% of the Pseudomonas aeruginosa strains were resistant to ciprofloxacin, 64.9% to ceftazidime and 42.0% to imipenem. LOS of patients was 4.4 days for those without HCAI, 9.4 days for those with CVC-BSI, 15.3 days for those with VAP and 12.4 days for those with CAUTI. Excess mortality was 19.0% [relative risk (RR) 3.87; P < or = 0.001] for VAP, 4.0% (RR 1.60; P=0.0174) for CVC-BSI, and 11.6% (RR 2.74; P=0.0102) for CAUTI. Data may not accurately reflect the clinical setting of the country and variations regarding surveillance may have affected HCAI rates. HCAI rates, LOS, mortality and bacterial resistance were high. Infection control programmes including surveillance and antibiotic policies are a priority in India.

The 'Delhi active test' for detection of anterior cruciate ligament tears

All the described tests for the detection of anterior cruciate ligament (ACL) tear are passive tests. For obese or muscular built patients these tests are cumbersome and often unsuitable. We describe for the first time a new method, the 'Delhi active test', by which a torn ACL can be detected irrespective of the size or build of the patient. The results of this test are reproducible and the diagnostic accuracy is comparable with other tests. We suggest that the 'Delhi active test' be used as one of the routine clinical methods to detect the torn ACL.

Haemodynamic responses to laryngoscopy and intubation in patients with pregnancy-induced hypertension: effect of intravenous esmolol with or without lidocaine

The pressor response is known to be exaggerated in patients with pregnancy-induced hypertension, which can result in increased morbidity and mortality in both mother and newborn. Various pharmacological agents have been used before induction in an attempt to attenuate the adrenergic response but with varying degree of success. Esmolol, an ultra short-acting cardioselective beta-blocker with rapid onset and short elimination half-life, is an attractive choice for attenuating the adrenergic response in pregnant patients. In a prospective, randomised double blind study we evaluated the efficacy of two bolus doses of esmolol with or without lidocaine, in patients with pregnancy-induced hypertension. Eighty such patients undergoing lower segmental caesarean section were randomly divided into four groups and received the following study drugs before intubation: group I, esmolol 1 mg.kg(-1); group II, esmolol 2 mg.kg(-1); group III, esmolol 1 mg.kg(-1) and lidocaine 1.5 mg.kg(-1); and group IV, esmolol 2 mg.kg(-1) and lidocaine 1.5 mg.kg(-1). In groups II, III and IV, the changes in maternal heart rate, systolic blood pressure and mean arterial pressure in response to laryngoscopy and intubation were attenuated to a comparable degree (P > 0.05). No adverse effects were noticed in mother or baby. We conclude that esmolol 1 mg.kg(-1) with lidocaine 1.5 mg.kg(-1) is effective in attenuating the adrenergic responses to laryngoscopy and intubation in patients with pregnancy-induced hypertension.

The styloid process in ossicular chain reconstruction. II: Long-term analysis

The human cadaveric styloid process was used as a homograft material for ossicular chain reconstruction in 41 cases of chronic suppurative otitis media over a five-year period. Although technical success was achieved in about 88.23 per cent of the cases, actual patient benefit was around 70.58 per cent; this is comparable to results obtained with most other homologous graft materials. The problem of breakage while remodelling the bone has also increased the risk of transmitting the AIDS virus and consequently undermines the use of this material.