What (not) to expect when classifying rare events

Suicide Death Prediction Using the Maryland Suicide Data Warehouse: A Sensitivity Analysis

OBJECTIVE

Nearly 50,000 Americans die each year from suicide, despite suicide death being a rare event in the context of health risk assessment and modeling. Prior research has underscored the need for contextualizing suicide risk models in terms of their potential uses and generalizability. This sensitivity analysis makes use of the Maryland Suicide Data Warehouse (MSDW) and illustrates how results inform clinical decision support.

METHOD

A cohort of 1 million living control patients were extracted from the MSDW in addition to 1,667 patients who had died by suicide between the years 2016 and 2019 according to the Maryland Office of the Medical Examiner (OCME). Data were extracted and aggregated as part of a 4-year retrospective design. Binary logistic and two penalized regression models were deployed in a repeated fivefold cross-validation. Model performances were evaluated using sensitivity, positive predictive value (PPV), and F1, and model coefficients were ranked according to coefficient size.

RESULTS

Several features were significantly associated with patients having died by suicide, including male sex, depressive and anxiety disorder diagnoses, social needs, and prior suicidal ideation and suicide attempt. Cross-validated binary logistic regression outperformed either ridge or LASSO (least absolute shrinkage and selection operator) models but generally achieved low-to-moderate PPV and sensitivity across most thresholds and a peak F1 of 0.323.

CONCLUSIONS

Suicide death prediction is constrained by the context of use, which determines the best balance of precision and recall. Predictive models must be evaluated close to the level of intervention. They may not hold up to different needs at different levels of care.

Magnetic Resonance Imaging in Osteochondritis Dissecans of the Humeral Capitellum: Preoperative Assessment of Lesion Size and Lateral Wall Integrity

PURPOSE

Magnetic resonance imaging (MRI) is a helpful tool in the evaluation of osteochondritis dissecans (OCD) of the humeral capitellum. The relationship between MRI appearance and clinically relevant intraoperative findings is incompletely understood. The goal of this study was to evaluate capitellar OCD lesions via preoperative MRI and to determine its accuracy in predicting lesion characteristics including lesion size and capitellar lateral wall integrity.

METHODS

Patients surgically treated for capitellar OCD between January 2010 and June 2018 were reviewed. Preoperative MRI images were assessed by a musculoskeletal radiologist with documentation of lesion size, location, violation of the lateral wall, and stage, in accordance with previously established criteria. These data were compared with intraoperative findings. Involvement of the lateral wall of the capitellum was defined using 2 methods: (1) subchondral edema or articular cartilage disruption in the lateral third of the capitellum (lateral third method) or (2) disruption of the proximal-distal subchondral line along the lateral border of the capitellum descending vertically before the subchondral bone turns horizontally to form the radiocapitellar articulation (capitellar lateral wall sign method). Diagnostic test characteristics were compared for both methods.

RESULTS

We reviewed the preoperative MRIs of 33 elbows with capitellar OCD and found no significant difference in mean lesion size between preoperative MRI (1.12 cm²) and intraoperative measurement (1.08 cm²). For detection of lateral wall integrity, preoperative MRI evaluation using the capitellar lateral wall sign method had an accuracy of 93%, sensitivity of 80%, specificity of 96%, positive predictive value of 80%, and negative predictive value of 96%.

CONCLUSIONS

Preoperative MRI is a valid predictor of lesion size. The capitellar lateral wall sign method is highly accurate in the identification of lateral wall involvement, which may influence preoperative planning.

TYPE OF STUDY/LEVEL OF EVIDENCE

Diagnostic II.

Risk for malnutrition in family practice non-attenders living in the community: A cross-sectional study from Slovenia

OBJECTIVES

The aim of this study was to determine the prevalence of malnutrition and the factors associated with malnutrition in a population of family practice non-attenders. This would enable us to recognize the extent of the problem and anticipate the measures needed to improve the current situation.

METHODS

This was a cross-sectional observational study in a family medicine setting. Participants were adults living in the community (≥18 y of age) who had not visited their chosen family physician in the previous 5 y (non-attenders) and who were able to participate in the study. Through the electronic system, we identified 2025 non-attenders. Data were collected by community nurses in the participants' homes. The main outcome measure was the Malnutrition Universal Screening Tool.

RESULTS

The sample consisted of 1641 participants (81% response rate). Increased risk for malnutrition was determined in 216 participants (13.2%; 95% confidence interval, 11.6-14.9). The multivariate model showed a significant association of increased risk for malnutrition with age and body mass index interaction (P = 0.022); number of chronic diseases (P = 0.001); poor assessment of current health (P = 0.001); a feelings of loneliness (P <0.001); and increased pain intensity (P = 0.003).

CONCLUSION

A screening program at the primary health care level could help identify patients at risk for malnutrition early. This could be followed by appropriate nutrition support, which may help to reverse or halt the malnutrition trajectory and the negative outcomes associated with poor nutritional status.

Comparison of treatment strategies and thresholds for optimizing fracture prevention in Canada: a simulation analysis

This comparison of osteoporosis treatment strategies and intervention thresholds highlights tradeoffs in terms of number of individuals qualifying for treatment and estimated fractures prevented.

PURPOSE

The current analysis was performed to inform the following key question as part of the Osteoporosis Canada's Osteoporosis Guidelines Update: "What is the best strategy to identify those at high fracture risk for pharmacotherapy in order to prevent the most fractures, considering both population and patient perspectives?"

METHODS

The study population consisted of 66,878 women age 50 years and older (mean age 66.0 ± 9.7 years) with documented fracture probability assessment (FRAX) and fracture outcomes. Fractures over the next 5 years were identified through linked administrative healthcare data. We estimated the fraction of the population that would warrant treatment and the number of fractures avoided per 1000 person-years according to multiple strategies and thresholds. Strategies were then rank ordered using 19 metrics.

RESULTS

During mean 4.4 years, 863 (3.5%) sustained one or more major osteoporotic fractures (MOF), 212 (0.8%) sustained a hip fracture, and 1210 (4.9%) sustained any incident fracture. For woman age 50-64 years, the highest ranked strategy was treatment based upon total hip T score ≤ -2.5, but several other strategies fell within 0.5 overall ranking. For women age 65 years and older, MOF > 20% was the highest ranked strategy with no closely ranked strategies. Pooling both age subgroups gave MOF > 20% as the highest ranked strategy, with several other strategies within 0.5 overall ranking.

CONCLUSIONS

Choice of treatment strategy and threshold for osteoporosis management strongly influences the number of individuals for whom pharmacologic treatment would be recommended and on estimated fracture rates in the population. This evidence-based approach to comparing these strategies will help to inform guidelines development in Canada and may be on interest elsewhere.

Predictive performances of STESS and EMSE in a Norwegian adult status epilepticus cohort

PURPOSE

"Status Epilepticus Severity Score" (STESS) and "Epidemiology-based Mortality Score in Status Epilepticus" (EMSE) are two clinical scoring systems aiming to predict mortality in status epilepticus (SE). The objective of this study was to compare their predictive performances in a cohort of 151 SE-patients from Oslo University Hospital in the period 2001-2017.

METHOD

Variables used to calculate STESS (age, previous seizures, worst SE-semiology, level of consciousness) and two different versions of EMSE, EMSE-EAC (etiology, age, comorbidities) and EMSE-EACE (etiology, age, comorbidities, EEG-pattern), as well as outcome were collected retrospectively. Receiver Operating Characteristic (ROC)-analyses, determination of best cut-off values, sensitivity (Se), specificity (Sp), positive predictive value (PPV) and negative predictive value (NPV) were performed. In addition, Precision-Recall curves (PRC) were produced, plotting PPV as a function of Se.

RESULTS

Thirteen patients (9%) died during their hospital stay. STESS did not accurately predict mortality, with a ROC-curve showing an area under the curve (AUC) of 0.625(95%CI = 0.472-0.783), p = 0.15. EMSE-EAC performed better with an AUC of 0.714(95%CI = 0.552-0.873), p = 0.01 and a best cut-off value of 37. Se was 69.2%, Sp 72.1%, PPV 19% and NPV 96.2%. EMSE-EACE performed best with an AUC of 0.855(95%CI = 0.736-0.976), p < 0.0005 and a best cut-off value of 79. Se was 77.8%, Sp 87.8%, PPV 36.8% and NPV 97.7%. The PRC showed areas under the PRC of 0.23 for EMSE-EAC and 0.46 for EMSE-EACE.

CONCLUSIONS

EMSE-EAC and EMSE-EACE performed better than STESS and may be useful in identifying the patients at risk of death in SE. PRC may give a more relevant visual representation of predictive utility than ROC-curves in situations of imbalanced datasets.