Number Needed to Diagnose, Predict, or Misdiagnose: Useful Metrics for Non-Canonical Signs of Cognitive Status?

Diagnostic tests performance indices: an overview

Diagnostic tests are important means in clinical practice. To assess the performance of a diagnostic test, we commonly need to compare its results to those obtained from a gold standard test. The test sensitivity is the probability of having a positive test in a diseased-patient; the specificity, a negative test result in a disease-free person. However, none of these indices are useful for clinicians who are looking for the inverse probabilities, i.e., the probabilities of the presence and absence of the disease in a person with a positive and negative test result, respectively, the so-called positive and negative predictive values. Likelihood ratios are other performance indices, which are not readily comprehensible to clinicians. There is another index proposed that looks more comprehensible to practicing physicians - the number needed to misdiagnose. It is the number of people who need to be tested in order to find one misdiagnosed (a false positive or a false negative result). For tests with continuous results, it is necessary to set a cut-off point, the choice of which affects the test performance. To arrive at a correct estimation of test performance indices, it is important to use a properly designed study and to consider various aspects that could potentially compromise the validity of the study, including the choice of the gold standard and the population study, among other things. Finally, it may be possible to derive the performance indices of a test solely based on the shape of the distribution of its results in a given group of people.

Reflectance confocal microscopy to diagnose malignant melanoma and lentigo maligna in the UK: a single-centre prospective observational trial

BACKGROUND

Previous work with reflectance confocal microscopy (RCM) has shown high sensitivity and specificity for malignant melanoma (MM); however, to date, there have been no studies with a UK cohort.

OBJECTIVES

To use RCM prospectively to accurately diagnose MM and lentigo maligna (LM) in a private UK secondary care, single-clinician setting; and to assess the potential of RCM to be used as a routine screening procedure.

METHODS

In total, 597 patients with a differential clinical diagnosis of MM or LM were consecutively recruited. A sequential record was made of the clinical, dermoscopy and RCM findings by a single dermatologist prior to biopsy. Imaging was done with an arm-mounted confocal microscope unless there was restricted access to a lesion that required a handheld probe. The likelihood of MM was scored for each diagnostic modality, with each diagnosis building on the last. Histology was assessed by a single blinded histopathologist. The trial was registered with ClinicalTrials.gov (NCT03508297).

RESULTS

Altogether, 733 lesions were included in the analysis, including 86 MM and LM (median diameter 7.0 mm). The benign-to-malignant ratio was 3 : 1 (nonmelanocytic malignancies included) and 8.3 : 1 for MM and LM only. The sensitivity and specificity for MM and LM, respectively, was 62.8% [95% confidence interval (CI) 51.7-73.0] and 63.1% (95% CI 59.3-66.8) for clinical examination; 91.9% (95% CI 84.0-96.7) and 42.0% (95% CI 38.1-45.9), respectively, for dermoscopy; and 94.2% (95% CI 87.0-98.1) and 83.0% (95% CI 79.9-85.8), respectively, for RCM. The positive predictive value of RCM in diagnosing MM and LM was 42.4% (95% CI 38.1-46.8) and the negative predictive value was 99.1% (95% CI 97.9-99.6).

CONCLUSIONS

This study demonstrates that RCM can reliably diagnose MM and is fast enough to be integrated into UK pigmented lesion clinics by dermatologists trained in RCM. The number needed to treat decreased from 3.86 with clinical examination to 2.96 with dermoscopy to 1.30 with RCM.

Evaluating the performance of a risk assessment score tool to predict HIV acquisition among pregnant and postpartum women in Kenya

BACKGROUND

Clinical risk score tools require validation in diverse settings and populations before they are widely implemented. We aimed to externally validate an HIV risk assessment tool for predicting HIV acquisition among pregnant and postpartum women. In the context of prevention of mother-to-child transmission programs, risk score tools could be used to prioritize retesting efforts and delivery of pre-exposure prophylaxis (PrEP) to pregnant and postpartum women most at risk for HIV acquisition while minimizing unnecessary perinatal exposure.

METHODS

Data from women enrolled in a cross-sectional study of programmatic HIV retesting and/or receiving maternal and child health care services at five facilities in Western Kenya were used to validate the predictive ability of a simplified risk score previously developed for pregnant/postpartum women. Incident HIV infections were defined as new HIV diagnoses following confirmed negative or unknown status during pregnancy. Predictive performance was assessed using the area under the receiver operating characteristic curve (AUC) and Brier score.

RESULTS

Among 1266 women with 35 incident HIV infections, we found an AUC for predicting HIV acquisition of 0.60 (95% CI, 0.51, 0.69), with a Brier score of 0.27. A risk score >6 was associated with a 2.9-fold increase in the odds of HIV acquisition (95% CI, 1.48, 5.70; p = 0.002) vs scores ≤6. Women with risk scores >6 were 27% (346/1266) of the population but accounted for 52% of HIV acquisitions. Syphilis, age at sexual debut, and unknown partner HIV status were significantly associated with increased risk of HIV in this cohort.

CONCLUSION

The simplified risk score performed moderately at predicting risk of HIV acquisition in this population of pregnant and postpartum women and may be useful to guide PrEP use or counseling.

Prediction of Osteoporotic Hip Fracture Outcome: Comparative Accuracy of 27 Immune-Inflammatory-Metabolic Markers and Related Conceptual Issues

Objectives: This study, based on the concept of immuno-inflammatory-metabolic (IIM) dysregulation, investigated and compared the prognostic impact of 27 indices at admission for prediction of postoperative myocardial injury (PMI) and/or hospital death in hip fracture (HF) patients. Methods: In consecutive HF patient (n = 1273, mean age 82.9 ± 8.7 years, 73.5% females) demographics, medical history, laboratory parameters, and outcomes were recorded prospectively. Multiple logistic regression and receiver-operating characteristic analyses (the area under the curve, AUC) were used to establish the predictive role for each biomarker. Results: Among 27 IIM biomarkers, 10 indices were significantly associated with development of PMI and 16 were indicative of a fatal outcome; in the subset of patients aged >80 years with ischaemic heart disease (IHD, the highest risk group: 90.2% of all deaths), the corresponding figures were 26 and 20. In the latter group, the five strongest preoperative predictors for PMI were anaemia (AUC 0.7879), monocyte/eosinophil ratio > 13.0 (AUC 0.7814), neutrophil/lymphocyte ratio > 7.5 (AUC 0.7784), eosinophil count < 1.1 × 109/L (AUC 0.7780), and neutrophil/albumin × 10 > 2.4 (AUC 0.7732); additionally, sensitivity was 83.1-75.4% and specificity was 82.1-75.0%. The highest predictors of in-hospital death were platelet/lymphocyte ratio > 280.0 (AUC 0.8390), lymphocyte/monocyte ratio < 1.1 (AUC 0.8375), albumin < 33 g/L (AUC 0.7889), red cell distribution width > 14.5% (AUC 0.7739), and anaemia (AUC 0.7604), sensitivity 88.2% and above, and specificity 85.1-79.3%. Internal validation confirmed the predictive value of the models. Conclusions: Comparison of 27 IIM indices in HF patients identified several simple, widely available, and inexpensive parameters highly predictive for PMI and/or in-hospital death. The applicability of IIM biomarkers to diagnose and predict risks for chronic diseases, including OP/OF, in the preclinical stages is discussed.

Chronic Kidney Disease in Patients with Hip Fracture: Prevalence and Outcomes

Objective. Although the association between chronic kidney disease (CKD) and osteoporotic fractures is well established, data on CKD combined with hip fracture (HF) are scarce and controversial. We aimed to assess in patients with HF the prevalence of CKD, its impact on hospital mortality and length of stay (LOS) and to determine the prognostic value of CKD to predict hospital outcomes. Methods. Prospectively collected clinical data were analysed in 3623 consecutive HF patients aged ≥65 years (mean age 83.4 ± 7.50 [standard deviation] years; 74.4% females). Results. CKD among older patients with HF is highly prevalent (39.9%), has different clinical characteristics, a 2.5-fold higher mortality rate, and 40% greater risk of prolonged LOS. The strongest risk for a poor outcome was advanced age (>80 years). The risk of death substantially increases in combination with chronic disorders, especially coronary artery disease, anaemia, hyperparathyroidism, and atrial fibrillation; models based only on three variables—CKD stage, age >80, and presence of a specific chronic condition—predicted in-hospital death with good discrimination capability (AUC ≥ 0.700) and reasonable accuracy, the number needed to predict ranged between 5.7 and 14.5. Only 12% of HF patients received osteoporotic drugs prefracture. Conclusion. In HF patients with CKD, the risk of adverse outcomes largely increases in parallel with worsening kidney function and, especially, in combination with comorbidities; models based on three admission variables predict a fatal outcome. Assessment of renal function is essential to preventing osteoporotic fractures.

The Price of Explainability in Machine Learning Models for 100-Day Readmission Prediction in Heart Failure: Retrospective, Comparative, Machine Learning Study

BACKGROUND

Sensitive and interpretable machine learning (ML) models can provide valuable assistance to clinicians in managing patients with heart failure (HF) at discharge by identifying individual factors associated with a high risk of readmission. In this cohort study, we delve into the factors driving the potential utility of classification models as decision support tools for predicting readmissions in patients with HF.

OBJECTIVE

The primary objective of this study is to assess the trade-off between using deep learning (DL) and traditional ML models to identify the risk of 100-day readmissions in patients with HF. Additionally, the study aims to provide explanations for the model predictions by highlighting important features both on a global scale across the patient cohort and on a local level for individual patients.

METHODS

The retrospective data for this study were obtained from the Regional Health Care Information Platform in Region Halland, Sweden. The study cohort consisted of patients diagnosed with HF who were over 40 years old and had been hospitalized at least once between 2017 and 2019. Data analysis encompassed the period from January 1, 2017, to December 31, 2019. Two ML models were developed and validated to predict 100-day readmissions, with a focus on the explainability of the model's decisions. These models were built based on decision trees and recurrent neural architecture. Model explainability was obtained using an ML explainer. The predictive performance of these models was compared against 2 risk assessment tools using multiple performance metrics.

RESULTS

The retrospective data set included a total of 15,612 admissions, and within these admissions, readmission occurred in 5597 cases, representing a readmission rate of 35.85%. It is noteworthy that a traditional and explainable model, informed by clinical knowledge, exhibited performance comparable to the DL model and surpassed conventional scoring methods in predicting readmission among patients with HF. The evaluation of predictive model performance was based on commonly used metrics, with an area under the precision-recall curve of 66% for the deep model and 68% for the traditional model on the holdout data set. Importantly, the explanations provided by the traditional model offer actionable insights that have the potential to enhance care planning.

CONCLUSIONS

This study found that a widely used deep prediction model did not outperform an explainable ML model when predicting readmissions among patients with HF. The results suggest that model transparency does not necessarily compromise performance, which could facilitate the clinical adoption of such models.

Derivation and Internal Validation of a Risk Score for Predicting Chlamydia trachomatis Infection in Kenyan Women Planning Conception

BACKGROUND

Availability of laboratory confirmation of sexually transmitted infections is increasing in low- and middle-income countries, but costs continue to limit their access. Chlamydia trachomatis (CT) is a sexually transmitted infection of significant clinical importance, particularly among women. This study aimed to develop a risk score to identify women with a higher likelihood of CT infection, who could then be prioritized for laboratory testing, in a population of Kenyan women planning pregnancies.

METHODS

Women with fertility intentions were included in this cross-sectional analysis. Logistic regression was used to estimate odds ratios for the association between demographic, medical, reproductive, and behavioral characteristics and the prevalence of CT infection. A risk score was developed and validated internally based on the regression coefficients in the final multivariable model.

RESULTS

The prevalence of CT was 7.4% (51 of 691). A risk score for predicting CT infection, with scores 0 to 6, was derived from participants' age, alcohol use, and presence of bacterial vaginosis. The prediction model yielded an area under the receiver operating curve of 0.78 (95% confidene interval [Cl], 0.72-0.84). A cutoff of ≤2 versus >2 identified 31.8% of women as higher risk with moderate sensitivity (70.6%; 95% Cl, 56.2-71.3) and specificity (71.3%; 95% Cl, 67.7-74.5). The bootstrap-corrected area under the receiver operating curve was 0.77 (95% Cl, 0.72-0.83).

CONCLUSIONS

In similar populations of women planning pregnancies, this type of risk score could be useful for prioritizing women for laboratory testing and would capture most women with CT infections while performing more costly testing in less than half of the population.

Clinical value of alarm features for colorectal cancer: a meta-analysis

BACKGROUND

Colorectal cancer (CRC) is a common neoplasm in Western countries. Prioritizing access to colonoscopy appears of critical relevance. Alarm features are considered to increase the likelihood of CRC. Our aim was to assess the diagnostic performance of alarm features for CRC diagnosis.

METHODS

We performed a systematic review and meta-analysis of studies reporting the diagnostic accuracy of alarm features (rectal bleeding, anemia, change in bowel habit, and weight loss) for CRC, published up to September 2021. Colonoscopy was required as the reference diagnostic test. Diagnostic accuracy measures were pooled by a bivariate mixed-effects regression model. The number needed to scope (NNS; i. e. the number of patients who need to undergo colonoscopy to diagnose one CRC) according to each alarm feature was calculated.

RESULTS

31 studies with 45 100 patients (mean age 31-88 years; men 36 %-63 %) were included. The prevalence of CRC ranged from 0.2 % to 22 %. Sensitivity was suboptimal, ranging from 12.4 % for weight loss to 49 % for rectal bleeding, whereas specificity ranged from 69.8 % for rectal bleeding to 91.9 % for weight loss. Taken individually, rectal bleeding and anemia would be the only practical alarm features mandating colonoscopy (NNS 5.3 and 6.7, respectively).

CONCLUSIONS

When considered independently, alarm features have variable accuracy for CRC, given the high heterogeneity of study populations reflected by wide variability in CRC prevalence. Rectal bleeding and anemia are the most practical to select patients for colonoscopy. Integration of alarm features in a comprehensive evaluation of patients should be considered.

Patterns of risk-Using machine learning and structural neuroimaging to identify pedophilic offenders

Background

Child sexual abuse (CSA) has become a focal point for lawmakers, law enforcement, and mental health professionals. With high prevalence rates around the world and far-reaching, often chronic, individual, and societal implications, CSA and its leading risk factor, pedophilia, have been well investigated. This has led to a wide range of clinical tools and actuarial instruments for diagnosis and risk assessment regarding CSA. However, the neurobiological underpinnings of pedosexual behavior, specifically regarding hands-on pedophilic offenders (PO), remain elusive. Such biomarkers for PO individuals could potentially improve the early detection of high-risk PO individuals and enhance efforts to prevent future CSA.

Aim

To use machine learning and MRI data to identify PO individuals.

Methods

From a single-center male cohort of 14 PO individuals and 15 matched healthy control (HC) individuals, we acquired diffusion tensor imaging data (anisotropy, diffusivity, and fiber tracking) in literature-based regions of interest (prefrontal cortex, anterior cingulate cortex, amygdala, and corpus callosum). We trained a linear support vector machine to discriminate between PO and HC individuals using these WM microstructure data. Post hoc, we investigated the PO model decision scores with respect to sociodemographic (age, education, and IQ) and forensic characteristics (psychopathy, sexual deviance, and future risk of sexual violence) in the PO subpopulation. We assessed model specificity in an external cohort of 53 HC individuals.

Results

The classifier discriminated PO from HC individuals with a balanced accuracy of 75.5% (sensitivity = 64.3%, specificity = 86.7%, P ₅₀₀₀ = 0.018) and an out-of-sample specificity to correctly identify HC individuals of 94.3%. The predictive brain pattern contained bilateral fractional anisotropy in the anterior cingulate cortex, diffusivity in the left amygdala, and structural prefrontal cortex-amygdala connectivity in both hemispheres. This brain pattern was associated with the number of previous child victims, the current stance on sexuality, and the professionally assessed risk of future sexual violent reoffending.

Conclusion

Aberrant white matter microstructure in the prefronto-temporo-limbic circuit could be a potential neurobiological correlate for PO individuals at high-risk of reoffending with CSA. Although preliminary and exploratory at this point, our findings highlight the general potential of MRI-based biomarkers and particularly WM microstructure patterns for future CSA risk assessment and preventive efforts.

Assessment of the Efficiency of Non-Invasive Diagnostic Imaging Modalities for Detecting Myocardial Ischemia in Patients Suspected of Having Stable Angina

This study aimed to assess and compare the efficiency of non-invasive imaging modalities in detecting myocardial ischemia in patients with suspected stable angina as easy-to-understand indices. Our study included 1000 patients with chest pain and possible stable myocardial ischemia. The modalities to be assessed were cardiac magnetic resonance imaging (CMRI), single-photon emission computed tomography, positron emission computed tomography (PET), stress echocardiography, and fractional flow reserve derived from coronary computed tomography angiography (FFRCT). As a simulation study, we assumed that all five imaging modalities were performed on these patients, and a decision tree analysis was conducted. From the results, the following efficiencies were assessed and compared: (1) number of true positive (TP), false positive (FP), false negative (FN), and true negative (TN) test results; (2) positive predictive value (PPV); (3) negative predictive value (NPV); (4) post-test probability; (5) diagnostic accuracy (DA); and (6) number needed to diagnose (NND). In the basic settings (pre-test probability: 30%), PET generated the highest TP (267) and NPV (95%, 95% confidence interval (CI): 93-96%). In contrast, CMRI produced the highest TN (616), PPV (76%, 95% CI: 71-80%), and DA (88%, 95% CI: 86-90%) and the lowest NND (1.33, 95% CI: 1.24-1.47). Although FFRCT generated the highest TP (267) and lowest FN (33), it generated the highest FP (168). In terms of detecting myocardial ischemia, compared with the other modalities, PET and CMRI were more efficient. The results of our study might be helpful for both patients and medical professionals associated with their examination.

Comparison of Prognostic Value of 10 Biochemical Indices at Admission for Prediction Postoperative Myocardial Injury and Hospital Mortality in Patients with Osteoporotic Hip Fracture

Aim: To evaluate the prognostic impact at admission of 10 biochemical indices for prediction postoperative myocardial injury (PMI) and/or hospital death in hip fracture (HF) patients. Methods: In 1273 consecutive patients with HF (mean age 82.9 ± 8.7 years, 73.5% women), clinical and laboratory parameters were collected prospectively, and outcomes were recorded. Multiple logistic regression and receiver-operating characteristic analyses (the area under the curve, AUC) were preformed, the number needed to predict (NNP) outcome was calculated. Results: Age ≥ 80 years and IHD were the most prominent clinical factors associated with both PMI (with cardiac troponin I rise) and in-hospital death. PMI occurred in 555 (43.6%) patients and contributed to 80.3% (49/61) of all deaths (mortality rate 8.8% vs. 1.9% in non-PMI patients). The most accurate biochemical predictive markers were parathyroid hormone > 6.8 pmol/L, urea > 7.5 mmol/L, 25(OH)vitamin D < 25 nmol/L, albumin < 33 g/L, and ratios gamma-glutamyl transferase (GGT) to alanine aminotransferase > 2.5, urea/albumin ≥ 2.0 and GGT/albumin ≥ 7.0; the AUC for developing PMI ranged between 0.782 and 0.742 (NNP: 1.84−2.13), the AUC for fatal outcome ranged from 0.803 to 0.722, (NNP: 3.77−9.52). Conclusions: In HF patients, easily accessible biochemical indices at admission substantially improve prediction of hospital outcomes, especially in the aged >80 years with IHD.

An 8-gene machine learning model improves clinical prediction of severe dengue progression

BACKGROUND

Each year 3-6 million people develop life-threatening severe dengue (SD). Clinical warning signs for SD manifest late in the disease course and are nonspecific, leading to missed cases and excess hospital burden. Better SD prognostics are urgently needed.

METHODS

We integrated 11 public datasets profiling the blood transcriptome of 365 dengue patients of all ages and from seven countries, encompassing biological, clinical, and technical heterogeneity. We performed an iterative multi-cohort analysis to identify differentially expressed genes (DEGs) between non-severe patients and SD progressors. Using only these DEGs, we trained an XGBoost machine learning model on public data to predict progression to SD. All model parameters were "locked" prior to validation in an independent, prospectively enrolled cohort of 377 dengue patients in Colombia. We measured expression of the DEGs in whole blood samples collected upon presentation, prior to SD progression. We then compared the accuracy of the locked XGBoost model and clinical warning signs in predicting SD.

RESULTS

We identified eight SD-associated DEGs in the public datasets and built an 8-gene XGBoost model that accurately predicted SD progression in the independent validation cohort with 86.4% (95% CI 68.2-100) sensitivity and 79.7% (95% CI 75.5-83.9) specificity. Given the 5.8% proportion of SD cases in this cohort, the 8-gene model had a positive and negative predictive value (PPV and NPV) of 20.9% (95% CI 16.7-25.6) and 99.0% (95% CI 97.7-100.0), respectively. Compared to clinical warning signs at presentation, which had 77.3% (95% CI 58.3-94.1) sensitivity and 39.7% (95% CI 34.7-44.9) specificity, the 8-gene model led to an 80% reduction in the number needed to predict (NNP) from 25.4 to 5.0. Importantly, the 8-gene model accurately predicted subsequent SD in the first three days post-fever onset and up to three days prior to SD progression.

CONCLUSIONS

The 8-gene XGBoost model, trained on heterogeneous public datasets, accurately predicted progression to SD in a large, independent, prospective cohort, including during the early febrile stage when SD prediction remains clinically difficult. The model has potential to be translated to a point-of-care prognostic assay to reduce dengue morbidity and mortality without overwhelming limited healthcare resources.

Cognitive screening instruments for dementia: comparing metrics of test limitation

Cognitive screening instruments (CSIs) for dementia and mild cognitive impairment are usually characterized in terms of measures of discrimination such as sensitivity, specificity, and likelihood ratios, but these CSIs also have limitations.

Communicating Risk: Developing an "Efficiency Index" for Dementia Screening Tests

Diagnostic and screening tests may have risks such as misdiagnosis, as well as the potential benefits of correct diagnosis. Effective communication of this risk to both clinicians and patients can be problematic. The purpose of this study was to develop a metric called the "efficiency index" (EI), defined as the ratio of test accuracy and inaccuracy, to evaluate screening tests for dementia. This measure was compared with a previously described "likelihood to be diagnosed or misdiagnosed" (LDM), also based on "numbers needed" metrics. Datasets from prospective pragmatic test accuracy studies examining four brief cognitive screening instruments (Mini-Mental State Examination; Montreal Cognitive Assessment; Mini-Addenbrooke's Cognitive Examination (MACE); and Free-Cog) were analysed to calculate values for EI and LDM, and to examine their variation with test cut-off for MACE and dementia prevalence. EI values were also calculated using a modification of McGee's heuristic for the simplification of likelihood ratios to estimate percentage change in diagnostic probability. The findings indicate that EI is easier to calculate than LDM and, unlike LDM, may be classified either qualitatively or quantitatively in a manner similar to likelihood ratios. EI shows the utility or inutility of diagnostic and screening tests, illustrating the inevitable trade-off between diagnosis and misdiagnosis. It may be a useful metric to communicate risk in a way that is easily intelligible for both clinicians and patients.

Building risk prediction models for daily use of marijuana using machine learning techniques

Identifying the characteristics of adults with recent marijuana use is limited by standard statistical methods and requires a unique approach. The objective of this study is to evaluate the efficiency of machine learning models in predicting daily marijuana use and identify factors associated with daily use among adults. The study analyzed pooled data from the 2016-2019 Behavioral Risk Factor Surveillance System (BRFSS) Survey in 2020. Prediction models were developed using four machine learning algorithms, including Logistic Regression, Decision Tree, and Random Forest with Gini function, and Naïve Bayes. Respondents were randomly divided into training and testing samples. The performance of all the models was compared using accuracy, AUC, precision, and recall. The study included 253,569 respondents, of whom 10,182 (5.9 %) reported daily marijuana use in the last 30 days. Of daily marijuana use, 53.4 % were young adults (age 18-34 years), 34.3 % female, 56.1 % non-Hispanic White, 15.2 % were college graduates, and 67.3 % were employed. Random Forest was the best performing model with AUC 0.97, followed by a Decision tree (AUC 0.95). The most important factors for daily marijuana use were the current use of e-cigarette and combustible cigarette use, male gender, unmarried, poor mental health, depression, cognitive decline, abnormal sleep pattern, and high-risk behavior. Data mining methods were useful in the discovery of behavior health-risk knowledge and to visualize the significance of predicting modeling from a multidimensional behavioral health survey.

MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values

The definition of test cut-offs is a critical determinant of many paired and unitary measures of diagnostic or screening test accuracy, such as sensitivity and specificity, positive and negative predictive values, and correct classification accuracy. Revision of test cut-offs from those defined in index studies is frowned upon as a potential source of bias, seemingly accepting any biases present in the index study, for example related to sample bias. Data from a large pragmatic test accuracy study examining the Mini-Addenbrooke's Cognitive Examination (MACE) were interrogated to determine optimal test cut-offs for the diagnosis of dementia and mild cognitive impairment (MCI) using either the maximal Youden index or the maximal correct classification accuracy. Receiver operating characteristic (ROC) and precision recall (PR) curves for dementia and MCI were also plotted, and MACE predictive values across a range of disease prevalences were calculated. Optimal cut-offs were found to be a point lower than those defined in the index study. MACE had good metrics for the area under the ROC curve and for the effect size (Cohen's d) for both dementia and MCI diagnosis, but PR curves suggested the superiority for MCI diagnosis. MACE had high negative predictive value at all prevalences, suggesting that a MACE test score above either cut-off excludes dementia and MCI in any setting.

'Likelihood to be diagnosed or misdiagnosed': application to meta-analytic data for cognitive screening instruments

Aim: To extend use of the recently described 'likelihood to be diagnosed or misdiagnosed' (LDM) metric for test accuracy studies through application to recent meta-analytic data of commonly used cognitive screening instruments. Methods: Raw data (true positives and negatives, false positives and negatives) were extracted from meta-analyses (minimum 5 studies or 1000 patients), from which LDM was calculated. LDM values were compared with those previously reported for single test accuracy studies. Results: LDM values for diagnosis of dementia ranged from around two to seven, and for diagnosis of mild cognitive impairment from two to three. LDM values based on meta-analytic data were larger than those reported for individual studies. Conclusion: LDM is an easily calculated and potentially useful unitary, global metric for test accuracy studies.

Evaluating cognitive screening instruments with the "likelihood to be diagnosed or misdiagnosed" measure

OBJECTIVES

To calculate "number needed to diagnose" (NND), "number needed to predict" (NNP), and "number needed to misdiagnose" (NNM) for cognitive screening instruments which are commonly used in suspected dementia and mild cognitive impairment, and from these to calculate a "likelihood to be diagnosed or misdiagnosed" (LDM) metric as the ratio of NNM to either NND or NNP.

METHODS

Datasets from pragmatic diagnostic test accuracy studies examining four commonly used cognitive screening instruments (Mini-Mental State Examination, MMSE; Montreal Cognitive Assessment, MoCA; Mini-Addenbrooke's Cognitive Examination, MACE; Six-item Cognitive Impairment Test, 6CIT) were analysed to calculate NND, NNP, and NNM, and from these derive values for LDM.

FINDINGS

Although all the tests had low NND and NNP as desired, NNM was also low. Hence, only MMSE and 6CIT achieved LDM > 1 for dementia diagnosis, and only MACE and 6CIT had LDM > 1 for diagnosis of mild cognitive impairment.

CONCLUSIONS

The likelihood to be diagnosed or misdiagnosed (LDM) metric may indicate the utility or inutility of diagnostic tests for clinicians and patients. LDM values may clarify the inevitable trade-off between sensitivity and specificity and hence clinician purpose in administering the diagnostic test (minimising false negatives or false positives).