Evaluating cognitive screening instruments with the "likelihood to be diagnosed or misdiagnosed" measure

Identification of amnestic mild cognitive impairment among Black and White community-dwelling older adults using NIH Toolbox Cognition tablet battery

OBJECTIVES

Identify which NIH Toolbox Cognition Battery (NIHTB-CB) subtest(s) best differentiate healthy controls (HC) from those with amnestic mild cognitive impairment (aMCI) and compare the discriminant accuracy between a model using a priori "Norm Adjusted" scores versus "Unadjusted" standard scores with age, sex, race/ethnicity, and education controlled for within the model. Racial differences were also examined.

METHODS

Participants were Black/African American (B/AA) and White consensus-confirmed (HC = 96; aMCI = 62) adults 60-85 years old that completed the NIHTB-CB for tablet. Discriminant function analysis (DFA) was used in the Total Sample and separately for B/AA (n = 80) and White participants (n = 78).

RESULTS

Picture Sequence Memory (an episodic memory task) was the highest loading coefficient across all DFA models. When stratified by race, differences were noted in the pattern of the highest loading coefficients within the DFAs. However, the overall discriminant accuracy of the DFA models in identifying HCs and those with aMCI did not differ significantly by race (B/AA, White) or model/score type (Norm Adjusted versus Unadjusted).

CONCLUSIONS

Racial differences were noted despite the use of normalized scores or demographic covariates-highlighting the importance of including underrepresented groups in research. While the models were fairly accurate at identifying consensus-confirmed HCs, the models proved less accurate at identifying White participants with an aMCI diagnosis. In clinical settings, further work is needed to optimize computerized batteries and the use of NIHTB-CB norm adjusted scores is recommended. In research settings, demographically corrected scores or within model correction is suggested.

Cognitive and behavioral abnormalities in individuals with Alzheimer’s disease, mild cognitive impairment, and subjective memory complaints

In this study, we investigated the ability of commonly used neuropsychological tests to detect cognitive and functional decline across the Alzheimer’s disease (AD) continuum. Moreover, as preclinical AD is a key area of investigation, we focused on the ability of neuropsychological tests to distinguish the early stages of the disease, such as individuals with Subjective Memory Complaints (SMC). This study included 595 participants from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset who were cognitively normal (CN), SMC, mild cognitive impairment (MCI; early or late stage), or AD. Our cognitive measures included the Rey Auditory Verbal Learning Test (RAVLT), the Everyday Cognition Questionnaire (ECog), the Functional Abilities Questionnaire (FAQ), the Alzheimer’s Disease Assessment Scale–Cognitive Subscale (ADAS-Cog), the Montreal Cognitive Assessment scale (MoCA), and the Trail Making test (TMT-B). Overall, our results indicated that the ADAS-13, RAVLT (learning), FAQ, ECog, and MoCA were all predictive of the AD progression continuum. However, TMT-B and the RAVLT (immediate and forgetting) were not significant predictors of the AD continuum. Indeed, contrary to our expectations ECog self-report (partner and patient) were the two strongest predictors in the model to detect the progression from CN to AD. Accordingly, we suggest using the ECog (both versions), RAVLT (learning), ADAS-13, and the MoCA to screen all stages of the AD continuum. In conclusion, we infer that these tests could help clinicians effectively detect the early stages of the disease (e.g., SMC) and distinguish the different stages of AD.

Functional magnetic resonance imaging, deep learning, and Alzheimer's disease: A systematic review

Alzheimer's disease (AD) is currently diagnosed using a mixture of psychological tests and clinical observations. However, these diagnoses are not perfect, and additional diagnostic tools (e.g., MRI) can help improve our understanding of AD as well as our ability to detect the disease. Accordingly, a large amount of research has been invested into innovative diagnostic methods for AD. Functional MRI (fMRI) is a form of neuroimaging technology that has been used to diagnose AD; however, fMRI is incredibly noisy, complex, and thus lacks clinical use. Nonetheless, recent innovations in deep learning technology could enable the simplified and streamlined analysis of fMRI. Deep learning is a form of artificial intelligence that uses computer algorithms based on human neural networks to solve complex problems. For example, in fMRI research, deep learning models can automatically denoise images and classify AD by detecting patterns in participants' brain scans. In this systematic review, we investigate how fMRI (specifically resting-state fMRI) and deep learning methods are used to diagnose AD. In turn, we outline the common deep neural network, preprocessing, and classification methods used in the literature. We also discuss the accuracy, strengths, limitations, and future direction of fMRI deep learning methods. In turn, we aim to summarize the current field for new researchers, suggest specific areas for future research, and highlight the potential of fMRI to aid AD diagnoses.

Applying ACE-III, M-ACE and MMSE to Diagnostic Screening Assessment of Cognitive Functions within the Polish Population

The research aims to compare the accuracy of the mini-mental state examination (MMSE), the Addenbrooke's cognitive examination III (ACE-III) and the mini-Addenbrooke's cognitive examination (M-ACE) within the Polish population. The model comprised several stages: the features of each test were compared; the shifts in result categorisations between the norm and below the norm were analysed; a third category-mild cognitive impairment (MCI)-was included. Additionally, particular ACE-III domains that scored below domain-specific norm thresholds were analysed to establish the potential early predictors of dementia. All tests correlated to a high and very high degree-cf. MMSE and ACE-III (r = 0.817; p < 0.001), MMSE and M-ACE (r = 0.753; p < 0.001), ACE-III and M-ACE (r = 0.942; p < 0.001). The area under the ROC curve for the ACE-III diagnostic variable had a high value (AUC = 0.920 ± 0.014). A cut-off point of 81 points was suggested for ACE-III; the M-ACE diagnostic variable had an equally high value (AUC = 0.891 ± 0.017). A cut-off point of 20 points was suggested. A significant decrease in the mean score values for people who scored norm or below the norm under ACE-III, as compared to the MMSE results for norm (p < 0.0001), occurred for speech fluency (which decreased by 26.4%). The tests in question are characterised by high sensitivity and specificity. Targeted ACE-III seems best recommended for use in specialised diagnostic centres, whereas M-ACE appears to be a better suited diagnostic alternative for primary health care centres in comparison to MMSE.

Cognitive screening instruments for dementia: comparing metrics of test limitation

Cognitive screening instruments (CSIs) for dementia and mild cognitive impairment are usually characterized in terms of measures of discrimination such as sensitivity, specificity, and likelihood ratios, but these CSIs also have limitations.

Communicating Risk: Developing an "Efficiency Index" for Dementia Screening Tests

Diagnostic and screening tests may have risks such as misdiagnosis, as well as the potential benefits of correct diagnosis. Effective communication of this risk to both clinicians and patients can be problematic. The purpose of this study was to develop a metric called the "efficiency index" (EI), defined as the ratio of test accuracy and inaccuracy, to evaluate screening tests for dementia. This measure was compared with a previously described "likelihood to be diagnosed or misdiagnosed" (LDM), also based on "numbers needed" metrics. Datasets from prospective pragmatic test accuracy studies examining four brief cognitive screening instruments (Mini-Mental State Examination; Montreal Cognitive Assessment; Mini-Addenbrooke's Cognitive Examination (MACE); and Free-Cog) were analysed to calculate values for EI and LDM, and to examine their variation with test cut-off for MACE and dementia prevalence. EI values were also calculated using a modification of McGee's heuristic for the simplification of likelihood ratios to estimate percentage change in diagnostic probability. The findings indicate that EI is easier to calculate than LDM and, unlike LDM, may be classified either qualitatively or quantitatively in a manner similar to likelihood ratios. EI shows the utility or inutility of diagnostic and screening tests, illustrating the inevitable trade-off between diagnosis and misdiagnosis. It may be a useful metric to communicate risk in a way that is easily intelligible for both clinicians and patients.

Harnessing forgetfulness: can episodic-memory tests predict early Alzheimer's disease?

A rapid increase in the number of patients with Alzheimer's disease (AD) is expected over the next decades. Accordingly, there is a critical need for early-stage AD detection methods that can enable effective treatment strategies. In this study, we consider the ability of episodic-memory measures to predict mild cognitive impairment (MCI) to AD conversion and thus, detect early-stage AD. For our analysis, we studied 307 participants with MCI across four years using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). Using a binary logistic regression, we compared episodic-memory tests to each other and to prominent neuroimaging methods in MCI converter (MCI participants who developed AD) and MCI non-converter groups (MCI participants who did not develop AD). We also combined variables to test the accuracy of mixed-predictor models. Our results indicated that the best predictors of MCI to AD conversion were the following: a combined episodic-memory and neuroimaging model in year one (59.8%), the Rey Auditory Verbal Learning Test in year two (71.7%), a mixed episodic-memory predictor model in year three (77.7%) and the Logical Memory Test in year four (77.2%) of ADNI. Overall, we found that individual episodic-memory measure and mixed models performed similarly when predicting MCI to AD conversion. Comparatively, individual neuroimaging measures predicted MCI conversion worse than chance. Accordingly, our results indicate that episodic-memory tests could be instrumental in detecting early-stage AD and enabling effective treatment.

Defining 'optimal' test cut-off using global test metrics: evidence from a cognitive screening instrument

Aim: To examine the variation of several global metrics of test accuracy with test cut-off for the diagnosis of dementia. These metrics included some based on the receiver operating characteristic curve, such as Youden index, and some independent of receiver operating characteristic curve, such as correct classification accuracy. Materials & methods: Data from a test accuracy study of Mini-Addenbrooke's Cognitive Examination were used to calculate and plot each global measure against test cut-off. Results: Different 'optimal' cut-points were identified for the different global measures, with a spread of ten points in observed optimal cut-off in the 30-point Mini-Addenbrooke's Cognitive Examination scale. Using these optima gave a large variation in test sensitivity from very high (diagnostic odds ratio) to very low (likelihood to be diagnosed or misdiagnosed), but all had high negative predictive value. Conclusion: The method used to determine the cut-off of cognitive screening instruments may have significant implications for test performance.

Predictors for Favorable Cognitive Outcome Post-Stroke: A-Seven-Year Follow-Up Study

BACKGROUND AND PURPOSE

Knowledge of the burden and development of post-stroke cognitive impairments (CIs) in the long-term after the first event is limited. We aimed to assess the prevalence of mild CI (MCI) and dementia 7 years after first-ever stroke or transient ischemic attack (TIA), to subclassify the impairments, and to identify predictors for a favorable cognitive outcome.

MATERIALS AND METHODS

During 2007 and 2008, 208 patients with first-ever stroke or TIA without preexisting CI were included. After 1 and 7 years, survivors were invited to a follow-up. Transitions of cognitive status from 1 to 7 years were recorded based on the 3 categories dementia, MCI, or none. Etiologic subclassification was based on clinical cognitive profile, magnetic resonance imaging (MRI) findings, and biomarkers at both time points. Favorable outcome was defined as normal cognitive function or MCI after 7 years with exclusion of those who had progression from normal to MCI.

RESULTS

Eighty patients died during follow-up, 12 patients refused further participation. After 7 years, 109 completed follow-up of whom 40 (37%) were diagnosed with MCI and 24 (22%) with dementia. Of the 64 patients diagnosed with CI, 9 were subclassified with degenerative cognitive disease, 13 with vascular disease, and 42 had mixed cognitive disease. In all, 65 patients (60%) had a favorable outcome. In multivariable logistic regression analysis, lower age and lower medial temporal lobe atrophy (MTLA) grade on MRI at 12 months were independently associated with a favorable outcome, adjusted OR (95% CI), 0.94 (0.86-0.92), and 0.55 (0.35-0.85), respectively.

CONCLUSIONS

Sixty percent of stroke survivors have a favorable cognitive outcome. Lower age and lower MTLA grade on MRI were associated with favorable outcome.

Free-Cog: Pragmatic Test Accuracy Study and Comparison with Mini-Addenbrooke's Cognitive Examination

BACKGROUND/AIMS

Canonical definitions of the dementia construct encompass deficits in both cognition and function, but most screening instruments for possible dementia address only cognitive abilities. Free-Cog is a recently described brief screening instrument for dementia designed to address not only cognitive but also functional abilities.

METHODS

A pragmatic test accuracy study of Free-Cog was undertaken in consecutive patients seen over 1 year in a secondary care setting. The performance of Free-Cog for diagnosis of dementia and mild cognitive impairment (MCI) was compared to that of Mini-Addenbrooke's Cognitive Examination (MACE).

RESULTS

In a cohort of 141 patients (prevalence of dementia and MCI 11 and 32%, respectively) both Free-Cog and MACE were quick and easy to use and acceptable to patients. Both tests had high sensitivity (1.00) and large effect sizes (Cohen's d) for diagnosis of dementia, but Free-Cog was more specific. For diagnosis of MCI, Free-Cog lacked sensitivity (0.58) but was specific (0.81), whereas MACE was sensitive (0.91) but not specific (0.35). Weighted comparison suggested equivalence for dementia diagnosis but a net benefit for MACE regarding MCI diagnosis.

CONCLUSION

Free-Cog is an acceptable and accurate test for dementia screening in a dedicated cognitive disorders clinic, but it appears less sensitive than MACE for the identification of MCI.

Codex (Cognitive Disorders Examination) Decision Tree Modified for the Detection of Dementia and MCI

Many cognitive screening instruments are available to assess patients with cognitive symptoms in whom a diagnosis of dementia or mild cognitive impairment is being considered. Most are quantitative scales with specified cut-off values. In contrast, the cognitive disorders examination or Codex is a two-step decision tree which incorporates components from the Mini-Mental State Examination (MMSE) (three word recall, spatial orientation) along with a simplified clock drawing test to produce categorical outcomes defining the probability of dementia diagnosis and, by implication, directing clinician response (reassurance, monitoring, further investigation, immediate treatment). Codex has been shown to have high sensitivity and specificity for dementia diagnosis but is less sensitive for the diagnosis of mild cognitive impairment (MCI). We examined minor modifications to the Codex decision tree to try to improve its sensitivity for the diagnosis of MCI, based on data extracted from studies of two other cognitive screening instruments, the Montreal Cognitive Assessment and Free-Cog, which are more stringent than MMSE in their tests of delayed recall. Neither modification proved of diagnostic value for mild cognitive impairment. Possible explanations for this failure are considered.

MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values

The definition of test cut-offs is a critical determinant of many paired and unitary measures of diagnostic or screening test accuracy, such as sensitivity and specificity, positive and negative predictive values, and correct classification accuracy. Revision of test cut-offs from those defined in index studies is frowned upon as a potential source of bias, seemingly accepting any biases present in the index study, for example related to sample bias. Data from a large pragmatic test accuracy study examining the Mini-Addenbrooke's Cognitive Examination (MACE) were interrogated to determine optimal test cut-offs for the diagnosis of dementia and mild cognitive impairment (MCI) using either the maximal Youden index or the maximal correct classification accuracy. Receiver operating characteristic (ROC) and precision recall (PR) curves for dementia and MCI were also plotted, and MACE predictive values across a range of disease prevalences were calculated. Optimal cut-offs were found to be a point lower than those defined in the index study. MACE had good metrics for the area under the ROC curve and for the effect size (Cohen's d) for both dementia and MCI diagnosis, but PR curves suggested the superiority for MCI diagnosis. MACE had high negative predictive value at all prevalences, suggesting that a MACE test score above either cut-off excludes dementia and MCI in any setting.

'Likelihood to be diagnosed or misdiagnosed': application to meta-analytic data for cognitive screening instruments

Aim: To extend use of the recently described 'likelihood to be diagnosed or misdiagnosed' (LDM) metric for test accuracy studies through application to recent meta-analytic data of commonly used cognitive screening instruments. Methods: Raw data (true positives and negatives, false positives and negatives) were extracted from meta-analyses (minimum 5 studies or 1000 patients), from which LDM was calculated. LDM values were compared with those previously reported for single test accuracy studies. Results: LDM values for diagnosis of dementia ranged from around two to seven, and for diagnosis of mild cognitive impairment from two to three. LDM values based on meta-analytic data were larger than those reported for individual studies. Conclusion: LDM is an easily calculated and potentially useful unitary, global metric for test accuracy studies.