An Artificial Intelligence-Assisted Method for Dementia Detection Using Images from the Clock Drawing Test

Comparison of multiple linear regression and machine learning methods in predicting cognitive function in older Chinese type 2 diabetes patients

INTRODUCTION

The prevalence of type 2 diabetes (T2D) has increased dramatically in recent decades, and there are increasing indications that dementia is related to T2D. Previous attempts to analyze such relationships principally relied on traditional multiple linear regression (MLR). However, recently developed machine learning methods (Mach-L) outperform MLR in capturing non-linear relationships. The present study applied four different Mach-L methods to analyze the relationships between risk factors and cognitive function in older T2D patients, seeking to compare the accuracy between MLR and Mach-L in predicting cognitive function and to rank the importance of risks factors for impaired cognitive function in T2D.

METHODS

We recruited older T2D between 60-95 years old without other major comorbidities. Demographic factors and biochemistry data were used as independent variables and cognitive function assessment (CFA) was conducted using the Montreal Cognitive Assessment as an independent variable. In addition to traditional MLR, we applied random forest (RF), stochastic gradient boosting (SGB), Naïve Byer's classifier (NB) and eXtreme gradient boosting (XGBoost).

RESULTS

Totally, the test cohort consisted of 197 T2D (98 men and 99 women). Results showed that all ML methods outperformed MLR, with symmetric mean absolute percentage errors for MLR, RF, SGB, NB and XGBoost respectively of 0.61, 0.599, 0.606, 0.599 and 0.2139. Education level, age, frailty score, fasting plasma glucose and body mass index were identified as key factors in descending order of importance.

CONCLUSION

In conclusion, our study demonstrated that RF, SGB, NB and XGBoost are more accurate than MLR for predicting CFA score, and identify education level, age, frailty score, fasting plasma glucose, body fat and body mass index as important risk factors in an older Chinese T2D cohort.

[Discriminant validity and inter-rater concordance of two scoring systems for the clock test]

OBJECTIVE

To compare the discriminant validity and inter-rater reliability of the two scoring systems for the Clock test that are most used in Spain.

METHODOLOGY

Two collections of clock drawings obtained in a clinical context (116 cases; 56.8% women, mean age 73.1±7.7 years) and in a cohort of volunteers (2039 drawings of 579 subjects; 59.5% women, mean age 78.3±3.8 years) have been assessed. All subjects were classified as cognitively normal (CN) or cognitively impaired (CI) after extensive clinical and neuropsychological evaluation. Expert raters have evaluated these drawings independently and without knowledge of the diagnosis using the Sunderland and Solomon systems standardized in Spanish by Cacho (range 0 to 10) and del Ser (range 0 to 7) respectively. The discriminant validity of each method was calculated in the two samples using the area under the ROC curve (aROC), and the inter-rater reliability was calculated in the clinical sample, that was assessed by the two evaluators, using the intraclass correlation coefficient (ICC) and the kappa coefficient.

RESULTS

There are no significant differences in the discriminant validity of the Sunderland and Solomon systems in any of the samples (clinical: aROC 0.73 [CI95%: 0.64-0.81] and 0.77 [CI95%: 0.69-0.85] respectively, P=.19; volunteers: aROC 0.69 [CI95%: 0.67-0.71] and 0.72 [CI95%: 0.69-0.73] respectively, P=.08). The cut-off points ≤8 and ≤5 correctly classify 71% and 73% of the clinical sample and 82% and 84% of the volunteer sample, respectively. Both systems have good agreement in the clinical sample (Sunderland: ICC 0.90 [CI95%: 0.81-0.93], kappa 0.76 [CI95%: 0.70-0.83]; Solomon: 0.92 [CI95%: 0.88-0.95] and 0.77 [CI95%: 0.71-0.83] respectively), somewhat higher in the second, although the differences are not significant.

CONCLUSIONS

The discriminant validity and inter-observer reliability of these two Clock Test correction systems are similar. Solomon's method, shorter and simpler, may be more advisable in pragmatic terms.

Automating Rey Complex Figure Test scoring using a deep learning-based approach: a potential large-scale screening tool for cognitive decline

BACKGROUND

The Rey Complex Figure Test (RCFT) has been widely used to evaluate the neurocognitive functions in various clinical groups with a broad range of ages. However, despite its usefulness, the scoring method is as complex as the figure. Such a complicated scoring system can lead to the risk of reducing the extent of agreement among raters. Although several attempts have been made to use RCFT in clinical settings in a digitalized format, little attention has been given to develop direct automatic scoring that is comparable to experienced psychologists. Therefore, we aimed to develop an artificial intelligence (AI) scoring system for RCFT using a deep learning (DL) algorithm and confirmed its validity.

METHODS

A total of 6680 subjects were enrolled in the Gwangju Alzheimer's and Related Dementia cohort registry, Korea, from January 2015 to June 2021. We obtained 20,040 scanned images using three images per subject (copy, immediate recall, and delayed recall) and scores rated by 32 experienced psychologists. We trained the automated scoring system using the DenseNet architecture. To increase the model performance, we improved the quality of training data by re-examining some images with poor results (mean absolute error (MAE) [Formula: see text] 5 [points]) and re-trained our model. Finally, we conducted an external validation with 150 images scored by five experienced psychologists.

RESULTS

For fivefold cross-validation, our first model obtained MAE = 1.24 [points] and R-squared ([Formula: see text]) = 0.977. However, after evaluating and updating the model, the performance of the final model was improved (MAE = 0.95 [points], [Formula: see text] = 0.986). Predicted scores among cognitively normal, mild cognitive impairment, and dementia were significantly different. For the 150 independent test sets, the MAE and [Formula: see text] between AI and average scores by five human experts were 0.64 [points] and 0.994, respectively.

CONCLUSION

We concluded that there was no fundamental difference between the rating scores of experienced psychologists and those of our AI scoring system. We expect that our AI psychologist will be able to contribute to screen the early stages of Alzheimer's disease pathology in medical checkup centers or large-scale community-based research institutes in a faster and cost-effective way.

Digital Cognitive Biomarker for Mild Cognitive Impairments and Dementia: A Systematic Review

The dementia population is increasing as the world’s population is growing older. The current systematic review aims to identify digital cognitive biomarkers from computerized tests for detecting dementia and its risk state of mild cognitive impairment (MCI), and to evaluate the diagnostic performance of digital cognitive biomarkers. A literature search was performed in three databases, and supplemented by a Google search for names of previously identified computerized tests. Computerized tests were categorized into five types, including memory tests, test batteries, other single/multiple cognitive tests, handwriting/drawing tests, and daily living tasks and serious games. Results showed that 78 studies were eligible. Around 90% of the included studies were rated as high quality based on the Newcastle–Ottawa Scale (NOS). Most of the digital cognitive biomarkers achieved comparable or even better diagnostic performance than traditional paper-and-pencil tests. Moderate to large group differences were consistently observed in cognitive outcomes related to memory and executive functions, as well as some novel outcomes measured by handwriting/drawing tests, daily living tasks, and serious games. These outcomes have the potential to be sensitive digital cognitive biomarkers for MCI and dementia. Therefore, digital cognitive biomarkers can be a sensitive and promising clinical tool for detecting MCI and dementia.

Modeling Users' Cognitive Performance Using Digital Pen Features

Digital pen features model characteristics of sketches and user behavior, and can be used for various supervised machine learning (ML) applications, such as multi-stroke sketch recognition and user modeling. In this work, we use a state-of-the-art set of more than 170 digital pen features, which we implement and make publicly available. The feature set is evaluated in the use case of analyzing paper-pencil-based neurocognitive assessments in the medical domain. Most cognitive assessments, for dementia screening for example, are conducted with a pen on normal paper. We record these tests with a digital pen as part of a new interactive cognitive assessment tool with automatic analysis of pen input. The physician can, first, observe the sketching process in real-time on a mobile tablet, e.g., in telemedicine settings or to follow Covid-19 distancing regulations. Second, the results of an automatic test analysis are presented to the physician in real-time, thereby reducing manual scoring effort and producing objective reports. As part of our evaluation we examine how accurately different feature-based, supervised ML models can automatically score cognitive tests, with and without semantic content analysis. A series of ML-based sketch recognition experiments is conducted, evaluating 10 modern off-the-shelf ML classifiers (i.e., SVMs, Deep Learning, etc.) on a sketch data set which we recorded with 40 subjects from a geriatrics daycare clinic. In addition, an automated ML approach (AutoML) is explored for fine-tuning and optimizing classification performance on the data set, achieving superior recognition accuracies. Using standard ML techniques our feature set outperforms all previous approaches on the cognitive tests considered, i.e., the Clock Drawing Test, the Rey-Osterrieth Complex Figure Test, and the Trail Making Test, by automatically scoring cognitive tests with up to 87.5% accuracy in a binary classification task.

Personalised treatment for cognitive impairment in dementia: development and validation of an artificial intelligence model

BACKGROUND

Donepezil, galantamine, rivastigmine and memantine are potentially effective interventions for cognitive impairment in dementia, but the use of these drugs has not been personalised to individual patients yet. We examined whether artificial intelligence-based recommendations can identify the best treatment using routinely collected patient-level information.

METHODS

Six thousand eight hundred four patients aged 59-102 years with a diagnosis of dementia from two National Health Service (NHS) Foundation Trusts in the UK were used for model training/internal validation and external validation, respectively. A personalised prescription model based on the Recurrent Neural Network machine learning architecture was developed to predict the Mini-Mental State Examination (MMSE) and Montreal Cognitive Assessment (MoCA) scores post-drug initiation. The drug that resulted in the smallest decline in cognitive scores between prescription and the next visit was selected as the treatment of choice. Change of cognitive scores up to 2 years after treatment initiation was compared for model evaluation.

RESULTS

Overall, 1343 patients with MMSE scores were identified for internal validation and 285 [21.22%] took the drug recommended. After 2 years, the reduction of mean [standard deviation] MMSE score in this group was significantly smaller than the remaining 1058 [78.78%] patients (0.60 [0.26] vs 2.80 [0.28]; P = 0.02). In the external validation cohort (N = 1772), 222 [12.53%] patients took the drug recommended and reported a smaller MMSE reduction compared to the 1550 [87.47%] patients who did not (1.01 [0.49] vs 4.23 [0.60]; P = 0.01). A similar performance gap was seen when testing the model on patients prescribed with AChEIs only.

CONCLUSIONS

It was possible to identify the most effective drug for the real-world treatment of cognitive impairment in dementia at an individual patient level. Routine care patients whose prescribed medications were the best fit according to the model had better cognitive performance after 2 years.

Redefining and Validating Digital Biomarkers as Fluid, Dynamic Multi-Dimensional Digital Signal Patterns

"Digital biomarker" is a term broadly and indiscriminately applied and often limited in its conceptualization to mimic well-established biomarkers as defined and approved by regulatory agencies such as the United States Food and Drug Administration (FDA). There is a practical urgency to revisit the definition of a digital biomarker and expand it beyond current methods of identification and validation. Restricting the promise of digital technologies within the realm of currently defined biomarkers creates a missed opportunity. A whole new field of prognostic and early diagnostic digital biomarkers driven by data science and artificial intelligence can break the current cycle of high healthcare costs and low health quality that is being driven by today's chronic disease detection and treatment approaches. This new class of digital biomarkers will be dynamic and require developing new FDA approval pathways and next-generation gold standards.

Developing an Image-Based Deep Learning Framework for Automatic Scoring of The Pentagon Drawing Test

BACKGROUND

The Pentagon Drawing Test (PDT) is a common assessment for visuospatial function. Evaluating the PDT by artificial intelligence can improve efficiency and reliability in the big data era. This study aimed to develop a deep learning (DL) framework for automatic scoring of the PDT based on image data.

METHODS

A total of 823 PDT photos were retrospectively collected and preprocessed into black-and-white, square-shape images. Stratified fivefold cross-validation was applied for training and testing. Two strategies based on convolutional neural networks were compared. The first strategy was to perform an image classification task using supervised transfer learning. The second strategy was designed with an object detection model for recognizing the geometric shapes in the figure, followed by a predetermined algorithm to score based on their classes and positions.

RESULTS

On average, the first framework demonstrated 62%accuracy, 62%recall, 65%precision, 63%specificity, and 0.72 area under the receiver operating characteristic curve. This performance was substantially outperformed by the second framework, with averages of 94%, 95%, 93%, 93%, and 0.95, respectively.

CONCLUSION

An image-based DL framework based on the object detection approach may be clinically applicable for automatic scoring of the PDT with high efficiency and reliability. With a limited sample size, transfer learning should be used with caution if the new images are distinct from the previous training data. Partitioning the problem-solving workflow into multiple simple tasks should facilitate model selection, improve performance, and allow comprehensible logic of the DL framework.