Machine Learning-Based Automatic Rating for Cardinal Symptoms of Parkinson Disease

Artificial intelligence-based video monitoring of movement disorders in the elderly: a review on current and future landscapes

ABSTRACT

Due to global ageing, the burden of chronic movement and neurological disorders (Parkinson's disease and essential tremor) is rapidly increasing. Current diagnosis and monitoring of these disorders rely largely on face-to-face assessments utilising clinical rating scales, which are semi-subjective and time-consuming. To address these challenges, the utilisation of artificial intelligence (AI) has emerged. This review explores the advantages and challenges associated with using AI-driven video monitoring to care for elderly patients with movement disorders. The AI-based video monitoring systems offer improved efficiency and objectivity in remote patient monitoring, enabling real-time analysis of data, more uniform outcomes and augmented support for clinical trials. However, challenges, such as video quality, privacy compliance and noisy training labels, during development need to be addressed. Ultimately, the advancement of video monitoring for movement disorders is expected to evolve towards discreet, home-based evaluations during routine daily activities. This progression must incorporate data security, ethical considerations and adherence to regulatory standards.

Machine learning-based gait analysis to predict clinical frailty scale in elderly patients with heart failure

Aims

Although frailty assessment is recommended for guiding treatment strategies and outcome prediction in elderly patients with heart failure (HF), most frailty scales are subjective, and the scores vary among raters. We sought to develop a machine learning-based automatic rating method/system/model of the clinical frailty scale (CFS) for patients with HF.

Methods and results

We prospectively examined 417 elderly (≥75 years) with symptomatic chronic HF patients from 7 centres between January 2019 and October 2023. The patients were divided into derivation (n = 194) and validation (n = 223) cohorts. We obtained body-tracking motion data using a deep learning-based pose estimation library, on a smartphone camera. Predicted CFS was calculated from 128 key features, including gait parameters, using the light gradient boosting machine (LightGBM) model. To evaluate the performance of this model, we calculated Cohen's weighted kappa (CWK) and intraclass correlation coefficient (ICC) between the predicted and actual CFSs. In the derivation and validation datasets, the LightGBM models showed excellent agreements between the actual and predicted CFSs [CWK 0.866, 95% confidence interval (CI) 0.807-0.911; ICC 0.866, 95% CI 0.827-0.898; CWK 0.812, 95% CI 0.752-0.868; ICC 0.813, 95% CI 0.761-0.854, respectively]. During a median follow-up period of 391 (inter-quartile range 273-617) days, the higher predicted CFS was independently associated with a higher risk of all-cause death (hazard ratio 1.60, 95% CI 1.02-2.50) after adjusting for significant prognostic covariates.

Conclusion

Machine learning-based algorithms of automatically CFS rating are feasible, and the predicted CFS is associated with the risk of all-cause death in elderly patients with HF.

Computer Vision for Parkinson's Disease Evaluation: A Survey on Finger Tapping

Parkinson's disease (PD) is a progressive neurodegenerative disorder whose prevalence has steadily been rising over the years. Specialist neurologists across the world assess and diagnose patients with PD, although the diagnostic process is time-consuming and various symptoms take years to appear, which means that the diagnosis is prone to human error. The partial automatization of PD assessment and diagnosis through computational processes has therefore been considered for some time. One well-known tool for PD assessment is finger tapping (FT), which can now be assessed through computer vision (CV). Artificial intelligence and related advances over recent decades, more specifically in the area of CV, have made it possible to develop computer systems that can help specialists assess and diagnose PD. The aim of this study is to review some advances related to CV techniques and FT so as to offer insight into future research lines that technological advances are now opening up.

Clinically Informed Automated Assessment of Finger Tapping Videos in Parkinson's Disease

The utilization of Artificial Intelligence (AI) for assessing motor performance in Parkinson's Disease (PD) offers substantial potential, particularly if the results can be integrated into clinical decision-making processes. However, the precise quantification of PD symptoms remains a persistent challenge. The current standard Unified Parkinson's Disease Rating Scale (UPDRS) and its variations serve as the primary clinical tools for evaluating motor symptoms in PD, but are time-intensive and prone to inter-rater variability. Recent work has applied data-driven machine learning techniques to analyze videos of PD patients performing motor tasks, such as finger tapping, a UPDRS task to assess bradykinesia. However, these methods often use abstract features that are not closely related to clinical experience. In this paper, we introduce a customized machine learning approach for the automated scoring of UPDRS bradykinesia using single-view RGB videos of finger tapping, based on the extraction of detailed features that rigorously conform to the established UPDRS guidelines. We applied the method to 75 videos from 50 PD patients collected in both a laboratory and a realistic clinic environment. The classification performance agreed well with expert assessors, and the features selected by the Decision Tree aligned with clinical knowledge. Our proposed framework was designed to remain relevant amid ongoing patient recruitment and technological progress. The proposed approach incorporates features that closely resonate with clinical reasoning and shows promise for clinical implementation in the foreseeable future.

Machine learning with optimization to create medicine intake schedules for Parkinson's disease patients

This paper presents a solution for creating individualized medicine intake schedules for Parkinson's disease patients. Dosing medicine in Parkinson's disease is a difficult and a time-consuming task and wrongly assigned therapy affects patient's quality of life making the disease more uncomfortable. The method presented in this paper may decrease errors in therapy and time required to establish a suitable medicine intake schedule by using objective measures to predict patient's response to medication. Firstly, it demonstrates the use of machine learning models to predict the patient's medicine response based on their state evaluation acquired during examination with biomedical sensors. Two architectures, a multilayer perceptron and a deep neural network with LSTM cells are proposed to evaluate the patient's future state based on their past condition and medication history, with the best patient-specific models achieving R2 value exceeding 0.96. These models serve as a foundation for conventional optimization, specifically genetic algorithm and differential evolution. These methods are applied to find optimal medicine intake schedules for patient's daily routine, resulting in a 7% reduction in the objective function value compared to existing approaches. To achieve this goal and be able to adapt the schedule during the day, reinforcement learning is also utilized. An agent is trained to suggest medicine doses that maintain the patient in an optimal state. The conducted experiments demonstrate that machine learning models can effectively model a patient's response to medication and both optimization approaches prove capable of finding optimal medicine schedules for patients. With further training on larger datasets from real patients the method has the potential to significantly improve the treatment of Parkinson's disease.

Gait video-based prediction of unified Parkinson's disease rating scale score: a retrospective study

BACKGROUND

The diagnosis of Parkinson's disease (PD) and evaluation of its symptoms require in-person clinical examination. Remote evaluation of PD symptoms is desirable, especially during a pandemic such as the coronavirus disease 2019 pandemic. One potential method to remotely evaluate PD motor impairments is video-based analysis. In this study, we aimed to assess the feasibility of predicting the Unified Parkinson's Disease Rating Scale (UPDRS) score from gait videos using a convolutional neural network (CNN) model.

METHODS

We retrospectively obtained 737 consecutive gait videos of 74 patients with PD and their corresponding neurologist-rated UPDRS scores. We utilized a CNN model for predicting the total UPDRS part III score and four subscores of axial symptoms (items 27, 28, 29, and 30), bradykinesia (items 23, 24, 25, 26, and 31), rigidity (item 22) and tremor (items 20 and 21). We trained the model on 80% of the gait videos and used 10% of the videos as a validation dataset. We evaluated the predictive performance of the trained model by comparing the model-predicted score with the neurologist-rated score for the remaining 10% of videos (test dataset). We calculated the coefficient of determination (R2) between those scores to evaluate the model's goodness of fit.

RESULTS

In the test dataset, the R2 values between the model-predicted and neurologist-rated values for the total UPDRS part III score and subscores of axial symptoms, bradykinesia, rigidity, and tremor were 0.59, 0.77, 0.56, 0.46, and 0.0, respectively. The performance was relatively low for videos from patients with severe symptoms.

CONCLUSIONS

Despite the low predictive performance of the model for the total UPDRS part III score, it demonstrated relatively high performance in predicting subscores of axial symptoms. The model approximately predicted the total UPDRS part III scores of patients with moderate symptoms, but the performance was low for patients with severe symptoms owing to limited data. A larger dataset is needed to improve the model's performance in clinical settings.

Multi-view emotional expressions dataset using 2D pose estimation

Human body expressions convey emotional shifts and intentions of action and, in some cases, are even more effective than other emotion models. Despite many datasets of body expressions incorporating motion capture available, there is a lack of more widely distributed datasets regarding naturalized body expressions based on the 2D video. In this paper, therefore, we report the multi-view emotional expressions dataset (MEED) using 2D pose estimation. Twenty-two actors presented six emotional (anger, disgust, fear, happiness, sadness, surprise) and neutral body movements from three viewpoints (left, front, right). A total of 4102 videos were captured. The MEED consists of the corresponding pose estimation results (i.e., 397,809 PNG files and 397,809 JSON files). The size of MEED exceeds 150 GB. We believe this dataset will benefit the research in various fields, including affective computing, human-computer interaction, social neuroscience, and psychiatry.

A Deep Learning Approach for Grading of Motor Impairment Severity in Parkinson's Disease

Objective and quantitative monitoring of movement impairments is crucial for detecting progression in neurological conditions such as Parkinson's disease (PD). This study examined the ability of deep learning approaches to grade motor impairment severity in a modified version of the Movement Disorders Society-sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS) using low-cost wearable sensors. A convolutional neural network architecture, XceptionTime, was used to classify lower and higher levels of motor impairment in persons with PD, across five distinct rhythmic tasks: finger tapping, hand movements, pronation-supination movements of the hands, toe tapping, and leg agility. In addition, an aggregate model was trained on data from all tasks together for evaluating bradykinesia symptom severity in PD. The model performance was highest in the hand movement tasks with an accuracy of 82.6% in the hold-out test dataset; the accuracy for the aggregate model was 79.7%, however, it demonstrated the lowest variability. Overall, these findings suggest the feasibility of integrating low-cost wearable technology and deep learning approaches to automatically and objectively quantify motor impairment in persons with PD. This approach may provide a viable solution for a widely deployable telemedicine solution.

A First Methodological Development and Validation of ReTap: An Open-Source UPDRS Finger Tapping Assessment Tool Based on Accelerometer-Data

Bradykinesia is a cardinal hallmark of Parkinson's disease (PD). Improvement in bradykinesia is an important signature of effective treatment. Finger tapping is commonly used to index bradykinesia, albeit these approaches largely rely on subjective clinical evaluations. Moreover, recently developed automated bradykinesia scoring tools are proprietary and are not suitable for capturing intraday symptom fluctuation. We assessed finger tapping (i.e., Unified Parkinson's Disease Rating Scale (UPDRS) item 3.4) in 37 people with Parkinson's disease (PwP) during routine treatment follow ups and analyzed their 350 sessions of 10-s tapping using index finger accelerometry. Herein, we developed and validated ReTap, an open-source tool for the automated prediction of finger tapping scores. ReTap successfully detected tapping blocks in over 94% of cases and extracted clinically relevant kinematic features per tap. Importantly, based on the kinematic features, ReTap predicted expert-rated UPDRS scores significantly better than chance in a hold out validation sample (n = 102). Moreover, ReTap-predicted UPDRS scores correlated positively with expert ratings in over 70% of the individual subjects in the holdout dataset. ReTap has the potential to provide accessible and reliable finger tapping scores, either in the clinic or at home, and may contribute to open-source and detailed analyses of bradykinesia.

Remote scoring models of rigidity and postural stability of Parkinson's disease based on indirect motions and a low-cost RGB algorithm

Background and objectives

The Movement Disorder Society's Unified Parkinson's Disease Rating Scale Part III (MDS-UPDRS III) is mostly common used for assessing the motor symptoms of Parkinson's disease (PD). In remote circumstances, vision-based techniques have many strengths over wearable sensors. However, rigidity (item 3.3) and postural stability (item 3.12) in the MDS-UPDRS III cannot be assessed remotely since participants need to be touched by a trained examiner during testing. We developed the four scoring models of rigidity of the neck, rigidity of the lower extremities, rigidity of the upper extremities, and postural stability based on features extracted from other available and touchless motions.

Methods

The red, green, and blue (RGB) computer vision algorithm and machine learning were combined with other available motions from the MDS-UPDRS III evaluation. A total of 104 patients with PD were split into a train set (89 individuals) and a test set (15 individuals). The light gradient boosting machine (LightGBM) multiclassification model was trained. Weighted kappa (k), absolute accuracy (ACC ± 0), and Spearman's correlation coefficient (rho) were used to evaluate the performance of model.

Results

For model of rigidity of the upper extremities, k = 0.58 (moderate), ACC ± 0 = 0.73, and rho = 0.64 (moderate). For model of rigidity of the lower extremities, k = 0.66 (substantial), ACC ± 0 = 0.70, and rho = 0.76 (strong). For model of rigidity of the neck, k = 0.60 (moderate), ACC ± 0 = 0.73, and rho = 0.60 (moderate). For model of postural stability, k = 0.66 (substantial), ACC ± 0 = 0.73, and rho = 0.68 (moderate).

Conclusion

Our study can be meaningful for remote assessments, especially when people have to maintain social distance, e.g., in situations such as the coronavirus disease-2019 (COVID-19) pandemic.

Quantifying neuro-motor correlations during awake deep brain stimulation surgery using markerless tracking

The expanding application of deep brain stimulation (DBS) therapy both drives and is informed by our growing understanding of disease pathophysiology and innovations in neurosurgical care. Neurophysiological targeting, a mainstay for identifying optimal, motor responsive targets, has remained largely unchanged for decades. Utilizing deep learning-based computer vision and related computational methods, we developed an effective and simple intraoperative approach to objectively correlate neural signals with movements, automating and standardizing the otherwise manual and subjective process of identifying ideal DBS electrode placements. Kinematics are extracted from video recordings of intraoperative motor testing using a trained deep neural network and compared to multi-unit activity recorded from the subthalamic nucleus. Neuro-motor correlations were quantified using dynamic time warping with the strength of a given comparison measured by comparing against a null distribution composed of related neuro-motor correlations. This objective measure was then compared to clinical determinations as recorded in surgical case notes. In seven DBS cases for treatment of Parkinson's disease, 100 distinct motor testing epochs were extracted for which clear clinical determinations were made. Neuro-motor correlations derived by our automated system compared favorably with expert clinical decision making in post-hoc comparisons, although follow-up studies are necessary to determine if improved correlation detection leads to improved outcomes. By improving the classification of neuro-motor relationships, the automated system we have developed will enable clinicians to maximize the therapeutic impact of DBS while also providing avenues for improving continued care of treated patients.

Deep Clinical Phenotyping of Parkinson's Disease: Towards a New Era of Research and Clinical Care

Despite recent advances in technology, clinical phenotyping of Parkinson's disease (PD) has remained relatively limited as current assessments are mainly based on empirical observation and subjective categorical judgment at the clinic. A lack of comprehensive, objective, and quantifiable clinical phenotyping data has hindered our capacity to diagnose, assess patients' conditions, discover pathogenesis, identify preclinical stages and clinical subtypes, and evaluate new therapies. Therefore, deep clinical phenotyping of PD patients is a necessary step towards understanding PD pathology and improving clinical care. In this review, we present a growing community consensus and perspective on how to clinically phenotype this disease, that is, to phenotype the entire course of disease progression by integrating capacity, performance, and perception approaches with state-of-the-art technology. We also explore the most studied aspects of PD deep clinical phenotypes, namely, bradykinesia, tremor, dyskinesia and motor fluctuation, gait impairment, speech impairment, and non-motor phenotypes.

A non-contact system for intraoperative quantitative assessment of bradykinesia in deep brain stimulation surgery

BACKGROUND AND OBJECTIVE

Deep brain stimulation (DBS) is an effective treatment for a number of neurological diseases, especially for the advanced stage of Parkinson's disease (PD). Objective assessment of patients' motor symptoms is crucial for accurate electrode targeting and treatment. Existing approaches suffer from subjective variability or interference with voluntary motion. This work is aimed to establish an objective assessment system to quantify bradykinesia in DBS surgery.

METHODS

Based on the analysis of the requirements for intraoperative assessment, we developed a system with non-contact measurement, online movement feature extraction, and interactive data analysis and visualization. An optical sensor, Leap Motion Controller (LMC), was taken to detect hand movement in three clinical tasks. A graphic user interface was designed to process, compare and visualize the collected data and assessment results online. Quantified movement features include amplitude, frequency, velocity, their decrement and variability, etc. Technical validation of the system was performed with a motion capture system (Mocap), with respect to data-level and feature-level accuracy and reliability. Clinical validation was conducted with 20 PD patients for intraoperative assessments in DBS surgery. Treatment responses with respect to the bradykinesia movement features were analyzed. Single case analysis and group statistical analysis were performed to examine the differences between preoperative and intraoperative performance, and the correlation between the clinical ratings and the quantified assessment was analyzed.

RESULTS

For the movements measured by LMC and Mocap, the average Pearson's correlation coefficient was 0.986, and the mean amplitude difference was 2.11 mm. No significant difference was found for all movement features quantified by LMC and Mocap. For the clinical tests, key movement features showed significant differences between the preoperative baseline and intraoperative performance when the brain stimulation was ON. The assessment results were significantly correlated with the MDS-UPDRS clinical ratings.

CONCLUSIONS

The proposed non-contact system has established itself as an objective intraoperative assessment, analysis, and visualization tool for DBS treatment of Parkinson's disease.

Reliability of a human pose tracking algorithm for measuring upper limb joints: comparison with photography-based goniometry

Background

Range of motion (ROM) measurements are essential for diagnosing and evaluating upper extremity conditions. Clinical goniometry is the most commonly used methods but it is time-consuming and skill-demanding. Recent advances in human tracking algorithm suggest potential for automatic angle measuring from RGB images. It provides an attractive alternative for at-distance measuring. However, the reliability of this method has not been fully established. The purpose of this study is to evaluate if the results of algorithm are as reliable as human raters in upper limb movements.

Methods

Thirty healthy young adults (20 males, 10 females) participated in this study. Participants were asked to performed a 6-motion task including movement of shoulder, elbow and wrist. Images of movements were captured by commercial digital cameras. Each movement was measured by a pose tracking algorithm (OpenPose) and compared with the surgeon-measurement results. The mean differences between the two measurements were compared. Pearson correlation coefficients were used to determine the relationship. Reliability was investigated by the intra-class correlation coefficients.

Results

Comparing this algorithm-based method with manual measurement, the mean differences were less than 3 degrees in 5 motions (shoulder abduction: 0.51; shoulder elevation: 2.87; elbow flexion:0.38; elbow extension:0.65; wrist extension: 0.78) except wrist flexion. All the intra-class correlation coefficients were larger than 0.60. The Pearson coefficients also showed high correlations between the two measurements (p < 0.001).

Conclusions

Our results indicated that pose estimation is a reliable method to measure the shoulder and elbow angles, supporting RGB images for measuring joint ROM. Our results presented the possibility that patients can assess their ROM by photos taken by a digital camera.

Trial registration

This study was registered in the Clinical Trials Center of The First Affiliated Hospital, Sun Yat-sen University (2021–387).

Supplementary Information

The online version contains supplementary material available at 10.1186/s12891-022-05826-4.

Identifying the Posture of Young Adults in Walking Videos by Using a Fusion Artificial Intelligent Method

Many neurological and musculoskeletal disorders are associated with problems related to postural movement. Noninvasive tracking devices are used to record, analyze, measure, and detect the postural control of the body, which may indicate health problems in real time. A total of 35 young adults without any health problems were recruited for this study to participate in a walking experiment. An iso-block postural identity method was used to quantitatively analyze posture control and walking behavior. The participants who exhibited straightforward walking and skewed walking were defined as the control and experimental groups, respectively. Fusion deep learning was applied to generate dynamic joint node plots by using OpenPose-based methods, and skewness was qualitatively analyzed using convolutional neural networks. The maximum specificity and sensitivity achieved using a combination of ResNet101 and the naïve Bayes classifier were 0.84 and 0.87, respectively. The proposed approach successfully combines cell phone camera recordings, cloud storage, and fusion deep learning for posture estimation and classification.

Acceptability of an In-Home Multimodal Sensor Platform in Parkinson’s Disease: A Qualitative Study (Preprint)

Background

Parkinson disease (PD) symptoms are complex, gradually progressive, and fluctuate hour by hour. Home-based technological sensors are being investigated to measure symptoms and track disease progression. A smart home sensor platform, with cameras and wearable devices, could be a useful tool to use to get a fuller picture of what someone’s symptoms are like. High-resolution video can capture the ground truth of symptoms and activities. There is a paucity of information about the acceptability of such sensors in PD.

Objective

The primary objective of our study was to explore the acceptability of living with a multimodal sensor platform in a naturalistic setting in PD. Two subobjectives are to identify any suggested limitations and to explore the sensors’ impact on participant behaviors.

Methods

A qualitative study was conducted with an inductive approach using semistructured interviews with a cohort of PD and control participants who lived freely for several days in a home-like environment while continuously being sensed.

Results

This study of 24 participants (12 with PD) found that it is broadly acceptable to use multimodal sensors including wrist-worn wearables, cameras, and other ambient sensors passively in free-living in PD. The sensor that was found to be the least acceptable was the wearable device. Suggested limitations on the platform for home deployment included camera-free time and space. Behavior changes were noted by the study participants, which may have related to being passively sensed. Recording high-resolution video in the home setting for limited periods of time was felt to be acceptable to all participants.

Conclusions

The results broaden the knowledge of what types of sensors are acceptable for use in research in PD and what potential limitations on these sensors should be considered in future work. The participants’ reported behavior change in this study should inform future similar research design to take this factor into account. Collaborative research study design, involving people living with PD at every stage, is important to ensure that the technology is acceptable and that the data outcomes produced are ecologically valid and accurate.

International Registered Report Identifier (IRRID)

RR2-10.1136/bmjopen-2020-041303

Video-Based Automated Assessment of Movement Parameters Consistent with MDS-UPDRS III in Parkinson's Disease

BACKGROUND

Among motor symptoms of Parkinson's disease (PD), including rigidity and resting tremor, bradykinesia is a mandatory feature to define the parkinsonian syndrome. MDS-UPDRS III is the worldwide reference scale to evaluate the parkinsonian motor impairment, especially bradykinesia. However, MDS-UPDRS III is an agent-based score making reproducible measurements and follow-up challenging.

OBJECTIVE

Using a deep learning approach, we developed a tool to compute an objective score of bradykinesia based on the guidelines of the gold-standard MDS-UPDRS III.

METHODS

We adapted and applied two deep learning algorithms to detect a two-dimensional (2D) skeleton of the hand composed of 21 predefined points, and transposed it into a three-dimensional (3D) skeleton for a large database of videos of parkinsonian patients performing MDS-UPDRS III protocols acquired in the Movement Disorder unit of Avicenne University Hospital.

RESULTS

We developed a 2D and 3D automated analysis tool to study the evolution of several key parameters during the protocol repetitions of the MDS-UPDRS III. Scores from 2D automated analysis showed a significant correlation with gold-standard ratings of MDS-UPDRS III, measured with coefficients of determination for the tapping (0.609) and hand movements (0.701) protocols using decision tree algorithms. The individual correlations of the different parameters measured with MDS-UPDRS III scores carry meaningful information and are consistent with MDS-UPDRS III guidelines.

CONCLUSION

We developed a deep learning-based tool to precisely analyze movement parameters allowing to reliably score bradykinesia for parkinsonian patients in a MDS-UPDRS manner.