Validation of a machine learning approach to estimate expanded disability status scale scores for multiple sclerosis

Natural language processing systems for extracting information from electronic health records about activities of daily living. A systematic review

Objective

Natural language processing (NLP) can enhance research on activities of daily living (ADL) by extracting structured information from unstructured electronic health records (EHRs) notes. This review aims to give insight into the state-of-the-art, usability, and performance of NLP systems to extract information on ADL from EHRs.

Materials and Methods

A systematic review was conducted based on searches in Pubmed, Embase, Cinahl, Web of Science, and Scopus. Studies published between 2017 and 2022 were selected based on predefined eligibility criteria.

Results

The review identified 22 studies. Most studies (65%) used NLP for classifying unstructured EHR data on 1 or 2 ADL. Deep learning, combined with a ruled-based method or machine learning, was the approach most commonly used. NLP systems varied widely in terms of the pre-processing and algorithms. Common performance evaluation methods were cross-validation and train/test datasets, with F1, precision, and sensitivity as the most frequently reported evaluation metrics. Most studies reported relativity high overall scores on the evaluation metrics.

Discussion

NLP systems are valuable for the extraction of unstructured EHR data on ADL. However, comparing the performance of NLP systems is difficult due to the diversity of the studies and challenges related to the dataset, including restricted access to EHR data, inadequate documentation, lack of granularity, and small datasets.

Conclusion

This systematic review indicates that NLP is promising for deriving information on ADL from unstructured EHR notes. However, what the best-performing NLP system is, depends on characteristics of the dataset, research question, and type of ADL.

Claims-based algorithm to estimate the Expanded Disability Status Scale for multiple sclerosis in a German health insurance fund: a validation study using patient medical records

Background

The Expanded Disability Status Scale (EDSS) quantifies disability and measures disease progression in multiple sclerosis (MS), however is not available in administrative claims databases.

Objectives

To develop a claims-based algorithm for deriving EDSS and validate it against a clinical dataset capturing true EDSS values from medical records.

Methods

We built a unique linked dataset combining claims data from the German AOK PLUS sickness fund and medical records from the Multiple Sclerosis Management System 3D (MSDS3D). Data were deterministically linked based on insurance numbers. We used 69 MS-related diagnostic indicators recorded with ICD-10-GM codes within 3 months before and after recorded true EDSS measures to estimate a claims-based EDSS proxy (pEDSS). Predictive performance of the pEDSS was assessed as an eight-fold (EDSS 1.0-7.0, ≥8.0), three-fold (EDSS 1.0-3.0, 4.0-5.0, ≥6.0), and binary classifier (EDSS <6.0, ≥6.0). For each classifier, predictive performance measures were determined, and overall performance was summarized using a macro F1-score. Finally, we implemented the algorithm to determine pEDSS among an overall cohort of patients with MS in AOK PLUS, who were alive and insured 12 months prior to and after index diagnosis.

Results

We recruited 100 people with MS insured by AOK PLUS who had ≥1 EDSS measure in MSDS3D between 01/10/2015 and 30/06/2019 (620 measurements overall). Patients had a mean rescaled EDSS of 3.2 and pEDSS of 3.0. The pEDSS deviated from the true EDSS by 1.2 points, resulting in a mean squared error of prediction of 2.6. For the eight-fold classifier, the macro F1-score of 0.25 indicated low overall predictive performance. Broader severity groupings were better performing, with the three-fold and binary classifiers for severe disability achieving a F1-score of 0.68 and 0.84, respectively. In the overall AOK PLUS cohort (3,756 patients, 71.9% female, mean 51.9 years), older patients, patients with progressive forms of MS and those with higher comorbidity burden showed higher pEDSS.

Conclusion

Generally, EDSS was underestimated by the algorithm as mild-to-moderate symptoms were poorly captured in claims across all functional systems. While the proxy-based approach using claims data may not allow for granular description of MS disability, broader severity groupings show good predictive performance.