Reverse Engineering of Digital Measures: Inviting Patients to the Conversation

Background

Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures.

Summary

In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled "Reverse Engineering of Digital Measures," was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools.

Key Messages

In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.

Digital Devices for Assessing Motor Functions in Mobility-Impaired and Healthy Populations: Systematic Literature Review

Background

With the advent of smart sensing technology, mobile and wearable devices can provide continuous and objective monitoring and assessment of motor function outcomes.

Objective

We aimed to describe the existing scientific literature on wearable and mobile technologies that are being used or tested for assessing motor functions in mobility-impaired and healthy adults and to evaluate the degree to which these devices provide clinically valid measures of motor function in these populations.

Methods

A systematic literature review was conducted by searching Embase, MEDLINE, CENTRAL (January 1, 2015, to June 24, 2020), the United States and European Union clinical trial registries, and the United States Food and Drug Administration website using predefined study selection criteria. Study selection, data extraction, and quality assessment were performed by 2 independent reviewers.

Results

A total of 91 publications representing 87 unique studies were included. The most represented clinical conditions were Parkinson disease (n=51 studies), followed by stroke (n=5), Huntington disease (n=5), and multiple sclerosis (n=2). A total of 42 motion-detecting devices were identified, and the majority (n=27, 64%) were created for the purpose of health care–related data collection, although approximately 25% were personal electronic devices (eg, smartphones and watches) and 11% were entertainment consoles (eg, Microsoft Kinect or Xbox and Nintendo Wii). The primary motion outcomes were related to gait (n=30), gross motor movements (n=25), and fine motor movements (n=23). As a group, sensor-derived motion data showed a mean sensitivity of 0.83 (SD 7.27), a mean specificity of 0.84 (SD 15.40), a mean accuracy of 0.90 (SD 5.87) in discriminating between diseased individuals and healthy controls, and a mean Pearson r validity coefficient of 0.52 (SD 0.22) relative to clinical measures. We did not find significant differences in the degree of validity between in-laboratory and at-home sensor-based assessments nor between device class (ie, health care–related device, personal electronic devices, and entertainment consoles).

Conclusions

Sensor-derived motion data can be leveraged to classify and quantify disease status for a variety of neurological conditions. However, most of the recent research on digital clinical measures is derived from proof-of-concept studies with considerable variation in methodological approaches, and much of the reviewed literature has focused on clinical validation, with less than one-quarter of the studies performing analytical validation. Overall, future research is crucially needed to further consolidate that sensor-derived motion data may lead to the development of robust and transformative digital measurements intended to predict, diagnose, and quantify neurological disease state and its longitudinal change.

A smartphone sensor-based digital outcome assessment of multiple sclerosis

Background:

Sensor-based monitoring tools fill a critical gap in multiple sclerosis (MS) research and clinical care.

Objective:

The aim of this study is to assess performance characteristics of the Floodlight Proof-of-Concept (PoC) app.

Methods:

In a 24-week study (clinicaltrials.gov: NCT02952911), smartphone-based active tests and passive monitoring assessed cognition (electronic Symbol Digit Modalities Test), upper extremity function (Pinching Test, Draw a Shape Test), and gait and balance (Static Balance Test, U-Turn Test, Walk Test, Passive Monitoring). Intraclass correlation coefficients (ICCs) and age- or sex-adjusted Spearman’s rank correlation determined test–retest reliability and correlations with clinical and magnetic resonance imaging (MRI) outcome measures, respectively.

Results:

Seventy-six people with MS (PwMS) and 25 healthy controls were enrolled. In PwMS, ICCs were moderate-to-good (ICC(2,1) = 0.61–0.85) across tests. Correlations with domain-specific standard clinical disability measures were significant for all tests in the cognitive (r = 0.82, p < 0.001), upper extremity function (|r|= 0.40–0.64, all p < 0.001), and gait and balance domains (r = −0.25 to −0.52, all p < 0.05; except for Static Balance Test: r = −0.20, p > 0.05). Most tests also correlated with Expanded Disability Status Scale, 29-item Multiple Sclerosis Impact Scale items or subscales, and/or normalized brain volume.

Conclusion:

The Floodlight PoC app captures reliable and clinically relevant measures of functional impairment in MS, supporting its potential use in clinical research and practice.

Smartphone- and Smartwatch-Based Remote Characterisation of Ambulation in Multiple Sclerosis During the Two-Minute Walk Test

Leveraging consumer technology such as smartphone and smartwatch devices to objectively assess people with multiple sclerosis (PwMS) remotely could capture unique aspects of disease progression. This study explores the feasibility of assessing PwMS and Healthy Control's (HC) physical function by characterising gait-related features, which can be modelled using machine learning (ML) techniques to correctly distinguish subgroups of PwMS from healthy controls. A total of 97 subjects (24 HC subjects, 52 mildly disabled (PwMSmild, EDSS [0-3]) and 21 moderately disabled (PwMSmod, EDSS [3.5-5.5]) contributed data which was recorded from a Two-Minute Walk Test (2MWT) performed out-of-clinic and daily over a 24-week period. Signal-based features relating to movement were extracted from sensors in smartphone and smartwatch devices. A large number of features (n = 156) showed fair-to-strong (R 0.3) correlations with clinical outcomes. LASSO feature selection was applied to select and rank subsets of features used for dichotomous classification between subject groups, which were compared using Logistic Regression (LR), Support Vector Machines (SVM) and Random Forest (RF) models. Classifications of subject types were compared using data obtained from smartphone, smartwatch and the fusion of features from both devices. Models built on smartphone features alone achieved the highest classification performance, indicating that accurate and remote measurement of the ambulatory characteristics of HC and PwMS can be achieved with only one device. It was observed however that smartphone-based performance was affected by inconsistent placement location (running belt versus pocket). Results show that PwMSmod could be distinguished from HC subjects (Acc. 82.2 ± 2.9%, Sen. 80.1 ± 3.9%, Spec. 87.2 ± 4.2%, F ₁ 84.3 ± 3.8), and PwMSmild (Acc. 82.3 ± 1.9%, Sen. 71.6 ± 4.2%, Spec. 87.0 ± 3.2%, F 1 75.1 ± 2.2) using an SVM classifier with a Radial Basis Function (RBF). PwMSmild were shown to exhibit HC-like behaviour and were thus less distinguishable from HC (Acc. 66.4 ± 4.5%, Sen. 67.5 ± 5.7%, Spec. 60.3 ± 6.7%, F ₁ 58.6 ± 5.8). Finally, it was observed that subjects in this study demonstrated low intra- and high inter-subject variability which was representative of subject-specific gait characteristics.

Gait Characteristics Harvested During a Smartphone-Based Self-Administered 2-Minute Walk Test in People with Multiple Sclerosis: Test-Retest Reliability and Minimum Detectable Change

The measurement of gait characteristics during a self-administered 2-minute walk test (2MWT), in persons with multiple sclerosis (PwMS), using a single body-worn device, has the potential to provide high-density longitudinal information on disease progression, beyond what is currently measured in the clinician-administered 2MWT. The purpose of this study is to determine the test-retest reliability, standard error of measurement (SEM) and minimum detectable change (MDC) of features calculated on gait characteristics, harvested during a self-administered 2MWT in a home environment, in 51 PwMS and 11 healthy control (HC) subjects over 24 weeks, using a single waist-worn inertial sensor-based smartphone. Excellent, or good to excellent test-retest reliability were observed in 58 of the 92 temporal, spatial and spatiotemporal gait features in PwMS. However, these were less reliable for HCs. Low SEM% and MDC% values were observed for most of the distribution measures for all gait characteristics for PwMS and HCs. This study demonstrates the inter-session test-retest reliability and provides an indication of clinically important change estimates, for interpreting the outcomes of gait characteristics measured using a body-worn smartphone, during a self-administered 2MWT. This system thus provides a reliable measure of gait characteristics in PwMS, supporting its application for the longitudinal assessment of gait deficits in this population.

Correction: Adherence and Satisfaction of Smartphone- and Smartwatch-Based Remote Active Testing and Passive Monitoring in People With Multiple Sclerosis: Nonrandomized Interventional Feasibility Study

[This corrects the article DOI: 10.2196/14863.].

Adherence and Satisfaction of Smartphone- and Smartwatch-Based Remote Active Testing and Passive Monitoring in People With Multiple Sclerosis: Nonrandomized Interventional Feasibility Study

Background

Current clinical assessments of people with multiple sclerosis are episodic and may miss critical features of functional fluctuations between visits.

Objective

The goal of the research was to assess the feasibility of remote active testing and passive monitoring using smartphones and smartwatch technology in people with multiple sclerosis with respect to adherence and satisfaction with the FLOODLIGHT test battery.

Methods

People with multiple sclerosis (aged 20 to 57 years; Expanded Disability Status Scale 0-5.5; n=76) and healthy controls (n=25) performed the FLOODLIGHT test battery, comprising active tests (daily, weekly, every two weeks, or on demand) and passive monitoring (sensor-based gait and mobility) for 24 weeks using a smartphone and smartwatch. The primary analysis assessed adherence (proportion of weeks with at least 3 days of completed testing and 4 hours per day passive monitoring) and questionnaire-based satisfaction. In-clinic assessments (clinical and magnetic resonance imaging) were performed.

Results

People with multiple sclerosis showed 70% (16.68/24 weeks) adherence to active tests and 79% (18.89/24 weeks) to passive monitoring; satisfaction score was on average 73.7 out of 100. Neither adherence nor satisfaction was associated with specific population characteristics. Test-battery assessments had an at least acceptable impact on daily activities in over 80% (61/72) of people with multiple sclerosis.

Conclusions

People with multiple sclerosis were engaged and satisfied with the FLOODLIGHT test battery. FLOODLIGHT sensor-based measures may enable continuous assessment of multiple sclerosis disease in clinical trials and real-world settings.

Trial Registration

ClinicalTrials.gov: NCT02952911; https://clinicaltrials.gov/ct2/show/NCT02952911

Evaluation of smartphone-based testing to generate exploratory outcome measures in a phase 1 Parkinson's disease clinical trial

BACKGROUND

Ubiquitous digital technologies such as smartphone sensors promise to fundamentally change biomedical research and treatment monitoring in neurological diseases such as PD, creating a new domain of digital biomarkers.

OBJECTIVES

The present study assessed the feasibility, reliability, and validity of smartphone-based digital biomarkers of PD in a clinical trial setting.

METHODS

During a 6-month, phase 1b clinical trial with 44 Parkinson participants, and an independent, 45-day study in 35 age-matched healthy controls, participants completed six daily motor active tests (sustained phonation, rest tremor, postural tremor, finger-tapping, balance, and gait), then carried the smartphone during the day (passive monitoring), enabling assessment of, for example, time spent walking and sit-to-stand transitions by gyroscopic and accelerometer data.

RESULTS

Adherence was acceptable: Patients completed active testing on average 3.5 of 7 times/week. Sensor-based features showed moderate-to-excellent test-retest reliability (average intraclass correlation coefficient = 0.84). All active and passive features significantly differentiated PD from controls with P < 0.005. All active test features except sustained phonation were significantly related to corresponding International Parkinson and Movement Disorder Society-Sponsored UPRDS clinical severity ratings. On passive monitoring, time spent walking had a significant (P = 0.005) relationship with average postural instability and gait disturbance scores. Of note, for all smartphone active and passive features except postural tremor, the monitoring procedure detected abnormalities even in those Parkinson participants scored as having no signs in the corresponding International Parkinson and Movement Disorder Society-Sponsored UPRDS items at the site visit.

CONCLUSIONS

These findings demonstrate the feasibility of smartphone-based digital biomarkers and indicate that smartphone-sensor technologies provide reliable, valid, clinically meaningful, and highly sensitive phenotypic data in Parkinson's disease. © 2018 The Authors. Movement Disorders published by Wiley Periodicals, Inc. on behalf of International Parkinson and Movement Disorder Society.