Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

Investigating Measurement Equivalence of Smartphone Sensor-Based Assessments: Remote, Digital, Bring-Your-Own-Device Study

BACKGROUND

Floodlight Open is a global, open-access, fully remote, digital-only study designed to understand the drivers and barriers in deployment and persistence of use of a smartphone app for measuring functional impairment in a naturalistic setting and broad study population.

OBJECTIVE

This study aims to assess measurement equivalence properties of the Floodlight Open app across operating system (OS) platforms, OS versions, and smartphone device models.

METHODS

Floodlight Open enrolled adult participants with and without self-declared multiple sclerosis (MS). The study used the Floodlight Open app, a "bring-your-own-device" (BYOD) solution that remotely measured MS-related functional ability via smartphone sensor-based active tests. Measurement equivalence was assessed in all evaluable participants by comparing the performance on the 6 active tests (ie, tests requiring active input from the user) included in the app across OS platforms (iOS vs Android), OS versions (iOS versions 11-15 and separately Android versions 8-10; comparing each OS version with the other OS versions pooled together), and device models (comparing each device model with all remaining device models pooled together). The tests in scope were Information Processing Speed, Information Processing Speed Digit-Digit (measuring reaction speed), Pinching Test (PT), Static Balance Test, U-Turn Test, and 2-Minute Walk Test. Group differences were assessed by permutation test for the mean difference after adjusting for age, sex, and self-declared MS disease status.

RESULTS

Overall, 1976 participants using 206 different device models were included in the analysis. Differences in test performance between subgroups were very small or small, with percent differences generally being ≤5% on the Information Processing Speed, Information Processing Speed Digit-Digit, U-Turn Test, and 2-Minute Walk Test; <20% on the PT; and <30% on the Static Balance Test. No statistically significant differences were observed between OS platforms other than on the PT (P<.001). Similarly, differences across iOS or Android versions were nonsignificant after correcting for multiple comparisons using false discovery rate correction (all adjusted P>.05). Comparing the different device models revealed a statistically significant difference only on the PT for 4 out of 17 models (adjusted P≤.001-.03).

CONCLUSIONS

Consistent with the hypothesis that smartphone sensor-based measurements obtained with different devices are equivalent, this study showed no evidence of a systematic lack of measurement equivalence across OS platforms, OS versions, and device models on 6 active tests included in the Floodlight Open app. These results are compatible with the use of smartphone-based tests in a bring-your-own-device setting, but more formal tests of equivalence would be needed.

The Ultra-Long-Term Sleep study: Design, rationale, data stability and user perspective

Sleep deprivation and poor sleep quality are significant societal challenges that negatively impact individuals' health. The interaction between subjective sleep quality, objective sleep measures, physical and cognitive performance, and their day-to-day variations remains poorly understood. Our year-long study of 20 healthy individuals, using subcutaneous electroencephalography, aimed to elucidate these interactions, assessing data stability and participant satisfaction, usability, well-being and adherence. In the study, 25 participants were fitted with a minimally invasive subcutaneous electroencephalography lead, with 20 completing the year of subcutaneous electroencephalography recording. Signal stability was measured using covariance of variation. Participant satisfaction, usability and well-being were measured with questionnaires: Perceived Ease of Use questionnaire, System Usability Scale, Headache questionnaire, Major Depression Inventory, World Health Organization 5-item Well-Being Index, and interviews. The subcutaneous electroencephalography signals remained stable for the entire year, with an average participant adherence rate of 91%. Participants rated their satisfaction with the subcutaneous electroencephalography device as easy to use with minimal or no discomfort. The System Usability Scale score was high at 86.3 ± 10.1, and interviews highlighted that participants understood how to use the subcutaneous electroencephalography device and described a period of acclimatization to sleeping with the device. This study provides compelling evidence for the feasibility of longitudinal sleep monitoring during everyday life utilizing subcutaneous electroencephalography in healthy subjects, showcasing excellent signal stability, adherence and user experience. The amassed subcutaneous electroencephalography data constitutes the largest dataset of its kind, and is poised to significantly advance our understanding of day-to-day variations in normal sleep and provide key insights into subjective and objective sleep quality.

Validity of remote live stream video evaluation of the North Star Ambulatory Assessment in patients with Duchenne muscular dystrophy

Conducting functional assessments remotely can help alleviate the burden of in-person assessment on patients with Duchenne muscular dystrophy and their caregivers. The objective of this study was to evaluate whether scores from remote functional assessment of patients with Duchenne muscular dystrophy correspond to in-person scores on the same functional assessments. Remote live stream versus in-person scores on the North Star Ambulatory Assessment (including time [seconds] to complete the 10-meter walk/run and time to rise from the floor [supine to stand]) were assessed using statistical analyses, including intraclass correlation coefficient, and Pearson, Spearman, and Bland-Altman analyses. The remote and in-clinic assessments had to occur within 2 weeks of one another to be considered for this analysis. This analysis included patients with Duchenne muscular dystrophy, aged 4 to 7 years. Participants in this analysis received delandistrogene moxeparvovec (as part of SRP-9001-101 [Study 101; NCT03375164] or SRP-9001-102 [Study 102; NCT03769116]) or were randomized to receive placebo (in Part 1 of Study 102). This study evaluates score reproducibility between live stream remote scoring versus in-person functional assessments as determined by intraclass correlation coefficient, and Pearson, Spearman, and Bland-Altman analyses. The results showed that scores from remote functional assessment of patients with Duchenne muscular dystrophy strongly correlated with those obtained in person. These findings demonstrate congruence between live stream remote and in-person functional assessment and suggest that remote assessment has the potential to reduce the burden on a family by supplementing in-clinic visits.

Real-world evidence for coverage determination of treatments for rare diseases

Health technology assessment (HTA) decisions for pharmaceuticals are complex and evolving. New rare disease treatments are often approved more quickly through accelerated approval schemes, creating more uncertainties about clinical evidence and budget impact at the time of market entry. The use of real-world evidence (RWE), including early coverage with evidence development, has been suggested as a means to support HTA decisions for rare disease treatments. However, the collection and use of RWE poses substantial challenges. These challenges are compounded when considered in the context of treatments for rare diseases. In this paper, we describe the methodological challenges to developing and using prospective and retrospective RWE for HTA decisions, for rare diseases in particular. We focus attention on key elements of study design and analyses, including patient selection and recruitment, appropriate adjustment for confounding and other sources of bias, outcome selection, and data quality monitoring. We conclude by offering suggestions to help address some of the most vexing challenges. The role of RWE in coverage and pricing determination will grow. It is, therefore, necessary for researchers, manufacturers, HTA agencies, and payers to ensure that rigorous and appropriate scientific principles are followed when using RWE as part of decision-making.

Tele-Assessment of Functional Capacity: Validity, Intra- and Inter-rater Reliability

BACKGROUND

Functional capacity evaluation is a standardized tool that assesses work-related skills. Although there are different test batteries, the most frequently used one is Work Well Systems. This study aims to determine the validity and inter- and intra-rater reliability of remote implementation of functional capacity tests (repetitive reaching, lifting object overhead, and working overhead) in asymptomatic individuals.

METHODS

A total of 51 asymptomatic individuals were included in the study. Participants completed all tests both face-to-face and remotely. Remote assessment videos were rewatched by the same researcher and different researchers for intra- and inter-rater reliability. All processes were scored by two independent researchers.

RESULTS

Remotely performing repetitive reaching (intraclass correlation coefficient [ICC]: 0.85-0.92, p < .001), lifting object overhead (ICC: 0.98, p < .001), and working overhead (ICC: 0.88 p < .001) tests are valid and reliable.

DISCUSSION

Repetitive reaching, lifting an object overhead, and sustained overhead work tests in the Work Well Systems-Functional Capacity Evaluation test battery can be performed remotely through videoconferencing. Remotely evaluating these tests, which are especially important in work-related situations, may be important in pandemic conditions and hybrid working conitions.

Adherence type impacts completion rates of frequent mobile cognitive assessments among older adults with and without cognitive impairment

Background

Prior to a diagnosis of Alzheimer's disease, many individuals experience cognitive and behavioral fluctuations that are not detected during a single session of traditional neuropsychological assessment. Mobile applications now enable high-frequency cognitive data to be collected remotely, introducing new opportunities and challenges. Emerging evidence suggests cognitively impaired older adults are capable of completing mobile assessments frequently, but no study has observed whether completion rates vary by assessment frequency or adherence type.

Methods

Thirty-three older adults were recruited from the Boston University Alzheimer's Disease Research Center (mean age = 73.5 years; 27.3% cognitively impaired; 57.6% female; 81.8% White, 18.2% Black). Participants remotely downloaded and completed the DANA Brain Vital application on their own mobile devices throughout the study. The study schedule included seventeen assessments to be completed over the course of a year. Specific periods during which assessments were expected to be completed were defined as subsegments, while segments consisted of multiple subsegments. The first segment included three subsegments to be completed within one week, the second segment included weekly subsegments and spanned three weeks, and the third and fourth segments included monthly subsegments spanning five and six months, respectively. Three distinct adherence types - subsegment adherence, segment adherence, and cumulative adherence - were examined to determine how completion rates varied depending on assessment frequency and adherence type.

Results

Adherence type significantly impacted whether the completion rates declined. When utilizing subsegment adherence, the completion rate significantly declined (p = 0.05) during the fourth segment. However, when considering completion rates from the perspective of segment adherence, a decline in completion rate was not observed. Overall adherence rates increased as adherence parameters were broadened from subsegment adherence (60.6%) to segment adherence (78.8%), to cumulative adherence (90.9%).

Conclusions

Older adults, including those with cognitive impairment, are able to complete remote cognitive assessments at a high-frequency, but may not necessarily adhere to prescribed schedules.

Evaluating the utility of daily speech assessments for monitoring depression symptoms

Objective

Depression is a common mental health disorder and a major public health concern, significantly interfering with the lives of those affected. The complex clinical presentation of depression complicates symptom assessments. Day-to-day fluctuations of depression symptoms within an individual bring an additional barrier, since infrequent testing may not reveal symptom fluctuation. Digital measures such as speech can facilitate daily objective symptom evaluation. Here, we evaluated the effectiveness of daily speech assessment in characterizing speech fluctuations in the context of depression symptoms, which can be completed remotely, at a low cost and with relatively low administrative resources.

Methods

Community volunteers (N = 16) completed a daily speech assessment, using the Winterlight Speech App, and Patient Health Questionnaire-9 (PHQ-9) for 30 consecutive business days. We calculated 230 acoustic and 290 linguistic features from individual's speech and investigated their relationship to depression symptoms at the intra-individual level through repeated measures analyses.

Results

We observed that depression symptoms were linked to linguistic features, such as less frequent use of dominant and positive words. Greater depression symptomatology was also significantly correlated with acoustic features: reduced variability in speech intensity and increased jitter.

Conclusions

Our findings support the feasibility of using acoustic and linguistic features as a measure of depression symptoms and propose daily speech assessment as a tool for better characterization of symptom fluctuations.

Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study

Background

Story recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring.

Objective

This study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD.

Methods

The “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire.

Results

Of the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests.

Conclusions

The unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.

Deployment of an End-to-End Remote, Digitalized Clinical Study Protocol in COVID-19: Process Evaluation

Background

The SARS-CoV-2 (COVID-19) pandemic may accelerate the adoption of digital, decentralized clinical trials. Conceptual recommendations for digitalized and remote clinical studies and technology are available to enable digitalization. Fully remote studies may break down some of the participation barriers in traditional trials. However, they add logistical complexity and offer fewer opportunities to intervene following a technical failure or adverse event.

Objective

Our group designed an end-to-end digitalized clinical study protocol, using the Food and Drug Administration (FDA)–cleared Current Health (CH) remote monitoring platform to collect symptoms and continuous physiological data of individuals recently infected with COVID-19 in the community. The purpose of this work is to provide a detailed example of an end-to-end digitalized protocol implementation based on conceptual recommendations by describing the study setup in detail, evaluating its performance, and identifying points of success and failure.

Methods

Primary recruitment was via social media and word of mouth. Informed consent was obtained during a virtual appointment, and the CH-monitoring kit was shipped directly to the participants. The wearable continuously recorded pulse rate (PR), respiratory rate (RR), oxygen saturation (SpO₂), skin temperature, and step count, while a tablet administered symptom surveys. Data were transmitted in real time to the CH cloud-based platform and displayed in the web-based dashboard, with alerts to the study team if the wearable was not charged or worn. The study duration was up to 30 days. The time to recruit, screen, consent, set up equipment, and collect data was quantified, and advertising engagement was tracked with a web analytics service.

Results

Of 13 different study advertisements, 5 (38.5%) were live on social media at any one time. In total, 38 eligibility forms were completed, and 19 (50%) respondents met the eligibility criteria. Of these, 9 (47.4%) were contactable and 8 (88.9%) provided informed consent. Deployment times ranged from 22 to 110 hours, and participants set up the equipment and started transmitting vital signs within 7.6 (IQR 6.3-10) hours of delivery. The mean wearable adherence was 70% (SD 19%), and the mean daily survey adherence was 88% (SD 21%) for the 8 participants. Vital signs were in normal ranges during study participation, and symptoms decreased over time.

Conclusions

Evaluation of clinical study implementation is important to capture what works and what might need to be modified. A well-calibrated approach to online advertising and enrollment can remove barriers to recruitment and lower costs but remains the most challenging part of research. Equipment was effectively and promptly shipped to participants and removed the risk of illness transmission associated with in-person encounters during a pandemic. Wearable technology incorporating continuous, clinical-grade monitoring offered an unprecedented level of detail and ecological validity. However, study planning, relationship building, and troubleshooting are more complex in the remote setting. The relevance of a study to potential participants remains key to its success.

Recruitment & Retention in Remote Research: Learnings from a Large Decentralized Real-World Study (Preprint)

BACKGROUND

Smartphones are increasingly used in health research. They provide a continuous connection between participants and researchers to monitor long-term health trajectories of large populations at a fraction of the cost of traditional research studies. However, despite the potential of using smartphones in remote research, there is an urgent need to develop effective strategies to reach, recruit, and retain the target populations in a representative and equitable manner.

OBJECTIVE

We aimed to investigate the impact of combining different recruitment and incentive distribution approaches used in remote research on cohort characteristics and long-term retention. The real-world factors significantly impacting active and passive data collection were also evaluated.

METHODS

We conducted a secondary data analysis of participant recruitment and retention using data from a large remote observation study aimed at understanding real-world factors linked to cold, influenza, and the impact of traumatic brain injury on daily functioning. We conducted recruitment in 2 phases between March 15, 2020, and January 4, 2022. Over 10,000 smartphone owners in the United States were recruited to provide 12 weeks of daily surveys and smartphone-based passive-sensing data. Using multivariate statistics, we investigated the potential impact of different recruitment and incentive distribution approaches on cohort characteristics. Survival analysis was used to assess the effects of sociodemographic characteristics on participant retention across the 2 recruitment phases. Associations between passive data-sharing patterns and demographic characteristics of the cohort were evaluated using logistic regression.

RESULTS

We analyzed over 330,000 days of engagement data collected from 10,000 participants. Our key findings are as follows: first, the overall characteristics of participants recruited using digital advertisements on social media and news media differed significantly from those of participants recruited using crowdsourcing platforms (Prolific and Amazon Mechanical Turk; P<.001). Second, participant retention in the study varied significantly across study phases, recruitment sources, and socioeconomic and demographic factors (P<.001). Third, notable differences in passive data collection were associated with device type (Android vs iOS) and participants' sociodemographic characteristics. Black or African American participants were significantly less likely to share passive sensor data streams than non-Hispanic White participants (odds ratio 0.44-0.49, 95% CI 0.35-0.61; P<.001). Fourth, participants were more likely to adhere to baseline surveys if the surveys were administered immediately after enrollment. Fifth, technical glitches could significantly impact real-world data collection in remote settings, which can severely impact generation of reliable evidence.

CONCLUSIONS

Our findings highlight several factors, such as recruitment platforms, incentive distribution frequency, the timing of baseline surveys, device heterogeneity, and technical glitches in data collection infrastructure, that could impact remote long-term data collection. Combined together, these empirical findings could help inform best practices for monitoring anomalies during real-world data collection and for recruiting and retaining target populations in a representative and equitable manner.