PsyCog: A computerised mini battery for assessing cognition in psychosis

Despite the functional impact of cognitive deficit in people with psychosis, objective cognitive assessment is not typically part of routine clinical care. This is partly due to the length of traditional assessments and the need for a highly trained administrator. Brief, automated computerised assessments could help to address this issue. We present data from an evaluation of PsyCog, a computerised, non-verbal, mini battery of cognitive tests. Healthy Control (HC) (N = 135), Clinical High Risk (CHR) (N = 233), and First Episode Psychosis (FEP) (N = 301) participants from a multi-centre prospective study were assessed at baseline, 6 months, and 12 months. PsyCog was used to assess cognitive performance at baseline and at up to two follow-up timepoints. Mean total testing time was 35.95 min (SD = 2.87). Relative to HCs, effect sizes of performance impairments were medium to large in FEP patients (composite score G = 1.21, subtest range = 0.52-0.88) and small to medium in CHR patients (composite score G = 0.59, subtest range = 0.18-0.49). Site effects were minimal, and test-retest reliability of the PsyCog composite was good (ICC = 0.82-0.89), though some practice effects and differences in data completion between groups were found. The present implementation of PsyCog shows it to be a useful tool for assessing cognitive function in people with psychosis. Computerised cognitive assessments have the potential to facilitate the evaluation of cognition in psychosis in both research and in clinical care, though caution should still be taken in terms of implementation and study design.

The Ultra-Long-Term Sleep study: Design, rationale, data stability and user perspective

Sleep deprivation and poor sleep quality are significant societal challenges that negatively impact individuals' health. The interaction between subjective sleep quality, objective sleep measures, physical and cognitive performance, and their day-to-day variations remains poorly understood. Our year-long study of 20 healthy individuals, using subcutaneous electroencephalography, aimed to elucidate these interactions, assessing data stability and participant satisfaction, usability, well-being and adherence. In the study, 25 participants were fitted with a minimally invasive subcutaneous electroencephalography lead, with 20 completing the year of subcutaneous electroencephalography recording. Signal stability was measured using covariance of variation. Participant satisfaction, usability and well-being were measured with questionnaires: Perceived Ease of Use questionnaire, System Usability Scale, Headache questionnaire, Major Depression Inventory, World Health Organization 5-item Well-Being Index, and interviews. The subcutaneous electroencephalography signals remained stable for the entire year, with an average participant adherence rate of 91%. Participants rated their satisfaction with the subcutaneous electroencephalography device as easy to use with minimal or no discomfort. The System Usability Scale score was high at 86.3 ± 10.1, and interviews highlighted that participants understood how to use the subcutaneous electroencephalography device and described a period of acclimatization to sleeping with the device. This study provides compelling evidence for the feasibility of longitudinal sleep monitoring during everyday life utilizing subcutaneous electroencephalography in healthy subjects, showcasing excellent signal stability, adherence and user experience. The amassed subcutaneous electroencephalography data constitutes the largest dataset of its kind, and is poised to significantly advance our understanding of day-to-day variations in normal sleep and provide key insights into subjective and objective sleep quality.

Prediction of mental effort derived from an automated vocal biomarker using machine learning in a large-scale remote sample

Introduction

Biomarkers of mental effort may help to identify subtle cognitive impairments in the absence of task performance deficits. Here, we aim to detect mental effort on a verbal task, using automated voice analysis and machine learning.

Methods

Audio data from the digit span backwards task were recorded and scored with automated speech recognition using the online platform NeuroVocalixTM, yielding usable data from 2,764 healthy adults (1,022 male, 1,742 female; mean age 31.4 years). Acoustic features were aggregated across each trial and normalized within each subject. Cognitive load was dichotomized for each trial by categorizing trials at >0.6 of each participants' maximum span as "high load." Data were divided into training (60%), test (20%), and validate (20%) datasets, each containing different participants. Training and test data were used in model building and hyper-parameter tuning. Five classification models (Logistic Regression, Naive Bayes, Support Vector Machine, Random Forest, and Gradient Boosting) were trained to predict cognitive load ("high" vs. "low") based on acoustic features. Analyses were limited to correct responses. The model was evaluated using the validation dataset, across all span lengths and within the subset of trials with a four-digit span. Classifier discriminant power was examined with Receiver Operating Curve (ROC) analysis.

Results

Participants reached a mean span of 6.34 out of 8 items (SD = 1.38). The Gradient Boosting classifier provided the best performing model on test data (AUC = 0.98) and showed excellent discriminant power for cognitive load on the validation dataset, across all span lengths (AUC = 0.99), and for four-digit only utterances (AUC = 0.95).

Discussion

A sensitive biomarker of mental effort can be derived from vocal acoustic features in remotely administered verbal cognitive tests. The use-case of this biomarker for improving sensitivity of cognitive tests to subtle pathology now needs to be examined.

Reverse Engineering of Digital Measures: Inviting Patients to the Conversation

Background

Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures.

Summary

In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled "Reverse Engineering of Digital Measures," was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools.

Key Messages

In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.

Description of the Method for Evaluating Digital Endpoints in Alzheimer Disease Study: Protocol for an Exploratory, Cross-sectional Study

Background

More sensitive and less burdensome efficacy end points are urgently needed to improve the effectiveness of clinical drug development for Alzheimer disease (AD). Although conventional end points lack sensitivity, digital technologies hold promise for amplifying the detection of treatment signals and capturing cognitive anomalies at earlier disease stages. Using digital technologies and combining several test modalities allow for the collection of richer information about cognitive and functional status, which is not ascertainable via conventional paper-and-pencil tests.

Objective

This study aimed to assess the psychometric properties, operational feasibility, and patient acceptance of 10 promising technologies that are to be used as efficacy end points to measure cognition in future clinical drug trials.

Methods

The Method for Evaluating Digital Endpoints in Alzheimer Disease study is an exploratory, cross-sectional, noninterventional study that will evaluate 10 digital technologies’ ability to accurately classify participants into 4 cohorts according to the severity of cognitive impairment and dementia. Moreover, this study will assess the psychometric properties of each of the tested digital technologies, including the acceptable range to assess ceiling and floor effects, concurrent validity to correlate digital outcome measures to traditional paper-and-pencil tests in AD, reliability to compare test and retest, and responsiveness to evaluate the sensitivity to change in a mild cognitive challenge model. This study included 50 eligible male and female participants (aged between 60 and 80 years), of whom 13 (26%) were amyloid-negative, cognitively healthy participants (controls); 12 (24%) were amyloid-positive, cognitively healthy participants (presymptomatic); 13 (26%) had mild cognitive impairment (predementia); and 12 (24%) had mild AD (mild dementia). This study involved 4 in-clinic visits. During the initial visit, all participants completed all conventional paper-and-pencil assessments. During the following 3 visits, the participants underwent a series of novel digital assessments.

Results

Participant recruitment and data collection began in June 2020 and continued until June 2021. Hence, the data collection occurred during the COVID-19 pandemic (SARS-CoV-2 virus pandemic). Data were successfully collected from all digital technologies to evaluate statistical and operational performance and patient acceptance. This paper reports the baseline demographics and characteristics of the population studied as well as the study's progress during the pandemic.

Conclusions

This study was designed to generate feasibility insights and validation data to help advance novel digital technologies in clinical drug development. The learnings from this study will help guide future methods for assessing novel digital technologies and inform clinical drug trials in early AD, aiming to enhance clinical end point strategies with digital technologies.

International Registered Report Identifier (IRRID)

DERR1-10.2196/35442

0020 Investigation of Seasonal Changes in Self-Reported Sleep Quality and Psychomotor Vigilance Task Outcomes: Results From the Ultra Long-Term Sleep (ULTS) Study

Introduction

Although sleep is fundamental for human well-being, factors that contribute to an individual’s experience and report of sleep quality remain poorly understood. Utilizing that sleepiness is known to impact vigilance performance, this study sets out to explore how self-reported sleep quality changes with behavioral performance and how this variation is affected by seasonal changes.

Methods

This work is an interim analysis of self-reported sleep quality and behavioral performance data collected in the Ultra Long-term Sleep (ULTS) study (ClinicalTrials.gov Identifier: NCT04513743). In the study 20 healthy participants (average 33±13 years of age) were enrolled for 365 continuous days to observe the seasonal variation in sleep and cognitive performance. The outcome from the daily psychomotor vigilance task (PVT) and sleep questionnaire is analyzed and reported. The sleep questionnaire is a composite of questions from the Sleep Satisfaction Tool, the Karolinska Sleep Diary, questions regarding feelings of pain and outside disturbances, easiness of waking up, and the Karolinska Sleepiness Scale (KSS). The PVT was designed for self-administered high-frequency testing and has a short 3-minute test period. Monthly changes were examined from June to November. This period was chosen since all subjects were active.

Results

The first eight participants have now completed the study. Repeated measures correlations between KSS and mean PVT reaction time showed moderate but highly robust associations between the two measures over time (r=0.2; 95% CI 0.18-0.23). From the data collected thus far from the entire population, fastest PVT mean reaction times were found on Saturdays and slowest on Tuesdays. Similarly, KSS had best scores in weekends. There was an overall increase in mean PVT reaction time during the investigated period from June to November. This was also observed in KSS.

Conclusion

Our findings show a moderate correlation between the mean PVT reaction time and Karolinska Sleepiness Scale with up to 365 datapoints per subject with weekly and seasonal trends observed.

Support (If Any)

This research is supported by Innovation Fund Denmark.

Correction: An Alternative to the Light Touch Digital Health Remote Study: The Stress and Recovery in Frontline COVID-19 Health Care Workers Study

[This corrects the article DOI: 10.2196/32165.].

Better Understanding of the Metamorphosis of Pregnancy (BUMP): protocol for a digital feasibility study in women from preconception to postpartum

The Better Understanding the Metamorphosis of Pregnancy (BUMP) study is a longitudinal feasibility study aimed to gain a deeper understanding of the pre-pregnancy and pregnancy symptom experience using digital tools. The present paper describes the protocol for the BUMP study. Over 1000 participants are being recruited through a patient provider-platform and through other channels in the United States (US). Participants in a preconception cohort (BUMP-C) are followed for 6 months, or until conception, while participants in a pregnancy cohort (BUMP) are followed into their fourth trimester. Participants are provided with a smart ring, a smartwatch (BUMP only), and a smart scale (BUMP only) alongside cohort-specific study apps. Participant centric engagement strategies are used that aim to co-design the digital approach with participants while providing knowledge and support. The BUMP study is intended to lay the foundational work for a larger study to determine whether participant co-designed digital tools can be used to detect, track and return multimodal symptoms during the perinatal window to inform individual level symptom trajectories.

Acoustic features of voice as a measure of cognitive load during performance of serial subtraction in a remote data collection context

BACKGROUND

Cognitive load is the mental demand a task imposes for a specific person. Performance declines when demand exceed capacity; therefore, increase of mental effort may precede measurable cognitive decline. Physiological indices of load (e.g. heart rate, skin conductance etc.) are sensitive to task demand (e.g. subtracting three vs seven), show increased cognitive load with ageing, and in MCI compared to healthy ageing. Voice features have promise as non-invasive and scalable indicators of mental effort. Here, we aim to classify serial subtraction at high and low cognitive load using voice recordings captured using an automated remote data collection system.

METHOD

Participants (aged 17-86) completed serial subtraction via the Neurovocalix web-app on their own devices. From a pool of 5,742 participants, 100 were randomly selected for manual review. Seven participants were excluded for audio or performance issues. Responses were transcribed and the start and end of each subtraction attempt marked, producing 3,254 attempts for analysis. Low-level acoustic features were extracted and aggregated over each attempt, then normalized within participant. Random Forest classifiers were trained and evaluated using Leave-One-Subject-Out-Cross-Validation (LOSOCV) to predict high vs low load. LOSOCV repeatedly splits the dataset by subject, with one participant at a time used for testing, and the remainder used for training the model. This produces model predictions for each participant and attempt.

RESULT

Average cross-validation accuracy was 0.81 (95% CI 0.78 to 0.84), with an average area under the curve (AUC) of 0.87 (95% CI 0.85 to 0.89). We tested predictions for specific numbers which appeared in both subtraction by seven and by three. Accuracy was 0.78, suggesting that predictions were not driven by specific numeric responses. We observed a significant negative correlation between behavioural performance on the task (response rate), and utterance load probability metric for utterances (ρ=-0.32, p<0.001), suggesting that participants who were more fluent in serial subtraction exhibited lower cognitive load.

CONCLUSION

Acoustic features of voice can distinguish between utterances generated under conditions of high and low cognitive load during serial subtraction, adding a novel, independent and sensitive outcome measure to a cognitive task with established utility in the context of neurodegeneration.

An Alternative to the Light Touch Digital Health Remote Study: The Stress and Recovery in Frontline COVID-19 Health Care Workers Study

Background

Several app-based studies share similar characteristics of a light touch approach that recruit, enroll, and onboard via a smartphone app and attempt to minimize burden through low-friction active study tasks while emphasizing the collection of passive data with minimal human contact. However, engagement is a common challenge across these studies, reporting low retention and adherence.

Objective

This study aims to describe an alternative to a light touch digital health study that involved a participant-centric design including high friction app-based assessments, semicontinuous passive data from wearable sensors, and a digital engagement strategy centered on providing knowledge and support to participants.

Methods

The Stress and Recovery in Frontline COVID-19 Health Care Workers Study included US frontline health care workers followed between May and November 2020. The study comprised 3 main components: (1) active and passive assessments of stress and symptoms from a smartphone app, (2) objective measured assessments of acute stress from wearable sensors, and (3) a participant codriven engagement strategy that centered on providing knowledge and support to participants. The daily participant time commitment was an average of 10 to 15 minutes. Retention and adherence are described both quantitatively and qualitatively.

Results

A total of 365 participants enrolled and started the study, and 81.0% (n=297) of them completed the study for a total study duration of 4 months. Average wearable sensor use was 90.6% days of total study duration. App-based daily, weekly, and every other week surveys were completed on average 69.18%, 68.37%, and 72.86% of the time, respectively.

Conclusions

This study found evidence for the feasibility and acceptability of a participant-centric digital health study approach that involved building trust with participants and providing support through regular phone check-ins. In addition to high retention and adherence, the collection of large volumes of objective measured data alongside contextual self-reported subjective data was able to be collected, which is often missing from light touch digital health studies.

Trial Registration

ClinicalTrials.gov NCT04713111; https://clinicaltrials.gov/ct2/show/NCT04713111

Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

The ability of remote research tools to collect granular, high-frequency data on symptoms and digital biomarkers is an important strength because it circumvents many limitations of traditional clinical trials and improves the ability to capture clinically relevant data. This approach allows researchers to capture more robust baselines and derive novel phenotypes for improved precision in diagnosis and accuracy in outcomes. The process for developing these tools however is complex because data need to be collected at a frequency that is meaningful but not burdensome for the participant or patient. Furthermore, traditional techniques, which rely on fixed conditions to validate assessments, may be inappropriate for validating tools that are designed to capture data under flexible conditions. This paper discusses the process for determining whether a digital assessment is suitable for remote research and offers suggestions on how to validate these novel tools.

Wearable Technology for High-Frequency Cognitive and Mood Assessment in Major Depressive Disorder: Longitudinal Observational Study

BACKGROUND

Cognitive symptoms are common in major depressive disorder and may help to identify patients who need treatment or who are not experiencing adequate treatment response. Digital tools providing real-time data assessing cognitive function could help support patient treatment and remediation of cognitive and mood symptoms.

OBJECTIVE

The aim of this study was to examine feasibility and validity of a wearable high-frequency cognitive and mood assessment app over 6 weeks, corresponding to when antidepressant pharmacotherapy begins to show efficacy.

METHODS

A total of 30 patients (aged 19-63 years; 19 women) with mild-to-moderate depression participated in the study. The new Cognition Kit app was delivered via the Apple Watch, providing a high-resolution touch screen display for task presentation and logging responses. Cognition was assessed by the n-back task up to 3 times daily and depressed mood by 3 short questions once daily. Adherence was defined as participants completing at least 1 assessment daily. Selected tests sensitive to depression from the Cambridge Neuropsychological Test Automated Battery and validated questionnaires of depression symptom severity were administered on 3 occasions (weeks 1, 3, and 6). Exploratory analyses examined the relationship between mood and cognitive measures acquired in low- and high-frequency assessment.

RESULTS

Adherence was excellent for mood and cognitive assessments (95% and 96%, respectively), did not deteriorate over time, and was not influenced by depression symptom severity or cognitive function at study onset. Analyses examining the relationship between high-frequency cognitive and mood assessment and validated measures showed good correspondence. Daily mood assessments correlated moderately with validated depression questionnaires (r=0.45-0.69 for total daily mood score), and daily cognitive assessments correlated moderately with validated cognitive tests sensitive to depression (r=0.37-0.50 for mean n-back).

CONCLUSIONS

This study supports the feasibility and validity of high-frequency assessment of cognition and mood using wearable devices over an extended period in patients with major depressive disorder.