Accuracy of 11 Wearable, Nearable, and Airable Consumer Sleep Trackers: Prospective Multicenter Validation Study

BACKGROUND

Consumer sleep trackers (CSTs) have gained significant popularity because they enable individuals to conveniently monitor and analyze their sleep. However, limited studies have comprehensively validated the performance of widely used CSTs. Our study therefore investigated popular CSTs based on various biosignals and algorithms by assessing the agreement with polysomnography.

OBJECTIVE

This study aimed to validate the accuracy of various types of CSTs through a comparison with in-lab polysomnography. Additionally, by including widely used CSTs and conducting a multicenter study with a large sample size, this study seeks to provide comprehensive insights into the performance and applicability of these CSTs for sleep monitoring in a hospital environment.

METHODS

The study analyzed 11 commercially available CSTs, including 5 wearables (Google Pixel Watch, Galaxy Watch 5, Fitbit Sense 2, Apple Watch 8, and Oura Ring 3), 3 nearables (Withings Sleep Tracking Mat, Google Nest Hub 2, and Amazon Halo Rise), and 3 airables (SleepRoutine, SleepScore, and Pillow). The 11 CSTs were divided into 2 groups, ensuring maximum inclusion while avoiding interference between the CSTs within each group. Each group (comprising 8 CSTs) was also compared via polysomnography.

RESULTS

The study enrolled 75 participants from a tertiary hospital and a primary sleep-specialized clinic in Korea. Across the 2 centers, we collected a total of 3890 hours of sleep sessions based on 11 CSTs, along with 543 hours of polysomnography recordings. Each CST sleep recording covered an average of 353 hours. We analyzed a total of 349,114 epochs from the 11 CSTs compared with polysomnography, where epoch-by-epoch agreement in sleep stage classification showed substantial performance variation. More specifically, the highest macro F1 score was 0.69, while the lowest macro F1 score was 0.26. Various sleep trackers exhibited diverse performances across sleep stages, with SleepRoutine excelling in the wake and rapid eye movement stages, and wearables like Google Pixel Watch and Fitbit Sense 2 showing superiority in the deep stage. There was a distinct trend in sleep measure estimation according to the type of device. Wearables showed high proportional bias in sleep efficiency, while nearables exhibited high proportional bias in sleep latency. Subgroup analyses of sleep trackers revealed variations in macro F1 scores based on factors, such as BMI, sleep efficiency, and apnea-hypopnea index, while the differences between male and female subgroups were minimal.

CONCLUSIONS

Our study showed that among the 11 CSTs examined, specific CSTs showed substantial agreement with polysomnography, indicating their potential application in sleep monitoring, while other CSTs were partially consistent with polysomnography. This study offers insights into the strengths of CSTs within the 3 different classes for individuals interested in wellness who wish to understand and proactively manage their own sleep.

Three Contactless Sleep Technologies Compared With Actigraphy and Polysomnography in a Heterogeneous Group of Older Men and Women in a Model of Mild Sleep Disturbance: Sleep Laboratory Study

BACKGROUND

Contactless sleep technologies (CSTs) hold promise for longitudinal, unobtrusive sleep monitoring in the community and at scale. They may be particularly useful in older populations wherein sleep disturbance, which may be indicative of the deterioration of physical and mental health, is highly prevalent. However, few CSTs have been evaluated in older people.

OBJECTIVE

This study evaluated the performance of 3 CSTs compared to polysomnography (PSG) and actigraphy in an older population.

METHODS

Overall, 35 older men and women (age: mean 70.8, SD 4.9 y; women: n=14, 40%), several of whom had comorbidities, including sleep apnea, participated in the study. Sleep was recorded simultaneously using a bedside radar (Somnofy [Vital Things]: n=17), 2 undermattress devices (Withings sleep analyzer [WSA; Withings Inc]: n=35; Emfit-QS [Emfit; Emfit Ltd]: n=17), PSG (n=35), and actigraphy (Actiwatch Spectrum [Philips Respironics]: n=18) during the first night in a 10-hour time-in-bed protocol conducted in a sleep laboratory. The devices were evaluated through performance metrics for summary measures and epoch-by-epoch classification. PSG served as the gold standard.

RESULTS

The protocol induced mild sleep disturbance with a mean sleep efficiency (SEFF) of 70.9% (SD 10.4%; range 52.27%-92.60%). All 3 CSTs overestimated the total sleep time (TST; bias: >90 min) and SEFF (bias: >13%) and underestimated wake after sleep onset (bias: >50 min). Sleep onset latency was accurately detected by the bedside radar (bias: <6 min) but overestimated by the undermattress devices (bias: >16 min). CSTs did not perform as well as actigraphy in estimating the all-night sleep summary measures. In an epoch-by-epoch concordance analysis, the bedside radar performed better in discriminating sleep versus wake (Matthew correlation coefficient [MCC]: mean 0.63, SD 0.12, 95% CI 0.57-0.69) than the undermattress devices (MCC of WSA: mean 0.41, SD 0.15, 95% CI 0.36-0.46; MCC of Emfit: mean 0.35, SD 0.16, 95% CI 0.26-0.43). The accuracy of identifying rapid eye movement and light sleep was poor across all CSTs, whereas deep sleep (ie, slow wave sleep) was predicted with moderate accuracy (MCC: >0.45) by both Somnofy and WSA. The deep sleep duration estimates of Somnofy correlated (r2=0.60; P<.01) with electroencephalography slow wave activity (0.75-4.5 Hz) derived from PSG, whereas for the undermattress devices, this correlation was not significant (WSA: r2=0.0096, P=.58; Emfit: r2=0.11, P=.21).

CONCLUSIONS

These CSTs overestimated the TST, and sleep stage prediction was unsatisfactory in this group of older people in whom SEFF was relatively low. Although it was previously shown that CSTs provide useful information on bed occupancy, which may be useful for particular use cases, the performance of these CSTs with respect to the TST and sleep stage estimation requires improvement before they can serve as an alternative to PSG in estimating most sleep variables in older individuals.

Contactless and longitudinal monitoring of nocturnal sleep and daytime naps in older men and women: a digital health technology evaluation study

STUDY OBJECTIVES

To compare the 24-hour sleep assessment capabilities of two contactless sleep technologies (CSTs) to actigraphy in community-dwelling older adults.

METHODS

We collected 7-14 days of data at home from 35 older adults (age: 65-83), some with medical conditions, using Withings Sleep Analyser (WSA, n = 29), Emfit QS (Emfit, n = 17), a standard actigraphy device (Actiwatch Spectrum [AWS, n = 34]), and a sleep diary (n = 35). We compared nocturnal and daytime sleep measures estimated by the CSTs and actigraphy without sleep diary information (AWS-A) against sleep-diary-assisted actigraphy (AWS|SD).

RESULTS

Compared to sleep diary, both CSTs accurately determined the timing of nocturnal sleep (intraclass correlation [ICC]: going to bed, getting out of bed, time in bed >0.75), whereas the accuracy of AWS-A was much lower. Compared to AWS|SD, the CSTs overestimated nocturnal total sleep time (WSA: +92.71 ± 81.16 minutes; Emfit: +101.47 ± 75.95 minutes) as did AWS-A (+46.95 ± 67.26 minutes). The CSTs overestimated sleep efficiency (WSA: +9.19% ± 14.26%; Emfit: +9.41% ± 11.05%), whereas AWS-A estimate (-2.38% ± 10.06%) was accurate. About 65% (n = 23) of participants reported daytime naps either in bed or elsewhere. About 90% in-bed nap periods were accurately determined by WSA while Emfit was less accurate. All three devices estimated 24-hour sleep duration with an error of ≈10% compared to the sleep diary.

CONCLUSIONS

CSTs accurately capture the timing of in-bed nocturnal sleep periods without the need for sleep diary information. However, improvements are needed in assessing parameters such as total sleep time, sleep efficiency, and naps before these CSTs can be fully utilized in field settings.

Pyjamas, Polysomnography and Professional Athletes: The Role of Sleep Tracking Technology in Sport

Technological advances in sleep monitoring have seen an explosion of devices used to gather important sleep metrics. These devices range from instrumented 'smart pyjamas' through to at-home polysomnography devices. Alongside these developments in sleep technologies, there have been concomitant increases in sleep monitoring in athletic populations, both in the research and in practical settings. The increase in sleep monitoring in sport is likely due to the increased knowledge of the importance of sleep in the recovery process and performance of an athlete, as well as the well-reported challenges that athletes can face with their sleep. This narrative review will discuss: (1) the importance of sleep to athletes; (2) the various wearable tools and technologies being used to monitor sleep in the sport setting; (3) the role that sleep tracking devices may play in gathering information about sleep; (4) the reliability and validity of sleep tracking devices; (5) the limitations and cautions associated with sleep trackers; and, (6) the use of sleep trackers to guide behaviour change in athletes. We also provide some practical recommendations for practitioners working with athletes to ensure that the selection of such devices and technology will meet the goals and requirements of the athlete.

Performance of seven consumer sleep-tracking devices compared with polysomnography

STUDY OBJECTIVES

Consumer sleep-tracking devices are widely used and becoming more technologically advanced, creating strong interest from researchers and clinicians for their possible use as alternatives to standard actigraphy. We, therefore, tested the performance of many of the latest consumer sleep-tracking devices, alongside actigraphy, versus the gold-standard sleep assessment technique, polysomnography (PSG).

METHODS

In total, 34 healthy young adults (22 women; 28.1 ± 3.9 years, mean ± SD) were tested on three consecutive nights (including a disrupted sleep condition) in a sleep laboratory with PSG, along with actigraphy (Philips Respironics Actiwatch 2) and a subset of consumer sleep-tracking devices. Altogether, four wearable (Fatigue Science Readiband, Fitbit Alta HR, Garmin Fenix 5S, Garmin Vivosmart 3) and three nonwearable (EarlySense Live, ResMed S+, SleepScore Max) devices were tested. Sleep/wake summary and epoch-by-epoch agreement measures were compared with PSG.

RESULTS

Most devices (Fatigue Science Readiband, Fitbit Alta HR, EarlySense Live, ResMed S+, SleepScore Max) performed as well as or better than actigraphy on sleep/wake performance measures, while the Garmin devices performed worse. Overall, epoch-by-epoch sensitivity was high (all ≥0.93), specificity was low-to-medium (0.18-0.54), sleep stage comparisons were mixed, and devices tended to perform worse on nights with poorer/disrupted sleep.

CONCLUSIONS

Consumer sleep-tracking devices exhibited high performance in detecting sleep, and most performed equivalent to (or better than) actigraphy in detecting wake. Device sleep stage assessments were inconsistent. Findings indicate that many newer sleep-tracking devices demonstrate promising performance for tracking sleep and wake. Devices should be tested in different populations and settings to further examine their wider validity and utility.