Agreement in computer-assisted manual scoring of polysomnograms across sleep centers

An approach for determining the reliability of manual and digital scoring of sleep stages

STUDY OBJECTIVES

Inter-scorer variability in sleep staging is largely due to equivocal epochs that contain features of more than one stage. We propose an approach that recognizes the existence of equivocal epochs and evaluates scorers accordingly.

METHODS

Epoch-by-epoch staging was performed on 70 polysomnograms by six qualified technologists and by a digital system (Michele Sleep Scoring [MSS]). Probability that epochs assigned the same stage by only two of the six technologists (minority score) resulted from random occurrence of two errors was calculated and found to be <5%, thereby indicating that the stage assigned is an acceptable variant for the epoch. Acceptable stages were identified in each epoch as stages assigned by at least two technologists. Percent agreement between each technologist and the other five technologists, acting as judges, was determined. Agreement was considered to exist if the stage assigned by the tested scorer was one of the acceptable stages for the epoch. Stage assigned by MSS was likewise considered in agreement if included in the acceptable stages made by the technologists.

RESULTS

Agreement of technologists tested against five qualified judges increased from 80.8% (range 70.5%-86.4% among technologists) when using the majority rule, to 96.1 (89.8%-98.5%) by the proposed approach. Agreement between unedited MSS and same judges was 90.0% and increased to 92.1% after brief editing.

CONCLUSIONS

Accounting for equivocal epochs provides a more accurate estimate of a scorer's (human or digital) competence in scoring sleep stages and reduces inter-scorer disagreements. The proposed approach can be implemented in sleep-scoring training and accreditation programs.

Modeling the Impact of Inter-Rater Disagreement on Sleep Statistics Using Deep Generative Learning

Sleep staging is the process by which an overnight polysomnographic measurement is segmented into epochs of 30 seconds, each of which is annotated as belonging to one of five discrete sleep stages. The resulting scoring is graphically depicted as a hypnogram, and several overnight sleep statistics are derived, such as total sleep time and sleep onset latency. Gold standard sleep staging as performed by human technicians is time-consuming, costly, and comes with imperfect inter-scorer agreement, which also results in inter-scorer disagreement about the overnight statistics. Deep learning algorithms have shown promise in automating sleep scoring, but struggle to model inter-scorer disagreement in sleep statistics. To that end, we introduce a novel technique using conditional generative models based on Normalizing Flows that permits the modeling of the inter-rater disagreement of overnight sleep statistics, termed U-Flow. We compare U-Flow to other automatic scoring methods on a hold-out test set of 70 subjects, each scored by six independent scorers. The proposed method achieves similar sleep staging performance in terms of accuracy and Cohen's kappa on the majority-voted hypnograms. At the same time, U-Flow outperforms the other methods in terms of modeling the inter-rater disagreement of overnight sleep statistics. The consequences of inter-rater disagreement about overnight sleep statistics may be great, and the disagreement potentially carries diagnostic and scientifically relevant information about sleep structure. U-Flow is able to model this disagreement efficiently and can support further investigations into the impact inter-rater disagreement has on sleep medicine and basic sleep research.

The short-form of the Cyberchondria Severity Scale (CSS-12): Adaptation and validation of the Spanish version in young Peruvian students

BACKGROUND

Cyberchondria is defined as the increase in health-related anxiety or anguish associated with excessive or repeated online searches for health-related information. Our objective was to cross-culturally adapt and validate the CSS-12 scale for Peruvian Spanish speakers, to determine whether the Bifactor model works as well in our population as in previous studies' and to explore whether the Bifactor-ESEM is a more suitable model.

METHODS

We performed a cultural adaptation using the Delphi method and a validation study on medical students between 2018 and 2019. Reliability was evaluated by using Cronbach's alpha (α) and McDonald's omega (Ω) for internal consistency, and Pearson's r and intraclass correlation coefficient (ICC), for test-retest reliability. We evaluated construct validity by contrasting four measurement models for the CSS-12 and the convergent validity against health anxiety.

RESULTS

The Spanish CSS-12 showed excellent reliability (α = .93; Ω = .93; ICC = .93; r = .96). The Bifactor ESEM model showed the best fit, supporting a unidimensional measure of the general cyberchondria. This measure was positively associated with health anxiety (r = .51).

CONCLUSIONS

The Spanish CSS-12 provides a valid and reliable unidimensional measure of cyberchondria, which is distinguishable from the more general health anxiety. This can be applied to similar populations and future research. The Bifactor-ESEM model appears to offer a more accurate and realistic representation of the multifaceted nature of cyberchondria. We provide a free-to-use form of the Spanish CSS-12 as supplemental material.

Challenges of Applying Automated Polysomnography Scoring at Scale

Automatic polysomnography analysis can be leveraged to shorten scoring times, reduce associated costs, and ultimately improve the overall diagnosis of sleep disorders. Multiple and diverse strategies have been attempted for implementation of this technology at scale in the routine workflow of sleep centers. The field, however, is complex and presents unsolved challenges in a number of areas. Recent developments in computer science and artificial intelligence are nevertheless closing the gap. Technological advances are also opening new pathways for expanding our current understanding of the domain and its analysis.

A vital sign signal noise suppression method for wearable piezoelectric devices

This paper tackles the problem of noise suppression during vital sign signal monitoring. Physiological signal monitoring is a significant and promising medical monitoring method, and wearable medical monitoring devices based on piezoelectric polymer sensors are a trending way for their advantages of being flexible in the shape, portable to use, and comfortable to wear. However, this raises the question that the measured signal contains much more noise components. To avoid the following shortcoming of low signal to noise ratio (SNR), a noise suppression method based on improved wavelet threshold and empirical mode decomposition combined with singular value decomposition (SVD) screening the intrinsic mode function (IMF) components is proposed. A wavelet transform is first used under the combination of hard and soft thresholds to focus the target range in the low-frequency region where the energy of the physiological signal is concentrated. Then, a complete ensemble empirical mode decomposition is used to decompose the signal effectively, which can resist the influence of random noises. Meanwhile, a SVD decomposition procedure was used to filter out the lower correlated IMF components to retain the validity of the original signal. We verified the effectiveness of the proposed method through simulated and measured experiments as well as the advantages and disadvantages of the algorithm compared with other physiological signal denoising algorithms through SNR filtering results, power spectrum distribution, and other perspectives. The results proved that the proposed method could effectively remove more detailed noise and improve the SNR of the signal efficiently, which is more conducive to the demand for auxiliary medical diagnosis in the future.

Autonomic arousal detection and cardio-respiratory sleep staging improve the accuracy of home sleep apnea tests

Introduction: The apnea-hypopnea index (AHI), defined as the number of apneas and hypopneas per hour of sleep, is still used as an important index to assess sleep disordered breathing (SDB) severity, where hypopneas are confirmed by the presence of an oxygen desaturation or an arousal. Ambulatory polygraphy without neurological signals, often referred to as home sleep apnea testing (HSAT), can potentially underestimate the severity of sleep disordered breathing (SDB) as sleep and arousals are not assessed. We aim to improve the diagnostic accuracy of HSATs by extracting surrogate sleep and arousal information derived from autonomic nervous system activity with artificial intelligence. Methods: We used polysomnographic (PSG) recordings from 245 subjects (148 with simultaneously recorded HSATs) to develop and validate a new algorithm to detect autonomic arousals using artificial intelligence. A clinically validated auto-scoring algorithm (Somnolyzer) scored respiratory events, cortical arousals, and sleep stages in PSGs, and provided respiratory events and sleep stages from cardio-respiratory signals in HSATs. In a four-fold cross validation of the newly developed algorithm, we evaluated the accuracy of the estimated arousal index and HSAT-derived surrogates for the AHI. Results: The agreement between the autonomic and cortical arousal index was moderate to good with an intraclass correlation coefficient of 0.73. When using thresholds of 5, 15, and 30 to categorize SDB into none, mild, moderate, and severe, the addition of sleep and arousal information significantly improved the classification accuracy from 70.2% (Cohen's κ = 0.58) to 80.4% (κ = 0.72), with a significant reduction of patients where the severity category was underestimated from 18.8% to 7.3%. Discussion: Extracting sleep and arousal information from autonomic nervous system activity can improve the diagnostic accuracy of HSATs by significantly reducing the probability of underestimating SDB severity without compromising specificity.

The Use of Respiratory Effort Improves an ECG-Based Deep Learning Algorithm to Assess Sleep-Disordered Breathing

BACKGROUND

Sleep apnea is a prevalent sleep-disordered breathing (SDB) condition that affects a large population worldwide. Research has demonstrated the potential of using electrocardiographic (ECG) signals (heart rate and ECG-derived respiration, EDR) to detect SDB. However, EDR may be a suboptimal replacement for respiration signals.

METHODS

We evaluated a previously described ECG-based deep learning algorithm in an independent dataset including 198 patients and compared performance for SDB event detection using thoracic respiratory effort versus EDR. We also evaluated the algorithm in terms of apnea-hypopnea index (AHI) estimation performance, and SDB severity classification based on the estimated AHI.

RESULTS

Using respiratory effort instead of EDR, we achieved an improved performance in SDB event detection (F1 score = 0.708), AHI estimation (Spearman's correlation = 0.922), and SDB severity classification (Cohen's kappa of 0.62 was obtained based on AHI).

CONCLUSION

Respiratory effort is superior to EDR to assess SDB. Using respiratory effort and ECG, the previously described algorithm achieves good performance in a new dataset from an independent laboratory confirming its adequacy for this task.

Validity study of a multiscaled fusion network using single-lead electrocardiogram signals for obstructive sleep apnea diagnosis

STUDY OBJECTIVES

We evaluated the validity of a squeeze-and-excitation and multiscaled fusion network (SE-MSCNN) using single-lead electrocardiogram (ECG) signals for obstructive sleep apnea detection and classification.

METHODS

Overnight polysomnographic data from 436 participants at the Sleep Center of the First Affiliated Hospital of Sun Yat-sen University were used to generate a new FAH-ECG dataset comprising 260, 88, and 88 single-lead ECG signal recordings for training, validation, and testing, respectively. The SE-MSCNN was employed for detection of apnea-hypopnea events from the acquired ECG segments. Sensitivity, specificity, accuracy, and F1 scores were assigned to assess algorithm performance. We also used the SE-MSCNN to estimate the apnea-hypopnea index, classify obstructive sleep apnea severity, and compare the agreement between 2 sleep technicians.

RESULTS

The SE-MSCNN's accuracy, sensitivity, specificity, and F1 score on the FAH-ECG dataset were 86.6%, 83.3%, 89.1%, and 0.843, respectively. Although slightly inferior to previously reported results using public datasets, it is superior to state-of-the-art open-source models. Furthermore, the SE-MSCNN had good agreement with manual scoring, such that the Spearman's correlations for the apnea-hypopnea index between the SE-MSCNN and 2 technicians were 0.93 and 0.94, respectively. Cohen's kappa scores in classifying the SE-MSCNN and the 2 sleep technicians were 0.72 and 0.78, respectively.

CONCLUSIONS

In this study, we validated the use of the SE-MSCNN in a clinical environment, and despite some limitations the network appeared to meet the performance standards for generalizability. Therefore, updating algorithms based on single-lead ECG signals can facilitate the development of novel wearable devices for efficient obstructive sleep apnea screening.

CITATION

Yue H, Li P, Li Y, et al. Validity study of a multiscaled fusion network using single-lead electrocardiogram signals for obstructive sleep apnea diagnosis. J Clin Sleep Med. 2023;19(6):1017-1025.

Consumer sleep technology for the screening of obstructive sleep apnea and snoring: current status and a protocol for a systematic review and meta-analysis of diagnostic test accuracy

There are concerns about the validation and accuracy of currently available consumer sleep technology for sleep-disordered breathing. The present report provides a background review of existing consumer sleep technologies and discloses the methods and procedures for a systematic review and meta-analysis of diagnostic test accuracy of these devices and apps for the detection of obstructive sleep apnea and snoring in comparison with polysomnography. The search will be performed in four databases (PubMed, Scopus, Web of Science, and the Cochrane Library). Studies will be selected in two steps, first by an analysis of abstracts followed by full-text analysis, and two independent reviewers will perform both phases. Primary outcomes include apnea-hypopnea index, respiratory disturbance index, respiratory event index, oxygen desaturation index, and snoring duration for both index and reference tests, as well as the number of true positives, false positives, true negatives, and false negatives for each threshold, as well as for epoch-by-epoch and event-by-event results, which will be considered for the calculation of surrogate measures (including sensitivity, specificity, and accuracy). Diagnostic test accuracy meta-analyses will be performed using the Chu and Cole bivariate binomial model. Mean difference meta-analysis will be performed for continuous outcomes using the DerSimonian and Laird random-effects model. Analyses will be performed independently for each outcome. Subgroup and sensitivity analyses will evaluate the effects of the types (wearables, nearables, bed sensors, smartphone applications), technologies (e.g., oximeter, microphone, arterial tonometry, accelerometer), the role of manufacturers, and the representativeness of the samples.

Scoring sleep with artificial intelligence enables quantification of sleep stage ambiguity: hypnodensity based on multiple expert scorers and auto-scoring

STUDY OBJECTIVES

To quantify the amount of sleep stage ambiguity across expert scorers and to validate a new auto-scoring platform against sleep staging performed by multiple scorers.

METHODS

We applied a new auto-scoring system to three datasets containing 95 PSGs scored by 6-12 scorers, to compare sleep stage probabilities (hypnodensity; i.e. the probability of each sleep stage being assigned to a given epoch) as the primary output, as well as a single sleep stage per epoch assigned by hierarchical majority rule.

RESULTS

The percentage of epochs with 100% agreement across scorers was 46 ± 9%, 38 ± 10% and 32 ± 9% for the datasets with 6, 9, and 12 scorers, respectively. The mean intra-class correlation coefficient between sleep stage probabilities from auto- and manual-scoring was 0.91, representing excellent reliability. Within each dataset, agreement between auto-scoring and consensus manual-scoring was significantly higher than agreement between manual-scoring and consensus manual-scoring (0.78 vs. 0.69; 0.74 vs. 0.67; and 0.75 vs. 0.67; all p < 0.01).

CONCLUSIONS

Analysis of scoring performed by multiple scorers reveals that sleep stage ambiguity is the rule rather than the exception. Probabilities of the sleep stages determined by artificial intelligence auto-scoring provide an excellent estimate of this ambiguity. Compared to consensus manual-scoring, sleep staging derived from auto-scoring is for each individual PSG noninferior to manual-scoring meaning that auto-scoring output is ready for interpretation without the need for manual adjustment.

Single-Night Diagnosis of Sleep Apnea Contributes to Inconsistent Cardiovascular Outcome Findings

BACKGROUND

Single-night disease misclassification of OSA due to night-to-night variability may contribute to inconsistent findings in OSA trials.

RESEARCH QUESTION

Does multinight quantification of OSA severity provide more precise estimates of associations with incident hypertension?

STUDY DESIGN AND METHODS

A total of 3,831 participants without hypertension at baseline were included in simulation analyses. Included participants had ≥ 28 days of nightly apnea-hypopnea index (AHI) recordings via an under-mattress sensor and ≥ 3 separate BP measurements over a 3-month baseline period followed by ≥ 3 separate BP measurements 6 to 9 months postbaseline. Incident hypertension was defined as a mean systolic BP ≥ 140 mm Hg or a mean diastolic BP ≥ 90 mm Hg. Simulated trials (1,000) were performed, using bootstrap methods to investigate the effect of variable numbers of nights (x = 1-56 per participant) to quantify AHI and the ability to detect associations between OSA and incident hypertension via logistic regression adjusted for age, sex, and BMI.

RESULTS

Participants were middle-aged (mean ± SD, 52 ± 12 y), mostly men (91%), and overweight (BMI, 28 ± 5 kg/m²). Single-night quantification of OSA failed to detect an association with hypertension risk in 42% of simulated trials (α = 0.05). Conversely, 100% of trials detected an association when AHI was quantified over ≥ 28 nights. Point estimates of hypertension risk were also 50% higher and uncertainty was 5 times lower during multinight vs single-night simulation trials.

INTERPRETATION

Multinight monitoring of OSA allows for better estimates of hypertension risk and potentially other adverse health outcomes associated with OSA. These findings have important implications for clinical care and OSA trial design.

Benchmarking performance of an automatic polysomnography scoring system in a population with suspected sleep disorders

Aim

The current gold standard for measuring sleep disorders is polysomnography (PSG), which is manually scored by a sleep technologist. Scoring a PSG is time-consuming and tedious, with substantial inter-rater variability. A deep-learning-based sleep analysis software module can perform autoscoring of PSG. The primary objective of the study is to validate the accuracy and reliability of the autoscoring software. The secondary objective is to measure workflow improvements in terms of time and cost via a time motion study.

Methodology

The performance of an automatic PSG scoring software was benchmarked against the performance of two independent sleep technologists on PSG data collected from patients with suspected sleep disorders. The technologists at the hospital clinic and a third-party scoring company scored the PSG records independently. The scores were then compared between the technologists and the automatic scoring system. An observational study was also performed where the time taken for sleep technologists at the hospital clinic to manually score PSGs was tracked, along with the time taken by the automatic scoring software to assess for potential time savings.

Results

Pearson's correlation between the manually scored apnea-hypopnea index (AHI) and the automatically scored AHI was 0.962, demonstrating a near-perfect agreement. The autoscoring system demonstrated similar results in sleep staging. The agreement between automatic staging and manual scoring was higher in terms of accuracy and Cohen's kappa than the agreement between experts. The autoscoring system took an average of 42.7 s to score each record compared with 4,243 s for manual scoring. Following a manual review of the auto scores, an average time savings of 38.6 min per PSG was observed, amounting to 0.25 full-time equivalent (FTE) savings per year.

Conclusion

The findings indicate a potential for a reduction in the burden of manual scoring of PSGs by sleep technologists and may be of operational significance for sleep laboratories in the healthcare setting.

Obstructive sleep apnea‐related intermittent hypoxaemia is associated with impaired vigilance

Obstructive sleep apnea (OSA)-related intermittent hypoxaemia is a potential risk factor for different OSA comorbidities, for example cardiovascular disease. However, conflicting results are found as to whether intermittent hypoxaemia is associated with impaired vigilance. Therefore, we aimed to investigate how desaturation characteristics differ between the non-impaired vigilance and impaired vigilance patient groups formed based on psychomotor vigilance task (PVT) performance and compared with traditional OSA severity parameters. The study population comprised 863 patients with suspected OSA who underwent a PVT test before polysomnography. The conventional OSA parameters, for example, the apnea-hypopnea index, oxygen desaturation index, and arousal index were computed. Furthermore, the median desaturation area, fall area, recovery area, and desaturation depth were computed with the pre-event baseline reference and with reference to the 100% oxygen saturation level. Patients were grouped into best- and worst-performing quartiles based on the number of lapses in PVT (Q1: PVT lapses <5 and Q4: PVT lapses >36). The association between parameters and impaired vigilance was evaluated by cumulative distribution functions (CDFs) and binomial logistic regression. Based on the CDFs, patients in Q4 had larger desaturation areas, recovery areas, and deeper desaturations when these were referenced to 100% saturation compared with Q1. The odds ratio (OR) of the median desaturation area (OR = 1.56), recovery area (OR = 1.71), and depth (OR = 1.65) were significantly elevated in Q4 in regression models. However, conventional OSA parameters were not significantly associated with impaired vigilance (ORs: 0.79-1.09). Considering desaturation parameters with a 100% SpO₂ reference in the diagnosis of OSA could provide additional information on the severity of OSA and related daytime vigilance impairment.

Certainty about uncertainty in sleep staging: a theoretical framework

Sleep stage classification is an important tool for the diagnosis of sleep disorders. Because sleep staging has such a high impact on clinical outcome, it is important that it is done reliably. However, it is known that uncertainty exists in both expert scorers and automated models. On average, the agreement between human scorers is only 82.6%. In this study, we provide a theoretical framework to facilitate discussion and further analyses of uncertainty in sleep staging. To this end, we introduce two variants of uncertainty, known from statistics and the machine learning community: aleatoric and epistemic uncertainty. We discuss what these types of uncertainties are, why the distinction is useful, where they arise from in sleep staging, and provide recommendations on how this framework can improve sleep staging in the future.

A systematic review of deep learning methods for modeling electrocardiograms during sleep

Sleep is one of the most important human physiological activities and plays an essential role in human health. Polysomnography (PSG) is the gold standard for measuring sleep quality and disorders, but it is time-consuming, labor-intensive, and prone to errors. Current research has confirmed the correlations between sleep and the respiratory/circulatory system. Electrocardiography (ECG) is convenient to perform, and ECG data are rich in breathing information. Therefore, sleep research based on ECG data has become popular. Currently, deep learning (DL) methods have achieved promising results on predictive health care tasks using ECG signals. Therefore, in this review, we systematically identify recent research studies and analyze them from the perspectives of data, model, and task. We discuss the shortcomings, summarize the findings, and highlight the potential opportunities. For sleep-related tasks, many ECG-based DL methods produce more accurate results than traditional approaches by combining multiple signal features and model structures. Methods that are more interpretable, scalable, and transferable will become ubiquitous in the daily practice of medicine and ambient-assisted-living applications. This paper is the first systematic review of ECG-based DL methods for sleep tasks.

Improving Machine Learning Technology in the Field of Sleep

The authors discuss the challenges of machine- and deep learning-based automatic analysis of obstructive sleep apnea with respect to known issues with the signal interpretation, patient physiology, and the apnea-hypopnea index. Their goal is to provide guidance for sleep and machine learning professionals working in this area of sleep medicine. They suggest that machine learning approaches may well be better targeted at examining and attempting to improve the diagnostic criteria, in order to build a more nuanced understanding of the detailed circumstances surrounding OSA, rather than merely attempting to reproduce human scoring.

Evaluation of Automated Hypnogram Analysis on Multi-Scored Polysomnographies

A new method for automated sleep stage scoring of polysomnographies is proposed that uses a random forest approach to model feature interactions and temporal effects. The model mostly relies on features based on the rules from the American Academy of Sleep Medicine, which allows medical experts to gain insights into the model. A common way to evaluate automated approaches to constructing hypnograms is to compare the one produced by the algorithm to an expert's hypnogram. However, given the same data, two expert annotators will construct (slightly) different hypnograms due to differing interpretations of the data or individual mistakes. A thorough evaluation of our method is performed on a multi-labeled dataset in which both the inter-rater variability as well as the prediction uncertainties are taken into account, leading to a new standard for the evaluation of automated sleep stage scoring algorithms. On all epochs, our model achieves an accuracy of 82.7%, which is only slightly lower than the inter-rater disagreement. When only considering the 63.3% of the epochs where both the experts and algorithm are certain, the model achieves an accuracy of 97.8%. Transition periods between sleep stages are identified and studied for the first time. Scoring guidelines for medical experts are provided to complement the certain predictions by scoring only a few epochs manually. This makes the proposed method highly time-efficient while guaranteeing a highly accurate final hypnogram.

A deep transfer learning approach for wearable sleep stage classification with photoplethysmography

Unobtrusive home sleep monitoring using wrist-worn wearable photoplethysmography (PPG) could open the way for better sleep disorder screening and health monitoring. However, PPG is rarely included in large sleep studies with gold-standard sleep annotation from polysomnography. Therefore, training data-intensive state-of-the-art deep neural networks is challenging. In this work a deep recurrent neural network is first trained using a large sleep data set with electrocardiogram (ECG) data (292 participants, 584 recordings) to perform 4-class sleep stage classification (wake, rapid-eye-movement, N1/N2, and N3). A small part of its weights is adapted to a smaller, newer PPG data set (60 healthy participants, 101 recordings) through three variations of transfer learning. Best results (Cohen's kappa of 0.65 ± 0.11, accuracy of 76.36 ± 7.57%) were achieved with the domain and decision combined transfer learning strategy, significantly outperforming the PPG-trained and ECG-trained baselines. This performance for PPG-based 4-class sleep stage classification is unprecedented in literature, bringing home sleep stage monitoring closer to clinical use. The work demonstrates the merit of transfer learning in developing reliable methods for new sensor technologies by reusing similar, older non-wearable data sets. Further study should evaluate our approach in patients with sleep disorders such as insomnia and sleep apnoea.

Landscape of biomedical informatics standards and terminologies for clinical sleep medicine research: A systematic review

A systematic literature review was conducted to understand the current landscape of standards and terminologies used in clinical sleep medicine. Literature search on PubMed, EMBASE, Medline and Web of Science was performed in March 2021 using terms related to sleep, terminologies, standards, harmonization, semantics, ontology, and electronic health records (EHR). Systematic review was carried out according to PRISMA. Among 128 included studies, 35 were eligible for review. Articles were broadly classified into six topics: standard terminology efforts, reporting standards, databases and resources, data integration efforts, EHR abstraction and standards for automated sleep scoring. This review highlights the progress and challenges related to establishing computable terminologies in sleep medicine, and identifies gaps, limitations and research opportunities related to data integration that could improve adoption of clinical research informatics in this field. There is a need for the systematic adoption of standardized terminologies in all areas of sleep medicine. Existing data aggregation resources could be leveraged to support the development of an integrated infrastructure and subsequent deployment in EHR systems within sleep centers. Ultimately, the adoption of standardized practices for documenting sleep disorders and related traits facilitates data sharing, thus accelerating discovery and clinical translation of informatics approaches applied to sleep medicine.

Interrater sleep stage scoring reliability between manual scoring from two European sleep centers and automatic scoring performed by the artificial intelligence-based Stanford-STAGES algorithm

STUDY OBJECTIVES

The objective of this study was to evaluate interrater reliability between manual sleep stage scoring performed in 2 European sleep centers and automatic sleep stage scoring performed by the previously validated artificial intelligence-based Stanford-STAGES algorithm.

METHODS

Full night polysomnographies of 1,066 participants were included. Sleep stages were manually scored in Berlin and Innsbruck sleep centers and automatically scored with the Stanford-STAGES algorithm. For each participant, we compared (1) Innsbruck to Berlin scorings (INN vs BER); (2) Innsbruck to automatic scorings (INN vs AUTO); (3) Berlin to automatic scorings (BER vs AUTO); (4) epochs where scorers from Innsbruck and Berlin had consensus to automatic scoring (CONS vs AUTO); and (5) both Innsbruck and Berlin manual scorings (MAN) to the automatic ones (MAN vs AUTO). Interrater reliability was evaluated with several measures, including overall and sleep stage-specific Cohen's κ.

RESULTS

Overall agreement across participants was substantial for INN vs BER (κ = 0.66 ± 0.13), INN vs AUTO (κ = 0.68 ± 0.14), CONS vs AUTO (κ = 0.73 ± 0.14), and MAN vs AUTO (κ = 0.61 ± 0.14), and moderate for BER vs AUTO (κ = 0.55 ± 0.15). Human scorers had the highest disagreement for N1 sleep (κ_N1 = 0.40 ± 0.16 for INN vs BER). Automatic scoring had lowest agreement with manual scorings for N1 and N3 sleep (κ_N1 = 0.25 ± 0.14 and κ_N3 = 0.42 ± 0.32 for MAN vs AUTO).

CONCLUSIONS

Interrater reliability for sleep stage scoring between human scorers was in line with previous findings, and the algorithm achieved an overall substantial agreement with manual scoring. In this cohort, the Stanford-STAGES algorithm showed similar performances to the ones achieved in the original study, suggesting that it is generalizable to new cohorts. Before its integration in clinical practice, future independent studies should further evaluate it in other cohorts.

Automatic analysis of single-channel sleep EEG in a large spectrum of sleep disorders

STUDY OBJECTIVES

To assess the performance of the single-channel automatic sleep staging (AS) software ASEEGA in adult patients diagnosed with various sleep disorders.

METHODS

Sleep recordings were included of 95 patients (38 women, 40.5 ± 13.7 years) diagnosed with insomnia (n = 23), idiopathic hypersomnia (n = 24), narcolepsy (n = 24), and obstructive sleep apnea (n = 24). Visual staging (VS) was performed by two experts (VS1 and VS2) according to the American Academy of Sleep Medicine rules. AS was based on the analysis of a single electroencephalogram channel (Cz-Pz), without any information from electro-oculography nor electromyography. The epoch-by-epoch agreement (concordance and Conger's coefficient [κ]) was compared pairwise (VS1-VS2, AS-VS1, AS-VS2) and between AS and consensual VS. Sleep parameters were also compared.

RESULTS

The pairwise agreements were: between AS and VS1, 78.6% (κ = 0.70); AS and VS2, 75.0% (0.65); and VS1 and VS2, 79.5% (0.72). Agreement between AS and consensual VS was 85.6% (0.80), with the following distribution: insomnia 85.5% (0.80), narcolepsy 83.8% (0.78), idiopathic hypersomnia 86.1% (0.68), and obstructive sleep disorder 87.2% (0.82). A significant low-amplitude scorer effect was observed for most sleep parameters, not always driven by the same scorer. Hypnograms obtained with AS and VS exhibited very close sleep organization, except for 80% of rapid eye movement sleep onset in the group diagnosed with narcolepsy missed by AS.

CONCLUSIONS

Agreement between AS and VS in sleep disorders is comparable to that reported in healthy individuals and to interexpert agreement in patients. ASEEGA could therefore be considered as a complementary sleep stage scoring tool in clinical practice, after improvement of rapid eye movement sleep onset detection.

Heart rate variability during wakefulness as a marker of obstructive sleep apnea severity

STUDY OBJECTIVES

Patients with obstructive sleep apnea (OSA) exhibit heterogeneous heart rate variability (HRV) during wakefulness and sleep. We investigated the influence of OSA severity on HRV parameters during wakefulness in a large international clinical sample.

METHODS

1247 subjects (426 without OSA and 821 patients with OSA) were enrolled from the Sleep Apnea Global Interdisciplinary Consortium. HRV parameters were calculated during a 5-minute wakefulness period with spontaneous breathing prior to the sleep study, using time-domain, frequency-domain and nonlinear methods. Differences in HRV were evaluated among groups using analysis of covariance, controlling for relevant covariates.

RESULTS

Patients with OSA showed significantly lower time-domain variations and less complexity of heartbeats compared to individuals without OSA. Those with severe OSA had remarkably reduced HRV compared to all other groups. Compared to non-OSA patients, those with severe OSA had lower HRV based on SDNN (adjusted mean: 37.4 vs. 46.2 ms; p < 0.0001), RMSSD (21.5 vs. 27.9 ms; p < 0.0001), ShanEn (1.83 vs. 2.01; p < 0.0001), and Forbword (36.7 vs. 33.0; p = 0.0001). While no differences were found in frequency-domain measures overall, among obese patients there was a shift to sympathetic dominance in severe OSA, with a higher LF/HF ratio compared to obese non-OSA patients (4.2 vs. 2.7; p = 0.009).

CONCLUSIONS

Time-domain and nonlinear HRV measures during wakefulness are associated with OSA severity, with severe patients having remarkably reduced and less complex HRV. Frequency-domain measures show a shift to sympathetic dominance only in obese OSA patients. Thus, HRV during wakefulness could provide additional information about cardiovascular physiology in OSA patients.

CLINICAL TRIAL INFORMATION

A Prospective Observational Cohort to Study the Genetics of Obstructive Sleep Apnea and Associated Co-Morbidities (German Clinical Trials Register - DKRS, DRKS00003966) https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00003966.

The Dreem Headband compared to polysomnography for electroencephalographic signal acquisition and sleep staging

Study Objectives

The development of ambulatory technologies capable of monitoring brain activity during sleep longitudinally is critical for advancing sleep science. The aim of this study was to assess the signal acquisition and the performance of the automatic sleep staging algorithms of a reduced-montage dry-electroencephalographic (EEG) device (Dreem headband, DH) compared to the gold-standard polysomnography (PSG) scored by five sleep experts.

Methods

A total of 25 subjects who completed an overnight sleep study at a sleep center while wearing both a PSG and the DH simultaneously have been included in the analysis. We assessed (1) similarity of measured EEG brain waves between the DH and the PSG; (2) the heart rate, breathing frequency, and respiration rate variability (RRV) agreement between the DH and the PSG; and (3) the performance of the DH’s automatic sleep staging according to American Academy of Sleep Medicine guidelines versus PSG sleep experts manual scoring.

Results

The mean percentage error between the EEG signals acquired by the DH and those from the PSG for the monitoring of α was 15 ± 3.5%, 16 ± 4.3% for β, 16 ± 6.1% for λ, and 10 ± 1.4% for θ frequencies during sleep. The mean absolute error for heart rate, breathing frequency, and RRV was 1.2 ± 0.5 bpm, 0.3 ± 0.2 cpm, and 3.2 ± 0.6%, respectively. Automatic sleep staging reached an overall accuracy of 83.5 ± 6.4% (F1 score: 83.8 ± 6.3) for the DH to be compared with an average of 86.4 ± 8.0% (F1 score: 86.3 ± 7.4) for the 5 sleep experts.

Conclusions

These results demonstrate the capacity of the DH to both monitor sleep-related physiological signals and process them accurately into sleep stages. This device paves the way for, large-scale, longitudinal sleep studies.

Clinical Trial Registration

NCT03725943.

Deep Learning for Diagnosis and Classification of Obstructive Sleep Apnea: A Nasal Airflow-Based Multi-Resolution Residual Network

PURPOSE

This study evaluated a novel approach for diagnosis and classification of obstructive sleep apnea (OSA), called Obstructive Sleep Apnea Smart System (OSASS), using residual networks and single-channel nasal pressure airflow signals.

METHODS

Data were collected from the sleep center of the First Affiliated Hospital, Sun Yat-sen University, and the Integrative Department of Guangdong Province Traditional Chinese Medical Hospital. We developed a new model called the multi-resolution residual network (Mr-ResNet) based on a residual network to detect nasal pressure airflow signals recorded by polysomnography (PSG) automatically. The performance of the model was assessed by its sensitivity, specificity, accuracy, and F1-score. We built OSASS based on Mr-ResNet to estimate the apnea‒hypopnea index (AHI) and to classify the severity of OSA, and compared the agreement between OSASS output and the registered polysomnographic technologist (RPSGT) score, assessed by two technologists.

RESULTS

In the primary test set, the sensitivity, specificity, accuracy, and F1-score of Mr-ResNet were 90.8%, 90.5%, 91.2%, and 90.5%, respectively. In the independent test set, the Spearman correlation for AHI between OSASS and the RPSGT score determined by two technologists was 0.94 (p < 0.001) and 0.96 (p < 0.001), respectively. Cohen's Kappa scores for classification between OSASS and the two technologists' scores were 0.81 and 0.84, respectively.

CONCLUSION

Our results indicated that OSASS can automatically diagnose and classify OSA using signals from a single-channel nasal pressure airflow, which is consistent with polysomnographic technologists' findings. Thus, OSASS holds promise for clinical application.

Validation of a new unattended sleep apnea monitor using two methods for the identification of hypopneas

STUDY OBJECTIVES

The objective of the present study was to evaluate the accuracy of a home sleep apnea test (HSAT), MATRx plus (Zephyr Sleep Technologies, Calgary, Alberta, Canada), in identifying apneas and hypopneas and estimating indices of obstructive sleep apnea (OSA).

METHODS

Individuals with suspected OSA underwent a one-night study wearing both HSAT and polysomnogram (PSG) sensors. The results provided by the overnight HSAT were compared with those from the simultaneously recorded PSG. The PSG data were scored manually, and the HSAT data were analyzed automatically using both preceding peak (PP) and moving average window (MW) methods for determining the reference oxyhemoglobin saturation (O₂ Sat). Accuracy of the HSAT in detecting individual apneic and hypopneic events was evaluated on an epoch-by-epoch basis. The apnea-hypopnea indices from the two recording systems were compared.

RESULTS

Agreement analysis for the individual apneic and hypopneic events yielded median values for sensitivity and specificity of 0.89 and 0.98 and positive and negative likelihood ratios of 37.35 and 0.11, respectively. Comparison of OSA indices between the two systems yielded correlation coefficients in the range of 0.95-0.96 and intraclass correlation coefficients ranging from 0.92-0.96. Bland-Altman analyses showed 0-2 cases lying outside the ± 2 standard deviation (SD) band and biases ranging from 2.1 to 5.3 events/h. The biases were larger for MW than PP.

CONCLUSIONS

The MATRx plus HSAT identifies apneic and hypopneic events and estimates OSA indices with accuracy suitable for clinical purposes but not in children, patients with underlying lung disease, and habitual mouth-breathers.

CLINICAL TRIAL REGISTRATION

Registry: ClinicalTrials.gov; Name: PSG Validation of MATRx plus AHI; Identifier: NCT03627169.

Estimation of the global prevalence and burden of obstructive sleep apnoea: a literature-based analysis

BACKGROUND

There is a scarcity of published data on the global prevalence of obstructive sleep apnoea, a disorder associated with major neurocognitive and cardiovascular sequelae. We used publicly available data and contacted key opinion leaders to estimate the global prevalence of obstructive sleep apnoea.

METHODS

We searched PubMed and Embase to identify published studies reporting the prevalence of obstructive sleep apnoea based on objective testing methods. A conversion algorithm was created for studies that did not use the American Academy of Sleep Medicine (AASM) 2012 scoring criteria to identify obstructive sleep apnoea, allowing determination of an equivalent apnoea-hypopnoea index (AHI) for publications that used different criteria. The presence of symptoms was not specifically analysed because of scarce information about symptoms in the reference studies and population data. Prevalence estimates for obstructive sleep apnoea across studies using different diagnostic criteria were standardised with a newly developed algorithm. Countries without obstructive sleep apnoea prevalence data were matched to a similar country with available prevalence data; population similarity was based on the population body-mass index, race, and geographical proximity. The primary outcome was prevalence of obstructive sleep apnoea based on AASM 2012 diagnostic criteria in individuals aged 30-69 years (as this age group generally had available data in the published studies and related to information from the UN for all countries).

FINDINGS

Reliable prevalence data for obstructive sleep apnoea were available for 16 countries, from 17 studies. Using AASM 2012 diagnostic criteria and AHI threshold values of five or more events per h and 15 or more events per h, we estimated that 936 million (95% CI 903-970) adults aged 30-69 years (men and women) have mild to severe obstructive sleep apnoea and 425 million (399-450) adults aged 30-69 years have moderate to severe obstructive sleep apnoea globally. The number of affected individuals was highest in China, followed by the USA, Brazil, and India.

INTERPRETATION

To our knowledge, this is the first study to report global prevalence of obstructive sleep apnoea; with almost 1 billion people affected, and with prevalence exceeding 50% in some countries, effective diagnostic and treatment strategies are needed to minimise the negative health impacts and to maximise cost-effectiveness.

FUNDING

ResMed.

Effect of Varying Definitions of Hypopnea on the Diagnosis and Clinical Outcomes of Sleep-Disordered Breathing: A Systematic Review and Meta-Analysis

STUDY OBJECTIVES

Various criteria have been used for scoring hypopneas, leading to difficulties when comparing results in clinical and research settings. We conducted a systematic review and meta-analysis to assess the effect of different hypopnea definitions on the diagnosis, severity, and clinical implications of sleep-disordered breathing (SDB).

METHODS

Ovid MEDLINE, Embase, and Scopus databases were queried for English-language publications from inception through March 7, 2017. Studies that directly compared various hypopnea definitions were eligible. The hierarchical summary receiver operating characteristic model was used to jointly estimate diagnostic performance for comparisons between criteria.

RESULTS

The initial search yielded 2,828 abstracts; 28 met inclusion criteria. After reviewing reference lists and expert review, five additional articles were identified. Most of the studies were cross-sectional or retrospective in nature. Eleven studies compared 2007 recommended criteria with 2012 criteria; 6 of these (evaluating 6,628 patients) were suitable for inclusion in the meta-analysis. Using the 2012 definition (≥ 3% desaturation or arousal) as the reference standard, the 2007 definition (≥ 4% desaturation) showed a sensitivity of 82.7% (95% confidence interval 0.72-0.90) and specificity of 93.2% (95% confidence interval 0.82-0.98). Although 2007 criteria were found to be associated with prevalent cardiovascular (CV) disease and increased risk of CV death, the 2012 criteria appeared to correspond better with intermediate CV risk markers based on two abstracts.

CONCLUSIONS

As expected, 2012 hypopnea scoring criteria resulted in a greater prevalence and severity of SDB. Data regarding the effect of varying hypopnea definitions on clinical outcomes, quality of life, health care costs, and mortality rates are limited.

COMMENTARY

A commentary on this article appears in this issue on page 683.

Neural network analysis of sleep stages enables efficient diagnosis of narcolepsy

Analysis of sleep for the diagnosis of sleep disorders such as Type-1 Narcolepsy (T1N) currently requires visual inspection of polysomnography records by trained scoring technicians. Here, we used neural networks in approximately 3,000 normal and abnormal sleep recordings to automate sleep stage scoring, producing a hypnodensity graph-a probability distribution conveying more information than classical hypnograms. Accuracy of sleep stage scoring was validated in 70 subjects assessed by six scorers. The best model performed better than any individual scorer (87% versus consensus). It also reliably scores sleep down to 5 s instead of 30 s scoring epochs. A T1N marker based on unusual sleep stage overlaps achieved a specificity of 96% and a sensitivity of 91%, validated in independent datasets. Addition of HLA-DQB1*06:02 typing increased specificity to 99%. Our method can reduce time spent in sleep clinics and automates T1N diagnosis. It also opens the possibility of diagnosing T1N using home sleep studies.

Reliability of the American Academy of Sleep Medicine Rules for Assessing Sleep Depth in Clinical Practice

STUDY OBJECTIVES

The American Academy of Sleep Medicine has published manuals for scoring polysomnograms that recommend time spent in non-rapid eye movement sleep stages (stage N1, N2, and N3 sleep) be reported. Given the well-established large interrater variability in scoring stage N1 and N3 sleep, we determined the range of time in stage N1 and N3 sleep scored by a large number of technologists when compared to reasonably estimated true values.

METHODS

Polysomnograms of 70 females were scored by 10 highly trained sleep technologists, two each from five different academic sleep laboratories. Range and confidence interval (CI = difference between the 5th and 95th percentiles) of the 10 times spent in stage N1 and N3 sleep assigned in each polysomnogram were determined. Average values of times spent in stage N1 and N3 sleep generated by the 10 technologists in each polysomnogram were considered representative of the true values for the individual polysomnogram. Accuracy of different technologists in estimating delta wave duration was determined by comparing their scores to digitally determined durations.

RESULTS

The CI range of the ten N1 scores was 4 to 39 percent of total sleep time (% TST) in different polysomnograms (mean CI ± standard deviation = 11.1 ± 7.1 % TST). Corresponding range for N3 was 1 to 28 % TST (14.4 ± 6.1 % TST). For stage N1 and N3 sleep, very low or very high values were reported for virtually all polysomnograms by different technologists. Technologists varied widely in their assignment of stage N3 sleep, scoring that stage when the digitally determined time of delta waves ranged from 3 to 17 seconds.

CONCLUSIONS

Manual scoring of non-rapid eye movement sleep stages is highly unreliable among highly trained, experienced technologists. Measures of sleep continuity and depth that are reliable and clinically relevant should be a focus of clinical research.

Optimal Classification of Respiratory Patterns From Manual Analyses Using Expectation-Maximization

Manual scoring (MS) of cardiorespiratory signals is the gold standard method for the analysis of respiratory data in sleep laboratories. In MS, trained, expert scorers characterize respiratory patterns by scrolling through a data record and visually identifying patterns. However, MS is limited by high intra- and inter-scorer variability and subjectivity. A strategy to mitigate this is to analyze the same respiratory data multiple times and generate a consensus. This consensus is generally determined by a majority vote (MV), where the most frequent pattern is selected as the true pattern. This paper presents expectation-maximization pattern sequence (EM-PSEQ), a novel method based on EM that estimates the true patterns optimally. A simulation study examined the accuracies of EM-PSEQ, MV, and individual scorers (IS) as a function of the number of analyses. Accuracy was measured with the Fleiss κ statistic, and is reported as , where , the median value, is the expected accuracy, and , the 5th percentile value, gives the minimum accuracy for 95% confidence. IS accuracy remained constant at as the number of analyses increased. MV accuracy increased slowly with the number of analyses and plateaued at after five analyses. In contrast, EM-PSEQ accuracy improved quickly, reaching an almost perfect value of with four analyses, and perfect accuracy after 25 analyses. EM-PSEQ performed much better than either MV or IS, and required only modest computational effort. Consequently, we believe EM-PSEQ will be a very valuable tool for clinical studies, as it can dramatically improve the accuracy of manual respiratory analysis with minimal additional cost.

Sleep apnea: a review of diagnostic sensors, algorithms, and therapies

While public awareness of sleep related disorders is growing, sleep apnea syndrome (SAS) remains a public health and economic challenge. Over the last two decades, extensive controlled epidemiologic research has clarified the incidence, risk factors including the obesity epidemic, and global prevalence of obstructive sleep apnea (OSA), as well as establishing a growing body of literature linking OSA with cardiovascular morbidity, mortality, metabolic dysregulation, and neurocognitive impairment. The US Institute of Medicine Committee on Sleep Medicine estimates that 50-70 million US adults have sleep or wakefulness disorders. Furthermore, the American Academy of Sleep Medicine (AASM) estimates that more than 29 million US adults suffer from moderate to severe OSA, with an estimated 80% of those individuals living unaware and undiagnosed, contributing to more than $149.6 billion in healthcare and other costs in 2015. Although various devices have been used to measure physiological signals, detect apneic events, and help treat sleep apnea, significant opportunities remain to improve the quality, efficiency, and affordability of sleep apnea care. As our understanding of respiratory and neurophysiological signals and sleep apnea physiological mechanisms continues to grow, and our ability to detect and process biomedical signals improves, novel diagnostic and treatment modalities emerge.

OBJECTIVE

This article reviews the current engineering approaches for the detection and treatment of sleep apnea.

APPROACH

It discusses signal acquisition and processing, highlights the current nonsurgical and nonpharmacological treatments, and discusses potential new therapeutic approaches.

MAIN RESULTS

This work has led to an array of validated signal and sensor modalities for acquiring, storing and viewing sleep data; a broad class of computational and signal processing approaches to detect and classify SAS disease patterns; and a set of distinctive therapeutic technologies whose use cases span the continuum of disease severity.

SIGNIFICANCE

This review provides a current perspective of the classes of tools at hand, along with a sense of their relative strengths and areas for further improvement.

Effect of moderate, 30 percent global maternal nutrient reduction on fetal and postnatal baboon phenotype

BACKGROUND

Most developmental programming studies on maternal nutrient reduction (MNR) are in altricial rodents whose maternal nutritional burden and offspring developmental trajectory differ from precocial non-human primates and humans.

METHODS

Control (CTR) baboon mothers ate ad libitum; MNR mothers ate 70% global control diet in pregnancy and lactation.

RESULTS

We present offspring morphometry, blood cortisol, and adrenocorticotropin (ACTH) during second half of gestation (G) and first three postnatal years. Moderate MNR produced intrauterine growth restriction (IUGR). IUGR males (n=43) and females (n=28) were smaller than CTR males (n=50) and females (n=47) in many measurements at many ages. In CTR, fetal ACTH increased 228% and cortisol 48% between 0.65G and 0.9G. IUGR ACTH was elevated at 0.65G and cortisol at 0.9G. 0.9G maternal gestational weight gain, fetal weight, and placenta weight were correlated.

CONCLUSIONS

Moderate IUGR decreased body weight and morphometric measurements at key time points and altered hypothalamo-pituitary-adrenal function.

The Accuracy, Night-to-Night Variability, and Stability of Frontopolar Sleep Electroencephalography Biomarkers

STUDY OBJECTIVES

To assess the validity of sleep architecture and sleep continuity biomarkers obtained from a portable, multichannel forehead electroencephalography (EEG) recorder.

METHODS

Forty-seven subjects simultaneously underwent polysomnography (PSG) while wearing a multichannel frontopolar EEG recording device (Sleep Profiler). The PSG recordings independently staged by 5 registered polysomnographic technologists were compared for agreement with the autoscored sleep EEG before and after expert review. To assess the night-to-night variability and first night bias, 2 nights of self-applied, in-home EEG recordings obtained from a clinical cohort of 63 patients were used (41% with a diagnosis of insomnia/depression, 35% with insomnia/obstructive sleep apnea, and 17.5% with all three). The between-night stability of abnormal sleep biomarkers was determined by comparing each night's data to normative reference values.

RESULTS

The mean overall interscorer agreements between the 5 technologists were 75.9%, and the mean kappa score was 0.70. After visual review, the mean kappa score between the autostaging and five raters was 0.67, and staging agreed with a majority of scorers in at least 80% of the epochs for all stages except stage N1. Sleep spindles, autonomic activation, and stage N3 exhibited the least between-night variability (P < .0001) and strongest between-night stability. Antihypertensive medications were found to have a significant effect on sleep quality biomarkers (P < .02).

CONCLUSIONS

A strong agreement was observed between the automated sleep staging and human-scored PSG. One night's recording appeared sufficient to characterize abnormal slow wave sleep, sleep spindle activity, and heart rate variability in patients, but a 2-night average improved the assessment of all other sleep biomarkers.

COMMENTARY

Two commentaries on this article appear in this issue on pages 771 and 773.

DRUG EVALUATION AND DECISION MAKING IN CATALONIA: DEVELOPMENT AND VALIDATION OF A METHODOLOGICAL FRAMEWORK BASED ON MULTI-CRITERIA DECISION ANALYSIS (MCDA) FOR ORPHAN DRUGS

OBJECTIVES

The aim of this study was to adapt and assess the value of a Multi-Criteria Decision Analysis (MCDA) framework (EVIDEM) for the evaluation of Orphan drugs in Catalonia (Catalan Health Service).

METHODS

The standard evaluation and decision-making procedures of CatSalut were compared with the EVIDEM methodology and contents. The EVIDEM framework was adapted to the Catalan context, focusing on the evaluation of Orphan drugs (PASFTAC program), during a Workshop with sixteen PASFTAC members. The criteria weighting was done using two different techniques (nonhierarchical and hierarchical). Reliability was assessed by re-test.

RESULTS

The EVIDEM framework and methodology was found useful and feasible for Orphan drugs evaluation and decision making in Catalonia. All the criteria considered for the development of the CatSalut Technical Reports and decision making were considered in the framework. Nevertheless, the framework could improve the reporting of some of these criteria (i.e., "unmet needs" or "nonmedical costs"). Some Contextual criteria were removed (i.e., "Mandate and scope of healthcare system", "Environmental impact") or adapted ("population priorities and access") for CatSalut purposes. Independently of the weighting technique considered, the most important evaluation criteria identified for orphan drugs were: "disease severity", "unmet needs" and "comparative effectiveness", while the "size of the population" had the lowest relevance for decision making. Test-retest analysis showed weight consistency among techniques, supporting reliability overtime.

CONCLUSIONS

MCDA (EVIDEM framework) could be a useful tool to complement the current evaluation methods of CatSalut, contributing to standardization and pragmatism, providing a method to tackle ethical dilemmas and facilitating discussions related to decision making.

Performance of a New Portable Wireless Sleep Monitor

STUDY OBJECTIVES

To determine if signals generated by a new sleep monitor (Prodigy) are comparable to signals generated during in-laboratory polysomnography (PSG).

METHODS

Fifty-nine patients with various sleep disorders (25 with moderate/severe sleep apnea) were studied. Full PSG was performed using standard acquisition systems. Prodigy was attached to the forehead with four disposable snap electrodes. Four additional electrodes were attached to monitor eye movements and muscle activity, and to serve as reference (mastoid). One frontal EEG signal was outputted in real time from the monitor and stored in the PSG record along with the other PSG signals. PSG was scored for sleep variables manually, and monitor records were scored by a validated automatic system (MSS) (MSS-Prodigy). MSS-Prodigy was briefly edited following suggestions of an Editing Helper feature of MSS.

RESULTS

Technical failures resulted in one study being unusable and another with data for only 3 hours. Prodigy EEG signal stored in the PSG record was visually indistinguishable from the PSG-derived EEG signals. Important differences between manual scores and unedited MSS-Prodigy were seen in a few patients in some sleep variables (notably onset latencies and REM time). Editing Helper issued 2.1 ± 0.8 suggestions/file. Only these suggestions were pursued during editing. Intraclass correlation coefficients for manual vs. edited MSS-Prodigy were > 0.83 for all sleep variables except for stages N1 and N3 (0.57 and 0.58).

CONCLUSIONS

When scored with MSS, and with only very minor editing, the monitor's results show excellent agreement with manual scoring of polysomnography data, even in patients with severe sleep disorders.

Minimizing Interrater Variability in Staging Sleep by Use of Computer-Derived Features

STUDY OBJECTIVES

Inter-scorer variability in sleep staging of polysomnograms (PSGs) results primarily from difficulty in determining whether: (1) an electroencephalogram pattern of wakefulness spans > 15 sec in transitional epochs, (2) spindles or K complexes are present, and (3) duration of delta waves exceeds 6 sec in a 30-sec epoch. We hypothesized that providing digitally derived information about these variables to PSG scorers may reduce inter-scorer variability.

METHODS

Fifty-six PSGs were scored (five-stage) by two experienced technologists, (first manual, M1). Months later, the technologists edited their own scoring (second manual, M2). PSGs were then scored with an automatic system and the same two technologists and an additional experienced technologist edited them, epoch-by-epoch (Edited-Auto). This resulted in seven manual scores for each PSG. The two M2 scores were then independently modified using digitally obtained values for sleep depth and delta duration and digitally identified spindles and K complexes.

RESULTS

Percent agreement between scorers in M2 was 78.9 ± 9.0% before modification and 96.5 ± 2.6% after. Errors of this approach were defined as a change in a manual score to a stage that was not assigned by any scorer during the seven manual scoring sessions. Total errors averaged 7.1 ± 3.7% and 6.9 ± 3.8% of epochs for scorers 1 and 2, respectively, and there was excellent agreement between the modified score and the initial manual score of each technologist.

CONCLUSIONS

Providing digitally obtained information about sleep depth, delta duration, spindles and K complexes during manual scoring can greatly reduce interrater variability in sleep staging by eliminating the guesswork in scoring epochs with equivocal features.

Personalized sleep staging system using evolutionary algorithm and symbolic fusion

This paper presents a novel system for automatic sleep staging based on evolutionary technique and symbolic intelligence. Proposed system mimics decision making process of clinical sleep staging using Symbolic Fusion and considers personal singularity with an adaptive thresholds setting up system using Evolutionary Algorithm. It proved to be an effective and promising system in personalizing sleep staging. This system can also be integrated with other medical systems to realize remote sleep monitoring or home-care.

Staging Sleep in Polysomnograms: Analysis of Inter-Scorer Variability

STUDY OBJECTIVES

To determine the reasons for inter-scorer variability in sleep staging of polysomnograms (PSGs).

METHODS

Fifty-six PSGs were scored (5-stage sleep scoring) by 2 experienced technologists, (first manual, M1). Months later, the technologists edited their own scoring (second manual, M2) based upon feedback from the investigators that highlighted differences between their scoring. The PSGs were then scored with an automatic system (Auto) and the technologists edited them, epoch-by-epoch (Edited-Auto). This resulted in 6 different manual scores for each PSG. Epochs were classified as scorer errors (one M1 score differed from the other 5 scores), scorer bias (all 3 scores of each technologist were similar, but differed from the other technologist) and equivocal (sleep scoring was inconsistent within and between technologists).

RESULTS

Percent agreement after M1 was 78.9% ± 9.0% and was unchanged after M2 (78.1% ± 9.7%) despite numerous edits (≈40/PSG) by the scorers. Agreement in Edited-Auto was higher (86.5% ± 6.4%, p < 1E-9). Scorer errors (< 2% of epochs) and scorer bias (3.5% ± 2.3% of epochs) together accounted for < 20% of M1 disagreements. A large number of epochs (92 ± 44/PSG) with scoring agreement in M1 were subsequently changed in M2 and/or Edited-Auto. Equivocal epochs, which showed scoring inconsistency, accounted for 28% ± 12% of all epochs, and up to 76% of all epochs in individual patients. Disagreements were largely between awake/NREM, N1/N2, and N2/N3 sleep.

CONCLUSION

Inter-scorer variability is largely due to epochs that are difficult to classify. Availability of digitally identified events (e.g., spindles) or calculated variables (e.g., depth of sleep, delta wave duration) during scoring may greatly reduce scoring variability.

An Examination of Methodological Paradigms for Calculating Upper Airway Critical Pressures during Sleep

STUDY OBJECTIVES

The goal of this study was to examine different paradigms for determining critical closing pressures (Pcrit). Methods of determining Pcrit were compared, including direct observation of occluded (no flow) breaths versus inferring Pcrit from extrapolated data, and Pcrit generated by aggregating pressure-flow data from multiple runs versus Pcrit averaged across individual pressure-flow runs. The relationship between Pcrit and obstructive sleep apnea (OSA) was examined.

METHODS

A total of 351 participants with and without OSA underwent overnight polysomnography with pressure-flow measurements to determine Pcrit. A series of filters were applied to raw data to provide consistent, objective criteria for determining which data to include in Pcrit calculations. Observed Pcrit values were computed as the mean nasal pressure level at which a subject had at least two breaths with peak inspiratory flow < 50 mL/sec. Extrapolated Pcrit was calculated in two ways: (1) separately for each individual run and then averaged; and (2) using all valid data from individual runs combined into one plot.

RESULTS

Observed Pcrit was calculated in 67% to 69% of participants, a similar or higher proportion of study subjects compared to extrapolated Pcrit values using a ± 3 cm H2O filter. Although raw (unfiltered) extrapolated Pcrit measures were able to be calculated among a greater proportion of participants than filtered, extrapolated Pcrit values, and thus had fewer missing values, they had larger variability. Both extrapolated and observed Pcrit were higher among individuals with OSA compared to those without OSA.

CONCLUSIONS

Observed Pcrit provides a reliable descriptor of hypotonic upper airway collapsibility. Different methods for determining Pcrit were able to distinguish subjects with and without OSA.

How to measure snoring? A comparison of the microphone, cannula and piezoelectric sensor

The objective of this study was to compare to each other the methods currently recommended by the American Academy of Sleep Medicine (AASM) to measure snoring: an acoustic sensor, a piezoelectric sensor and a nasal pressure transducer (cannula). Ten subjects reporting habitual snoring were included in the study, performed at Landspitali-University Hospital, Iceland. Snoring was assessed by listening to the air medium microphone located on a patient's chest, compared to listening to two overhead air medium microphones (stereo) and manual scoring of a piezoelectric sensor and nasal cannula vibrations. The chest audio picked up the highest number of snore events of the different snore sensors. The sensitivity and positive predictive value of scoring snore events from the different sensors was compared to the chest audio: overhead audio (0.78, 0.98), cannula (0.55, 0.67) and piezoelectric sensor (0.78, 0.92), respectively. The chest audio was capable of detecting snore events with lower volume and higher fundamental frequency than the other sensors. The 200 Hz sampling rate of the cannula and piezoelectric sensor was one of their limitations for detecting snore events. The different snore sensors do not measure snore events in the same manner. This lack of consistency will affect future research on the clinical significance of snoring. Standardization of objective snore measurements is therefore needed. Based on this paper, snore measurements should be audio-based and the use of the cannula as a snore sensor be discontinued, but the piezoelectric sensor could possibly be modified for improvement.

Obstructive sleep apnoea in the general population: highly prevalent but minimal symptoms

The aim was to assess the prevalence of obstructive sleep apnoea (OSA) as defined by an apnoea-hypopnea index (AHI) ≥15 in the middle-aged general population, and the interrelationship between OSA, sleep-related symptoms, sleepiness and vigilance.A general population sample of 40-65-year-old Icelanders was invited to participate in a study protocol that included a type 3 sleep study, questionnaire and a psychomotor vigilance test (PVT).Among the 415 subjects included in the study, 56.9% had no OSA (AHI <5), 24.1% had mild OSA (AHI 5-14.9), 12.5% had moderate OSA (AHI 15-29.9), 2.9% had severe OSA (AHI ≥30) and 3.6% were already diagnosed and receiving OSA treatment. However, no significant relationship was found between AHI and subjective sleepiness or clinical symptoms. A relationship with objective vigilance assessed by PVT was only found for those with AHI ≥30. Subjects already on OSA treatment and those accepting OSA treatment after participating in the study were more symptomatic and sleepier than others with similar OSA severity, as assessed by the AHI.In a middle-aged general population, approximately one in five subjects had moderate-to-severe OSA, but the majority of them were neither symptomatic nor sleepy and did not have impaired vigilance.

Computer-Assisted Automated Scoring of Polysomnograms Using the Somnolyzer System

STUDY OBJECTIVES

Manual scoring of polysomnograms is a time-consuming and tedious process. To expedite the scoring of polysomnograms, several computerized algorithms for automated scoring have been developed. The overarching goal of this study was to determine the validity of the Somnolyzer system, an automated system for scoring polysomnograms.

DESIGN

The analysis sample comprised of 97 sleep studies. Each polysomnogram was manually scored by certified technologists from four sleep laboratories and concurrently subjected to automated scoring by the Somnolyzer system. Agreement between manual and automated scoring was examined. Sleep staging and scoring of disordered breathing events was conducted using the 2007 American Academy of Sleep Medicine criteria.

SETTING

Clinical sleep laboratories.

MEASUREMENTS AND RESULTS

A high degree of agreement was noted between manual and automated scoring of the apnea-hypopnea index (AHI). The average correlation between the manually scored AHI across the four clinical sites was 0.92 (95% confidence interval: 0.90-0.93). Similarly, the average correlation between the manual and Somnolyzer-scored AHI values was 0.93 (95% confidence interval: 0.91-0.96). Thus, interscorer correlation between the manually scored results was no different than that derived from manual and automated scoring. Substantial concordance in the arousal index, total sleep time, and sleep efficiency between manual and automated scoring was also observed. In contrast, differences were noted between manually and automated scored percentages of sleep stages N1, N2, and N3.

CONCLUSION

Automated analysis of polysomnograms using the Somnolyzer system provides results that are comparable to manual scoring for commonly used metrics in sleep medicine. Although differences exist between manual versus automated scoring for specific sleep stages, the level of agreement between manual and automated scoring is not significantly different than that between any two human scorers. In light of the burden associated with manual scoring, automated scoring platforms provide a viable complement of tools in the diagnostic armamentarium of sleep medicine.