Evaluating sleep-stage classification: how age and early-late sleep affects classification performance

Sleep stage classification is a common method used by experts to monitor the quantity and quality of sleep in humans, but it is a time-consuming and labour-intensive task with high inter- and intra-observer variability. Using wavelets for feature extraction and random forest for classification, an automatic sleep-stage classification method was sought and assessed. The age of the subjects, as well as the moment of sleep (early-night and late-night), were confronted to the performance of the classifier. From this study, we observed that these variables do affect the automatic model performance, improving the classification of some sleep stages and worsening others.

Current status and prospects of automatic sleep stages scoring: Review

The scoring of sleep stages is one of the essential tasks in sleep analysis. Since a manual procedure requires considerable human and financial resources, and incorporates some subjectivity, an automated approach could result in several advantages. There have been many developments in this area, and in order to provide a comprehensive overview, it is essential to review relevant recent works and summarise the characteristics of the approaches, which is the main aim of this article. To achieve it, we examined articles published between 2018 and 2022 that dealt with the automated scoring of sleep stages. In the final selection for in-depth analysis, 125 articles were included after reviewing a total of 515 publications. The results revealed that automatic scoring demonstrates good quality (with Cohen's kappa up to over 0.80 and accuracy up to over 90%) in analysing EEG/EEG + EOG + EMG signals. At the same time, it should be noted that there has been no breakthrough in the quality of results using these signals in recent years. Systems involving other signals that could potentially be acquired more conveniently for the user (e.g. respiratory, cardiac or movement signals) remain more challenging in the implementation with a high level of reliability but have considerable innovation capability. In general, automatic sleep stage scoring has excellent potential to assist medical professionals while providing an objective assessment.

Optimization of real-time analysis of sleep-wake cycle in mice

Studying the biology of sleep requires accurate and efficient assessment of the sleep stages. However, analysis of sleep-wake cycles in mice and other laboratory animals remains a time-consuming and laborious process. In this study, we developed a Python script and a process for the streamlined analysis of sleep data that includes real-time processing of electroencephalogram (EEG) and electromyogram (EMG) signals that is compatible with commercial sleep-recording software that supports user datagram protocol (UDP) communication. The process consists of EEG/EMG data acquisition, automated threshold calculation for real-time determination of sleep stages, sleep staging and EEG power spectrum analysis. It also allows data storage in the format that facilitates further analysis of the sleep pattern in mice. The described method is aimed at increasing efficiency of sleep stage scoring and analysis in mice thus facilitating sleep research. • A process of EEG/EMG recording and streamline analysis of sleep-wake cycle in real time in mice. • The compatibility with commercial sleep-recording software that can generate a UDP stream. • The capability of further analysis of recorded data by an open-source software.

Beyond traditional sleep scoring: Massive feature extraction and data-driven clustering of sleep time series

The widely used guidelines for sleep staging were developed for the visual inspection of electrophysiological recordings by the human eye. As such, these rules reflect a limited range of features in these data and are therefore restricted in accurately capturing the physiological changes associated with sleep. Here we present a novel analysis framework that extensively characterizes sleep dynamics using over 7700 time-series features from the hctsa software. We used clustering to categorize sleep epochs based on the similarity of their time-series features, without relying on established scoring conventions. The resulting sleep structure overlapped substantially with that defined by visual scoring. However, we also observed discrepancies between our approach and traditional scoring. This divergence principally stemmed from the extensive characterization by hctsa features, which captured distinctive time-series properties within the traditionally defined sleep stages that are overlooked with visual scoring. Lastly, we report time-series features that are highly discriminative of stages. Our framework lays the groundwork for a data-driven exploration of sleep sub-stages and has significant potential to identify new signatures of sleep disorders and conscious sleep states.

Scoring sleep using respiration and movement-based features

Rules derived from standard Rechtschaffen and Kales criteria were developed to accurately score rodent sleep into wake, rapid eye movement (REM) sleep, and non-REM sleep using movements detected by non-contact electric field (EF) sensors. • Using this method, rodent sleep can be scored using only respiratory and gross body movements as a validated, non-invasive alternative to electrode techniques. • The methodology and rules established for EF sensor-based sleep scoring were easily learned and implemented. • Examples of expert-scored files are included here to help novice scorers self-train to score sleep. Though validated in mice, sleep scoring using respiratory movements has the potential for application in other species and through other movement-based technologies beyond EF sensors.

Automatic sleep staging of EEG signals: recent development, challenges, and future directions

Modern deep learning holds a great potential to transform clinical practice on human sleep. Teaching a machine to carry out routine tasks would be a tremendous reduction in workload for clinicians. Sleep staging, a fundamental step in sleep practice, is a suitable task for this and will be the focus in this article. Recently, automatic sleep staging systems have been trained to mimic manual scoring, leading to similar performance to human sleep experts, at least on scoring of healthy subjects. Despite tremendous progress, we have not seen automatic sleep scoring adopted widely in clinical environments. This review aims to give a shared view of the authors on the most recent state-of-the-art development in automatic sleep staging, the challenges that still need to be addressed, and the future directions for automatic sleep scoring to achieve clinical value.

An artificial neural network for automated behavioral state classification in rats

Accurate behavioral state classification is critical for many research applications. Researchers typically rely upon manual identification of behavioral state through visual inspection of electrophysiological signals, but this approach is time intensive and subject to low inter-rater reliability. To overcome these limitations, a diverse set of algorithmic approaches have been put forth to automate the classification process. Recently, novel machine learning approaches have been detailed that produce rapid and highly accurate classifications. These approaches however, are often computationally expensive, require significant expertise to implement, and/or require proprietary software that limits broader adoption. Here we detail a novel artificial neural network that uses electrophysiological features to automatically classify behavioral state in rats with high accuracy, sensitivity, and specificity. Common parameters of interest to sleep scientists, including state-dependent power spectra and homeostatic non-REM slow wave activity, did not significantly differ when using this automated classifier as compared to manual scoring. Flexible options enable researchers to further increase classification accuracy through manual rescoring of a small subset of time intervals with low model prediction certainty or further decrease researcher time by generalizing trained networks across multiple recording days. The algorithm is fully open-source and coded within a popular, and freely available, software platform to increase access to this research tool and provide additional flexibility for future researchers. In sum, we have developed a readily implementable, efficient, and effective approach for automated behavioral state classification in rats.

STQS: Interpretable multi-modal Spatial-Temporal-seQuential model for automatic Sleep scoring

Sleep scoring is an important step for the detection of sleep disorders and usually performed by visual analysis. Since manual sleep scoring is time consuming, machine-learning based approaches have been proposed. Though efficient, these algorithms are black-box in nature and difficult to interpret by clinicians. In this paper, we propose a deep learning architecture for multi-modal sleep scoring, investigate the model's decision making process, and compare the model's reasoning with the annotation guidelines in the AASM manual. Our architecture, called STQS, uses convolutional neural networks (CNN) to automatically extract spatio-temporal features from 3 modalities (EEG, EOG and EMG), a bidirectional long short-term memory (Bi-LSTM) to extract sequential information, and residual connections to combine spatio-temporal and sequential features. We evaluated our model on two large datasets, obtaining an accuracy of 85% and 77% and a macro F1 score of 79% and 73% on SHHS and an in-house dataset, respectively. We further quantify the contribution of various architectural components and conclude that adding LSTM layers improves performance over a spatio-temporal CNN, while adding residual connections does not. Our interpretability results show that the output of the model is well aligned with AASM guidelines, and therefore, the model's decisions correspond to domain knowledge. We also compare multi-modal models and single-channel models and suggest that future research should focus on improving multi-modal models.

Effects of new PLM scoring rules on PLM rate in relation to sleep and resting wake for RLS and healthy controls

PURPOSE

This study evaluates the differences in periodic leg movement (PLM) rates for Restless Legs Syndrome (RLS) and healthy controls when using the updated PLM scoring criteria developed by IRLSSG in 2016 versus the prior PLM scoring criteria developed by IRLSSG in 2006. Four major problems with the prior standards had been objectively identified, i.e. minimum inter-movement interval should be 10 not 5 s, non-PLM leg movements should end any preceding PLM sequence, a leg movement (LM) can be any length > 0.5 s, and a PLM should be a persisting movement not a couple or a series of closely spaced, very brief events. Each of these led to including, erroneously, various random leg movements as PLM. Correcting these problems was expected to increase specificity, reducing the number of PLM detected, particularly in situations producing relatively more random leg movements, e.g. wake vs. sleep and controls without PLMD vs. RLS patients.

METHODS

This study evaluated the putative benefits of the updated, 2016-scoring criteria. The LMs from 42 RLS patients and 30 age- and gender-matched controls were scored for PLMS and PLMW from standard all-night PSG recordings using both 2006 and 2016 WASM criteria.

RESULTS/CONCLUSION

The results confirmed that that the 2016 compared to the 2006 criteria generally decreased the PLM rates with particularly large decreases for the conditions with more random non-PLM events, e.g. wake times and normal healthy controls. This supported the view that the new criteria succeeded in increasing the specificity of PLM detection. Moreover, the changes in PLM rates were generally small for the conditions with relatively few random LM, e.g. RLS and sleep. Thus the bulk of existing PLMS research does not require reconsideration of results, with possible exception of special situations with relatively more random leg movements than periodic leg movements, e.g. wake, healthy normals and children.

Ambulatory sleep scoring using accelerometers-distinguishing between nonwear and sleep/wake states

Background

Differentiating nonwear time from sleep and wake times is essential for the estimation of sleep duration based on actigraphy data. To efficiently analyze large-scale data sets, an automatic method of identifying these three different states is required. Therefore, we developed a classification algorithm to determine nonwear, sleep and wake periods from accelerometer data. Our work aimed to (I) develop a new pattern recognition algorithm for identifying nonwear periods from actigraphy data based on the influence of respiration rate on the power spectrum of the acceleration signal and implement it in an automatic classification algorithm for nonwear/sleep/wake states; (II) address motion artifacts that occur during nonwear periods and are known to cause misclassification of these periods; (III) adjust the algorithm depending on the sensor position (wrist, chest); and (IV) validate the algorithm on both healthy individuals and patients with sleep disorders.

Methods

The study involved 98 participants who wore wrist and chest acceleration sensors for one day of measurements. They spent one night in the sleep laboratory and continued to wear the sensors outside of the laboratory for the remainder of the day. The results of the classification algorithm were compared to those of the reference source: polysomnography for wake/sleep and manual annotations for nonwear/wear classification.

Results

The median kappa values for the two locations were 0.83 (wrist) and 0.84 (chest). The level of agreement did not vary significantly by sleep health (good sleepers vs. subjects with sleep disorders) (p = 0.348, p = 0.118) or by sex (p = 0.442, p = 0.456). The intraclass correlation coefficients of nonwear total time between the reference and the algorithm were 0.92 and 0.97 with the outliers and 0.95 and 0.98 after the outliers were removed for the wrist and chest, respectively. There was no evidence of an association between the mean difference (and 95% limits of agreement) and the mean of the two methods for either sensor position (wrist p = 0.110, chest p = 0.164), and the mean differences (algorithm minus reference) were 5.11 [95% LoA −15.4–25.7] and 1.32 [95% LoA −9.59–12.24] min/day, respectively, after the outliers were removed.

Discussion

We studied the influence of the respiration wave on the power spectrum of the acceleration signal for the differentiation of nonwear periods from sleep and wake periods. The algorithm combined both spectral analysis of the acceleration signal and rescoring. Based on the Bland-Altman analysis, the chest-worn accelerometer showed better results than the wrist-worn accelerometer.

Performance comparison of machine learning techniques in sleep scoring based on wavelet features and neighboring component analysis

Introduction

Sleep scoring is an important step in the treatment of sleep disorders. Manual annotation of sleep stages is time-consuming and experience-relevant and, therefore, needs to be done using machine learning techniques.

Methods

Sleep-EDF polysomnography was used in this study as a dataset. Support vector machines and artificial neural network performance were compared in sleep scoring using wavelet tree features and neighborhood component analysis.

Results

Neighboring component analysis as a combination of linear and non-linear feature selection method had a substantial role in feature dimension reduction. Artificial neural network and support vector machine achieved 90.30% and 89.93% accuracy, respectively.

Discussion and Conclusion

Similar to the state of the art performance, the introduced method in the present study achieved an acceptable performance in sleep scoring. Furthermore, its performance can be enhanced using a technique combined with other techniques in feature generation and dimension reduction. It is hoped that, in the future, intelligent techniques can be used in the process of diagnosing and treating sleep disorders.

Automatic sleep staging using ear-EEG

Background

Sleep and sleep quality assessment by means of sleep stage analysis is important for both scientific and clinical applications. Unfortunately, the presently preferred method, polysomnography (PSG), requires considerable expert assistance and significantly affects the sleep of the person under observation. A reliable, accurate and mobile alternative to the PSG would make sleep information much more readily available in a wide range of medical circumstances.

New method

Using an already proven method, ear-EEG, in which electrodes are placed inside the concha and ear canal, we measure cerebral activity and automatically score the sleep into up to five stages. These results are compared to manual scoring by trained clinicians, based on a simultaneously recorded PSG.

Results

The correspondence between manually scored sleep, based on the PSG, and the automatic labelling, based on ear-EEG data, was evaluated using Cohen’s kappa coefficient. Kappa values are in the range 0.5–0.8, making ear-EEG relevant for both scientific and clinical applications. Furthermore, a sleep-wake classifier with leave-one-out cross validation yielded specificity of 0.94 and sensitivity of 0.52 for the sleep stage.

Comparison with existing method(s)

Ear-EEG based scoring has clear advantages when compared to both the PSG and other mobile solutions, such as actigraphs. It is far more mobile, and potentially cheaper than the PSG, and the information on sleep stages is far superior to a wrist-based actigraph, or other devices based solely on body movement.

Conclusions

This study shows that ear-EEG recordings carry information about sleep stages, and indicates that automatic sleep staging based on ear-EEG can classify sleep stages with a level of accuracy that makes it relevant for both scientific and clinical sleep assessment.

Temporal correlation between two channels EEG of bipolar lead in the head midline is associated with sleep-wake stages

Whether the temporal correlation between inter-leads Electroencephalogram (EEG) that located on the boundary between left and right brain hemispheres is associated with sleep stages or not is still unknown. The purpose of this paper is to evaluate the role of correlation coefficients between EEG leads Fpz-Cz and Pz-Oz for automatic classification of sleep stages. A total number of 39 EEG recordings (about 20 h each) were selected from the expanded sleep database in European data format for temporal correlation analysis. Original waveform of EEG was decomposed into sub-bands δ (1-4 Hz), θ (4-8 Hz), α (8-13 Hz) and β (13-30 Hz). The correlation coefficient between original EEG leads Fpz-Cz and Pz-Oz within frequency band 0.5-30 Hz was defined as r(EEG) and was calculated every 30 s, while that between the two leads EEG in sub-bands δ, θ, α and β were defined as r(δ), r(θ), r(α) and r(β), respectively. Classification of wakefulness and sleep was processed by fixed threshold that derived from the probability density function of correlation coefficients. There was no correlation between EEG leads Fpz-Cz and Pz-Oz during wakefulness (|r| < 0.1 for r(θ), r(α) and r(β), while 0.3 > r > 0.1 for r(EEG) and r(δ)), while low correlation existed during sleep (r ≈ -0.4 for r(EEG), r(δ), r(θ), r(α) and r(β)). There were significant differences (analysis of variance, P < 0.001) for r(EEG), r(δ), r(θ), r(α) and r(β) during sleep when in comparison with that during wakefulness, respectively. The accuracy for distinguishing states between wakefulness and sleep was 94.2, 93.4, 89.4, 85.2 and 91.4% in terms of r(EEG), r(δ), r(θ), r(α) and r(β), respectively. However, no correlation index between EEG leads Fpz-Cz and Pz-Oz could distinguish all five types of wakefulness, rapid eye movement (REM) sleep, N1 sleep, N2 sleep and N3 sleep. In conclusion, the temporal correlation between EEG bipolar leads Fpz-Cz and Pz-Oz are highly associated with sleep-wake stages. Moreover, high accuracy of sleep-wake classification could be achieved by the temporal correlation within frequency band 0.5-30 Hz between EEG leads Fpz-Cz and Pz-Oz.

Multiple classifier systems for automatic sleep scoring in mice

BACKGROUND

Electroencephalogram (EEG) and electromyogram (EMG) recordings are often used in rodents to study sleep architecture and sleep-associated neural activity. These recordings must be scored to designate what sleep/wake state the animal is in at each time point. Manual sleep-scoring is very time-consuming, so machine-learning classifier algorithms have been used to automate scoring.

NEW METHOD

Instead of using single classifiers, we implement a multiple classifier system. The multiple classifier is built from six base classifiers: decision tree, k-nearest neighbors, naïve Bayes, support vector machine, neural net, and linear discriminant analysis. Decision tree and k-nearest neighbors were improved into ensemble classifiers by using bagging and random subspace. Confidence scores from each classifier were combined to determine the final classification. Ambiguous epochs can be rejected and left for a human to classify.

RESULTS

Support vector machine was the most accurate base classifier, and had error rate of 0.054. The multiple classifier system reduced the error rate to 0.049, which was not significantly different from a second human scorer. When 10% of epochs were rejected, the remaining epochs' error rate dropped to 0.018.

COMPARISON WITH EXISTING METHOD(S)

Compared with the most accurate single classifier (support vector machine), the multiple classifier reduced errors by 9.4%. The multiple classifier surpassed the accuracy of a second human scorer after rejecting only 2% of epochs.

CONCLUSIONS

Multiple classifier systems are an effective way to increase automated sleep scoring accuracy. Improvements in autoscoring will allow sleep researchers to increase sample sizes and recording lengths, opening new experimental possibilities.

SegWay: A simple framework for unsupervised sleep segmentation in experimental EEG recordings

Sleep analysis in animal models typically involves recording an electroencephalogram (EEG) and electromyogram (EMG) and scoring vigilance state in brief epochs of data as Wake, REM (rapid eye movement sleep) or NREM (non-REM) either manually or using a computer algorithm. Computerized methods usually estimate features from each epoch like the spectral power associated with distinctive cortical rhythms and dissect the feature space into regions associated with different states by applying thresholds, or by using supervised/unsupervised statistical classifiers; but there are some factors to consider when using them:•

Most classifiers require scored sample data, elaborate heuristics or computational steps not easily reproduced by the average sleep researcher, who is the targeted end user.

•

Even when prediction is reasonably accurate, small errors can lead to large discrepancies in estimates of important sleep metrics such as the number of bouts or their duration.

•

As we show here, besides partitioning the feature space by vigilance state, modeling transitions between the states can give more accurate scores and metrics.

An unsupervised sleep segmentation framework, “SegWay”, is demonstrated by applying the algorithm step-by-step to unlabeled EEG recordings in mice. The accuracy of sleep scoring and estimation of sleep metrics is validated against manual scores.

Assessing sleep architecture and continuity measures through the analysis of heart rate and wrist movement recordings in healthy subjects: comparison with results based on polysomnography

OBJECTIVE

The objective of the study was to evaluate the reliability of a new methodology for assessing sleep architecture descriptors based on heart rate and body movement recordings.

METHODS

Twelve healthy male and female subjects between 18 and 40 years of age, without sleep disorders and not taking any drug or medication that could affect sleep, were recorded continuously during five consecutive nights. Together with the standard polysomnography, heart rate was recorded with a Holter and wrist movements by actimetry. Of the 60 recorded nights, 48 artifact-free nights were analyzed by two independent and well-trained visual scorers according to the rules of the American Academy of Sleep Medicine. Sleep stages were assigned to every 30-s epoch. In parallel, the same nights were analyzed by the new methodology using only heart rate and actimetry data, allowing a 1-s epoch sleep stage classification. Sleep architecture was measured for 48 nights, independently for the two manual scorings and the automatic analysis.

RESULTS

Over 42 nights, the intra-class correlation coefficient, used to assess the consistency or reproducibility of quantitative measurements made by different observers, was classified as excellent when all 12 descriptors were combined. Analyses of the individual descriptors showed excellent interclass correlation for eight and good for four of the 12.

CONCLUSION

The automatic analysis of heart rate and body movement during sleep allows for the evaluation of sleep architecture and continuity that is equivalent to those obtained by manual scoring of polysomnography. The technique used here is simple and robust to allow for home sleep monitoring.

Automatic detection of rapid eye movements (REMs): A machine learning approach

BACKGROUND

Rapid eye movements (REMs) are a defining feature of REM sleep. The number of discrete REMs over time, or REM density, has been investigated as a marker of clinical psychopathology and memory consolidation. However, human detection of REMs is a time-consuming and subjective process. Therefore, reliable, automated REM detection software is a valuable research tool.

NEW METHOD

We developed an automatic REM detection algorithm combining a novel set of extracted features and the 'AdaBoost' classification algorithm to detect the presence of REMs in Electrooculogram data collected from the right and left outer canthi (ROC/LOC). Algorithm performance measures of Recall (percentage of REMs detected) and Precision (percentage of REMs detected that are true REMs) were calculated and compared to the gold standard of human detection by three expert sleep scorers. REM detection by four non-experts were also investigated and compared to expert raters and the algorithm.

RESULTS

The algorithm performance (78.1% Recall, 82.6% Precision) surpassed that of the average (expert & non-expert) single human detection performance (76% Recall, 83% Precision). Agreement between non-experts (Cronbach Alpha=0.65) is markedly lower than experts (Cronbach Alpha=0.80).

COMPARISON WITH EXISTING METHOD(S)

By following reported methods, we implemented all previously published LOC and ROC based detection algorithms on our dataset. Our algorithm performance exceeded all others.

CONCLUSIONS

The automatic detection algorithm presented is a viable and efficient method of REM detection as it reliably matches the performance of human scorers and outperforms all other known LOC- and ROC-based detection algorithms.

Sleep scoring made easy-Semi-automated sleep analysis software and manual rescoring tools for basic sleep research in mice

Studying sleep behavior in animal models demands clear separation of vigilance states. Pure manual scoring is time-consuming and commercial scoring software is costly. We present a LabVIEW-based, semi-automated scoring routine using recorded EEG and EMG signals. This scoring routine is •designed to reliably assign the vigilance/sleep states wakefulness (WAKE), non-rapid eye movement sleep (NREMS) and rapid eye movement sleep (REMS) to defined EEG/EMG episodes.•straightforward to use even for beginners in the field of sleep research.•freely available upon request. Chronic recordings from mice were used to design and evaluate the scoring routine consisting of an artifact-removal, a scoring- and a rescoring routine. The scoring routine processes EMG and different EEG frequency bands. Amplitude-based thresholds for EEG and EMG parameters trigger a decision tree assigning each EEG episode to a defined vigilance/sleep state automatically. Using the rescoring routine individual episodes or particular state transitions can be re-evaluated manually. High agreements between auto-scored and manual sleep scoring could be shown for experienced scorers and for beginners quickly and reliably. With small modifications to the software, it can be easily adapted for sleep analysis in other animal models.

Principal component analysis of the EEG spectrum can provide yes-or-no criteria for demarcation of boundaries between NREM sleep stages

Human sleep begins in stage 1 and progresses into stages 2 and 3 of Non-Rapid-Eye-Movement (NREM) sleep. These stages were defined using several arbitrarily-defined thresholds for subdivision of albeit continuous process of sleep deepening. Since recent studies indicate that stage 3 (slow wave sleep) has unique vital functions, more accurate measurement of this stage duration and continuity might be required for both research and practical purposes. However, the true neurophysiological boundary between stages 2 and 3 remains unknown. In a search for non-arbitrary threshold criteria for distinguishing the boundaries between NREM sleep stages, scores on the principal components of the electroencephalographic (EEG) spectrum were analyzed in relation to stage onsets. Eighteen young men made 12–20-minute attempts to nap during 24-hour wakefulness. Single-minute intervals of the nap EEG records were assigned relative to the minute of onsets of polysomnographically determined stages 1, 2, and 3. The analysis of within-nap time courses of principal components scores revealed that, unlike any conventional spectral EEG index, score on the 4th principal component exhibited a rather rapid rise on the boundary between stages 2 and 3. This was mostly a change from negative to positive score. Therefore, it might serve as yes-or-no criterion of stage 3 onset. Additionally, similarly rapid changes in sign of scores were exhibited by the 1st and 2nd principal components on the boundary of stages 2 and 1 and on the boundary between stage 1 and wakefulness, respectively.

Learning machines and sleeping brains: Automatic sleep stage classification using decision-tree multi-class support vector machines

BACKGROUND

Sleep staging is a critical step in a range of electrophysiological signal processing pipelines used in clinical routine as well as in sleep research. Although the results currently achievable with automatic sleep staging methods are promising, there is need for improvement, especially given the time-consuming and tedious nature of visual sleep scoring.

NEW METHOD

Here we propose a sleep staging framework that consists of a multi-class support vector machine (SVM) classification based on a decision tree approach. The performance of the method was evaluated using polysomnographic data from 15 subjects (electroencephalogram (EEG), electrooculogram (EOG) and electromyogram (EMG) recordings). The decision tree, or dendrogram, was obtained using a hierarchical clustering technique and a wide range of time and frequency-domain features were extracted. Feature selection was carried out using forward sequential selection and classification was evaluated using k-fold cross-validation.

RESULTS

The dendrogram-based SVM (DSVM) achieved mean specificity, sensitivity and overall accuracy of 0.92, 0.74 and 0.88 respectively, compared to expert visual scoring. Restricting DSVM classification to data where both experts' scoring was consistent (76.73% of the data) led to a mean specificity, sensitivity and overall accuracy of 0.94, 0.82 and 0.92 respectively.

COMPARISON WITH EXISTING METHODS

The DSVM framework outperforms classification with more standard multi-class "one-against-all" SVM and linear-discriminant analysis.

CONCLUSION

The promising results of the proposed methodology suggest that it may be a valuable alternative to existing automatic methods and that it could accelerate visual scoring by providing a robust starting hypnogram that can be further fine-tuned by expert inspection.

Inter-expert and intra-expert reliability in sleep spindle scoring

OBJECTIVES

To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings.

METHODS

The EEG dataset was comprised of 400 randomly selected 115s segments of stage 2 sleep from 110 sleeping subjects in the general population (57±8, range: 42-72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F1-scores, Cohen's kappa (κ), and intra-class correlation coefficient (ICC).

RESULTS

We found an average intra-expert F1-score agreement of 72±7% (κ: 0.66±0.07). The average inter-expert agreement was 61±6% (κ: 0.52±0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system.

CONCLUSIONS

We estimate that 2-3 experts are needed to build a spindle scoring dataset with 'substantial' reliability (κ: 0.61-0.8), and 4 or more experts are needed to build a dataset with 'almost perfect' reliability (κ: 0.81-1).

SIGNIFICANCE

Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.

When does this cortical area drop off? Principal component structuring of the EEG spectrum yields yes-or-no criteria of local sleep onset

The traditional sleep scoring approach has been invented long before the recognition of strictly local nature of the sleep process. It considers sleep as a whole-organism behavior state, and, thus, it cannot be used for identification of sleep onset in a separate brain region. Therefore, this paper was aimed on testing whether the practically useful, simple and reliable yes-or-no criterion of sleep onset in a particular cortical region might be developed through applying principal component analysis to the electroencephalographic (EEG) spectra. The resting EEG was recorded with 2-hour intervals throughout 43-61-hour prolongation of wakefulness, and during 12 20-minute attempts to nap in the course of 24-hour wakefulness (15 and 18 adults, respectively). The EEG power spectra were averaged on 1-min intervals of each resting EEG record and on 1-min intervals of each napping attempt, respectively. Since we earlier demonstrated that scores on the first and second principal components of the EEG spectrum exhibit dramatic changes during the sleep onset period, a zero-crossing buildup of the first score and a zero-crossing decline of the second score were examined as possible yes-or-no markers of regional sleep onsets. The results suggest that, irrespective of electrode location, sleep onset criterion and duration of preceding wakefulness, a highly significant zero-crossing decline of the second principal component score always occurred within 1-minute interval of transition from wakefulness to sleep. Therefore, it was concluded that such zero-crossing decline can serve as a reliable, simple, and practically useful yes-or-no marker of drop off event in a given cortical area.