Standardized database development for EEG epileptiform transient detection: EEGnet scoring system and machine learning analysis

Interrater reliability of interictal EEG waveforms in Lennox-Gastaut Syndrome

OBJECTIVE

Identification of EEG waveforms is critical for diagnosing Lennox-Gastaut Syndrome (LGS) but is complicated by the progressive nature of the disease. Here, we assess the interrater reliability (IRR) among pediatric epileptologists for classifying EEG waveforms associated with LGS.

METHODS

A novel automated algorithm was used to objectively identify epochs of EEG with transient high power, which were termed events of interest (EOIs). The algorithm was applied to EEG from 20 LGS subjects and 20 healthy controls during NREM sleep, and 1350 EOIs were identified. Three raters independently reviewed the EOIs within isolated 15-second EEG segments in a randomized, blinded fashion. For each EOI, the raters assigned a waveform label (spike and slow wave, generalized paroxysmal fast activity, seizure, spindle, vertex, muscle, artifact, nothing, or other) and indicated the perceived subject type (LGS or control).

RESULTS

Labeling of subject type had 85% accuracy across all EOIs and an IRR of κ =0.790, suggesting that brief segments of EEG containing high-power waveforms can be reliably classified as pathological or normal. Waveform labels were less consistent, with κ =0.558, and the results were highly variable for different categories of waveforms. Label mismatches typically occurred when one reviewer selected "nothing," suggesting that reviewers had different thresholds for applying named labels.

SIGNIFICANCE

Classification of EEG waveforms associated with LGS has weak IRR, due in part to varying thresholds applied during visual review. Computational methods to objectively define EEG biomarkers of LGS may improve IRR and aid clinical decision-making.

A review of signal processing and machine learning techniques for interictal epileptiform discharge detection

Brain interictal epileptiform discharges (IEDs), as one of the hallmarks of epileptic brain, are transient events captured by electroencephalogram (EEG). IEDs are generated by seizure networks, and they occur between seizures (interictal periods). The development of a robust method for IED detection could be highly informative for clinical treatment procedures and epileptic patient management. Since 1972, different machine learning techniques, from template matching to deep learning, have been developed to automatically detect IEDs from scalp EEG (scEEG) and intracranial EEG (iEEG). While the scEEG signals suffer from low information details and high attenuation of IEDs due to the high skull electrical impedance, the iEEG signals recorded using implanted electrodes enjoy higher details and are more suitable for identifying the IEDs. In this review paper, we group IED detection techniques into six categories: (1) template matching, (2) feature representation (mimetic, time-frequency, and nonlinear features), (3) matrix decomposition, (4) tensor factorization, (5) neural networks, and (6) estimation of the iEEG from the concurrent scEEG followed by detection and classification. The methods are compared quantitatively (e.g., in terms of accuracy, sensitivity, and specificity), and their general advantages and limitations are described. Finally, current limitations and possible future research paths related to this field are mentioned.

Pitfalls in scalp EEG: Current obstacles and future directions

Although electroencephalography (EEG) serves a critical role in the evaluation and management of seizure disorders, it is commonly misinterpreted, resulting in avoidable medical, social, and financial burdens to patients and health care systems. Overinterpretation of sharply contoured transient waveforms as being representative of interictal epileptiform abnormalities lies at the core of this problem. However, the magnitude of these errors is amplified by the high prevalence of paroxysmal events exhibited in clinical practice that compel investigation with EEG. Neurology training programs, which vary considerably both in the degree of exposure to EEG and the composition of EEG didactics, have not effectively addressed this widespread issue. Implementation of competency-based curricula in lieu of traditional educational approaches may enhance proficiency in EEG interpretation amongst general neurologists in the absence of formal subspecialty training. Efforts in this regard have led to the development of a systematic, high-fidelity approach to the interpretation of epileptiform discharges that is readily employable across medical centers. Additionally, machine learning techniques hold promise for accelerating accurate and reliable EEG interpretation, particularly in settings where subspecialty interpretive EEG services are not readily available. This review highlights common diagnostic errors in EEG interpretation, limitations in current educational paradigms, and initiatives aimed at resolving these challenges.

Epileptic seizure focus detection from interictal electroencephalogram: a survey

Electroencephalogram (EEG) is one of most effective clinical diagnosis modalities for the localization of epileptic focus. Most current AI solutions use this modality to analyze the EEG signals in an automated manner to identify the epileptic seizure focus. To develop AI system for identifying the epileptic focus, there are many recently-published AI solutions based on biomarkers or statistic features that utilize interictal EEGs. In this review, we survey these solutions and find that they can be divided into three main categories: (i) those that use of biomarkers in EEG signals, including high-frequency oscillation, phase-amplitude coupling, and interictal epileptiform discharges, (ii) others that utilize feature-extraction methods, and (iii) solutions based upon neural networks (an end-to-end approach). We provide a detailed description of seizure focus with clinical diagnosis methods, a summary of the public datasets that seek to reduce the research gap in epilepsy, recent novel performance evaluation criteria used to evaluate the AI systems, and guidelines on when and how to use them. This review also suggests a number of future research challenges that must be overcome in order to design more efficient computer-aided solutions to epilepsy focus detection.

Fast parametric curve matching (FPCM) for automatic spike detection

Epilepsy is a widely spread neurological disease, whose treatment often requires resection of the pathological cortical tissue. Interictal spike analysis observed in the non-invasively collected EEG or MEG data offers a way to localize epileptogenic cortical structures for surgery planning purposes. While a plethora of automatic spike detection techniques have been developed each with its own assumptions and limitations, non of them is ideal and the best results are achieved when the output of several automatic spike detectors are combined. This is especially true in the low signal-to-noise ratio conditions. To this end we propose a novel biomimetic approach for automatic spike detection based on a constrained mixed spline machinery that we dub as fast parametric curve matching (FPCM). Using the peak-wave shape parametrization, the constrained parametric morphological model is constructed and convolved with the observed multichannel data to very efficiently determine mixed spline parameters corresponding to each time-point in the dataset. Then the logical predicates that directly map to the expected interictal event morphology allow us to accomplish the spike detection task. The results of simulations mimicking typical low SNR scenario show the robustness and high ROC AUC values of the FPCM method as compared to the spike detection performed by the means of more conventional approaches such as wavelet decomposition, template matching or simple amplitude thresholding. Applied to the real MEG and EEG data from the human patients and to ECoG data from the rat, the FPCM technique demonstrates reliable detection of the interictal events and localization of epileptogenic zones concordant with independent conclusions made by the epileptologist. Since the FPCM is computationally light, tolerant to high amplitude artifacts and flexible to accommodate verbalized descriptions of the arbitrary target morphology, it may complement the existing arsenal of means for analysis of noisy interictal datasets.

Semiology, EEG, and neuroimaging findings in temporal lobe epilepsies

Temporal lobe epilepsy (TLE) is the most common type of focal epilepsy. First descriptions of TLE date back in time and detailed portraits of epileptic seizures of temporal origin can be found in early medical reports as well as in the works of various artists and dramatists. Depending on the seizure onset zone, several subtypes of TLE have been identified, each one associated with peculiar ictal semiology. TLE can result from multiple etiological causes, ranging from genetic to lesional ones. While the diagnosis of TLE relies on detailed analysis of clinical as well as electroencephalographic (EEG) features, the lesions responsible for seizure generation can be highlighted by multiple brain imaging modalities or, in selected cases, by genetic investigations. TLE is the most common cause of refractory epilepsy and despite the great advances in diagnostic tools, no lesion is found in around one-third of patients. Surgical treatment is a safe and effective option, requiring presurgical investigations to accurately identify the seizure onset zone (SOZ). In selected cases, presurgical investigations need intracerebral investigations (such as stereoelectroencephalography) or dedicated metabolic imaging techniques (interictal PET and ictal SPECT) to correctly identify the brain structures to be removed.

Machine learning for detection of interictal epileptiform discharges

The electroencephalogram (EEG) is a fundamental tool in the diagnosis and classification of epilepsy. In particular, Interictal Epileptiform Discharges (IEDs) reflect an increased likelihood of seizures and are routinely assessed by visual analysis of the EEG. Visual assessment is, however, time consuming and prone to subjectivity, leading to a high misdiagnosis rate and motivating the development of automated approaches. Research towards automating IED detection started 45 years ago. Approaches range from mimetic methods to deep learning techniques. We review different approaches to IED detection, discussing their performance and limitations. Traditional machine learning and deep learning methods have yielded the best results so far and their application in the field is still growing. Standardization of datasets and outcome measures is necessary to compare models more objectively and decide which should be implemented in a clinical setting.

Incorporating Uncertainty in Data Labeling into Automatic Detection of Interictal Epileptiform Discharges from Concurrent Scalp-EEG via Multi-way Analysis

Interictal epileptiform discharges (IEDs) are elicited from an epileptic brain, whereas they can also be due to other neurological abnormalities. The diversity in their morphologies, their strengths, and their sources within the brain cause a great deal of uncertainty in their labeling by clinicians. The aim of this study is therefore to exploit and incorporate this uncertainty (the probability of the waveform being an IED) in the IED detection system which combines spatial component analysis (SCA) with the IED probabilities referred to as SCA-IEDP-based method. For comparison, we also propose and study SCA-based method in which probability of the waveform being an IED is ignored. The proposed models are employed to detect IEDs in two different classification approaches: (1) subject-dependent and (2) subject-independent classification approaches. The proposed methods are compared with two other state-of-the-art methods namely, time-frequency features and tensor factorization methods. The proposed SCA-IEDP model has achieved superior performance in comparison with the traditional SCA and other competing methods. It achieved 79.9% and 63.4% accuracy values in subject-dependent and subject-independent classification approaches, respectively. This shows that considering the IED probabilities in designing an IED detection system can boost its performance.

Interrater Reliability of Experts in Identifying Interictal Epileptiform Discharges in Electroencephalograms

Importance

The validity of using electroencephalograms (EEGs) to diagnose epilepsy requires reliable detection of interictal epileptiform discharges (IEDs). Prior interrater reliability (IRR) studies are limited by small samples and selection bias.

Objective

To assess the reliability of experts in detecting IEDs in routine EEGs.

Design, Setting, and Participants

This prospective analysis conducted in 2 phases included as participants physicians with at least 1 year of subspecialty training in clinical neurophysiology. In phase 1, 9 experts independently identified candidate IEDs in 991 EEGs (1 expert per EEG) reported in the medical record to contain at least 1 IED, yielding 87 636 candidate IEDs. In phase 2, the candidate IEDs were clustered into groups with distinct morphological features, yielding 12 602 clusters, and a representative candidate IED was selected from each cluster. We added 660 waveforms (11 random samples each from 60 randomly selected EEGs reported as being free of IEDs) as negative controls. Eight experts independently scored all 13 262 candidates as IEDs or non-IEDs. The 1051 EEGs in the study were recorded at the Massachusetts General Hospital between 2012 and 2016.

Main Outcomes and Measures

Primary outcome measures were percentage of agreement (PA) and beyond-chance agreement (Gwet κ) for individual IEDs (IED-wise IRR) and for whether an EEG contained any IEDs (EEG-wise IRR). Secondary outcomes were the correlations between numbers of IEDs marked by experts across cases, calibration of expert scoring to group consensus, and receiver operating characteristic analysis of how well multivariate logistic regression models may account for differences in the IED scoring behavior between experts.

Results

Among the 1051 EEGs assessed in the study, 540 (51.4%) were those of females and 511 (48.6%) were those of males. In phase 1, 9 experts each marked potential IEDs in a median of 65 (interquartile range [IQR], 28-332) EEGs. The total number of IED candidates marked was 87 636. Expert IRR for the 13 262 individually annotated IED candidates was fair, with the mean PA being 72.4% (95% CI, 67.0%-77.8%) and mean κ being 48.7% (95% CI, 37.3%-60.1%). The EEG-wise IRR was substantial, with the mean PA being 80.9% (95% CI, 76.2%-85.7%) and mean κ being 69.4% (95% CI, 60.3%-78.5%). A statistical model based on waveform morphological features, when provided with individualized thresholds, explained the median binary scores of all experts with a high degree of accuracy of 80% (range, 73%-88%).

Conclusions and Relevance

This study's findings suggest that experts can identify whether EEGs contain IEDs with substantial reliability. Lower reliability regarding individual IEDs may be largely explained by various experts applying different thresholds to a common underlying statistical model.

Automated Detection of Interictal Epileptiform Discharges from Scalp Electroencephalograms by Convolutional Neural Networks

Visual evaluation of electroencephalogram (EEG) for Interictal Epileptiform Discharges (IEDs) as distinctive biomarkers of epilepsy has various limitations, including time-consuming reviews, steep learning curves, interobserver variability, and the need for specialized experts. The development of an automated IED detector is necessary to provide a faster and reliable diagnosis of epilepsy. In this paper, we propose an automated IED detector based on Convolutional Neural Networks (CNNs). We have evaluated the proposed IED detector on a sizable database of 554 scalp EEG recordings (84 epileptic patients and 461 nonepileptic subjects) recorded at Massachusetts General Hospital (MGH), Boston. The proposed CNN IED detector has achieved superior performance in comparison with conventional methods with a mean cross-validation area under the precision-recall curve (AUPRC) of 0.838[Formula: see text]±[Formula: see text]0.040 and false detection rate of 0.2[Formula: see text]±[Formula: see text]0.11 per minute for a sensitivity of 80%. We demonstrated the proposed system to be noninferior to 30 neurologists on a dataset from the Medical University of South Carolina (MUSC). Further, we clinically validated the system at National University Hospital (NUH), Singapore, with an agreement accuracy of 81.41% with a clinical expert. Moreover, the proposed system can be applied to EEG recordings with any arbitrary number of channels.

Detection of mesial temporal lobe epileptiform discharges on intracranial electrodes using deep learning

OBJECTIVE

Develop a high-performing algorithm to detect mesial temporal lobe (mTL) epileptiform discharges on intracranial electrode recordings.

METHODS

An epileptologist annotated 13,959 epileptiform discharges from a dataset of intracranial EEG recordings from 46 epilepsy patients. Using this dataset, we trained a convolutional neural network (CNN) to recognize mTL epileptiform discharges from a single intracranial bipolar channel. The CNN outputs from multiple bipolar channel inputs were averaged to generate the final detector output. Algorithm performance was estimated using a nested 5-fold cross-validation.

RESULTS

On the receiver-operating characteristic curve, our algorithm achieved an area under the curve (AUC) of 0.996 and a partial AUC (for specificity > 0.9) of 0.981. AUC on a precision-recall curve was 0.807. A sensitivity of 84% was attained at a false positive rate of 1 per minute. 35.9% of the false positive detections corresponded to epileptiform discharges that were missed during expert annotation.

CONCLUSIONS

Using deep learning, we developed a high-performing, patient non-specific algorithm for detection of mTL epileptiform discharges on intracranial electrodes.

SIGNIFICANCE

Our algorithm has many potential applications for understanding the impact of mTL epileptiform discharges in epilepsy and on cognition, and for developing therapies to specifically reduce mTL epileptiform activity.

EEG CLassification Via Convolutional Neural Network-Based Interictal Epileptiform Event Detection

Diagnosis of epilepsy based on visual inspection of electroencephalogram (EEG) abnormalities is an inefficient, time-consuming, and expert-centered process. Moreover, the diagnosis based on ictal epileptiform events is challenging as the ictal patterns are infrequent. Consequently, the development of an automated, fast, and reliable epileptic EEG diagnostic system is essential. The interictal epileptiform discharges (IEDs) are recurring patterns that are highly suggestive of epilepsy. In this paper, we propose an epileptic EEG classification system based on IED detection. The proposed system comprises of three modules: pre-processing, waveform-level classification, and EEG-level classification. We employ a Convolutional Neural Network (CNN) for waveform-level classification and a Support Vector Machine (SVM) for EEG-level classification. We evaluated the proposed system on a dataset of 156 EEGs recorded at Massachusetts General Hospital (MGH), Boston. The system achieved a mean 4-fold classification accuracy of 83.86% for classifying EEGs with and without IEDs.

Unsupervised Learning of Spatiotemporal Interictal Discharges in Focal Epilepsy

BACKGROUND

Interictal epileptiform discharges are an important biomarker for localization of focal epilepsy, especially in patients who undergo chronic intracranial monitoring. Manual detection of these pathophysiological events is cumbersome, but is still superior to current rule-based approaches in most automated algorithms.

OBJECTIVE

To develop an unsupervised machine-learning algorithm for the improved, automated detection and localization of interictal epileptiform discharges based on spatiotemporal pattern recognition.

METHODS

We decomposed 24 h of intracranial electroencephalography signals into basis functions and activation vectors using non-negative matrix factorization (NNMF). Thresholding the activation vector and the basis function of interest detected interictal epileptiform discharges in time and space (specific electrodes), respectively. We used convolutive NNMF, a refined algorithm, to add a temporal dimension to basis functions.

RESULTS

The receiver operating characteristics for NNMF-based detection are close to the gold standard of human visual-based detection and superior to currently available alternative automated approaches (93% sensitivity and 97% specificity). The algorithm successfully identified thousands of interictal epileptiform discharges across a full day of neurophysiological recording and accurately summarized their localization into a single map. Adding a temporal window allowed for visualization of the archetypal propagation network of these epileptiform discharges.

CONCLUSION

Unsupervised learning offers a powerful approach towards automated identification of recurrent pathological neurophysiological signals, which may have important implications for precise, quantitative, and individualized evaluation of focal epilepsy.

A fast machine learning approach to facilitate the detection of interictal epileptiform discharges in the scalp electroencephalogram

BACKGROUND

Finding interictal epileptiform discharges (IEDs) in the EEG is a part of diagnosing epilepsy. Automated software for annotating EEGs of patients with suspected epilepsy can therefore help with reaching a diagnosis. A large amount of data is required for training and evaluating an effective IED detection system. IEDs occur infrequently in the most patients' EEG, therefore, interictal EEG recordings contain mostly background waveforms.

NEW METHOD

As the first step to detect IEDs, we propose a machine learning technique eliminating most EEG background data using an ensemble of simple fast classifiers based on several EEG features. This could save computation time for an IED detection method, allowing the remaining waveforms to be classified by more computationally intensive methods. We consider several efficient features and reject background by applying thresholds on them in consecutive steps.

RESULTS

We applied the proposed algorithm on a dataset of 156 EEGs (93 and 63 with and without IEDs, respectively). We were able to eliminate 78% of background waveforms while retaining 97% of IEDs on our cross-validated dataset.

COMPARISON WITH EXISTING METHODS

We applied support vector machine, k-nearest neighbours, and random forest classifiers to detect IEDs with and without initial background rejection. Results show that rejecting background by our proposed method speeds up the overall classification by a factor ranging from 1.8 to 4.7 for the considered classifiers.

CONCLUSIONS

The proposed method successfully reduces computation time of an IED detection system. Therefore, it is beneficial in speeding up IED detection especially when utilizing large EEG datasets.

Interictal Epileptiform Discharge Detection in EEG in Different Practice Settings

OBJECTIVE

The goal of the study was to measure the performance of academic and private practice (PP) neurologists in detecting interictal epileptiform discharges in routine scalp EEG recordings.

METHODS

Thirty-five EEG scorers (EEGers) participated (19 academic and 16 PP) and marked the location of ETs in 200 30-second EEG segments using a web-based EEG annotation system. All participants provided board certification status, years of Epilepsy Fellowship Training (EFT), and years in practice. The Persyst P13 automated IED detection algorithm was also run on the EEG segments for comparison.

RESULTS

Academic EEGers had an average of 1.66 years of EFT versus 0.50 years of EFT for PP EEGers (P < 0.0001) and had higher rates of board certification. Inter-rater agreement for the 35 EEGers was fair. There was higher performance for EEGers in academics, with at least 1.5 years of EFT, and with American Board of Clinical Neurophysiology and American Board of Psychiatry and Neurology-E specialty board certification. The Persyst P13 algorithm at its default setting (perception value = 0.4) did not perform as well at the EEGers, but at substantially higher perception value settings, the algorithm performed almost as well human experts.

CONCLUSIONS

Inter-rater agreement among EEGers in both academic and PP settings varies considerably. Practice location, years of EFT, and board certification are associated with significantly higher performance for IED detection in routine scalp EEG. Continued medical education of PP neurologists and neurologists without EFT is needed to improve routine scalp EEG interpretation skills. The performance of automated detection algorithms is approaching that of human experts.

Detecting abnormal electroencephalograms using deep convolutional networks

OBJECTIVES

Electroencephalography (EEG) is a central part of the medical evaluation for patients with neurological disorders. Training an algorithm to label the EEG normal vs abnormal seems challenging, because of EEG heterogeneity and dependence of contextual factors, including age and sleep stage. Our objectives were to validate prior work on an independent data set suggesting that deep learning methods can discriminate between normal vs abnormal EEGs, to understand whether age and sleep stage information can improve discrimination, and to understand what factors lead to errors.

METHODS

We train a deep convolutional neural network on a heterogeneous set of 8522 routine EEGs from the Massachusetts General Hospital. We explore several strategies for optimizing model performance, including accounting for age and sleep stage.

RESULTS

The area under the receiver operating characteristic curve (AUC) on an independent test set (n = 851) is 0.917 marginally improved by including age (AUC = 0.924), and both age and sleep stages (AUC = 0.925), though not statistically significant.

CONCLUSIONS

The model architecture generalizes well to an independent dataset. Adding age and sleep stage to the model does not significantly improve performance.

SIGNIFICANCE

Insights learned from misclassified examples, and minimal improvement by adding sleep stage and age suggest fruitful directions for further research.

CLASSIFIER CASCADE TO AID IN DETECTION OF EPILEPTIFORM TRANSIENTS IN INTERICTAL EEG

The presence of Epileptiform Transients (ET) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. Automated ET detection can increase the uniformity and speed of ET detection. Current ET detection methods suffer from insufficient precision and high false positive rates. Since ETs occur infrequently in the EEG of most patients, the majority of recordings comprise background EEG waveforms. In this work we establish a method to exclude as much background data as possible from EEG recordings by applying a classifier cascade. The remaining data can then be classified using other ET detection methods. We compare a single Support Vector Machine (SVM) to a cascade of SVMs for detecting ETs. Our results show that the precision and false positive rate improve significantly by incorporating a classifier cascade before ET detection. Our method can help improve the precision and false positive rate of an ET detection system. At a fixed sensitivity, we were able to improve precision by 6.78%; and at a fixed false positive rate, the sensitivity improved by 2.83%.

Interictal epileptiform discharge characteristics underlying expert interrater agreement

OBJECTIVE

The presence of interictal epileptiform discharges (IED) in the electroencephalogram (EEG) is a key finding in the medical workup of a patient with suspected epilepsy. However, inter-rater agreement (IRA) regarding the presence of IED is imperfect, leading to incorrect and delayed diagnoses. An improved understanding of which IED attributes mediate expert IRA might help in developing automatic methods for IED detection able to emulate the abilities of experts. Therefore, using a set of IED scored by a large number of experts, we set out to determine which attributes of IED predict expert agreement regarding the presence of IED.

METHODS

IED were annotated on a 5-point scale by 18 clinical neurophysiologists within 200 30-s EEG segments from recordings of 200 patients. 5538 signal analysis features were extracted from the waveforms, including wavelet coefficients, morphological features, signal energy, nonlinear energy operator response, electrode location, and spectrogram features. Feature selection was performed by applying elastic net regression and support vector regression (SVR) was applied to predict expert opinion, with and without the feature selection procedure and with and without several types of signal normalization.

RESULTS

Multiple types of features were useful for predicting expert annotations, but particular types of wavelet features performed best. Local EEG normalization also enhanced best model performance. As the size of the group of EEGers used to train the models was increased, the performance of the models leveled off at a group size of around 11.

CONCLUSIONS

The features that best predict inter-rater agreement among experts regarding the presence of IED are wavelet features, using locally standardized EEG. Our models for predicting expert opinion based on EEGer's scores perform best with a large group of EEGers (more than 10).

SIGNIFICANCE

By examining a large group of EEG signal analysis features we found that wavelet features with certain wavelet basis functions performed best to identify IEDs. Local normalization also improves predictability, suggesting the importance of IED morphology over amplitude-based features. Although most IED detection studies in the past have used opinion from three or fewer experts, our study suggests a "wisdom of the crowd" effect, such that pooling over a larger number of expert opinions produces a better correlation between expert opinion and objectively quantifiable features of the EEG.

Crowdsourcing seizure detection: algorithm development and validation on human implanted device recordings

There exist significant clinical and basic research needs for accurate, automated seizure detection algorithms. These algorithms have translational potential in responsive neurostimulation devices and in automatic parsing of continuous intracranial electroencephalography data. An important barrier to developing accurate, validated algorithms for seizure detection is limited access to high-quality, expertly annotated seizure data from prolonged recordings. To overcome this, we hosted a kaggle.com competition to crowdsource the development of seizure detection algorithms using intracranial electroencephalography from canines and humans with epilepsy. The top three performing algorithms from the contest were then validated on out-of-sample patient data including standard clinical data and continuous ambulatory human data obtained over several years using the implantable NeuroVista seizure advisory system. Two hundred teams of data scientists from all over the world participated in the kaggle.com competition. The top performing teams submitted highly accurate algorithms with consistent performance in the out-of-sample validation study. The performance of these seizure detection algorithms, achieved using freely available code and data, sets a new reproducible benchmark for personalized seizure detection. We have also shared a 'plug and play' pipeline to allow other researchers to easily use these algorithms on their own datasets. The success of this competition demonstrates how sharing code and high quality data results in the creation of powerful translational tools with significant potential to impact patient care.

Automated epileptiform spike detection via affinity propagation-based template matching

Interictal epileptiform spikes are the key diagnostic biomarkers for epilepsy. The clinical gold standard of spike detection is visual inspection performed by neurologists. This is a tedious, time-consuming, and expert-centered process. The development of automated spike detection systems is necessary in order to provide a faster and more reliable diagnosis of epilepsy. In this paper, we propose an efficient template matching spike detector based on a combination of spike and background waveform templates. We generate a template library by clustering a collection of spikes and background waveforms extracted from a database of 50 patients with epilepsy. We benchmark the performance of five clustering techniques based on the receiver operating characteristic (ROC) curves. In addition, background templates are integrated with existing spike templates to improve the overall performance. The affinity propagation-based template matching system with a combination of spike and background templates is shown to outperform the other four conventional methods with the highest area-under-curve (AUC) of 0.953.

Characteristics of EEG Interpreters Associated With Higher Interrater Agreement

PURPOSE

The goal of the project is to determine characteristics of academic neurophysiologist EEG interpreters (EEGers), which predict good interrater agreement (IRA) and to determine the number of EEGers needed to develop an ideal standardized testing and training data set for epileptiform transient (ET) detection algorithms.

METHODS

A three-phase scoring method was used. In phase 1, 19 EEGers marked the location of ETs in two hundred 30-second segments of EEG from 200 different patients. In phase 2, EEG events marked by at least 2 EEGers were annotated by 18 EEGers on a 5-point scale to indicate whether they were ETs. In phase 3, a third opinion was obtained from EEGers on any inconsistencies between phase 1 and phase 2 scoring.

RESULTS

The IRA for the 18 EEGers was only fair. A select group of the EEGers had good IRA and the other EEGers had low IRA. Board certification by the American Board of Clinical Neurophysiology was associated with better IRA performance but other board certifications, years of fellowship training, and years of practice were not. As the number of EEGers used for scoring is increased, the amount of change in the consensus opinion decreases steadily and is quite low as the group size approaches 10.

CONCLUSIONS

The IRA among EEGers varies considerably. The EEGers must be tested before use as scorers for ET annotation research projects. The American Board of Clinical Neurophysiology certification is associated with improved performance. The optimal size for a group of experts scoring ETs in EEG is probably in the 6 to 10 range.

Using Supervised Machine Learning to Classify Real Alerts and Artifact in Online Multisignal Vital Sign Monitoring Data

OBJECTIVE

The use of machine-learning algorithms to classify alerts as real or artifacts in online noninvasive vital sign data streams to reduce alarm fatigue and missed true instability.

DESIGN

Observational cohort study.

SETTING

Twenty-four-bed trauma step-down unit.

PATIENTS

Two thousand one hundred fifty-three patients.

INTERVENTION

Noninvasive vital sign monitoring data (heart rate, respiratory rate, peripheral oximetry) recorded on all admissions at 1/20 Hz, and noninvasive blood pressure less frequently, and partitioned data into training/validation (294 admissions; 22,980 monitoring hours) and test sets (2,057 admissions; 156,177 monitoring hours). Alerts were vital sign deviations beyond stability thresholds. A four-member expert committee annotated a subset of alerts (576 in training/validation set, 397 in test set) as real or artifact selected by active learning, upon which we trained machine-learning algorithms. The best model was evaluated on test set alerts to enact online alert classification over time.

MEASUREMENTS AND MAIN RESULTS

The Random Forest model discriminated between real and artifact as the alerts evolved online in the test set with area under the curve performance of 0.79 (95% CI, 0.67-0.93) for peripheral oximetry at the instant the vital sign first crossed threshold and increased to 0.87 (95% CI, 0.71-0.95) at 3 minutes into the alerting period. Blood pressure area under the curve started at 0.77 (95% CI, 0.64-0.95) and increased to 0.87 (95% CI, 0.71-0.98), whereas respiratory rate area under the curve started at 0.85 (95% CI, 0.77-0.95) and increased to 0.97 (95% CI, 0.94-1.00). Heart rate alerts were too few for model development.

CONCLUSIONS

Machine-learning models can discern clinically relevant peripheral oximetry, blood pressure, and respiratory rate alerts from artifacts in an online monitoring dataset (area under the curve > 0.87).

A novel scheme for the validation of an automated classification method for epileptic spikes by comparison with multiple observers

•

We created a validation method for the evaluation of automated classification of interictal spikes.

•

We used a modified version of Wave_clus (WC) to automatically classify the data of 5 patients.

•

WC classification was similar to EEG reviewers providing an unbiased evaluation of the clinical data.

Objective

To validate the application of an automated neuronal spike classification algorithm, Wave_clus (WC), on interictal epileptiform discharges (IED) obtained from human intracranial EEG (icEEG) data.

Method

Five 10-min segments of icEEG recorded in 5 patients were used. WC and three expert EEG reviewers independently classified one hundred IED events into IED classes or non-IEDs. First, we determined whether WC-human agreement variability falls within inter-reviewer agreement variability by calculating the variation of information for each classifier pair and quantifying the overlap between all WC-reviewer and all reviewer-reviewer pairs. Second, we compared WC and EEG reviewers’ spike identification and individual spike class labels visually and quantitatively.

Results

The overlap between all WC-human pairs and all human pairs was >80% for 3/5 patients and >58% for the other 2 patients demonstrating WC falling within inter-human variation. The average sensitivity of spike marking for WC was 91% and >87% for all three EEG reviewers. Finally, there was a strong visual and quantitative similarity between WC and EEG reviewers.

Conclusions

WC performance is indistinguishable to that of EEG reviewers’ suggesting it could be a valid clinical tool for the assessment of IEDs.

Significance

WC can be used to provide quantitative analysis of epileptic spikes.

Comparison of a Novel Dry Electrode Headset to Standard Routine EEG in Veterans

OBJECTIVE

This purpose of this study was to evaluate the usefulness of a prototype battery-powered dry electrode system (DES) EEG recording headset in Veteran patients by comparing it with standard EEG.

METHODS

Twenty-one Veterans had both a standard electrode system recording and DES recording in nine different patient states at the same encounter. Setup time, patient comfort, and subject preference were measured. Three experts performed technical quality rating of each EEG recording in a blinded fashion using the web-based EEGnet system. Power spectra were compared between DES and standard electrode system recordings.

RESULTS

The average time for DES setup was 5.7 minutes versus 21.1 minutes for standard electrode system. Subjects reported that the DES was more comfortable during setup. Most subjects (15 of 21) preferred the DES. On a five-point scale (1-best quality to 5-worst quality), the technical quality of the standard electrode system recordings was significantly better than for the DES recordings, at 1.25 versus 2.41 (P < 0.0001). But experts found that 87% of the DES EEG segments were of sufficient technical quality to be interpretable.

CONCLUSIONS

This DES offers quick and easy setup and is well tolerated by subjects. Although the technical quality of DES recordings was less than standard EEG, most of the DES recordings were rated as interpretable by experts.

SIGNIFICANCE

This DES, if improved, could be useful for a telemedicine approach to outpatient routine EEG recording within the Veterans Administration or other health system.

Detection of Intracranial Signatures of Interictal Epileptiform Discharges from Concurrent Scalp EEG

Interictal epileptiform discharges (IEDs) are transient neural electrical activities that occur in the brain of patients with epilepsy. A problem with the inspection of IEDs from the scalp electroencephalogram (sEEG) is that for a subset of epileptic patients, there are no visually discernible IEDs on the scalp, rendering the above procedures ineffective, both for detection purposes and algorithm evaluation. On the other hand, intracranially placed electrodes yield a much higher incidence of visible IEDs as compared to concurrent scalp electrodes. In this work, we utilize concurrent scalp and intracranial EEG (iEEG) from a group of temporal lobe epilepsy (TLE) patients with low number of scalp-visible IEDs. The aim is to determine whether by considering the timing information of the IEDs from iEEG, the resulting concurrent sEEG contains enough information for the IEDs to be reliably distinguished from non-IED segments. We develop an automatic detection algorithm which is tested in a leave-subject-out fashion, where each test subject’s detection algorithm is based on the other patients’ data. The algorithm obtained a [Formula: see text] accuracy in recognizing scalp IED from non-IED segments with [Formula: see text] accuracy when trained and tested on the same subject. Also, it was able to identify nonscalp-visible IED events for most patients with a low number of false positive detections. Our results represent a proof of concept that IED information for TLE patients is contained in scalp EEG even if they are not visually identifiable and also that between subject differences in the IED topology and shape are small enough such that a generic algorithm can be used.

Real alerts and artifact classification in archived multi-signal vital sign monitoring data: implications for mining big data

Huge hospital information system databases can be mined for knowledge discovery and decision support, but artifact in stored non-invasive vital sign (VS) high-frequency data streams limits its use. We used machine-learning (ML) algorithms trained on expert-labeled VS data streams to automatically classify VS alerts as real or artifact, thereby "cleaning" such data for future modeling. 634 admissions to a step-down unit had recorded continuous noninvasive VS monitoring data [heart rate (HR), respiratory rate (RR), peripheral arterial oxygen saturation (SpO₂) at 1/20 Hz, and noninvasive oscillometric blood pressure (BP)]. Time data were across stability thresholds defined VS event epochs. Data were divided Block 1 as the ML training/cross-validation set and Block 2 the test set. Expert clinicians annotated Block 1 events as perceived real or artifact. After feature extraction, ML algorithms were trained to create and validate models automatically classifying events as real or artifact. The models were then tested on Block 2. Block 1 yielded 812 VS events, with 214 (26 %) judged by experts as artifact (RR 43 %, SpO₂ 40 %, BP 15 %, HR 2 %). ML algorithms applied to the Block 1 training/cross-validation set (tenfold cross-validation) gave area under the curve (AUC) scores of 0.97 RR, 0.91 BP and 0.76 SpO₂. Performance when applied to Block 2 test data was AUC 0.94 RR, 0.84 BP and 0.72 SpO₂. ML-defined algorithms applied to archived multi-signal continuous VS monitoring data allowed accurate automated classification of VS alerts as real or artifact, and could support data mining for future model building.

A study of morphology-based wavelet features and multiple-wavelet strategy for EEG signal classification: results and selected statistical analysis

Automatic detection and classification of Epileptiform transients is an open and important clinical issue. In this paper, we test 5 feature sets derived from a group of morphology-based wavelet features and compare the results with that of a Guler-suggested feature set. We also implement a multiple-mother-wavelet strategy and compare performance with the usual single-mother-wavelet strategy. The results indicate that both the derived features and the multiple-mother-wavelet strategy improved classifier performance, using a variety of performance measures. We assess the statistical significance of the performance improvement of the new feature sets/strategy. In most cases, the performance improvement is either significant or highly significant.

Generalized periodic discharges and 'triphasic waves': A blinded evaluation of inter-rater agreement and clinical significance

OBJECTIVES

Generalized periodic discharges (GPDs) are associated with nonconvulsive seizures. Triphasic waves (TWs), a subtype of GPDs, have been described in relation to metabolic encephalopathy and not felt to be associated with seizures. We sought to establish the consistency of use of this descriptive term and its association with seizures.

METHODS

11 experts in continuous EEG monitoring scored 20 cEEG samples containing GPDs using Standardized Critical Care EEG Terminology. In the absence of patient information, the inter-rater agreement (IRA) for EEG descriptors including TWs was assessed along with raters' clinical EEG interpretation and compared with actual patient information.

RESULTS

The IRA for 'generalized' and 'periodic' was near-perfect (kappa=0.81), but fair for 'triphasic' (kappa=0.33). Patients with TWs were as likely to develop seizures as those without (25% vs 26%, N.S.) and surprisingly, patients with TWs were less likely to have toxic-metabolic encephalopathy than those without TWs (55% vs 79%, p<0.01).

CONCLUSIONS

While IRA for the terms "generalized" and "periodic" is high, it is only fair for TWs. EEG interpreted as TWs presents similar risk for seizures as GPDs without triphasic appearance. GPDs are commonly associated with metabolic encephalopathy, but 'triphasic' appearance is not predictive.

SIGNIFICANCE

Conventional association of 'triphasic waves' with specific clinical conditions may lead to inaccurate EEG interpretation.

Inter-rater agreement on identification of electrographic seizures and periodic discharges in ICU EEG recordings

OBJECTIVE

This study investigated inter-rater agreement (IRA) among EEG experts for the identification of electrographic seizures and periodic discharges (PDs) in continuous ICU EEG recordings.

METHODS

Eight board-certified EEG experts independently identified seizures and PDs in thirty 1-h EEG segments which were selected from ICU EEG recordings collected from three medical centers. IRA was compared between seizure and PD identifications, as well as among rater groups that have passed an ICU EEG Certification Test, developed by the Critical Care EEG Monitoring Research Consortium (CCEMRC).

RESULTS

Both kappa and event-based IRA statistics showed higher mean values in identification of seizures compared to PDs (k=0.58 vs. 0.38; p<0.001). The group of rater pairs who had both passed the ICU EEG Certification Test had a significantly higher mean IRA in comparison to rater pairs in which neither had passed the test.

CONCLUSIONS

IRA among experts is significantly higher for identification of electrographic seizures compared to PDs. Additional instruction, such as the training module and certification test developed by the CCEMRC, could enhance this IRA.

SIGNIFICANCE

This study demonstrates more disagreement in the labeling of PDs in comparison to seizures. This may be improved by education about standard EEG nomenclature.

Inter-expert and intra-expert reliability in sleep spindle scoring

OBJECTIVES

To measure the inter-expert and intra-expert agreement in sleep spindle scoring, and to quantify how many experts are needed to build a reliable dataset of sleep spindle scorings.

METHODS

The EEG dataset was comprised of 400 randomly selected 115s segments of stage 2 sleep from 110 sleeping subjects in the general population (57±8, range: 42-72 years). To assess expert agreement, a total of 24 Registered Polysomnographic Technologists (RPSGTs) scored spindles in a subset of the EEG dataset at a single electrode location (C3-M2). Intra-expert and inter-expert agreements were calculated as F1-scores, Cohen's kappa (κ), and intra-class correlation coefficient (ICC).

RESULTS

We found an average intra-expert F1-score agreement of 72±7% (κ: 0.66±0.07). The average inter-expert agreement was 61±6% (κ: 0.52±0.07). Amplitude and frequency of discrete spindles were calculated with higher reliability than the estimation of spindle duration. Reliability of sleep spindle scoring can be improved by using qualitative confidence scores, rather than a dichotomous yes/no scoring system.

CONCLUSIONS

We estimate that 2-3 experts are needed to build a spindle scoring dataset with 'substantial' reliability (κ: 0.61-0.8), and 4 or more experts are needed to build a dataset with 'almost perfect' reliability (κ: 0.81-1).

SIGNIFICANCE

Spindle scoring is a critical part of sleep staging, and spindles are believed to play an important role in development, aging, and diseases of the nervous system.

A self-adapting system for the automated detection of inter-ictal epileptiform discharges

PURPOSE

Scalp EEG remains the standard clinical procedure for the diagnosis of epilepsy. Manual detection of inter-ictal epileptiform discharges (IEDs) is slow and cumbersome, and few automated methods are used to assist in practice. This is mostly due to low sensitivities, high false positive rates, or a lack of trust in the automated method. In this study we aim to find a solution that will make computer assisted detection more efficient than conventional methods, while preserving the detection certainty of a manual search.

METHODS

Our solution consists of two phases. First, a detection phase finds all events similar to epileptiform activity by using a large database of template waveforms. Individual template detections are combined to form "IED nominations", each with a corresponding certainty value based on the reliability of their contributing templates. The second phase uses the ten nominations with highest certainty and presents them to the reviewer one by one for confirmation. Confirmations are used to update certainty values of the remaining nominations, and another iteration is performed where ten nominations with the highest certainty are presented. This continues until the reviewer is satisfied with what has been seen. Reviewer feedback is also used to update template accuracies globally and improve future detections.

KEY FINDINGS

Using the described method and fifteen evaluation EEGs (241 IEDs), one third of all inter-ictal events were shown after one iteration, half after two iterations, and 74%, 90%, and 95% after 5, 10 and 15 iterations respectively. Reviewing fifteen iterations for the 20-30 min recordings 1 took approximately 5 min.

SIGNIFICANCE

The proposed method shows a practical approach for combining automated detection with visual searching for inter-ictal epileptiform activity. Further evaluation is needed to verify its clinical feasibility and measure the added value it presents.

Mobile EEG in epilepsy

The sensitivity of routine EEG recordings for interictal epileptiform discharges in epilepsy is limited. In some patients, inpatient video-EEG may be performed to increase the likelihood of finding abnormalities. Although many agree that home EEG recordings may provide a cost-effective alternative to these recordings, their use is still not introduced everywhere. We surveyed Dutch neurologists and patients and evaluated a novel mobile EEG device (Mobita, TMSi). Key specifications were compared with three other current mobile EEG devices. We shortly discuss algorithms to assist in the review process. Thirty percent (33 out of 109) of Dutch neurologists reported that home EEG recordings are used in their hospital. The majority of neurologists think that mobile EEG can have additional value in investigation of unclear paroxysms, but not in the initial diagnosis after a first seizure. Poor electrode contacts and signal quality, limited recording time and absence of software for reliable and effective assistance in the interpretation of EEGs have been important constraints for usage, but in recent devices discussed here, many of these problems have been solved. The majority of our patients were satisfied with the home EEG procedure and did not think that our EEG device was uncomfortable to wear, but they did feel uneasy wearing it in public.