Quantifying Motion in Video Recordings of Neonatal Seizures by Robust Motion Trackers Based on Block Motion Models

Video and audio processing in paediatrics: a review

OBJECTIVE

Video and sound acquisition and processing technologies have seen great improvements in recent decades, with many applications in the biomedical area. The aim of this paper is to review the overall state of the art of advances within these topics in paediatrics and to evaluate their potential application for monitoring in the neonatal intensive care unit (NICU).

APPROACH

For this purpose, more than 150 papers dealing with video and audio processing were reviewed. For both topics, clinical applications are described according to the considered cohorts-full-term newborns, infants and toddlers or preterm newborns. Then, processing methods are presented, in terms of data acquisition, feature extraction and characterization.

MAIN RESULTS

The paper first focuses on the exploitation of video recordings; these began to be automatically processed in the 2000s and we show that they have mainly been used to characterize infant motion. Other applications, including respiration and heart rate estimation and facial analysis, are also presented. Audio processing is then reviewed, with a focus on the analysis of crying. The first studies in this field focused on induced-pain cries and the newest ones deal with spontaneous cries; the analyses are mainly based on frequency features. Then, some papers dealing with non-cry signals are also discussed.

SIGNIFICANCE

Finally, we show that even if recent improvements in digital video and signal processing allow for increased automation of processing, the context of the NICU makes a fully automated analysis of long recordings problematic. A few proposals for overcoming some of the limitations are given.

Video surveillance of epilepsy patients using color image processing

This paper introduces a method for tracking patients under video surveillance based on a color marker system. The patients are not restricted in their movements, which requires a tracking system that can overcome non-ideal scenes e.g. occlusions, very fast movements, lighting issues and other moving objects. The suggested marker system consists of twelve unique markers that are located at each joint. By using a color marker system, each marker (if visible) can be found in every frame disregarding the possibility that it was occluded in the previous frame, compared to other tracking systems.

Non-intrusive head movement analysis of videotaped seizures of epileptic origin

In this work we propose a non-intrusive video analytic system for patient's body parts movement analysis in Epilepsy Monitoring Unit. The system utilizes skin color modeling, head/face pose template matching and face detection to analyze and quantify the head movements. Epileptic patients' heads are analyzed holistically to infer seizure and normal random movements. The patient does not require to wear any special clothing, markers or sensors, hence it is totally non-intrusive. The user initializes the person-specific skin color and selects few face/head poses in the initial few frames. The system then tracks the head/face and extracts spatio-temporal features. Support vector machines are then used on these features to classify seizure-like movements from normal random movements. Experiments are performed on numerous long hour video sequences captured in an Epilepsy Monitoring Unit at a local hospital. The results demonstrate the feasibility of the proposed system in pediatric epilepsy monitoring and seizure detection.

Vision-based motion detection, analysis and recognition of epileptic seizures--a systematic review

The analysis of human motion from video has been the object of interest for many application areas, these including surveillance, control, biomedical analysis, video annotation etc. This paper addresses the advances within this topic in relation to epilepsy, a domain where human motion is with no doubt one of the most important elements of a patient's clinical image. It describes recent achievements in vision-based detection, analysis and recognition of human motion in epilepsy for marker-based and marker-free systems. An overview of motion-characterizing features extracted so far is presented separately. The objective is to gain existing knowledge in this field and set the route marks for the future development of an integrated decision support system for epilepsy diagnosis and disease management based on automated video analysis. This review revealed that the quantification of motion patterns of selected epileptic seizures has been studied thoroughly while the recognition of seizures is currently in its beginnings, but however feasible. Moreover, only a limited set of seizure types have been analyzed so far, indicating that a holistic approach addressing all epileptic syndromes is still missing.

Ethernet Motion-Sensor Based Alarm System for Epilepsy Monitoring

In several biomedical domains, it would be interesting to monitor subjects over night time using wearable motion sensors and trigger an alarm if a specific movement has been detected by processing the accelerometer readings. In this paper, the authors describe an innovative architecture for such an alarm system in the context of epilepsy monitoring. The main ingredients of the proposed system are wireless motion sensors, a radio-frequency transceiver linked to an Ethernet gateway and an acquisition server that incorporates real-time detection method. This motion analysis system is further integrated in the dataflow of an existing medicalized alarm system and an event is sent to healthcare professionals every time a seizure is detected by the expert system. The EPIMOUV system has been evaluated, during a 6-month period, in a specialized institution with epilepsy pharmaco-resistant residents.

Classification of epileptic motor manifestations using inertial and magnetic sensors

In order to characterize objectively the succession of movements observed during motor seizures, inertial and magnetic sensors were placed on epileptic patients. Video recordings synchronized with motion recordings were analyzed visually during seizures and divided, for each limb, into events corresponding to different classes of motor manifestations. For each classified event, features were extracted and a subset selection was automated using artificial neural networks. The best artificial neural network was simulated on whole recordings to generate a stereotypic evolution of motor manifestations that we called motorograms. It is shown that motorograms can point out seizure movements and emphasize epileptic patterns.

Detection system of motor epileptic seizures through motion analysis with 3D accelerometers

A system of epilepsy seizure detection in real life conditions and based on inertial sensors is presented in this paper with a focus on the signal processing to recognize seizure moves. This system is based on several models of signals, one corresponding to general movements, and two others describing seizures moves. The detection algorithm evaluates for a given time window which model fits the best with the observed signals and trigger an alarm if this model is a seizure model. The signal processing algorithm is based on hidden Markov models.

Automated Detection of Videotaped Neonatal Seizures Based on Motion Tracking Methods

This study was carried out during the second phase of the project "Video Technologies for Neonatal Seizures" and aimed at the development of a seizure detection system by training neural networks, using quantitative motion information extracted by motion tracking methods from short video segments of infants monitored for seizures. The motion of the infants' body parts was quantified by temporal motion trajectory signals extracted from video recordings by robust motion trackers, based on block motion models. These motion trackers were developed to autonomously adjust to illumination and contrast changes that may occur during the video frame sequence. The computational tools and procedures developed for automated seizure detection were evaluated on short video segments selected and labeled by physicians from a set of 240 video recordings of 54 patients exhibiting myoclonic seizures (80 segments), focal clonic seizures (80 segments), and random infant movements (80 segments). This evaluation provided the basis for selecting the most effective strategy for training neural networks to detect neonatal seizures as well as the decision scheme used for interpreting the responses of the trained neural networks. The best neural networks exhibited sensitivity and specificity above 90%. The best among the motion trackers developed in this study produced quantitative features that constitute a reliable basis for detecting myoclonic and focal clonic neonatal seizures. The performance targets of the second phase of the project may be achieved by combining the quantitative features described in this paper with those obtained by analyzing motion strength signals produced by motion segmentation methods.

Training reformulated radial basis function neural networks capable of identifying uncertainty in data classification

This paper introduces a learning algorithm that can be used for training reformulated radial basis function neural networks (RBFNNs) capable of identifying uncertainty in data classification. This learning algorithm trains a special class of reformulated RBFNNs, known as cosine RBFNNs, by updating selected adjustable parameters to minimize the class-conditional variances at the outputs of their radial basis functions (RBFs). The experiments verify that quantum neural networks (QNNs) and cosine RBFNNs trained by the proposed learning algorithm are capable of identifying uncertainty in data classification, a property that is not shared by cosine RBFNNs trained by the original learning algorithm and conventional feed-forward neural networks (FFNNs). Finally, this study leads to a simple classification strategy that can be used to improve the classification accuracy of QNNs and cosine RBFNNs by rejecting ambiguous feature vectors based on their responses.

Automated detection of videotaped neonatal seizures based on motion segmentation methods

OBJECTIVE

This study was aimed at the development of a seizure detection system by training neural networks using quantitative motion information extracted by motion segmentation methods from short video recordings of infants monitored for seizures.

METHODS

The motion of the infants' body parts was quantified by temporal motion strength signals extracted from video recordings by motion segmentation methods based on optical flow computation. The area of each frame occupied by the infants' moving body parts was segmented by direct thresholding, by clustering of the pixel velocities, and by clustering the motion parameters obtained by fitting an affine model to the pixel velocities. The computational tools and procedures developed for automated seizure detection were tested and evaluated on 240 short video segments selected and labeled by physicians from a set of video recordings of 54 patients exhibiting myoclonic seizures (80 segments), focal clonic seizures (80 segments), and random infant movements (80 segments).

RESULTS

The experimental study described in this paper provided the basis for selecting the most effective strategy for training neural networks to detect neonatal seizures as well as the decision scheme used for interpreting the responses of the trained neural networks. Depending on the decision scheme used for interpreting the responses of the trained neural networks, the best neural networks exhibited sensitivity above 90% or specificity above 90%.

CONCLUSIONS

The best among the motion segmentation methods developed in this study produced quantitative features that constitute a reliable basis for detecting myoclonic and focal clonic neonatal seizures. The performance targets of this phase of the project may be achieved by combining the quantitative features described in this paper with those obtained by analyzing motion trajectory signals produced by motion tracking methods.

SIGNIFICANCE

A video system based upon automated analysis potentially offers a number of advantages. Infants who are at risk for seizures could be monitored continuously using relatively inexpensive and non-invasive video techniques that supplement direct observation by nursery personnel. This would represent a major advance in seizure surveillance and offers the possibility for earlier identification of potential neurological problems and subsequent intervention.

Automated Detection of Videotaped Neonatal Seizures of Epileptic Origin

PURPOSE

This study aimed at the development of a seizure-detection system by training neural networks with quantitative motion information extracted from short video segments of neonatal seizures of the myoclonic and focal clonic types and random infant movements.

METHODS

The motion of the infants' body parts was quantified by temporal motion-strength signals extracted from video segments by motion-segmentation methods based on optical flow computation. The area of each frame occupied by the infants' moving body parts was segmented by clustering the motion parameters obtained by fitting an affine model to the pixel velocities. The motion of the infants' body parts also was quantified by temporal motion-trajectory signals extracted from video recordings by robust motion trackers based on block-motion models. These motion trackers were developed to adjust autonomously to illumination and contrast changes that may occur during the video-frame sequence. Video segments were represented by quantitative features obtained by analyzing motion-strength and motion-trajectory signals in both the time and frequency domains. Seizure recognition was performed by conventional feed-forward neural networks, quantum neural networks, and cosine radial basis function neural networks, which were trained to detect neonatal seizures of the myoclonic and focal clonic types and to distinguish them from random infant movements.

RESULTS

The computational tools and procedures developed for automated seizure detection were evaluated on a set of 240 video segments of 54 patients exhibiting myoclonic seizures (80 segments), focal clonic seizures (80 segments), and random infant movements (80 segments). Regardless of the decision scheme used for interpreting the responses of the trained neural networks, all the neural network models exhibited sensitivity and specificity>90%. For one of the decision schemes proposed for interpreting the responses of the trained neural networks, the majority of the trained neural-network models exhibited sensitivity>90% and specificity>95%. In particular, cosine radial basis function neural networks achieved the performance targets of this phase of the project (i.e., sensitivity>95% and specificity>95%).

CONCLUSIONS

The best among the motion segmentation and tracking methods developed in this study produced quantitative features that constitute a reliable basis for detecting neonatal seizures. The performance targets of this phase of the project were achieved by combining the quantitative features obtained by analyzing motion-strength signals with those produced by analyzing motion-trajectory signals. The computational procedures and tools developed in this study to perform off-line analysis of short video segments will be used in the next phase of this project, which involves the integration of these procedures and tools into a system that can process and analyze long video recordings of infants monitored for seizures in real time.