AVIM—A contactless system for infant data acquisition and analysis: Software architecture and first results

How can cry acoustics associate newborns' distress levels with neurophysiological and behavioral signals?

Introduction

Even though infant crying is a common phenomenon in humans' early life, it is still a challenge for researchers to properly understand it as a reflection of complex neurophysiological functions. Our study aims to determine the association between neonatal cry acoustics with neurophysiological signals and behavioral features according to different cry distress levels of newborns.

Methods

Multimodal data from 25 healthy term newborns were collected simultaneously recording infant cry vocalizations, electroencephalography (EEG), near-infrared spectroscopy (NIRS) and videos of facial expressions and body movements. Statistical analysis was conducted on this dataset to identify correlations among variables during three different infant conditions (i.e., resting, cry, and distress). A Deep Learning (DL) algorithm was used to objectively and automatically evaluate the level of cry distress in infants.

Results

We found correlations between most of the features extracted from the signals depending on the infant's arousal state, among them: fundamental frequency (F0), brain activity (delta, theta, and alpha frequency bands), cerebral and body oxygenation, heart rate, facial tension, and body rigidity. Additionally, these associations reinforce that what is occurring at an acoustic level can be characterized by behavioral and neurophysiological patterns. Finally, the DL audio model developed was able to classify the different levels of distress achieving 93% accuracy.

Conclusion

Our findings strengthen the potential of crying as a biomarker evidencing the physical, emotional and health status of the infant becoming a crucial tool for caregivers and clinicians.

NeoCam: An Edge-Cloud Platform for Non-Invasive Real-Time Monitoring in Neonatal Intensive Care Units

In this work we introduce NeoCam, an open source hardware-software platform for video-based monitoring of preterms infants in Neonatal Intensive Care Units (NICUs). NeoCam includes an edge computing device that performs video acquisition and processing in real-time. Compared to other proposed solutions, it has the advantage of handling data more efficiently by performing most of the processing on the device, including proper anonymisation for better compliance with privacy regulations. In addition, it allows to perform various video analysis tasks of clinical interest in parallel at speeds of between 20 and 30 frames-per-second. We introduce algorithms to measure without contact the breathing rate, motor activity, body pose and emotional status of the infants. For breathing rate, our system shows good agreement with existing methods provided there is sufficient light and proper imaging conditions. Models for motor activity and stress detection are new to the best of our knowledge. NeoCam has been tested on preterms in the NICU of the University Hospital Puerta del Mar (Cádiz, Spain), and we report the lessons learned from this trial.

Healthcare applications of single camera markerless motion capture: a scoping review

Background

Single camera markerless motion capture has the potential to facilitate at home movement assessment due to the ease of setup, portability, and affordable cost of the technology. However, it is not clear what the current healthcare applications of single camera markerless motion capture are and what information is being collected that may be used to inform clinical decision making. This review aims to map the available literature to highlight potential use cases and identify the limitations of the technology for clinicians and researchers interested in the collection of movement data.

Survey Methodology

Studies were collected up to 14 January 2022 using Pubmed, CINAHL and SPORTDiscus using a systematic search. Data recorded included the description of the markerless system, clinical outcome measures, and biomechanical data mapped to the International Classification of Functioning, Disability and Health Framework (ICF). Studies were grouped by patient population.

Results

A total of 50 studies were included for data collection. Use cases for single camera markerless motion capture technology were identified for Neurological Injury in Children and Adults; Hereditary/Genetic Neuromuscular Disorders; Frailty; and Orthopaedic or Musculoskeletal groups. Single camera markerless systems were found to perform well in studies involving single plane measurements, such as in the analysis of infant general movements or spatiotemporal parameters of gait, when evaluated against 3D marker-based systems and a variety of clinical outcome measures. However, they were less capable than marker-based systems in studies requiring the tracking of detailed 3D kinematics or fine movements such as finger tracking.

Conclusions

Single camera markerless motion capture offers great potential for extending the scope of movement analysis outside of laboratory settings in a practical way, but currently suffers from a lack of accuracy where detailed 3D kinematics are required for clinical decision making. Future work should therefore focus on improving tracking accuracy of movements that are out of plane relative to the camera orientation or affected by occlusion, such as supination and pronation of the forearm.

Extraction of Premature Newborns' Spontaneous Cries in the Real Context of Neonatal Intensive Care Units

Cry analysis is an important tool to evaluate the development of preterm infants. However, the context of Neonatal Intensive Care Units is challenging, since a wide variety of sounds can occur (e.g., alarms and adult voices). In this paper, a method to extract cries is proposed. It is based on an initial segmentation between silence and sound events, followed by feature extraction on the resulting audio segments and a cry and non-cry classification. A database of 198 cry events coming from 21 newborns and 439 non-cry events was created. Then, a set of features—including Mel-Frequency Cepstral Coefficients—issued from principal component analysis, was computed to describe each audio segment. For the first time in cry analysis, noise was handled using harmonic plus noise analysis. Several machine learning models have been compared. The K-Nearest Neighbours approach showed the best results with a precision of 92.9%. To test the approach in a monitoring application, 412 h of recordings were automatically processed. The cries automatically selected were replayed and a precision of 92.2% was obtained. The impact of errors on the fundamental frequency characterisation was also studied. Results show that despite a difficult context, automatic cry extraction for non-invasive monitoring of vocal development of preterm infants is achievable.

Video-Based Automatic Baby Motion Analysis for Early Neurological Disorder Diagnosis: State of the Art and Future Directions

Neurodevelopmental disorders (NDD) are impairments of the growth and development of the brain and/or central nervous system. In the light of clinical findings on early diagnosis of NDD and prompted by recent advances in hardware and software technologies, several researchers tried to introduce automatic systems to analyse the baby's movement, even in cribs. Traditional technologies for automatic baby motion analysis leverage contact sensors. Alternatively, remotely acquired video data (e.g., RGB or depth) can be used, with or without active/passive markers positioned on the body. Markerless approaches are easier to set up and maintain (without any human intervention) and they work well on non-collaborative users, making them the most suitable technologies for clinical applications involving children. On the other hand, they require complex computational strategies for extracting knowledge from data, and then, they strongly depend on advances in computer vision and machine learning, which are among the most expanding areas of research. As a consequence, also markerless video-based analysis of movements in children for NDD has been rapidly expanding but, to the best of our knowledge, there is not yet a survey paper providing a broad overview of how recent scientific developments impacted it. This paper tries to fill this gap and it lists specifically designed data acquisition tools and publicly available datasets as well. Besides, it gives a glimpse of the most promising techniques in computer vision, machine learning and pattern recognition which could be profitably exploited for children motion analysis in videos.

Voxyvi: A system for long-term audio and video acquisitions in neonatal intensive care units

BACKGROUND

In the European Union, 300,000 newborn babies are born prematurely every year. Their care is ensured in Neonatal Intensive Care Units (NICU) where vital signs are constantly monitored. In addition, other descriptors such as motion, facial and vocal activities have been shown to be essential to assess neurobehavioral development.

AIM

In the scope of the European project Digi-NewB, we aimed to develop and evaluate a new audio-video device designed to non-invasively acquire multi-modal data (audio, video and thermal images), while fitting the wide variety of bedding environment in NICU.

METHODS

Firstly, a multimodal system and associated software and guidelines to collect data in neonatal intensive care unit were proposed. Secondly, methods for post-evaluation of the acquisition phase were developed, including the study of clinician feedback and a qualitative analysis of the data.

RESULTS

The deployment of 19 acquisition devices in six French hospitals allowed to record more than 500 newborns of different gestational and postmenstrual ages. After the acquisition phase, clinical feedback was mostly positive. In addition, quality of more than 300 recordings was inspected and showed that 77% of the data is exploitable. In depth, the percentage of sole presence of the newborn was estimated at 62% within recordings.

CONCLUSIONS

This study demonstrates that audio-video acquisitions are feasible on a large scale in real life in NICU. The experience also allowed us to make a clear observation of the requirements and challenges that will have to be overcome in order to set up audio-video monitoring methods.

Spontaneous movements in the newborns: a tool of quantitative video analysis of preterm babies

BACKGROUND AND OBJECTIVES

The number of preterm babies is steadily growing world-wide and these neonates are at risk of neuro-motor-cognitive deficits. The observation of spontaneous movements in the first three months of age is known to predict such risk. However, the analysis by specifically trained physiotherapists is not suited for the clinical routine, motivating the development of simple computerized video analysis systems, integrated with a well-structured Biobank to make available for preterm babies a growing service with diagnostic, prognostic and epidemiological purposes.

METHODS

MIMAS (Markerless Infant Movement Analysis System) is a simple, low-cost system of video analysis of spontaneous movements of newborns in their natural environment, based on a single standard RGB camera, without markers attached to the body. The original videos are transformed into binarized sequences highlighting the silhouette of the baby, in order to minimize the illumination effects and increase the robustness of the analysis; such sequences are then coded by a large set of parameters (39) related to the spatial and spectral changes of the silhouette. The parameter vectors of each baby were stored in the Biobank together with related clinical information.

RESULTS

The preliminary test of the system was carried out at the Gaslini Pediatric Hospital in Genoa, where 46 preterm (PT) and 21 full-term (FT) babies (as controls) were recorded at birth (T0) and 8-12 weeks thereafter (T1). A simple statistical analysis of the data showed that the coded parameters are sensitive to the degree of maturation of the newborns (comparing T0 with T1, for both PT and FT babies), and to the conditions at birth (PT vs. FT at T0), whereas this difference tends to vanish at T1. Moreover, the coding method seems also able to detect the few 'abnormal' preterm babies in the PT populations that were analyzed as specific case studies.

CONCLUSIONS

Preliminary results motivate the adoption of this tool in clinical practice allowing for a systematic accumulation of cases in the Biobank, thus for improving the accuracy of data analysis performed by MIMAS and ultimately allowing the adoption of data mining techniques.

General Movement Assessment from videos of computed 3D infant body models is equally effective compared to conventional RGB video rating

BACKGROUND

General Movement Assessment (GMA) is a powerful tool to predict Cerebral Palsy (CP). Yet, GMA requires substantial training challenging its broad implementation in clinical routine. This inspired a world-wide quest for automated GMA.

AIMS

To test whether a low-cost, marker-less system for three-dimensional motion capture from RGB depth sequences using a whole body infant model may serve as the basis for automated GMA.

STUDY DESIGN

Clinical case study at an academic neurodevelopmental outpatient clinic.

SUBJECTS

Twenty-nine high risk infants were assessed at their clinical follow-up at 2-4 month corrected age (CA). Their neurodevelopmental outcome was assessed regularly up to 12-31 months CA.

OUTCOME MEASURES

GMA according to Hadders-Algra by a masked GMA-expert of conventional and computed 3D body model ("SMIL motion") videos of the same GMs. Agreement between both GMAs was tested using dichotomous and graded scaling with Kappa and intraclass correlations, respectively. Sensitivity and specificity to predict CP at ≥12 months CA were assessed.

RESULTS

Agreement of the two GMA ratings was moderate-good for GM-complexity (κ = 0.58; ICC = 0.874 [95%CI 0.730; 0.941]) and substantial-good for fidgety movements (FMs; Kappa = 0.78, ICC = 0.926 [95%CI 0.843; 0.965]). Five children were diagnosed with CP (four bilateral, one unilateral CP). The GMs of the child with unilateral CP were twice rated as mildly abnormal with FMs. GM-complexity and somewhat less FMs, of both conventional and SMIL motion videos predicted bilateral CP comparably to published literature.

CONCLUSIONS

Our computed infant 3D full body model is an attractive starting point for automated GMA in infants at risk of CP.

Movidea: A Software Package for Automatic Video Analysis of Movements in Infants at Risk for Neurodevelopmental Disorders

Early detecting the presence of neurodevelopmental disorders plays an important role in the effectiveness of the treatment. In this paper, we present a novel tool to extract motion features using single camera video recordings of infants. The Movidea software was developed to allow the operator to track the movement of end-effectors of infants in free moving conditions and extract movement features automatically. Movidea was used by different operators to analyze a set of video recordings and its performance was evaluated. The results showed that Movidea performance did not vary with the operator, and the tracking was also stable in home-video recordings. Even if the setup allowed for a two-dimensional analysis, most of the informative content of the movement was maintained. The reliability of the measures and features extracted, as well as the easiness of use, may boost the uptake of the proposed solution in clinical settings. Movidea overcomes the current limitation in the clinical practice in early detection of neurodevelopmental disorders by providing objective measures based on reliable data, and adds a new tool for the motor analysis of infants through unobtrusive technology.

Detection of Atypical and Typical Infant Movements using Computer-based Video Analysis

The diagnosis of cerebral palsy (CP) is difficult before 2 years of age. The general movements assessment (GMA) is a method for predicting CP from the spontaneous movements of infants in the first months of life. This assessment has shown high accuracy in predicting CP, but its use is limited by a lack of trained clinicians and its subjective nature. An objective and cost-effective alternative is the automatic videobased assessment of infant movements. Retrospective videos with clinical GMA outcomes were evaluated against eligibility criteria for the automatic analysis consisting of a skin model for segmentation and large displacement optical flow (LDOF) for motion tracking. Kinematic features were extracted to classify the movements as typical or atypical using different classification algorithms. Preliminary classification results obtained from the analysis of 127 videos of preterm infants showed up to 92% of accuracy in predicting CP. A computerbased assessment would provide clinicians with an objective tool for early diagnosis of CP, to facilitate early intervention and improve functional outcomes.

Video and audio processing in paediatrics: a review

OBJECTIVE

Video and sound acquisition and processing technologies have seen great improvements in recent decades, with many applications in the biomedical area. The aim of this paper is to review the overall state of the art of advances within these topics in paediatrics and to evaluate their potential application for monitoring in the neonatal intensive care unit (NICU).

APPROACH

For this purpose, more than 150 papers dealing with video and audio processing were reviewed. For both topics, clinical applications are described according to the considered cohorts-full-term newborns, infants and toddlers or preterm newborns. Then, processing methods are presented, in terms of data acquisition, feature extraction and characterization.

MAIN RESULTS

The paper first focuses on the exploitation of video recordings; these began to be automatically processed in the 2000s and we show that they have mainly been used to characterize infant motion. Other applications, including respiration and heart rate estimation and facial analysis, are also presented. Audio processing is then reviewed, with a focus on the analysis of crying. The first studies in this field focused on induced-pain cries and the newest ones deal with spontaneous cries; the analyses are mainly based on frequency features. Then, some papers dealing with non-cry signals are also discussed.

SIGNIFICANCE

Finally, we show that even if recent improvements in digital video and signal processing allow for increased automation of processing, the context of the NICU makes a fully automated analysis of long recordings problematic. A few proposals for overcoming some of the limitations are given.

A Novel Way to Measure and Predict Development: A Heuristic Approach to Facilitate the Early Detection of Neurodevelopmental Disorders

PURPOSE OF REVIEW

Substantial research exists focusing on the various aspects and domains of early human development. However, there is a clear blind spot in early postnatal development when dealing with neurodevelopmental disorders, especially those that manifest themselves clinically only in late infancy or even in childhood.

RECENT FINDINGS

This early developmental period may represent an important timeframe to study these disorders but has historically received far less research attention. We believe that only a comprehensive interdisciplinary approach will enable us to detect and delineate specific parameters for specific neurodevelopmental disorders at a very early age to improve early detection/diagnosis, enable prospective studies and eventually facilitate randomised trials of early intervention. In this article, we propose a dynamic framework for characterising neurofunctional biomarkers associated with specific disorders in the development of infants and children. We have named this automated detection 'Fingerprint Model', suggesting one possible approach to accurately and early identify neurodevelopmental disorders.