Automatic and Continuous Discomfort Detection for Premature Infants in a NICU Using Video-Based Motion Analysis

Living-Skin Detection Based on Spatio-Temporal Analysis of Structured Light Pattern

Living-skin detection is an important step for imaging photoplethysmography and biometric anti-spoofing. In this paper, we propose a new approach that exploits spatio-temporal characteristics of structured light patterns projected on the skin surface for living-skin detection. We observed that due to the interactions between laser photons and tissues inside a multi-layer skin structure, the frequency-domain sharpness feature of laser spots on skin and non-skin surfaces exhibits clear difference. Additionally, the subtle physiological motion of living-skin causes laser interference, leading to brightness fluctuations of laser spots projected on the skin surface. Based on these two observations, we designed a new living-skin detection algorithm to distinguish skin from non-skin using spatio-temporal features of structured laser spots. Experiments in the dark chamber and Neonatal Intensive Care Unit (NICU) demonstrated that the proposed setup and method performed well, achieving a precision of 85.32%, recall of 83.87%, and F1-score of 83.03% averaged over these two scenes. Compared to the approach that only leverages the property of multilayer skin structure, the hybrid approach obtains an averaged improvement of 8.18% in precision, 3.93% in recall, and 8.64% in F1-score. These results validate the efficacy of using frequency domain sharpness and brightness fluctuations to augment the features of living-skin tissues irradiated by structured light, providing a solid basis for structured light based physiological imaging.

Video-Based versus On-Site Neonatal Pain Assessment in Neonatal Intensive Care Units: The Impact of Video-Based Neonatal Pain Assessment in Real-World Scenario on Pain Diagnosis and Its Artificial Intelligence Application

BACKGROUND

Neonatal pain assessment (NPA) represents a huge global problem of essential importance, as a timely and accurate assessment of neonatal pain is indispensable for implementing pain management.

PURPOSE

To investigate the consistency of pain scores derived through video-based NPA (VB-NPA) and on-site NPA (OS-NPA), providing the scientific foundation and feasibility of adopting VB-NPA results in a real-world scenario as the gold standard for neonatal pain in clinical studies and labels for artificial intelligence (AI)-based NPA (AI-NPA) applications.

SETTING

A total of 598 neonates were recruited from a pediatric hospital in China.

METHODS

This observational study recorded 598 neonates who underwent one of 10 painful procedures, including arterial blood sampling, heel blood sampling, fingertip blood sampling, intravenous injection, subcutaneous injection, peripheral intravenous cannulation, nasopharyngeal suctioning, retention enema, adhesive removal, and wound dressing. Two experienced nurses performed OS-NPA and VB-NPA at a 10-day interval through double-blind scoring using the Neonatal Infant Pain Scale to evaluate the pain level of the neonates. Intra-rater and inter-rater reliability were calculated and analyzed, and a paired samples t-test was used to explore the bias and consistency of the assessors' pain scores derived through OS-NPA and VB-NPA. The impact of different label sources was evaluated using three state-of-the-art AI methods trained with labels given by OS-NPA and VB-NPA, respectively.

RESULTS

The intra-rater reliability of the same assessor was 0.976-0.983 across different times, as measured by the intraclass correlation coefficient. The inter-rater reliability was 0.983 for single measures and 0.992 for average measures. No significant differences were observed between the OS-NPA scores and the assessment of an independent VB-NPA assessor. The different label sources only caused a limited accuracy loss of 0.022-0.044 for the three AI methods.

CONCLUSION

VB-NPA in a real-world scenario is an effective way to assess neonatal pain due to its high intra-rater and inter-rater reliability compared to OS-NPA and could be used for the labeling of large-scale NPA video databases for clinical studies and AI training.

Continuous sensing and quantification of body motion in infants: A systematic review

Abnormal body motion in infants may be associated with neurodevelopmental delay or critical illness. In contrast to continuous patient monitoring of the basic vitals, the body motion of infants is only determined by discrete periodic clinical observations of caregivers, leaving the infants unattended for observation for a longer time. One step to fill this gap is to introduce and compare different sensing technologies that are suitable for continuous infant body motion quantification. Therefore, we conducted this systematic review for infant body motion quantification based on the PRISMA method (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). In this systematic review, we introduce and compare several sensing technologies with motion quantification in different clinical applications. We discuss the pros and cons of each sensing technology for motion quantification. Additionally, we highlight the clinical value and prospects of infant motion monitoring. Finally, we provide suggestions with specific needs in clinical practice, which can be referred by clinical users for their implementation. Our findings suggest that motion quantification can improve the performance of vital sign monitoring, and can provide clinical value to the diagnosis of complications in infants.

Multimodality Video Acquisition System for the Assessment of Vital Distress in Children

In children, vital distress events, particularly respiratory, go unrecognized. To develop a standard model for automated assessment of vital distress in children, we aimed to construct a prospective high-quality video database for critically ill children in a pediatric intensive care unit (PICU) setting. The videos were acquired automatically through a secure web application with an application programming interface (API). The purpose of this article is to describe the data acquisition process from each PICU room to the research electronic database. Using an Azure Kinect DK and a Flir Lepton 3.5 LWIR attached to a Jetson Xavier NX board and the network architecture of our PICU, we have implemented an ongoing high-fidelity prospectively collected video database for research, monitoring, and diagnostic purposes. This infrastructure offers the opportunity to develop algorithms (including computational models) to quantify vital distress in order to evaluate vital distress events. More than 290 RGB, thermographic, and point cloud videos of each 30 s have been recorded in the database. Each recording is linked to the patient's numerical phenotype, i.e., the electronic medical health record and high-resolution medical database of our research center. The ultimate goal is to develop and validate algorithms to detect vital distress in real time, both for inpatient care and outpatient management.

A self-training automatic infant-cry detector

Infant cry is one of the first distinctive and informative life signals observed after birth. Neonatologists and automatic assistive systems can analyse infant cry to early-detect pathologies. These analyses extensively use reference expert-curated databases containing annotated infant-cry audio samples. However, these databases are not publicly accessible because of their sensitive data. Moreover, the recorded data can under-represent specific phenomena or the operational conditions required by other medical teams. Additionally, building these databases requires significant investments that few hospitals can afford. This paper describes an open-source workflow for infant-cry detection, which identifies audio segments containing high-quality infant-cry samples with no other overlapping audio events (e.g. machine noise or adult speech). It requires minimal training because it trains an LSTM-with-self-attention model on infant-cry samples automatically detected from the recorded audio through cluster analysis and HMM classification. The audio signal processing uses energy and intonation acoustic features from 100-ms segments to improve spectral robustness to noise. The workflow annotates the input audio with intervals containing infant-cry samples suited for populating a database for neonatological and early diagnosis studies. On 16 min of hospital phone-audio recordings, it reached sufficient infant-cry detection accuracy in 3 neonatal care environments (nursery—69%, sub-intensive—82%, intensive—77%) involving 20 infants subject to heterogeneous cry stimuli, and had substantial agreement with an expert’s annotation. Our workflow is a cost-effective solution, particularly suited for a sub-intensive care environment, scalable to monitor from one to many infants. It allows a hospital to build and populate an extensive high-quality infant-cry database with a minimal investment.

Asymmetric Three-dimensional Convolutions For Preterm Infants' Pose Estimation

Computer-assisted tools for preterm infants' movement monitoring in neonatal intensive care unit (NICU) could support clinicians in highlighting preterm-birth complications. With such a view, in this work we propose a deep-learning framework for preterm infants' pose estimation from depth videos acquired in the actual clinical practice. The pipeline consists of two consecutive convolutional neural networks (CNNs). The first CNN (inherited from our previous work) acts to roughly predict joints and joint-connections position, while the second CNN (Asy-regression CNN) refines such predictions to trace the limb pose. Asy-regression relies on asymmetric convolutions to temporally optimize both the training and predictions phase. Compared to its counterpart without asymmetric convolutions, Asy-regression experiences a reduction in training and prediction time of 66% , while keeping the root mean square error, computed against manual pose annotation, merely unchanged. Research mostly works to develop highly accurate models, few efforts have been invested to make the training and deployment of such models time-effective. With a view to make these monitoring technologies sustainable, here we focused on the second aspect and addressed the problem of designing a framework as trade-off between reliability and efficiency.

Camera-based discomfort detection using multi-channel attention 3D-CNN for hospitalized infants

Background

Detecting discomfort in infants is an important topic for their well-being and development. In this paper, we present an automatic and continuous video-based system for monitoring and detecting discomfort in infants.

Methods

The proposed system employs a novel and efficient 3D convolutional neural network (CNN), which achieves an end-to-end solution without the conventional face detection and tracking steps. In the scheme of this study, we thoroughly investigate the video characteristics (e.g., intensity images and motion images) and CNN architectures (e.g., 2D and 3D) for infant discomfort detection. The realized improvements of the 3D-CNN are based on capturing both the motion and the facial expression information of the infants.

Results

The performance of the system is assessed using videos recorded from 24 hospitalized infants by visualizing receiver operating characteristic (ROC) curves and measuring the values of area under the ROC curve (AUC). Additional performance metrics (labeling accuracy) are also calculated. Experimental results show that the proposed system achieves an AUC of 0.99, while the overall labeling accuracy is 0.98.

Conclusions

These results confirms the robustness by using the 3D-CNN for infant discomfort monitoring and capturing both motion and facial expressions simultaneously.

Tic Detection in Tourette Syndrome Patients Based on Unsupervised Visual Feature Learning

A clinical diagnosis of tic disorder involves several complex processes, among which observation and evaluation of patient behavior usually require considerable time and effective cooperation between the doctor and the patient. The existing assessment scale has been simplified into qualitative and quantitative assessments of movements and sound twitches over a certain period, but it must still be completed manually. Therefore, we attempt to find an automatic method for detecting tic movement to assist in diagnosis and evaluation. Based on real clinical data, we propose a deep learning architecture that combines both unsupervised and supervised learning methods and learns features from videos for tic motion detection. The model is trained using leave-one-subject-out cross-validation for both binary and multiclass classification tasks. For these tasks, the model reaches average recognition precisions of 86.33% and 86.26% and recalls of 77.07% and 78.78%, respectively. The visualization of features learned from the unsupervised stage indicates the distinguishability of the two types of tics and the nontic. Further evaluation results suggest its potential clinical application for auxiliary diagnoses and evaluations of treatment effects.

Towards Continuous Camera-Based Respiration Monitoring in Infants

Aiming at continuous unobtrusive respiration monitoring, motion robustness is paramount. However, some types of motion can completely hide the respiration information and the detection of these events is required to avoid incorrect rate estimations. Therefore, this work proposes a motion detector optimized to specifically detect severe motion of infants combined with a respiration rate detection strategy based on automatic pixels selection, which proved to be robust to motion of the infants involving head and limbs. A dataset including both thermal and RGB (Red Green Blue) videos was used amounting to a total of 43 h acquired on 17 infants. The method was successfully applied to both RGB and thermal videos and compared to the chest impedance signal. The Mean Absolute Error (MAE) in segments where some motion is present was 1.16 and 1.97 breaths/min higher than the MAE in the ideal moments where the infants were still for testing and validation set, respectively. Overall, the average MAE on the testing and validation set are 3.31 breaths/min and 5.36 breaths/min, using 64.00% and 69.65% of the included video segments (segments containing events such as interventions were excluded based on a manual annotation), respectively. Moreover, we highlight challenges that need to be overcome for continuous camera-based respiration monitoring. The method can be applied to different camera modalities, does not require skin visibility, and is robust to some motion of the infants.

Voxyvi: A system for long-term audio and video acquisitions in neonatal intensive care units

BACKGROUND

In the European Union, 300,000 newborn babies are born prematurely every year. Their care is ensured in Neonatal Intensive Care Units (NICU) where vital signs are constantly monitored. In addition, other descriptors such as motion, facial and vocal activities have been shown to be essential to assess neurobehavioral development.

AIM

In the scope of the European project Digi-NewB, we aimed to develop and evaluate a new audio-video device designed to non-invasively acquire multi-modal data (audio, video and thermal images), while fitting the wide variety of bedding environment in NICU.

METHODS

Firstly, a multimodal system and associated software and guidelines to collect data in neonatal intensive care unit were proposed. Secondly, methods for post-evaluation of the acquisition phase were developed, including the study of clinician feedback and a qualitative analysis of the data.

RESULTS

The deployment of 19 acquisition devices in six French hospitals allowed to record more than 500 newborns of different gestational and postmenstrual ages. After the acquisition phase, clinical feedback was mostly positive. In addition, quality of more than 300 recordings was inspected and showed that 77% of the data is exploitable. In depth, the percentage of sole presence of the newborn was estimated at 62% within recordings.

CONCLUSIONS

This study demonstrates that audio-video acquisitions are feasible on a large scale in real life in NICU. The experience also allowed us to make a clear observation of the requirements and challenges that will have to be overcome in order to set up audio-video monitoring methods.

Machine Learning-Based Automatic Classification of Video Recorded Neonatal Manipulations and Associated Physiological Parameters: A Feasibility Study

Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO₂)) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks’ gestation, diaper changes were associated with significant changes in HR and SpO₂, and, for neonates ≥32 weeks’ gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.