Preterm Infants' Pose Estimation With Spatio-Temporal Features

Comparison of marker-less 2D image-based methods for infant pose estimation

In this study we compare the performance of available generic- and specialized infant-pose estimators for a video-based automated general movement assessment (GMA), and the choice of viewing angle for optimal recordings, i.e., conventional diagonal view used in GMA vs. top-down view. We used 4500 annotated video-frames from 75 recordings of infant spontaneous motor functions from 4 to 16 weeks. To determine which pose estimation method and camera angle yield the best pose estimation accuracy on infants in a GMA related setting, the error with respect to human annotations and the percentage of correct key-points (PCK) were computed and compared. The results show that the best performing generic model trained on adults, ViTPose, also performs best on infants. We see no improvement from using specific infant-pose estimators over the generic pose estimators on our infant dataset. However, when retraining a generic model on our data, there is a significant improvement in pose estimation accuracy. This indicates limited generalization capabilities of infant-pose estimators to other infant datasets, meaning that one should be careful when choosing infant pose estimators and using them on infant datasets which they were not trained on. The pose estimation accuracy obtained from the top-down view is significantly better than that obtained from the diagonal view (the standard view for GMA). This suggests that a top-down view should be included in recording setups for automated GMA research.

The Role of Artificial Intelligence in Pediatric Intensive Care: A Systematic Review

Pediatric intensive care units (PICUs) could transform due to artificial intelligence (AI), which could improve patient outcomes, increase diagnostic accuracy, and streamline repetitive procedures. The goal of this systematic review was to outline how AI can be used to enhance any health outcomes in pediatric intensive care. We searched four databases (PubMed, Scopus, Web of Science, and IEEE Xplore) for relevant studies using a predefined systematic search. We found 267 studies in these four databases. The studies were first screened to remove the duplicates and then screened by titles to remove irrelevant studies. The studies were further screened based on inclusion and exclusion criteria, in which 32 studies were found suitable for inclusion in this study. The studies were assessed for risk of bias using the Prediction Model Risk Of Bias Assessment Tool (PROBAST) tool. After AI was implemented, almost 22% (n = 7) of studies showed an immediate effect and enhanced health outcomes. A small number of studies involved AI implementation in actual PICUs, while the majority focused on experimental testing. AI models outperformed conventional clinical modalities among the remaining 78% (n = 25) and might have indirectly impacted patient outcomes. Significant variation in metrics and standardization was found when health outcomes were quantitatively assessed using statistical measures, including specificity (38%; n = 12) and area under the receiver operating characteristic curve (AUROC) (56%; n = 18). There are not sufficient studies showing that AI has significantly enhanced pediatric critical care patients' health outcomes. To evaluate AI's impact, more prospective, experimental research is required, utilizing verified outcome measures, defined metrics, and established application frameworks.

A Deep-Learning Approach for Vocal Fold Pose Estimation in Videoendoscopy

Accurate vocal fold (VF) pose estimation is crucial for diagnosing larynx diseases that can eventually lead to VF paralysis. The videoendoscopic examination is used to assess VF motility, usually estimating the change in the anterior glottic angle (AGA). This is a subjective and time-consuming procedure requiring extensive expertise. This research proposes a deep learning framework to estimate VF pose from laryngoscopy frames acquired in the actual clinical practice. The framework performs heatmap regression relying on three anatomically relevant keypoints as a prior for AGA computation, which is estimated from the coordinates of the predicted points. The assessment of the proposed framework is performed using a newly collected dataset of 471 laryngoscopy frames from 124 patients, 28 of whom with cancer. The framework was tested in various configurations and compared with other state-of-the-art approaches (direct keypoints regression and glottal segmentation) for both pose estimation, and AGA evaluation. The proposed framework obtained the lowest root mean square error (RMSE) computed on all the keypoints (5.09, 6.56, and 6.40 pixels, respectively) among all the models tested for VF pose estimation. Also for the AGA evaluation, heatmap regression reached the lowest mean average error (MAE) ( 5 . 87 ∘ ). Results show that relying on keypoints heatmap regression allows to perform VF pose estimation with a small error, overcoming drawbacks of state-of-the-art algorithms, especially in challenging images such as pathologic subjects, presence of noise, and occlusion.

Artificial intelligence detects awareness of functional relation with the environment in 3 month old babies

A recent experiment probed how purposeful action emerges in early life by manipulating infants' functional connection to an object in the environment (i.e., tethering an infant's foot to a colorful mobile). Vicon motion capture data from multiple infant joints were used here to create Histograms of Joint Displacements (HJDs) to generate pose-based descriptors for 3D infant spatial trajectories. Using HJDs as inputs, machine and deep learning systems were tasked with classifying the experimental state from which snippets of movement data were sampled. The architectures tested included k-Nearest Neighbour (kNN), Linear Discriminant Analysis (LDA), Fully connected network (FCNet), 1D-Convolutional Neural Network (1D-Conv), 1D-Capsule Network (1D-CapsNet), 2D-Conv and 2D-CapsNet. Sliding window scenarios were used for temporal analysis to search for topological changes in infant movement related to functional context. kNN and LDA achieved higher classification accuracy with single joint features, while deep learning approaches, particularly 2D-CapsNet, achieved higher accuracy on full-body features. For each AI architecture tested, measures of foot activity displayed the most distinct and coherent pattern alterations across different experimental stages (reflected in the highest classification accuracy rate), indicating that interaction with the world impacts the infant behaviour most at the site of organism~world connection.

Spatio-temporal layers based intra-operative stereo depth estimation network via hierarchical prediction and progressive training

BACKGROUND AND OBJECTIVE

Safety of robotic surgery can be enhanced through augmented vision or artificial constraints to the robotl motion, and intra-operative depth estimation is the cornerstone of these applications because it provides precise position information of surgical scenes in 3D space. High-quality depth estimation of endoscopic scenes has been a valuable issue, and the development of deep learning provides more possibility and potential to address this issue.

METHODS

In this paper, a deep learning-based approach is proposed to recover 3D information of intra-operative scenes. To this aim, a fully 3D encoder-decoder network integrating spatio-temporal layers is designed, and it adopts hierarchical prediction and progressive learning to enhance prediction accuracy and shorten training time.

RESULTS

Our network gets the depth estimation accuracy of MAE 2.55±1.51 (mm) and RMSE 5.23±1.40 (mm) using 8 surgical videos with a resolution of 1280×1024, which performs better compared with six other state-of-the-art methods that were trained on the same data.

CONCLUSIONS

Our network can implement a promising depth estimation performance in intra-operative scenes using stereo images, allowing the integration in robot-assisted surgery to enhance safety.

FRSR: Framework for real-time scene reconstruction in robot-assisted minimally invasive surgery

3D reconstruction of the intra-operative scenes provides precise position information which is the foundation of various safety related applications in robot-assisted surgery, such as augmented reality. Herein, a framework integrated into a known surgical system is proposed to enhance the safety of robotic surgery. In this paper, we present a scene reconstruction framework to restore the 3D information of the surgical site in real time. In particular, a lightweight encoder-decoder network is designed to perform disparity estimation, which is the key component of the scene reconstruction framework. The stereo endoscope of da Vinci Research Kit (dVRK) is adopted to explore the feasibility of the proposed approach, and it provides the possibility for the migration to other Robot Operating System (ROS) based robot platforms due to the strong independence on hardware. The framework is evaluated using three different scenarios, including a public dataset (3018 pairs of endoscopic images), the scene from the dVRK endoscope in our lab as well as a self-made clinical dataset captured from an oncology hospital. Experimental results show that the proposed framework can reconstruct 3D surgical scenes in real time (25 FPS), and achieve high accuracy (2.69 ± 1.48 mm in MAE, 5.47 ± 1.34 mm in RMSE and 0.41 ± 0.23 in SRE, respectively). It demonstrates that our framework can reconstruct intra-operative scenes with high reliability of both accuracy and speed, and the validation of clinical data also shows its potential in surgery. This work enhances the state of art in 3D intra-operative scene reconstruction based on medical robot platforms. The clinical dataset has been released to promote the development of scene reconstruction in the medical image community.

Classifying Vocal Folds Fixation from Endoscopic Videos with Machine Learning

Vocal folds motility evaluation is paramount in both the assessment of functional deficits and in the accurate staging of neoplastic disease of the glottis. Diagnostic endoscopy, and in particular videoendoscopy, is nowadays the method through which the motility is estimated. The clinical diagnosis, however, relies on the examination of the videoendoscopic frames, which is a subjective and professional-dependent task. Hence, a more rigorous, objective, reliable, and repeatable method is needed. To support clinicians, this paper proposes a machine learning (ML) approach for vocal cords motility classification. From the endoscopic videos of 186 patients with both vocal cords preserved motility and fixation, a dataset of 558 images relative to the two classes was extracted. Successively, a number of features was retrieved from the images and used to train and test four well-grounded ML classifiers. From test results, the best performance was achieved using XGBoost, with precision = 0.82, recall = 0.82, F1 score = 0.82, and accuracy = 0.82. After comparing the most relevant ML models, we believe that this approach could provide precise and reliable support to clinical evaluation.Clinical Relevance- This research represents an important advancement in the state-of-the-art of computer-assisted otolaryngology, to develop an effective tool for motility assessment in the clinical practice.

NeoCam: An Edge-Cloud Platform for Non-Invasive Real-Time Monitoring in Neonatal Intensive Care Units

In this work we introduce NeoCam, an open source hardware-software platform for video-based monitoring of preterms infants in Neonatal Intensive Care Units (NICUs). NeoCam includes an edge computing device that performs video acquisition and processing in real-time. Compared to other proposed solutions, it has the advantage of handling data more efficiently by performing most of the processing on the device, including proper anonymisation for better compliance with privacy regulations. In addition, it allows to perform various video analysis tasks of clinical interest in parallel at speeds of between 20 and 30 frames-per-second. We introduce algorithms to measure without contact the breathing rate, motor activity, body pose and emotional status of the infants. For breathing rate, our system shows good agreement with existing methods provided there is sufficient light and proper imaging conditions. Models for motor activity and stress detection are new to the best of our knowledge. NeoCam has been tested on preterms in the NICU of the University Hospital Puerta del Mar (Cádiz, Spain), and we report the lessons learned from this trial.

Open video data sharing in developmental science and clinical practice

In behavioral research and clinical practice video data has rarely been shared or pooled across sites due to ethical concerns of confidentiality, although the need of shared large-scaled datasets remains increasing. This demand is even more imperative when data-heavy computer-based approaches are involved. To share data while abiding by privacy protection rules, a critical question arises whether efforts at data de-identification reduce data utility? We addressed this question by showcasing an established and video-based diagnostic tool for detecting neurological deficits. We demonstrated for the first time that, for analyzing infant neuromotor functions, pseudonymization by face-blurring video recordings is a viable approach. The redaction did not affect classification accuracy for either human assessors or artificial intelligence methods, suggesting an adequate and easy-to-apply solution for sharing behavioral video data. Our work shall encourage more innovative solutions to share and merge stand-alone video datasets into large data pools to advance science and public health.

TwinEDA: a sustainable deep-learning approach for limb-position estimation in preterm infants' depth images

Early diagnosis of neurodevelopmental impairments in preterm infants is currently based on the visual analysis of newborns' motion patterns by trained operators. To help automatize this time-consuming and qualitative procedure, we propose a sustainable deep-learning algorithm for accurate limb-pose estimation from depth images. The algorithm consists of a convolutional neural network (TwinEDA) relying on architectural blocks that require limited computation while ensuring high performance in prediction. To ascertain its low computational costs and assess its application in on-the-edge computing, TwinEDA was additionally deployed on a cost-effective single-board computer. The network was validated on a dataset of 27,000 depth video frames collected during the actual clinical practice from 27 preterm infants. When compared to the main state-of-the-art competitor, TwinEDA is twice as fast to predict a single depth frame and four times as light in terms of memory, while performing similarly in terms of Dice similarity coefficient (0.88). This result suggests that the pursuit of efficiency does not imply the detriment of performance. This work is among the first to propose an automatic and sustainable limb-position estimation approach for preterm infants. This represents a significant step towards the development of broadly accessible clinical monitoring applications.

Artificial intelligence in the diagnosis of necrotising enterocolitis in newborns

Necrotising enterocolitis (NEC) is one of the most common diseases in neonates and predominantly affects premature or very-low-birth-weight infants. Diagnosis is difficult and needed in hours since the first symptom onset for the best therapeutic effects. Artificial intelligence (AI) may play a significant role in NEC diagnosis. A literature search on the use of AI in the diagnosis of NEC was performed. Four databases (PubMed, Embase, arXiv, and IEEE Xplore) were searched with the appropriate MeSH terms. The search yielded 118 publications that were reduced to 8 after screening and checking for eligibility. Of the eight, five used classic machine learning (ML), and three were on the topic of deep ML. Most publications showed promising results. However, no publications with evident clinical benefits were found. Datasets used for training and testing AI systems were small and typically came from a single institution. The potential of AI to improve the diagnosis of NEC is evident. The body of literature on this topic is scarce, and more research in this area is needed, especially with a focus on clinical utility. Cross-institutional data for the training and testing of AI algorithms are required to make progress in this area. IMPACT: Only a few publications on the use of AI in NEC diagnosis are available although they offer some evidence that AI may be helpful in NEC diagnosis. AI requires large, multicentre, and multimodal datasets of high quality for model training and testing. Published results in the literature are based on data from single institutions and, as such, have limited generalisability. Large multicentre studies evaluating broad datasets are needed to evaluate the true potential of AI in diagnosing NEC in a clinical setting.

Generating depth images of preterm infants in given poses using GANs

BACKGROUND AND OBJECTIVES

The use of deep learning for preterm infant's movement monitoring has the potential to support clinicians in early recognizing motor and behavioural disorders. The development of deep learning algorithms is, however, hampered by the lack of publicly available annotated datasets.

METHODS

To mitigate the issue, this paper presents a Generative Adversarial Network-based framework to generate images of preterm infants in a given pose. The framework consists of a bibranch encoder and a conditional Generative Adversarial Network, to generate a rough image and a refined version of it, respectively.

RESULTS

Evaluation was performed on the Moving INfants In RGB-D dataset which has 12.000 depth frames from 12 preterm infants. A low Fréchet inception distance (142.9) and an inception score (2.8) close to that of real-image distribution (2.6) are obtained. The results achieved show the potentiality of the framework in generating realistic depth images of preterm infants in a given pose.

CONCLUSIONS

Pursuing research on the generation of new data may enable researchers to propose increasingly advanced and effective deep learning-based monitoring systems.

Deep learning-based quantitative analyses of spontaneous movements and their association with early neurological development in preterm infants

This study aimed to develop quantitative assessments of spontaneous movements in high-risk preterm infants based on a deep learning algorithm. Video images of spontaneous movements were recorded in very preterm infants at the term-equivalent age. The Hammersmith Infant Neurological Examination (HINE) was performed in infants at 4 months of corrected age. Joint positional data were extracted using a pretrained pose-estimation model. Complexity and similarity indices of joint angle and angular velocity in terms of sample entropy and Pearson correlation coefficient were compared between the infants with HINE < 60 and ≥ 60. Video images of spontaneous movements were recorded in 65 preterm infants at term-equivalent age. Complexity indices of joint angles and angular velocities differed between the infants with HINE < 60 and ≥ 60 and correlated positively with HINE scores in most of the joints at the upper and lower extremities (p < 0.05). Similarity indices between each joint angle or joint angular velocity did not differ between the two groups in most of the joints at the upper and lower extremities. Quantitative assessments of spontaneous movements in preterm infants are feasible using a deep learning algorithm and sample entropy. The results indicated that complexity indices of joint movements at both the upper and lower extremities can be potential candidates for detecting developmental outcomes in preterm infants.

Video-Based Automatic Baby Motion Analysis for Early Neurological Disorder Diagnosis: State of the Art and Future Directions

Neurodevelopmental disorders (NDD) are impairments of the growth and development of the brain and/or central nervous system. In the light of clinical findings on early diagnosis of NDD and prompted by recent advances in hardware and software technologies, several researchers tried to introduce automatic systems to analyse the baby's movement, even in cribs. Traditional technologies for automatic baby motion analysis leverage contact sensors. Alternatively, remotely acquired video data (e.g., RGB or depth) can be used, with or without active/passive markers positioned on the body. Markerless approaches are easier to set up and maintain (without any human intervention) and they work well on non-collaborative users, making them the most suitable technologies for clinical applications involving children. On the other hand, they require complex computational strategies for extracting knowledge from data, and then, they strongly depend on advances in computer vision and machine learning, which are among the most expanding areas of research. As a consequence, also markerless video-based analysis of movements in children for NDD has been rapidly expanding but, to the best of our knowledge, there is not yet a survey paper providing a broad overview of how recent scientific developments impacted it. This paper tries to fill this gap and it lists specifically designed data acquisition tools and publicly available datasets as well. Besides, it gives a glimpse of the most promising techniques in computer vision, machine learning and pattern recognition which could be profitably exploited for children motion analysis in videos.

Artificial Intelligence to Improve Health Outcomes in the NICU and PICU: A Systematic Review

CONTEXT

Artificial intelligence (AI) technologies are increasingly used in pediatrics and have the potential to help inpatient physicians provide high-quality care for critically ill children.

OBJECTIVE

We aimed to describe the use of AI to improve any health outcome(s) in neonatal and pediatric intensive care.

DATA SOURCE

PubMed, IEEE Xplore, Cochrane, and Web of Science databases.

STUDY SELECTION

We used peer-reviewed studies published between June 1, 2010, and May 31, 2020, in which researchers described (1) AI, (2) pediatrics, and (3) intensive care. Studies were included if researchers assessed AI use to improve at least 1 health outcome (eg, mortality).

DATA EXTRACTION

Data extraction was conducted independently by 2 researchers. Articles were categorized by direct or indirect impact of AI, defined by the European Institute of Innovation and Technology Health joint report.

RESULTS

Of the 287 publications screened, 32 met inclusion criteria. Approximately 22% (n = 7) of studies revealed a direct impact and improvement in health outcomes after AI implementation. Majority were in prototype testing, and few were deployed into an ICU setting. Among the remaining 78% (n = 25) AI models outperformed standard clinical modalities and may have indirectly influenced patient outcomes. Quantitative assessment of health outcomes using statistical measures, such as area under the receiver operating curve (56%; n = 18) and specificity (38%; n = 12), revealed marked heterogeneity in metrics and standardization.

CONCLUSIONS

Few studies have revealed that AI has directly improved health outcomes for pediatric critical care patients. Further prospective, experimental studies are needed to assess AI's impact by using established implementation frameworks, standardized metrics, and validated outcome measures.

A Pose-based Feature Fusion and Classification Framework for the Early Prediction of Cerebral Palsy in Infants

The early diagnosis of cerebral palsy is an area which has recently seen significant multi-disciplinary research. Diagnostic tools such as the General Movements Assessment (GMA), have produced some very promising results. However, the prospect of automating these processes may improve accessibility of the assessment and also enhance the understanding of movement development of infants. Previous works have established the viability of using pose-based features extracted from RGB video sequences to undertake classification of infant body movements based upon the GMA. In this paper, we propose a series of new and improved features, and a feature fusion pipeline for this classification task. We also introduce the RVI-38 dataset, a series of videos captured as part of routine clinical care. By utilising this challenging dataset we establish the robustness of several motion features for classification, subsequently informing the design of our proposed feature fusion framework based upon the GMA. We evaluate our proposed framework's classification performance using both the RVI-38 dataset and the publicly available MINI-RGBD dataset. We also implement several other methods from the literature for direct comparison using these two independent datasets. Our experimental results and feature analysis show that our proposed pose-based method performs well across both datasets. The proposed features afford us the opportunity to include finer detail than previous methods, and further model GMA specific body movements. These new features also allow us to take advantage of additional body-part specific information as a means of improving the overall classification performance, whilst retaining GMA relevant, interpretable, and shareable features.

Asymmetric Three-dimensional Convolutions For Preterm Infants' Pose Estimation

Computer-assisted tools for preterm infants' movement monitoring in neonatal intensive care unit (NICU) could support clinicians in highlighting preterm-birth complications. With such a view, in this work we propose a deep-learning framework for preterm infants' pose estimation from depth videos acquired in the actual clinical practice. The pipeline consists of two consecutive convolutional neural networks (CNNs). The first CNN (inherited from our previous work) acts to roughly predict joints and joint-connections position, while the second CNN (Asy-regression CNN) refines such predictions to trace the limb pose. Asy-regression relies on asymmetric convolutions to temporally optimize both the training and predictions phase. Compared to its counterpart without asymmetric convolutions, Asy-regression experiences a reduction in training and prediction time of 66% , while keeping the root mean square error, computed against manual pose annotation, merely unchanged. Research mostly works to develop highly accurate models, few efforts have been invested to make the training and deployment of such models time-effective. With a view to make these monitoring technologies sustainable, here we focused on the second aspect and addressed the problem of designing a framework as trade-off between reliability and efficiency.

Improving Preterm Infants' Joint Detection in Depth Images Via Dense Convolutional Neural Networks

Preterm infants' spontaneous motility is a valuable diagnostic and prognostic index of motor and cognitive impairments. Despite being recognized as crucial, preterm infant's movement assessment is mostly based on clinicians' visual inspection. The aim of this work is to present a 2D dense convolutional neural network (denseCNN) to detect preterm infant's joints in depth images acquired in neonatal intensive care units. The denseCNN allows to improve the performance of our previous model in the detection of joints and joint connections, reaching a median recall value equal to 0.839. With a view to monitor preterm infants in a scenario where computational resources are scarce, we tested the architecture on a mid-range laptop. The prediction occurs in real-time (0.014 s per image), opening up the possibility of integrating such monitoring system in a domestic environment.

Using spatial-temporal ensembles of convolutional neural networks for lumen segmentation in ureteroscopy

Purpose

Ureteroscopy is an efficient endoscopic minimally invasive technique for the diagnosis and treatment of upper tract urothelial carcinoma. During ureteroscopy, the automatic segmentation of the hollow lumen is of primary importance, since it indicates the path that the endoscope should follow. In order to obtain an accurate segmentation of the hollow lumen, this paper presents an automatic method based on convolutional neural networks (CNNs).

Methods

The proposed method is based on an ensemble of 4 parallel CNNs to simultaneously process single and multi-frame information. Of these, two architectures are taken as core-models, namely U-Net based in residual blocks (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_1$$\end{document}m1) and Mask-RCNN (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$m_2$$\end{document}m2), which are fed with single still-frames I(t). The other two models (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_1$$\end{document}M1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document}M2) are modifications of the former ones consisting on the addition of a stage which makes use of 3D convolutions to process temporal information. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_1$$\end{document}M1, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$M_2$$\end{document}M2 are fed with triplets of frames (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I(t-1)$$\end{document}I(t-1), I(t), \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I(t+1)$$\end{document}I(t+1)) to produce the segmentation for I(t).

Results

The proposed method was evaluated using a custom dataset of 11 videos (2673 frames) which were collected and manually annotated from 6 patients. We obtain a Dice similarity coefficient of 0.80, outperforming previous state-of-the-art methods.

Conclusion

The obtained results show that spatial-temporal information can be effectively exploited by the ensemble model to improve hollow lumen segmentation in ureteroscopic images. The method is effective also in the presence of poor visibility, occasional bleeding, or specular reflections.

Supplementary Information

The online version supplementary material available at 10.1007/s11548-021-02376-3.

The future of General Movement Assessment: The role of computer vision and machine learning - A scoping review

BACKGROUND

The clinical and scientific value of Prechtl general movement assessment (GMA) has been increasingly recognised, which has extended beyond the detection of cerebral palsy throughout the years. With advancing computer science, a surging interest in developing automated GMA emerges.

AIMS

In this scoping review, we focused on video-based approaches, since it remains authentic to the non-intrusive principle of the classic GMA. Specifically, we aimed to provide an overview of recent video-based approaches targeting GMs; identify their techniques for movement detection and classification; examine if the technological solutions conform to the fundamental concepts of GMA; and discuss the challenges of developing automated GMA.

METHODS AND PROCEDURES

We performed a systematic search for computer vision-based studies on GMs.

OUTCOMES AND RESULTS

We identified 40 peer-reviewed articles, most (n = 30) were published between 2017 and 2020. A wide variety of sensing, tracking, detection, and classification tools for computer vision-based GMA were found. Only a small portion of these studies applied deep learning approaches. A comprehensive comparison between data acquisition and sensing setups across the reviewed studies, highlighting limitations and advantages of each modality in performing automated GMA is provided.

CONCLUSIONS AND IMPLICATIONS

A "method-of-choice" for automated GMA does not exist. Besides creating large datasets, understanding the fundamental concepts and prerequisites of GMA is necessary for developing automated solutions. Future research shall look beyond the narrow field of detecting cerebral palsy and open up to the full potential of applying GMA to enable an even broader application.

Machine Learning-Based Automatic Classification of Video Recorded Neonatal Manipulations and Associated Physiological Parameters: A Feasibility Study

Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO₂)) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks’ gestation, diaper changes were associated with significant changes in HR and SpO₂, and, for neonates ≥32 weeks’ gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.

The babyPose dataset

The database here described contains data relevant to preterm infants' movement acquired in neonatal intensive care units (NICUs). The data consists of 16 depth videos recorded during the actual clinical practice. Each video consists of 1000 frames (i.e., 100s). The dataset was acquired at the NICU of the Salesi Hospital, Ancona (Italy). Each frame was annotated with the limb-joint location. Twelve joints were annotated, i.e., left and right shoul- der, elbow, wrist, hip, knee and ankle. The database is freely accessible at http://doi.org/10.5281/zenodo.3891404. This dataset represents a unique resource for artificial intelligence researchers that want to develop algorithms to provide healthcare professionals working in NICUs with decision support. Hence, the babyPose dataset is the first annotated dataset of depth images relevant to preterm infants' movement analysis.

Can human posture and range of motion be measured automatically by smart mobile applications?

Human posture and Range of Motion (ROM) are important components of a physical assessment and, from the collected data, it is possible to identify postural deviations such as scoliosis or joint and muscle limitations, hence identifying risks of more serious injuries. Posture assessment and ROM measures are also necessary metrics to monitor the effect of treatments used in the motor rehabilitation of patients, as well as to monitor their clinical progress. These evaluation processes are more frequently performed through visual inspection and manual palpation, which are simple and low cost methods. These methods, however, can be optimized with the use of tools such as photogrammetry and goniometry. Mobile solutions have also been developed to help health professionals to capture more objective data and with less risk of bias. Although there are already several systems proposed for assessing human posture and ROM in the literature, they have not been able to automatically identify and mark Anatomical and Segment Points (ASPs). The hypothesis presented here considers the development of a mobile application for automatic identification of ASPs by using machine learning algorithms and computer vision models associated with technologies embedded in smartphones. From ASPs identification, it will be possible to identify changes in postural alignment and ROM. In this context, our view is that an application derived from the hypothesis will serve as an additional tool to assist in the physical assessment process and, consequently, in the diagnosis of disorders related to postural and movement changes.