1
|
Huang B, Hu S, Liu Z, Lin CL, Su J, Zhao C, Wang L, Wang W. Challenges and prospects of visual contactless physiological monitoring in clinical study. NPJ Digit Med 2023; 6:231. [PMID: 38097771 PMCID: PMC10721846 DOI: 10.1038/s41746-023-00973-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Accepted: 11/21/2023] [Indexed: 12/17/2023] Open
Abstract
The monitoring of physiological parameters is a crucial topic in promoting human health and an indispensable approach for assessing physiological status and diagnosing diseases. Particularly, it holds significant value for patients who require long-term monitoring or with underlying cardiovascular disease. To this end, Visual Contactless Physiological Monitoring (VCPM) is capable of using videos recorded by a consumer camera to monitor blood volume pulse (BVP) signal, heart rate (HR), respiratory rate (RR), oxygen saturation (SpO2) and blood pressure (BP). Recently, deep learning-based pipelines have attracted numerous scholars and achieved unprecedented development. Although VCPM is still an emerging digital medical technology and presents many challenges and opportunities, it has the potential to revolutionize clinical medicine, digital health, telemedicine as well as other areas. The VCPM technology presents a viable solution that can be integrated into these systems for measuring vital parameters during video consultation, owing to its merits of contactless measurement, cost-effectiveness, user-friendly passive monitoring and the sole requirement of an off-the-shelf camera. In fact, the studies of VCPM technologies have been rocketing recently, particularly AI-based approaches, but few are employed in clinical settings. Here we provide a comprehensive overview of the applications, challenges, and prospects of VCPM from the perspective of clinical settings and AI technologies for the first time. The thorough exploration and analysis of clinical scenarios will provide profound guidance for the research and development of VCPM technologies in clinical settings.
Collapse
Affiliation(s)
- Bin Huang
- AI Research Center, Hangzhou Innovation Institute, Beihang University, 99 Juhang Rd., Binjiang Dist., Hangzhou, Zhejiang, China.
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China.
| | - Shen Hu
- Department of Obstetrics, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
- Department of Epidemiology, The Harvard T.H. Chan School of Public Health, Boston, MA, USA
| | - Zimeng Liu
- School of Automation Science and Electrical Engineering, Beihang University, Beijing, China
| | - Chun-Liang Lin
- College of Electrical Engineering and Computer Science, National Chung Hsing University, 145 Xingda Rd., South Dist., Taichung, Taiwan.
| | - Junfeng Su
- Department of General Intensive Care Unit, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
- Key Laboratory of Early Warning and Intervention of Multiple Organ Failure, China National Ministry of Education, Hangzhou, Zhejiang, China
| | - Changchen Zhao
- AI Research Center, Hangzhou Innovation Institute, Beihang University, 99 Juhang Rd., Binjiang Dist., Hangzhou, Zhejiang, China
| | - Li Wang
- Department of Rehabilitation Medicine, The First Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
| | - Wenjin Wang
- Department of Biomedical Engineering, Southern University of Science and Technology, 1088 Xueyuan Ave, Nanshan Dist., Shenzhen, Guangdong, China.
| |
Collapse
|
2
|
Peng Z, Kommers D, Liang RH, Long X, Cottaar W, Niemarkt H, Andriessen P, van Pul C. Continuous sensing and quantification of body motion in infants: A systematic review. Heliyon 2023; 9:e18234. [PMID: 37501976 PMCID: PMC10368857 DOI: 10.1016/j.heliyon.2023.e18234] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Revised: 06/26/2023] [Accepted: 07/12/2023] [Indexed: 07/29/2023] Open
Abstract
Abnormal body motion in infants may be associated with neurodevelopmental delay or critical illness. In contrast to continuous patient monitoring of the basic vitals, the body motion of infants is only determined by discrete periodic clinical observations of caregivers, leaving the infants unattended for observation for a longer time. One step to fill this gap is to introduce and compare different sensing technologies that are suitable for continuous infant body motion quantification. Therefore, we conducted this systematic review for infant body motion quantification based on the PRISMA method (Preferred Reporting Items for Systematic Reviews and Meta-Analyses). In this systematic review, we introduce and compare several sensing technologies with motion quantification in different clinical applications. We discuss the pros and cons of each sensing technology for motion quantification. Additionally, we highlight the clinical value and prospects of infant motion monitoring. Finally, we provide suggestions with specific needs in clinical practice, which can be referred by clinical users for their implementation. Our findings suggest that motion quantification can improve the performance of vital sign monitoring, and can provide clinical value to the diagnosis of complications in infants.
Collapse
Affiliation(s)
- Zheng Peng
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Clinical Physics, Máxima Medical Centre, Veldhoven, the Netherlands
| | - Deedee Kommers
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Neonatology, Máxima Medical Centre, Veldhoven, the Netherlands
| | - Rong-Hao Liang
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Industrial Design, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Xi Long
- Department of Electrical Engineering, Eindhoven University of Technology, Eindhoven, the Netherlands
- Philips Research, Eindhoven, the Netherlands
| | - Ward Cottaar
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
| | - Hendrik Niemarkt
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Neonatology, Máxima Medical Centre, Veldhoven, the Netherlands
| | - Peter Andriessen
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Neonatology, Máxima Medical Centre, Veldhoven, the Netherlands
| | - Carola van Pul
- Department of Applied Physics, Eindhoven University of Technology, Eindhoven, the Netherlands
- Department of Clinical Physics, Máxima Medical Centre, Veldhoven, the Netherlands
| |
Collapse
|
3
|
Zeng Y, Song X, Yang J, Wang W. Time-domain Features of Angular-velocity Signals for Camera-based Respiratory RoI detection: A Clinical Study in NICU. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-6. [PMID: 38083770 DOI: 10.1109/embc40787.2023.10340063] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Camera-based measurement of respiratory rate (RR) is emerging for preterm infants monitoring in Neonatal Intensive Care Units (NICU). Accurate detection of respiratory region of interest (Resp-RoI), e.g. thorax and abdomen of infants, is essential for achieving a fully-automatic solution and for high-quality RR estimation. However, the application of fast Fourier transform (FFT) for detecting Resp-RoI in premature infants may not be appropriate due to their irregular breathing patterns. This study proposes a new method for detecting Resp-RoIs in premature infants that uses time-domain features of angular-velocity of respiration. By fusing respiratory motion on orthogonal directions, the proposed method is more robust to variations of infant posture in the incubator.. In addition, using inter-beat interval (IBI) features in the time domain helps to distinguish between Resp-RoI and background. The proposed method was validated on 20 preterm infants in NICU. It obtains a clear improvement on Resp-RoI detection (RoI correspondence = 0.74) and RR estimation (MAE = 3.62 bpm) against the benchmarked approaches (maxFFT: RoI correspondence = 0.45, MAE = 5.61 bpm).
Collapse
|
4
|
Walker SB, Badke CM, Carroll MS, Honegger KS, Fawcett A, Weese-Mayer DE, Sanchez-Pinto LN. Novel approaches to capturing and using continuous cardiorespiratory physiological data in hospitalized children. Pediatr Res 2023; 93:396-404. [PMID: 36329224 DOI: 10.1038/s41390-022-02359-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/02/2022] [Revised: 08/16/2022] [Accepted: 10/11/2022] [Indexed: 11/06/2022]
Abstract
Continuous cardiorespiratory physiological monitoring is a cornerstone of care in hospitalized children. The data generated by monitoring devices coupled with machine learning could transform the way we provide care. This scoping review summarizes existing evidence on novel approaches to continuous cardiorespiratory monitoring in hospitalized children. We aimed to identify opportunities for the development of monitoring technology and the use of machine learning to analyze continuous physiological data to improve the outcomes of hospitalized children. We included original research articles published on or after January 1, 2001, involving novel approaches to collect and use continuous cardiorespiratory physiological data in hospitalized children. OVID Medline, PubMed, and Embase databases were searched. We screened 2909 articles and performed full-text extraction of 105 articles. We identified 58 articles describing novel devices or approaches, which were generally small and single-center. In addition, we identified 47 articles that described the use of continuous physiological data in prediction models, but only 7 integrated multidimensional data (e.g., demographics, laboratory results). We identified three areas for development: (1) further validation of promising novel devices; (2) more studies of models integrating multidimensional data with continuous cardiorespiratory data; and (3) further dissemination, implementation, and validation of prediction models using continuous cardiorespiratory data. IMPACT: We performed a comprehensive scoping review of novel approaches to capture and use continuous cardiorespiratory physiological data for monitoring, diagnosis, providing care, and predicting events in hospitalized infants and children, from novel devices to machine learning-based prediction models. We identified three key areas for future development: (1) further validation of promising novel devices; (2) more studies of models integrating multidimensional data with continuous cardiorespiratory data; and (3) further dissemination, implementation, and validation of prediction models using cardiorespiratory data.
Collapse
Affiliation(s)
- Sarah B Walker
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA. .,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA.
| | - Colleen M Badke
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| | - Michael S Carroll
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| | - Kyle S Honegger
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| | - Andrea Fawcett
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| | - Debra E Weese-Mayer
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| | - L Nelson Sanchez-Pinto
- Department of Pediatrics, Northwestern University Feinberg School of Medicine, Chicago, IL, USA.,Stanley Manne Children's Research Institute, Ann & Robert H. Lurie Children's Hospital of Chicago, Chicago, IL, USA
| |
Collapse
|
5
|
Grech N, Agius JC, Sciberras S, Micallef N, Camilleri K, Falzon O. Non-contact Vital Signs Monitoring in Paediatric Anaesthesia - Current Challenges and Future Direction. ACTA MEDICA (HRADEC KRALOVE) 2023; 66:39-46. [PMID: 37930092 DOI: 10.14712/18059694.2023.14] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2023]
Abstract
Non-contact vital sign monitoring is an area of increasing interest in the clinical scenario since it offers advantages over traditional monitoring using leads and wires. These advantages include reduction in transmission of infection and more freedom of movement. Yet there is a paucity of studies available in the clinical setting particularly in paediatric anaesthesia. This scoping review aims to investigate why contactless monitoring, specifically with red-green-blue cameras, is not implemented in mainstream practise. The challenges, drawbacks and limitations of non-contact vital sign monitoring, will be outlined, together with future direction on how it can potentially be implemented in the setting of paediatric anaesthesia, and in the critical care scenario.
Collapse
Affiliation(s)
- Nicole Grech
- Department of Anaesthesia and Intensive Care Medicine, Mater Dei Hospital, Malta.
| | - Jean Calleja Agius
- Department of Anatomy, Faculty of Medicine and Surgery, University of Malta
| | - Stephen Sciberras
- Department of Anaesthesia and Intensive Care Medicine, Mater Dei Hospital, Malta
| | - Neil Micallef
- Centre for Biomedical Cybernetics, Faculty of Engineering, University of Malta
| | - Kenneth Camilleri
- Centre for Biomedical Cybernetics, Faculty of Engineering, University of Malta
| | - Owen Falzon
- Centre for Biomedical Cybernetics, Faculty of Engineering, University of Malta
| |
Collapse
|
6
|
Cao L, Zhang Z, Li J, Wang Z, Ren Y, Wang Q, Huang D, Li Z. A Low-Cost Flexible Perforated Respiratory Sensor Based on Platinum for Continuous Respiratory Monitoring. MICROMACHINES 2022; 13:1743. [PMID: 36296096 PMCID: PMC9611104 DOI: 10.3390/mi13101743] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/27/2022] [Revised: 09/26/2022] [Accepted: 10/10/2022] [Indexed: 06/16/2023]
Abstract
Monitoring sleep conditions is of importance for sleep quality evaluation and sleep disease diagnosis. Accurate respiration detection provides key information about sleep conditions. Here, we propose a perforated temperature sensor that can be worn below the nasal cavity to monitor breath. The sensing system consists of two perforated temperature sensors, signal conditioning circuits, a transmission module, and a supporting analysis algorithm. The perforated structure effectively enhances the sensitivity of the system and shortens the response time. The sensor's response time is 0.07 s in air and sensitivity is 1.4‱°C-1. The device can achieve a monitoring respiratory temperature range between normal room temperature and 40 °C. The simple and standard micromachining process ensures low cost and high reproducibility. We achieved the monitoring of different breathing patterns, such as normal breathing, panting, and apnea, which can be applied to sleep breath monitoring and exercise information recording.
Collapse
Affiliation(s)
- Lu Cao
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- College of Engineering, Peking University, Beijing 100871, China
| | - Zhitong Zhang
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| | - Junshi Li
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| | - Zhongyan Wang
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| | - Yingjie Ren
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| | - Qining Wang
- College of Engineering, Peking University, Beijing 100871, China
| | - Dong Huang
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| | - Zhihong Li
- National Key Laboratory of Science and Technology on Micro/Nano Fabrication, Beijing 100871, China
- School of Integrated Circuits, Peking University, Beijing 100871, China
| |
Collapse
|
7
|
Botina-Monsalve D, Benezeth Y, Miteran J. Performance analysis of remote photoplethysmography deep filtering using long short-term memory neural network. Biomed Eng Online 2022; 21:69. [PMID: 36123747 PMCID: PMC9487135 DOI: 10.1186/s12938-022-01037-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 09/06/2022] [Indexed: 11/16/2022] Open
Abstract
Background Remote photoplethysmography (rPPG) is a technique developed to estimate heart rate using standard video cameras and ambient light. Due to the multiple sources of noise that deteriorate the quality of the signal, conventional filters such as the bandpass and wavelet-based filters are commonly used. However, after using conventional filters, some alterations remain, but interestingly an experienced eye can easily identify them. Results We studied a long short-term memory (LSTM) network in the rPPG filtering task to identify these alterations using many-to-one and many-to-many approaches. We used three public databases in intra-dataset and cross-dataset scenarios, along with different protocols to analyze the performance of the method. We demonstrate how the network can be easily trained with a set of 90 signals totaling around 45 min. On the other hand, we show the stability of the LSTM performance with six state-of-the-art rPPG methods. Conclusions This study demonstrates the superiority of the LSTM-based filter experimentally compared with conventional filters in an intra-dataset scenario. For example, we obtain on the VIPL database an MAE of 3.9 bpm, whereas conventional filtering improves performance on the same dataset from 10.3 bpm to 7.7 bpm. The cross-dataset approach presents a dependence in the network related to the average signal-to-noise ratio on the rPPG signals, where the closest signal-to-noise ratio values in the training and testing set the better. Moreover, it was demonstrated that a relatively small amount of data are sufficient to successfully train the network and outperform the results obtained by classical filters. More precisely, we have shown that about 45 min of rPPG signal could be sufficient to train an effective LSTM deep-filter. Supplementary Information The online version contains supplementary material available at 10.1186/s12938-022-01037-z.
Collapse
Affiliation(s)
| | | | - Johel Miteran
- Univ. Bourgogne Franche-Comté, ImViA EA7535, Dijon, France
| |
Collapse
|
8
|
Ahmed S, Park J, Cho SH. Effects of Receiver Beamforming for Vital Sign Measurements Using FMCW Radar at Various Distances and Angles. SENSORS (BASEL, SWITZERLAND) 2022; 22:6877. [PMID: 36146226 PMCID: PMC9503483 DOI: 10.3390/s22186877] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/20/2022] [Revised: 09/04/2022] [Accepted: 09/08/2022] [Indexed: 06/16/2023]
Abstract
Short-range millimeter wave radar sensors provide a reliable, continuous and non-contact solution for vital sign extraction. Off-The-Shelf (OTS) radars often have a directional antenna (beam) pattern. The transmitted wave has a conical main lobe, and power of the received target echoes deteriorate as we move away from the center point of the lobe. While measuring vital signs, the human subject is often located at the center of the antenna lobe. Since beamforming can increase signal quality at the side (azimuth) angles, this paper aims to provide an experimental comparison of vital sign extraction with and without beamforming. The experimental confirmation that beamforming can decrease the error in the vital sign extraction through radar has so far not been performed by researchers. A simple, yet effective receiver beamformer was designed and a concurrent measurement with and without beamforming was made for the comparative analysis. Measurements were made at three different distances and five different arrival angles, and the preliminary results suggest that as the observation angle increases, the effectiveness of beamforming increases. At an extreme angle of 40 degrees, the beamforming showed above 20% improvement in heart rate estimation. Heart rate measurement error was reduced significantly in comparison with the breathing rate.
Collapse
|
9
|
Yang X, Zhang Z, Huang Y, Zheng Y, Shen Y. Using a graph-based image segmentation algorithm for remote vital sign estimation and monitoring. Sci Rep 2022; 12:15197. [PMID: 36071124 PMCID: PMC9451121 DOI: 10.1038/s41598-022-19198-1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 08/25/2022] [Indexed: 12/02/2022] Open
Abstract
Reliable and contactless measurements of vital signs, such as respiration and heart rate, are still unmet needs in clinical and home settings. Mm-wave radar and video-based technologies are promising, but currently, the signal processing-based vital sign extraction methods are prone to body motion disruptions or illumination variations in the surrounding environment. Here we propose an image segmentation-based method to extract vital signs from the recorded video and mm-wave radar signals. The proposed method analyses time-frequency spectrograms obtained from Short-Time Fourier Transform rather than individual time-domain signals. This leads to much-improved robustness and accuracy of the heart rate and respiration rate extraction over existing methods. The experiments were conducted under pre- and post-exercise conditions and were repeated on multiple individuals. The results are evaluated by using four metrics against the gold standard contact-based measurements. Significant improvements were observed in terms of precision, accuracy, and stability. The performance was reflected by achieving an averaged Pearson correlation coefficient (PCC) of 93.8% on multiple subjects. We believe that the proposed estimation method will help address the needs for the increasingly popular remote cardiovascular sensing and diagnosing posed by Covid-19.
Collapse
Affiliation(s)
- Xingyu Yang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Zijian Zhang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Yi Huang
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK
| | - Yalin Zheng
- Department of Eye and Vision Science, University of Liverpool, Liverpool, L7 8TX, UK
| | - Yaochun Shen
- Department of Electrical Engineering and Electronics, University of Liverpool, Liverpool, L69 3GJ, UK.
| |
Collapse
|
10
|
Lyra S, Rixen J, Heimann K, Karthik S, Joseph J, Jayaraman K, Orlikowsky T, Sivaprakasam M, Leonhardt S, Hoog Antink C. Camera fusion for real-time temperature monitoring of neonates using deep learning. Med Biol Eng Comput 2022; 60:1787-1800. [PMID: 35505175 PMCID: PMC9079037 DOI: 10.1007/s11517-022-02561-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2021] [Accepted: 03/25/2022] [Indexed: 11/23/2022]
Abstract
Abstract The continuous monitoring of vital signs is a crucial aspect of medical care in neonatal intensive care units. Since cable-based sensors pose a potential risk for the immature skin of preterm infants, unobtrusive monitoring techniques using camera systems are increasingly investigated. The combination of deep learning–based algorithms and camera modalities such as RGB and infrared thermography can improve the development of cable-free methods for the extraction of vital parameters. In this study, a real-time approach for local extraction of temperatures on the body surface of neonates using a multi-modal clinical dataset was implemented. Therefore, a trained deep learning–based keypoint detector was used for body landmark prediction in RGB. Image registration was conducted to transfer the RGB points to the corresponding thermographic recordings. These landmarks were used to extract the body surface temperature in various regions to determine the central-peripheral temperature difference. A validation of the keypoint detector showed a mean average precision of 0.82. The registration resulted in mean absolute errors of 16.4 px (8.2 mm) for x and 22.4 px (11.2 mm) for y. The evaluation of the temperature extraction revealed a mean absolute error of 0.55 \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$^{\circ }$$\end{document}∘C. A final performance of 31 fps was observed on the NVIDIA Jetson Xavier NX module, which proves real-time capability on an embedded GPU system. As a result, the approach can perform real-time temperature extraction on a low-cost GPU module. Graphical abstract ![]()
Collapse
|
11
|
Selvaraju V, Spicher N, Wang J, Ganapathy N, Warnecke JM, Leonhardt S, Swaminathan R, Deserno TM. Continuous Monitoring of Vital Signs Using Cameras: A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2022; 22:4097. [PMID: 35684717 PMCID: PMC9185528 DOI: 10.3390/s22114097] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Revised: 05/18/2022] [Accepted: 05/18/2022] [Indexed: 02/04/2023]
Abstract
In recent years, noncontact measurements of vital signs using cameras received a great amount of interest. However, some questions are unanswered: (i) Which vital sign is monitored using what type of camera? (ii) What is the performance and which factors affect it? (iii) Which health issues are addressed by camera-based techniques? Following the preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement, we conduct a systematic review of continuous camera-based vital sign monitoring using Scopus, PubMed, and the Association for Computing Machinery (ACM) databases. We consider articles that were published between January 2018 and April 2021 in the English language. We include five vital signs: heart rate (HR), respiratory rate (RR), blood pressure (BP), body skin temperature (BST), and oxygen saturation (SpO2). In total, we retrieve 905 articles and screened them regarding title, abstract, and full text. One hundred and four articles remained: 60, 20, 6, 2, and 1 of the articles focus on HR, RR, BP, BST, and SpO2, respectively, and 15 on multiple vital signs. HR and RR can be measured using red, green, and blue (RGB) and near-infrared (NIR) as well as far-infrared (FIR) cameras. So far, BP and SpO2 are monitored with RGB cameras only, whereas BST is derived from FIR cameras only. Under ideal conditions, the root mean squared error is around 2.60 bpm, 2.22 cpm, 6.91 mm Hg, 4.88 mm Hg, and 0.86 °C for HR, RR, systolic BP, diastolic BP, and BST, respectively. The estimated error for SpO2 is less than 1%, but it increases with movements of the subject and the camera-subject distance. Camera-based remote monitoring mainly explores intensive care, post-anaesthesia care, and sleep monitoring, but also explores special diseases such as heart failure. The monitored targets are newborn and pediatric patients, geriatric patients, athletes (e.g., exercising, cycling), and vehicle drivers. Camera-based techniques monitor HR, RR, and BST in static conditions within acceptable ranges for certain applications. The research gaps are large and heterogeneous populations, real-time scenarios, moving subjects, and accuracy of BP and SpO2 monitoring.
Collapse
Affiliation(s)
- Vinothini Selvaraju
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
- Non-Invasive Imaging and Diagnostic Laboratory, Biomedical Engineering, Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India;
| | - Nicolai Spicher
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Ju Wang
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Nagarajan Ganapathy
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Joana M. Warnecke
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| | - Steffen Leonhardt
- Chair for Medical Information Technology, Helmholtz-Institute for Biomedical Engineering, RWTH Aachen University, D-52074 Aachen, Germany;
| | - Ramakrishnan Swaminathan
- Non-Invasive Imaging and Diagnostic Laboratory, Biomedical Engineering, Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India;
| | - Thomas M. Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, D-38106 Braunschweig, Germany; (V.S.); (N.S.); (J.W.); (N.G.); (J.M.W.)
| |
Collapse
|
12
|
Molinaro N, Schena E, Silvestri S, Massaroni C. Multi-ROI Spectral Approach for the Continuous Remote Cardio-Respiratory Monitoring from Mobile Device Built-In Cameras. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22072539. [PMID: 35408151 PMCID: PMC9002464 DOI: 10.3390/s22072539] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Revised: 03/16/2022] [Accepted: 03/23/2022] [Indexed: 05/05/2023]
Abstract
Heart rate (HR) and respiratory rate (fR) can be estimated by processing videos framing the upper body and face regions without any physical contact with the subject. This paper proposed a technique for continuously monitoring HR and fR via a multi-ROI approach based on the spectral analysis of RGB video frames recorded with a mobile device (i.e., a smartphone's camera). The respiratory signal was estimated by the motion of the chest, whereas the cardiac signal was retrieved from the pulsatile activity at the level of right and left cheeks and forehead. Videos were recorded from 18 healthy volunteers in four sessions with different user-camera distances (i.e., 0.5 m and 1.0 m) and illumination conditions (i.e., natural and artificial light). For HR estimation, three approaches were investigated based on single or multi-ROI approaches. A commercially available multiparametric device was used to record reference respiratory signals and electrocardiogram (ECG). The results demonstrated that the multi-ROI approach outperforms the single-ROI approach providing temporal trends of both the vital parameters comparable to those provided by the reference, with a mean absolute error (MAE) consistently below 1 breaths·min-1 for fR in all the scenarios, and a MAE between 0.7 bpm and 6 bpm for HR estimation, whose values increase at higher distances.
Collapse
Affiliation(s)
- Nunzia Molinaro
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, 00128 Rome, Italy
| | - Emiliano Schena
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, 00128 Rome, Italy
| | - Sergio Silvestri
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, 00128 Rome, Italy
| | - Carlo Massaroni
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, 00128 Rome, Italy
| |
Collapse
|
13
|
Molinaro N, Schena E, Silvestri S, Bonotti F, Aguzzi D, Viola E, Buccolini F, Massaroni C. Contactless Vital Signs Monitoring From Videos Recorded With Digital Cameras: An Overview. Front Physiol 2022; 13:801709. [PMID: 35250612 PMCID: PMC8895203 DOI: 10.3389/fphys.2022.801709] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2021] [Accepted: 01/20/2022] [Indexed: 01/26/2023] Open
Abstract
The measurement of physiological parameters is fundamental to assess the health status of an individual. The contactless monitoring of vital signs may provide benefits in various fields of application, from healthcare and clinical setting to occupational and sports scenarios. Recent research has been focused on the potentiality of camera-based systems working in the visible range (380-750 nm) for estimating vital signs by capturing subtle color changes or motions caused by physiological activities but invisible to human eyes. These quantities are typically extracted from videos framing some exposed body areas (e.g., face, torso, and hands) with adequate post-processing algorithms. In this review, we provided an overview of the physiological and technical aspects behind the estimation of vital signs like respiratory rate, heart rate, blood oxygen saturation, and blood pressure from digital images as well as the potential fields of application of these technologies. Per each vital sign, we provided the rationale for the measurement, a classification of the different techniques implemented for post-processing the original videos, and the main results obtained during various applications or in validation studies. The available evidence supports the premise of digital cameras as an unobtrusive and easy-to-use technology for physiological signs monitoring. Further research is needed to promote the advancements of the technology, allowing its application in a wide range of population and everyday life, fostering a biometrical holistic of the human body (BHOHB) approach.
Collapse
Affiliation(s)
- Nunzia Molinaro
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy
| | - Emiliano Schena
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy
| | - Sergio Silvestri
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy
| | | | - Damiano Aguzzi
- BHOHB – Biometrical Holistic of Human Body S.r.l., Rome, Italy
| | - Erika Viola
- BHOHB – Biometrical Holistic of Human Body S.r.l., Rome, Italy
| | - Fabio Buccolini
- BHOHB – Biometrical Holistic of Human Body S.r.l., Rome, Italy
| | - Carlo Massaroni
- Unit of Measurements and Biomedical Instrumentation, Departmental Faculty of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy
| |
Collapse
|
14
|
Lucy FK, Suha KT, Dipty ST, Wadud MSI, Kadir MA. Video based non-contact monitoring of respiratory rate and chest indrawing in children with pneumonia. Physiol Meas 2021; 42. [PMID: 34715683 DOI: 10.1088/1361-6579/ac34eb] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2021] [Accepted: 10/29/2021] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Pneumonia is the single largest cause of death in children worldwide due to infectious diseases. According to WHO guidelines, fast breathing and chest indrawing are the key indicators of pneumonia in children requiring antibiotic treatments. The aim of this study was to develop a video-based novel method for simultaneous monitoring of respiratory rate and chest indrawing without upsetting babies. APPROACH Respiratory signals, corresponding to periodic movements of chest-abdominal walls during breathing, were extracted by analyzing RGB (red, green, blue) components in video frames captured by a smartphone camera. Respiratory rate was then obtained by applying fast fourier transform on the de-noised respiratory signal. Chest indrawing was detected by analysing relative phases of regional chest-abdominal wall mobility. The performance of the developed algorithm was evaluated on both healthy and pneumonia children. MAIN RESULTS The proposed method can measure respiratory rate with an overall mean absolute error of 1.8 bpm in the range 18-105 bpm. Phase difference between regional chest wall movements in the chest indrawing (pneumonia) cases was found to be 143±23.9 degrees, which was significantly higher than that in the healthy cases 52.3 ±32.6 degrees (p<0.001). SIGNIFICANCE Being non-intrusive and non-subjective, this computer-aided method can be useful in the monitoring for respiratory rate and chest indrawing for the diagnosis of pneumonia and its severity in children.
Collapse
Affiliation(s)
- Ferdous Karim Lucy
- Biomedical Engineering, Military Institute of Science and Technology, Dhaka, BANGLADESH
| | - Khadiza Tun Suha
- Department of Biomedical Engineering, Military Institute of Science and Technology, Dhaka, Dhaka District, BANGLADESH
| | - Sumaiya Tabassum Dipty
- biomedical engineering, Military Institute of Science and Technology, Dhaka, 1216, BANGLADESH
| | - Md Sharjis Ibne Wadud
- Department of Biomedical Engineering, Military Institute of Science and Technology, Dhaka, Dhaka District, BANGLADESH
| | - Muhammad Abdul Kadir
- Department of Biomedical Physics & Technology, University of Dhaka, Dhaka, BANGLADESH
| |
Collapse
|
15
|
Cheng CH, Wong KL, Chin JW, Chan TT, So RHY. Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda. SENSORS (BASEL, SWITZERLAND) 2021; 21:6296. [PMID: 34577503 PMCID: PMC8473186 DOI: 10.3390/s21186296] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Revised: 09/13/2021] [Accepted: 09/16/2021] [Indexed: 01/05/2023]
Abstract
Heart rate (HR) is one of the essential vital signs used to indicate the physiological health of the human body. While traditional HR monitors usually require contact with skin, remote photoplethysmography (rPPG) enables contactless HR monitoring by capturing subtle light changes of skin through a video camera. Given the vast potential of this technology in the future of digital healthcare, remote monitoring of physiological signals has gained significant traction in the research community. In recent years, the success of deep learning (DL) methods for image and video analysis has inspired researchers to apply such techniques to various parts of the remote physiological signal extraction pipeline. In this paper, we discuss several recent advances of DL-based methods specifically for remote HR measurement, categorizing them based on model architecture and application. We further detail relevant real-world applications of remote physiological monitoring and summarize various common resources used to accelerate related research progress. Lastly, we analyze the implications of research findings and discuss research gaps to guide future explorations.
Collapse
Affiliation(s)
- Chun-Hong Cheng
- Department of Computer Science, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China;
- PanopticAI, Hong Kong Science and Technology Parks, New Territories, Hong Kong, China; (J.-W.C.); (T.-T.C.); (R.H.Y.S.)
| | - Kwan-Long Wong
- PanopticAI, Hong Kong Science and Technology Parks, New Territories, Hong Kong, China; (J.-W.C.); (T.-T.C.); (R.H.Y.S.)
- Department of Bioengineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China
| | - Jing-Wei Chin
- PanopticAI, Hong Kong Science and Technology Parks, New Territories, Hong Kong, China; (J.-W.C.); (T.-T.C.); (R.H.Y.S.)
- Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China
| | - Tsz-Tai Chan
- PanopticAI, Hong Kong Science and Technology Parks, New Territories, Hong Kong, China; (J.-W.C.); (T.-T.C.); (R.H.Y.S.)
- Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China
| | - Richard H. Y. So
- PanopticAI, Hong Kong Science and Technology Parks, New Territories, Hong Kong, China; (J.-W.C.); (T.-T.C.); (R.H.Y.S.)
- Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China
| |
Collapse
|
16
|
Cabon S, Porée F, Cuffel G, Rosec O, Geslin F, Pladys P, Simon A, Carrault G. Voxyvi: A system for long-term audio and video acquisitions in neonatal intensive care units. Early Hum Dev 2021; 153:105303. [PMID: 33453631 DOI: 10.1016/j.earlhumdev.2020.105303] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/27/2020] [Revised: 11/04/2020] [Accepted: 12/21/2020] [Indexed: 11/18/2022]
Abstract
BACKGROUND In the European Union, 300,000 newborn babies are born prematurely every year. Their care is ensured in Neonatal Intensive Care Units (NICU) where vital signs are constantly monitored. In addition, other descriptors such as motion, facial and vocal activities have been shown to be essential to assess neurobehavioral development. AIM In the scope of the European project Digi-NewB, we aimed to develop and evaluate a new audio-video device designed to non-invasively acquire multi-modal data (audio, video and thermal images), while fitting the wide variety of bedding environment in NICU. METHODS Firstly, a multimodal system and associated software and guidelines to collect data in neonatal intensive care unit were proposed. Secondly, methods for post-evaluation of the acquisition phase were developed, including the study of clinician feedback and a qualitative analysis of the data. RESULTS The deployment of 19 acquisition devices in six French hospitals allowed to record more than 500 newborns of different gestational and postmenstrual ages. After the acquisition phase, clinical feedback was mostly positive. In addition, quality of more than 300 recordings was inspected and showed that 77% of the data is exploitable. In depth, the percentage of sole presence of the newborn was estimated at 62% within recordings. CONCLUSIONS This study demonstrates that audio-video acquisitions are feasible on a large scale in real life in NICU. The experience also allowed us to make a clear observation of the requirements and challenges that will have to be overcome in order to set up audio-video monitoring methods.
Collapse
Affiliation(s)
- S Cabon
- Univ Rennes, Inserm, LTSI - UMR 1099, F-35000, France.
| | - F Porée
- Univ Rennes, Inserm, LTSI - UMR 1099, F-35000, France
| | - G Cuffel
- Voxygen, Pleumeur-Bodou F-22560, France
| | - O Rosec
- Voxygen, Pleumeur-Bodou F-22560, France
| | - F Geslin
- CHU Rennes, Rennes F-35000, France
| | - P Pladys
- CHU Rennes, Rennes F-35000, France
| | - A Simon
- Univ Rennes, Inserm, LTSI - UMR 1099, F-35000, France
| | - G Carrault
- Univ Rennes, Inserm, LTSI - UMR 1099, F-35000, France
| |
Collapse
|