Conventional and deep learning methods in heart rate estimation from RGB face videos

3D DenseNet with temporal transition layer for heart rate estimation from real-life RGB videos

BACKGROUND

Deep learning has demonstrated superior performance over traditional methods for the estimation of heart rates in controlled contexts. However, in less controlled scenarios this performance seems to vary based on the training dataset and the architecture of the deep learning models.

OBJECTIVES

In this paper, we develop a deep learning-based model leveraging the power of 3D convolutional neural networks (3DCNN) to extract temporal and spatial features that lead to an accurate heart rates estimation from RGB no pre-defined region of interest (ROI) videos.

METHODS

We propose a 3D DenseNet with a 3D temporal transition layer for the estimation of heart rates from a large-scale dataset of videos that appear more hospital-like and real-life than other existing facial video-based datasets.

RESULTS

Experimentally, our model was trained and tested on this less controlled dataset and showed heart rate estimation performance with root mean square error (RMSE) of 8.68 BPM and mean absolute error (MAE) of 3.34 BPM.

CONCLUSION

Moreover, we show that such a model can also achieve better results than the state-of-the-art models when tested on the VIPL-HR public dataset.

A numerical evaluation of real-time workloads for ramp controller through optimization of multi-type feature combinations derived from eye tracker, respiratory, and fatigue patterns

Ramp controllers are required to manage their workloads effectively while handling complex operational tasks, a crucial part of improving aviation safety. The ability to detect their instantaneous workload is vital for ensuring operational effectiveness and preventing hazardous incidents. This paper introduces a novel methodology aimed at enhancing the evaluation of the ramp controller's cumulative workload by incorporating and optimizing the feature combination from eye movement, respiratory, and fatigue characteristics. Specifically, a 90-minute simulated experiment related to ramp control tasks, using real data from Shanghai Hongqiao Airport, is conducted to collect multi-type data from 8 controllers. Following data construction and the extraction of multi-type, the workloads of all samples are categorized through unsupervised learning. Subsequently, supervised learning techniques are used to calculate feature weights and train classifiers after data alignment. The optimal feature combination is established by calculating feature weights, and the best classification accuracy is over 98%, achieved by the KNN classifier. Furthermore, numerical evaluation and threshold calculations for different workload levels are interpreted. It is promising to provide insights into future works towards human-centered data construction, processing, and interpretation to promote the progress of workload assessment within the aviation industry.

A Multimodal Transformer Model for Recognition of Images from Complex Laparoscopic Surgical Videos

The determination of the potential role and advantages of artificial intelligence-based models in the field of surgery remains uncertain. This research marks an initial stride towards creating a multimodal model, inspired by the Video-Audio-Text Transformer, that aims to reduce negative occurrences and enhance patient safety. The model employs text and image embedding state-of-the-art models (ViT and BERT) to assess their efficacy in extracting the hidden and distinct features from the surgery video frames. These features are then used as inputs for convolution-free Transformer architectures to extract comprehensive multidimensional representations. A joint space is then used to combine the text and image features extracted from both Transformer encoders. This joint space ensures that the relationships between the different modalities are preserved during the combination process. The entire model was trained and tested on laparoscopic cholecystectomy (LC) videos encompassing various levels of complexity. Experimentally, a mean accuracy of 91.0%, a precision of 81%, and a recall of 83% were reached by the model when tested on 30 videos out of 80 from the Cholec80 dataset.