Pulmonary COVID-19: Learning Spatiotemporal Features Combining CNN and LSTM Networks for Lung Ultrasound Video Classification

Automatic diagnosis of abdominal pathologies in untrimmed ultrasound videos

PURPOSE

Despite major advances in Computer Assisted Diagnosis (CAD), the need for carefully labeled training data remains an important clinical translation barrier. This work aims to overcome this barrier for ultrasound video-based CAD, using video-level classification labels combined with a novel training strategy to improve the generalization performance of state-of-the-art (SOTA) video classifiers.

METHODS

SOTA video classifiers were trained and evaluated on a novel ultrasound video dataset of liver and kidney pathologies, and they all struggled to generalize, especially for kidney pathologies. A new training strategy is presented, wherein a frame relevance assessor is trained to score the video frames in a video by diagnostic relevance. This is used to automatically generate diagnostically-relevant video clips (DR-Clips), which guide a video classifier during training and inference.

RESULTS

Using DR-Clips with a Video Swin Transformer, we achieved a 0.92 ROC-AUC for kidney pathology detection in videos, compared to 0.72 ROC-AUC with a Swin Transformer and standard video clips. For liver steatosis detection, due to the diffuse nature of the pathology, the Video Swin Transformer, and other video classifiers, performed similarly well, generally exceeding a 0.92 ROC-AUC.

CONCLUSION

In theory, video classifiers, such as video transformers, should be able to solve ultrasound CAD tasks with video labels. However, in practice, video labels provide weaker supervision compared to image labels, resulting in worse generalization, as demonstrated. The additional frame guidance provided by DR-Clips enhances performance significantly. The results highlight current limits and opportunities to improve frame guidance.

Mapping artificial intelligence models in emergency medicine: A scoping review on artificial intelligence performance in emergency care and education

Artificial intelligence (AI) is increasingly improving the processes such as emergency patient care and emergency medicine education. This scoping review aims to map the use and performance of AI models in emergency medicine regarding AI concepts. The findings show that AI-based medical imaging systems provide disease detection with 85%-90% accuracy in imaging techniques such as X-ray and computed tomography scans. In addition, AI-supported triage systems were found to be successful in correctly classifying low- and high-urgency patients. In education, large language models have provided high accuracy rates in evaluating emergency medicine exams. However, there are still challenges in the integration of AI into clinical workflows and model generalization capacity. These findings demonstrate the potential of updated AI models, but larger-scale studies are still needed.

Semi-supervised Ensemble Learning for Automatic Interpretation of Lung Ultrasound Videos

Point-of-care ultrasound (POCUS) stands as a safe, portable, and cost-effective imaging modality for swift bedside patient examinations. Specifically, lung ultrasonography (LUS) has proven useful in evaluating both acute and chronic pulmonary conditions. Despite its clinical value, automatic LUS interpretation remains relatively unexplored, particularly in multi-label contexts. This work proposes a novel deep learning (DL) framework tailored for interpreting lung POCUS videos, whose outputs are the finding(s) present in these videos (such as A-lines, B-lines, or consolidations). The pipeline, based on a residual (2+1)D architecture, initiates with a pre-processing routine for video masking and standardisation, and employs a semi-supervised approach to harness available unlabeled data. Additionally, we introduce an ensemble modeling strategy that aggregates outputs from models trained to predict distinct label sets, thereby leveraging the hierarchical nature of LUS findings. The proposed framework and its building blocks were evaluated through extensive experiments with both multi-class and multi-label models, highlighting its versatility. In a held-out test set, the categorical proposal, suited for expedite triage, achieved an average F1-score of 92.4%, while the multi-label proposal, helpful for patient management and referral, achieved an average F1-score of 70.5% across five relevant LUS findings. Overall, the semi-supervised methodology contributed significantly to improved performance, while the proposed hierarchy-aware ensemble provided moderate additional gains.

FLUEnT: Transformer for detecting lung consolidations in videos using fused lung ultrasound encodings

Pneumonia is the leading cause of death among children around the world. According to WHO, a total of 740,180 lives under the age of five were lost due to pneumonia in 2019. Lung ultrasound (LUS) has been shown to be particularly useful for supporting the diagnosis of pneumonia in children and reducing mortality in resource-limited settings. The wide application of point-of-care ultrasound at the bedside is limited mainly due to a lack of training for data acquisition and interpretation. Artificial Intelligence can serve as a potential tool to automate and improve the LUS data interpretation process, which mainly involves analysis of hyper-echoic horizontal and vertical artifacts, and hypo-echoic small to large consolidations. This paper presents, Fused Lung Ultrasound Encoding-based Transformer (FLUEnT), a novel pediatric LUS video scoring framework for detecting lung consolidations using fused LUS encodings. Frame-level embeddings from a variational autoencoder, features from a spatially attentive ResNet-18, and encoded patient information as metadata combiningly form the fused encodings. These encodings are then passed on to the transformer for binary classification of the presence or absence of consolidations in the video. The video-level analysis using fused encodings resulted in a mean balanced accuracy of 89.3 %, giving an average improvement of 4.7 % points in comparison to when using these encodings individually. In conclusion, outperforming the state-of-the-art models by an average margin of 8 % points, our proposed FLUEnT framework serves as a benchmark for detecting lung consolidations in LUS videos from pediatric pneumonia patients.

Video Classification of Cloth Simulations: Deep Learning and Position-Based Dynamics for Stiffness Prediction

In virtual reality, augmented reality, or animation, the goal is to represent the movement of deformable objects in the real world as similar as possible in the virtual world. Therefore, this paper proposed a method to automatically extract cloth stiffness values from video scenes, and then they are applied as material properties for virtual cloth simulation. We propose the use of deep learning (DL) models to tackle this issue. The Transformer model, in combination with pre-trained architectures like DenseNet121, ResNet50, VGG16, and VGG19, stands as a leading choice for video classification tasks. Position-Based Dynamics (PBD) is a computational framework widely used in computer graphics and physics-based simulations for deformable entities, notably cloth. It provides an inherently stable and efficient way to replicate complex dynamic behaviors, such as folding, stretching, and collision interactions. Our proposed model characterizes virtual cloth based on softness-to-stiffness labels and accurately categorizes videos using this labeling. The cloth movement dataset utilized in this research is derived from a meticulously designed stiffness-oriented cloth simulation. Our experimental assessment encompasses an extensive dataset of 3840 videos, contributing to a multi-label video classification dataset. Our results demonstrate that our proposed model achieves an impressive average accuracy of 99.50%. These accuracies significantly outperform alternative models such as RNN, GRU, LSTM, and Transformer.

Pulmonary disease detection and classification in patient respiratory audio files using long short-term memory neural networks

Introduction

In order to improve the diagnostic accuracy of respiratory illnesses, our research introduces a novel methodology to precisely diagnose a subset of lung diseases using patient respiratory audio recordings. These lung diseases include Chronic Obstructive Pulmonary Disease (COPD), Upper Respiratory Tract Infections (URTI), Bronchiectasis, Pneumonia, and Bronchiolitis.

Methods

Our proposed methodology trains four deep learning algorithms on an input dataset consisting of 920 patient respiratory audio files. These audio files were recorded using digital stethoscopes and comprise the Respiratory Sound Database. The four deployed models are Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), CNN ensembled with unidirectional LSTM (CNN-LSTM), and CNN ensembled with bidirectional LSTM (CNN-BLSTM).

Results

The aforementioned models are evaluated using metrics such as accuracy, precision, recall, and F1-score. The best performing algorithm, LSTM, has an overall accuracy of 98.82% and F1-score of 0.97.

Discussion

The LSTM algorithm's extremely high predictive accuracy can be attributed to its penchant for capturing sequential patterns in time series based audio data. In summary, this algorithm is able to ingest patient audio recordings and make precise lung disease predictions in real-time.

FP-CNN: Fuzzy pooling-based convolutional neural network for lung ultrasound image classification with explainable AI

The COVID-19 pandemic wreaks havoc on healthcare systems all across the world. In pandemic scenarios like COVID-19, the applicability of diagnostic modalities is crucial in medical diagnosis, where non-invasive ultrasound imaging has the potential to be a useful biomarker. This research develops a computer-assisted intelligent methodology for ultrasound lung image classification by utilizing a fuzzy pooling-based convolutional neural network FP-CNN with underlying evidence of particular decisions. The fuzzy-pooling method finds better representative features for ultrasound image classification. The FPCNN model categorizes ultrasound images into one of three classes: covid, disease-free (normal), and pneumonia. Explanations of diagnostic decisions are crucial to ensure the fairness of an intelligent system. This research has used Shapley Additive Explanation (SHAP) to explain the prediction of the FP-CNN models. The prediction of the black-box model is illustrated using the SHAP explanation of the intermediate layers of the black-box model. To determine the most effective model, we have tested different state-of-the-art convolutional neural network architectures with various training strategies, including fine-tuned models, single-layer fuzzy pooling models, and fuzzy pooling at all pooling layers. Among different architectures, the Xception model with all pooling layers having fuzzy pooling achieves the best classification results of 97.2% accuracy. We hope our proposed method will be helpful for the clinical diagnosis of covid-19 from lung ultrasound (LUS) images.

Automated Lung Ultrasound Pulmonary Disease Quantification Using an Unsupervised Machine Learning Technique for COVID-19

COVID-19 is an ongoing global health pandemic. Although COVID-19 can be diagnosed with various tests such as PCR, these tests do not establish pulmonary disease burden. Whereas point-of-care lung ultrasound (POCUS) can directly assess the severity of characteristic pulmonary findings of COVID-19, the advantage of using US is that it is inexpensive, portable, and widely available for use in many clinical settings. For automated assessment of pulmonary findings, we have developed an unsupervised learning technique termed the calculated lung ultrasound (CLU) index. The CLU can quantify various types of lung findings, such as A or B lines, consolidations, and pleural effusions, and it uses these findings to calculate a CLU index score, which is a quantitative measure of pulmonary disease burden. This is accomplished using an unsupervised, patient-specific approach that does not require training on a large dataset. The CLU was tested on 52 lung ultrasound examinations from several institutions. CLU demonstrated excellent concordance with radiologist findings in different pulmonary disease states. Given the global nature of COVID-19, the CLU would be useful for sonographers and physicians in resource-strapped areas with limited ultrasound training and diagnostic capacities for more accurate assessment of pulmonary status.

LSTM model for predicting the daily number of asthma patients in Seoul, South Korea, using meteorological and air pollution data

Asthma is a common respiratory disease that is affected by air pollutants and meteorological factors. In this study, we developed models that predict the daily number of patients receiving treatment for asthma using air pollution and meteorological data. A neural network with long short-term memory (LSTM) and fully connected (FC) layers was used. The daily number of asthma patients in the city of Seoul, the capital of South Korea, was collected from the National Health Insurance Service. The data from 2015 to 2018 were used as the training and validation datasets for model development. Unseen data from 2019 were used for testing. The daily number of asthma patients per 100,000 inhabitants was predicted. The LSTM-FC neural network model achieved a Pearson correlation coefficient of 0.984 (P < 0.001) and root mean square error of 3.472 between the predicted and original values on the unseen testing dataset. The factors that impacted the prediction were the number of asthma patients in the previous time step before the predicted date, type of day (regular day and day after a holiday), minimum temperature, SO₂, daily changes in the amount of cloud, and daily changes in diurnal temperature range. We successfully developed a neural network that predicts the onset and exacerbation of asthma, and we identified the crucial influencing air pollutants and meteorological factors. This study will help us to establish appropriate measures according to the daily predicted number of asthma patients and reduce the daily onset and exacerbation of asthma in the susceptible population.

Automated detection of human mental disorder

The pressures of daily life result in a proliferation of terms such as stress, anxiety, and mood swings. These feelings may be developed to depression and more complicated mental problems. Unfortunately, the mood and emotional changes are difficult to notice and considered a disease that must be treated until late. The late diagnosis appears in suicidal intensions and harmful behaviors. In this work, main human observable facial behaviors are detected and classified by a model that has developed to assess a person’s mental health. Haar feature-based cascade is used to extract the features from the detected faces from FER+ dataset. VGG model classifies if the user is normal or abnormal. Then in the case of abnormal, the model predicts if he has depression, anxiety, or other disorder according to the detected facial expression. The required assistance and support can be provided in a timely manner with this prediction. The system has achieved a 95% of overall prediction accuracy.

Deep learning classifiers for computer-aided diagnosis of multiple lungs disease

BACKGROUND

Computer aided diagnosis has gained momentum in the recent past. The advances in deep learning and availability of huge volumes of data along with increased computational capabilities has reshaped the diagnosis and prognosis procedures.

OBJECTIVE

These methods are proven to be relatively less expensive and safer alternatives of the otherwise traditional approaches. This study is focused on efficient diagnosis of three very common diseases: lung cancer, pneumonia and Covid-19 using X-ray images.

METHODS

Three different deep learning models are designed and developed to perform 4-way classification. Inception V3, Convolutional Neural Networks (CNN) and Long Short Term Memory models (LSTM) are used as building blocks. The performance of these models is evaluated using three publicly available datasets, the first dataset contains images for Lung cancer, second contains images for Covid-19 and third dataset contains images for Pneumonia and normal subjects. Combining three datasets creates a class imbalance problem which is resolved using pre-processing and data augmentation techniques. After data augmentation 1386 subjects are randomly chosen for each class.

RESULTS

It is observed that CNN when combined with LSTM (CNN-LSTM) produces significantly improved results (accuracy of 94.5 %) which is better than CNN and InceptionV3-LSTM. 3,5, and 10 fold cross validation is performed to verify all results calculated using three different classifiersConclusions:This research concludes that a single computer-aided diagnosis system can be developed for diagnosing multiple diseases.

Non-contact detection method of pregnant sows backfat thickness based on two-dimensional images

Since sow backfat thickness (BFT) is highly correlated with its service life and reproductive effectiveness, dynamic monitoring of BFT is a critical component of large-scale sow farm productivity. Existing contact measures of sow BFT have their problems including, high measurement intensity and sows' stress reaction, low biological safety, and difficulty in meeting the requirements for multiple measurements. This article presents a two-dimensional (2D) image-based approach for determining the BFT of pregnant sows when combined with the backfat growth rate (BGR). The 2D image features of sows extracted by convolutional neural networks (CNN) and the artificially defined phenotypic features of sows such as hip width, hip height, body length, hip height-width ratio, length-width ratio, and waist-hip ratio, were used respectively, combined with BGR, to construct a prediction model for sow BFT using support vector regression (SVR). Following testing and comparison, it was shown that using CNN to extract features from images could effectively replace artificially defined features, BGR contributed to the model's accuracy improvement. The CNN-BGR-SVR model performed the best, with R² of 0.72 and mean absolute error of 1.21 mm, and root mean square error of 1.50 mm, and mean absolute percentage error of 7.57%. The results demonstrated that the CNN-BGR-SVR model based on 2D images was capable of detecting sow BFT, establishing a new reference for non-contact sow BFT detection technology.

Image Sensing and Processing with Convolutional Neural Networks

Convolutional neural networks are a class of deep neural networks that leverage spatial information, and they are therefore well suited to classifying images for a range of applications [...].

Applications of artificial intelligence in lung ultrasound: Review of deep learning methods for COVID-19 fighting

BACKGROUND

The pandemic outbreak of the novel coronavirus disease (COVID-19) has highlighted the need to combine rapid, non-invasive and widely accessible techniques with the least risk of patient’s cross-infection to achieve a successful early detection and surveillance of the disease. In this regard, the lung ultrasound (LUS) technique has been proved invaluable in both the differential diagnosis and the follow-up of COVID-19 patients, and its potential may be destined to evolve. Recently, indeed, LUS has been empowered through the development of automated image processing techniques.

AIM

To provide a systematic review of the application of artificial intelligence (AI) technology in medical LUS analysis of COVID-19 patients using the preferred reporting items of systematic reviews and meta-analysis (PRISMA) guidelines.

METHODS

A literature search was performed for relevant studies published from March 2020 - outbreak of the pandemic - to 30 September 2021. Seventeen articles were included in the result synthesis of this paper.

RESULTS

As part of the review, we presented the main characteristics related to AI techniques, in particular deep learning (DL), adopted in the selected articles. A survey was carried out on the type of architectures used, availability of the source code, network weights and open access datasets, use of data augmentation, use of the transfer learning strategy, type of input data and training/test datasets, and explainability.

CONCLUSION

Finally, this review highlighted the existing challenges, including the lack of large datasets of reliable COVID-19-based LUS images to test the effectiveness of DL methods and the ethical/regulatory issues associated with the adoption of automated systems in real clinical scenarios.

Review of Machine Learning in Lung Ultrasound in COVID-19 Pandemic

Ultrasound imaging of the lung has played an important role in managing patients with COVID-19-associated pneumonia and acute respiratory distress syndrome (ARDS). During the COVID-19 pandemic, lung ultrasound (LUS) or point-of-care ultrasound (POCUS) has been a popular diagnostic tool due to its unique imaging capability and logistical advantages over chest X-ray and CT. Pneumonia/ARDS is associated with the sonographic appearances of pleural line irregularities and B-line artefacts, which are caused by interstitial thickening and inflammation, and increase in number with severity. Artificial intelligence (AI), particularly machine learning, is increasingly used as a critical tool that assists clinicians in LUS image reading and COVID-19 decision making. We conducted a systematic review from academic databases (PubMed and Google Scholar) and preprints on arXiv or TechRxiv of the state-of-the-art machine learning technologies for LUS images in COVID-19 diagnosis. Openly accessible LUS datasets are listed. Various machine learning architectures have been employed to evaluate LUS and showed high performance. This paper will summarize the current development of AI for COVID-19 management and the outlook for emerging trends of combining AI-based LUS with robotics, telehealth, and other techniques.

A Review of Deep Learning Applications in Lung Ultrasound Imaging of COVID-19 Patients

The massive and continuous spread of COVID-19 has motivated researchers around the world to intensely explore, understand, and develop new techniques for diagnosis and treatment. Although lung ultrasound imaging is a less established approach when compared to other medical imaging modalities such as X-ray and CT, multiple studies have demonstrated its promise to diagnose COVID-19 patients. At the same time, many deep learning models have been built to improve the diagnostic efficiency of medical imaging. The integration of these initially parallel efforts has led multiple researchers to report deep learning applications in medical imaging of COVID-19 patients, most of which demonstrate the outstanding potential of deep learning to aid in the diagnosis of COVID-19. This invited review is focused on deep learning applications in lung ultrasound imaging of COVID-19 and provides a comprehensive overview of ultrasound systems utilized for data acquisition, associated datasets, deep learning models, and comparative performance.