Automatic Classification of Cardiac Views in Echocardiogram Using Histogram and Statistical Features

CACTUS: An open dataset and framework for automated Cardiac Assessment and Classification of Ultrasound images using deep transfer learning

Cardiac ultrasound (US) scanning is one of the most commonly used techniques in cardiology to diagnose the health of the heart and its proper functioning. During a typical US scan, medical professionals take several images of the heart to be classified based on the cardiac views they contain, with a focus on high-quality images. However, this task is time consuming and error prone. Therefore, it is necessary to consider ways to automate these tasks and assist medical professionals in classifying and assessing cardiac US images. Machine learning (ML) techniques are regarded as a prominent solution due to their success in the development of numerous applications aimed at enhancing the medical field, including addressing the shortage of echography technicians. However, the limited availability of medical data presents a significant barrier to the application of ML in the field of cardiology, particularly regarding US images of the heart. This paper addresses this challenge by introducing the first open graded dataset for Cardiac Assessment and ClassificaTion of UltraSound (CACTUS), which is available online. This dataset contains images obtained from scanning a CAE Blue Phantom and representing various heart views and different quality levels, exceeding the conventional cardiac views typically found in literature. Additionally, the paper introduces a Deep Learning (DL) framework consisting of two main components. The first component is responsible for classifying cardiac US images based on the heart view using a Convolutional Neural Network (CNN) architecture. The second component uses the concept of Transfer Learning (TL) to utilize knowledge from the first component and fine-tune it to create a model for grading and assessing cardiac images. The framework demonstrates high performance in both classification and grading, achieving up to 99.43% accuracy and as low as 0.3067 error, respectively. To showcase its robustness, the framework is further fine-tuned using new images representing additional cardiac views and also compared to several other state-of-the-art architectures. The framework's outcomes and its performance in handling real-time scans were also assessed using a questionnaire answered by cardiac experts.

Deep learning supported echocardiogram analysis: A comprehensive review

An echocardiogram is a sophisticated ultrasound imaging technique employed to diagnose heart conditions. The transthoracic echocardiogram, one of the most prevalent types, is instrumental in evaluating significant cardiac diseases. However, interpreting its results heavily relies on the clinician's expertise. In this context, artificial intelligence has emerged as a vital tool for helping clinicians. This study critically analyzes key state-of-the-art research that uses deep learning techniques to automate transthoracic echocardiogram analysis and support clinical judgments. We have systematically organized and categorized articles that proffer solutions for view classification, enhancement of image quality and dataset, segmentation and identification of cardiac structures, detection of cardiac function abnormalities, and quantification of cardiac functions. We compared the performance of various deep learning approaches within each category, identifying the most promising methods. Additionally, we highlight limitations in current research and explore promising avenues for future exploration. These include addressing generalizability issues, incorporating novel AI approaches, and tackling the analysis of rare cardiac diseases.

Performance evaluation of computer-aided automated master frame selection techniques for fetal echocardiography

PURPOSE

Fetal echocardiography is widely used for the assessment of fetal heart development and detection of congenital heart disease (CHD). Preliminary examination of the fetal heart involves the four-chamber view which indicates the presence of all the four chambers and its structural symmetry. Examination of various cardiac parameters is generally done using the clinically selected diastole frame. This largely depends on the expertise of the sonographer and is prone to intra- and interobservational errors. To overcome this, automated frame selection technique is proposed for the recognition of fetal cardiac chamber from fetal echocardiography.

METHODS

Three techniques have been proposed in this research study to automate the process of determining the frame referred as "Master Frame" that can be used for the measurement of the cardiac parameters. The first method uses frame similarity measures (FSM) for the determination of the master frame from the given cine loop ultrasonic sequences. FSM makes use of similarity measures such as correlation, structural similarity index (SSIM), peak signal to noise ratio (PSNR), and mean square error (MSE) to identify the cardiac cycle, and all the frames in one cardiac cycle are superimposed to form the master frame. The final master frame is obtained by considering the average of the master frame obtained using each similarity measure. The second method uses averaging of ± 20% from the midframes (AMF). The third method uses averaging of all the frames (AAF) of the cine loop sequence. Both diastole and master frames have been annotated by the clinical experts, and their ground truths are compared for validation. No segmentation techniques have been used to avoid the variability of the performance of various segmentation techniques. All the proposed schemes were evaluated using six fidelity metrics such as Dice coefficient, Jaccard ratio, Hausdorff distance, structural similarity index, mean absolute error, and Pratt figure of merit.

RESULTS

The three proposed techniques were tested on the frames extracted from 95 ultrasound cine loop sequences between 19 and 32 weeks of gestation. The feasibility of the techniques was determined by the computation of fidelity metrics between the master frame derived and the diastole frame chosen by the clinical experts. The FSM-based identified master frame found to closely match with manually chosen diastole frame and also ensures statistically significant. The method also detects automatically the cardiac cycle. The resultant master frame obtained through AMF though found to be identical to that of the diastole frame, the size of the chambers found to be reduced that can lead to inaccurate chamber measurement. The master frame obtained through AAF was not found to be identical to that of clinical diastole frame.

CONCLUSION

It can be concluded that the frame similarity measure (FSM)-based master frame can be introduced in the clinical routine for segmentation followed by cardiac chamber measurements. Such automated master frame selection also overcomes the manual intervention of earlier reported techniques in the literature. The fidelity metrics assessment further confirms the suitability of proposed master frame for automated fetal chamber recognition.

Assisted probe guidance in cardiac ultrasound: A review

Echocardiography is the most frequently used imaging modality in cardiology. However, its acquisition is affected by inter-observer variability and largely dependent on the operator's experience. In this context, artificial intelligence techniques could reduce these variabilities and provide a user independent system. In recent years, machine learning (ML) algorithms have been used in echocardiography to automate echocardiographic acquisition. This review focuses on the state-of-the-art studies that use ML to automate tasks regarding the acquisition of echocardiograms, including quality assessment (QA), recognition of cardiac views and assisted probe guidance during the scanning process. The results indicate that performance of automated acquisition was overall good, but most studies lack variability in their datasets. From our comprehensive review, we believe automated acquisition has the potential not only to improve accuracy of diagnosis, but also help novice operators build expertise and facilitate point of care healthcare in medically underserved areas.

Advancing cardiovascular medicine with machine learning: Progress, potential, and perspective

Recent advances in machine learning (ML) have made it possible to analyze high-dimensional and complex data-such as free text, images, waveforms, videos, and sound-in an automated manner by successfully learning complex associations within these data. Cardiovascular medicine is particularly well poised to take advantage of these ML advances, due to the widespread digitization of medical data and the large number of diagnostic tests used to evaluate cardiovascular disease. Various ML approaches have successfully been applied to cardiovascular tests and diseases to automate interpretation, accurately perform measurements, and, in some cases, predict novel diagnoses from less invasive tests, effectively expanding the utility of more widely accessible diagnostic tests. Here, we present examples of some impactful advances in cardiovascular medicine using ML across a variety of modalities, with a focus on deep learning applications.

Master Frame Extraction of Fetal Cardiac Images Using B Mode Ultrasound Images

Fetal Echocardiography is used for monitoring the fetal heart and for detection of Congenital Heart Disease (CHD). It is well known that fetal cardiac four chamber view has been widely used for preliminary examination for the detection of CHD. The end diastole frame is generally used for the analysis of the fetal cardiac chambers which is manually picked by the clinician during examination/screening. This method is subjected to intra and inter observer errors and also time consuming. The proposed study aims to automate this process by determining the frame, referred to as the Master frame from the cine loop sequences that can be used for the analysis of the fetal heart chambers instead of the clinically chosen diastole frame. The proposed framework determines the correlation between the reference (first) frame with the successive frames to identify one cardiac cycle. Then the Master frame is formed by superimposing all the frames belonging to one cardiac cycle. The master frame is then compared with the clinically chosen diastole frame in terms of fidelity metrics such as Dice coefficient, Hausdorff distance, mean square error and structural similarity index. The average value of the fidelity metrics considering the dataset used for this study 0.73 for Dice, 13.94 for Hausdorff distance, 0.99 for Structural Similarity Index and 0.035 for mean square error confirms the suitability of the proposed master frame extraction thereby avoiding manual intervention by the clinician. .

Automated interpretation of systolic and diastolic function on the echocardiogram: a multicohort study

BACKGROUND

Echocardiography is the diagnostic modality for assessing cardiac systolic and diastolic function to diagnose and manage heart failure. However, manual interpretation of echocardiograms can be time consuming and subject to human error. Therefore, we developed a fully automated deep learning workflow to classify, segment, and annotate two-dimensional (2D) videos and Doppler modalities in echocardiograms.

METHODS

We developed the workflow using a training dataset of 1145 echocardiograms and an internal test set of 406 echocardiograms from the prospective heart failure research platform (Asian Network for Translational Research and Cardiovascular Trials; ATTRaCT) in Asia, with previous manual tracings by expert sonographers. We validated the workflow against manual measurements in a curated dataset from Canada (Alberta Heart Failure Etiology and Analysis Research Team; HEART; n=1029 echocardiograms), a real-world dataset from Taiwan (n=31 241), the US-based EchoNet-Dynamic dataset (n=10 030), and in an independent prospective assessment of the Asian (ATTRaCT) and Canadian (Alberta HEART) datasets (n=142) with repeated independent measurements by two expert sonographers.

FINDINGS

In the ATTRaCT test set, the automated workflow classified 2D videos and Doppler modalities with accuracies (number of correct predictions divided by the total number of predictions) ranging from 0·91 to 0·99. Segmentations of the left ventricle and left atrium were accurate, with a mean Dice similarity coefficient greater than 93% for all. In the external datasets (n=1029 to 10 030 echocardiograms used as input), automated measurements showed good agreement with locally measured values, with a mean absolute error range of 9-25 mL for left ventricular volumes, 6-10% for left ventricular ejection fraction (LVEF), and 1·8-2·2 for the ratio of the mitral inflow E wave to the tissue Doppler e' wave (E/e' ratio); and reliably classified systolic dysfunction (LVEF <40%, area under the receiver operating characteristic curve [AUC] range 0·90-0·92) and diastolic dysfunction (E/e' ratio ≥13, AUC range 0·91-0·91), with narrow 95% CIs for AUC values. Independent prospective evaluation confirmed less variance of automated compared with human expert measurements, with all individual equivalence coefficients being less than 0 for all measurements.

INTERPRETATION

Deep learning algorithms can automatically annotate 2D videos and Doppler modalities with similar accuracy to manual measurements by expert sonographers. Use of an automated workflow might accelerate access, improve quality, and reduce costs in diagnosing and managing heart failure globally.

FUNDING

A*STAR Biomedical Research Council and A*STAR Exploit Technologies.

Artificial intelligence applied to support medical decisions for the automatic analysis of echocardiogram images: A systematic review

The echocardiogram is a test that is widely used in Heart Disease Diagnoses. However, its analysis is largely dependent on the physician's experience. In this regard, artificial intelligence has become an essential technology to assist physicians. This study is a Systematic Literature Review (SLR) of primary state-of-the-art studies that used Artificial Intelligence (AI) techniques to automate echocardiogram analyses. Searches on the leading scientific article indexing platforms using a search string returned approximately 1400 articles. After applying the inclusion and exclusion criteria, 118 articles were selected to compose the detailed SLR. This SLR presents a thorough investigation of AI applied to support medical decisions for the main types of echocardiogram (Transthoracic, Transesophageal, Doppler, Stress, and Fetal). The article's data extraction indicated that the primary research interest of the studies comprised four groups: 1) Improvement of image quality; 2) identification of the cardiac window vision plane; 3) quantification and analysis of cardiac functions, and; 4) detection and classification of cardiac diseases. The articles were categorized and grouped to show the main contributions of the literature to each type of ECHO. The results indicate that the Deep Learning (DL) methods presented the best results for the detection and segmentation of the heart walls, right and left atrium and ventricles, and classification of heart diseases using images/videos obtained by echocardiography. The models that used Convolutional Neural Network (CNN) and its variations showed the best results for all groups. The evidence produced by the results presented in the tabulation of the studies indicates that the DL contributed significantly to advances in echocardiogram automated analysis processes. Although several solutions were presented regarding the automated analysis of ECHO, this area of research still has great potential for further studies to improve the accuracy of results already known in the literature.

Machine learning: at the heart of failure diagnosis

PURPOSE OF REVIEW

Refinement in machine learning (ML) techniques and approaches has rapidly expanded artificial intelligence applications for the diagnosis and classification of heart failure (HF). This review is designed to provide the clinician with the basics of ML, as well as this technologies future utility in HF diagnosis and the potential impact on patient outcomes.

RECENT FINDINGS

Recent studies applying ML methods to unique data sets available from electrocardiography, vectorcardiography, echocardiography, and electronic health records show significant promise for improving diagnosis, enhancing detection, and advancing treatment of HF. Innovations in both supervised and unsupervised methods have heightened the diagnostic accuracy of models developed to identify the presence of HF and further augmentation of model capabilities are likely utilizing ensembles of ML algorithms derived from different techniques.

SUMMARY

This article is an overview of recent applications of ML to achieve improved diagnosis of HF and the resultant implications for patient management.

Deep Learning Methods for Classification of Certain Abnormalities in Echocardiography

This article experiments with deep learning methodologies in echocardiogram (echo), a promising and vigorously researched technique in the preponderance field. This paper involves two different kinds of classification in the echo. Firstly, classification into normal (absence of abnormalities) or abnormal (presence of abnormalities) has been done, using 2D echo images, 3D Doppler images, and videographic images. Secondly, based on different types of regurgitation, namely, Mitral Regurgitation (MR), Aortic Regurgitation (AR), Tricuspid Regurgitation (TR), and a combination of the three types of regurgitation are classified using videographic echo images. Two deep-learning methodologies are used for these purposes, a Recurrent Neural Network (RNN) based methodology (Long Short Term Memory (LSTM)) and an Autoencoder based methodology (Variational AutoEncoder (VAE)). The use of videographic images distinguished this work from the existing work using SVM (Support Vector Machine) and also application of deep-learning methodologies is the first of many in this particular field. It was found that deep-learning methodologies perform better than SVM methodology in normal or abnormal classification. Overall, VAE performs better in 2D and 3D Doppler images (static images) while LSTM performs better in the case of videographic images.

Harnessing Machine Intelligence in Automatic Echocardiogram Analysis: Current Status, Limitations, and Future Directions

Echocardiography (echo) is a critical tool in diagnosing various cardiovascular diseases. Despite its diagnostic and prognostic value, interpretation and analysis of echo images are still widely performed manually by echocardiographers. A plethora of algorithms has been proposed to analyze medical ultrasound data using signal processing and machine learning techniques. These algorithms provided opportunities for developing automated echo analysis and interpretation systems. The automated approach can significantly assist in decreasing the variability and burden associated with manual image measurements. In this paper, we review the state-of-the-art automatic methods for analyzing echocardiography data. Particularly, we comprehensively and systematically review existing methods of four major tasks: echo quality assessment, view classification, boundary segmentation, and disease diagnosis. Our review covers three echo imaging modes, which are B-mode, M-mode, and Doppler. We also discuss the challenges and limitations of current methods and outline the most pressing directions for future research. In summary, this review presents the current status of automatic echo analysis and discusses the challenges that need to be addressed to obtain robust systems suitable for efficient use in clinical settings or point-of-care testing.

White striping degree assessment using computer vision system and consumer acceptance test

OBJECTIVE

The objective of this study was to evaluate three different degrees of white striping (WS) addressing their automatic assessment and customer acceptance. The WS classification was performed based on a computer vision system (CVS), exploring different machine learning (ML) algorithms and the most important image features. Moreover, it was verified by consumer acceptance and purchase intent.

METHODS

The samples for image analysis were classified by trained specialists, according to severity degrees regarding visual and firmness aspects. Samples were obtained with a digital camera, and 25 features were extracted from these images. ML algorithms were applied aiming to induce a model capable of classifying the samples into three severity degrees. In addition, two sensory analyses were performed: 75 samples properly grilled were used for the first sensory test, and 9 photos for the second. All tests were performed using a 10-cm hybrid hedonic scale (acceptance test) and a 5-point scale (purchase intention).

RESULTS

The information gain metric ranked 13 attributes. However, just one type of image feature was not enough to describe the phenomenon. The classification models support vector machine, fuzzy-W, and random forest showed the best results with similar general accuracy (86.4%). The worst performance was obtained by multilayer perceptron (70.9%) with the high error rate in normal (NORM) sample predictions. The sensory analysis of acceptance verified that WS myopathy negatively affects the texture of the broiler breast fillets when grilled and the appearance attribute of the raw samples, which influenced the purchase intention scores of raw samples.

CONCLUSION

The proposed system has proved to be adequate (fast and accurate) for the classification of WS samples. The sensory analysis of acceptance showed that WS myopathy negatively affects the tenderness of the broiler breast fillets when grilled, while the appearance attribute of the raw samples eventually influenced purchase intentions.

Automatic apical view classification of echocardiograms using a discriminative learning dictionary

As part of striving towards fully automatic cardiac functional assessment of echocardiograms, automatic classification of their standard views is essential as a pre-processing stage. The similarity among three of the routinely acquired longitudinal scans: apical two-chamber (A2C), apical four-chamber (A4C) and apical long-axis (ALX), and the noise commonly inherent to these scans - make the classification a challenge. Here we introduce a multi-stage classification algorithm that employs spatio-temporal feature extraction (Cuboid Detector) and supervised dictionary learning (LC-KSVD) approaches to uniquely enhance the automatic recognition and classification accuracy of echocardiograms. The algorithm incorporates both discrimination and labelling information to allow a discriminative and sparse representation of each view. The advantage of the spatio-temporal feature extraction as compared to spatial processing is then validated. A set of 309 clinical clips (103 for each view), were labeled by 2 experts. A subset of 70 clips of each class was used as a training set and the rest as a test set. The recognition accuracies achieved were: 97%, 91% and 97% of A2C, A4C and ALX respectively, with average recognition rate of 95%. Thus, automatic classification of echocardiogram views seems promising, despite the inter-view similarity between the classes and intra-view variability among clips belonging to the same class.