1
|
Nascimento JC, Carneiro G. One Shot Segmentation: Unifying Rigid Detection and Non-Rigid Segmentation Using Elastic Regularization. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2020; 42:3054-3070. [PMID: 31217094 DOI: 10.1109/tpami.2019.2922959] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
This paper proposes a novel approach for the non-rigid segmentation of deformable objects in image sequences, which is based on one-shot segmentation that unifies rigid detection and non-rigid segmentation using elastic regularization. The domain of application is the segmentation of a visual object that temporally undergoes a rigid transformation (e.g., affine transformation) and a non-rigid transformation (i.e., contour deformation). The majority of segmentation approaches to solve this problem are generally based on two steps that run in sequence: a rigid detection, followed by a non-rigid segmentation. In this paper, we propose a new approach, where both the rigid and non-rigid segmentation are performed in a single shot using a sparse low-dimensional manifold that represents the visual object deformations. Given the multi-modality of these deformations, the manifold partitions the training data into several patches, where each patch provides a segmentation proposal during the inference process. These multiple segmentation proposals are merged using the classification results produced by deep belief networks (DBN) that compute the confidence on each segmentation proposal. Thus, an ensemble of DBN classifiers is used for estimating the final segmentation. Compared to current methods proposed in the field, our proposed approach is advantageous in four aspects: (i) it is a unified framework to produce rigid and non-rigid segmentations; (ii) it uses an ensemble classification process, which can help the segmentation robustness; (iii) it provides a significant reduction in terms of the number of dimensions of the rigid and non-rigid segmentations search spaces, compared to current approaches that divide these two problems; and (iv) this lower dimensionality of the search space can also reduce the need for large annotated training sets to be used for estimating the DBN models. Experiments on the problem of left ventricle endocardial segmentation from ultrasound images, and lip segmentation from frontal facial images using the extended Cohn-Kanade (CK+) database, demonstrate the potential of the methodology through qualitative and quantitative evaluations, and the ability to reduce the search and training complexities without a significant impact on the segmentation accuracy.
Collapse
|
2
|
Medley DO, Santiago C, Nascimento JC. Deep Active Shape Model for Robust Object Fitting. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2019; 29:2380-2394. [PMID: 31670673 DOI: 10.1109/tip.2019.2948728] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Object recognition and localization is still a very challenging problem, despite recent advances in deep learning (DL) approaches, especially for objects with varying shapes and appearances. Statistical models, such as an Active Shape Model (ASM), rely on a parametric model of the object, allowing an easy incorporation of prior knowledge about shape and appearance in a principled way. To take advantage of these benefits, this paper proposes a new ASM framework that addresses two tasks: (i) comparing the performance of several image features used to extract observations from an input image; and (ii) improving the performance of the model fitting by relying on a probabilistic framework that allows the use of multiple observations and is robust to the presence of outliers. The goal in (i) is to maximize the quality of the observations by exploring a wide set of handcrafted features (HOG, SIFT, and texture templates) and more recent DL-based features. Regarding (ii), we use the Generalized Expectation-Maximization algorithm to deal with outliers and to extend the fitting process to multiple observations. The proposed framework is evaluated in the context of facial landmark fitting and the segmentation of the endocardium of the left ventricle in cardiac magnetic resonance volumes. We experimentally observe that the proposed approach is robust not only to outliers, but also to adverse initialization conditions and to large search regions (from where the observations are extracted from the image). Furthermore, the results of the proposed combination of the ASM with DL-based features are competitive with more recent DL approaches (e.g. FCN [1], U-Net [2] and CNN Cascade [3]), showing that it is possible to combine the benefits of statistical models and DL into a new deep ASM probabilistic framework.
Collapse
|
3
|
Liu Z, Chan SC, Zhang S, Zhang Z, Chen X. Automatic Muscle Fiber Orientation Tracking in Ultrasound Images Using a New Adaptive Fading Bayesian Kalman Smoother. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2019; 28:3714-3727. [PMID: 30794172 DOI: 10.1109/tip.2019.2899941] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
This paper proposes a new algorithm for automatic estimation of muscle fiber orientation (MFO) in musculoskeletal ultrasound images, which is commonly used for both diagnosis and rehabilitation assessment of patients. The algorithm is based on a novel adaptive fading Bayesian Kalman filter (AF-BKF) and an automatic region of interest (ROI) extraction method. The ROI is first enhanced by the Gabor filter (GF) and extracted automatically using the revoting constrained Radon transform (RCRT) approach. The dominant MFO in the ROI is then detected by the RT and tracked by the proposed AF-BKF, which employs simplified Gaussian mixtures to approximate the non-Gaussian state densities and a new adaptive fading method to update the mixture parameters. An AF-BK smoother (AF-BKS) is also proposed by extending the AF-BKF using the concept of Rauch-Tung-Striebel smoother for further smoothing the fascicle orientations. The experimental results and comparisons show that: 1) the maximum segmentation error of the proposed RCRT is below nine pixels, which is sufficiently small for MFO tracking; 2) the accuracy of MFO gauged by RT in the ROI enhanced by the GF is comparable to that of using multiscale vessel enhancement filter-based method and better than those of local RT and revoting Hough transform approaches; and 3) the proposed AF-BKS algorithm outperforms the other tested approaches and achieves a performance close to those obtained by experienced operators (the overall covariance obtained by the AF-BKS is 3.19, which is rather close to that of the operators, 2.86). It, thus, serves as a valuable tool for automatic estimation of fascicle orientations and possibly for other applications in musculoskeletal ultrasound images.
Collapse
|
4
|
Kocev B, Hahn HK, Linsen L, Wells WM, Kikinis R. Uncertainty-aware asynchronous scattered motion interpolation using Gaussian process regression. Comput Med Imaging Graph 2019; 72:1-12. [PMID: 30654093 PMCID: PMC6433137 DOI: 10.1016/j.compmedimag.2018.12.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2018] [Revised: 08/16/2018] [Accepted: 12/03/2018] [Indexed: 11/28/2022]
Abstract
We address the problem of interpolating randomly non-uniformly spatiotemporally scattered uncertain motion measurements, which arises in the context of soft tissue motion estimation. Soft tissue motion estimation is of great interest in the field of image-guided soft-tissue intervention and surgery navigation, because it enables the registration of pre-interventional/pre-operative navigation information on deformable soft-tissue organs. To formally define the measurements as spatiotemporally scattered motion signal samples, we propose a novel motion field representation. To perform the interpolation of the motion measurements in an uncertainty-aware optimal unbiased fashion, we devise a novel Gaussian process (GP) regression model with a non-constant-mean prior and an anisotropic covariance function and show through an extensive evaluation that it outperforms the state-of-the-art GP models that have been deployed previously for similar tasks. The employment of GP regression enables the quantification of uncertainty in the interpolation result, which would allow the amount of uncertainty present in the registered navigation information governing the decisions of the surgeon or intervention specialist to be conveyed.
Collapse
Affiliation(s)
- Bojan Kocev
- Department of Mathematics and Computer Science, University of Bremen, Bremen, Germany; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany; Department of Computer Science and Electrical Engineering, Jacobs University Bremen, Bremen, Germany.
| | - Horst Karl Hahn
- Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany; Department of Computer Science and Electrical Engineering, Jacobs University Bremen, Bremen, Germany
| | - Lars Linsen
- Institute of Computer Science, Westfälische Wilhelms-Universität Münster, Germany
| | - William M Wells
- Department of Radiology, Harvard Medical School and Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Ron Kikinis
- Department of Mathematics and Computer Science, University of Bremen, Bremen, Germany; Fraunhofer Institute for Medical Image Computing MEVIS, Bremen, Germany; Department of Radiology, Harvard Medical School and Brigham and Women's Hospital, Boston, MA 02115, USA
| |
Collapse
|
5
|
|
6
|
Stegmaier J, Mikut R. Fuzzy-based propagation of prior knowledge to improve large-scale image analysis pipelines. PLoS One 2017; 12:e0187535. [PMID: 29095927 PMCID: PMC5667823 DOI: 10.1371/journal.pone.0187535] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2017] [Accepted: 10/20/2017] [Indexed: 11/19/2022] Open
Abstract
Many automatically analyzable scientific questions are well-posed and a variety of information about expected outcomes is available a priori. Although often neglected, this prior knowledge can be systematically exploited to make automated analysis operations sensitive to a desired phenomenon or to evaluate extracted content with respect to this prior knowledge. For instance, the performance of processing operators can be greatly enhanced by a more focused detection strategy and by direct information about the ambiguity inherent in the extracted data. We present a new concept that increases the result quality awareness of image analysis operators by estimating and distributing the degree of uncertainty involved in their output based on prior knowledge. This allows the use of simple processing operators that are suitable for analyzing large-scale spatiotemporal (3D+t) microscopy images without compromising result quality. On the foundation of fuzzy set theory, we transform available prior knowledge into a mathematical representation and extensively use it to enhance the result quality of various processing operators. These concepts are illustrated on a typical bioimage analysis pipeline comprised of seed point detection, segmentation, multiview fusion and tracking. The functionality of the proposed approach is further validated on a comprehensive simulated 3D+t benchmark data set that mimics embryonic development and on large-scale light-sheet microscopy data of a zebrafish embryo. The general concept introduced in this contribution represents a new approach to efficiently exploit prior knowledge to improve the result quality of image analysis pipelines. The generality of the concept makes it applicable to practically any field with processing strategies that are arranged as linear pipelines. The automated analysis of terabyte-scale microscopy data will especially benefit from sophisticated and efficient algorithms that enable a quantitative and fast readout.
Collapse
Affiliation(s)
- Johannes Stegmaier
- Institute for Applied Computer Science, Karlsruhe Institute of Technology, Eggenstein-Leopoldshafen, Germany
- * E-mail:
| | - Ralf Mikut
- Institute for Applied Computer Science, Karlsruhe Institute of Technology, Eggenstein-Leopoldshafen, Germany
| |
Collapse
|
7
|
Nascimento JC, Carneiro G. Deep Learning on Sparse Manifolds for Faster Object Segmentation. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2017; 26:4978-4990. [PMID: 28708556 DOI: 10.1109/tip.2017.2725582] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
We propose a new combination of deep belief networks and sparse manifold learning strategies for the 2D segmentation of non-rigid visual objects. With this novel combination, we aim to reduce the training and inference complexities while maintaining the accuracy of machine learning-based non-rigid segmentation methodologies. Typical non-rigid object segmentation methodologies divide the problem into a rigid detection followed by a non-rigid segmentation, where the low dimensionality of the rigid detection allows for a robust training (i.e., a training that does not require a vast amount of annotated images to estimate robust appearance and shape models) and a fast search process during inference. Therefore, it is desirable that the dimensionality of this rigid transformation space is as small as possible in order to enhance the advantages brought by the aforementioned division of the problem. In this paper, we propose the use of sparse manifolds to reduce the dimensionality of the rigid detection space. Furthermore, we propose the use of deep belief networks to allow for a training process that can produce robust appearance models without the need of large annotated training sets. We test our approach in the segmentation of the left ventricle of the heart from ultrasound images and lips from frontal face images. Our experiments show that the use of sparse manifolds and deep belief networks for the rigid detection stage leads to segmentation results that are as accurate as the current state of the art, but with lower search complexity and training processes that require a small amount of annotated training data.
Collapse
|
8
|
Allan G, Nouranian S, Tsang T, Seitel A, Mirian M, Jue J, Hawley D, Fleming S, Gin K, Swift J, Rohling R, Abolmaesumi P. Simultaneous Analysis of 2D Echo Views for Left Atrial Segmentation and Disease Detection. IEEE TRANSACTIONS ON MEDICAL IMAGING 2017; 36:40-50. [PMID: 27455520 DOI: 10.1109/tmi.2016.2593900] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
We propose a joint information approach for automatic analysis of 2D echocardiography (echo) data. The approach combines a priori images, their segmentations and patient diagnostic information within a unified framework to determine various clinical parameters, such as cardiac chamber volumes, and cardiac disease labels. The main idea behind the approach is to employ joint Independent Component Analysis of both echo image intensity information and corresponding segmentation labels to generate models that jointly describe the image and label space of echo patients on multiple apical views, instead of independently. These models are then both used for segmentation and volume estimation of cardiac chambers such as the left atrium and for detecting pathological abnormalities such as mitral regurgitation. We validate the approach on a large cohort of echoes obtained from 6,993 studies. We report performance of the proposed approach in estimation of the left-atrium volume and detection of mitral-regurgitation severity. A correlation coefficient of 0.87 was achieved for volume estimation of the left atrium when compared to the clinical report. Moreover, we classified patients that suffer from moderate or severe mitral regurgitation with an average accuracy of 82%.
Collapse
|
9
|
|
10
|
Santiago C, Nascimento JC, Marques JS. A new ASM framework for left ventricle segmentation exploring slice variability in cardiac MRI volumes. Neural Comput Appl 2016. [DOI: 10.1007/s00521-016-2337-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/03/2023]
|
11
|
Balaji G, Subashini T, Chidambaram N. Automatic Classification of Cardiac Views in Echocardiogram Using Histogram and Statistical Features. ACTA ACUST UNITED AC 2015. [DOI: 10.1016/j.procs.2015.02.084] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
12
|
Fusing correspondenceless 3D point distribution models. ACTA ACUST UNITED AC 2014. [PMID: 24505673 DOI: 10.1007/978-3-642-40811-3_32] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register]
Abstract
This paper presents a framework for the fusion of multiple point distribution models (PDMs) with unknown point correspondences. With this work, models built from distinct patient groups and imaging modalities can be merged, with the aim to obtain a PDM that encodes a wider range of anatomical variability. To achieve this, two technical challenges are addressed in this work. Firstly, the model fusion must be carried out directly on the corresponding means and eigenvectors as the original data is not always available and cannot be freely exchanged across centers for various legal and practical reasons. Secondly, the PDMs need to be normalized before fusion as the point correspondence is unknown. The proposed framework is validated by integrating statistical models of the left and right ventricles of the heart constructed from different imaging modalities (MRI and CT) and with different landmark representations of the data. The results show that the integration is statistically and anatomically meaningful and that the quality of the resulting model is significantly improved.
Collapse
|
13
|
Dietenbeck T, Barbosa D, Alessandrini M, Jasaityte R, Robesyn V, D'hooge J, Friboulet D, Bernard O. Whole myocardium tracking in 2D-echocardiography in multiple orientations using a motion constrained level-set. Med Image Anal 2014; 18:500-14. [PMID: 24561989 DOI: 10.1016/j.media.2014.01.005] [Citation(s) in RCA: 14] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2013] [Revised: 01/08/2014] [Accepted: 01/24/2014] [Indexed: 11/18/2022]
Abstract
The segmentation and tracking of the myocardium in echocardiographic sequences is an important task for the diagnosis of heart disease. This task is difficult due to the inherent problems of echographic images (i.e. low contrast, speckle noise, signal dropout, presence of shadows). In this article, we extend a level-set method recently proposed in Dietenbeck et al. (2012) in order to track the whole myocardium in echocardiographic sequences. To this end, we enforce temporal coherence by adding a new motion prior energy to the existing framework. This motion prior term is expressed as new constraint that enforces the conservation of the levels of the implicit function along the image sequence. Moreover, the robustness of the proposed method is improved by adjusting the associated hyperparameters in a spatially adaptive way, using the available strong a priori about the echocardiographic regions to be segmented. The accuracy and robustness of the proposed method is evaluated by comparing the obtained segmentation with experts references and to another state-of-the-art method on a dataset of 15 sequences (≃ 900 images) acquired in three echocardiographic views. We show that the algorithm provides results that are consistent with the inter-observer variability and outperforms the state-of-the-art method. We also carry out a complete study on the influence of the parameters settings. The obtained results demonstrate the stability of our method according to those values.
Collapse
Affiliation(s)
- T Dietenbeck
- Université de Lyon, CREATIS, CNRS UMR5220, INSERM U1044, Université Lyon 1, INSA-LYON, France.
| | - D Barbosa
- Université de Lyon, CREATIS, CNRS UMR5220, INSERM U1044, Université Lyon 1, INSA-LYON, France; Cardiovascular Imaging and Dynamics, KU Leuven, Leuven, Belgium
| | - M Alessandrini
- Université de Lyon, CREATIS, CNRS UMR5220, INSERM U1044, Université Lyon 1, INSA-LYON, France
| | - R Jasaityte
- Cardiovascular Imaging and Dynamics, KU Leuven, Leuven, Belgium
| | - V Robesyn
- Cardiovascular Imaging and Dynamics, KU Leuven, Leuven, Belgium
| | - J D'hooge
- Cardiovascular Imaging and Dynamics, KU Leuven, Leuven, Belgium
| | - D Friboulet
- Université de Lyon, CREATIS, CNRS UMR5220, INSERM U1044, Université Lyon 1, INSA-LYON, France
| | - O Bernard
- Université de Lyon, CREATIS, CNRS UMR5220, INSERM U1044, Université Lyon 1, INSA-LYON, France
| |
Collapse
|
14
|
Carneiro G, Nascimento JC. Combining multiple dynamic models and deep learning architectures for tracking the left ventricle endocardium in ultrasound data. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2013; 35:2592-2607. [PMID: 24051722 DOI: 10.1109/tpami.2013.96] [Citation(s) in RCA: 61] [Impact Index Per Article: 5.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/02/2023]
Abstract
We present a new statistical pattern recognition approach for the problem of left ventricle endocardium tracking in ultrasound data. The problem is formulated as a sequential importance resampling algorithm such that the expected segmentation of the current time step is estimated based on the appearance, shape, and motion models that take into account all previous and current images and previous segmentation contours produced by the method. The new appearance and shape models decouple the affine and nonrigid segmentations of the left ventricle to reduce the running time complexity. The proposed motion model combines the systole and diastole motion patterns and an observation distribution built by a deep neural network. The functionality of our approach is evaluated using a dataset of diseased cases containing 16 sequences and another dataset of normal cases comprised of four sequences, where both sets present long axis views of the left ventricle. Using a training set comprised of diseased and healthy cases, we show that our approach produces more accurate results than current state-of-the-art endocardium tracking methods in two test sequences from healthy subjects. Using three test sequences containing different types of cardiopathies, we show that our method correlates well with interuser statistics produced by four cardiologists.
Collapse
|
15
|
Jiang N, Su H, Liu W, Wu Y. Discriminative metric preservation for tracking low-resolution targets. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2012; 21:1284-1297. [PMID: 21908254 DOI: 10.1109/tip.2011.2167345] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/31/2023]
Abstract
Tracking low-resolution (LR) targets is a practical yet quite challenging problem in real video analysis applications. Lack of discriminative details in the visual appearance of the LR target leads to the matching ambiguity, which confronts most existing tracking methods. Although artificially enhancing the video resolution by superresolution (SR) techniques before analyzing might be an option, the high demand of computational cost can hardly meet the requirements of the tracking scenario. This paper presents a novel solution to track LR targets without explicitly performing SR. This new approach is based on discriminative metric preservation that preserves the data affinity structure in the high-resolution (HR) feature space for effective and efficient matching of LR images. In addition, we substantialize this new approach in a solid case study of differential tracking under metric preservation and derive a closed-form solution to motion estimation for LR video. In addition, this paper extends the basic linear metric preservation method to a more powerful nonlinear kernel metric preservation method. Such a solution to LR target tracking is discriminative, robust, and efficient. Extensive experiments validate the entrustments and effectiveness of the proposed approach and demonstrate the improved performance of the proposed method in tracking LR targets.
Collapse
Affiliation(s)
- Nan Jiang
- Department of Electronics and Information Engineering, Huazhong University of Science and Technology, Wuhan, China.
| | | | | | | |
Collapse
|
16
|
Carneiro G, Nascimento JC, Freitas A. The segmentation of the left ventricle of the heart from ultrasound data using deep learning architectures and derivative-based search methods. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2012; 21:968-982. [PMID: 21947526 DOI: 10.1109/tip.2011.2169273] [Citation(s) in RCA: 93] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/31/2023]
Abstract
We present a new supervised learning model designed for the automatic segmentation of the left ventricle (LV) of the heart in ultrasound images. We address the following problems inherent to supervised learning models: 1) the need of a large set of training images; 2) robustness to imaging conditions not present in the training data; and 3) complex search process. The innovations of our approach reside in a formulation that decouples the rigid and nonrigid detections, deep learning methods that model the appearance of the LV, and efficient derivative-based search algorithms. The functionality of our approach is evaluated using a data set of diseased cases containing 400 annotated images (from 12 sequences) and another data set of normal cases comprising 80 annotated images (from two sequences), where both sets present long axis views of the LV. Using several error measures to compute the degree of similarity between the manual and automatic segmentations, we show that our method not only has high sensitivity and specificity but also presents variations with respect to a gold standard (computed from the manual annotations of two experts) within interuser variability on a subset of the diseased cases. We also compare the segmentations produced by our approach and by two state-of-the-art LV segmentation models on the data set of normal cases, and the results show that our approach produces segmentations that are comparable to these two approaches using only 20 training images and increasing the training set to 400 images causes our approach to be generally more accurate. Finally, we show that efficient search methods reduce up to tenfold the complexity of the method while still producing competitive segmentations. In the future, we plan to include a dynamical model to improve the performance of the algorithm, to use semisupervised learning methods to reduce even more the dependence on rich and large training sets, and to design a shape model less dependent on the training set.
Collapse
Affiliation(s)
- Gustavo Carneiro
- Australian Centre for Visual Technologies, University of Adelaide, Adelaide, SA 5005, Australia.
| | | | | |
Collapse
|
17
|
|
18
|
Saragih JM, Lucey S, Cohn JF. Deformable Model Fitting by Regularized Landmark Mean-Shift. Int J Comput Vis 2010. [DOI: 10.1007/s11263-010-0380-4] [Citation(s) in RCA: 620] [Impact Index Per Article: 41.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
19
|
Brox T, Rosenhahn B, Gall J, Cremers D. Combined region and motion-based 3D tracking of rigid and articulated objects. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2010; 32:402-415. [PMID: 20075468 DOI: 10.1109/tpami.2009.32] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/28/2023]
Abstract
In this paper, we propose the combined use of complementary concepts for 3D tracking: region fitting on one side and dense optical flow as well as tracked SIFT features on the other. Both concepts are chosen such that they can compensate for the shortcomings of each other. While tracking by the object region can prevent the accumulation of errors, optical flow and SIFT can handle larger transformations. Whereas segmentation works best in case of homogeneous objects, optical flow computation and SIFT tracking rely on sufficiently structured objects. We show that a sensible combination yields a general tracking system that can be applied in a large variety of scenarios without the need to manually adjust weighting parameters.
Collapse
Affiliation(s)
- Thomas Brox
- University of California, Berkeley, CA, USA.
| | | | | | | |
Collapse
|
20
|
Abstract
The finite mixture model is widely used in various statistical learning problems. However, the model obtained may contain a large number of components, making it inefficient in practical applications. In this paper, we propose to simplify the mixture model by minimizing an upper bound of the approximation error between the original and the simplified model, under the use of the L (2) distance measure. This is achieved by first grouping similar components together and then performing local fitting through function approximation. The simplified model obtained can then be used as a replacement of the original model to speed up various algorithms involving mixture models during training (e.g., Bayesian filtering, belief propagation) and testing [e.g., kernel density estimation, support vector machine (SVM) testing]. Encouraging results are observed in the experiments on density estimation, clustering-based image segmentation, and simplification of SVM decision functions.
Collapse
Affiliation(s)
- James T Kwok
- Lawrence Berkeley National Laboratory,Berkeley, CA 94720, USA.
| | | |
Collapse
|
21
|
Yang M, Wu Y, Hua G. Context-aware visual tracking. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2009; 31:1195-1209. [PMID: 19443919 DOI: 10.1109/tpami.2008.146] [Citation(s) in RCA: 26] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
Enormous uncertainties in unconstrained environments lead to a fundamental dilemma that many tracking algorithms have to face in practice: Tracking has to be computationally efficient, but verifying whether or not the tracker is following the true target tends to be demanding, especially when the background is cluttered and/or when occlusion occurs. Due to the lack of a good solution to this problem, many existing methods tend to be either effective but computationally intensive by using sophisticated image observation models or efficient but vulnerable to false alarms. This greatly challenges long-duration robust tracking. This paper presents a novel solution to this dilemma by considering the context of the tracking scene. Specifically, we integrate into the tracking process a set of auxiliary objects that are automatically discovered in the video on the fly by data mining. Auxiliary objects have three properties, at least in a short time interval: 1) persistent co-occurrence with the target, 2) consistent motion correlation to the target, and 3) easy to track. Regarding these auxiliary objects as the context of the target, the collaborative tracking of these auxiliary objects leads to efficient computation as well as strong verification. Our extensive experiments have exhibited exciting performance in very challenging real-world testing cases.
Collapse
Affiliation(s)
- Ming Yang
- Electrical Engineering and Computer Science Department, Northwestern University, Evanston, IL 60208, USA.
| | | | | |
Collapse
|
22
|
Yun Lei, Xiaoqing Ding, Shengjin Wang. Visual Tracker Using Sequential Bayesian Learning: Discriminative, Generative, and Hybrid. ACTA ACUST UNITED AC 2008; 38:1578-91. [DOI: 10.1109/tsmcb.2008.928226] [Citation(s) in RCA: 20] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
23
|
Sun W, Cetin M, Chan R, Willsky AS. Learning the dynamics and time-recursive boundary detection of deformable objects. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2008; 17:2186-2200. [PMID: 18972658 DOI: 10.1109/tip.2008.2004638] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
We propose a principled framework for recursively segmenting deformable objects across a sequence of frames. We demonstrate the usefulness of this method on left ventricular segmentation across a cardiac cycle. The approach involves a technique for learning the system dynamics together with methods of particle-based smoothing as well as nonparametric belief propagation on a loopy graphical model capturing the temporal periodicity of the heart. The dynamic system state is a low-dimensional representation of the boundary, and the boundary estimation involves incorporating curve evolution into recursive state estimation. By formulating the problem as one of state estimation, the segmentation at each particular time is based not only on the data observed at that instant, but also on predictions based on past and future boundary estimates. Although this paper focuses on left ventricle segmentation, the method generalizes to temporally segmenting any deformable object.
Collapse
Affiliation(s)
- Walter Sun
- Laboratory for Information and Decision Systems (LIDS), Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA.
| | | | | | | |
Collapse
|
24
|
Clinical utility of automated assessment of left ventricular ejection fraction using artificial intelligence-assisted border detection. Am Heart J 2008; 155:562-70. [PMID: 18294497 DOI: 10.1016/j.ahj.2007.11.002] [Citation(s) in RCA: 25] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/19/2007] [Accepted: 11/02/2007] [Indexed: 11/20/2022]
Abstract
BACKGROUND Ejection fraction (EF) calculated from 2-dimensional echocardiography provides important prognostic and therapeutic information in patients with heart disease. However, quantification of EF requires planimetry and is time-consuming. As a result, visual assessment is frequently used but is subjective and requires extensive experience. New computer software to assess EF automatically is now available and could be used routinely in busy digital laboratories (>15,000 studies per year) and in core laboratories running large clinical trials. We tested Siemens AutoEF software (Siemens Medical Solutions, Erlangen, Germany) to determine whether it correlated with visual estimates of EF, manual planimetry, and cardiac magnetic resonance (CMR). METHODS Siemens AutoEF is based on learned patterns and artificial intelligence. An expert and a novice reader assessed EF visually by reviewing transthoracic echocardiograms from consecutive patients. An experienced sonographer quantified EF in all studies using Simpson's method of disks. AutoEF results were compared to CMR. RESULTS Ninety-two echocardiograms were analyzed. Visual assessment by the expert (R = 0.86) and the novice reader (R = 0.80) correlated more closely with manual planimetry using Simpson's method than did AutoEF (R = 0.64). The correlation between AutoEF and CMR was 0.63, 0.28, and 0.51 for EF, end-diastolic and end-systolic volumes, respectively. CONCLUSION The discrepancies in EF estimates between AutoEF and manual tracing using Simpson's method and between AutoEF and CMR preclude routine clinical use of AutoEF until it has been validated in a number of large, busy echocardiographic laboratories. Visual assessment of EF, with its strong correlation with quantitative EF, underscores its continued clinical utility.
Collapse
|
25
|
Nascimento JC, Marques JS. Robust shape tracking with multiple models in ultrasound images. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2008; 17:392-406. [PMID: 18270127 DOI: 10.1109/tip.2007.915552] [Citation(s) in RCA: 30] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
This paper addresses object tracking in ultrasound images using a robust multiple model tracker. The proposed tracker has the following features: 1) it uses multiple dynamic models to track the evolution of the object boundary, and 2) it models invalid observations (outliers), reducing their influence on the shape estimates. The problem considered in this paper is the tracking of the left ventricle which is known to be a challenging problem. The heart motion presents two phases (diastole and systole) with different dynamics, the multiple models used in this tracker try to solve this difficulty. In addition, ultrasound images are corrupted by strong multiplicative noise which prevents the use of standard deformable models. Robust estimation techniques are used to address this difficulty. The multiple model data association (MMDA) tracker proposed in this paper is based on a bank of nonlinear filters, organized in a tree structure. The algorithm determines which model is active at each instant of time and updates its state by propagating the probability distribution, using robust estimation techniques.
Collapse
Affiliation(s)
- Jacinto C Nascimento
- Instituto Superior Tecnico, Instituta de Sistemas e Robotica, 1049-001 Lisboa, Portugal
| | | |
Collapse
|
26
|
Stiglmayr M, Schwarz R, Klamroth K, Leugering G, Lohscheller J. Registration of PE segment contour deformations in digital high-speed videos. Med Image Anal 2008; 12:318-34. [PMID: 18304855 DOI: 10.1016/j.media.2007.12.001] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2006] [Revised: 12/12/2007] [Accepted: 12/13/2007] [Indexed: 11/16/2022]
Abstract
Oncologic therapy of laryngeal cancer may necessitate a total excision of the larynx which results in loss of voice. Voice rehabilitation can be achieved using mucosal tissue vibrations at the upper part of the esophagus which serves as substitute voice generating element (PE segment). The quality of the substitute voice is closely related to vibratory characteristics of the PE segment. By means of a high-speed camera the dynamics of the PE segment can be recorded in real-time. Using image processing the deformations of the PE segment are extracted from the image series as deforming contours. Commonly, the characterization of PE dynamics bases on the spectral analysis of the time varying contour area. However, this constitutes an integral approach which masks most of the specific dynamics of PE deformations. We present an algorithm that automatically registers one segmented contour in a frame of the video sequence to the contour in the next frame to derive discrete 2-D trajectories of PE vibrations. By concatenation of the obtained transformations this approach provides a total registration of PE segment contours. We suggest a mixed-integer programming formulation for the problem that combines an advanced outlier and deformation handling with the introduction of dummy points in regions that newly open up, and that includes normal information in the objective function to avoid unwanted deformations. Numerical experiments show that the implemented alternate convex search algorithm produces robust results which is demonstrated at the example of five high-speed recordings of laryngectomee subjects.
Collapse
Affiliation(s)
- Michael Stiglmayr
- Institute of Applied Mathematics, University of Erlangen - Nürnberg, Martensstasse 3, 91058 Erlangen, Germany.
| | | | | | | | | |
Collapse
|
27
|
Shao J, Porikli F, Chellappa R. Estimation of contour motion and deformation for nonrigid object tracking. JOURNAL OF THE OPTICAL SOCIETY OF AMERICA. A, OPTICS, IMAGE SCIENCE, AND VISION 2007; 24:2109-21. [PMID: 17621317 DOI: 10.1364/josaa.24.002109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2023]
Abstract
We present an algorithm for nonrigid contour tracking in heavily cluttered background scenes. Based on the properties of nonrigid contour movements, a sequential framework for estimating contour motion and deformation is proposed. We solve the nonrigid contour tracking problem by decomposing it into three subproblems: motion estimation, deformation estimation, and shape regulation. First, we employ a particle filter to estimate the global motion parameters of the affine transform between successive frames. Then we generate a probabilistic deformation map to deform the contour. To improve robustness, multiple cues are used for deformation probability estimation. Finally, we use a shape prior model to constrain the deformed contour. This enables us to retrieve the occluded parts of the contours and accurately track them while allowing shape changes specific to the given object types. Our experiments show that the proposed algorithm significantly improves the tracker performance.
Collapse
Affiliation(s)
- Jie Shao
- Center for Automation Research and Department of Electrical and Computer Engineering, University of Maryland, College Park, Maryland 20742, USA.
| | | | | |
Collapse
|
28
|
Sun W, Qetin M, Chan R, Reddy V, Holmvang G, Chandar V, Willsky A. Segmenting and tracking the left ventricle by learning the dynamics in cardiac images. ACTA ACUST UNITED AC 2007; 19:553-65. [PMID: 17354725 DOI: 10.1007/11505730_46] [Citation(s) in RCA: 26] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/16/2023]
Abstract
Having accurate left ventricle (LV) segmentations across a cardiac cycle provides useful quantitative (e.g. ejection fraction) and qualitative information for diagnosis of certain heart conditions. Existing LV segmentation techniques are founded mostly upon algorithms for segmenting static images. In order to exploit the dynamic structure of the heart in a principled manner, we approach the problem of LV segmentation as a recursive estimation problem. In our framework, LV boundaries constitute the dynamic system state to be estimated, and a sequence of observed cardiac images constitute the data. By formulating the problem as one of state estimation, the segmentation at each particular time is based not only on the data observed at that instant, but also on predictions based on past segmentations. This requires a dynamical system model of the LV, which we propose to learn from training data through an information-theoretic approach. To incorporate the learned dynamic model into our segmentation framework and obtain predictions, we use ideas from particle filtering. Our framework uses a curve evolution method to combine such predictions with the observed images to estimate the LV boundaries at each time. We demonstrate the effectiveness of the proposed approach on a large set of cardiac images. We observe that our approach provides more accurate segmentations than those from static image segmentation techniques, especially when the observed data are of limited quality.
Collapse
Affiliation(s)
- W Sun
- Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA.
| | | | | | | | | | | | | |
Collapse
|
29
|
Qazi M, Fung G, Krishnan S, Bi J, Rao RB, Katz AS. Automated heart abnormality detection using sparse linear classifiers. ACTA ACUST UNITED AC 2007; 26:56-63. [PMID: 17441609 DOI: 10.1109/memb.2007.335591] [Citation(s) in RCA: 17] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Affiliation(s)
- Maleeha Qazi
- Siemens Medical Solutions, Malvern, Pennsylvania, USA
| | | | | | | | | | | |
Collapse
|
30
|
Butakoff C, Frangi AF. A framework for weighted fusion of multiple statistical models of shape and appearance. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2006; 28:1847-57. [PMID: 17063688 DOI: 10.1109/tpami.2006.215] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2023]
Abstract
This paper presents a framework for weighted fusion of several Active Shape and Active Appearance Models. The approach is based on the eigenspace fusion method proposed by Hall et al., which has been extended to fuse more than two weighted eigenspaces using unbiased mean and covariance matrix estimates. To evaluate the performance of fusion, a comparative assessment on segmentation precision as well as facial verification tests are performed using the AR, EQUINOX, and XM2VTS databases. Based on the results, it is concluded that the fusion is useful when the model needs to be updated online or when the original observations are absent.
Collapse
Affiliation(s)
- Constantine Butakoff
- Computational Imaging Lab, Department of Technology, University Pompeu Fabra, Barcelona, Spain.
| | | |
Collapse
|
31
|
Zhou SK, Shao J, Georgescu B, Comaniciu D. Pairwise active appearance model and its application to echocardiography tracking. MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION : MICCAI ... INTERNATIONAL CONFERENCE ON MEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION 2006; 9:736-43. [PMID: 17354956 DOI: 10.1007/11866565_90] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
We propose a pairwise active appearance model (PAAM) to characterize statistical regularities in shape, appearance, and motion presented by a target that undergoes a series of motion phases, such as the left ventricle in echocardiography. The PAAM depicts the transition in motion phase through a Markov chain and the transition in both shape and appearance through a conditional Gaussian distribution. We learn from a database the joint Gaussian distribution of the shapes and appearances belonging to two consecutive motion phases (i.e., a pair of motion phases), from which we analytically compute the conditional Gaussian distribution. We utilize the PAAM in tracking the left ventricle contour in echocardiography and obtain improved tracking results in terms of localization accuracy when compared with expert-specified contours.
Collapse
Affiliation(s)
- S Kevin Zhou
- Integrated Data Systems, Siemens Corporate Research, Inc., Princeton, NJ, USA.
| | | | | | | |
Collapse
|
32
|
Nascimento JC, Sanches JM, Marques JS. A method for the dynamic analysis of the heart using a Lyapounov based denoising algorithm. CONFERENCE PROCEEDINGS : ... ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL CONFERENCE 2006; 2006:4828-4831. [PMID: 17946654 DOI: 10.1109/iembs.2006.260183] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/25/2023]
Abstract
Heart tracking in ultrasound sequences is a difficult task due to speckle noise, low SNR and lack of contrast. Therefore it is usually difficult to obtain robust estimates of the heart cavities since feature detectors produce a large number of outliers. This paper presents an algorithm which combines two main operations: i) a novel denoising algorithm based on the Lyapounov equation and ii) a robust tracker, recently proposed by the authors, based on a model of the outlier features. Experimental results are provided, showing that the proposed algorithm is computationally efficient and leads to accurate estimates of the left ventricle during the cardiac cycle.
Collapse
|
33
|
Identification of Luminal and Medial Adventitial Borders in Intravascular Ultrasound Images Using Level Sets. ACTA ACUST UNITED AC 2006. [DOI: 10.1007/11902140_61] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register]
|