Image segmentation based on GrabCut framework integrating multiscale nonlinear structure tensor

Efficientand Robust Automated Segmentation of Nanoparticles and Aggregates from Transmission Electron Microscopy Images with Highly Complex Backgrounds

Morphologies of nanoparticles and aggregates play an important role in their properties for a range of applications. In particular, significant synthesis efforts have been directed toward controlling nanoparticle morphology and aggregation behavior in biomedical applications, as their size and shape have a significant impact on cellular uptake. Among several techniques for morphological characterization, transmission electron microscopy (TEM) can provide direct and accurate characterization of nanoparticle/aggregate morphology details. Nevertheless, manually analyzing a large number of TEM images is still a laborious process. Hence, there has been a surge of interest in employing machine learning methods to analyze nanoparticle size and shape. In order to achieve accurate nanoparticle analysis using machine learning methods, reliable and automated nanoparticle segmentation from TEM images is critical, especially when the nanoparticle image contrast is weak and the background is complex. These challenges are particularly pertinent in biomedical applications. In this work, we demonstrate an efficient, robust, and automated nanoparticle image segmentation method suitable for subsequent machine learning analysis. Our method is robust for noisy, low-electron-dose cryo-TEM images and for TEM cell images with complex, strong-contrast background features. Moreover, our method does not require any a priori training datasets, making it efficient and general. The ability to automatically, reliably, and efficiently segment nanoparticle/aggregate images is critical for advancing precise particle/aggregate control in biomedical applications.

Graphical Image Region Extraction with K-Means Clustering and Watershed

With a wide range of applications, image segmentation is a complex and difficult preprocessing step that plays an important role in automatic visual systems, which accuracy impacts, not only on segmentation results, but directly affects the effectiveness of the follow-up tasks. Despite the many advances achieved in the last decades, image segmentation remains a challenging problem, particularly, the segmenting of color images due to the diverse inhomogeneities of color, textures and shapes present in the descriptive features of the images. In trademark graphic images segmentation, beyond these difficulties, we must also take into account the high noise and low resolution, which are often present. Trademark graphic images can also be very heterogeneous with regard to the elements that make them up, which can be overlapping and with varying lighting conditions. Due to the immense variation encountered in corporate logos and trademark graphic images, it is often difficult to select a single method for extracting relevant image regions in a way that produces satisfactory results. Many of the hybrid approaches that integrate the Watershed and K-Means algorithms involve processing very high quality and visually similar images, such as medical images, meaning that either approach can be tweaked to work on images that follow a certain pattern. Trademark images are totally different from each other and are usually fully colored. Our system solves this difficulty given it is a generalized implementation designed to work in most scenarios, through the use of customizable parameters and completely unbiased for an image type. In this paper, we propose a hybrid approach to Image Region Extraction that focuses on automated region proposal and segmentation techniques. In particular, we analyze popular techniques such as K-Means Clustering and Watershedding and their effectiveness when deployed in a hybrid environment to be applied to a highly variable dataset. The proposed system consists of a multi-stage algorithm that takes as input an RGB image and produces multiple outputs, corresponding to the extracted regions. After preprocessing steps, a K-Means function with random initial centroids and a user-defined value for k is executed over the RGB image, generating a gray-scale segmented image, to which a threshold method is applied to generate a binary mask, containing the necessary information to generate a distance map. Then, the Watershed function is performed over the distance map, using the markers defined by the Connected Component Analysis function that labels regions on 8-way pixel connectivity, ensuring that all regions are correctly found. Finally, individual objects are labelled for extraction through a contour method, based on border following. The achieved results show adequate region extraction capabilities when processing graphical images from different datasets, where the system correctly distinguishes the most relevant visual elements of images with minimal tweaking.

3D Interactive Segmentation With Semi-Implicit Representation and Active Learning

Segmenting complex 3D geometry is a challenging task due to rich structural details and complex appearance variations of target object. Shape representation and foreground-background delineation are two of the core components of segmentation. Explicit shape models, such as mesh based representations, suffer from poor handling of topological changes. On the other hand, implicit shape models, such as level-set based representations, have limited capacity for interactive manipulation. Fully automatic segmentation for separating foreground objects from background generally utilizes non-interoperable machine learning methods, which heavily rely on the off-line training dataset and are limited to the discrimination power of the chosen model. To address these issues, we propose a novel semi-implicit representation method, namely Non-Uniform Implicit B-spline Surface (NU-IBS), which adaptively distributes parametrically blended patches according to geometrical complexity. Then, a two-stage cascade classifier is introduced to carry out efficient foreground and background delineation, where a simplistic Naïve-Bayesian model is trained for fast background elimination, followed by a stronger pseudo-3D Convolutional Neural Network (CNN) multi-scale classifier to precisely identify the foreground objects. A localized interactive and adaptive segmentation scheme is incorporated to boost the delineation accuracy by utilizing the information iteratively gained from user intervention. The segmentation result is obtained via deforming an NU-IBS according to the probabilistic interpretation of delineated regions, which also imposes a homogeneity constrain for individual segments. The proposed method is evaluated on a 3D cardiovascular Computed Tomography Angiography (CTA) image dataset and Brain Tumor Image Segmentation Benchmark 2015 (BraTS2015) 3D Magnetic Resonance Imaging (MRI) dataset.

Ensemble Learning with Multiclassifiers on Pediatric Hand Radiograph Segmentation for Bone Age Assessment

In the study of pediatric automatic bone age assessment (BAA) in clinical practice, the extraction of the object area in hand radiographs is an important part, which directly affects the prediction accuracy of the BAA. But no perfect segmentation solution has been found yet. This work is to develop an automatic hand radiograph segmentation method with high precision and efficiency. We considered the hand segmentation task as a classification problem. The optimal segmentation threshold for each image was regarded as the prediction target. We utilized the normalized histogram, mean value, and variance of each image as input features to train the classification model, based on ensemble learning with multiple classifiers. 600 left-hand radiographs with the bone age ranging from 1 to 18 years old were included in the dataset. Compared with traditional segmentation methods and the state-of-the-art U-Net network, the proposed method performed better with a higher precision and less computational load, achieving an average PSNR of 52.43 dB, SSIM of 0.97, DSC of 0.97, and JSI of 0.91, which is more suitable in clinical application. Furthermore, the experimental results also verified that hand radiograph segmentation could bring an average improvement for BAA performance of at least 13%.

Probabilistic Diffusion for Interactive Image Segmentation

This paper presents an interactive image segmentation approach in which we formulate segmentation as a probabilistic estimation problem based on the prior user intention. Instead of directly measuring the relationship between pixels and labels, we first estimate the distances between pixel pairs and label pairs using a probabilistic framework. Then, binary probabilities with label pairs are naturally converted to unary probabilities with labels. The higher order relationship helps improve the robustness to user inputs. To improve segmentation accuracy, a likelihood learning framework is proposed to fuse the region and the boundary information of the image by imposing a smoothing constraint on the unary potentials. Furthermore, we establish an equivalence relationship between likelihood learning and likelihood diffusion and propose an iterative diffusion-based optimization strategy to maintain computational efficiency. Experiments on the Berkeley segmentation data set and Microsoft GrabCut database demonstrate that the proposed method can obtain better performance than the state-of-the-art methods.

Automatic Segmentation of Ultrasound Tomography Image

Ultrasound tomography (UST) image segmentation is fundamental in breast density estimation, medicine response analysis, and anatomical change quantification. Existing methods are time consuming and require massive manual interaction. To address these issues, an automatic algorithm based on GrabCut (AUGC) is proposed in this paper. The presented method designs automated GrabCut initialization for incomplete labeling and is sped up with multicore parallel programming. To verify performance, AUGC is applied to segment thirty-two in vivo UST volumetric images. The performance of AUGC is validated with breast overlapping metrics (Dice coefficient (D), Jaccard (J), and False positive (FP)) and time cost (TC). Furthermore, AUGC is compared to other methods, including Confidence Connected Region Growing (CCRG), watershed, and Active Contour based Curve Delineation (ACCD). Experimental results indicate that AUGC achieves the highest accuracy (D = 0.9275 and J = 0.8660 and FP = 0.0077) and takes on average about 4 seconds to process a volumetric image. It was said that AUGC benefits large-scale studies by using UST images for breast cancer screening and pathological quantification.

Feasible and Robust Optimization Framework for Auxiliary Information Refinement in Spatially-Varying Image Enhancement

In content-based image processing, the precise inference of auxiliary information dominates various image enhancement applications. Given the rough auxiliary information provided by users or inference algorithms, a common scenario is to refine it with respect to the image content. Quadratic Laplacian regularization is generally used as the refinement framework because of the availability of closed-form solutions. However, solving the resultant large linear system imposes a great burden on commodity computing hardware systems in the form of computational time and memory consumption, so efficient computing algorithms without losing precision are required, especially for large images. In this paper, we first analyze the geometric nature of the quadratic Laplacian regularization associated with the algebraic property of the corresponding linear system, which clarifies the essential issues causing ineffective solutions for conventional optimization algorithms. Correspondingly, we propose an optimization scheme that is capable of approaching the closed-form solution in an efficient manner using existing fast local filters, and we perform a spectral analysis to validate the robustness of this method in severe conditions. Finally, experimental results show that the proposed scheme is more feasible for large input images and is more robust to obtain the effective refinement than conventional algorithms.

A Novel PDE-Based Single Image Super-Resolution Reconstruction Method

For applications such as remote sensing imaging and medical imaging, high-resolution (HR) images are urgently required. Image Super-Resolution (SR) reconstruction has great application prospects in optical imaging. In this paper, we propose a novel unified Partial Differential Equation (PDE)-based method to single image SR reconstruction. Firstly, two directional diffusion terms calculated by Anisotropic Nonlinear Structure Tensor (ANLST) are constructed, combing information of all channels to prevent singular results, making full use of its directional diffusion feature. Secondly, by introducing multiple orientations estimation using high order matrix-valued tensor instead of gradient, orientations can be estimated more precisely for junctions or corners. As a unique descriptor of orientations, mixed orientation parameter (MOP) is separated into two orientations by finding roots of a second-order polynomial in the nonlinear part. Then, we synthesize a Gradient Vector Flow (GVF) shock filter to balance edge enhancement and de-noising process. Experimental results confirm the validity of the method and show that the method enhances image edges, restores corners or junctions, and suppresses noise robustness, which is competitive with the existing methods.

Neutro-Connectedness Cut

Interactive image segmentation is a challenging task and receives increasing attention recently; however, two major drawbacks exist in interactive segmentation approaches. First, the segmentation performance of region of interest (ROI)-based methods is sensitive to the initial ROI: different ROIs may produce results with great difference. Second, most seed-based methods need intense interactions, and are not applicable in many cases. In this paper, we generalize the neutro-connectedness (NC) to be independent of top-down priors of objects and to model image topology with indeterminacy measurement on image regions, propose a novel method for determining object and background regions, which is applied to exclude isolated background regions and enforce label consistency, and put forward a hybrid interactive segmentation method, NC Cut (NC-Cut), which can overcome the above two problems by utilizing both pixelwise appearance information and region-based NC properties. We evaluate the proposed NC-Cut by employing two image data sets (265 images), and demonstrate that the proposed approach outperforms the state-of-the-art interactive image segmentation methods (Grabcut, MILCut, One-Cut, MGC_max^sum, and pPBC).

Semiautomated hybrid algorithm for estimation of three-dimensional liver surface in CT using dynamic cellular automata and level-sets

Liver segmentation continues to remain a major challenge, largely due to its intense complexity with surrounding anatomical structures (stomach, kidney, and heart), high noise level and lack of contrast in pathological computed tomography (CT) data. We present an approach to reconstructing the liver surface in low contrast CT. The main contributions are: (1) a stochastic resonance-based methodology in discrete cosine transform domain is developed to enhance the contrast of pathological liver images, (2) a new formulation is proposed to prevent the object boundary, resulting from the cellular automata method, from leaking into the surrounding areas of similar intensity, and (3) a level-set method is suggested to generate intermediate segmentation contours from two segmented slices distantly located in a subject sequence. We have tested the algorithm on real datasets obtained from two sources, Hamad General Hospital and medical image computing and computer-assisted interventions grand challenge workshop. Various parameters in the algorithm, such as [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], [Formula: see text], and [Formula: see text], play imperative roles, thus their values are precisely selected. Both qualitative and quantitative evaluation performed on liver data show promising segmentation accuracy when compared with ground truth data reflecting the potential of the proposed method.

Tensor scale: An analytic approach with efficient computation and applications

Scale is a widely used notion in computer vision and image understanding that evolved in the form of scale-space theory where the key idea is to represent and analyze an image at various resolutions. Recently, we introduced a notion of local morphometric scale referred to as "tensor scale" using an ellipsoidal model that yields a unified representation of structure size, orientation and anisotropy. In the previous work, tensor scale was described using a 2-D algorithmic approach and a precise analytic definition was missing. Also, the application of tensor scale in 3-D using the previous framework is not practical due to high computational complexity. In this paper, an analytic definition of tensor scale is formulated for n-dimensional (n-D) images that captures local structure size, orientation and anisotropy. Also, an efficient computational solution in 2- and 3-D using several novel differential geometric approaches is presented and the accuracy of results is experimentally examined. Also, a matrix representation of tensor scale is derived facilitating several operations including tensor field smoothing to capture larger contextual knowledge. Finally, the applications of tensor scale in image filtering and n-linear interpolation are presented and the performance of their results is examined in comparison with respective state-of-art methods. Specifically, the performance of tensor scale based image filtering is compared with gradient and Weickert's structure tensor based diffusive filtering algorithms. Also, the performance of tensor scale based n-linear interpolation is evaluated in comparison with standard n-linear and windowed-sinc interpolation methods.

Iterative narrowband-based graph cuts optimization for geodesic active contours with region forces (GACWRF)

In this paper, an iterative narrow-band-based graph cuts (INBBGC) method is proposed to optimize the geodesic active contours with region forces (GACWRF) model for interactive object segmentation. Based on cut metric on graphs proposed by Boykov and Kolmogorov, an NBBGC method is devised to compute the local minimization of GAC. An extension to an iterative manner, namely, INBBGC, is developed for less sensitivity to the initial curve. The INBBGC method is similar to graph-cuts-based active contour (GCBAC) presented by Xu , and their differences have been analyzed and discussed. We then integrate the region force into GAC. An improved INBBGC (IINBBGC) method is proposed to optimize the GACWRF model, thus can effectively deal with the concave region and complicated real-world images segmentation. Two region force models such as mean and probability models are studied. Therefore, the GCBAC method can be regarded as the special case of our proposed IINBBGC method without region force. Our proposed algorithm has been also analyzed to be similar to the Grabcut method when the Gaussian mixture model region force is adopted, and the band region is extended to the whole image. Thus, our proposed IINBBGC method can be regarded as narrow-band-based Grabcut method or GCBAC with region force method. We apply our proposed IINBBGC algorithm on synthetic and real-world images to emphasize its performance, compared with other segmentation methods, such as GCBAC and Grabcut methods.

Designing and testing scene enhancement algorithms for patients with retina degenerative disorders

Background

Retina degenerative disorders represent the primary cause of blindness in UK and in the developed world. In particular, Age Related Macular Degeneration (AMD) and Retina Pigmentosa (RP) diseases are of interest to this study. We have therefore created new image processing algorithms for enhancing the visual scenes for them.

Methods

In this paper we present three novel image enhancement techniques aimed at enhancing the remaining visual information for patients suffering from retina dystrophies. Currently, the only effective way to test novel technology for visual enhancement is to undergo testing on large numbers of patients. To test our techniques, we have therefore built a retinal image processing model and compared the results to data from patient testing. In particular we focus on the ability of our image processing techniques to achieve improved face detection and enhanced edge perception.

Results

Results from our model are compared to actual data obtained from testing the performance of these algorithms on 27 patients with an average visual acuity of 0.63 and an average contrast sensitivity of 1.22. Results show that Tinted Reduced Outlined Nature (TRON) and Edge Overlaying algorithms are most beneficial for dynamic scenes such as motion detection. Image Cartoonization was most beneficial for spatial feature detection such as face detection. Patient's stated that they would most like to see Cartoonized images for use in daily life.

Conclusions

Results obtained from our retinal model and from patients show that there is potential for these image processing techniques to improve visual function amongst the visually impaired community. In addition our methodology using face detection and efficiency of perceived edges in determining potential benefit derived from different image enhancement algorithms could also prove to be useful in quantitatively assessing algorithms in future studies.