Uncertainty-aware guided volume segmentation

Estimation and Analysis of Slice Propagation Uncertainty in 3D Anatomy Segmentation

Supervised methods for 3D anatomy segmentation demonstrate superior performance but are often limited by the availability of annotated data. This limitation has led to a growing interest in self-supervised approaches in tandem with the abundance of available unannotated data. Slice propagation has emerged as a self-supervised approach that leverages slice registration as a self-supervised task to achieve full anatomy segmentation with minimal supervision. This approach significantly reduces the need for domain expertise, time, and the cost associated with building fully annotated datasets required for training segmentation networks. However, this shift toward reduced supervision via deterministic networks raises concerns about the trustworthiness and reliability of predictions, especially when compared with more accurate supervised approaches. To address this concern, we propose integrating calibrated uncertainty quantification (UQ) into slice propagation methods, which would provide insights into the model's predictive reliability and confidence levels. Incorporating uncertainty measures enhances user confidence in self-supervised approaches, thereby improving their practical applicability. We conducted experiments on three datasets for 3D abdominal segmentation using five UQ methods. The results illustrate that incorporating UQ improves not only model trustworthiness but also segmentation accuracy. Furthermore, our analysis reveals various failure modes of slice propagation methods that might not be immediately apparent to end-users. This study opens up new research avenues to improve the accuracy and trustworthiness of slice propagation methods.

Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation

Deep learning based methods for automatic organ segmentation have shown promise in aiding diagnosis and treatment planning. However, quantifying and understanding the uncertainty associated with model predictions is crucial in critical clinical applications. While many techniques have been proposed for epistemic or model-based uncertainty estimation, it is unclear which method is preferred in the medical image analysis setting. This paper presents a comprehensive benchmarking study that evaluates epistemic uncertainty quantification methods in organ segmentation in terms of accuracy, uncertainty calibration, and scalability. We provide a comprehensive discussion of the strengths, weaknesses, and out-of-distribution detection capabilities of each method as well as recommendations for future improvements. These findings contribute to the development of reliable and robust models that yield accurate segmentations while effectively quantifying epistemic uncertainty.

Finding Nano-Ötzi: Cryo-Electron Tomography Visualization Guided by Learned Segmentation

Cryo-electron tomography (cryo-ET) is a new 3D imaging technique with unprecedented potential for resolving submicron structural details. Existing volume visualization methods, however, are not able to reveal details of interest due to low signal-to-noise ratio. In order to design more powerful transfer functions, we propose leveraging soft segmentation as an explicit component of visualization for noisy volumes. Our technical realization is based on semi-supervised learning, where we combine the advantages of two segmentation algorithms. First, the weak segmentation algorithm provides good results for propagating sparse user-provided labels to other voxels in the same volume and is used to generate dense pseudo-labels. Second, the powerful deep-learning-based segmentation algorithm learns from these pseudo-labels to generalize the segmentation to other unseen volumes, a task that the weak segmentation algorithm fails at completely. The proposed volume visualization uses deep-learning-based segmentation as a component for segmentation-aware transfer function design. Appropriate ramp parameters can be suggested automatically through frequency distribution analysis. Furthermore, our visualization uses gradient-free ambient occlusion shading to further suppress the visual presence of noise, and to give structural detail the desired prominence. The cryo-ET data studied in our technical experiments are based on the highest-quality tilted series of intact SARS-CoV-2 virions. Our technique shows the high impact in target sciences for visual data analysis of very noisy volumes that cannot be visualized with existing techniques.

Hierarchical Random Walker Segmentation for Large Volumetric Biomedical Images

The random walker method for image segmentation is a popular tool for semi-automatic image segmentation, especially in the biomedical field. However, its linear asymptotic run time and memory requirements make application to 3D datasets of increasing sizes impractical. We propose a hierarchical framework that, to the best of our knowledge, is the first attempt to overcome these restrictions for the random walker algorithm and achieves sublinear run time and constant memory complexity. The goal of this framework is- rather than improving the segmentation quality compared to the baseline method- to make interactive segmentation on out-of-core datasets possible. The method is evaluated quantitatively on synthetic data and the CT-ORG dataset where the expected improvements in algorithm run time while maintaining high segmentation quality are confirmed. The incremental (i.e., interaction update) run time is demonstrated to be in seconds on a standard PC even for volumes of hundreds of gigabytes in size. In a small case study the applicability to large real world from current biomedical research is demonstrated. An implementation of the presented method is publicly available in version 5.2 of the widely used volume rendering and processing software Voreen (https://www.uni-muenster.de/Voreen/).

Epistemic and aleatoric uncertainties reduction with rotation variation for medical image segmentation with ConvNets

The deep convolutional neural network (ConvNet) achieves significant segmentation performance on medical images of various modalities. However, the isolated errors in a large testing set with various tumor conditions are not acceptable in clinical practice. This is usually caused in inadequate training and noise inherent during data collection, which are recognized as epistemic and aleatoric uncertainties in deep learning-based approaches. In this paper, we analyze the two types of uncertainties in medical image segmentation tasks and propose a reduction method by training models with data augmentation. The shelter zones in images are reduced with 2D imaging on surfaces of different angles from 3D organs. Rotation transformation and noise are estimated by Monte Carlo simulation with prior parameter distributions, and the aleatoric uncertainty is quantized in this process. Experiments on segmentation of computed tomography images demonstrate that overconfident incorrect predictions are reduced through uncertainty reduction and that our method outperforms prediction baselines based on epistemic and aleatoric estimation.

MHSU-Net: A more versatile neural network for medical image segmentation

BACKGROUND AND OBJECTIVE

Medical image segmentation plays an important role in clinic. Recently, with the development of deep learning, many convolutional neural network (CNN)-based medical image segmentation algorithms have been proposed. Among them, U-Net is one of the most famous networks. However, the standard convolutional layers used by U-Net limit its capability to capture abundant features. Additionally, the consecutive maximum pooling operations in U-Net cause certain features to be lost. This paper aims to improve the feature extraction capability of U-Net and reduce the feature loss during the segmentation process. Meanwhile, the paper also focuses on improving the versatility of the proposed segmentation model.

METHODS

Firstly, in order to enable the model to capture richer features, we have proposed a novel multiscale convolutional block (MCB). MCB adopts a wider and deeper structure, which can be applied to different types of segmentation tasks. Secondly, a hybrid down-sampling block (HDSB) has been proposed to reduce the feature loss via replacing the maximum pooling layer. Thirdly, we have proposed a context module (CIF) based on atrous convolution and SKNet to extract sufficient context information. Finally, we combined the CIF module with Skip Connection of U-Net, and further proposed the Skip Connection+ structure.

RESULTS

We name the proposed network MHSU-Net. MHSU-Net has been evaluated on three different datasets, including lung, cell contour, and pancreas. Experimental results demonstrate that MHSU-Net outperforms U-Net and other state-of-the-art models under various evaluation metrics, and owns greater potential in clinical applications.

CONCLUSIONS

The proposed modules can greatly improve the feature extraction capability of the segmentation model and effectively reduce the feature loss during the segmentation process. MHSU-Net can also be applied to different types of medical image segmentation tasks.

Living Donor-Recipient Pair Matching for Liver Transplant via Ternary Tree Representation With Cascade Incremental Learning

Visual understanding of liver vessels anatomy between the living donor-recipient (LDR) pair can assist surgeons to optimize transplant planning by avoiding non-targeted arteries which can cause severe complications. We propose to visually analyze the anatomical variants of the liver vessels anatomy to maximize similarity for finding a suitable Living Donor-Recipient (LDR) pair. Liver vessels are segmented from computed tomography angiography (CTA) volumes by employing a cascade incremental learning (CIL) model. Our CIL architecture is able to find optimal solutions, which we use to update the model with liver vessel CTA images. A novel ternary tree based algorithm is proposed to map all the possible liver vessel variants into their respective tree topologies. The tree topologies of the recipient's and donor's liver vessels are then used for an appropriate matching. The proposed algorithm utilizes a set of defined vessel tree variants which are updated to maintain the maximum matching options by leveraging the accurate segmentation results of the vessels derived from the incremental learning ability of the CIL. We introduce a novel concept of in-order digital string based comparison to match the geometry of two anatomically varied trees. Experiments through visual illustrations and quantitative analysis demonstrated the effectiveness of our approach compared to state-of-the-art.

Rapid methods for the evaluation of fluorescent reporters in tissue clearing and the segmentation of large vascular structures

Light sheet fluorescence microscopy (LSFM) of large tissue samples does not require mechanical sectioning and allows efficient visualization of spatially complex or rare structures. Therefore, LSFM has become invaluable in developmental and biomedical research. Because sample size may limit whole-mount staining, LSFM benefits from transgenic reporter organisms expressing fluorescent proteins (FPs) and, however, requires optical clearing and computational data visualization and analysis. The former often interferes with FPs, while the latter requires massive computing resources. Here, we describe 3D-polymerized cell dispersions, a rapid and straightforward method, based on recombinant FP expression in freely selectable tester cells, to evaluate and compare fluorescence retention in different tissue-clearing protocols. For the analysis of large LSFM data, which usually requires huge computing resources, we introduce a refined, interactive, hierarchical random walker approach that is capable of efficient segmentation of the vasculature in data sets even on a consumer grade PC.

Confidence-Controlled Local Isosurfacing

This article presents a novel framework that can generate a high-fidelity isosurface model of X-ray computed tomography (CT) data. CT surfaces with subvoxel precision and smoothness can be simply modeled via isosurfacing, where a single CT value represents an isosurface. However, this inevitably results in geometric distortion of the CT data containing CT artifacts. An alternative is to treat this challenge as a segmentation problem. However, in general, segmentation techniques are not robust against noisy data and require heavy computation to handle the artifacts that occur in three-dimensional CT data. Furthermore, the surfaces generated from segmentation results may contain jagged, overly smooth, or distorted geometries. We present a novel local isosurfacing framework that can address these issues simultaneously. The proposed framework exploits two primary techniques: 1) Canny edge approach for obtaining surface candidate boundary points and evaluating their confidence and 2) screened Poisson optimization for fitting a surface to the boundary points in which the confidence term is incorporated. This combination facilitates local isosurfacing that can produce high-fidelity surface models. We also implement an intuitive user interface to alleviate the burden of selecting the appropriate confidence computing parameters. Our experimental results demonstrate the effectiveness of the proposed framework.

Visualisation of Spatial Data Uncertainty. A Case Study of a Database of Topographic Objects

The Database of Topographic Objects (DTO) is the official database of Poland for collecting and providing spatial data with the detail level of a topographic map. Polish national DTOs manage information about the spatial location and attribute values of geographic objects. Data in the DTO are the starting point for geographic information systems (GISs) for various central and local governments as well as private institutions. Every set of spatial data based on measurement-derived data is susceptible to uncertainty. Therefore, the widespread awareness of data uncertainty is of vital importance to all GIS users. Cartographic visualisation techniques are an effective approach to informing spatial dataset users about the uncertainty of the data. The objective of the research was to define a set of methods for visualising the DTO data uncertainty using expert know-how and experience. This set contains visualisation techniques for presenting three types of uncertainty: positional, attribute, and temporal. The positional uncertainty for point objects was presented using visual variables, object fill with hue colour and lightness, and glyphs placed at map symbol positions. The positional uncertainty for linear objects was presented using linear object contours made of dotted lines and glyphs at vertices. Fill grain density and contour crispness were employed to represent the positional uncertainty for surface objects. The attribute value uncertainty and the temporal uncertainty were represented using fill grain density and fill colour value. The proposed set of the DTO uncertainty visualisation methods provides a finite array of visualisation techniques that can be tested and juxtaposed. The visualisation methods were comprehensively evaluated in a survey among experts who use spatial databases. Results of user preference analysis have demonstrated that the set of the DTO data uncertainty visualisation techniques may be applied to the full extent. The future implementation of the proposed visualisation methods in GIS databases will help data users interpret values correctly.

Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks

Despite the state-of-the-art performance for medical image segmentation, deep convolutional neural networks (CNNs) have rarely provided uncertainty estimations regarding their segmentation outputs, e.g., model (epistemic) and image-based (aleatoric) uncertainties. In this work, we analyze these different types of uncertainties for CNN-based 2D and 3D medical image segmentation tasks at both pixel level and structure level. We additionally propose a test-time augmentation-based aleatoric uncertainty to analyze the effect of different transformations of the input image on the segmentation output. Test-time augmentation has been previously used to improve segmentation accuracy, yet not been formulated in a consistent mathematical framework. Hence, we also propose a theoretical formulation of test-time augmentation, where a distribution of the prediction is estimated by Monte Carlo simulation with prior distributions of parameters in an image acquisition model that involves image transformations and noise. We compare and combine our proposed aleatoric uncertainty with model uncertainty. Experiments with segmentation of fetal brains and brain tumors from 2D and 3D Magnetic Resonance Images (MRI) showed that 1) the test-time augmentation-based aleatoric uncertainty provides a better uncertainty estimation than calculating the test-time dropout-based model uncertainty alone and helps to reduce overconfident incorrect predictions, and 2) our test-time augmentation outperforms a single-prediction baseline and dropout-based multiple predictions.

Multimodal Imaging of Patients With Gliomas Confirms 11C-MET PET as a Complementary Marker to MRI for Noninvasive Tumor Grading and Intraindividual Follow-Up After Therapy

The value of combined L-( methyl-[¹¹C]) methionine positron-emitting tomography (MET-PET) and magnetic resonance imaging (MRI) with regard to tumor extent, entity prediction, and therapy effects in clinical routine in patients with suspicion of a brain tumor was investigated. In n = 65 patients with histologically verified brain lesions n = 70 MET-PET and MRI (T1-weighted gadolinium-enhanced [T1w-Gd] and fluid-attenuated inversion recovery or T2-weighted [FLAIR/T2w]) examinations were performed. The computer software "visualization and analysis framework volume rendering engine (Voreen)" was used for analysis of extent and intersection of tumor compartments. Binary logistic regression models were developed to differentiate between World Health Organization (WHO) tumor types/grades. Tumor sizes as defined by thresholding based on tumor-to-background ratios were significantly different as determined by MET-PET (21.6 ± 36.8 cm³), T1w-Gd-MRI (3.9 ± 7.8 cm³), and FLAIR/T2-MRI (64.8 ± 60.4 cm³; P < .001). The MET-PET visualized tumor activity where MRI parameters were negative: PET positive tumor volume without Gd enhancement was 19.8 ± 35.0 cm³ and without changes in FLAIR/T2 10.3 ± 25.7 cm³. FLAIR/T2-MRI visualized greatest tumor extent with differences to MET-PET being greater in posttherapy (64.6 ± 62.7 cm³) than in newly diagnosed patients (20.5 ± 52.6 cm³). The binary logistic regression model differentiated between WHO tumor types (fibrillary astrocytoma II n = 10 from other gliomas n = 16) with an accuracy of 80.8% in patients at primary diagnosis. Combined PET and MRI improve the evaluation of tumor activity, extent, type/grade prediction, and therapy-induced changes in patients with glioma and serve information highly relevant for diagnosis and management.

TopoAngler: Interactive Topology-Based Extraction of Fishes

We present TopoAngler, a visualization framework that enables an interactive user-guided segmentation of fishes contained in a micro-CT scan. The inherent noise in the CT scan coupled with the often disconnected (and sometimes broken) skeletal structure of fishes makes an automatic segmentation of the volume impractical. To overcome this, our framework combines techniques from computational topology with an interactive visual interface, enabling the human-in-the-Ioop to effectively extract fishes from the volume. In the first step, the join tree of the input is used to create a hierarchical segmentation of the volume. Through the use of linked views, the visual interface then allows users to interactively explore this hierarchy, and gather parts of individual fishes into a coherent sub-volume, thus reconstructing entire fishes. Our framework was primarily developed for its application to CT scans of fishes, generated as part of the ScanAllFish project, through close collaboration with their lead scientist. However, we expect it to also be applicable in other biological applications where a single dataset contains multiple specimen; a common routine that is now widely followed in laboratories to increase throughput of expensive CT scanners.

VIPAR, a quantitative approach to 3D histopathology applied to lymphatic malformations

BACKGROUND

Lack of investigatory and diagnostic tools has been a major contributing factor to the failure to mechanistically understand lymphedema and other lymphatic disorders in order to develop effective drug and surgical therapies. One difficulty has been understanding the true changes in lymph vessel pathology from standard 2D tissue sections.

METHODS

VIPAR (volume information-based histopathological analysis by 3D reconstruction and data extraction), a light-sheet microscopy-based approach for the analysis of tissue biopsies, is based on digital reconstruction and visualization of microscopic image stacks. VIPAR allows semiautomated segmentation of the vasculature and subsequent nonbiased extraction of characteristic vessel shape and connectivity parameters. We applied VIPAR to analyze biopsies from healthy lymphedematous and lymphangiomatous skin.

RESULTS

Digital 3D reconstruction provided a directly visually interpretable, comprehensive representation of the lymphatic and blood vessels in the analyzed tissue volumes. The most conspicuous features were disrupted lymphatic vessels in lymphedematous skin and a hyperplasia (4.36-fold lymphatic vessel volume increase) in the lymphangiomatous skin. Both abnormalities were detected by the connectivity analysis based on extracted vessel shape and structure data. The quantitative evaluation of extracted data revealed a significant reduction of lymphatic segment length (51.3% and 54.2%) and straightness (89.2% and 83.7%) for lymphedematous and lymphangiomatous skin, respectively. Blood vessel length was significantly increased in the lymphangiomatous sample (239.3%).

CONCLUSION

VIPAR is a volume-based tissue reconstruction data extraction and analysis approach that successfully distinguished healthy from lymphedematous and lymphangiomatous skin. Its application is not limited to the vascular systems or skin.

FUNDING

Max Planck Society, DFG (SFB 656), and Cells-in-Motion Cluster of Excellence EXC 1003.

Comparative Local Quality Assessment of 3D Medical Image Segmentations with Focus on Statistical Shape Model-Based Algorithms

The quality of automatic 3D medical segmentation algorithms needs to be assessed on test datasets comprising several 3D images (i.e., instances of an organ). The experts need to compare the segmentation quality across the dataset in order to detect systematic segmentation problems. However, such comparative evaluation is not supported well by current methods. We present a novel system for assessing and comparing segmentation quality in a dataset with multiple 3D images. The data is analyzed and visualized in several views. We detect and show regions with systematic segmentation quality characteristics. For this purpose, we extended a hierarchical clustering algorithm with a connectivity criterion. We combine quality values across the dataset for determining regions with characteristic segmentation quality across instances. Using our system, the experts can also identify 3D segmentations with extraordinary quality characteristics. While we focus on algorithms based on statistical shape models, our approach can also be applied to cases, where landmark correspondences among instances can be established. We applied our approach to three real datasets: liver, cochlea and facial nerve. The segmentation experts were able to identify organ regions with systematic segmentation characteristics as well as to detect outlier instances.

FISICO: Fast Image SegmentatIon COrrection

Background and Purpose

In clinical diagnosis, medical image segmentation plays a key role in the analysis of pathological regions. Despite advances in automatic and semi-automatic segmentation techniques, time-effective correction tools are commonly needed to improve segmentation results. Therefore, these tools must provide faster corrections with a lower number of interactions, and a user-independent solution to reduce the time frame between image acquisition and diagnosis.

Methods

We present a new interactive method for correcting image segmentations. Our method provides 3D shape corrections through 2D interactions. This approach enables an intuitive and natural corrections of 3D segmentation results. The developed method has been implemented into a software tool and has been evaluated for the task of lumbar muscle and knee joint segmentations from MR images.

Results

Experimental results show that full segmentation corrections could be performed within an average correction time of 5.5±3.3 minutes and an average of 56.5±33.1 user interactions, while maintaining the quality of the final segmentation result within an average Dice coefficient of 0.92±0.02 for both anatomies. In addition, for users with different levels of expertise, our method yields a correction time and number of interaction decrease from 38±19.2 minutes to 6.4±4.3 minutes, and 339±157.1 to 67.7±39.6 interactions, respectively.

Characterizing and visualizing predictive uncertainty in numerical ensembles through Bayesian model averaging

Numerical ensemble forecasting is a powerful tool that drives many risk analysis efforts and decision making tasks. These ensembles are composed of individual simulations that each uniquely model a possible outcome for a common event of interest: e.g., the direction and force of a hurricane, or the path of travel and mortality rate of a pandemic. This paper presents a new visual strategy to help quantify and characterize a numerical ensemble's predictive uncertainty: i.e., the ability for ensemble constituents to accurately and consistently predict an event of interest based on ground truth observations. Our strategy employs a Bayesian framework to first construct a statistical aggregate from the ensemble. We extend the information obtained from the aggregate with a visualization strategy that characterizes predictive uncertainty at two levels: at a global level, which assesses the ensemble as a whole, as well as a local level, which examines each of the ensemble's constituents. Through this approach, modelers are able to better assess the predictive strengths and weaknesses of the ensemble as a whole, as well as individual models. We apply our method to two datasets to demonstrate its broad applicability.

Simulation based planning of surgical interventions in pediatric cardiology

Hemodynamics plays an essential role in the progression and treatment of cardiovascular disease. However, while medical imaging provides increasingly detailed anatomical information, clinicians often have limited access to hemodynamic data that may be crucial to patient risk assessment and treatment planning. Computational simulations can now provide detailed hemodynamic data to augment clinical knowledge in both adult and pediatric applications. There is a particular need for simulation tools in pediatric cardiology, due to the wide variation in anatomy and physiology in congenital heart disease patients, necessitating individualized treatment plans. Despite great strides in medical imaging, enabling extraction of flow information from magnetic resonance and ultrasound imaging, simulations offer predictive capabilities that imaging alone cannot provide. Patient specific simulations can be used for in silico testing of new surgical designs, treatment planning, device testing, and patient risk stratification. Furthermore, simulations can be performed at no direct risk to the patient. In this paper, we outline the current state of the art in methods for cardiovascular blood flow simulation and virtual surgery. We then step through pressing challenges in the field, including multiscale modeling, boundary condition selection, optimization, and uncertainty quantification. Finally, we summarize simulation results of two representative examples from pediatric cardiology: single ventricle physiology, and coronary aneurysms caused by Kawasaki disease. These examples illustrate the potential impact of computational modeling tools in the clinical setting.

Hierarchical Exploration of Volumes Using Multilevel Segmentation of the Intensity-Gradient Histograms

Visual exploration of volumetric datasets to discover the embedded features and spatial structures is a challenging and tedious task. In this paper we present a semi-automatic approach to this problem that works by visually segmenting the intensity-gradient 2D histogram of a volumetric dataset into an exploration hierarchy. Our approach mimics user exploration behavior by analyzing the histogram with the normalized-cut multilevel segmentation technique. Unlike previous work in this area, our technique segments the histogram into a reasonable set of intuitive components that are mutually exclusive and collectively exhaustive. We use information-theoretic measures of the volumetric data segments to guide the exploration. This provides a data-driven coarse-to-fine hierarchy for a user to interactively navigate the volume in a meaningful manner.