Estimation of inferential uncertainty in assessing expert segmentation performance from STAPLE

Data-driven modelling for assessing trophic status in marine ecosystems using machine learning approaches

Assessing eutrophication in coastal and transitional waters is of utmost importance, yet existing Trophic Status Index (TSI) models face challenges like multicollinearity, data redundancy, inappropriate aggregation methods, and complex classification schemes. To tackle these issues, we developed a novel tool that harnesses machine learning (ML) and artificial intelligence (AI), enhancing the reliability and accuracy of trophic status assessments. Our research introduces an improved data-driven methodology specifically tailored for transitional and coastal (TrC) waters, with a focus on Cork Harbour, Ireland, as a case study. Our innovative approach, named the Assessment Trophic Status Index (ATSI) model, comprises three main components: the selection of pertinent water quality indicators, the computation of ATSI scores, and the implementation of a new classification scheme. To optimize input data and minimize redundancy, we employed ML techniques, including advanced deep learning methods. Specifically, we developed a CHL prediction model utilizing ten algorithms, among which XGBoost demonstrated exceptional performance, showcasing minimal errors during both training (RMSE = 0.0, MSE = 0.0, MAE = 0.01) and testing (RMSE = 0.0, MSE = 0.0, MAE = 0.01) phases. Utilizing a novel linear rescaling interpolation function, we calculated ATSI scores and evaluated the model's sensitivity and efficiency across diverse application domains, employing metrics such as R2, the Nash-Sutcliffe efficiency (NSE), and the model efficiency factor (MEF). The results consistently revealed heightened sensitivity and efficiency across all application domains. Additionally, we introduced a brand new classification scheme for ranking the trophic status of transitional and coastal waters. To assess spatial sensitivity, we applied the ATSI model to four distinct waterbodies in Ireland, comparing trophic assessment outcomes with the Assessment of Trophic Status of Estuaries and Bays in Ireland (ATSEBI) System. Remarkably, significant disparities between the ATSI and ATSEBI System were evident in all domains, except for Mulroy Bay. Overall, our research significantly enhances the accuracy of trophic status assessments in marine ecosystems. The ATSI model, combined with cutting-edge ML techniques and our new classification scheme, represents a promising avenue for evaluating and monitoring trophic conditions in TrC waters. The study also demonstrated the effectiveness of ATSI in assessing trophic status across various waterbodies, including lakes, rivers, and more. These findings make substantial contributions to the field of marine ecosystem management and conservation.

Two-dimensional segmentation fusion tool: an extensible, free-to-use, user-friendly tool for combining different bidimensional segmentations

Introduction: In several fields, the process of fusing multiple two-dimensional (2D) closed lines is an important step. For instance, this is fundamental in histology and oncology in general. The treatment of a tumor consists of numerous steps and activities. Among them, segmenting the cancer area, that is, the correct identification of its spatial location by the segmentation technique, is one of the most important and at the same time complex and delicate steps. The difficulty in deriving reliable segmentations stems from the lack of a standard for identifying the edges and surrounding tissues of the tumor area. For this reason, the entire process is affected by considerable subjectivity. Given a tumor image, different practitioners can associate different segmentations with it, and the diagnoses produced may differ. Moreover, experimental data show that the analysis of the same area by the same physician at two separate timepoints may result in different lines being produced. Accordingly, it is challenging to establish which contour line is the ground truth. Methods: Starting from multiple segmentations related to the same tumor, statistical metrics and computational procedures could be exploited to combine them for determining the most reliable contour line. In particular, numerous algorithms have been developed over time for this procedure, but none of them is validated yet. Accordingly, in this field, there is no ground truth, and research is still active. Results: In this work, we developed the Two-Dimensional Segmentation Fusion Tool (TDSFT), a user-friendly tool distributed as a free-to-use standalone application for MAC, Linux, and Windows, which offers a simple and extensible interface where numerous algorithms are proposed to "compute the mean" (i.e., the process to fuse, combine, and "average") multiple 2D lines. Conclusions: The TDSFT can support medical specialists, but it can also be used in other fields where it is required to combine 2D close lines. In addition, the TDSFT is designed to be easily extended with new algorithms thanks to a dedicated graphical interface for configuring new parameters. The TDSFT can be downloaded from the following link: https://sourceforge.net/p/tdsft.

A sophisticated model for rating water quality

Here, we present the Irish Water Quality Index (IEWQI) model for assessing transitional and coastal water quality in an effort to improve the method and develop a tool that can be used by environmental regulators to abate water pollution in Ireland. The developed model has been associated with the adoption of water quality standards formulated for coastal and transitional waterbodies according to the water framework directive legislation by the environmental regulator of Irish water. The model consists of five identical components, including (i) indicator selection technique is to select the crucial water quality indicator; (ii) sub-index (SI) function for rescaling various water quality indicators' information into a uniform scale; (iii) indicators' weight method for estimating the weight values based on the relative significance of real-time information on water quality; (iii) aggregation function for computing the water quality index (WQI) score; and (v) score interpretation scheme for assessing the state of water quality. The IEWQI model was developed based on Cork Harbour, Ireland. The developed IEWQI model was applied to four coastal waterbodies in Ireland, for assessing water quality using 2021 water quality data for the summer and winter seasons in order to evaluate model sensitivity in terms of spatio-temporal resolution of various waterbodies. The model efficiency and uncertainty were also analysed in this research. In terms of different spatio-temporal magnitudes of various domains, the model shows higher sensitivity in four application domains during the summer and winter. In addition, the results of uncertainty reveal that the IEWQI model architecture may be effective for reducing model uncertainty in order to avoid model eclipsing and ambiguity problems. The findings of this study reveal that the IEWQI model could be an efficient and reliable technique for the assessment of transitional and coastal water quality more accurately in any geospatial domain.

A four-way model (FEST) for source apportionment: Development, verification, and application

In studying the spatial, temporal, and particle size variations heavy metal sources, a source apportionment model for a four-way array of data is required. In this study, referencing two-way and three-way models, a four-way (particle fractions, elements, sites, and time) source apportionment model (FEST) was developed. Errors in the three-way models solving four-way problems verified the necessity of developing the FEST model. The results showed that the FEST model had a higher accuracy than the existing models, which was probably because of more constraints and input data in the FEST model. Based on the sampled data in Beijing, sources were apportioned for the four-way array of data using the FEST model, and the spatial, temporal, and particle size variations of sources were evaluated. The main sources of heavy metals were similar to those in our prior studies, whereas the contributions of sources to specific heavy metals differed. Traffic exhaust and fuel combustion contributed more to fine particles than coarse particles, indicating that the two should be controlled preferentially among all sources. The management of traffic exhaust should be focused on the central and northern areas in each season, and the control of fuel combustion should be strengthened in the southern area in winter.

The effect of imaging modality (magnetic resonance imaging vs. computed tomography) and patient position (supine vs. prone) on target and organ at risk doses in partial breast irradiation

INTRODUCTION

Conventionally computed tomography (CT) has been used to delineate target volumes in radiotherapy; however, magnetic resonance imaging (MRI) is being continually integrated into clinical practice; therefore, the investigation into targets derived from MRI is warranted. The purpose of this study was to evaluate the impact of imaging modality (MRI vs. CT) and patient positioning (supine vs. prone) on planning target volumes (PTVs) and organs at risk (OARs) for partial breast irradiation (PBI).

METHODS

A retrospective data set, of 35 patients, was accessed where each patient had undergone MRI and CT imaging for tangential whole breast radiotherapy in both the supine and prone position. PTVs were defined from seroma cavity (SC) volumes delineated on each respective image, resulting in 4 PTVs per patient. PBI plans were generated with 6MV external beam radiotherapy (EBRT) using the TROG 06.02 protocol guidelines. A prescription of 38.5Gy in 10 fractions was used for all cases. The impact analysis of imaging modality and patient positioning included dose to PTVs, and OARs based on agreed criteria. Statistical analysis was conducted though Mann-Whitey U, Fisher's exact and chi-squared testing (P < 0.005).

RESULTS

Twenty-four patients were eligible for imaging analysis. However, positioning analysis could only be investigated on 19 of these data sets. No statistically significant difference was found in OAR doses based on imaging modality. Supine patient position resulted in lower contralateral breast dose (0.10Gy ± 0.35 vs. 0.33Gy ± 0.78, p = 0.011). Prone positioning resulted in a lower dose to ipsilateral lung volumes (10.85Gy ± 11.37 vs. 3.41Gy ± 3.93, P = <0.001).

CONCLUSIONS

PBI plans with PTVs derived from MRI exhibited no clinically significant differences when compared to plans created from CT in relation to plan compliance and OAR dose. Patient position requires careful consideration regardless of imaging modality chosen. Although there was no proven superiority of MRI derived target volumes, it indicates that MRI could be considered for PBI target delineation.

Evaluating diffusion-weighted magnetic resonance imaging for target volume delineation in head and neck radiotherapy

INTRODUCTION

Inter-observer variability (IOV) in target volume delineation is a source of error in head and neck radiotherapy. Diffusion-weighted imaging (DWI) has been shown to be useful in detecting recurrent head and neck cancer. This study aims to determine whether DWI improves target volume delineation and IOV.

METHODS

Four radiation oncologists delineated the gross tumour volume (GTV) for ten head and neck cancer patients. Delineation was performed on CT alone as well as fused image sets which incorporated fluorodeoxyglucose (FDG)-positron emission tomography (PET) and magnetic resonance imaging (MRI) in the form of CT/PET, CT/PET/T2W and CT/PET/T2W/DWI image sets. Analysis of the variability of contour volumes was completed by comparison to the simultaneous truth and performance level estimation (STAPLE) volumes. The DICE Similarity Coefficient (DSC) and other IOV metrics for each observer's contour were compared to the STAPLE for each patient and image dataset. A DWI usability scoresheet for delineation was completed.

RESULTS

The CT/PET/T2W/DWI mean GTV volume of 13.37 (10.35-16.39)cm³ was shown to be different to the mean GTV of 10.92 (8.32-13.51)cm³ when using CT alone (P < 0.001). The GTV DSC amongst observers for CT alone was 0.72 (0.65-0.79), CT/PET was 0.73 (0.67-0.80), CT/PET/T2W was 0.71 (0.64-0.77) and CT/PET/T2W/DWI was 0.69 (0.61-0.75).

CONCLUSION

Mean GTVs with the addition of DWI had slightly larger volumes compared to standard CT and CT/PET volumes. DWI may add supplemental visual information for GTV delineation while having a small impact on IOV, therefore potentially improving target volume delineation.

The impact of a radiologist-led workshop on MRI target volume delineation for radiotherapy

INTRODUCTION

Magnetic resonance imaging (MRI) is increasingly used for target volume delineation in radiotherapy due to its superior soft tissue visualisation compared to computed tomography (CT). The aim of this study was to assess the impact of a radiologist-led workshop on inter-observer variability in volume delineation on MRI.

METHODS

Data from three separate studies evaluating the impact of MRI in lung, breast and cervix were collated. At pre-workshop evaluation, observers involved in each clinical site were instructed to delineate specified volumes. Radiologists specialising in each cancer site conducted an interactive workshop on interpretation of images and anatomy for each clinical site. At post-workshop evaluation, observers repeated delineation a minimum of 2 weeks after the workshops. Inter-observer variability was evaluated using dice similarity coefficient (DSC) and volume similarity (VOLSIM) index comparing reference and observer volumes.

RESULTS

Post-workshop primary gross tumour volumes (GTV) were smaller than pre-workshop volumes for lung with a mean percentage reduction of 10.4%. Breast clinical target volumes (CTV) were similar but seroma volumes were smaller post-workshop on both supine (65% reduction) and prone MRI (73% reduction). Based on DSC scores, improvement in inter-observer variability was seen for the seroma cavity volume on prone MRI with a reduction in DSC score range from 0.4-0.8 to 0.7-0.9. Breast CTV demonstrated good inter-observer variability scores (mean DSC 0.9) for both pre- and post-workshop. Post-workshop observer delineated cervix GTV was smaller than pre-workshop by 26.9%.

CONCLUSION

A radiologist-led workshop did not significantly reduce inter-observer variability in volume delineation for the three clinical sites. However, some improvement was noted in delineation of breast CTV, seroma volumes and cervix GTV.

Comparative performance evaluation of automated segmentation methods of hippocampus from magnetic resonance images of temporal lobe epilepsy patients

PURPOSE

Segmentation of the hippocampus from magnetic resonance (MR) images is a key task in the evaluation of mesial temporal lobe epilepsy (mTLE) patients. Several automated algorithms have been proposed although manual segmentation remains the benchmark. Choosing a reliable algorithm is problematic since structural definition pertaining to multiple edges, missing and fuzzy boundaries, and shape changes varies among mTLE subjects. Lack of statistical references and guidance for quantifying the reliability and reproducibility of automated techniques has further detracted from automated approaches. The purpose of this study was to develop a systematic and statistical approach using a large dataset for the evaluation of automated methods and establish a method that would achieve results better approximating those attained by manual tracing in the epileptogenic hippocampus.

METHODS

A template database of 195 (81 males, 114 females; age range 32-67 yr, mean 49.16 yr) MR images of mTLE patients was used in this study. Hippocampal segmentation was accomplished manually and by two well-known tools (FreeSurfer and hammer) and two previously published methods developed at their institution [Automatic brain structure segmentation (ABSS) and LocalInfo]. To establish which method was better performing for mTLE cases, several voxel-based, distance-based, and volume-based performance metrics were considered. Statistical validations of the results using automated techniques were compared with the results of benchmark manual segmentation. Extracted metrics were analyzed to find the method that provided a more similar result relative to the benchmark.

RESULTS

Among the four automated methods, ABSS generated the most accurate results. For this method, the Dice coefficient was 5.13%, 14.10%, and 16.67% higher, Hausdorff was 22.65%, 86.73%, and 69.58% lower, precision was 4.94%, -4.94%, and 12.35% higher, and the root mean square (RMS) was 19.05%, 61.90%, and 65.08% lower than LocalInfo, FreeSurfer, and hammer, respectively. The Bland-Altman similarity analysis revealed a low bias for the ABSS and LocalInfo techniques compared to the others.

CONCLUSIONS

The ABSS method for automated hippocampal segmentation outperformed other methods, best approximating what could be achieved by manual tracing. This study also shows that four categories of input data can cause automated segmentation methods to fail. They include incomplete studies, artifact, low signal-to-noise ratio, and inhomogeneity. Different scanner platforms and pulse sequences were considered as means by which to improve reliability of the automated methods. Other modifications were specially devised to enhance a particular method assessed in this study.

Comparison of Magnetic Resonance Imaging and Computed Tomography for Breast Target Volume Delineation in Prone and Supine Positions

PURPOSE

To determine whether T2-weighted MRI improves seroma cavity (SC) and whole breast (WB) interobserver conformity for radiation therapy purposes, compared with the gold standard of CT, both in the prone and supine positions.

METHODS AND MATERIALS

Eleven observers (2 radiologists and 9 radiation oncologists) delineated SC and WB clinical target volumes (CTVs) on T2-weighted MRI and CT supine and prone scans (4 scans per patient) for 33 patient datasets. Individual observer's volumes were compared using the Dice similarity coefficient, volume overlap index, center of mass shift, and Hausdorff distances. An average cavity visualization score was also determined.

RESULTS

Imaging modality did not affect interobserver variation for WB CTVs. Prone WB CTVs were larger in volume and more conformal than supine CTVs (on both MRI and CT). Seroma cavity volumes were larger on CT than on MRI. Seroma cavity volumes proved to be comparable in interobserver conformity in both modalities (volume overlap index of 0.57 (95% Confidence Interval (CI) 0.54-0.60) for CT supine and 0.52 (95% CI 0.48-0.56) for MRI supine, 0.56 (95% CI 0.53-0.59) for CT prone and 0.55 (95% CI 0.51-0.59) for MRI prone); however, after registering modalities together the intermodality variation (Dice similarity coefficient of 0.41 (95% CI 0.36-0.46) for supine and 0.38 (0.34-0.42) for prone) was larger than the interobserver variability for SC, despite the location typically remaining constant.

CONCLUSIONS

Magnetic resonance imaging interobserver variation was comparable to CT for the WB CTV and SC delineation, in both prone and supine positions. Although the cavity visualization score and interobserver concordance was not significantly higher for MRI than for CT, the SCs were smaller on MRI, potentially owing to clearer SC definition, especially on T2-weighted MR images.

Multivariate Analyses Applied to Healthy Neurodevelopment in Fetal, Neonatal, and Pediatric MRI

Multivariate analysis (MVA) is a class of statistical and pattern recognition techniques that involve the processing of data that contains multiple measurements per sample. MVA can be used to address a wide variety of neurological medical imaging related challenges including the evaluation of healthy brain development, the automated analysis of brain tissues and structures through image segmentation, evaluating the effects of genetic and environmental factors on brain development, evaluating sensory stimulation's relationship with functional brain activity and much more. Compared to adult imaging, pediatric, neonatal and fetal imaging have attracted less attention from MVA researchers, however, recent years have seen remarkable MVA research growth in pre-adult populations. This paper presents the results of a systematic review of the literature focusing on MVA applied to healthy subjects in fetal, neonatal and pediatric magnetic resonance imaging (MRI) of the brain. While the results of this review demonstrate considerable interest from the scientific community in applications of MVA technologies in brain MRI, the field is still young and significant research growth will continue into the future.

Multiatlas segmentation as nonparametric regression

This paper proposes a novel theoretical framework to model and analyze the statistical characteristics of a wide range of segmentation methods that incorporate a database of label maps or atlases; such methods are termed as label fusion or multiatlas segmentation. We model these multiatlas segmentation problems as nonparametric regression problems in the high-dimensional space of image patches. We analyze the nonparametric estimator's convergence behavior that characterizes expected segmentation error as a function of the size of the multiatlas database. We show that this error has an analytic form involving several parameters that are fundamental to the specific segmentation problem (determined by the chosen anatomical structure, imaging modality, registration algorithm, and label-fusion algorithm). We describe how to estimate these parameters and show that several human anatomical structures exhibit the trends modeled analytically. We use these parameter estimates to optimize the regression estimator. We show that the expected error for large database sizes is well predicted by models learned on small databases. Thus, a few expert segmentations can help predict the database sizes required to keep the expected error below a specified tolerance level. Such cost-benefit analysis is crucial for deploying clinical multiatlas segmentation systems.

A collaborative resource to build consensus for automated left ventricular segmentation of cardiac MR images

A collaborative framework was initiated to establish a community resource of ground truth segmentations from cardiac MRI. Multi-site, multi-vendor cardiac MRI datasets comprising 95 patients (73 men, 22 women; mean age 62.73±11.24years) with coronary artery disease and prior myocardial infarction, were randomly selected from data made available by the Cardiac Atlas Project (Fonseca et al., 2011). Three semi- and two fully-automated raters segmented the left ventricular myocardium from short-axis cardiac MR images as part of a challenge introduced at the STACOM 2011 MICCAI workshop (Suinesiaputra et al., 2012). Consensus myocardium images were generated based on the Expectation-Maximization principle implemented by the STAPLE algorithm (Warfield et al., 2004). The mean sensitivity, specificity, positive predictive and negative predictive values ranged between 0.63 and 0.85, 0.60 and 0.98, 0.56 and 0.94, and 0.83 and 0.92, respectively, against the STAPLE consensus. Spatial and temporal agreement varied in different amounts for each rater. STAPLE produced high quality consensus images if the region of interest was limited to the area of discrepancy between raters. To maintain the quality of the consensus, an objective measure based on the candidate automated rater performance distribution is proposed. The consensus segmentation based on a combination of manual and automated raters were more consistent than any particular rater, even those with manual input. The consensus is expected to improve with the addition of new automated contributions. This resource is open for future contributions, and is available as a test bed for the evaluation of new segmentation algorithms, through the Cardiac Atlas Project (www.cardiacatlas.org).

Towards a multi-site international public dataset for the validation of retinal image analysis software

This paper discusses concisely the main issues and challenges posed by the validation of retinal image analysis algorithms. It is designed to set the discussion for the IEEE EBMC 2013 invited session "From laboratory to clinic: the validation of retinal image processing tools ". The session carries forward an international initiative started at EMBC 2011, Boston, which resulted in the first large-consensus paper (14 international sites) on the validation of retinal image processing software, appearing in IOVS. This paper is meant as a focus for the session discussion, but the ubiquity and importance of validation makes its contents, arguably, of interest for the wider medical image processing community.

Groupwise conditional random forests for automatic shape classification and contour quality assessment in radiotherapy planning

Radiation therapy is used to treat cancer patients around the world. High quality treatment plans maximally radiate the targets while minimally radiating healthy organs at risk. In order to judge plan quality and safety, segmentations of the targets and organs at risk are created, and the amount of radiation that will be delivered to each structure is estimated prior to treatment. If the targets or organs at risk are mislabelled, or the segmentations are of poor quality, the safety of the radiation doses will be erroneously reviewed and an unsafe plan could proceed. We propose a technique to automatically label groups of segmentations of different structures from a radiation therapy plan for the joint purposes of providing quality assurance and data mining. Given one or more segmentations and an associated image we seek to assign medically meaningful labels to each segmentation and report the confidence of that label. Our method uses random forests to learn joint distributions over the training features, and then exploits a set of learned potential group configurations to build a conditional random field (CRF) that ensures the assignment of labels is consistent across the group of segmentations. The CRF is then solved via a constrained assignment problem. We validate our method on 1574 plans, consisting of 17[Formula: see text] 579 segmentations, demonstrating an overall classification accuracy of 91.58%. Our results also demonstrate the stability of RF with respect to tree depth and the number of splitting variables in large data sets.

Validating retinal fundus image analysis algorithms: issues and a proposal

This paper concerns the validation of automatic retinal image analysis (ARIA) algorithms. For reasons of space and consistency, we concentrate on the validation of algorithms processing color fundus camera images, currently the largest section of the ARIA literature. We sketch the context (imaging instruments and target tasks) of ARIA validation, summarizing the main image analysis and validation techniques. We then present a list of recommendations focusing on the creation of large repositories of test data created by international consortia, easily accessible via moderated Web sites, including multicenter annotations by multiple experts, specific to clinical tasks, and capable of running submitted software automatically on the data stored, with clear and widely agreed-on performance criteria, to provide a fair comparison.

Estimating a reference standard segmentation with spatially varying performance parameters: local MAP STAPLE

We present a new algorithm, called local MAP STAPLE, to estimate from a set of multi-label segmentations both a reference standard segmentation and spatially varying performance parameters. It is based on a sliding window technique to estimate the segmentation and the segmentation performance parameters for each input segmentation. In order to allow for optimal fusion from the small amount of data in each local region, and to account for the possibility of labels not being observed in a local region of some (or all) input segmentations, we introduce prior probabilities for the local performance parameters through a new maximum a posteriori formulation of STAPLE. Further, we propose an expression to compute confidence intervals in the estimated local performance parameters. We carried out several experiments with local MAP STAPLE to characterize its performance and value for local segmentation evaluation. First, with simulated segmentations with known reference standard segmentation and spatially varying performance, we show that local MAP STAPLE performs better than both STAPLE and majority voting. Then we present evaluations with data sets from clinical applications. These experiments demonstrate that spatial adaptivity in segmentation performance is an important property to capture. We compared the local MAP STAPLE segmentations to STAPLE, and to previously published fusion techniques and demonstrate the superiority of local MAP STAPLE over other state-of-the-art algorithms.

How Many Templates Does It Take for a Good Segmentation?: Error Analysis in Multiatlas Segmentation as a Function of Database Size

This paper proposes a novel formulation to model and analyze the statistical characteristics of some types of segmentation problems that are based on combining label maps / templates / atlases. Such segmentation-by-example approaches are quite powerful on their own for several clinical applications and they provide prior information, through spatial context, when combined with intensity-based segmentation methods. The proposed formulation models a class of multiatlas segmentation problems as nonparametric regression problems in the high-dimensional space of images. The paper presents a systematic analysis of the nonparametric estimation's convergence behavior (i.e. characterizing segmentation error as a function of the size of the multiatlas database) and shows that it has a specific analytic form involving several parameters that are fundamental to the specific segmentation problem (i.e. chosen anatomical structure, imaging modality, registration method, label-fusion algorithm, etc.). We describe how to estimate these parameters and show that several brain anatomical structures exhibit the trends determined analytically. The proposed framework also provides per-voxel confidence measures for the segmentation. We show that the segmentation error for large database sizes can be predicted using small-sized databases. Thus, small databases can be exploited to predict the database sizes required ("how many templates") to achieve "good" segmentations having errors lower than a specified tolerance. Such cost-benefit analysis is crucial for designing and deploying multiatlas segmentation systems.