Model selection on solid ground: Rigorous comparison of nine ways to evaluate Bayesian model evidence

Characterisation of the normal human ganglion cell-inner plexiform layer using widefield optical coherence tomography

PURPOSE

To describe variations in ganglion cell-inner plexiform layer (GCIPL) thickness in a healthy cohort from widefield optical coherence tomography (OCT) scans.

METHODS

Widefield OCT scans spanning 55° × 45° were acquired from 470 healthy eyes. The GCIPL was automatically segmented using deep learning methods. Thickness measurements were extracted after correction for warpage and retinal tilt. Multiple linear regression analysis was applied to discern trends between global GCIPL thickness and age, axial length and sex. To further characterise age-related change, hierarchical and two-step cluster algorithms were applied to identify locations sharing similar ageing properties, and rates of change were quantified using regression analyses with data pooled by cluster analysis outcomes.

RESULTS

Declines in widefield GCIPL thickness with age, increasing axial length and female sex were observed (parameter estimates -0.053, -0.436 and -0.464, p-values <0.001, <0.001 and 0.02, respectively). Cluster analyses revealed concentric, slightly nasally displaced, horseshoe patterns of age-related change in the GCIPL, with up to four statistically distinct clusters outside the macula. Linear regression analyses revealed significant ageing decline in GCIPL thickness across all clusters, with faster rates of change observed at central locations when expressed as absolute (slope = -0.19 centrally vs. -0.04 to -0.12 peripherally) and percentage rates of change (slope = -0.001 centrally vs. -0.0005 peripherally).

CONCLUSIONS

Normative variations in GCIPL thickness from widefield OCT with age, axial length and sex were noted, highlighting factors worth considering in further developments. Widefield OCT has promising potential to facilitate quantitative detection of abnormal GCIPL outside standard fields of view.

Wetland functional assessment and uncertainty analysis using fuzzy α-cut-based modified hydrogeomorphic approach

Wetlands are significant ecosystems which perform several functions such as ground water recharge, flood control, carbon sequestration, and pollution reduction. Accurate evaluation of wetland functions is challenging, due to uncertainty associated with variables such as vegetation, soil, hydrology, land use, and landscape. Uncertainty is due to the factors such as the cost of evaluating quality parameters, measurement, and human errors. This study proposes an innovative framework based on modified hydrogeomorphic approach (HGMA) using fuzzy α-cut technique. HGMA has been used for wetland functional assessment and α-cut technique is used to characterize uncertainty corresponding to the input variables and wetland functions. The most uncertain variables were found to be the density of wetlands and basin count in the landscape assessment area with the scores of 4.38% and 3.614% respectively. Among the functions, the highest uncertainty is found in functional capacity index (FCI) corresponding to water storage (1.697%) and retain particulate (1.577%). The quantified uncertainty can help the practitioners to make informed decisions regarding planning best management practices for preserving and restoring the wetland functionality.

High sampling resolution optical coherence tomography reveals potential concurrent reductions in ganglion cell-inner plexiform and inner nuclear layer thickness but not in outer retinal thickness in glaucoma

PURPOSE

To analyse optical coherence tomography (OCT)-derived inner nuclear layer (INL) and outer retinal complex (ORC) measurements relative to ganglion cell-inner plexiform layer (GCIPL) measurements in glaucoma.

METHODS

Glaucoma participants (n = 271) were categorised by 10-2 visual field defect type. Differences in GCIPL, INL and ORC thickness were calculated between glaucoma and matched healthy eyes (n = 548). Hierarchical cluster algorithms were applied to generate topographic patterns of retinal thickness change, with agreement between layers assessed using Cohen's kappa (κ). Differences in GCIPL, INL and ORC thickness within and outside GCIPL regions showing the greatest reductions and Spearman's correlations between layer pairs were compared with 10-2 mean deviation (MD) and pattern standard deviation (PSD) to determine trends with glaucoma severity.

RESULTS

Glaucoma participants with inferior and superior defects presented with concordant GCIPL and INL defects demonstrating mostly fair-to-moderate agreement (κ = 0.145-0.540), which was not observed in eyes with no or ring defects (κ = -0.067-0.230). Correlations (r) with MD and PSD were moderate and weak in GCIPL and INL thickness differences, respectively (GCIPL vs. MD r = 0.479, GCIPL vs. PSD r = -0.583, INL vs. MD r = 0.259, INL vs. PSD r = -0.187, p = <0.0001-0.002), and weak in GCIPL-INL correlations (MD r = 0.175, p = 0.004 and PSD r = 0.154, p = 0.01). No consistent patterns in ORC thickness or correlations were observed.

CONCLUSIONS

In glaucoma, concordant reductions in macular INL and GCIPL thickness can be observed, but reductions in ORC thickness appear unlikely. These findings suggest that trans-synaptic retrograde degeneration may occur in glaucoma and could indicate the usefulness of INL thickness in evaluating glaucomatous damage.

Bayesian inversion of laboratory experiments of transport through limestone fractures

In this study, a novel experimental setup is proposed for which a column filled with glass beads and parallelepiped-shaped limestone beams is used to reconstruct a multiple fracture limestone media. The proposed setup produces asymmetric breakthrough curves (BTCs) that are consistent with the shape expected from the past field and lab-scale studies. Three transport experiments have been conducted under fast, medium, and slow flow velocity conditions. The research focuses on parameter and state estimation using Bayesian inference via Markov Chain Monte Carlo (MCMC) sampler, investigating the degree to which three models of transport through fractured media can reproduce the experimental results under the three flow conditions. The first transport model, named ADE, is based on the equivalent porous medium (EPM) approach and corresponds to the linear advection dispersion equation (ADE). The second model, named FOMIM (first-order mobile immobile), is based on the mobile/immobile approach and uses the dual porosity model with a linear first-order transfer between mobile and immobile regions. The third model, named NLMIM (non-linear mobile-immobile), uses a nonlinear transfer function between these two regions. The results of the three models show that almost all the unknown model input parameters can be well-estimated with narrow confidence intervals using the MCMC method. With respect to state estimation, the ADE model fails to reproduce correctly the tail of the BTCs observed under slow and medium flow conditions. The FOMIM model improves the tailing of the BTCs, but significant discrepancies remain between simulated and measured concentrations. The NLMIM model with velocity-dependent parameters is the only model that captures BTCs under all three conditions of slow, medium, and fast flow velocities.

A note on tools for prediction under uncertainty and identifiability of SIR-like dynamical systems for epidemiology

We provide an overview of the methods that can be used for prediction under uncertainty and data fitting of dynamical systems, and of the fundamental challenges that arise in this context. The focus is on SIR-like models, that are being commonly used when attempting to predict the trend of the COVID-19 pandemic. In particular, we raise a warning flag about identifiability of the parameters of SIR-like models; often, it might be hard to infer the correct values of the parameters from data, even for very simple models, making it non-trivial to use these models for meaningful predictions. Most of the points that we touch upon are actually generally valid for inverse problems in more general setups.

Gene-Centric Model Approaches for Accurate Prediction of Pesticide Biodegradation in Soils

Pesticides are widely used in agriculture despite their negative impact on ecosystems and human health. Biogeochemical modeling facilitates the mechanistic understanding of microbial controls on pesticide turnover in soils. We propose to inform models of coupled microbial dynamics and pesticide turnover with measurements of the abundance and expression of functional genes. To assess the advantages of informing models with genetic data, we developed a novel "gene-centric" model and compared model variants of differing structural complexity against a standard biomass-based model. The models were calibrated and validated using data from two batch experiments in which the degradation of the pesticides dichlorophenoxyacetic acid (2,4-D) and 2-methyl-4-chlorophenoxyacetic acid (MCPA) were observed in soil. When calibrating against data on pesticide mineralization, the gene-centric and biomass-based models performed equally well. However, accounting for pesticide-triggered gene regulation allows improved performance in capturing microbial dynamics and in predicting pesticide mineralization. This novel modeling approach also reveals a hysteretic relationship between pesticide degradation rates and gene expression, implying that the biodegradation performance in soils cannot be directly assessed by measuring the expression of functional genes. Our gene-centric model provides an effective approach for exploiting molecular biology data to simulate pesticide degradation in soils.

Disentangling model complexity in green roof hydrological analysis: A Bayesian perspective

Green Roofs (GRs) have proven to be a sustainable solution to stormwater management in urban areas. To boost their adoption at the large scale, there is a need to develop numerical models, which are accurate, computationally cheap, and as complex as needed to reproduce the hydrological behavior of GRs. Alternative conceptual and mechanistic approaches have been proposed and tested, however the most appropriate level of model complexity for GRs' analysis is still unknown. To cover this scientific gap, we provide a Bayesian comprehensive perspective of GR hydrological modeling, which includes a statistically rigorous Bayesian comparison of one conceptual and multiple Richards-based mechanistic GR models, and a probabilistic assessment of the information content of different observations. The analysis of the marginal likelihoods reveals that the conceptual and the unimodal van Genuchten - Mualem models are the most appropriate parameterizations, and that further layers of model complexity are not fully supported by the measurements. In addition to that, the estimated Kullback-Leibler divergences suggest that the measured volumetric water content outperforms the measured subsurface outflow and tracer concentrations in terms of informativeness, leading to the lowest model predictive uncertainty for the simulation of water fluxes. The findings of this study represent a first step to clarify the role of model complexity in GRs' analysis, and open new perspective on GRs' model-based experimental design.

Robust Model Selection: Flatness-Based Optimal Experimental Design for a Biocatalytic Reaction

Considering the competitive and strongly regulated pharmaceutical industry, mathematical modeling and process systems engineering might be useful tools for implementing quality by design (QbD) and quality by control (QbC) strategies for low-cost but high-quality drugs. However, a crucial task in modeling (bio)pharmaceutical manufacturing processes is the reliable identification of model candidates from a set of various model hypotheses. To identify the best experimental design suitable for a reliable model selection and system identification is challenging for nonlinear (bio)pharmaceutical process models in general. This paper is the first to exploit differential flatness for model selection problems under uncertainty, and thus translates the model selection problem to advanced concepts of systems theory and controllability aspects, respectively. Here, the optimal controls for improved model selection trajectories are expressed analytically with low computational costs. We further demonstrate the impact of parameter uncertainties on the differential flatness-based method and provide an effective robustification strategy with the point estimate method for uncertainty quantification. In a simulation study, we consider a biocatalytic reaction step simulating the carboligation of aldehydes, where we successfully derive optimal controls for improved model selection trajectories under uncertainty.

Evaluating Subsurface Parameterization to Simulate Hyporheic Exchange: The Steinlach River Test Site

Hyporheic exchange is the interaction of river water and groundwater, and is difficult to predict. One of the largest contributions to predictive uncertainty for hyporheic exchange has been attributed to the representation of heterogeneous subsurface properties. Our study evaluates the trade-offs between intrinsic (irreducible) and epistemic (reducible) model errors when choosing between homogeneous and highly complex subsurface parameter structures. We modeled the Steinlach River Test Site in Southwest Germany using a fully coupled surface water-groundwater model to simulate hyporheic exchange and to assess the predictive errors and uncertainties of transit time distributions. A highly parameterized model was built, treated as a "virtual reality" and used as a reference. We found that if the parameter structure is too simple, it will be limited by intrinsic model errors. By increasing subsurface complexity through the addition of zones or heterogeneity, we can begin to exchange intrinsic for epistemic errors. Thus, the appropriate level of detail to represent the subsurface depends on the acceptable range of intrinsic structural errors for the given modeling objectives and the available site data. We found that a zonated model is capable of reproducing the transit time distributions of a more detailed model, but only if the geological structures are known. An interpolated heterogeneous parameter field (cf. pilot points) showed the best trade-offs between the two errors, indicating fitness for practical applications. Parameter fields generated by multiple-point geostatistics (MPS) produce transit time distributions with the largest uncertainties, however, these are reducible by additional hydrogeological data, particularly flux measurements.

The Connection between Bayesian Inference and Information Theory for Model Selection, Information Gain and Experimental Design

We show a link between Bayesian inference and information theory that is useful for model selection, assessment of information entropy and experimental design. We align Bayesian model evidence (BME) with relative entropy and cross entropy in order to simplify computations using prior-based (Monte Carlo) or posterior-based (Markov chain Monte Carlo) BME estimates. On the one hand, we demonstrate how Bayesian model selection can profit from information theory to estimate BME values via posterior-based techniques. Hence, we use various assumptions including relations to several information criteria. On the other hand, we demonstrate how relative entropy can profit from BME to assess information entropy during Bayesian updating and to assess utility in Bayesian experimental design. Specifically, we emphasize that relative entropy can be computed avoiding unnecessary multidimensional integration from both prior and posterior-based sampling techniques. Prior-based computation does not require any assumptions, however posterior-based estimates require at least one assumption. We illustrate the performance of the discussed estimates of BME, information entropy and experiment utility using a transparent, non-linear example. The multivariate Gaussian posterior estimate includes least assumptions and shows the best performance for BME estimation, information entropy and experiment utility from posterior-based sampling.

Improving the Accuracy of Hydrodynamic Simulations in Data Scarce Environments Using Bayesian Model Averaging: A Case Study of the Inner Niger Delta, Mali, West Africa

In this paper, the study area was the Inner Niger Delta (IND) in Mali, West Africa. The IND is threatened by climate change, increasing irrigation, and dam operations. 2D hydrodynamic modelling was used to simulate water levels, discharge, and inundation extent in the IND. Three different digital elevation models (DEM) (SRTM, MERIT, and a DEM derived from satellite images were used as a source of elevation data. Six different models were created, with different sources of elevation data and different downstream boundary conditions. Given that the performance of the models varies according to the location in the IND, the variable under consideration and the performance criteria, Bayesian Model Averaging (BMA) was used to assess the relative performance of each of the six models. The BMA weights, along with deterministic performance measures, such as the Nash Sutcliffe coefficient (NS) and the Pearson’s correlation coefficient (r), provide quantitative evidence as to which model is the best when simulating a particular hydraulic variable at a particular location. After the models were combined with BMA, both discharge and water levels could be simulated with reasonable precision (NS > 0.8). The results of this work can contribute to the more efficient management of water resources in the IND.

Making Steppingstones out of Stumbling Blocks: A Bayesian Model Evidence Estimator with Application to Groundwater Transport Model Selection

Bayesian model evidence (BME) is a measure of the average fit of a model to observation data given all the parameter values that the model can assume. By accounting for the trade-off between goodness-of-fit and model complexity, BME is used for model selection and model averaging purposes. For strict Bayesian computation, the theoretically unbiased Monte Carlo based numerical estimators are preferred over semi-analytical solutions. This study examines five BME numerical estimators and asks how accurate estimation of the BME is important for penalizing model complexity. The limiting cases for numerical BME estimators are the prior sampling arithmetic mean estimator (AM) and the posterior sampling harmonic mean (HM) estimator, which are straightforward to implement, yet they result in underestimation and overestimation, respectively. We also consider the path sampling methods of thermodynamic integration (TI) and steppingstone sampling (SS) that sample multiple intermediate distributions that link the prior and the posterior. Although TI and SS are theoretically unbiased estimators, they could have a bias in practice arising from numerical implementation. For example, sampling errors of some intermediate distributions can introduce bias. We propose a variant of SS, namely the multiple one-steppingstone sampling (MOSS) that is less sensitive to sampling errors. We evaluate these five estimators using a groundwater transport model selection problem. SS and MOSS give the least biased BME estimation at an efficient computational cost. If the estimated BME has a bias that covariates with the true BME, this would not be a problem because we are interested in BME ratios and not their absolute values. On the contrary, the results show that BME estimation bias can be a function of model complexity. Thus, biased BME estimation results in inaccurate penalization of more complex models, which changes the model ranking. This was less observed with SS and MOSS as with the three other methods.

Model selection criteria for dynamic brain PET studies

Background

 Several criteria exist to identify the optimal model for quantification of tracer kinetics. The purpose of this study was to evaluate the correspondence in kinetic model preference identification for brain PET studies among five model selection criteria: Akaike Information Criterion (AIC), AIC unbiased (AIC_C), model selection criterion (MSC), Schwartz Criterion (SC), and F-test.

Materials and Methods

Six tracers were evaluated: [¹¹C]FMZ, [¹¹C]GMOM, [¹¹C]PK11195, [¹¹C]Raclopride, [¹⁸F]FDG, and [¹¹C]PHT, including data from five subjects per tracer. Time activity curves (TACs) were analysed using six plasma input models: reversible single-tissue model (1T2k), irreversible two-tissue model (2T3k), and reversible two-tissue model (2T4k), all with and without blood volume fraction parameter (V_B). For each tracer and criterion, the percentage of TACs preferring a certain model was calculated.

Results

For all radiotracers, strong agreement was seen across the model selection criteria. The F-test was considered as the reference, as it is a frequently used hypothesis test. The F-test confirmed the AIC preferred model in 87% of all cases. The strongest (but minimal) disagreement across regional TACs was found when comparing AIC with AIC_C. Despite these regional discrepancies, same preferred kinetic model was obtained using all criteria, with an exception of one FMZ subject.

Conclusion

In conclusion, all five model selection criteria resulted in similar conclusions with only minor differences that did not affect overall model selection.

Electronic supplementary material

The online version of this article (10.1186/s40658-017-0197-0) contains supplementary material, which is available to authorized users.