When Gaussian Process Meets Big Data: A Review of Scalable GPs

Fast and accurate Bayesian optimization with pre-trained transformers for constrained engineering problems

Bayesian Optimization (BO) is a foundational strategy in engineering design optimization for efficiently handling black-box functions with many constraints and expensive evaluations. This paper introduces a novel constraint-handling framework for Bayesian Optimization (BO) using Prior-data Fitted Networks (PFNs), a foundation transformer model. Unlike traditional approaches requiring separate Gaussian Process (GP) models for each constraint, our framework leverages PFN's transformer architecture to evaluate objectives and constraints simultaneously in a single forward pass using in-context learning. Through comprehensive benchmarking across 15 test problems spanning synthetic, structural, and engineering design challenges, we demonstrate an order of magnitude speedup while maintaining or improving solution quality compared to conventional GP-based methods with constrained expected improvement (CEI). Our approach particularly excels at engineering problems by rapidly finding feasible, optimal solutions. This benchmark framework for evaluating new BO algorithms in engineering design will be published at https://github.com/rosenyu304/BOEngineeringBenchmark.

High entropy alloy property predictions using a transformer-based language model

This study introduces a language transformer-based machine learning model to predict key mechanical properties of high-entropy alloys (HEAs), addressing the challenges due to their complex, multi-principal element compositions and limited experimental data. By pre-training the transformer on extensive synthetic materials data and fine-tuning it with specific HEA datasets, the model effectively captures intricate elemental interactions through self-attention mechanisms. This approach mitigates data scarcity issues via transfer learning, enhancing predictive accuracy for properties like elongation (%) and ultimate tensile strength compared to traditional regression models such as random forests and Gaussian processes. The model's interpretability is enhanced by visualizing attention weights, revealing significant elemental relationships that align with known metallurgical principles. This work demonstrates the potential of transformer models to accelerate materials discovery and optimization, enabling accurate property predictions, thereby advancing the field of materials informatics. To fully realize the model's potential in practical applications, future studies should incorporate more advanced preprocessing methods, realistic constraints during synthetic dataset generation, and more refined tokenization techniques.

On the Robustness of Bayesian Neural Networks to Adversarial Attacks

Vulnerability to adversarial attacks is one of the principal hurdles to the adoption of deep learning in safety-critical applications. Despite significant efforts, both practical and theoretical, training deep learning models robust to adversarial attacks is still an open problem. In this article, we analyse the geometry of adversarial attacks in the over-parameterized limit for Bayesian neural networks (BNNs). We show that, in the limit, vulnerability to gradient-based attacks arises as a result of degeneracy in the data distribution, i.e., when the data lie on a lower dimensional submanifold of the ambient space. As a direct consequence, we demonstrate that in this limit, BNN posteriors are robust to gradient-based adversarial attacks. Crucially, by relying on the convergence of infinitely-wide BNNs to Gaussian processes (GPs), we prove that, under certain relatively mild assumptions, the expected gradient of the loss with respect to the BNN posterior distribution is vanishing, even when each NN sampled from the BNN posterior does not have vanishing gradients. The experimental results on the MNIST, Fashion MNIST, and a synthetic dataset with BNNs trained with Hamiltonian Monte Carlo and variational inference support this line of arguments, empirically showing that BNNs can display both high accuracy on clean data and robustness to both gradient-based and gradient-free adversarial attacks.

Neural Operator Variational Inference Based on Regularized Stein Discrepancy for Deep Gaussian Processes

Deep Gaussian process (DGP) models offer a powerful nonparametric approach for Bayesian inference, but exact inference is typically intractable, motivating the use of various approximations. However, existing approaches, such as mean-field Gaussian assumptions, limit the expressiveness and efficacy of DGP models, while stochastic approximation can be computationally expensive. To tackle these challenges, we introduce neural operator variational inference (NOVI) for DGPs. NOVI uses a neural generator to obtain a sampler and minimizes the regularized Stein discrepancy (RSD) between the generated distribution and true posterior in $\mathcal {L}_{2}$ space. We solve the minimax problem using Monte Carlo estimation and subsampling stochastic optimization techniques and demonstrate that the bias introduced by our method can be controlled by multiplying the Fisher divergence with a constant, which leads to robust error control and ensures the stability and precision of the algorithm. Our experiments on datasets ranging from hundreds to millions demonstrate the effectiveness and the faster convergence rate of the proposed method. We achieve a classification accuracy of 93.56 on the CIFAR10 dataset, outperforming state-of-the-art (SOTA) Gaussian process (GP) methods. We are optimistic that NOVI possesses the potential to enhance the performance of deep Bayesian nonparametric models and could have significant implications for various practical applications.

Stagewise Training With Exponentially Growing Training Sets

In the world of big data, training large-scale machine learning problems has gained considerable attention. Numerous innovative optimization strategies have been presented in recent years to accelerate the large-scale training process. However, the possibility of further accelerating the training process of various optimization algorithms remains an unresolved subject. To begin addressing this difficult problem, we exploit the researched findings that when training data are independent and identically distributed, the learning problem on a smaller dataset is not significantly different from the original one. Upon that, we propose a stagewise training technique that grows the size of the training set exponentially while solving nonsmooth subproblem. We demonstrate that our stagewise training via exponentially growing the size of the training sets (STEGSs) are compatible with a large number of proximal gradient descent and gradient hard thresholding (GHT) techniques. Interestingly, we demonstrate that STEGS can greatly reduce overall complexity while maintaining statistical accuracy or even surpassing the intrinsic error introduced by GHT approaches. In addition, we analyze the effect of the training data growth rate on the overall complexity. The practical results of applying $l_{2,1}$ - and $l_{0}$ -norms to a variety of large-scale real-world datasets not only corroborate our theories but also demonstrate the benefits of our STEGS framework.

Multilabel Classification for Entry-Dependent Expert Selection in Distributed Gaussian Processes

By distributing the training process, local approximation reduces the cost of the standard Gaussian process. An ensemble method aggregates predictions from local Gaussian experts, each trained on different data partitions, under the assumption of perfect diversity among them. While this assumption ensures tractable aggregation, it is frequently violated in practice. Although ensemble methods provide consistent results by modeling dependencies among experts, they incur a high computational cost, scaling cubically with the number of experts. Implementing an expert-selection strategy reduces the number of experts involved in the final aggregation step, thereby improving efficiency. However, selection approaches that assign a fixed set of experts to each data point cannot account for the unique properties of individual data points. This paper introduces a flexible expert-selection approach tailored to the characteristics of individual data points. To achieve this, we frame the selection task as a multi-label classification problem in which experts define the labels, and each data point is associated with specific experts. We discuss in detail the prediction quality, efficiency, and asymptotic properties of the proposed solution. We demonstrate the efficiency of the proposed method through extensive numerical experiments on synthetic and real-world datasets. This strategy is easily extendable to distributed learning scenarios and multi-agent models, regardless of Gaussian assumptions regarding the experts.

Bayesian Optimization over Multiple Experimental Fidelities Accelerates Automated Discovery of Drug Molecules

Different experiments of differing fidelities are commonly used in the search for new drug molecules. In classic experimental funnels, libraries of molecules undergo sequential rounds of virtual, coarse, and refined experimental screenings, with each level balanced between the cost of experiments and the number of molecules screened. Bayesian optimization offers an alternative approach, using iterative experiments to locate optimal molecules with fewer experiments than large-scale screening, but without the ability to weigh the costs and benefits of different types of experiments. In this work, we combine the multifidelity approach of the experimental funnel with Bayesian optimization to search for drug molecules iteratively, taking full advantage of different types of experiments, their costs, and the quality of the data they produce. We first demonstrate the utility of the multifidelity Bayesian optimization (MF-BO) approach on a series of drug targets with data reported in ChEMBL, emphasizing what properties of the chemical search space result in substantial acceleration with MF-BO. Then we integrate the MF-BO experiment selection algorithm into an autonomous molecular discovery platform to illustrate the prospective search for new histone deacetylase inhibitors using docking scores, single-point percent inhibitions, and dose-response IC50 values as low-, medium-, and high-fidelity experiments. A chemical search space with appropriate diversity and fidelity correlation for use with MF-BO was constructed with a genetic generative algorithm. The MF-BO integrated platform then docked more than 3,500 molecules, automatically synthesized and screened more than 120 molecules for percent inhibition, and selected a handful of molecules for manual evaluation at the highest fidelity. Many of the molecules screened have never been reported in any capacity. At the end of the search, several new histone deacetylase inhibitors were found with submicromolar inhibition, free of problematic hydroxamate moieties that constrain the use of current inhibitors.

Fault detection for Li-ion batteries of electric vehicles with segmented regression method

Electric vehicles are increasingly popular for their environmental benefits and cost savings, but the reliability and safety of their lithium-ion batteries are critical concerns. Current regression methods for battery fault detection often analyze charging and discharging as a single continuous process, missing important phase differences. This paper proposes segmented regression to better capture these distinct characteristics for accurate fault detection. The focus is on detecting voltage deviations caused by internal short circuits, external short circuits, and capacity degradation, which are primary indicators of battery faults. Firstly, data from real electric vehicles, operating under normal and faulty conditions, is collected over a period of 18 months. Secondly, the segmented regression method is utilized to segment the data based on the charging and discharging cycles and capture potential dependencies in battery behavior within each cycle. Thirdly, an optimized gated recurrent unit network is developed and integrated with the segmented regression to enable accurate cell voltage estimation. Lastly, an adaptive threshold algorithm is proposed to integrate driving behavior and environmental factors into a Gaussian process regression model. The integrated model dynamically estimates the normal fluctuation range of battery cell voltages for fault detection. The effectiveness of the proposed method is validated on a comprehensive dataset, achieving superior accuracy with values of 99.803% and 99.507% during the charging and discharging phases, respectively.

Efficient large-scale genomic prediction in approximate genome-based kernel model

KEY MESSAGE

Three computationally efficient algorithms of GP including RHBK, RHDK, and RHPK were developed in approximate genome-based kernel model. The drastically growing amount of genomic information contributes to increasing computational burden of genomic prediction (GP). In this study, we developed three computationally efficient algorithms of GP including RHBK, RHDK, and RHPK in approximate genome-based kernel model, which reduces dimension of genomic data via Nyström approximation and decreases the computational cost significantly thereby. According to the simulation study and real datasets, our three methods demonstrated predictive accuracy similar to or better than RHAPY, GBLUP, and rrBLUP in most cases. They also demonstrated a substantial reduction in computational time compared to GBLUP and rrBLUP in simulation. Due to their advanced computing efficiency, our three methods can be used in a wide range of application scenarios in the future.

Bayesian Regression Analysis for Dependent Data with an Elliptical Shape

This paper proposes a parametric hierarchical model for functional data with an elliptical shape, using a Gaussian process prior to capturing the data dependencies that reflect systematic errors while modeling the underlying curved shape through a von Mises-Fisher distribution. The model definition, Bayesian inference, and MCMC algorithm are discussed. The effectiveness of the model is demonstrated through the reconstruction of curved trajectories using both simulated and real-world examples. The discussion in this paper focuses on two-dimensional problems, but the framework can be extended to higher-dimensional spaces, making it adaptable to a wide range of applications.

GTR: An SQL Generator With Transition Representation in Cross-Domain Database Systems

Recent studies have focused on using natural language (NL) to automatically retrieve useful data from database (DB) systems. As an important component of autonomous DB systems, the NL-to-SQL technique can assist DB administrators in writing high-quality SQL statements and make persons with no SQL background knowledge learn complex SQL languages. However, existing studies cannot deal with the issue that the expression of NL inevitably mismatches the implementation details of SQLs, and the large number of out-of-domain (OOD) words makes it difficult to predict table columns. In particular, it is difficult to accurately convert NL into SQL in an end-to-end fashion. Intuitively, it facilitates the model to understand the relations if a "bridge" [transition representation (TR)] is employed to make it compatible with both NL and SQL in the phase of conversion. In this article, we propose an automatic SQL generator with TR called GTR in cross-domain DB systems. Specifically, GTR contains three SQL generation steps: 1) GTR learns the relation between questions and DB schemas; 2) GTR uses a grammar-based model to synthesize a TR; and 3) GTR predicts SQL from TR based on the rules. We conduct extensive experiments on two commonly used datasets, that is, WikiSQL and Spider. On the testing set of the Spider and WikiSQL datasets, the results show that GTR achieves 58.32% and 71.29% exact matching accuracy which outperforms the state-of-the-art methods, respectively.

Towards planetary boundary sustainability of food processing wastewater, by resource recovery & emission reduction: A process system engineering perspective

Meeting the needs of a growing population calls for a change from linear production systems that exacerbate the depletion of finite natural resources and the emission of environmental pollutants. These linear production systems have resulted in the human-driven perturbation of the Earth's natural biogeochemical cycles and the transgression of environmentally safe operating limits. One solution that can help alleviate the environmental issues associated both with resource stress and harmful emissions is resource recovery from waste. In this review, we address the recovery of resources from food and beverage processing wastewater (FPWW), which offers a synergistic solution to some of the environmental issues with traditional food production. Research on resource recovery from FPWW typically focuses on technologies to recover specific resources without considering integrative process systems to recover multiple resources while simultaneously satisfying regulations on final effluent quality. Process Systems Engineering (PSE) offers methodologies able to address this holistic process design problem, including modelling the trade-offs between competing objectives. Optimisation of FPWW treatment and resource recovery has significant scope to reduce the environmental impacts of food production systems. There is significant potential to recover carbon, nitrogen, and phosphorus resources while respecting effluent quality limits, even when the significant uncertainties inherent to wastewater systems are considered. This review article gives an overview of the environmental challenges we face, discussed within the framework of the planetary boundary, and highlights the impacts caused by the agri-food sector. This paper also presents a comprehensive review of the characteristics of FPWW and available technologies to recover carbon and nutrient resources from wastewater streams with a particular focus on bioprocesses. PSE research and modelling advances are discussed in this review. Based on this discussion, we conclude the article with future research directions.

Partial Multiview Representation Learning With Cross-View Generation

Multiview learning has made significant progress in recent years. However, an implicit assumption is that multiview data are complete, which is often contrary to practical applications. Due to human or data acquisition equipment errors, what we actually get is partial multiview data, which existing multiview algorithms are limited to processing. Modeling complex dependencies between views in terms of consistency and complementarity remains challenging, especially in partial multiview data scenarios. To address the above issues, this article proposes a deep Gaussian cross-view generation model (named PMvCG), which aims to model views according to the principles of consistency and complementarity and eventually learn the comprehensive representation of partial multiview data. PMvCG can discover cross-view associations by learning view-sharing and view-specific features of different views in the representation space. The missing views can be reconstructed and are applied in turn to further optimize the model. The estimated uncertainty in the model is also considered and integrated into the representation to improve the performance. We design a variational inference and iterative optimization algorithm to solve PMvCG effectively. We conduct comprehensive experiments on multiple real-world datasets to validate the performance of PMvCG. We compare the PMvCG with various methods by applying the learned representation to clustering and classification. We also provide more insightful analysis to explore the PMvCG, such as convergence analysis, parameter sensitivity analysis, and the effect of uncertainty in the representation. The experimental results indicate that PMvCG obtains promising results and surpasses other comparative methods under different experimental settings.

Task decomposition strategy based on machine learning for boosting performance and identifying mechanisms in heterogeneous activation of peracetic acid process

Heterogeneous activation of peracetic acid (PAA) process is a promising method for removing organic pollutants from water. Nevertheless, this process is constrained by several complex factors, such as the selection of catalysts, optimization of reaction conditions, and identification of mechanism. In this study, a task decomposition strategy was adopted by combining a catalyst and reaction condition optimization machine learning (CRCO-ML) model and a mechanism identification machine learning (MI-ML) model to address these issues. The Categorical Boosting (CatBoost) model was identified as the best-performing model for the dataset (1024 sets and 7122 data points) in this study, achieving an R2 of 0.92 and an RMSE of 1.28. Catalyst composition, PAA dosage, and catalyst dosage were identified as the three most important features through SHAP analysis in the CRCO-ML model. The HCO3- is considered the most influential water matrix affecting the k value. The errors between all reverse experiment results and the predictions of the CRCO-ML and MI-ML models were <10 % and 15 %, respectively. This interdisciplinary work provides novel insights into the design and application of the heterogeneous activation of PAA process, significantly contributing to the rapid development of this technology.

Forward variable selection enables fast and accurate dynamic system identification with Karhunen-Loève decomposed Gaussian processes

A promising approach for scalable Gaussian processes (GPs) is the Karhunen-Loève (KL) decomposition, in which the GP kernel is represented by a set of basis functions which are the eigenfunctions of the kernel operator. Such decomposed kernels have the potential to be very fast, and do not depend on the selection of a reduced set of inducing points. However KL decompositions lead to high dimensionality, and variable selection thus becomes paramount. This paper reports a new method of forward variable selection, enabled by the ordered nature of the basis functions in the KL expansion of the Bayesian Smoothing Spline ANOVA kernel (BSS-ANOVA), coupled with fast Gibbs sampling in a fully Bayesian approach. It quickly and effectively limits the number of terms, yielding a method with competitive accuracies, training and inference times for tabular datasets of low feature set dimensionality. Theoretical computational complexities are [Formula: see text] in training and [Formula: see text] per point in inference, where N is the number of instances and P the number of expansion terms. The inference speed and accuracy makes the method especially useful for dynamic systems identification, by modeling the dynamics in the tangent space as a static problem, then integrating the learned dynamics using a high-order scheme. The methods are demonstrated on two dynamic datasets: a 'Susceptible, Infected, Recovered' (SIR) toy problem, along with the experimental 'Cascaded Tanks' benchmark dataset. Comparisons on the static prediction of time derivatives are made with a random forest (RF), a residual neural network (ResNet), and the Orthogonal Additive Kernel (OAK) inducing points scalable GP, while for the timeseries prediction comparisons are made with LSTM and GRU recurrent neural networks (RNNs) along with the SINDy package.

Probabilistic Attention Based on Gaussian Processes for Deep Multiple Instance Learning

Multiple instance learning (MIL) is a weakly supervised learning paradigm that is becoming increasingly popular because it requires less labeling effort than fully supervised methods. This is especially interesting for areas where the creation of large annotated datasets remains challenging, as in medicine. Although recent deep learning MIL approaches have obtained state-of-the-art results, they are fully deterministic and do not provide uncertainty estimations for the predictions. In this work, we introduce the attention Gaussian process (AGP) model, a novel probabilistic attention mechanism based on Gaussian processes (GPs) for deep MIL. AGP provides accurate bag-level predictions as well as instance-level explainability and can be trained end-to-end. Moreover, its probabilistic nature guarantees robustness to overfit on small datasets and uncertainty estimations for the predictions. The latter is especially important in medical applications, where decisions have a direct impact on the patient's health. The proposed model is validated experimentally as follows. First, its behavior is illustrated in two synthetic MIL experiments based on the well-known MNIST and CIFAR-10 datasets, respectively. Then, it is evaluated in three different real-world cancer detection experiments. AGP outperforms state-of-the-art MIL approaches, including deterministic deep learning ones. It shows a strong performance even on a small dataset with less than 100 labels and generalizes better than competing methods on an external test set. Moreover, we experimentally show that predictive uncertainty correlates with the risk of wrong predictions, and therefore it is a good indicator of reliability in practice. Our code is publicly available.

Machine learning on longitudinal multi-modal data enables the understanding and prognosis of Alzheimer's disease progression

Alzheimer's disease (AD) is a complex pathophysiological disease. Allowing for heterogeneity, not only in disease manifestations but also in different progression patterns, is critical for developing effective disease models that can be used in clinical and research settings. We introduce a machine learning model for identifying underlying patterns in Alzheimer's disease (AD) trajectory using longitudinal multi-modal data from the ADNI cohort and the AIBL cohort. Ten biologically and clinically meaningful disease-related states were identified from data, which constitute three non-overlapping stages (i.e., neocortical atrophy [NCA], medial temporal atrophy [MTA], and whole brain atrophy [WBA]) and two distinct disease progression patterns (i.e., NCA → WBA and MTA → WBA). The index of disease-related states provided a remarkable performance in predicting the time to conversion to AD dementia (C-Index: 0.923 ± 0.007). Our model shows potential for promoting the understanding of heterogeneous disease progression and early predicting the conversion time to AD dementia.

Multitask methods for predicting molecular properties from heterogeneous data

Data generation remains a bottleneck in training surrogate models to predict molecular properties. We demonstrate that multitask Gaussian process regression overcomes this limitation by leveraging both expensive and cheap data sources. In particular, we consider training sets constructed from coupled-cluster (CC) and density functional theory (DFT) data. We report that multitask surrogates can predict at CC-level accuracy with a reduction in data generation cost by over an order of magnitude. Of note, our approach allows the training set to include DFT data generated by a heterogeneous mix of exchange-correlation functionals without imposing any artificial hierarchy on functional accuracy. More generally, the multitask framework can accommodate a wider range of training set structures-including the full disparity between the different levels of fidelity-than existing kernel approaches based on Δ-learning although we show that the accuracy of the two approaches can be similar. Consequently, multitask regression can be a tool for reducing data generation costs even further by opportunistically exploiting existing data sources.

AI-Based Metamaterial Design

The use of metamaterials in various devices has revolutionized applications in optics, healthcare, acoustics, and power systems. Advancements in these fields demand novel or superior metamaterials that can demonstrate targeted control of electromagnetic, mechanical, and thermal properties of matter. Traditional design systems and methods often require manual manipulations which is time-consuming and resource intensive. The integration of artificial intelligence (AI) in optimizing metamaterial design can be employed to explore variant disciplines and address bottlenecks in design. AI-based metamaterial design can also enable the development of novel metamaterials by optimizing design parameters that cannot be achieved using traditional methods. The application of AI can be leveraged to accelerate the analysis of vast data sets as well as to better utilize limited data sets via generative models. This review covers the transformative impact of AI and AI-based metamaterial design for optics, acoustics, healthcare, and power systems. The current challenges, emerging fields, future directions, and bottlenecks within each domain are discussed.

Asynchronous Parallel Large-Scale Gaussian Process Regression

Gaussian process regression (GPR) is an important nonparametric learning method in machine learning research with many real-world applications. It is well known that training large-scale GPR is a challenging task due to the required heavy computational cost and large volume memory. To address this challenging problem, in this article, we propose an asynchronous doubly stochastic gradient algorithm to handle the large-scale training of GPR. We formulate the GPR to a convex optimization problem, i.e., kernel ridge regression. After that, in order to efficiently solve this convex kernel problem, we first use the random feature mapping method to approximate the kernel model and then utilize two unbiased stochastic approximations, i.e., stochastic variance reduced gradient and stochastic coordinate descent, to update the solution asynchronously and in parallel. In this way, our algorithm scales well in both sample size and dimensionality, and speeds up the training computation. More importantly, we prove that our algorithm has a global linear convergence rate. Our experimental results on eight large-scale benchmark datasets with both regression and classification tasks show that the proposed algorithm outperforms the existing state-of-the-art GPR methods.

Discrimination of Genetic Biomarkers of Disease through Machine-Learning-Based Hypothesis Testing of Direct SERS Spectra of DNA and RNA

Cancer is globally a leading cause of death that would benefit from diagnostic approaches detecting it in its early stages. However, despite much research and investment, cancer early diagnosis is still underdeveloped. Owing to its high sensitivity, surface-enhanced Raman spectroscopy (SERS)-based detection of biomarkers has attracted growing interest in this area. Oligonucleotides are an important type of genetic biomarkers as their alterations can be linked to the disease prior to symptom onset. We propose a machine-learning (ML)-enabled framework to analyze complex direct SERS spectra of short, single-stranded DNA and RNA targets to identify relevant mutations occurring in genetic biomarkers, which are key disease indicators. First, by employing ad hoc-synthesized colloidal silver nanoparticles as SERS substrates, we analyze single-base mutations in ssDNA and RNA sequences using a direct SERS-sensing approach. Then, an ML-based hypothesis test is proposed to identify these changes and differentiate the mutated sequences from the corresponding native ones. Rooted in "functional data analysis," this ML approach fully leverages the rich information and dependencies within SERS spectral data for improved modeling and detection capability. Tested on a large set of DNA and RNA SERS data, including from miR-21 (a known cancer miRNA biomarker), our approach is shown to accurately differentiate SERS spectra obtained from different oligonucleotides, outperforming various data-driven methods across several performance metrics, including accuracy, sensitivity, specificity, and F1-scores. Hence, this work represents a step forward in the development of the combined use of SERS and ML as effective methods for disease diagnosis with real applicability in the clinic.

Estimating plastic waste generation using supervised time-series learning techniques in Johannesburg, South Africa

In recent times, many investigators have delved into plastic waste (PW) research, both locally and internationally. Many of these studies have focused on problems related to land-based and marine-based PW management with its attendant impact on public health and the ecosystem. Hitherto, there have been little or no studies on forecasting PW quantities in developing countries (DCs). The key objective of this study is to provide a forecast on PW generation in the city of Johannesburg (CoJ), South Africa over the next three decades. The data used for the forecasting were historical data obtained from Statistics South Africa (StatsSA). For effective prediction and comparison, three-time series models were employed in this study. They include exponential smoothing (ETS), Artificial Neural Network (ANN), and the Gaussian Process Regression (GPR). The exponential kernel GPR model performed best on the overall plastic prediction with a determination coefficient (R2) of 0.96, however, on individual PW estimation, ANN was better with an overall R2 of 0.93. From the result, it is predicted that between 2021 and 2050, the total PW generated in CoJ is forecasted to be around 6.7 megatonnes with an average of 0.22 megatonnes/year. In addition, the estimated plastic composition is 17,910 tonnes PS per year; 13,433 tonnes PP per year; 59,440 tonnes HDPE per year; 4478 tonnes PVC per year; 85,074 tonnes PET per year; 34,590 tonnes LDPE per year and 8955 tonnes other PWs per year.

Data-Driven Design for Metamaterials and Multiscale Systems: A Review

Metamaterials are artificial materials designed to exhibit effective material parameters that go beyond those found in nature. Composed of unit cells with rich designability that are assembled into multiscale systems, they hold great promise for realizing next-generation devices with exceptional, often exotic, functionalities. However, the vast design space and intricate structure-property relationships pose significant challenges in their design. A compelling paradigm that could bring the full potential of metamaterials to fruition is emerging: data-driven design. This review provides a holistic overview of this rapidly evolving field, emphasizing the general methodology instead of specific domains and deployment contexts. Existing research is organized into data-driven modules, encompassing data acquisition, machine learning-based unit cell design, and data-driven multiscale optimization. The approaches are further categorized within each module based on shared principles, analyze and compare strengths and applicability, explore connections between different modules, and identify open research questions and opportunities.

Accurate COP Trajectory Estimation in Healthy and Pathological Gait Using Multimodal Instrumented Insoles and Deep Learning Models

Measuring center-of-pressure (COP) trajectories in out-of-the-lab environments may provide valuable information about changes in gait and balance function related to natural disease progression or treatment in neurological disorders. Traditional equipment to acquire COP trajectories includes stationary force plates, instrumented treadmills, electronic walkways, and insoles featuring high-density force sensing arrays, all of which are expensive and not widely accessible. This study introduces novel deep recurrent neural networks that can accurately estimate dynamic COP trajectories by fusing data from affordable and heterogeneous insole-embedded sensors (namely, an eight-cell array of force sensitive resistors (FSRs) and an inertial measurement unit (IMU)). The method was validated against gold-standard equipment during out-of-the-lab ambulatory tasks that simulated real-world walking. Root-mean-square errors (RMSE) in the mediolateral (ML) and anteroposterior (AP) directions obtained from healthy individuals (ML: 0.51 cm, AP: 1.44 cm) and individuals with neuromuscular conditions (ML: 0.59 cm, AP: 1.53 cm) indicated technical validity. In individuals with neuromuscular conditions, COP-derived metrics showed significant correlations with validated clinical measures of ambulatory function and lower-extremity muscle strength, providing proof-of-concept evidence of the convergent validity of the proposed method for clinical applications.

Multi-View Deep Gaussian Processes for Supervised Learning

Multi-view learning is a widely studied topic in machine learning, which considers learning with multiple views of samples to improve the prediction performance. Even though some approaches have sprung up recently, it is still challenging to jointly explore information contained in different views. Multi-view deep Gaussian processes have shown strong advantages in unsupervised representation learning. However, they are limited when dealing with labeled multi-view data for supervised learning, and ignore the application potential of uncertainty estimation. In this paper, we propose a supervised multi-view deep Gaussian process model (named SupMvDGP), which uses the label of the views to further improve the performance, and takes the quantitative uncertainty estimation as a supplement to assist humans to make better use of prediction. According to the diversity of views, the SupMvDGP can establish asymmetric depth structure to better model different views, so as to make full use of the property of each view. We provide a variational inference method to effectively solve the complex model. Finally, we conduct comprehensive comparative experiments on multiple real world datasets to evaluate the performance of SupMvDGP. The experimental results show that the SupMvDGP achieves the state-of-the-art results in multiple tasks, which verifies the effectiveness and superiority of the proposed approach. Meanwhile, we provide a case study to show that the SupMvDGP has the ability to provide uncertainty estimation than alternative deep models, which can alert people to better treat the prediction results in high-risk applications.

Variational log-Gaussian point-process methods for grid cells

We present practical solutions to applying Gaussian-process (GP) methods to calculate spatial statistics for grid cells in large environments. GPs are a data efficient approach to inferring neural tuning as a function of time, space, and other variables. We discuss how to design appropriate kernels for grid cells, and show that a variational Bayesian approach to log-Gaussian Poisson models can be calculated quickly. This class of models has closed-form expressions for the evidence lower-bound, and can be estimated rapidly for certain parameterizations of the posterior covariance. We provide an implementation that operates in a low-rank spatial frequency subspace for further acceleration, and demonstrate these methods on experimental data.

Developmental changes in exploration resemble stochastic optimization

Human development is often described as a 'cooling off' process, analogous to stochastic optimization algorithms that implement a gradual reduction in randomness over time. Yet there is ambiguity in how to interpret this analogy, due to a lack of concrete empirical comparisons. Using data from n = 281 participants ages 5 to 55, we show that cooling off does not only apply to the single dimension of randomness. Rather, human development resembles an optimization process of multiple learning parameters, for example, reward generalization, uncertainty-directed exploration and random temperature. Rapid changes in parameters occur during childhood, but these changes plateau and converge to efficient values in adulthood. We show that while the developmental trajectory of human parameters is strikingly similar to several stochastic optimization algorithms, there are important differences in convergence. None of the optimization algorithms tested were able to discover reliably better regions of the strategy space than adult participants on this task.

Image Matting With Deep Gaussian Process

We observe a common characteristic between the classical propagation-based image matting and the Gaussian process (GP)-based regression. The former produces closer alpha matte values for pixels associated with a higher affinity, while the outputs regressed by the latter are more correlated for more similar inputs. Based on this observation, we reformulate image matting as GP and find that this novel matting-GP formulation results in a set of attractive properties. First, it offers an alternative view on and approach to propagation-based image matting. Second, an application of kernel learning in GP brings in a novel deep matting-GP technique, which is pretty powerful for encapsulating the expressive power of deep architecture on the image relative to its matting. Third, an existing scalable GP technique can be incorporated to further reduce the computational complexity to O(n) from O(n3) of many conventional matting propagation techniques. Our deep matting-GP provides an attractive strategy toward addressing the limit of widespread adoption of deep learning techniques to image matting for which a sufficiently large labeled dataset is lacking. A set of experiments on both synthetically composited images and real-world images show the superiority of the deep matting-GP to not only the classical propagation-based matting techniques but also modern deep learning-based approaches.

An improved rhythmicity analysis method using Gaussian Processes detects cell-density dependent circadian oscillations in stem cells

MOTIVATION

Detecting oscillations in time series remains a challenging problem even after decades of research. In chronobiology, rhythms (for instance in gene expression, eclosion, egg-laying, and feeding) tend to be low amplitude, display large variations amongst replicates, and often exhibit varying peak-to-peak distances (non-stationarity). Most currently available rhythm detection methods are not specifically designed to handle such datasets, and are also limited by their use of P-values in detecting oscillations.

RESULTS

We introduce a new method, ODeGP (Oscillation Detection using Gaussian Processes), which combines Gaussian Process regression and Bayesian inference to incorporate measurement errors, non-uniformly sampled data, and a recently developed non-stationary kernel to improve detection of oscillations. By using Bayes factors, ODeGP models both the null (non-rhythmic) and the alternative (rhythmic) hypotheses, thus providing an advantage over P-values. Using synthetic datasets, we first demonstrate that ODeGP almost always outperforms eight commonly used methods in detecting stationary as well as non-stationary symmetric oscillations. Next, by analyzing existing qPCR datasets, we demonstrate that our method is more sensitive compared to the existing methods at detecting weak and noisy oscillations. Finally, we generate new qPCR data on mouse embryonic stem cells. Surprisingly, we discover using ODeGP that increasing cell-density results in rapid generation of oscillations in the Bmal1 gene, thus highlighting our method's ability to discover unexpected and new patterns. In its current implementation, ODeGP is meant only for analyzing single or a few time-trajectories, not genome-wide datasets.

AVAILABILITY AND IMPLEMENTATION

ODeGP is available at https://github.com/Shaonlab/ODeGP.

MVIRA: A model based on Missing Value Imputation and Reliability Assessment for mortality risk prediction

BACKGROUND

Mortality risk prediction is to predict whether a patient has the risk of death based on relevant diagnosis and treatment data. How to accurately predict patient mortality risk based on electronic health records (EHR) is currently a hot research topic in the healthcare field. In actual medical datasets, there are often many missing values, which can seriously interfere with the effect of model prediction. However, when missing values are interpolated, most existing methods do not take into account the fidelity or confidence of the interpolated values. Misestimation of missing variables can lead to modeling difficulties and performance degradation, while the reliability of the model may be compromised in clinical environments.

MATERIALS AND METHODS

We propose a model based on Missing Value Imputation and Reliability Assessment for mortality risk prediction (MVIRA). The model uses a combination of variational autoencoder and recurrent neural networks to complete the interpolation of missing values and enhance the characterization ability of EHR data, thus improving the performance of mortality risk prediction. In addition, we also introduce the Monte Carlo Dropout method to calculate the uncertainty of the model prediction results and thus achieve the reliability assessment of the model.

RESULTS

We perform performance validation of the model on the public datasets MIMIC-III and MIMIC-IV. The proposed model showed improved performance compared with competitive models in terms of overall specialties.

CONCLUSION

The proposed model can effectively improve the accuracy of mortality risk prediction, and can help medical institutions assess the condition of patients.

Demonstration of an AI-driven workflow for autonomous high-resolution scanning microscopy

Modern scanning microscopes can image materials with up to sub-atomic spatial and sub-picosecond time resolutions, but these capabilities come with large volumes of data, which can be difficult to store and analyze. We report the Fast Autonomous Scanning Toolkit (FAST) that addresses this challenge by combining a neural network, route optimization, and efficient hardware controls to enable a self-driving experiment that actively identifies and measures a sparse but representative data subset in lieu of the full dataset. FAST requires no prior information about the sample, is computationally efficient, and uses generic hardware controls with minimal experiment-specific wrapping. We test FAST in simulations and a dark-field X-ray microscopy experiment of a WSe2 film. Our studies show that a FAST scan of <25% is sufficient to accurately image and analyze the sample. FAST is easy to adapt for any scanning microscope; its broad adoption will empower general multi-level studies of materials evolution with respect to time, temperature, or other parameters.

Fast Bayesian inference for large occupancy datasets

In recent years, the study of species' occurrence has benefited from the increased availability of large-scale citizen-science data. While abundance data from standardized monitoring schemes are biased toward well-studied taxa and locations, opportunistic data are available for many taxonomic groups, from a large number of locations and across long timescales. Hence, these data provide opportunities to measure species' changes in occurrence, particularly through the use of occupancy models, which account for imperfect detection. These opportunistic datasets can be substantially large, numbering hundreds of thousands of sites, and hence present a challenge from a computational perspective, especially within a Bayesian framework. In this paper, we develop a unifying framework for Bayesian inference in occupancy models that account for both spatial and temporal autocorrelation. We make use of the Pólya-Gamma scheme, which allows for fast inference, and incorporate spatio-temporal random effects using Gaussian processes (GPs), for which we consider two efficient approximations: subset of regressors and nearest neighbor GPs. We apply our model to data on two UK butterfly species, one common and widespread and one rare, using records from the Butterflies for the New Millennium database, producing occupancy indices spanning 45 years. Our framework can be applied to a wide range of taxa, providing measures of variation in species' occurrence, which are used to assess biodiversity change.

Recent Advances in 2D Material Theory, Synthesis, Properties, and Applications

Two-dimensional (2D) material research is rapidly evolving to broaden the spectrum of emergent 2D systems. Here, we review recent advances in the theory, synthesis, characterization, device, and quantum physics of 2D materials and their heterostructures. First, we shed insight into modeling of defects and intercalants, focusing on their formation pathways and strategic functionalities. We also review machine learning for synthesis and sensing applications of 2D materials. In addition, we highlight important development in the synthesis, processing, and characterization of various 2D materials (e.g., MXnenes, magnetic compounds, epitaxial layers, low-symmetry crystals, etc.) and discuss oxidation and strain gradient engineering in 2D materials. Next, we discuss the optical and phonon properties of 2D materials controlled by material inhomogeneity and give examples of multidimensional imaging and biosensing equipped with machine learning analysis based on 2D platforms. We then provide updates on mix-dimensional heterostructures using 2D building blocks for next-generation logic/memory devices and the quantum anomalous Hall devices of high-quality magnetic topological insulators, followed by advances in small twist-angle homojunctions and their exciting quantum transport. Finally, we provide the perspectives and future work on several topics mentioned in this review.

Pareto optimization with small data by learning across common objective spaces

In multi-objective optimization, it becomes prohibitively difficult to cover the Pareto front (PF) as the number of points scales exponentially with the dimensionality of the objective space. The challenge is exacerbated in expensive optimization domains where evaluation data is at a premium. To overcome insufficient representations of PFs, Pareto estimation (PE) invokes inverse machine learning to map preferred but unexplored regions along the front to the Pareto set in decision space. However, the accuracy of the inverse model depends on the training data, which is inherently scarce/small given high-dimensional/expensive objectives. To alleviate this small data challenge, this paper marks a first study on multi-source inverse transfer learning for PE. A method to maximally utilize experiential source tasks to augment PE in the target optimization task is proposed. Information transfers between heterogeneous source-target pairs is uniquely enabled in the inverse setting through the unification provided by common objective spaces. Our approach is tested experimentally on benchmark functions as well as on high-fidelity, multidisciplinary simulation data of composite materials manufacturing processes, revealing significant gains to the predictive accuracy and PF approximation capacity of Pareto set learning. With such accurate inverse models made feasible, a future of on-demand human-machine interaction facilitating multi-objective decisions is envisioned.

Reconstructing Kernel-Based Machine Learning Force Fields with Superlinear Convergence

Kernel machines have sustained continuous progress in the field of quantum chemistry. In particular, they have proven to be successful in the low-data regime of force field reconstruction. This is because many equivariances and invariances due to physical symmetries can be incorporated into the kernel function to compensate for much larger data sets. So far, the scalability of kernel machines has however been hindered by its quadratic memory and cubical runtime complexity in the number of training points. While it is known that iterative Krylov subspace solvers can overcome these burdens, their convergence crucially relies on effective preconditioners, which are elusive in practice. Effective preconditioners need to partially presolve the learning problem in a computationally cheap and numerically robust manner. Here, we consider the broad class of Nyström-type methods to construct preconditioners based on successively more sophisticated low-rank approximations of the original kernel matrix, each of which provides a different set of computational trade-offs. All considered methods aim to identify a representative subset of inducing (kernel) columns to approximate the dominant kernel spectrum.

The Impact of Supervised Learning Methods in Ultralarge High-Throughput Docking

Structure-based virtual screening methods are, nowadays, one of the key pillars of computational drug discovery. In recent years, a series of studies have reported docking-based virtual screening campaigns of large databases ranging from hundreds to thousands of millions compounds, further identifying novel hits after experimental validation. As these larg-scale efforts are not generally accessible, machine learning-based protocols have emerged to accelerate the identification of virtual hits within an ultralarge chemical space, reaching impressive reductions in computational time. Herein, we illustrate the motivation and the problem behind the screening of large databases, providing an overview of key concepts and essential applications of machine learning-accelerated protocols, specifically concerning supervised learning methods. We also discuss where the field stands with these novel developments, highlighting possible insights for future studies.

On Nonstationary Gaussian Process Model for Solving Data-Driven Optimization Problems

In data-driven evolutionary optimization, most existing Gaussian processes (GPs)-assisted evolutionary algorithms (EAs) adopt stationary GPs (SGPs) as surrogate models, which might be insufficient for solving most optimization problems. This article finds that GPs in the optimization problems are nonstationary with great probability. We propose to employ a nonstationary GP (NSGP) surrogate model for data-driven evolutionary optimization, where the mean of the NSGP is allowed to vary with the decision variables, while its residue variance follows an SGP. In this article, the nonstationarity of GPs in the tested functions is theoretically analyzed. In addition, this article constructs an NSGP where the SGP is a degenerate case. Performance comparisons of the NSGP with the SGP and the NSGP-assisted EA (NSGP-MAEA) with the SGP-assisted EA (SGP-MAEA) are carried out on a set of benchmark problems and an antenna design problem. These comparison results demonstrate the competitiveness of the NSGP model.

Optimizing autoinjector devices using physics-based simulations and Gaussian processes

Autoinjectors are becoming a primary drug delivery option to the subcutaneous space. These devices need to work robustly and autonomously to maximize drug bio-availability. However, current designs ignore the coupling between autoinjector dynamics and tissue biomechanics. Here we present a Bayesian framework for optimization of autoinjector devices that can account for the coupled autoinjector-tissue biomechanics and uncertainty in tissue mechanical behavior. The framework relies on replacing the high fidelity model of tissue insertion with a Gaussian process (GP). The GP model is accurate yet computationally affordable, enabling a thorough sensitivity analysis that identified tissue properties, which are not part of the autoinjector design space, as important variables for the injection process. Higher fracture toughness decreases the crack depth, while tissue shear modulus has the opposite effect. The sensitivity analysis also shows that drug viscosity and spring force, which are part of the design space, affect the location and timing of drug delivery. Low viscosity could lead to premature delivery, but can be prevented with smaller spring forces, while higher viscosity could prevent premature delivery while demanding larger spring forces and increasing the time of injection. Increasing the spring force guarantees penetration to the desired depth, but it can result in undesirably high accelerations. The Bayesian optimization framework tackles the challenge of designing devices with performance metrics coupled to uncertain tissue properties. This work is important for the design of other medical devices for which optimization in the presence of material behavior uncertainty is needed.

Big problems in spatio-temporal disease mapping: Methods and software

BACKGROUND AND OBJECTIVE

Fitting spatio-temporal models for areal data is crucial in many fields such as cancer epidemiology. However, when data sets are very large, many issues arise. The main objective of this paper is to propose a general procedure to analyze high-dimensional spatio-temporal areal data, with special emphasis on mortality/incidence relative risk estimation.

METHODS

We present a pragmatic and simple idea that permits hierarchical spatio-temporal models to be fitted when the number of small areas is very large. Model fitting is carried out using integrated nested Laplace approximations over a partition of the spatial domain. We also use parallel and distributed strategies to speed up computations in a setting where Bayesian model fitting is generally prohibitively time-consuming or even unfeasible.

RESULTS

Using simulated and real data, we show that our method outperforms classical global models. We implement the methods and algorithms that we develop in the open-source R package bigDM where specific vignettes have been included to facilitate the use of the methodology for non-expert users.

CONCLUSIONS

Our scalable methodology proposal provides reliable risk estimates when fitting Bayesian hierarchical spatio-temporal models for high-dimensional data.

CT-based deep learning model for the prediction of DNA mismatch repair deficient colorectal cancer: a diagnostic study

BACKGROUND

Stratification of DNA mismatch repair (MMR) status in patients with colorectal cancer (CRC) enables individual clinical treatment decision making. The present study aimed to develop and validate a deep learning (DL) model based on the pre-treatment CT images for predicting MMR status in CRC.

METHODS

1812 eligible participants (training cohort: n = 1124; internal validation cohort: n = 482; external validation cohort: n = 206) with CRC were enrolled from two institutions. All pretherapeutic CT images from three dimensions were trained by the ResNet101, then integrated by Gaussian process regression (GPR) to develop a full-automatic DL model for MMR status prediction. The predictive performance of the DL model was evaluated using the area under the receiver operating characteristic curve (AUC) and then tested in the internal and external validation cohorts. Additionally, the participants from institution 1 were sub-grouped by various clinical factors for subgroup analysis, then the predictive performance of the DL model for identifying MMR status between participants in different groups were compared.

RESULTS

The full-automatic DL model was established in the training cohort to stratify the MMR status, which presented promising discriminative ability with the AUCs of 0.986 (95% CI 0.971-1.000) in the internal validation cohort and 0.915 (95% CI 0.870-0.960) in the external validation cohort. In addition, the subgroup analysis based on the thickness of CT images, clinical T and N stages, gender, the longest diameter, and the location of tumors revealed that the DL model showed similar satisfying prediction performance.

CONCLUSIONS

The DL model may potentially serve as a noninvasive tool to facilitate the pre-treatment individualized prediction of MMR status in patients with CRC, which could promote the personalized clinical-making decision.

Exact Gaussian processes for massive datasets via non-stationary sparsity-discovering kernels

A Gaussian Process (GP) is a prominent mathematical framework for stochastic function approximation in science and engineering applications. Its success is largely attributed to the GP's analytical tractability, robustness, and natural inclusion of uncertainty quantification. Unfortunately, the use of exact GPs is prohibitively expensive for large datasets due to their unfavorable numerical complexity of [Formula: see text] in computation and [Formula: see text] in storage. All existing methods addressing this issue utilize some form of approximation-usually considering subsets of the full dataset or finding representative pseudo-points that render the covariance matrix well-structured and sparse. These approximate methods can lead to inaccuracies in function approximations and often limit the user's flexibility in designing expressive kernels. Instead of inducing sparsity via data-point geometry and structure, we propose to take advantage of naturally-occurring sparsity by allowing the kernel to discover-instead of induce-sparse structure. The premise of this paper is that the data sets and physical processes modeled by GPs often exhibit natural or implicit sparsities, but commonly-used kernels do not allow us to exploit such sparsity. The core concept of exact, and at the same time sparse GPs relies on kernel definitions that provide enough flexibility to learn and encode not only non-zero but also zero covariances. This principle of ultra-flexible, compactly-supported, and non-stationary kernels, combined with HPC and constrained optimization, lets us scale exact GPs well beyond 5 million data points.

Global sensitivity analysis based on Gaussian-process metamodelling for complex biomechanical problems

Biomechanical models often need to describe very complex systems, organs or diseases, and hence also include a large number of parameters. One of the attractive features of physics-based models is that in those models (most) parameters have a clear physical meaning. Nevertheless, the determination of these parameters is often very elaborate and costly and shows a large scatter within the population. Hence, it is essential to identify the most important parameters (worth the effort) for a particular problem at hand. In order to distinguish parameters which have a significant influence on a specific model output from non-influential parameters, we use sensitivity analysis, in particular the Sobol method as a global variance-based method. However, the Sobol method requires a large number of model evaluations, which is prohibitive for computationally expensive models. We therefore employ Gaussian processes as a metamodel for the underlying full model. Metamodelling introduces further uncertainty, which we also quantify. We demonstrate the approach by applying it to two different problems: nanoparticle-mediated drug delivery in a complex, multiphase tumour-growth model, and arterial growth and remodelling. Even relatively small numbers of evaluations of the full model suffice to identify the influential parameters in both cases and to separate them from non-influential parameters. The approach also allows the quantification of higher-order interaction effects. We thus show that a variance-based global sensitivity analysis is feasible for complex, computationally expensive biomechanical models. Different aspects of sensitivity analysis are covered including a transparent declaration of the uncertainties involved in the estimation process. Such a global sensitivity analysis not only helps to massively reduce costs for experimental determination of parameters but is also highly beneficial for inverse analysis of such complex models.

Active Learning for Efficient Soil Monitoring in Large Terrain with Heterogeneous Sensor Network

Soils are a complex ecosystem that provides critical services, such as growing food, supplying antibiotics, filtering wastes, and maintaining biodiversity; hence monitoring soil health and domestication is required for sustainable human development. Low-cost and high-resolution soil monitoring systems are challenging to design and build. Compounded by the sheer size of the monitoring area of interest and the variety of biological, chemical, and physical parameters to monitor, naive approaches to adding or scheduling more sensors will suffer from cost and scalability problems. We investigate a multi-robot sensing system integrated with an active learning-based predictive modeling technique. Taking advantage of advances in machine learning, the predictive model allows us to interpolate and predict soil attributes of interest from the data collected by sensors and soil surveys. The system provides high-resolution prediction when the modeling output is calibrated with static land-based sensors. The active learning modeling technique allows our system to be adaptive in data collection strategy for time-varying data fields, utilizing aerial and land robots for new sensor data. We evaluated our approach using numerical experiments with a soil dataset focusing on heavy metal concentration in a flooded area. The experimental results demonstrate that our algorithms can reduce sensor deployment costs via optimized sensing locations and paths while providing high-fidelity data prediction and interpolation. More importantly, the results verify the adapting behavior of the system to the spatial and temporal variations of soil conditions.

Correlated product of experts for sparse Gaussian process regression

Gaussian processes (GPs) are an important tool in machine learning and statistics. However, off-the-shelf GP inference procedures are limited to datasets with several thousand data points because of their cubic computational complexity. For this reason, many sparse GPs techniques have been developed over the past years. In this paper, we focus on GP regression tasks and propose a new approach based on aggregating predictions from several local and correlated experts. Thereby, the degree of correlation between the experts can vary between independent up to fully correlated experts. The individual predictions of the experts are aggregated taking into account their correlation resulting in consistent uncertainty estimates. Our method recovers independent Product of Experts, sparse GP and full GP in the limiting cases. The presented framework can deal with a general kernel function and multiple variables, and has a time and space complexity which is linear in the number of experts and data samples, which makes our approach highly scalable. We demonstrate superior performance, in a time vs. accuracy sense, of our proposed method against state-of-the-art GP approximations for synthetic as well as several real-world datasets with deterministic and stochastic optimization.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10994-022-06297-3.

Surrogated-assisted multimodal multi-objective optimization for hybrid renewable energy system

Hybrid renewable energy system (HRES) is an effective tool to improve the utilization of renewable energy so as to enhance the quality of energy supply. The optimization of HRES includes a simulation process during a long time span, which is time-consuming. So far, introducing a surrogate model to replace the objective evaluation is an effective way to solve such problems. However, existing methods focused few on the diversity of solutions in the decision space. Based on this motivation, we proposed a novel surrogated-assisted multi-objective evolutionary algorithm that focuses on solving multimodal and time-expensive problems, termed SaMMEA. Specifically, we use a Gaussian process model to replace the calculation of the objective values. In addition, a special environmental selection strategy is proposed to enhance the diversity of solutions in the decision space and a model management method is proposed to better train the surrogate model. The proposed algorithm is then compared to several state-of-the-art algorithms on HRES problems, which indicates that the proposed algorithm is competitive.

Optimisation of the core subset for the APY approximation of genomic relationships

BACKGROUND

By entering the era of mega-scale genomics, we are facing many computational issues with standard genomic evaluation models due to their dense data structure and cubic computational complexity. Several scalable approaches have been proposed to address this challenge, such as the Algorithm for Proven and Young (APY). In APY, genotyped animals are partitioned into core and non-core subsets, which induces a sparser inverse of the genomic relationship matrix. This partitioning is often done at random. While APY is a good approximation of the full model, random partitioning can make results unstable, possibly affecting accuracy or even reranking animals. Here we present a stable optimisation of the core subset by choosing animals with the most informative genotype data.

METHODS

We derived a novel algorithm for optimising the core subset based on a conditional genomic relationship matrix or a conditional single nucleotide polymorphism (SNP) genotype matrix. We compared the accuracy of genomic predictions with different core subsets for simulated and real pig data sets. The core subsets were constructed (1) at random, (2) based on the diagonal of the genomic relationship matrix, (3) at random with weights from (2), or (4) based on the novel conditional algorithm. To understand the different core subset constructions, we visualise the population structure of the genotyped animals with linear Principal Component Analysis and non-linear Uniform Manifold Approximation and Projection.

RESULTS

All core subset constructions performed equally well when the number of core animals captured most of the variation in the genomic relationships, both in simulated and real data sets. When the number of core animals was not sufficiently large, there was substantial variability in the results with the random construction but no variability with the conditional construction. Visualisation of the population structure and chosen core animals showed that the conditional construction spreads core animals across the whole domain of genotyped animals in a repeatable manner.

CONCLUSIONS

Our results confirm that the size of the core subset in APY is critical. Furthermore, the results show that the core subset can be optimised with the conditional algorithm that achieves an optimal and repeatable spread of core animals across the domain of genotyped animals.