Harnessing multiple models for outbreak management

An analysis of spatial and temporal uncertainty propagation in agent-based models

Spatially explicit simulations of complex systems lead to inherent uncertainties in spatial outcomes. Visualizing the temporal propagation of spatial uncertainties is crucial to communicate the reliability of such models. However, the current Uncertainty Analyses (UAs) either consider spatial uncertainty at the end of model runs, or consider non-spatial uncertainties at different model states. To address this, we propose a Spatio-Temporal UA (ST-UA) approach to generate an uncertainty propagation index and visualize the temporal propagation of different uncertainty measures between two temporal model states. We select the total effects sensitivity measure (a Sobol index) for a sample application within the ST-UA approach. The application is the Tobacco Town ABM, a spatial model simulating smoking behaviours. We showcase the effect of the statistical distributions of wages and smoking rates on the propensity to buy cigarettes, which leads to the propagation of uncertainty in the number of purchased cigarettes by individuals. The findings highlight the usefulness of the ST-UA in (i) communicating the reliability of the spatial outcomes of the model; and (ii) guiding modellers towards the spatial areas with relatively high uncertainties at different temporal steps. This approach can be readily transferred to other application areas that are characterized with spatio-temporal uncertainty.This article is part of the theme issue 'Uncertainty quantification for healthcare and biological systems (Part 2)'.

Collaborative forecasting of influenza-like illness in Italy: The Influcast experience

Collaborative hubs that integrate multiple teams to generate ensemble projections and forecasts for shared targets are now regarded as state-of-the-art in epidemic predictive modeling. In this paper, we introduce Influcast, Italy's first epidemic forecasting hub for influenza-like illness. During the 2023/2024 winter season, Influcast provided 20 rounds of forecasts, involving five teams and eight models to predict influenza-like illness incidence up to four weeks in advance at the national and regional administrative level. The individual forecasts were synthesized into an ensemble and benchmarked against a baseline model. Across all models, the ensemble most frequently ranks among the top performers at the national level considering different metrics and forecasting rounds. Additionally, the ensemble outperforms the baseline and most individual models across all regions. Despite a decline in absolute performance over longer horizons, the ensemble model outperformed the baseline in all considered horizons. These findings show the importance of multimodel forecasting hubs in producing reliable short-term influenza-like illnesses forecasts that can inform public health preparedness and mitigation strategies.

Artificial intelligence for modelling infectious disease epidemics

Infectious disease threats to individual and public health are numerous, varied and frequently unexpected. Artificial intelligence (AI) and related technologies, which are already supporting human decision making in economics, medicine and social science, have the potential to transform the scope and power of infectious disease epidemiology. Here we consider the application to infectious disease modelling of AI systems that combine machine learning, computational statistics, information retrieval and data science. We first outline how recent advances in AI can accelerate breakthroughs in answering key epidemiological questions and we discuss specific AI methods that can be applied to routinely collected infectious disease surveillance data. Second, we elaborate on the social context of AI for infectious disease epidemiology, including issues such as explainability, safety, accountability and ethics. Finally, we summarize some limitations of AI applications in this field and provide recommendations for how infectious disease epidemiology can harness most effectively current and future developments in AI.

Wastewater-based surveillance of SARS-CoV-2: Short-term projection (forecasting), smoothing and outlier identification using Bayesian smoothing

BACKGROUND

Day-to-day variation in the measurement of SARS-CoV-2 in wastewater can challenge public health interpretation. We assessed a Bayesian smoothing and forecasting method previously used for surveillance and short-term projection of COVID-19 cases, hospitalizations, and deaths.

METHODS

SARS-CoV-2 viral measurement from the sewershed in Ottawa, Canada, sampled at the municipal wastewater treatment plant from July 1, 2020, to February 15, 2022, was used to assess and internally validate measurement averaging and prediction. External validation was performed using viral measurement data from influent wastewater samples from 15 wastewater treatment plants and municipalities across Ontario.

RESULTS

Plots of SARS-CoV-2 viral measurement over time using Bayesian smoothing visually represented distinct COVID-19 "waves" described by case and hospitalization data in both initial (Ottawa) and external validation in 15 Ontario communities. The time-varying growth rate of viral measurement in wastewater samples approximated the growth rate observed for cases and hospitalization. One-week predicted viral measurement approximated the observed viral measurement throughout the assessment period from December 23, 2020, to August 8, 2022. An uncalibrated model showed underprediction during rapid increases in viral measurement (positive growth) and overprediction during rapid decreases. After recalibration, the model showed a close approximation between observed and predicted estimates.

CONCLUSION

Bayesian smoothing of wastewater surveillance data of SARS-CoV-2 allows for accurate estimates of COVID-19 growth rates and one- and two-week forecasting of SARS-CoV-2 in wastewater for 16 municipalities in Ontario, Canada. Further assessment is warranted in other communities representing different sewersheds and environmental conditions.

Modular Synthesis of High Performance Energetic Biocidal Bicyclic Compounds Derivated from Iodopyrazoles

Modular synthesis can combine different functional modules to flexibly regulate comprehensive properties and study the diversity of compounds. This study established a modular bicyclic synthesis strategy of combining polynitro energetic module with iodine-containing biocidal module. Compounds 1-6 with high iodine content (48.72-69.56 %) and high thermal stability (Td: 172-304 °C) were synthesized and exhaustively identified. By modular synthesis, the detonation properties and gas-production of 3-6 improved greatly expanding their biocidal efficacy and maintained the iodine atomic utilization of iodine-containing module. Notably, 4,5-diiodo-3,4',5'-trinitro-1,3'-bipyrazole (5) and 3,5-diiodo-4,4',5'-trinitro-1,3'-bipyrazole (6) exhibit high detonation velocities (D: 5903 m s-1, 5769 m s-1, respectively) and highest gas production of 212.85 L mol-1 and 217.66 L mol-1 after decomposition. This study diversifies polyiodio-nitro compounds, and also inspire the implementation of similar synthesis strategies to provide family-level synthetic solutions to energetic biocidal materials.

Canada's provincial COVID-19 pandemic modelling efforts: A review of mathematical models and their impacts on the responses

SETTING

Mathematical modelling played an important role in the public health response to COVID-19 in Canada. Variability in epidemic trajectories, modelling approaches, and data infrastructure across provinces provides a unique opportunity to understand the factors that shaped modelling strategies.

INTERVENTION

Provinces implemented stringent pandemic interventions to mitigate SARS-CoV-2 transmission, considering evidence from epidemic models. This study aimed to summarize provincial COVID-19 modelling efforts. We identified modelling teams working with provincial decision-makers, through referrals and membership in Canadian modelling networks. Information on models, data sources, and knowledge translation were abstracted using standardized instruments.

OUTCOMES

We obtained information from six provinces. For provinces with sustained community transmission, initial modelling efforts focused on projecting epidemic trajectories and healthcare demands, and evaluating impacts of proposed interventions. In provinces with low community transmission, models emphasized quantifying importation risks. Most of the models were compartmental and deterministic, with projection horizons of a few weeks. Models were updated regularly or replaced by new ones, adapting to changing local epidemic dynamics, pathogen characteristics, vaccines, and requests from public health. Surveillance datasets for cases, hospitalizations and deaths, and serological studies were the main data sources for model calibration. Access to data for modelling and the structure for knowledge translation differed markedly between provinces.

IMPLICATION

Provincial modelling efforts during the COVID-19 pandemic were tailored to local contexts and modulated by available resources. Strengthening Canadian modelling capacity, developing and sustaining collaborations between modellers and governments, and ensuring earlier access to linked and timely surveillance data could help improve pandemic preparedness.

Scenario design for infectious disease projections: Integrating concepts from decision analysis and experimental design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, situational awareness, horizon scanning, forecasting, and value of information) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Characterising information gains and losses when collecting multiple epidemic model outputs

BACKGROUND

Collaborative comparisons and combinations of epidemic models are used as policy-relevant evidence during epidemic outbreaks. In the process of collecting multiple model projections, such collaborations may gain or lose relevant information. Typically, modellers contribute a probabilistic summary at each time-step. We compared this to directly collecting simulated trajectories. We aimed to explore information on key epidemic quantities; ensemble uncertainty; and performance against data, investigating potential to continuously gain information from a single cross-sectional collection of model results.

METHODS

We compared projections from the European COVID-19 Scenario Modelling Hub. Five teams modelled incidence in Belgium, the Netherlands, and Spain. We compared July 2022 projections by incidence, peaks, and cumulative totals. We created a probabilistic ensemble drawn from all trajectories, and compared to ensembles from a median across each model's quantiles, or a linear opinion pool. We measured the predictive accuracy of individual trajectories against observations, using this in a weighted ensemble. We repeated this sequentially against increasing weeks of observed data. We evaluated these ensembles to reflect performance with varying observed data.

RESULTS

By collecting modelled trajectories, we showed policy-relevant epidemic characteristics. Trajectories contained a right-skewed distribution well represented by an ensemble of trajectories or a linear opinion pool, but not models' quantile intervals. Ensembles weighted by performance typically retained the range of plausible incidence over time, and in some cases narrowed this by excluding some epidemic shapes.

CONCLUSIONS

We observed several information gains from collecting modelled trajectories rather than quantile distributions, including potential for continuously updated information from a single model collection. The value of information gains and losses may vary with each collaborative effort's aims, depending on the needs of projection users. Understanding the differing information potential of methods to collect model projections can support the accuracy, sustainability, and communication of collaborative infectious disease modelling efforts.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

An inaugural forum on epidemiological modeling for public health stakeholders in Arizona

Epidemiological models-which help us understand and forecast the spread of infectious disease-can be valuable tools for public health. However, barriers exist that can make it difficult to employ epidemiological models routinely within the repertoire of public health planning. These barriers include technical challenges associated with constructing the models, challenges in obtaining appropriate data for model parameterization, and problems with clear communication of modeling outputs and uncertainty. To learn about the unique barriers and opportunities within the state of Arizona, we gathered a diverse set of 48 public health stakeholders for a day-and-a-half forum. Our research group was motivated specifically by our work building software for public health-relevant modeling and by our earnest desire to collaborate closely with stakeholders to ensure that our software tools are practical and useful in the face of evolving public health needs. Here we outline the planning and structure of the forum, and we highlight as a case study some of the lessons learned from breakout discussions. While unique barriers exist for implementing modeling for public health, there is also keen interest in doing so across diverse sectors of State and Local government, although issues of equal and fair access to modeling knowledge and technologies remain key issues for future development. We found this forum to be useful for building relationships and informing our software development, and we plan to continue such meetings annually to create a continual feedback loop between academic molders and public health practitioners.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Spatiotemporal relative risk distribution of porcine reproductive and respiratory syndrome virus in the United States

Porcine reproductive and respiratory syndrome virus (PRRSV) remains widely distributed across the U.S. swine industry. Between-farm movements of animals and transportation vehicles, along with local transmission are the primary routes by which PRRSV is spread. Given the farm-to-farm proximity in high pig production areas, local transmission is an important pathway in the spread of PRRSV; however, there is limited understanding of the role local transmission plays in the dissemination of PRRSV, specifically, the distance at which there is increased risk for transmission from infected to susceptible farms. We used a spatial and spatiotemporal kernel density approach to estimate PRRSV relative risk and utilized a Bayesian spatiotemporal hierarchical model to assess the effects of environmental variables, between-farm movement data and on-farm biosecurity features on PRRSV outbreaks. The maximum spatial distance calculated through the kernel density approach was 15.3 km in 2018, 17.6 km in 2019, and 18 km in 2020. Spatiotemporal analysis revealed greater variability throughout the study period, with significant differences between the different farm types. We found that downstream farms (i.e., finisher and nursery farms) were located in areas of significant-high relative risk of PRRSV. Factors associated with PRRSV outbreaks were farms with higher number of access points to barns, higher numbers of outgoing movements of pigs, and higher number of days where temperatures were between 4°C and 10°C. Results obtained from this study may be used to guide the reinforcement of biosecurity and surveillance strategies to farms and areas within the distance threshold of PRRSV positive farms.

Operational analysis for COVID-19 testing: Determining the risk from asymptomatic infections

Testing remains a key tool for managing health care and making health policy during the coronavirus pandemic, and it will probably be important in future pandemics. Because of false negative and false positive tests, the observed fraction of positive tests-the surface positivity-is generally different from the fraction of infected individuals (the incidence rate of the disease). In this paper a previous method for translating surface positivity to a point estimate for incidence rate, then to an appropriate range of values for the incidence rate consistent with the model and data (the test range), and finally to the risk (the probability of including one infected individual) associated with groups of different sizes is illustrated. The method is then extended to include asymptomatic infections. To do so, the process of testing is modeled using both analysis and Monte Carlo simulation. Doing so shows that it is possible to determine point estimates for the fraction of infected and symptomatic individuals, the fraction of uninfected and symptomatic individuals, and the ratio of infected asymptomatic individuals to infected symptomatic individuals. Inclusion of symptom status generalizes the test range from an interval to a region in the plane determined by the incidence rate and the ratio of asymptomatic to symptomatic infections; likelihood methods can be used to determine the contour of the rest region. Points on this contour can be used to compute the risk (defined as the probability of including one asymptomatic infected individual) in groups of different sizes. These results have operational implications that include: positivity rate is not incidence rate; symptom status at testing can provide valuable information about asymptomatic infections; collecting information on time since putative virus exposure at testing is valuable for determining point estimates and test ranges; risk is a graded (rather than binary) function of group size; and because the information provided by testing becomes more accurate with more tests but at a decreasing rate, it is possible to over-test fixed spatial regions. The paper concludes with limitations of the method and directions for future work.

Context-dependent representation of within- and between-model uncertainty: aggregating probabilistic predictions in infectious disease epidemiology

Probabilistic predictions support public health planning and decision making, especially in infectious disease emergencies. Aggregating outputs from multiple models yields more robust predictions of outcomes and associated uncertainty. While the selection of an aggregation method can be guided by retrospective performance evaluations, this is not always possible. For example, if predictions are conditional on assumptions about how the future will unfold (e.g. possible interventions), these assumptions may never materialize, precluding any direct comparison between predictions and observations. Here, we summarize literature on aggregating probabilistic predictions, illustrate various methods for infectious disease predictions via simulation, and present a strategy for choosing an aggregation method when empirical validation cannot be used. We focus on the linear opinion pool (LOP) and Vincent average, common methods that make different assumptions about between-prediction uncertainty. We contend that assumptions of the aggregation method should align with a hypothesis about how uncertainty is expressed within and between predictions from different sources. The LOP assumes that between-prediction uncertainty is meaningful and should be retained, while the Vincent average assumes that between-prediction uncertainty is akin to sampling error and should not be preserved. We provide an R package for implementation. Given the rising importance of multi-model infectious disease hubs, our work provides useful guidance on aggregation and a deeper understanding of the benefits and risks of different approaches.

Vote-processing rules for combining control recommendations from multiple models

Mathematical modelling is used during disease outbreaks to compare control interventions. Using multiple models, the best method to combine model recommendations is unclear. Existing methods weight model projections, then rank control interventions using the combined projections, presuming model outputs are directly comparable. However, the way each model represents the epidemiological system will vary. We apply electoral vote-processing rules to combine model-generated rankings of interventions. Combining rankings of interventions, instead of combining model projections, avoids assuming that projections are comparable as all comparisons of projections are made within each model. We investigate four rules: First-past-the-post, Alternative Vote (AV), Coombs Method and Borda Count. We investigate rule sensitivity by including models that favour only one action or including those that rank interventions randomly. We investigate two case studies: the 2014 Ebola outbreak in West Africa (37 compartmental models) and a hypothetical foot-and-mouth disease outbreak in UK (four individual-based models). The Coombs Method was least susceptible to adding models that favoured a single action, Borda Count and AV were most susceptible to adding models that ranked interventions randomly. Each rule chose the same intervention as when ranking interventions by mean projections, suggesting that combining rankings provides similar recommendations with fewer assumptions about model comparability. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

A retrospective assessment of COVID-19 model performance in the USA

Coronavirus disease 2019 (COVID-19) forecasts from over 100 models are readily available. However, little published information exists regarding the performance of their uncertainty estimates (i.e. probabilistic performance). To evaluate their probabilistic performance, we employ the classical model (CM), an established method typically used to validate expert opinion. In this analysis, we assess both the predictive and probabilistic performance of COVID-19 forecasting models during 2021. We also compare the performance of aggregated forecasts (i.e. ensembles) based on equal and CM performance-based weights to an established ensemble from the Centers for Disease Control and Prevention (CDC). Our analysis of forecasts of COVID-19 mortality from 22 individual models and three ensembles across 49 states indicates that-(i) good predictive performance does not imply good probabilistic performance, and vice versa; (ii) models often provide tight but inaccurate uncertainty estimates; (iii) most models perform worse than a naive baseline model; (iv) both the CDC and CM performance-weighted ensembles perform well; but (v) while the CDC ensemble was more informative, the CM ensemble was more statistically accurate across states. This study presents a worthwhile method for appropriately assessing the performance of probabilistic forecasts and can potentially improve both public health decision-making and COVID-19 modelling.

Optimizing management of invasions in an uncertain world using dynamic spatial models

Dispersal drives invasion dynamics of nonnative species and pathogens. Applying knowledge of dispersal to optimize the management of invasions can mean the difference between a failed and a successful control program and dramatically improve the return on investment of control efforts. A common approach to identifying optimal management solutions for invasions is to optimize dynamic spatial models that incorporate dispersal. Optimizing these spatial models can be very challenging because the interaction of time, space, and uncertainty rapidly amplifies the number of dimensions being considered. Addressing such problems requires advances in and the integration of techniques from multiple fields, including ecology, decision analysis, bioeconomics, natural resource management, and optimization. By synthesizing recent advances from these diverse fields, we provide a workflow for applying ecological theory to advance optimal management science and highlight priorities for optimizing the control of invasions. One of the striking gaps we identify is the extremely limited consideration of dispersal uncertainty in optimal management frameworks, even though dispersal estimates are highly uncertain and greatly influence invasion outcomes. In addition, optimization frameworks rarely consider multiple types of uncertainty (we describe five major types) and their interrelationships. Thus, feedbacks from management or other sources that could magnify uncertainty in dispersal are rarely considered. Incorporating uncertainty is crucial for improving transparency in decision risks and identifying optimal management strategies. We discuss gaps and solutions to the challenges of optimization using dynamic spatial models to increase the practical application of these important tools and improve the consistency and robustness of management recommendations for invasions.

SPARSEMODr: Rapidly simulate spatially explicit and stochastic models of COVID-19 and other infectious diseases

Building realistically complex models of infectious disease transmission that are relevant for informing public health is conceptually challenging and requires knowledge of coding architecture that can implement key modeling conventions. For example, many of the models built to understand COVID-19 dynamics have included stochasticity, transmission dynamics that change throughout the epidemic due to changes in host behavior or public health interventions, and spatial structures that account for important spatio-temporal heterogeneities. Here we introduce an R package, SPARSEMODr, that allows users to simulate disease models that are stochastic and spatially explicit, including a model for COVID-19 that was useful in the early phases of the epidemic. SPARSEMOD stands for SPAtial Resolution-SEnsitive Models of Outbreak Dynamics, and our goal is to demonstrate particular conventions for rapidly simulating the dynamics of more complex, spatial models of infectious disease. In this report, we outline the features and workflows of our software package that allow for user-customized simulations. We believe the example models provided in our package will be useful in educational settings, as the coding conventions are adaptable, and will help new modelers to better understand important assumptions that were built into sophisticated COVID-19 models.

An expert judgment model to predict early stages of the COVID-19 pandemic in the United States

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single "linear pool" by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.

Robust models of disease heterogeneity and control, with application to the SARS-CoV-2 epidemic

In light of the continuing emergence of new SARS-CoV-2 variants and vaccines, we create a robust simulation framework for exploring possible infection trajectories under various scenarios. The situations of primary interest involve the interaction between three components: vaccination campaigns, non-pharmaceutical interventions (NPIs), and the emergence of new SARS-CoV-2 variants. Additionally, immunity waning and vaccine boosters are modeled to account for their growing importance. New infections are generated according to a hierarchical model in which people have a random, individual infectiousness. The model thus includes super-spreading observed in the COVID-19 pandemic which is important for accurate uncertainty prediction. Our simulation functions as a dynamic compartment model in which an individual's history of infection, vaccination, and possible reinfection all play a role in their resistance to further infections. We present a risk measure for each SARS-CoV-2 variant, [Formula: see text], that accounts for the amount of resistance within a population and show how this risk changes as the vaccination rate increases. [Formula: see text] highlights that different variants may become dominant in different countries-and in different times-depending on the population compositions in terms of previous infections and vaccinations. We compare the efficacy of control strategies which act to both suppress COVID-19 outbreaks and relax restrictions when possible. We demonstrate that a controller that responds to the effective reproduction number in addition to case numbers is more efficient and effective in controlling new waves than monitoring case numbers alone. This not only reduces the median total infections and peak quarantine cases, but also controls outbreaks much more reliably: such a controller entirely prevents rare but large outbreaks. This is important as the majority of public discussions about efficient control of the epidemic have so far focused primarily on thresholds for case numbers.

The ecology of COVID-19 and related environmental and sustainability issues

Around the globe, human behavior and ecosystem health have been extensively and sometimes severely affected by the unprecedented COVID-19 pandemic. Most efforts to study these complex and heterogenous effects to date have focused on public health and economics. Some studies have evaluated the pandemic's influences on the environment, but often on a single aspect such as air or water pollution. The related research opportunities are relatively rare, and the approaches are unique in multiple aspects and mostly retrospective. Here, we focus on the diverse research opportunities in disease ecology and ecosystem sustainability related to the (intermittent) lockdowns that drastically reduced human activities. We discuss several key knowledge gaps and questions to address amid the ongoing pandemic. In principle, the common knowledge accumulated from invasion biology could also be effectively applied to COVID-19, and the findings could offer much-needed information for future pandemic prevention and management.

Effectiveness of Localized Lockdowns in the COVID-19 Pandemic

Nonpharmaceutical interventions, such as social distancing and lockdowns, have been essential to control of the coronavirus disease 2019 (COVID-19) pandemic. In particular, localized lockdowns in small geographic areas have become an important policy intervention for preventing viral spread in cases of resurgence. These localized lockdowns can result in lower social and economic costs compared with larger-scale suppression strategies. Using an integrated data set from Chile (March 3-June 15, 2020) and a novel synthetic control approach, we estimated the effect of localized lockdowns, disentangling its direct and indirect causal effects on transmission of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Our results showed that the effects of localized lockdowns are strongly modulated by their duration and are influenced by indirect effects from neighboring geographic areas. Our estimates suggest that extending localized lockdowns can slow down SARS-CoV-2 transmission; however, localized lockdowns on their own are insufficient to control pandemic growth in the presence of indirect effects from contiguous neighboring areas that do not have lockdowns. These results provide critical empirical evidence about the effectiveness of localized lockdowns in interconnected geographic areas.

Challenges in estimation, uncertainty quantification and elicitation for pandemic modelling

The estimation of parameters and model structure for informing infectious disease response has become a focal point of the recent pandemic. However, it has also highlighted a plethora of challenges remaining in the fast and robust extraction of information using data and models to help inform policy. In this paper, we identify and discuss four broad challenges in the estimation paradigm relating to infectious disease modelling, namely the Uncertainty Quantification framework, data challenges in estimation, model-based inference and prediction, and expert judgement. We also postulate priorities in estimation methodology to facilitate preparation for future pandemics.

From Climate Change to Pandemics: Decision Science Can Help Scientists Have Impact

Scientific knowledge and advances are a cornerstone of modern society. They improve our understanding of the world we live in and help us navigate global challenges including emerging infectious diseases, climate change and the biodiversity crisis. However, there is a perpetual challenge in translating scientific insight into policy. Many articles explain how to better bridge the gap through improved communication and engagement, but we believe that communication and engagement are only one part of the puzzle. There is a fundamental tension between science and policy because scientific endeavors are rightfully grounded in discovery, but policymakers formulate problems in terms of objectives, actions and outcomes. Decision science provides a solution by framing scientific questions in a way that is beneficial to policy development, facilitating scientists’ contribution to public discussion and policy. At its core, decision science is a field that aims to pinpoint evidence-based management strategies by focussing on those objectives, actions, and outcomes defined through the policy process. The importance of scientific discovery here is in linking actions to outcomes, helping decision-makers determine which actions best meet their objectives. In this paper we explain how problems can be formulated through the structured decision-making process. We give our vision for what decision science may grow to be, describing current gaps in methodology and application. By better understanding and engaging with the decision-making processes, scientists can have greater impact and make stronger contributions to important societal problems.

Communicating uncertainty in epidemic models

While mathematical models of disease transmission are widely used to inform public health decision-makers globally, the uncertainty inherent in results are often poorly communicated. We outline some potential sources of uncertainty in epidemic models, present traditional methods used to illustrate uncertainty and discuss alternative presentation formats used by modelling groups throughout the COVID-19 pandemic. Then, by drawing on the experience of our own recent modelling, we seek to contribute to the ongoing discussion of how to improve upon traditional methods used to visualise uncertainty by providing a suggestion of how this can be presented in a clear and simple manner.

Reopening California: Seeking robust, non-dominated COVID-19 exit strategies

The COVID-19 pandemic required significant public health interventions from local governments. Although nonpharmaceutical interventions often were implemented as decision rules, few studies evaluated the robustness of those reopening plans under a wide range of uncertainties. This paper uses the Robust Decision Making approach to stress-test 78 alternative reopening strategies, using California as an example. This study uniquely considers a wide range of uncertainties and demonstrates that seemingly sensible reopening plans can lead to both unnecessary COVID-19 deaths and days of interventions. We find that plans using fixed COVID-19 case thresholds might be less effective than strategies with time-varying reopening thresholds. While we use California as an example, our results are particularly relevant for jurisdictions where vaccination roll-out has been slower. The approach used in this paper could also prove useful for other public health policy problems in which policymakers need to make robust decisions in the face of deep uncertainty.

Burden is in the eye of the beholder: Sensitivity of yellow fever disease burden estimates to modeling assumptions

Estimates of disease burden are important for setting public health priorities. These estimates involve numerous modeling assumptions, whose uncertainties are not always well described. We developed a framework for estimating the burden of yellow fever in Africa and evaluated its sensitivity to modeling assumptions that are often overlooked. We found that alternative interpretations of serological data resulted in a nearly 20-fold difference in burden estimates (range of central estimates, 8.4 × 104 to 1.5 × 106 deaths in 2021–2030). Uncertainty about the vaccination status of serological study participants was the primary driver of this uncertainty. Even so, statistical uncertainty was even greater than uncertainty due to modeling assumptions, accounting for a total of 87% of variance in burden estimates. Combined with estimates that most infections go unreported (range of 95% credible intervals, 99.65 to 99.99%), our results suggest that yellow fever’s burden will remain highly uncertain without major improvements in surveillance.

Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Probabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. We found that which model assumptions had the most ensemble weight changed through time. We additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Our results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases.

Coordinated Strategy for a Model-Based Decision Support Tool for Coronavirus Disease, Utah, USA

The coronavirus disease pandemic has highlighted the key role epidemiologic models play in supporting public health decision-making. In particular, these models provide estimates of outbreak potential when data are scarce and decision-making is critical and urgent. We document the integrated modeling response used in the US state of Utah early in the coronavirus disease pandemic, which brought together a diverse set of technical experts and public health and healthcare officials and led to an evidence-based response to the pandemic. We describe how we adapted a standard epidemiologic model; harmonized the outputs across modeling groups; and maintained a constant dialogue with policymakers at multiple levels of government to produce timely, evidence-based, and coordinated public health recommendations and interventions during the first wave of the pandemic. This framework continues to support the state’s response to ongoing outbreaks and can be applied in other settings to address unique public health challenges.

Afforestation, reforestation and new challenges from COVID-19: Thirty-three recommendations to support civil society organizations (CSOs)

Afforestation/reforestation (A/R) programs spearheaded by Civil Society Organizations (CSOs) play a significant role in reaching global climate policy targets and helping low-income nations meet the United Nations (UN) Sustainable Development Goals (SDGs). However, these organizations face unprecedented challenges due to the COVID-19 pandemic. Consequently, these challenges affect their ability to address issues associated with deforestation and forest degradation in a timely manner. We discuss the influence COVID-19 can have on previous, present and future A/R initiatives, in particular, the ones led by International Non-governmental Organizations (INGOs). We provide thirty-three recommendations for exploring underlying deforestation patterns and optimizing forest policy reforms to support forest cover expansion during the pandemic. The recommendations are classified into four groups - i) curbing deforestation and improving A/R, ii) protecting the environment and mitigating climate change, iii) enhancing socio-economic conditions, and iv) amending policy and law enforcement practices.

Assessing fitness-for-purpose and comparing the suitability of COVID-19 multi-country models for local contexts and users

Background: Mathematical models have been used throughout the COVID-19 pandemic to inform policymaking decisions. The COVID-19 Multi-Model Comparison Collaboration (CMCC) was established to provide country governments, particularly low- and middle-income countries (LMICs), and other model users with an overview of the aims, capabilities and limits of the main multi-country COVID-19 models to optimise their usefulness in the COVID-19 response. Methods: Seven models were identified that satisfied the inclusion criteria for the model comparison and had creators that were willing to participate in this analysis. A questionnaire, extraction tables and interview structure were developed to be used for each model, these tools had the aim of capturing the model characteristics deemed of greatest importance based on discussions with the Policy Group. The questionnaires were first completed by the CMCC Technical group using publicly available information, before further clarification and verification was obtained during interviews with the model developers. The fitness-for-purpose flow chart for assessing the appropriateness for use of different COVID-19 models was developed jointly by the CMCC Technical Group and Policy Group. Results: A flow chart of key questions to assess the fitness-for-purpose of commonly used COVID-19 epidemiological models was developed, with focus placed on their use in LMICs. Furthermore, each model was summarised with a description of the main characteristics, as well as the level of engagement and expertise required to use or adapt these models to LMIC settings. Conclusions: This work formalises a process for engagement with models, which is often done on an ad-hoc basis, with recommendations for both policymakers and model developers and should improve modelling use in policy decision making.

Data-Texts in the Sciences: The Evidence-Explanation Continuum

Grounded within current reform recommendations and built upon Giere's views (1986, 1999) on model-based science, we propose an alternative approach to science education which we refer to as the Evidence-Explanation (EE) Continuum. The approach addresses conceptual, epistemological, and social domains of knowledge, and places emphasis on the epistemological conversations about data acquisitions and transformations in the sciences. The steps of data transformation, which we refer to as data-texts, we argue, unfold the processes of using evidence during knowledge building and reveal the dynamics of scientific practices. Data-texts involve (a) obtaining observations/measurements to become data; (b) selecting and interpreting data to become evidence; (c) using evidence to ascertain patterns and develop models; and (d) utilizing the patterns and models to propose and refine explanations. Throughout the transformations of the EE continuum, there are stages of transition that foster the engagement of learners in negotiations of meaning and collective construction of knowledge. A focus on the EE continuum facilitates the emergence of further insights, both by questioning the nature of the data and its multiple possibilities for change and representations and by reflecting on the nature of the explanations. The shift of emphasis to the epistemics of science holds implications for the design of learning environments that support learners in developing contemporary understandings of the nature and processes of scientific practices.

Reopening California : Seeking Robust, Non-Dominated COVID-19 Exit Strategies

Amid global scarcity of COVID-19 vaccines and the threat of new variant strains, California and other jurisdictions face the question of when and how to implement and relax COVID-19 Nonpharmaceutical Interventions (NPIs). While policymakers have attempted to balance the health and economic impacts of the pandemic, decentralized decision-making, deep uncertainty, and the lack of widespread use of comprehensive decision support methods can lead to the choice of fragile or inefficient strategies. This paper uses simulation models and the Robust Decision Making (RDM) approach to stress-test California's reopening strategy and other alternatives over a wide range of futures. We find that plans which respond aggressively to initial outbreaks are required to robustly control the pandemic. Further, the best plans adapt to changing circumstances, lowering their stringent requirements to reopen over time or as more constituents are vaccinated. While we use California as an example, our results are particularly relevant for jurisdictions where vaccination roll-out has been slower.

A scenario modeling pipeline for COVID-19 emergency planning

Coronavirus disease 2019 (COVID-19) has caused strain on health systems worldwide due to its high mortality rate and the large portion of cases requiring critical care and mechanical ventilation. During these uncertain times, public health decision makers, from city health departments to federal agencies, sought the use of epidemiological models for decision support in allocating resources, developing non-pharmaceutical interventions, and characterizing the dynamics of COVID-19 in their jurisdictions. In response, we developed a flexible scenario modeling pipeline that could quickly tailor models for decision makers seeking to compare projections of epidemic trajectories and healthcare impacts from multiple intervention scenarios in different locations. Here, we present the components and configurable features of the COVID Scenario Pipeline, with a vignette detailing its current use. We also present model limitations and active areas of development to meet ever-changing decision maker needs.

“Saving Lives, Protecting Livelihoods, and Safeguarding Nature”: Risk-Based Wildlife Trade Policy for Sustainable Development Outcomes Post-COVID-19

The COVID-19 pandemic has caused huge loss of life, and immense social and economic harm. Wildlife trade has become central to discourse on COVID-19, zoonotic pandemics, and related policy responses, which must focus on “saving lives, protecting livelihoods, and safeguarding nature.” Proposed policy responses have included extreme measures such as banning all use and trade of wildlife, or blanket measures for entire Classes. However, different trades pose varying degrees of risk for zoonotic pandemics, while some trades also play critical roles in delivering other key aspects of sustainable development, particularly related to poverty and hunger alleviation, decent work, responsible consumption and production, and life on land and below water. Here we describe how wildlife trade contributes to the UN Sustainable Development Goals (SDGs) in diverse ways, with synergies and trade-offs within and between the SDGs. In doing so, we show that prohibitions could result in severe trade-offs against some SDGs, with limited benefits for public health via pandemic prevention. This complexity necessitates context-specific policies, with multi-sector decision-making that goes beyond simple top-down solutions. We encourage decision-makers to adopt a risk-based approach to wildlife trade policy post-COVID-19, with policies formulated via participatory, evidence-based approaches, which explicitly acknowledge uncertainty, complexity, and conflicting values across different components of the SDGs. This should help to ensure that future use and trade of wildlife is safe, environmentally sustainable and socially just.

The effect of COVID-19 on the economy: Evidence from an early adopter of localized lockdowns

BACKGROUND

Governments worldwide have implemented large-scale non-pharmaceutical interventions, such as social distancing or school closures, to prevent and control the growth of the COVID-19 pandemic. These strategies, implemented with varying stringency, have imposed substantial social and economic costs to society. As some countries begin to reopen and ease mobility restrictions, lockdowns in smaller geographic areas are increasingly considered an attractive policy intervention to mitigate societal costs while controlling epidemic growth. Nevertheless, there is a lack of empirical evidence to support these decisions.

METHODS

Drawing from a rich data set of localized lockdowns in Chile, we used econometric methods to measure the reduction in local economic activity from lockdowns when applied to smaller or larger geographical areas. We measured economic activity by tax collection at the municipality-level.

RESULTS

Our results show that lockdowns were associated with a 10%-15% drop in local economic activity, which is twice the reduction in local economic activity suffered by municipalities that were not under lockdown. A three-to-four-month lockdown had a similar effect on economic activity than a year of the 2009 great recession. We found costs are proportional to the population under lockdown, without differences when lockdowns were measured at the municipality or city-wide levels.

CONCLUSIONS

Our findings suggest that localized lockdowns have a large effect on local economic activity, but these effects are proportional to the population under lockdown. Our results suggest that epidemiological criteria should guide decisions about the optimal size of lockdown areas since the proportional impact of lockdowns on the economy seems to be unchanged by scale.

COVID-19 reopening strategies at the county level in the face of uncertainty: Multiple Models for Outbreak Decision Support

Policymakers make decisions about COVID-19 management in the face of considerable uncertainty. We convened multiple modeling teams to evaluate reopening strategies for a mid-sized county in the United States, in a novel process designed to fully express scientific uncertainty while reducing linguistic uncertainty and cognitive biases. For the scenarios considered, the consensus from 17 distinct models was that a second outbreak will occur within 6 months of reopening, unless schools and non-essential workplaces remain closed. Up to half the population could be infected with full workplace reopening; non-essential business closures reduced median cumulative infections by 82%. Intermediate reopening interventions identified no win-win situations; there was a trade-off between public health outcomes and duration of workplace closures. Aggregate results captured twice the uncertainty of individual models, providing a more complete expression of risk for decision-making purposes.

Three questions to ask before using model outputs for decision support

Decision makers must have sufficient confidence in models if they are to influence their decisions. We propose three screening questions to critically evaluate models with respect to their purpose, organization, and evidence. They enable a more transparent, robust, and secure use of model outputs.

Key questions for modelling COVID-19 exit strategies

Combinations of intense non-pharmaceutical interventions (lockdowns) were introduced worldwide to reduce SARS-CoV-2 transmission. Many governments have begun to implement exit strategies that relax restrictions while attempting to control the risk of a surge in cases. Mathematical modelling has played a central role in guiding interventions, but the challenge of designing optimal exit strategies in the face of ongoing transmission is unprecedented. Here, we report discussions from the Isaac Newton Institute 'Models for an exit strategy' workshop (11-15 May 2020). A diverse community of modellers who are providing evidence to governments worldwide were asked to identify the main questions that, if answered, would allow for more accurate predictions of the effects of different exit strategies. Based on these questions, we propose a roadmap to facilitate the development of reliable models to guide exit strategies. This roadmap requires a global collaborative effort from the scientific community and policymakers, and has three parts: (i) improve estimation of key epidemiological parameters; (ii) understand sources of heterogeneity in populations; and (iii) focus on requirements for data collection, particularly in low-to-middle-income countries. This will provide important information for planning exit strategies that balance socio-economic benefits with public health.