Multiple models for outbreak decision support in the face of uncertainty

Leveraging artificial intelligence and software engineering methods in epidemiology for the co-creation of decision-support tools based on mechanistic models

Epidemiological modeling is a key lever for infectious disease control and prevention on farms. It makes it possible to understand the spread of pathogens, but also to compare intervention scenarios even in counterfactual situations. However, the actual capability of decision makers to use mechanistic models to support timely interventions is limited. This study demonstrates how artificial intelligence (AI) techniques can make mechanistic epidemiological models more accessible to farmers and veterinarians, and how to transform such models into user-friendly decision-support tools (DST). By leveraging knowledge representation methods, such as the textual formalization of model components through a domain-specific language (DSL), the co-design of mechanistic models and DST becomes more efficient and collaborative. This facilitates the integration of explicit expert knowledge and practical insights into the modeling process. Furthermore, the utilization of AI and software engineering enables the automation of web application generation based on existing mechanistic models. This automation simplifies the development of DST, as tool designers can focus on identifying users' needs and specifying expected features and meaningful presentations of outcomes, instead of wasting time in writing code to wrap models into web apps. To illustrate the practical application of this approach, we consider the example of Bovine Respiratory Disease (BRD), a tough challenge in fattening farms where young beef bulls often develop BRD shortly after being allocated into pens. BRD is a multi-factorial, multi-pathogen disease that is difficult to anticipate and control, often resulting in the massive use of antimicrobials to mitigate its impact on animal health, welfare, and economic losses. The DST developed from an existing mechanistic BRD model empowers users, including farmers and veterinarians, to customize scenarios based on their specific farm conditions. It enables them to anticipate the effects of various pathogens, compare the epidemiological and economic outcomes associated with different farming practices, and decide how to balance the reduction of disease impact and the reduction of antimicrobial usage (AMU). The generic method presented in this article illustrates the potential of artificial intelligence (AI) and software engineering methods to enhance the co-creation of DST based on mechanistic models in veterinary epidemiology. The corresponding pipeline is distributed as an open-source software. By leveraging these advancements, this research aims to bridge the gap between theoretical models and the practical usage of their outcomes on the field.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

Projecting the future impact of emerging SARS-CoV-2 variants under uncertainty: Modeling the initial Omicron outbreak

Over the past several years, the emergence of novel SARS-CoV-2 variants has led to multiple waves of increased COVID-19 incidence. When the Omicron variant emerged, there was considerable concern about its potential impact in the winter of 2021-2022 due to its increased fitness. However, there was also considerable uncertainty regarding its likely impact due to questions about its relative transmissibility, severity, and degree of immune escape. We sought to evaluate the ability of an agent-based model to forecast incidence in the context of this emerging pathogen variant. To project COVID-19 cases and deaths in Indiana, we calibrated our model to COVID-19 hospitalizations, deaths, and test-positivity rates through November 2021, and then projected COVID-19 incidence through April 2022 under four different scenarios that covered the plausible ranges of Omicron's severity, transmissibility, and degree of immune escape. Our initial projections from December 2021 through March 2022 indicated that under a pessimistic scenario with high disease severity, the peak in weekly COVID-19 deaths in Indiana would be larger than the previous peak in December 2020. However, retrospective analyses indicate that Omicron's severity was closer to the optimistic scenario, and even though cases and hospitalizations reached a new peak, fewer deaths occurred than during the previous peak. According to our results, Omicron's rapid spread was consistent with a combination of higher transmissibility and immune escape relative to earlier variants. Our updated projections starting in January 2022 accurately predicted that cases would peak in mid-January and decline rapidly over the next several months. The performance of our projections shows that following the emergence of a new pathogen variant, models can help quantify the potential range of outbreak magnitudes and trajectories. Agent-based models are particularly useful in these scenarios because they can efficiently track individual vaccination and infection histories with multiple variants with varying degrees of cross-protection.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Scenario Design for Infectious Disease Projections: Integrating Concepts from Decision Analysis and Experimental Design

Across many fields, scenario modeling has become an important tool for exploring long-term projections and how they might depend on potential interventions and critical uncertainties, with relevance to both decision makers and scientists. In the past decade, and especially during the COVID-19 pandemic, the field of epidemiology has seen substantial growth in the use of scenario projections. Multiple scenarios are often projected at the same time, allowing important comparisons that can guide the choice of intervention, the prioritization of research topics, or public communication. The design of the scenarios is central to their ability to inform important questions. In this paper, we draw on the fields of decision analysis and statistical design of experiments to propose a framework for scenario design in epidemiology, with relevance also to other fields. We identify six different fundamental purposes for scenario designs (decision making, sensitivity analysis, value of information, situational awareness, horizon scanning, and forecasting) and discuss how those purposes guide the structure of scenarios. We discuss other aspects of the content and process of scenario design, broadly for all settings and specifically for multi-model ensemble projections. As an illustrative case study, we examine the first 17 rounds of scenarios from the U.S. COVID-19 Scenario Modeling Hub, then reflect on future advancements that could improve the design of scenarios in epidemiological settings.

Informing pandemic response in the face of uncertainty. An evaluation of the U.S. COVID-19 Scenario Modeling Hub

Our ability to forecast epidemics more than a few weeks into the future is constrained by the complexity of disease systems, our limited ability to measure the current state of an epidemic, and uncertainties in how human action will affect transmission. Realistic longer-term projections (spanning more than a few weeks) may, however, be possible under defined scenarios that specify the future state of critical epidemic drivers, with the additional benefit that such scenarios can be used to anticipate the comparative effect of control measures. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make 6-month ahead projections of the number of SARS-CoV-2 cases, hospitalizations and deaths. The SMH released nearly 1.8 million national and state-level projections between February 2021 and November 2022. SMH performance varied widely as a function of both scenario validity and model calibration. Scenario assumptions were periodically invalidated by the arrival of unanticipated SARS-CoV-2 variants, but SMH still provided projections on average 22 weeks before changes in assumptions (such as virus transmissibility) invalidated scenarios and their corresponding projections. During these periods, before emergence of a novel variant, a linear opinion pool ensemble of contributed models was consistently more reliable than any single model, and projection interval coverage was near target levels for the most plausible scenarios (e.g., 79% coverage for 95% projection interval). SMH projections were used operationally to guide planning and policy at different stages of the pandemic, illustrating the value of the hub approach for long-term scenario projections.

Spatial dynamics of malaria transmission

The Ross-Macdonald model has exerted enormous influence over the study of malaria transmission dynamics and control, but it lacked features to describe parasite dispersal, travel, and other important aspects of heterogeneous transmission. Here, we present a patch-based differential equation modeling framework that extends the Ross-Macdonald model with sufficient skill and complexity to support planning, monitoring and evaluation for Plasmodium falciparum malaria control. We designed a generic interface for building structured, spatial models of malaria transmission based on a new algorithm for mosquito blood feeding. We developed new algorithms to simulate adult mosquito demography, dispersal, and egg laying in response to resource availability. The core dynamical components describing mosquito ecology and malaria transmission were decomposed, redesigned and reassembled into a modular framework. Structural elements in the framework-human population strata, patches, and aquatic habitats-interact through a flexible design that facilitates construction of ensembles of models with scalable complexity to support robust analytics for malaria policy and adaptive malaria control. We propose updated definitions for the human biting rate and entomological inoculation rates. We present new formulas to describe parasite dispersal and spatial dynamics under steady state conditions, including the human biting rates, parasite dispersal, the "vectorial capacity matrix," a human transmitting capacity distribution matrix, and threshold conditions. An [Formula: see text] package that implements the framework, solves the differential equations, and computes spatial metrics for models developed in this framework has been developed. Development of the model and metrics have focused on malaria, but since the framework is modular, the same ideas and software can be applied to other mosquito-borne pathogen systems.