Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Informing the development of transmission modelling guidance for global immunization decision-making: A qualitative needs assessment

In recent years, mathematical transmission models have been increasingly used to support immunization program decisions and to measure the impact and cost-effectiveness of interventions. However, countries face expertise-and resource-related barriers that limit the use and application of modelled evidence to inform decisions. The World Health Organization (WHO) established an Immunization and Vaccines Implementation Research advisory committee subgroup in 2023 to support immunization decision-makers to effectively generate, translate and use such evidence for strategies, policies, and programs. This study supports this effort, detailing the needs of end-users to inform content and format of the guidance. Fifteen in-depth interviews were conducted with vaccination decision-makers and modelers from all six WHO regions and across low-, middle- and high-income countries. Interviews explored: (i) how modelling is understood and used; (ii) the challenges faced when using modelled evidence; (iii) the types of guidance that would be most useful to enhance the use of modelled evidence. Analysis of transcripts was guided by the framework method, which structures the analysis of qualitative data. Participants with modelling expertise used it firsthand, systematically, and often in an advisory capacity. Less experienced users, often in policy advisory roles, were less confident in their understanding of modelling and some did not use it at all. Decision-makers with little or no modelling experience cited a need for more information to help them understand the value of modelling in their context and many supported its potential. All participants saw a need for capacity strengthening and localised application to instil confidence in using modelled evidence. Those with less experience expressed a need for ongoing interactive engagement with knowledge brokers and training. Insights from this study are being integrated into the development of guidance by WHO. By considering the diverse challenges and needs of both experienced and inexperienced users of modelling, the guidance will support immunization strategy and policy by responding specifically to immunization decision-makers information needs.

Adopting Technological Innovations to Enhance Disaster Event Response

The convergence of medical and technological developments has continued to transform the delivery of medical care in disaster environments, incorporating advances from telecommunications to physiologic monitoring, artificial intelligence, and computer vision. However, unless the interconnected nature of these developments is conceptualized with a proper framework, there is a risk of overlooking applications, developing silos, and limiting interoperability between innovations. To develop such a framework, this piece integrated a review of current literature, expert insights, and global market trends to propose 4 categories of innovations: (1) Enabling Technologies, (2) Signal Acquisition, (3) Data Utilization, and (4) Applications. Applications can be further subdivided into 4 use cases: (1) Disease and Injury Surveillance and Detection, (2) Population Protection, (3) Responder Protection, and (4) Disease and Injury Management. Practitioners, policymakers, and private sector counterparts can utilize this framework to change their clinical practices, allocate funds in a stepwise fashion, or prioritize development projects, respectively.

Scenario construction and evolutionary analysis of nonconventional public health emergencies based on Bayesian networks

Objectives

The objective was to aggregate the various scenarios that occur during nonconventional public health emergencies (NCPHEs) and analyze the evolutionary patterns of NCPHEs to better avoid risks and reduce social impacts. The aim was to enhance strategies for handling NCPHEs.

Study design

News reports were crawled to obtain the scenario elements of NCPHEs and categorized into the spreading stage or derivation stage. Finally, the key scenario nodes and scenario evolution process were analyzed in combination with a corresponding emergency response assessment of each scenario by experts.

Methods

Dempster-Shafer (DS) theory and Bayesian networks (BNs) were applied for data reasoning, and a spread-derived coupled scenario-response theoretical model of NCPHEs for major public health emergencies was constructed. The scenario evolution path of COVID-19 was derived by combining seven types of major scenario states and corresponding emergency response measures extracted from 952 spreading scenarios.

Results

The 26 NCPHE spread scenarios and 41 NCPHE derivation scenarios were summarized. Optimized and pessimistic NCPHE scenario pathways were generated by combining the seven major spreading scenarios to help decision makers predict the development of NCPHEs and take timely and effective emergency response measures for key scenario nodes.

Conclusion

This study provides a new approach for understanding and managing NCPHEs, emphasizing the need to consider the specificity and complexity of such emergencies when developing decision-making strategies. Our contextual derivation model and emergency decision-making system provide practical tools with which to enhance NCPHE response capabilities and promote public health and safety.

Expanding the use of mathematical modeling in healthcare epidemiology and infection prevention and control

During the coronavirus disease 2019 pandemic, mathematical modeling has been widely used to understand epidemiological burden, trends, and transmission dynamics, to facilitate policy decisions, and, to a lesser extent, to evaluate infection prevention and control (IPC) measures. This review highlights the added value of using conventional epidemiology and modeling approaches to address the complexity of healthcare-associated infections (HAI) and antimicrobial resistance. It demonstrates how epidemiological surveillance data and modeling can be used to infer transmission dynamics in healthcare settings and to forecast healthcare impact, how modeling can be used to improve the validity of interpretation of epidemiological surveillance data, how modeling can be used to estimate the impact of IPC interventions, and how modeling can be used to guide IPC and antimicrobial treatment and stewardship decision-making. There are several priority areas for expanding the use of modeling in healthcare epidemiology and IPC. Importantly, modeling should be viewed as complementary to conventional healthcare epidemiological approaches, and this requires collaboration and active coordination between IPC, healthcare epidemiology, and mathematical modeling groups.

Inference of epidemic dynamics in the COVID-19 era and beyond

The COVID-19 pandemic demonstrated the key role that epidemiology and modelling play in analysing infectious threats and supporting decision making in real-time. Motivated by the unprecedented volume and breadth of data generated during the pandemic, we review modern opportunities for analysis to address questions that emerge during a major modern epidemic. Following the broad chronology of insights required - from understanding initial dynamics to retrospective evaluation of interventions, we describe the theoretical foundations of each approach and the underlying intuition. Through a series of case studies, we illustrate real life applications, and discuss implications for future work.

Title evaluation of FluSight influenza forecasting in the 2021-22 and 2022-23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021-22 and 2022-23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021-22 and 12 out of 18 models in 2022-23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021-22 and the 5th most accurate in the 2022-23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

Fitting Epidemic Models to Data: A Tutorial in Memory of Fred Brauer

Fred Brauer was an eminent mathematician who studied dynamical systems, especially differential equations. He made many contributions to mathematical epidemiology, a field that is strongly connected to data, but he always chose to avoid data analysis. Nevertheless, he recognized that fitting models to data is usually necessary when attempting to apply infectious disease transmission models to real public health problems. He was curious to know how one goes about fitting dynamical models to data, and why it can be hard. Initially in response to Fred's questions, we developed a user-friendly R package, fitode, that facilitates fitting ordinary differential equations to observed time series. Here, we use this package to provide a brief tutorial introduction to fitting compartmental epidemic models to a single observed time series. We assume that, like Fred, the reader is familiar with dynamical systems from a mathematical perspective, but has limited experience with statistical methodology or optimization techniques.

Infectious disease surveillance needs for the United States: lessons from Covid-19

The COVID-19 pandemic has highlighted the need to upgrade systems for infectious disease surveillance and forecasting and modeling of the spread of infection, both of which inform evidence-based public health guidance and policies. Here, we discuss requirements for an effective surveillance system to support decision making during a pandemic, drawing on the lessons of COVID-19 in the U.S., while looking to jurisdictions in the U.S. and beyond to learn lessons about the value of specific data types. In this report, we define the range of decisions for which surveillance data are required, the data elements needed to inform these decisions and to calibrate inputs and outputs of transmission-dynamic models, and the types of data needed to inform decisions by state, territorial, local, and tribal health authorities. We define actions needed to ensure that such data will be available and consider the contribution of such efforts to improving health equity.

Data-driven mechanistic framework with stratified immunity and effective transmissibility for COVID-19 scenario projections

Scenario-based modeling frameworks have been widely used to support policy-making at state and federal levels in the United States during the COVID-19 response. While custom-built models can be used to support one-off studies, sustained updates to projections under changing pandemic conditions requires a robust, integrated, and adaptive framework. In this paper, we describe one such framework, UVA-adaptive, that was built to support the CDC-aligned Scenario Modeling Hub (SMH) across multiple rounds, as well as weekly/biweekly projections to Virginia Department of Health (VDH) and US Department of Defense during the COVID-19 response. Building upon an existing metapopulation framework, PatchSim, UVA-adaptive uses a calibration mechanism relying on adjustable effective transmissibility as a basis for scenario definition while also incorporating real-time datasets on case incidence, seroprevalence, variant characteristics, and vaccine uptake. Through the pandemic, our framework evolved by incorporating available data sources and was extended to capture complexities of multiple strains and heterogeneous immunity of the population. Here we present the version of the model that was used for the recent projections for SMH and VDH, describe the calibration and projection framework, and demonstrate that the calibrated transmissibility correlates with the evolution of the pathogen as well as associated societal dynamics.

When do we need multiple infectious disease models? Agreement between projection rank and magnitude in a multi-model setting

Mathematical models are useful for public health planning and response to infectious disease threats. However, different models can provide differing results, which can hamper decision making if not synthesized appropriately. To address this challenge, multi-model hubs convene independent modeling groups to generate ensembles, known to provide more accurate predictions of future outcomes. Yet, these hubs are resource intensive, and how many models are sufficient in a hub is not known. Here, we compare the benefit of predictions from multiple models in different contexts: (1) decision settings that depend on predictions of quantitative outcomes (e.g., hospital capacity planning), where assessments of the benefits of multi-model ensembles have largely focused; and (2) decisions settings that require the ranking of alternative epidemic scenarios (e.g., comparing outcomes under multiple possible interventions and biological uncertainties). We develop a mathematical framework to mimic a multi-model prediction setting, and use this framework to quantify how frequently predictions from different models agree. We further explore multi-model agreement using real-world, empirical data from 14 rounds of U.S. COVID-19 Scenario Modeling Hub projections. Our results suggest that the value of multiple models could be different in different decision contexts, and if only a few models are available, focusing on the rank of alternative epidemic scenarios could be more robust than focusing on quantitative outcomes. Although additional exploration of the sufficient number of models for different contexts is still needed, our results indicate that it may be possible to identify decision contexts where it is robust to rely on fewer models, a finding that can inform the use of modeling resources during future public health crises.

A multiscale modeling framework for Scenario Modeling: Characterizing the heterogeneity of the COVID-19 epidemic in the US

The Scenario Modeling Hub (SMH) initiative provides projections of potential epidemic scenarios in the United States (US) by using a multi-model approach. Our contribution to the SMH is generated by a multiscale model that combines the global epidemic metapopulation modeling approach (GLEAM) with a local epidemic and mobility model of the US (LEAM-US), first introduced here. The LEAM-US model consists of 3142 subpopulations each representing a single county across the 50 US states and the District of Columbia, enabling us to project state and national trajectories of COVID-19 cases, hospitalizations, and deaths under different epidemic scenarios. The model is age-structured, and multi-strain. It integrates data on vaccine administration, human mobility, and non-pharmaceutical interventions. The model contributed to all 17 rounds of the SMH, and allows for the mechanistic characterization of the spatio-temporal heterogeneities observed during the COVID-19 pandemic. Here we describe the mathematical and computational structure of our model, and present the results concerning the emergence of the SARS-CoV-2 Alpha variant (lineage designation B.1.1.7) as a case study. Our findings show considerable spatial and temporal heterogeneity in the introduction and diffusion of the Alpha variant, both at the level of individual states and combined statistical areas, as it competes against the ancestral lineage. We discuss the key factors driving the time required for the Alpha variant to rise to dominance within a population, and quantify the impact that the emergence of the Alpha variant had on the effective reproduction number at the state level. Overall, we show that our multiscale modeling approach is able to capture the complexity and heterogeneity of the COVID-19 pandemic response in the US.

The Limitations of a Hypothetical All-Variant COVID-19 Vaccine: A Simulation Study

This paper simulates a hypothetical pan-coronavirus vaccine that confers immediate sterilizing immunity against all SARS-CoV-2 variants. Simulations used a SEIIS (susceptible, exposed, infective, immune, susceptible) spreadsheet model that ran two parallel subpopulations: one that accepted vaccination and another that refused it. The two subpopulations could transmit infections to one another. Using data from the United States (US), the simulated vaccine was tested against limiting factors such as vaccine hesitancy, slow vaccination distribution, and the development of high-transmission variants. The vaccine was often successful at reducing cases, but high-transmission variants and discontinuation of non-pharmaceutical interventions (NPIs) such as masking greatly elevated cases. A puzzling outcome was that if NPIs were discontinued and high-transmission variants became common, the model predicted consistently higher rates of disease than are actually observed in the US in 2024. However, if cumulative exposure to virus antigens increased the duration of immunity or decreased the infectivity of the virus, the model predictions were brought back into a more realistic range. The major finding was that even when a COVID-19 vaccine always produces sterilizing immunity against every SARS-CoV-2 variant, its ability to control the epidemic can be compromised by multiple common conditions.

Combining models to generate consensus medium-term projections of hospital admissions, occupancy and deaths relating to COVID-19 in England

Mathematical modelling has played an important role in offering informed advice during the COVID-19 pandemic. In England, a cross government and academia collaboration generated medium-term projections (MTPs) of possible epidemic trajectories over the future 4-6 weeks from a collection of epidemiological models. In this article, we outline this collaborative modelling approach and evaluate the accuracy of the combined and individual model projections against the data over the period November 2021-December 2022 when various Omicron subvariants were spreading across England. Using a number of statistical methods, we quantify the predictive performance of the model projections for both the combined and individual MTPs, by evaluating the point and probabilistic accuracy. Our results illustrate that the combined MTPs, produced from an ensemble of heterogeneous epidemiological models, were a closer fit to the data than the individual models during the periods of epidemic growth or decline, with the 90% confidence intervals widest around the epidemic peaks. We also show that the combined MTPs increase the robustness and reduce the biases associated with a single model projection. Learning from our experience of ensemble modelling during the COVID-19 epidemic, our findings highlight the importance of developing cross-institutional multi-model infectious disease hubs for future outbreak control.

Potential impact of annual vaccination with reformulated COVID-19 vaccines: Lessons from the US COVID-19 scenario modeling hub

BACKGROUND

Coronavirus Disease 2019 (COVID-19) continues to cause significant hospitalizations and deaths in the United States. Its continued burden and the impact of annually reformulated vaccines remain unclear. Here, we present projections of COVID-19 hospitalizations and deaths in the United States for the next 2 years under 2 plausible assumptions about immune escape (20% per year and 50% per year) and 3 possible CDC recommendations for the use of annually reformulated vaccines (no recommendation, vaccination for those aged 65 years and over, vaccination for all eligible age groups based on FDA approval).

METHODS AND FINDINGS

The COVID-19 Scenario Modeling Hub solicited projections of COVID-19 hospitalization and deaths between April 15, 2023 and April 15, 2025 under 6 scenarios representing the intersection of considered levels of immune escape and vaccination. Annually reformulated vaccines are assumed to be 65% effective against symptomatic infection with strains circulating on June 15 of each year and to become available on September 1. Age- and state-specific coverage in recommended groups was assumed to match that seen for the first (fall 2021) COVID-19 booster. State and national projections from 8 modeling teams were ensembled to produce projections for each scenario and expected reductions in disease outcomes due to vaccination over the projection period. From April 15, 2023 to April 15, 2025, COVID-19 is projected to cause annual epidemics peaking November to January. In the most pessimistic scenario (high immune escape, no vaccination recommendation), we project 2.1 million (90% projection interval (PI) [1,438,000, 4,270,000]) hospitalizations and 209,000 (90% PI [139,000, 461,000]) deaths, exceeding pre-pandemic mortality of influenza and pneumonia. In high immune escape scenarios, vaccination of those aged 65+ results in 230,000 (95% confidence interval (CI) [104,000, 355,000]) fewer hospitalizations and 33,000 (95% CI [12,000, 54,000]) fewer deaths, while vaccination of all eligible individuals results in 431,000 (95% CI: 264,000-598,000) fewer hospitalizations and 49,000 (95% CI [29,000, 69,000]) fewer deaths.

CONCLUSIONS

COVID-19 is projected to be a significant public health threat over the coming 2 years. Broad vaccination has the potential to substantially reduce the burden of this disease, saving tens of thousands of lives each year.

The US COVID-19 and Influenza Scenario Modeling Hubs: Delivering long-term projections to guide policy

Between December 2020 and April 2023, the COVID-19 Scenario Modeling Hub (SMH) generated operational multi-month projections of COVID-19 burden in the US to guide pandemic planning and decision-making in the context of high uncertainty. This effort was born out of an attempt to coordinate, synthesize and effectively use the unprecedented amount of predictive modeling that emerged throughout the COVID-19 pandemic. Here we describe the history of this massive collective research effort, the process of convening and maintaining an open modeling hub active over multiple years, and attempt to provide a blueprint for future efforts. We detail the process of generating 17 rounds of scenarios and projections at different stages of the COVID-19 pandemic, and disseminating results to the public health community and lay public. We also highlight how SMH was expanded to generate influenza projections during the 2022-23 season. We identify key impacts of SMH results on public health and draw lessons to improve future collaborative modeling efforts, research on scenario projections, and the interface between models and policy.

Ensemble2: Scenarios ensembling for communication and performance analysis

Throughout the COVID-19 pandemic, scenario modeling played a crucial role in shaping the decision-making process of public health policies. Unlike forecasts, scenario projections rely on specific assumptions about the future that consider different plausible states-of-the-world that may or may not be realized and that depend on policy interventions, unpredictable changes in the epidemic outlook, etc. As a consequence, long-term scenario projections require different evaluation criteria than the ones used for traditional short-term epidemic forecasts. Here, we propose a novel ensemble procedure for assessing pandemic scenario projections using the results of the Scenario Modeling Hub (SMH) for COVID-19 in the United States (US). By defining a "scenario ensemble" for each model and the ensemble of models, termed "Ensemble2", we provide a synthesis of potential epidemic outcomes, which we use to assess projections' performance, bypassing the identification of the most plausible scenario. We find that overall the Ensemble2 models are well-calibrated and provide better performance than the scenario ensemble of individual models. The ensemble procedure accounts for the full range of plausible outcomes and highlights the importance of scenario design and effective communication. The scenario ensembling approach can be extended to any scenario design strategy, with potential refinements including weighting scenarios and allowing the ensembling process to evolve over time.