Characterising information gains and losses when collecting multiple epidemic model outputs

BACKGROUND

Collaborative comparisons and combinations of epidemic models are used as policy-relevant evidence during epidemic outbreaks. In the process of collecting multiple model projections, such collaborations may gain or lose relevant information. Typically, modellers contribute a probabilistic summary at each time-step. We compared this to directly collecting simulated trajectories. We aimed to explore information on key epidemic quantities; ensemble uncertainty; and performance against data, investigating potential to continuously gain information from a single cross-sectional collection of model results.

METHODS

We compared projections from the European COVID-19 Scenario Modelling Hub. Five teams modelled incidence in Belgium, the Netherlands, and Spain. We compared July 2022 projections by incidence, peaks, and cumulative totals. We created a probabilistic ensemble drawn from all trajectories, and compared to ensembles from a median across each model's quantiles, or a linear opinion pool. We measured the predictive accuracy of individual trajectories against observations, using this in a weighted ensemble. We repeated this sequentially against increasing weeks of observed data. We evaluated these ensembles to reflect performance with varying observed data.

RESULTS

By collecting modelled trajectories, we showed policy-relevant epidemic characteristics. Trajectories contained a right-skewed distribution well represented by an ensemble of trajectories or a linear opinion pool, but not models' quantile intervals. Ensembles weighted by performance typically retained the range of plausible incidence over time, and in some cases narrowed this by excluding some epidemic shapes.

CONCLUSIONS

We observed several information gains from collecting modelled trajectories rather than quantile distributions, including potential for continuously updated information from a single model collection. The value of information gains and losses may vary with each collaborative effort's aims, depending on the needs of projection users. Understanding the differing information potential of methods to collect model projections can support the accuracy, sustainability, and communication of collaborative infectious disease modelling efforts.

Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations

Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts' predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022.

Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance.

Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models.

Conclusions: Our results support combining independent models into an ensemble forecast to improve epidemiological predictions, and suggest that median averages yield better performance than methods based on means. We highlight that forecast consumers should place more weight on incident death forecasts than case forecasts at horizons greater than two weeks.

Funding: European Commission, Ministerio de Ciencia, Innovación y Universidades, FEDER; Agència de Qualitat i Avaluació Sanitàries de Catalunya; Netzwerk Universitätsmedizin; Health Protection Research Unit; Wellcome Trust; European Centre for Disease Prevention and Control; Ministry of Science and Higher Education of Poland; Federal Ministry of Education and Research; Los Alamos National Laboratory; German Free State of Saxony; NCBiR; FISR 2020 Covid-19 I Fase; Spanish Ministry of Health / REACT-UE (FEDER); National Institutes of General Medical Sciences; Ministerio de Sanidad/ISCIII; PERISCOPE European H2020; PERISCOPE European H2021; InPresa; National Institutes of Health, NSF, US Centers for Disease Control and Prevention, Google, University of Virginia, Defense Threat Reduction Agency.

Corrigendum: Simulation of COVID-19 propagation scenarios in the Madrid metropolitan area

[This corrects the article DOI: 10.3389/fpubh.2021.636023.].

Evaluation of vaccination strategies for the metropolitan area of Madrid via agent-based simulation

OBJECTIVE

We analyse the impact of different vaccination strategies on the propagation of COVID-19 within the Madrid metropolitan area, starting on 27 December 2020 and ending in Summer of 2021.

MATERIALS AND METHODS

The predictions are based on simulation using EpiGraph, an agent-based COVID-19 simulator. We first summarise the different models implemented in the simulator, then provide a comprehensive description of the vaccination model and define different vaccination strategies. The simulator-including the vaccination model-is validated by comparing its results with real data from the metropolitan area of Madrid during the third COVID-19 wave. This work considers different COVID-19 propagation scenarios for a simulated population of about 5 million.

RESULTS

The main result shows that the best strategy is to vaccinate first the elderly with the two doses spaced 56 days apart; this approach reduces the final infection rate by an additional 6% and the number of deaths by an additional 3% with respect to vaccinating first the elderly at the interval recommended by the vaccine producer. The reason is the increase in the number of vaccinated individuals at any time during the simulation.

CONCLUSION

The existing level of detail and maturity of EpiGraph allowed us to evaluate complex scenarios and thus use it successfully to help guide the strategy for the COVID-19 vaccination campaign of the Spanish health authorities.

Assessing population-sampling strategies for reducing the COVID-19 incidence

As long as critical levels of vaccination have not been reached to ensure heard immunity, and new SARS-CoV-2 strains are developing, the only realistic way to reduce the infection speed in a population is to track the infected individuals before they pass on the virus. Testing the population via sampling has shown good results in slowing the epidemic spread. Sampling can be implemented at different times during the epidemic and may be done either per individual or for combined groups of people at a time. The work we present here makes two main contributions. We first extend and refine our scalable agent-based COVID-19 simulator to incorporate an improved socio-demographic model which considers professions, as well as a more realistic population mixing model based on contact matrices per country. These extensions are necessary to develop and test various sampling strategies in a scenario including the 62 largest cities in Spain; this is our second contribution. As part of the evaluation, we also analyze the impact of different parameters, such as testing frequency, quarantine time, percentage of quarantine breakers, or group testing, on sampling efficacy. Our results show that the most effective strategies are pooling, rapid antigen test campaigns, and requiring negative testing for access to public areas. The effectiveness of all these strategies can be greatly increased by reducing the number of contacts for infected individual.

Simulation of COVID-19 Propagation Scenarios in the Madrid Metropolitan Area

This work presents simulation results for different mitigation and confinement scenarios for the propagation of COVID-19 in the metropolitan area of Madrid. These scenarios were implemented and tested using EpiGraph, an epidemic simulator which has been extended to simulate COVID-19 propagation. EpiGraph implements a social interaction model, which realistically captures a large number of characteristics of individuals and groups, as well as their individual interconnections, which are extracted from connection patterns in social networks. Besides the epidemiological and social interaction components, it also models people's short and long-distance movements as part of a transportation model. These features, together with the capacity to simulate scenarios with millions of individuals and apply different contention and mitigation measures, gives EpiGraph the potential to reproduce the COVID-19 evolution and study medium-term effects of the virus when applying mitigation methods. EpiGraph, obtains closely aligned infected and death curves related to the first wave in the Madrid metropolitan area, achieving similar seroprevalence values. We also show that selective lockdown for people over 60 would reduce the number of deaths. In addition, evaluate the effect of the use of face masks after the first wave, which shows that the percentage of people that comply with mask use is a crucial factor for mitigating the infection's spread.

Evaluating the impact of the weather conditions on the influenza propagation

Background

Predicting the details of how an epidemic evolves is highly valuable as health institutions need to better plan towards limiting the infection propagation effects and optimizing their prediction and response capabilities. Simulation is a cost- and time-effective way of predicting the evolution of the infection as the joint influence of many different factors: interaction patterns, personal characteristics, travel patterns, meteorological conditions, previous vaccination, etc. The work presented in this paper extends EpiGraph, our influenza epidemic simulator, by introducing a meteorological model as a modular component that interacts with the rest of EpiGraph’s modules to refine our previous simulation results. Our goal is to estimate the effects of changes in temperature and relative humidity on the patterns of epidemic influenza based on data provided by the Spanish Influenza Sentinel Surveillance System (SISSS) and the Spanish Meteorological Agency (AEMET).

Methods

Our meteorological model is based on the regression model developed by AB and JS, and it is tuned with influenza surveillance data obtained from SISSS. After pre-processing this data to clean it and reconstruct missing samples, we obtain new values for the reproduction number of each urban region in Spain, every 10 minutes during 2011. We simulate the propagation of the influenza by setting the date of the epidemic onset and the initial influenza-illness rates for each urban region.

Results

We show that the simulation results have the same propagation shape as the weekly influenza rates as recorded by SISSS. We perform experiments for a realistic scenario based on actual meteorological data from 2010-2011, and for synthetic values assumed under simplified predicted climate change conditions. Results show that a diminishing relative humidity of 10% produces an increment of about 1.6% in the final infection rate. The effect of temperature changes on the infection spread is also noticeable, with a decrease of 1.1% per extra degree.Conclusions: Using a tool like ours could help predict the shape of developing epidemics and its peaks, and would permit to quickly run scenarios to determine the evolution of the epidemic under different conditions. We make EpiGraph source code and epidemic data publicly available.

Leveraging social networks for understanding the evolution of epidemics

BACKGROUND

To understand how infectious agents disseminate throughout a population it is essential to capture the social model in a realistic manner. This paper presents a novel approach to modeling the propagation of the influenza virus throughout a realistic interconnection network based on actual individual interactions which we extract from online social networks. The advantage is that these networks can be extracted from existing sources which faithfully record interactions between people in their natural environment. We additionally allow modeling the characteristics of each individual as well as customizing his daily interaction patterns by making them time-dependent. Our purpose is to understand how the infection spreads depending on the structure of the contact network and the individuals who introduce the infection in the population. This would help public health authorities to respond more efficiently to epidemics.

RESULTS

We implement a scalable, fully distributed simulator and validate the epidemic model by comparing the simulation results against the data in the 2004-2005 New York State Department of Health Report (NYSDOH), with similar temporal distribution results for the number of infected individuals. We analyze the impact of different types of connection models on the virus propagation. Lastly, we analyze and compare the effects of adopting several different vaccination policies, some of them based on individual characteristics -such as age- while others targeting the super-connectors in the social model.

CONCLUSIONS

This paper presents an approach to modeling the propagation of the influenza virus via a realistic social model based on actual individual interactions extracted from online social networks. We implemented a scalable, fully distributed simulator and we analyzed both the dissemination of the infection and the effect of different vaccination policies on the progress of the epidemics. The epidemic values predicted by our simulator match real data from NYSDOH. Our results show that our simulator can be a useful tool in understanding the differences in the evolution of an epidemic within populations with different characteristics and can provide guidance with regard to which, and how many, individuals should be vaccinated to slow down the virus propagation and reduce the number of infections.