2
|
Sherratt K, Gruson H, Grah R, Johnson H, Niehus R, Prasse B, Sandmann F, Deuschel J, Wolffram D, Abbott S, Ullrich A, Gibson G, Ray EL, Reich NG, Sheldon D, Wang Y, Wattanachit N, Wang L, Trnka J, Obozinski G, Sun T, Thanou D, Pottier L, Krymova E, Meinke JH, Barbarossa MV, Leithäuser N, Mohring J, Schneider J, Włazło J, Fuhrmann J, Lange B, Rodiah I, Baccam P, Gurung H, Stage S, Suchoski B, Budzinski J, Walraven R, Villanueva I, Tucek V, Smid M, Zajíček M, Pérez Álvarez C, Reina B, Bosse NI, Meakin SR, Castro L, Fairchild G, Michaud I, Osthus D, Alaimo Di Loro P, Maruotti A, Eclerová V, Kraus A, Kraus D, Pribylova L, Dimitris B, Li ML, Saksham S, Dehning J, Mohr S, Priesemann V, Redlarski G, Bejar B, Ardenghi G, Parolini N, Ziarelli G, Bock W, Heyder S, Hotz T, Singh DE, Guzman-Merino M, Aznarte JL, Moriña D, Alonso S, Álvarez E, López D, Prats C, Burgard JP, Rodloff A, Zimmermann T, Kuhlmann A, Zibert J, Pennoni F, Divino F, Català M, Lovison G, Giudici P, Tarantino B, Bartolucci F, Jona Lasinio G, Mingione M, Farcomeni A, Srivastava A, Montero-Manso P, Adiga A, Hurt B, Lewis B, Marathe M, Porebski P, Venkatramanan S, Bartczuk RP, Dreger F, Gambin A, Gogolewski K, Gruziel-Słomka M, Krupa B, Moszyński A, Niedzielewski K, Nowosielski J, Radwan M, Rakowski F, Semeniuk M, Szczurek E, Zieliński J, Kisielewski J, Pabjan B, Kirsten H, Kheifetz Y, Scholz M, Biecek P, Bodych M, Filinski M, Idzikowski R, Krueger T, Ozanski T, Bracher J, Funk S. Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations. eLife 2023; 12:81916. [PMID: 37083521 DOI: 10.7554/elife.81916] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 02/20/2023] [Indexed: 04/22/2023] Open
Abstract
Background: Short-term forecasts of infectious disease contribute to situational awareness and capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise forecasts' predictive performance by combining independent models into an ensemble. Here we report the performance of ensemble predictions of COVID-19 cases and deaths across Europe from March 2021 to March 2022. Methods: We created the European COVID-19 Forecast Hub, an online open-access platform where modellers upload weekly forecasts for 32 countries with results publicly visualised and evaluated. We created a weekly ensemble forecast from the equally-weighted average across individual models' predictive quantiles. We measured forecast accuracy using a baseline and relative Weighted Interval Score (rWIS). We retrospectively explored ensemble methods, including weighting by past performance. Results: We collected weekly forecasts from 48 models, of which we evaluated 29 models alongside the ensemble model. The ensemble had a consistently strong performance across countries over time, performing better on rWIS than 91% of forecasts for deaths (N=763 predictions from 20 models), and 83% forecasts for cases (N=886 predictions from 23 models). Performance remained stable over a 4-week horizon for death forecasts but declined with longer horizons for cases. Among ensemble methods, the most influential choice came from using a median average instead of the mean, regardless of weighting component models. Conclusions: Our results support combining independent models into an ensemble forecast to improve epidemiological predictions, and suggest that median averages yield better performance than methods based on means. We highlight that forecast consumers should place more weight on incident death forecasts than case forecasts at horizons greater than two weeks. Funding: European Commission, Ministerio de Ciencia, Innovación y Universidades, FEDER; Agència de Qualitat i Avaluació Sanitàries de Catalunya; Netzwerk Universitätsmedizin; Health Protection Research Unit; Wellcome Trust; European Centre for Disease Prevention and Control; Ministry of Science and Higher Education of Poland; Federal Ministry of Education and Research; Los Alamos National Laboratory; German Free State of Saxony; NCBiR; FISR 2020 Covid-19 I Fase; Spanish Ministry of Health / REACT-UE (FEDER); National Institutes of General Medical Sciences; Ministerio de Sanidad/ISCIII; PERISCOPE European H2020; PERISCOPE European H2021; InPresa; National Institutes of Health, NSF, US Centers for Disease Control and Prevention, Google, University of Virginia, Defense Threat Reduction Agency.
Collapse
Affiliation(s)
- Katharine Sherratt
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | - Hugo Gruson
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | - Rok Grah
- European Centre for Disease Prevention and Control, Stockholm, Sweden
| | - Helen Johnson
- European Centre for Disease Prevention and Control, Stockholm, Sweden
| | - Rene Niehus
- European Centre for Disease Prevention and Control, Stockholm, Sweden
| | - Bastian Prasse
- European Centre for Disease Prevention and Control, Stockholm, Sweden
| | - Frank Sandmann
- European Centre for Disease Prevention and Control, Stockholm, Sweden
| | | | | | - Sam Abbott
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | | | - Graham Gibson
- University of Massachusetts Amherst, Amherst, United States
| | - Evan L Ray
- University of Massachusetts Amherst, Amherst, United States
| | | | - Daniel Sheldon
- University of Massachusetts Amherst, Amherst, United States
| | - Yijin Wang
- University of Massachusetts Amherst, Amherst, United States
| | | | - Lijing Wang
- Boston Children's Hospital, Boston, United States
| | - Jan Trnka
- Department of Biochemistry, Cell and Molecular Biology, Charles University, Prague, Czech Republic
| | | | - Tao Sun
- École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Dorina Thanou
- École Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | | | | | | | | | - Neele Leithäuser
- Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
| | - Jan Mohring
- Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
| | - Johanna Schneider
- Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
| | - Jaroslaw Włazło
- Fraunhofer Institute for Industrial Mathematics, Kaiserslautern, Germany
| | | | - Berit Lange
- Helmholtz Centre for Infection Research, Braunschweig, Germany
| | - Isti Rodiah
- Helmholtz Centre for Infection Research, Braunschweig, Germany
| | | | | | | | | | | | | | - Inmaculada Villanueva
- Institut d'Investigacions Biomediques August Pi i Sunyer, Universitat Pompeu Fabra, Barcelona, Spain
| | - Vit Tucek
- Institute of Computer Science, Prague, Czech Republic
| | - Martin Smid
- Institute of Information Theory and Automation, Prague, Czech Republic
| | - Milan Zajíček
- Institute of Information Theory and Automation, Prague, Czech Republic
| | | | | | - Nikos I Bosse
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | - Sophie R Meakin
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| | - Lauren Castro
- Los Alamos National Laboratory, Los Alamos, United States
| | | | - Isaac Michaud
- Los Alamos National Laboratory, Los Alamos, United States
| | - Dave Osthus
- Los Alamos National Laboratory, Los Alamos, United States
| | | | | | | | | | | | | | | | | | - Soni Saksham
- Massachusetts Institute of Technology, Cambridge, United States
| | - Jonas Dehning
- Max-Planck-Institut fur Dynamik und Selbstorganisation, Göttingen, Germany
| | - Sebastian Mohr
- Max-Planck-Institut fur Dynamik und Selbstorganisation, Göttingen, Germany
| | - Viola Priesemann
- MPRG Priesemann, Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
| | | | | | | | | | | | - Wolfgang Bock
- Technical University of Kaiserlautern, Kaiserslautern, Germany
| | | | - Thomas Hotz
- Technische Universitat Ilmenau, Ilmenau, Germany
| | | | | | - Jose L Aznarte
- Universidad Nacional de Educacion a Distancia, Madrid, Spain
| | | | - Sergio Alonso
- Universitat Politecnica de Catalunya, Barcelona, Spain
| | - Enric Álvarez
- Universitat Politecnica de Catalunya, Barcelona, Spain
| | - Daniel López
- Universitat Politecnica de Catalunya, Barcelona, Spain
| | - Clara Prats
- Universitat Politecnica de Catalunya, Barcelona, Spain
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Benjamin Hurt
- University of Virginia, Charlottesville, United States
| | - Bryan Lewis
- University of Virginia, Charlottesville, United States
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | - Marcin Bodych
- Wroclaw University of Science and Technology, Wroclaw, Poland
| | - Maciej Filinski
- Wroclaw University of Science and Technology, Wroclaw, Poland
| | | | - Tyll Krueger
- Wroclaw University of Science and Technology, Wroclaw, Poland
| | - Tomasz Ozanski
- Wroclaw University of Science and Technology, Wroclaw, Poland
| | | | - Sebastian Funk
- Centre for the Mathematical Modelling of Infectious Diseases, London School of Hygiene and Tropical Medicine, London, United Kingdom
| |
Collapse
|
4
|
Parolini N, Dede’ L, Antonietti PF, Ardenghi G, Manzoni A, Miglio E, Pugliese A, Verani M, Quarteroni A. SUIHTER: a new mathematical model for COVID-19. Application to the analysis of the second epidemic outbreak in Italy. Proc Math Phys Eng Sci 2021; 477:20210027. [PMID: 35153578 PMCID: PMC8441130 DOI: 10.1098/rspa.2021.0027] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Accepted: 08/24/2021] [Indexed: 11/30/2022] Open
Abstract
The COVID-19 epidemic is the latest in a long list of pandemics that have affected humankind in the last century. In this paper, we propose a novel mathematical epidemiological model named SUIHTER from the names of the seven compartments that it comprises: susceptible uninfected individuals (S), undetected (both asymptomatic and symptomatic) infected (U), isolated infected (I), hospitalized (H), threatened (T), extinct (E) and recovered (R). A suitable parameter calibration that is based on the combined use of the least-squares method and the Markov chain Monte Carlo method is proposed with the aim of reproducing the past history of the epidemic in Italy, which surfaced in late February and is still ongoing to date, and of validating SUIHTER in terms of its predicting capabilities. A distinctive feature of the new model is that it allows a one-to-one calibration strategy between the model compartments and the data that are made available daily by the Italian Civil Protection Department. The new model is then applied to the analysis of the Italian epidemic with emphasis on the second outbreak, which emerged in autumn 2020. In particular, we show that the epidemiological model SUIHTER can be suitably used in a predictive manner to perform scenario analysis at a national level.
Collapse
Affiliation(s)
- N. Parolini
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - L. Dede’
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - P. F. Antonietti
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - G. Ardenghi
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - A. Manzoni
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - E. Miglio
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - A. Pugliese
- Department of Mathematics, University of Trento, Trento, Italy
| | - M. Verani
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
| | - A. Quarteroni
- MOX, Department of Mathematics, Politecnico di Milano, Milano, Italy
- Institute of Mathematics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| |
Collapse
|