An expert judgment model to predict early stages of the COVID-19 pandemic in the United States

Chimeric Forecasting: An experiment to leverage human judgment to improve forecasts of infectious disease using simulated surveillance data

Forecasts of infectious agents provide public health officials advanced warning about the intensity and timing of the spread of disease. Past work has found that accuracy and calibration of forecasts is weakest when attempting to predict an epidemic peak. Forecasts from a mechanistic model would be improved if there existed accurate information about the timing and intensity of an epidemic. We presented 3000 humans with simulated surveillance data about the number of incident hospitalizations from a current and two past seasons, and asked that they predict the peak time and intensity of the underlying epidemic. We found that in comparison to two control models, a model including human judgment produced more accurate forecasts of peak time and intensity of hospitalizations during an epidemic. Chimeric models have the potential to improve our ability to predict targets of public health interest which may in turn reduce infectious disease burden.

Human judgement forecasting of COVID-19 in the UK

Background

In the past, two studies found ensembles of human judgement forecasts of COVID-19 to show predictive performance comparable to ensembles of computational models, at least when predicting case incidences. We present a follow-up to a study conducted in Germany and Poland and investigate a novel joint approach to combine human judgement and epidemiological modelling.

Methods

From May 24th to August 16th 2021, we elicited weekly one to four week ahead forecasts of cases and deaths from COVID-19 in the UK from a crowd of human forecasters. A median ensemble of all forecasts was submitted to the European Forecast Hub. Participants could use two distinct interfaces: in one, forecasters submitted a predictive distribution directly, in the other forecasters instead submitted a forecast of the effective reproduction number R t. This was then used to forecast cases and deaths using simulation methods from the EpiNow2 R package. Forecasts were scored using the weighted interval score on the original forecasts, as well as after applying the natural logarithm to both forecasts and observations.

Results

The ensemble of human forecasters overall performed comparably to the official European Forecast Hub ensemble on both cases and deaths, although results were sensitive to changes in details of the evaluation. R t forecasts performed comparably to direct forecasts on cases, but worse on deaths. Self-identified "experts" tended to be better calibrated than "non-experts" for cases, but not for deaths.

Conclusions

Human judgement forecasts and computational models can produce forecasts of similar quality for infectious disease such as COVID-19. The results of forecast evaluations can change depending on what metrics are chosen and judgement on what does or doesn't constitute a "good" forecast is dependent on the forecast consumer. Combinations of human and computational forecasts hold potential but present real-world challenges that need to be solved.

When expert predictions fail

We examine the opportunities and challenges of expert judgment in the social sciences, scrutinizing the way social scientists make predictions. While social scientists show above-chance accuracy in predicting laboratory-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are oversimplified, confuse levels of analysis to which a model applies, misalign the nature of the model with the nature of the phenomena, and fail to consider factors beyond the scientist's pet theory. Taking cues from physical sciences and meteorology, we advocate an approach that integrates broad foundational models with context-specific time series data. We call for a shift in the social sciences towards more precise, daring predictions and greater intellectual humility.

Disentangling the role of virus infectiousness and awareness-based human behavior during the early phase of the COVID-19 pandemic in the European Union

In this work, we manage to disentangle the role of virus infectiousness and awareness-based human behavior in the COVID-19 pandemic. Using Bayesian inference, we quantify the uncertainty of a state-space model whose propagator is based on an unusual SEIR-type model since it incorporates the effective population fraction as a parameter. Within the Markov Chain Monte Carlo (MCMC) algorithm, Unscented Kalman Filter (UKF) may be used to evaluate the likelihood approximately. UKF is a suitable strategy in many cases, but it is not well-suited to deal with non-negativity restrictions on the state variables. To overcome this difficulty, we modify the UKF, conveniently truncating Gaussian distributions, which allows us to deal with such restrictions. We use official infection notification records to analyze the first 22 weeks of infection spread in each of the 27 countries of the European Union (EU). It is known that such records are the primary source of information to assess the early evolution of the pandemic and, at the same time, usually suffer underreporting and backlogs. Our model explicitly accounts for uncertainty in the dynamic model parameters, the dynamic model adequacy, and the infection observation process. We argue that this modeling paradigm allows us to disentangle the role of the contact rate, the effective population fraction, and the infection observation probability across time and space with an imperfect first principles model. Our findings agree with phylogenetic evidence showing little variability in the contact rate, or virus infectiousness, across EU countries during the early phase of the pandemic, highlighting the advantage of incorporating the effective population fraction into pandemic modeling for heterogeneity in both human behavior and reporting. Finally, to evaluate the consistency of our data assimilation method, we performed a forecast that adequately fits the actual data.

Statement of significance

Data-driven and model-based epidemiological studies aimed at learning the number of people infected early during a pandemic should explicitly consider the behavior-induced effective population effect. Indeed, the non-isolated, or effective, fraction of the population during the early phase of the pandemic is time-varying, and first-principles modeling with quantified uncertainty is imperative for an adequate analysis across time and space. We argue that, although good inference results may be obtained using the classical SEIR type model, the model posed in this work has allowed us to disentangle the role of virus infectiousness and awareness-based human behavior during the early phase of the COVID-19 pandemic in the European Union from official infection notification records.

Unraveling the COVID-19 hospitalization dynamics in Spain using Bayesian inference

BACKGROUND

One of the main challenges of the COVID-19 pandemic is to make sense of available, but often heterogeneous and noisy data. This contribution presents a data-driven methodology that allows exploring the hospitalization dynamics of COVID-19, exemplified with a study of 17 autonomous regions in Spain from summer 2020 to summer 2021.

METHODS

We use data on new daily cases and hospitalizations reported by the Spanish Ministry of Health to implement a Bayesian inference method that allows making short-term predictions of bed occupancy of COVID-19 patients in each of the autonomous regions of the country.

RESULTS

We show how to use the temporal series for the number of daily admissions and discharges from hospital to reproduce the hospitalization dynamics of COVID-19 patients. For the case-study of the region of Aragon, we estimate that the probability of being admitted to hospital care upon infection is 0.090 [0.086-0.094], (95% C.I.), with the distribution governing hospital admission yielding a median interval of 3.5 days and an IQR of 7 days. Likewise, the distribution on the length of stay produces estimates of 12 days for the median and 10 days for the IQR. A comparison between model parameters for the regions analyzed allows to detect differences and changes in policies of the health authorities.

CONCLUSIONS

We observe important regional differences, signaling that to properly compare very different populations, it is paramount to acknowledge all the diversity in terms of culture, socio-economic status, and resource availability. To better understand the impact of this pandemic, much more data, disaggregated and properly annotated, should be made available.

Chimeric forecasting: combining probabilistic predictions from computational models and human judgment

Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble-a combination of computational and human judgment forecasts-as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.