The wisdom of crowds in action: Forecasting epidemic diseases with a web-based prediction market system

Analytical reference framework to analyze non-COVID-19 events

BACKGROUND

The COVID-19 pandemic has disrupted the healthcare system, leading to delays in detection of other non-COVID-19 diseases. This paper presents ANE Framework (Analytics for Non-COVID-19 Events), a reliable and user-friendly analytical forecasting framework designed to predict the number of patients with non-COVID-19 diseases. Prior to 2020, there were analytical models focused on specific illnesses and contexts. Then, most models have focused on understanding COVID-19 behavior. There is a lack of analytical frameworks that enable disease forecasting for non-COVID-19 diseases.

METHODS

The ANE Framework utilizes time series analysis to generate forecasting models. The framework leverages daily data from official government sources and employs SARIMA models to forecast the number of non-COVID-19 cases, such as tuberculosis and suicide attempts.

RESULTS

The framework was tested on five different non-COVID-19 events. The framework performs well across all events, including tuberculosis and suicide attempts, with a Mean Absolute Percentage Error (MAPE) of up to 20% and the consistency remains independent of the behavior of each event. Moreover, a pairwise comparison of averages can lead to over or underestimation of the impact. The disruption caused by the pandemic resulted in a 17% gap (2383 cases) between expected and reported tuberculosis cases, and a 19% gap (2464 cases) for suicide attempts. These gaps varied between 20 and 64% across different cities and regions. The ANE Framework has proven to be reliable for analyzing several diseases and exhibits the flexibility to incorporate new data from various sources. Regular updates and the inclusion of new associated data enhance the framework's effectiveness.

CONCLUSIONS

Current pandemic shows the necessity of developing flexible models to be adapted to different illness data. The framework developed proved to be reliable for the different diseases analyzed, presenting enough flexibility to update with new data or even include new data from different databases. To keep updated on the result of the project allows the inclusion of new data associated with it. Similarly, the proposed strategy in the ANE framework allows for improving the quality of the obtained results with news events.

Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID events

BACKGROUND

The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning.

METHODS

We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised.

FINDINGS

A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (pboth <1 × 10-9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10-14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10-7.

INTERPRETATION

Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks.

FUNDING

This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.

Chimeric forecasting: combining probabilistic predictions from computational models and human judgment

Forecasts of the trajectory of an infectious agent can help guide public health decision making. A traditional approach to forecasting fits a computational model to structured data and generates a predictive distribution. However, human judgment has access to the same data as computational models plus experience, intuition, and subjective data. We propose a chimeric ensemble-a combination of computational and human judgment forecasts-as a novel approach to predicting the trajectory of an infectious agent. Each month from January, 2021 to June, 2021 we asked two generalist crowds, using the same criteria as the COVID-19 Forecast Hub, to submit a predictive distribution over incident cases and deaths at the US national level either two or three weeks into the future and combined these human judgment forecasts with forecasts from computational models submitted to the COVID-19 Forecasthub into a chimeric ensemble. We find a chimeric ensemble compared to an ensemble including only computational models improves predictions of incident cases and shows similar performance for predictions of incident deaths. A chimeric ensemble is a flexible, supportive public health tool and shows promising results for predictions of the spread of an infectious agent.

STAMINA: Bioinformatics Platform for Monitoring and Mitigating Pandemic Outbreaks

This paper presents the components and integrated outcome of a system that aims to achieve early detection, monitoring and mitigation of pandemic outbreaks. The architecture of the platform aims at providing a number of pandemic-response-related services, on a modular basis, that allows for the easy customization of the platform to address user’s needs per case. This customization is achieved through its ability to deploy only the necessary, loosely coupled services and tools for each case, and by providing a common authentication, data storage and data exchange infrastructure. This way, the platform can provide the necessary services without the burden of additional services that are not of use in the current deployment (e.g., predictive models for pathogens that are not endemic to the deployment area). All the decisions taken for the communication and integration of the tools that compose the platform adhere to this basic principle. The tools presented here as well as their integration is part of the project STAMINA.

Forecasts for the Attainment of Major Research Milestones in Parkinson's Disease

BACKGROUND

Projections about when research milestones will be attained are often of interest to patients and can help inform decisions about research funding and health system planning.

OBJECTIVE

To collect aggregated expert forecasts on the attainment of 11 major research milestones in Parkinson's disease (PD).

METHODS

Experts were asked to provide predictions about the attainment of 11 milestones in PD research in an online survey. PD experts were identified from: 1) The Michael J. Fox Foundation for Parkinson's Research data base, 2) doctors specializing in PD at top ranked neurology centers in the US and Canada, and 3) corresponding authors of articles on PD in top medical journals. Judgments were aggregated using coherence weighting. We tested the relationship between demographic variables and individual judgments using a linear regression.

RESULTS

249 PD experts completed the survey. In the aggregate, experts believed that new treatments like gene therapy for monogenic PD, immunotherapy and cell therapy had 56.1%, 59.7%, and 66.6% probability, respectively of progressing in the clinical approval process within the next 10 years. Milestones involving existing management approaches, like the approval of a deep brain stimulation device or a body worn sensor had 78.4% and 82.2% probability of occurring within the next 10 years. Demographic factors were unable to explain deviations from the aggregate forecast (R2 = 0.029).

CONCLUSIONS

Aggregated expert opinion suggests that milestones for the advancement of new treatment options for PD are still many years away. However, other improvements in PD diagnosis and management are believed to be near at hand.

Diagnosing Coronavirus Disease 2019 (COVID-19): Efficient Harris Hawks-Inspired Fuzzy K-Nearest Neighbor Prediction Methods

This study is devoted to proposing a useful intelligent prediction model to distinguish the severity of COVID-19, to provide a more fair and reasonable reference for assisting clinical diagnostic decision-making. Based on patients' necessary information, pre-existing diseases, symptoms, immune indexes, and complications, this article proposes a prediction model using the Harris hawks optimization (HHO) to optimize the Fuzzy K-nearest neighbor (FKNN), which is called HHO-FKNN. This model is utilized to distinguish the severity of COVID-19. In HHO-FKNN, the purpose of introducing HHO is to optimize the FKNN's optimal parameters and feature subsets simultaneously. Also, based on actual COVID-19 data, we conducted a comparative experiment between HHO-FKNN and several well-known machine learning algorithms, which result shows that not only the proposed HHO-FKNN can obtain better classification performance and higher stability on the four indexes but also screen out the key features that distinguish severe COVID-19 from mild COVID-19. Therefore, we can conclude that the proposed HHO-FKNN model is expected to become a useful tool for COVID-19 prediction.

Wisdom of the expert crowd prediction of response for 3 neurology randomized trials

OBJECTIVE

To explore the accuracy of combined neurology expert forecasts in predicting primary endpoints for trials.

METHODS

We identified one major randomized trial each in stroke, multiple sclerosis (MS), and amyotrophic lateral sclerosis (ALS) that was closing within 6 months. After recruiting a sample of neurology experts for each disease, we elicited forecasts for the primary endpoint outcomes in the trial placebo and treatment arms. Our main outcome was the accuracy of averaged predictions, measured using ordered Brier scores. Scores were compared against an algorithm that offered noncommittal predictions.

RESULTS

Seventy-one neurology experts participated. Combined forecasts of experts were less accurate than a noncommittal prediction algorithm for the stroke trial (pooled Brier score = 0.340, 95% subjective probability interval [sPI] 0.340 to 0.340 vs 0.185 for the uninformed prediction), and approximately as accurate for the MS study (pooled Brier score = 0.107, 95% confidence interval [CI] 0.081 to 0.133 vs 0.098 for the noncommittal prediction) and the ALS study (pooled Brier score = 0.090, 95% CI 0.081 to 0.185 vs 0.090). The 95% sPIs of individual predictions contained actual trial outcomes among 44% of experts. Only 18% showed prediction skill exceeding the noncommittal prediction. Independent experts and coinvestigators achieved similar levels of accuracy.

CONCLUSION

In this first-of-kind exploratory study, averaged expert judgments rarely outperformed noncommittal forecasts. However, experts at least anticipated the possibility of effects observed in trials. Our findings, if replicated in different trial samples, caution against the reliance on simple approaches for combining expert opinion in making research and policy decisions.

Syndromic surveillance using web data: a systematic review

During the recent years, a lot of debate is taken place about the evolution of Smart Healthcare systems. Particularly, how these systems can help people improve human conditions of health, by taking advantages of the new Information and Communication Technologies (ICT), regarding early prediction and efficient treatment. The purpose of this study is to provide a systematic review of the current literature available that focuses on information systems on syndromic surveillance using web data. All published items concern articles, books, reviews, reports, conference announcements, and dissertations. We used a variation of PRISMA Statements methodology to conduct a systematic review. The review identifies the relevant published papers from the year 2004 to 2018, systematically includes and explores them to extract similarities, gaps, and conclusions on the research that has been done so far. The results presented concern the year, the examined disease, the web data source, the geographic location/country, and the data analysis method used. The results show that influenza is the most examined infectious disease. The internet tools most used are Twitter and Google. Regarding the geographical areas explored in the published papers, the most examined country is the United States, since many scientists come from this country. There is a significant growth of articles since 2009. There are also various statistical methods used to correlate the data retrieved from the internet to the data from national authorities. The conclusion of all researches is that the Web can be a useful tool for the detection of serious epidemics and for a creation of a syndromic surveillance system using the Web, since we can predict epidemics from web data before they are officially detected in population. With the advance of ICT, Smart Healthcare can benefit from the monitoring of epidemics and the early prediction of such a system, improving national or international health strategies and policy decision. This can be achieved through the provision of new technology tools to enhance health monitoring systems toward the new innovations of Smart Health or eHealth, even with the emerging technologies of Internet of Things. The challenges and impacts of an electronic system based on internet data include the social, medical, and technological disciplines. These can be further extended to Smart Healthcare, as the data streaming can provide with real-time information, awareness on epidemics and alerts for both patients or medical scientists. Finally, these new systems can help improve the standards of human life.

Main factors influencing recovery in MERS Co-V patients using machine learning

BACKGROUND

Middle East Respiratory Syndrome (MERS) is a major infectious disease which has affected the Middle Eastern countries, especially the Kingdom of Saudi Arabia (KSA) since 2012. The high mortality rate associated with this disease has been a major cause of concern. This paper aims at identifying the major factors influencing MERS recovery in KSA.

METHODS

The data used for analysis was collected from the Ministry of Health website, KSA. The important factors impelling the recovery are found using machine learning. Machine learning models such as support vector machine, conditional inference tree, naïve Bayes and J48 are modelled to identify the important factors. Univariate and multivariate logistic regression analysis is also carried out to identify the significant factors statistically.

RESULT

The main factors influencing MERS recovery rate are identified as age, pre-existing diseases, severity of disease and whether the patient is a healthcare worker or not. In spite of MERS being a zoonotic disease, contact with camels is not a major factor influencing recovery.

CONCLUSION

The methods used were able to determine the prime factors influencing MERS recovery. It can be comprehended that awareness about symptoms and seeking medical intervention at the onset of development of symptoms will make a long way in reducing the mortality rate.

Forecasting infectious disease emergence subject to seasonal forcing

BACKGROUND

Despite high vaccination coverage, many childhood infections pose a growing threat to human populations. Accurate disease forecasting would be of tremendous value to public health. Forecasting disease emergence using early warning signals (EWS) is possible in non-seasonal models of infectious diseases. Here, we assessed whether EWS also anticipate disease emergence in seasonal models.

METHODS

We simulated the dynamics of an immunizing infectious pathogen approaching the tipping point to disease endemicity. To explore the effect of seasonality on the reliability of early warning statistics, we varied the amplitude of fluctuations around the average transmission. We proposed and analyzed two new early warning signals based on the wavelet spectrum. We measured the reliability of the early warning signals depending on the strength of their trend preceding the tipping point and then calculated the Area Under the Curve (AUC) statistic.

RESULTS

Early warning signals were reliable when disease transmission was subject to seasonal forcing. Wavelet-based early warning signals were as reliable as other conventional early warning signals. We found that removing seasonal trends, prior to analysis, did not improve early warning statistics uniformly.

CONCLUSIONS

Early warning signals anticipate the onset of critical transitions for infectious diseases which are subject to seasonal forcing. Wavelet-based early warning statistics can also be used to forecast infectious disease.

Utility and potential of rapid epidemic intelligence from internet-based sources

OBJECTIVES

Rapid epidemic detection is an important objective of surveillance to enable timely intervention, but traditional validated surveillance data may not be available in the required timeframe for acute epidemic control. Increasing volumes of data on the Internet have prompted interest in methods that could use unstructured sources to enhance traditional disease surveillance and gain rapid epidemic intelligence. We aimed to summarise Internet-based methods that use freely-accessible, unstructured data for epidemic surveillance and explore their timeliness and accuracy outcomes.

METHODS

Steps outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist were used to guide a systematic review of research related to the use of informal or unstructured data by Internet-based intelligence methods for surveillance.

RESULTS

We identified 84 articles published between 2006-2016 relating to Internet-based public health surveillance methods. Studies used search queries, social media posts and approaches derived from existing Internet-based systems for early epidemic alerts and real-time monitoring. Most studies noted improved timeliness compared to official reporting, such as in the 2014 Ebola epidemic where epidemic alerts were generated first from ProMED-mail. Internet-based methods showed variable correlation strength with official datasets, with some methods showing reasonable accuracy.

CONCLUSION

The proliferation of publicly available information on the Internet provided a new avenue for epidemic intelligence. Methodologies have been developed to collect Internet data and some systems are already used to enhance the timeliness of traditional surveillance systems. To improve the utility of Internet-based systems, the key attributes of timeliness and data accuracy should be included in future evaluations of surveillance systems.