Estimating the cumulative incidence of SARS-CoV-2 with imperfect serological tests: Exploiting cutoff-free approaches

A unified framework for diagnostic test development and evaluation during outbreaks of emerging infections

Evaluating diagnostic test accuracy during epidemics is difficult due to an urgent need for test availability, changing disease prevalence and pathogen characteristics, and constantly evolving testing aims and applications. Based on lessons learned during the SARS-CoV-2 pandemic, we introduce a framework for rapid diagnostic test development, evaluation, and validation during outbreaks of emerging infections. The framework is based on the feedback loop between test accuracy evaluation, modelling studies for public health decision-making, and impact of public health interventions. We suggest that building on this feedback loop can help future diagnostic test evaluation platforms better address the requirements of both patient care and public health.

Serodynamics: A primer and synthetic review of methods for epidemiological inference using serological data

We present a review and primer of methods to understand epidemiological dynamics and identify past exposures from serological data, referred to as serodynamics. We discuss processing and interpreting serological data prior to fitting serodynamical models, and review approaches for estimating epidemiological trends and past exposures, ranging from serocatalytic models applied to binary serostatus data, to more complex models incorporating quantitative antibody measurements and immunological understanding. Although these methods are seemingly disparate, we demonstrate how they are derived within a common mathematical framework. Finally, we discuss key areas for methodological development to improve scientific discovery and public health insights in seroepidemiology.

Correcting for Antibody Waning in Cumulative Incidence Estimation From Sequential Serosurveys

Serosurveys are a widely used tool to estimate the cumulative incidence-the fraction of a population that has been infected by a given pathogen. These surveys rely on serological assays that measure the level of pathogen-specific antibodies. Because antibody levels are waning, the fraction of previously infected individuals that have seroreverted increases with time past infection. To avoid underestimating the true cumulative incidence, it is therefore essential to correct for waning antibody levels. We present an empirically supported approach for seroreversion correction in cumulative incidence estimation when sequential serosurveys are conducted in the context of a newly emerging infectious disease. The correction is based on the observed dynamics of antibody titers in seropositive cases and validated using several in silico test scenarios. Furthermore, through this approach we revise a previous cumulative incidence estimate relying on the assumption of an exponentially declining probability of seroreversion over time, of severe acute respiratory syndrome coronavirus 2, of 76% in Manaus, Brazil, by October 2020 to 47.6% (95% confidence region: 43.5-53.5). This estimate has implications, for example, for the proximity to herd immunity in Manaus in late 2020.

Estimating SARS-CoV-2 infection probabilities with serological data and a Bayesian mixture model

The individual results of SARS-CoV-2 serological tests measured after the first pandemic wave of 2020 cannot be directly interpreted as a probability of having been infected. Plus, these results are usually returned as a binary or ternary variable, relying on predefined cut-offs. We propose a Bayesian mixture model to estimate individual infection probabilities, based on 81,797 continuous anti-spike IgG tests from Euroimmun collected in France after the first wave. This approach used serological results as a continuous variable, and was therefore not based on diagnostic cut-offs. Cumulative incidence, which is necessary to compute infection probabilities, was estimated according to age and administrative region. In France, we found that a "negative" or a "positive" test, as classified by the manufacturer, could correspond to a probability of infection as high as 61.8% or as low as 67.7%, respectively. "Indeterminate" tests encompassed probabilities of infection ranging from 10.8 to 96.6%. Our model estimated tailored individual probabilities of SARS-CoV-2 infection based on age, region, and serological result. It can be applied in other contexts, if estimates of cumulative incidence are available.

A Bayesian approach to estimating COVID-19 incidence and infection fatality rates

Naive estimates of incidence and infection fatality rates (IFR) of coronavirus disease 2019 suffer from a variety of biases, many of which relate to preferential testing. This has motivated epidemiologists from around the globe to conduct serosurveys that measure the immunity of individuals by testing for the presence of SARS-CoV-2 antibodies in the blood. These quantitative measures (titer values) are then used as a proxy for previous or current infection. However, statistical methods that use this data to its full potential have yet to be developed. Previous researchers have discretized these continuous values, discarding potentially useful information. In this article, we demonstrate how multivariate mixture models can be used in combination with post-stratification to estimate cumulative incidence and IFR in an approximate Bayesian framework without discretization. In doing so, we account for uncertainty from both the estimated number of infections and incomplete deaths data to provide estimates of IFR. This method is demonstrated using data from the Action to Beat Coronavirus erosurvey in Canada.

Estimation of the infection attack rate of mumps in an outbreak among college students using paired serology

Mumps virus is a highly transmissible pathogen that is effectively controlled in countries with high vaccination coverage. Nevertheless, outbreaks have occurred worldwide over the past decades in vaccinated populations. Here we analyse an outbreak of mumps virus genotype G among college students in the Netherlands over the period 2009-2012 using paired serological data. To identify infections in the presence of preexisting antibodies we compared mumps specific serum IgG concentrations in two consecutive samples (n=746), whereby the first sample was taken when students started their study prior to the outbreaks, and the second sample was taken 2-5 years later. We fit a binary mixture model to the data. The two mixing distributions represent uninfected and infected classes. Throughout we assume that the infection probability increases with the ratio of antibody concentrations of the second to first sample. The estimated infection attack rate in this study is higher than reported earlier (0.095 versus 0.042). The analyses yield probabilistic classifications of participants, which are mostly quite precise owing to the high intraclass correlation of samples in uninfected participants (0.85, 95%CrI: 0.82-0.87). The estimated probability of infection increases with decreasing antibody concentration in the pre-outbreak sample, such that the probability of infection is 0.12 (95%CrI: 0.10-0.13) for the lowest quartile of the pre-outbreak samples and 0.056 (95%CrI: 0.044-0.068) for the highest quartile. We discuss the implications of these insights for the design of booster vaccination strategies.

Estimating cutoff values for diagnostic tests to achieve target specificity using extreme value theory

BACKGROUND

Rapidly developing tests for emerging diseases is critical for early disease monitoring. In the early stages of an epidemic, when low prevalences are expected, high specificity tests are desired to avoid numerous false positives. Selecting a cutoff to classify positive and negative test results that has the desired operating characteristics, such as specificity, is challenging for new tests because of limited validation data with known disease status. While there is ample statistical literature on estimating quantiles of a distribution, there is limited evidence on estimating extreme quantiles from limited validation data and the resulting test characteristics in the disease testing context.

METHODS

We propose using extreme value theory to select a cutoff with predetermined specificity by fitting a Pareto distribution to the upper tail of the negative controls. We compared this method to five previously proposed cutoff selection methods in a data analysis and simulation study. We analyzed COVID-19 enzyme linked immunosorbent assay antibody test results from long-term care facilities and skilled nursing staff in Colorado between May and December of 2020.

RESULTS

We found the extreme value approach had minimal bias when targeting a specificity of 0.995. Using the empirical quantile of the negative controls performed well when targeting a specificity of 0.95. The higher target specificity is preferred for overall test accuracy when prevalence is low, whereas the lower target specificity is preferred when prevalence is higher and resulted in less variable prevalence estimation.

DISCUSSION

While commonly used, the normal based methods showed considerable bias compared to the empirical and extreme value theory-based methods.

CONCLUSIONS

When determining disease testing cutoffs from small training data samples, we recommend using the extreme value based-methods when targeting a high specificity and the empirical quantile when targeting a lower specificity.

Estimating SARS-CoV-2 seroprevalence

Governments and public health authorities use seroprevalence studies to guide responses to the COVID-19 pandemic. Seroprevalence surveys estimate the proportion of individuals who have detectable SARS-CoV-2 antibodies. However, serologic assays are prone to misclassification error, and non-probability sampling may induce selection bias. In this paper, non-parametric and parametric seroprevalence estimators are considered that address both challenges by leveraging validation data and assuming equal probabilities of sample inclusion within covariate-defined strata. Both estimators are shown to be consistent and asymptotically normal, and consistent variance estimators are derived. Simulation studies are presented comparing the estimators over a range of scenarios. The methods are used to estimate severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) seroprevalence in New York City, Belgium, and North Carolina.

A Mixture Model for Estimating SARS-CoV-2 Seroprevalence in Chennai, India

Serological assays used to estimate the prevalence of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) often rely on manufacturers' cutoffs established on the basis of severe cases. We conducted a household-based serosurvey of 4,677 individuals in Chennai, India, from January to May 2021. Samples were tested for SARS-CoV-2 immunoglobulin G (IgG) antibodies to the spike (S) and nucleocapsid (N) proteins. We calculated seroprevalence, defining seropositivity using manufacturer cutoffs and using a mixture model based on measured IgG level. Using manufacturer cutoffs, there was a 5-fold difference in seroprevalence estimated by each assay. This difference was largely reconciled using the mixture model, with estimated anti-S and anti-N IgG seroprevalence of 64.9% (95% credible interval (CrI): 63.8, 66.0) and 51.5% (95% CrI: 50.2, 52.9), respectively. Age and socioeconomic factors showed inconsistent relationships with anti-S and anti-N IgG seropositivity using manufacturer cutoffs. In the mixture model, age was not associated with seropositivity, and improved household ventilation was associated with lower seropositivity odds. With global vaccine scale-up, the utility of the more stable anti-S IgG assay may be limited due to the inclusion of the S protein in several vaccines. Estimates of SARS-CoV-2 seroprevalence using alternative targets must consider heterogeneity in seroresponse to ensure that seroprevalence is not underestimated and correlates are not misinterpreted.

Serological testing of blood donors to characterise the impact of COVID-19 in Melbourne, Australia, 2020

Rapidly identifying and isolating people with acute SARS-CoV-2 infection has been a core strategy to contain COVID-19 in Australia, but a proportion of infections go undetected. We estimated SARS-CoV-2 specific antibody prevalence (seroprevalence) among blood donors in metropolitan Melbourne following a COVID-19 outbreak in the city between June and September 2020. The aim was to determine the extent of infection spread and whether seroprevalence varied demographically in proportion to reported cases of infection. The design involved stratified sampling of residual specimens from blood donors (aged 20-69 years) in three postcode groups defined by low (<3 cases/1,000 population), medium (3-7 cases/1,000 population) and high (>7 cases/1,000 population) COVID-19 incidence based on case notification data. All specimens were tested using the Wantai SARS-CoV-2 total antibody assay. Seroprevalence was estimated with adjustment for test sensitivity and specificity for the Melbourne metropolitan blood donor and residential populations, using multilevel regression and poststratification. Overall, 4,799 specimens were collected between 23 November and 17 December 2020. Seroprevalence for blood donors was 0.87% (90% credible interval: 0.25-1.49%). The highest estimates, of 1.13% (0.25-2.15%) and 1.11% (0.28-1.95%), respectively, were observed among donors living in the lowest socioeconomic areas (Quintiles 1 and 2) and lowest at 0.69% (0.14-1.39%) among donors living in the highest socioeconomic areas (Quintile 5). When extrapolated to the Melbourne residential population, overall seroprevalence was 0.90% (0.26-1.51%), with estimates by demography groups similar to those for the blood donors. The results suggest a lack of extensive community transmission and good COVID-19 case ascertainment based on routine testing during Victoria's second epidemic wave. Residual blood donor samples provide a practical epidemiological tool for estimating seroprevalence and information on population patterns of infection, against which the effectiveness of ongoing responses to the pandemic can be assessed.

Applying mixture model methods to SARS-CoV-2 serosurvey data from Geneva

Serosurveys are an important tool to estimate the true extent of the current SARS-CoV-2 pandemic. So far, most serosurvey data have been analyzed with cutoff-based methods, which dichotomize individual measurements into sero-positives or negatives based on a predefined cutoff. However, mixture model methods can gain additional information from the same serosurvey data. Such methods refrain from dichotomizing individual values and instead use the full distribution of the serological measurements from pre-pandemic and COVID-19 controls to estimate the cumulative incidence. This study presents an application of mixture model methods to SARS-CoV-2 serosurvey data from the SEROCoV-POP study from April and May 2020 in Geneva (2766 individuals). Besides estimating the total cumulative incidence in these data (8.1% (95% CI: 6.8%–9.9%)), we applied extended mixture model methods to estimate an indirect indicator of disease severity, which is the fraction of cases with a distribution of antibody levels similar to hospitalized COVID-19 patients. This fraction is 51.2% (95% CI: 15.2%–79.5%) across the full serosurvey, but differs between three age classes: 21.4% (95% CI: 0%–59.6%) for individuals between 5 and 40 years old, 60.2% (95% CI: 21.5%–100%) for individuals between 41 and 65 years old and 100% (95% CI: 20.1%–100%) for individuals between 66 and 90 years old. Additionally, we find a mismatch between the inferred negative distribution of the serosurvey and the validation data of pre-pandemic controls. Overall, this study illustrates that mixture model methods can provide additional insights from serosurvey data.

Quantifying previous SARS-CoV-2 infection through mixture modelling of antibody levels

As countries decide on vaccination strategies and how to ease movement restrictions, estimating the proportion of the population previously infected with SARS-CoV-2 is important for predicting the future burden of COVID-19. This proportion is usually estimated from serosurvey data in two steps: first the proportion above a threshold antibody level is calculated, then the crude estimate is adjusted using external estimates of sensitivity and specificity. A drawback of this approach is that the PCR-confirmed cases used to estimate the sensitivity of the threshold may not be representative of cases in the wider population-e.g., they may be more recently infected and more severely symptomatic. Mixture modelling offers an alternative approach that does not require external data from PCR-confirmed cases. Here we illustrate the bias in the standard threshold-based approach by comparing both approaches using data from several Kenyan serosurveys. We show that the mixture model analysis produces estimates of previous infection that are often substantially higher than the standard threshold analysis.

Herpesviruses Serology Distinguishes Different Subgroups of Patients From the United Kingdom Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Biobank

The evidence of an association between Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS) and chronic herpesviruses infections remains inconclusive. Two reasons for the lack of consistent evidence are the large heterogeneity of the patients' population with different disease triggers and the use of arbitrary cutoffs for defining seropositivity. In this work we re-analyzed previously published serological data related to 7 herpesvirus antigens. Patients with ME/CFS were subdivided into four subgroups related to the disease triggers: S0-42 patients who did not know their disease trigger; S1-43 patients who reported a non-infection trigger; S2-93 patients who reported an infection trigger, but that infection was not confirmed by a lab test; and S3-48 patients who reported an infection trigger and that infection was confirmed by a lab test. In accordance with a sensitivity analysis, the data were compared to those from 99 healthy controls allowing the seropositivity cutoffs to vary within a wide range of possible values. We found a negative association between S1 and seropositivity to Epstein-Barr virus (VCA and EBNA1 antigens) and Varicella-Zoster virus using specific seropositivity cutoff. However, this association was not significant when controlling for multiple testing. We also found that S3 had a lower seroprevalence to the human cytomegalovirus when compared to healthy controls for all cutoffs used for seropositivity and after adjusting for multiple testing using the Benjamini-Hochberg procedure. However, this association did not reach statistical significance when using Benjamini-Yekutieli procedure. In summary, herpesviruses serology could distinguish subgroups of ME/CFS patients according to their disease trigger, but this finding could be eventually affected by the problem of multiple testing.