Comparing Prevalence Estimates From Population-Based Surveys to Inform Surveillance Using Electronic Health Records

INTRODUCTION

Electronic health record (EHR) systems provide an opportunity to use a novel data source for population health surveillance. Validation studies that compare prevalence estimates from EHRs and surveys most often use difference testing, which can, because of large sample sizes, lead to detection of significant differences that are not meaningful. We explored a novel application of the two one-sided t test (TOST) to assess the equivalence of prevalence estimates in 2 population-based surveys to inform margin selection for validating EHR-based surveillance prevalence estimates derived from large samples.

METHODS

We compared prevalence estimates of health indicators in the 2013 Community Health Survey (CHS) and the 2013-2014 New York City Health and Nutrition Examination Survey (NYC HANES) by using TOST, a 2-tailed t test, and other goodness-of-fit measures.

RESULTS

A ±5 percentage-point equivalence margin for a TOST performed well for most health indicators. For health indicators with a prevalence estimate of less than 10% (extreme obesity [CHS, 3.5%; NYC HANES, 5.1%] and serious psychological distress [CHS, 5.2%; NYC HANES, 4.8%]), a ±2.5 percentage-point margin was more consistent with other goodness-of-fit measures than the larger percentage-point margins.

CONCLUSION

A TOST with a ±5 percentage-point margin was useful in establishing equivalence, but a ±2.5 percentage-point margin may be appropriate for health indicators with a prevalence estimate of less than 10%. Equivalence testing can guide future efforts to validate EHR data.

Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data

INTRODUCTION

Electronic health records (EHRs) offer potential for population health surveillance but EHR-based surveillance measures require validation prior to use. We assessed the validity of obesity, smoking, depression, and influenza vaccination indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This report is the second in a 3-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the infrastructure underlying the NYC Macroscope; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. This second report, which addresses concerns related to sampling bias and data quality, describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods described in this report to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

METHODS

NYC Macroscope prevalence estimates, overall and stratified by sex and age group, were compared to reference survey estimates for adult New Yorkers who reported visiting a doctor in the past year. Agreement was evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by examining individual EHR records in a subsample of 48 survey participants.

RESULTS

Among adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%) fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination prevalence were more than 10 percentage points lower than the estimates from either reference survey. While specificity was > 0.90 for both of these indicators, sensitivity was < 0.70.

DISCUSSION

Through this work we have demonstrated that EHR data from a convenience sample of providers can produce acceptable estimates of smoking and obesity prevalence among adult New Yorkers in care; gained a better understanding of the challenges involved in estimating depression prevalence from EHRs; and identified areas for additional research regarding estimation of influenza vaccination prevalence. We have also shared lessons learned about how EHR indicators should be constructed and offer methodologic suggestions for validating them.

CONCLUSIONS

This work adds to a rapidly emerging body of literature about how to define, collect and interpret EHR-based surveillance measures and may help guide other jurisdictions.

Design of the New York City Macroscope: Innovations in Population Health Surveillance Using Electronic Health Records

Introduction:

Electronic health records (EHRs) have the potential to offer real-time, inexpensive standardized health data about chronic health conditions. Despite rapid expansion, EHR data evaluations for chronic disease surveillance have been limited. We present design and methods for the New York City (NYC) Macroscope, an EHR-based chronic disease surveillance system.

This methods report is the first in a three part series describing the development and validation of the NYC Macroscope. This report describes in detail the infrastructure underlying the NYC Macroscope; indicator definitions; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. The second report describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

Methods:

We designed the NYC Macroscope for comparison to a local “gold standard,” the 2013–14 NYC Health and Nutrition Examination Survey, and the telephonic 2013 Community Health Survey. NYC Macroscope indicators covered prevalence, treatment, and control of diabetes, hypertension, and hyperlipidemia; and prevalence of influenza vaccination, obesity, depression and smoking. Indicators were stratified by age, sex, and neighborhood poverty, and weighted to the in-care NYC population and limited to primary care patients. Indicator queries were distributed to a virtual network of primary care practices; 392 practices and 716,076 adult patients were retained in the final sample.

Findings:

The NYC Macroscope covered 10% of primary care providers and 15% of all adult patients in NYC in 2013 (8–47% of patients by neighborhood). Data completeness varied by domain from 98% for blood pressure among patients with hypertension to 33% for depression screening.

Discussion:

Design and validation efforts undertaken by NYC are described here to provide one potential blueprint for leveraging EHRs for population health monitoring. To replicate a model like NYC Macroscope, jurisdictions should establish buy-in; build informatics capacity; use standard, simple case defnitions; establish documentation quality thresholds; restrict to primary care providers; and weight the sample to a target population.

Characterizing Adults Receiving Primary Medical Care in New York City: Implications for Using Electronic Health Records for Chronic Disease Surveillance

Introduction

Electronic health records (EHRs) from primary care providers can be used for chronic disease surveillance; however, EHR-based prevalence estimates may be biased toward people who seek care. This study sought to describe the characteristics of an in-care population and compare them with those of a not-in-care population to inform interpretation of EHR data.

Methods

We used data from the 2013–2014 New York City Health and Nutrition Examination Survey (NYC HANES), considered the gold standard for estimating disease prevalence, and the 2013 Community Health Survey, and classified participants as in care or not in care, on the basis of their report of seeing a health care provider in the previous year. We used χ² tests to compare the distribution of demographic characteristics, health care coverage and access, and chronic conditions between the 2 populations.

Results

According to the Community Health Survey, approximately 4.1 million (71.7%) adults aged 20 or older had seen a health care provider in the previous year; according to NYC HANES, approximately 4.7 million (75.1%) had. In both surveys, the in-care population was more likely to be older, female, non-Hispanic, and insured compared with the not-in-care population. The in-care population from the NYC HANES also had a higher prevalence of diabetes (16.7% vs 6.9%; P < .001), hypercholesterolemia (35.7% vs 22.3%; P < .001), and hypertension (35.5% vs 26.4%; P < .001) than the not-in-care population.

Conclusion

Systematic differences between in-care and not-in-care populations warrant caution in using primary care data to generalize to the population at large. Future efforts to use primary care data for chronic disease surveillance need to consider the intended purpose of data collected in these systems as well as the characteristics of the population using primary care.