Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data

INTRODUCTION

Electronic health records (EHRs) offer potential for population health surveillance but EHR-based surveillance measures require validation prior to use. We assessed the validity of obesity, smoking, depression, and influenza vaccination indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This report is the second in a 3-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the infrastructure underlying the NYC Macroscope; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. This second report, which addresses concerns related to sampling bias and data quality, describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods described in this report to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

METHODS

NYC Macroscope prevalence estimates, overall and stratified by sex and age group, were compared to reference survey estimates for adult New Yorkers who reported visiting a doctor in the past year. Agreement was evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by examining individual EHR records in a subsample of 48 survey participants.

RESULTS

Among adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%) fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination prevalence were more than 10 percentage points lower than the estimates from either reference survey. While specificity was > 0.90 for both of these indicators, sensitivity was < 0.70.

DISCUSSION

Through this work we have demonstrated that EHR data from a convenience sample of providers can produce acceptable estimates of smoking and obesity prevalence among adult New Yorkers in care; gained a better understanding of the challenges involved in estimating depression prevalence from EHRs; and identified areas for additional research regarding estimation of influenza vaccination prevalence. We have also shared lessons learned about how EHR indicators should be constructed and offer methodologic suggestions for validating them.

CONCLUSIONS

This work adds to a rapidly emerging body of literature about how to define, collect and interpret EHR-based surveillance measures and may help guide other jurisdictions.

Design of the New York City Macroscope: Innovations in Population Health Surveillance Using Electronic Health Records

Introduction:

Electronic health records (EHRs) have the potential to offer real-time, inexpensive standardized health data about chronic health conditions. Despite rapid expansion, EHR data evaluations for chronic disease surveillance have been limited. We present design and methods for the New York City (NYC) Macroscope, an EHR-based chronic disease surveillance system.

This methods report is the first in a three part series describing the development and validation of the NYC Macroscope. This report describes in detail the infrastructure underlying the NYC Macroscope; indicator definitions; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. The second report describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

Methods:

We designed the NYC Macroscope for comparison to a local “gold standard,” the 2013–14 NYC Health and Nutrition Examination Survey, and the telephonic 2013 Community Health Survey. NYC Macroscope indicators covered prevalence, treatment, and control of diabetes, hypertension, and hyperlipidemia; and prevalence of influenza vaccination, obesity, depression and smoking. Indicators were stratified by age, sex, and neighborhood poverty, and weighted to the in-care NYC population and limited to primary care patients. Indicator queries were distributed to a virtual network of primary care practices; 392 practices and 716,076 adult patients were retained in the final sample.

Findings:

The NYC Macroscope covered 10% of primary care providers and 15% of all adult patients in NYC in 2013 (8–47% of patients by neighborhood). Data completeness varied by domain from 98% for blood pressure among patients with hypertension to 33% for depression screening.

Discussion:

Design and validation efforts undertaken by NYC are described here to provide one potential blueprint for leveraging EHRs for population health monitoring. To replicate a model like NYC Macroscope, jurisdictions should establish buy-in; build informatics capacity; use standard, simple case defnitions; establish documentation quality thresholds; restrict to primary care providers; and weight the sample to a target population.

Monitoring Prevalence, Treatment, and Control of Metabolic Conditions in New York City Adults Using 2013 Primary Care Electronic Health Records: A Surveillance Validation Study

Introduction:

Electronic health records (EHRs) can potentially extend chronic disease surveillance, but few EHR-based initiatives tracking population-based metrics have been validated for accuracy. We designed a new EHR-based population health surveillance system for New York City (NYC) known as NYC Macroscope. This report is the third in a 3-part series describing the development and validation of that system. The first report describes governance and technical infrastructure underlying the NYC Macroscope. The second report describes validation methods and presents validation results for estimates of obesity, smoking, depression and influenza vaccination. In this third paper we present validation findings for metabolic indicators (hypertension, hyperlipidemia, diabetes).

Methods:

We compared EHR-based estimates to those from a gold standard surveillance source - the 2013–2014 NYC Health and Nutrition Examination Survey (NYC HANES) - overall and stratified by sex and age group, using the two one-sided test of equivalence and other validation criteria.

Results:

EHR-based hypertension prevalence estimates were highly concordant with NYC HANES estimates. Diabetes prevalence estimates were highly concordant when measuring diagnosed diabetes but less so when incorporating laboratory results. Hypercholesterolemia prevalence estimates were less concordant overall. Measures to assess treatment and control of the 3 metabolic conditions performed poorly.

Discussion:

While indicator performance was variable, findings here confirm that a carefully constructed EHR-based surveillance system can generate prevalence estimates comparable to those from gold-standard examination surveys for certain metabolic conditions such as hypertension and diabetes.

Conclusions:

Standardized EHR metrics have potential utility for surveillance at lower annual costs than surveys, especially as representativeness of contributing clinical practices to EHR-based surveillance systems increases.

Characterizing Adults Receiving Primary Medical Care in New York City: Implications for Using Electronic Health Records for Chronic Disease Surveillance

Introduction

Electronic health records (EHRs) from primary care providers can be used for chronic disease surveillance; however, EHR-based prevalence estimates may be biased toward people who seek care. This study sought to describe the characteristics of an in-care population and compare them with those of a not-in-care population to inform interpretation of EHR data.

Methods

We used data from the 2013–2014 New York City Health and Nutrition Examination Survey (NYC HANES), considered the gold standard for estimating disease prevalence, and the 2013 Community Health Survey, and classified participants as in care or not in care, on the basis of their report of seeing a health care provider in the previous year. We used χ² tests to compare the distribution of demographic characteristics, health care coverage and access, and chronic conditions between the 2 populations.

Results

According to the Community Health Survey, approximately 4.1 million (71.7%) adults aged 20 or older had seen a health care provider in the previous year; according to NYC HANES, approximately 4.7 million (75.1%) had. In both surveys, the in-care population was more likely to be older, female, non-Hispanic, and insured compared with the not-in-care population. The in-care population from the NYC HANES also had a higher prevalence of diabetes (16.7% vs 6.9%; P < .001), hypercholesterolemia (35.7% vs 22.3%; P < .001), and hypertension (35.5% vs 26.4%; P < .001) than the not-in-care population.

Conclusion

Systematic differences between in-care and not-in-care populations warrant caution in using primary care data to generalize to the population at large. Future efforts to use primary care data for chronic disease surveillance need to consider the intended purpose of data collected in these systems as well as the characteristics of the population using primary care.

Assessing capacity and disease burden in a virtual network of New York City primary care providers following Hurricane Sandy

Urban contexts introduce unique challenges that must be addressed to ensure that areas of high population density can function when disasters occur. The ability to generate useful data to guide decision-making is critical in this context. Widespread adoption of electronic health record (EHR) systems in recent years has created electronic data sources and networks that may play an important role in public health surveillance efforts, including in post-disaster situations. The Primary Care Information Project (PCIP) at the New York City Department of Health and Mental Hygiene has partnered with local clinicians to establish an electronic data system, and this network provides infrastructure to support primary care surveillance activities in New York City. After Hurricane Sandy, PCIP generated several sets of data to contribute to the city's efforts to assess the impact of the storm, including daily connectivity data to establish practice operations, data to examine patterns of primary care utilization in severely affected and less affected areas, and data on the frequency of respiratory infection diagnosis in the primary care setting. Daily patient visit data from three heavily affected neighborhoods showed the health department where primary care capacity was most affected in the weeks following Sandy. Overall transmission data showed that practices in less affected areas were quicker to return to normal reporting patterns, while those in more affected areas did not resume normal data transmissions for a few months. Rates of bronchitis increased after Sandy compared to the two prior years; while this was most likely attributable to a more severe flu season, it demonstrates the capacity of primary care networks to pick up on these types of post-emergency trends. Hurricane Sandy was the first disaster situation where PCIP was asked to assess public health impact, generating information that could contribute to aid and recovery efforts. This experience allowed us to explore the strengths and weaknesses of ambulatory EHR data in post-disaster settings. Data from ambulatory EHR networks can augment existing surveillance streams by providing sentinel population snapshots on clinically available indicators in near real time.