Evaluation of a health information exchange system for microcephaly case-finding - New York City, 2013-2015

Background

Birth defects surveillance in the United States is conducted principally by review of routine but lagged reporting to statewide congenital malformations registries of diagnoses by hospitals or other health care providers, a process that is not designed to rapidly detect changes in prevalence. Health information exchange (HIE) systems are well suited for rapid surveillance, but information is limited about their effectiveness at detecting birth defects. We evaluated HIE data to detect microcephaly diagnosed at birth during January 1, 2013–December 31, 2015 before known introduction of Zika virus in North America.

Methods

Data from an HIE system were queried for microcephaly diagnostic codes on day of birth or during the first two days after birth at three Bronx hospitals for births to New York City resident mothers. Suspected cases identified by HIE data were compared with microcephaly cases that had been identified through direct inquiry of hospital records and confirmed by chart abstraction in a previous study of the same cohort.

Results

Of 16,910 live births, 43 suspected microcephaly cases were identified through an HIE system compared to 67 confirmed cases that had been identified as part of the prior study. A total of 39 confirmed cases were found by both studies (sensitivity = 58.21%, 95% CI: 45.52–70.15%; positive predictive value = 90.70%, 95% CI: 77.86–97.41%; negative predictive value = 99.83%, 95% CI: 99.76–99.89% for HIE data).

Conclusion

Despite limitations, HIE systems could be used for rapid newborn microcephaly surveillance, especially in the many jurisdictions where more labor-intensive approaches are not feasible. Future work is needed to improve electronic medical record documentation quality to improve sensitivity and reduce misclassification.

Can Electronic Health Records Be Used for Population Health Surveillance? Validating Population Health Metrics Against Established Survey Data

INTRODUCTION

Electronic health records (EHRs) offer potential for population health surveillance but EHR-based surveillance measures require validation prior to use. We assessed the validity of obesity, smoking, depression, and influenza vaccination indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This report is the second in a 3-part series describing the development and validation of the NYC Macroscope. The first report describes in detail the infrastructure underlying the NYC Macroscope; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. This second report, which addresses concerns related to sampling bias and data quality, describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods described in this report to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

METHODS

NYC Macroscope prevalence estimates, overall and stratified by sex and age group, were compared to reference survey estimates for adult New Yorkers who reported visiting a doctor in the past year. Agreement was evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by examining individual EHR records in a subsample of 48 survey participants.

RESULTS

Among adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%) fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination prevalence were more than 10 percentage points lower than the estimates from either reference survey. While specificity was > 0.90 for both of these indicators, sensitivity was < 0.70.

DISCUSSION

Through this work we have demonstrated that EHR data from a convenience sample of providers can produce acceptable estimates of smoking and obesity prevalence among adult New Yorkers in care; gained a better understanding of the challenges involved in estimating depression prevalence from EHRs; and identified areas for additional research regarding estimation of influenza vaccination prevalence. We have also shared lessons learned about how EHR indicators should be constructed and offer methodologic suggestions for validating them.

CONCLUSIONS

This work adds to a rapidly emerging body of literature about how to define, collect and interpret EHR-based surveillance measures and may help guide other jurisdictions.

Design of the New York City Macroscope: Innovations in Population Health Surveillance Using Electronic Health Records

Introduction:

Electronic health records (EHRs) have the potential to offer real-time, inexpensive standardized health data about chronic health conditions. Despite rapid expansion, EHR data evaluations for chronic disease surveillance have been limited. We present design and methods for the New York City (NYC) Macroscope, an EHR-based chronic disease surveillance system.

This methods report is the first in a three part series describing the development and validation of the NYC Macroscope. This report describes in detail the infrastructure underlying the NYC Macroscope; indicator definitions; design decisions that were made to maximize data quality; characteristics of the population sampled; completeness of data collected; and lessons learned from doing this work. The second report describes the methods used to evaluate the validity and robustness of NYC Macroscope prevalence estimates; presents validation results for estimates of obesity, smoking, depression and influenza vaccination; and discusses the implications of our findings for NYC and for other jurisdictions embarking on similar work. The third report applies the same validation methods to metabolic outcomes, including the prevalence, treatment and control of diabetes, hypertension and hyperlipidemia.

Methods:

We designed the NYC Macroscope for comparison to a local “gold standard,” the 2013–14 NYC Health and Nutrition Examination Survey, and the telephonic 2013 Community Health Survey. NYC Macroscope indicators covered prevalence, treatment, and control of diabetes, hypertension, and hyperlipidemia; and prevalence of influenza vaccination, obesity, depression and smoking. Indicators were stratified by age, sex, and neighborhood poverty, and weighted to the in-care NYC population and limited to primary care patients. Indicator queries were distributed to a virtual network of primary care practices; 392 practices and 716,076 adult patients were retained in the final sample.

Findings:

The NYC Macroscope covered 10% of primary care providers and 15% of all adult patients in NYC in 2013 (8–47% of patients by neighborhood). Data completeness varied by domain from 98% for blood pressure among patients with hypertension to 33% for depression screening.

Discussion:

Design and validation efforts undertaken by NYC are described here to provide one potential blueprint for leveraging EHRs for population health monitoring. To replicate a model like NYC Macroscope, jurisdictions should establish buy-in; build informatics capacity; use standard, simple case defnitions; establish documentation quality thresholds; restrict to primary care providers; and weight the sample to a target population.

Monitoring Prevalence, Treatment, and Control of Metabolic Conditions in New York City Adults Using 2013 Primary Care Electronic Health Records: A Surveillance Validation Study

Introduction:

Electronic health records (EHRs) can potentially extend chronic disease surveillance, but few EHR-based initiatives tracking population-based metrics have been validated for accuracy. We designed a new EHR-based population health surveillance system for New York City (NYC) known as NYC Macroscope. This report is the third in a 3-part series describing the development and validation of that system. The first report describes governance and technical infrastructure underlying the NYC Macroscope. The second report describes validation methods and presents validation results for estimates of obesity, smoking, depression and influenza vaccination. In this third paper we present validation findings for metabolic indicators (hypertension, hyperlipidemia, diabetes).

Methods:

We compared EHR-based estimates to those from a gold standard surveillance source - the 2013–2014 NYC Health and Nutrition Examination Survey (NYC HANES) - overall and stratified by sex and age group, using the two one-sided test of equivalence and other validation criteria.

Results:

EHR-based hypertension prevalence estimates were highly concordant with NYC HANES estimates. Diabetes prevalence estimates were highly concordant when measuring diagnosed diabetes but less so when incorporating laboratory results. Hypercholesterolemia prevalence estimates were less concordant overall. Measures to assess treatment and control of the 3 metabolic conditions performed poorly.

Discussion:

While indicator performance was variable, findings here confirm that a carefully constructed EHR-based surveillance system can generate prevalence estimates comparable to those from gold-standard examination surveys for certain metabolic conditions such as hypertension and diabetes.

Conclusions:

Standardized EHR metrics have potential utility for surveillance at lower annual costs than surveys, especially as representativeness of contributing clinical practices to EHR-based surveillance systems increases.

Preliminary evaluation of a comprehensive provider feedback report

Provider feedback reports (PFRs) offer one way for providers to use their electronic health record (EHR) data to understand aspects of their clinical performance and improve quality of care. The Primary Care Information Project (PCIP) serves as a bureau of the New York City Department of Health and Mental Hygiene and as a Regional Extension Center that helps area healthcare providers adopt and achieve Meaningful Use of EHR systems. This study analyzes improvement on multiple quality measures pre- and post-receipt of a comprehensive, EHR-based PFR that PCIP created for its member providers. We analyzed improvement among low- versus high-performing providers pre- and post-receipt of the PFR. Pre-PFR receipt, improvement between high and low performers varied per measure. Post-PFR receipt, low performers improved more than high performers on all measures, and more than themselves in the pre-PFR period. Findings suggest PFRs derived directly from provider EHRs may particularly assist lower-performing providers to improve performance.