How good are the data? Feasible approach to validation of metrics of quality derived from an outpatient electronic health record

Assessing hospital differences in low-risk cesarean delivery metrics in Florida

BACKGROUND

Unnecessary cesarean deliveries lead to increased maternal and neonatal morbidities and mortalities. In 2020, Florida had a cesarean delivery rate of 35.9%, the third highest in the nation. An effective quality improvement strategy to reduce overall cesarean delivery rates is to decrease primary cesarean deliveries in low-risk births (nulliparous, term, singleton, vertex). Of note, 3 nationally accepted hospital measures of low-risk cesarean delivery rates include the nulliparous, term, singleton, vertex; Joint Commission; and Society for Maternal-Fetal Medicine metrics. Comparing metrics is necessary because accurate and timely measurement is essential to support multihospital quality improvement efforts to reduce low-risk cesarean delivery rates and improve the quality of maternal care.

OBJECTIVE

This study aimed to assess differences in hospital low-risk cesarean delivery rates in Florida using 5 different metrics of low-risk cesarean delivery rate based on (1) risk methodology, nulliparous, term, singleton, vertex; Joint Commission; and Society for Maternal-Fetal Medicine metrics, and (2) data source, linked birth certificate and hospital discharge records and hospital discharge records only.

STUDY DESIGN

This was a population-based study of live Florida births from 2016 to 2019 to compare 5 approaches to calculating low-risk cesarean delivery rates. Analyses were performed using linked birth certificate data and inpatient hospital discharge data. The 5 low-risk cesarean delivery measures were defined as follows: nulliparous, term, singleton, vertex birth certificate; Joint Commission-linked used Joint Commission exclusions; Society for Maternal-Fetal Medicine-linked used Society for Maternal-Fetal Medicine exclusions; Joint Commission hospital discharge with Joint Commission exclusions; and Society for Maternal-Fetal Medicine hospital discharge with Society for Maternal-Fetal Medicine exclusions. Nulliparous, term, singleton, vertex birth certificate was based on data from birth certificates and not using linked hospital discharge data. Designated as nulliparous, term, singleton, vertex, it does not exclude other high-risk conditions. The second and third measures (Joint Commission-linked used Joint Commission exclusions and Society for Maternal-Fetal Medicine-linked used Society for Maternal-Fetal Medicine exclusions) use data elements from the full-linked dataset to designate nulliparous, term, singleton, vertex and excluded several high-risk conditions. The last 2 measures (Joint Commission hospital discharge with Joint Commission exclusions; and Society for Maternal-Fetal Medicine hospital discharge with Society for Maternal-Fetal Medicine exclusions) were based on data from hospital discharge data only and not using linked birth certificate data. These measures generally reflect term, singleton, and vertex because parity could not be assessed adequately on hospital discharge data. Hospital differences between these 5 measures were calculated overall and by neonatal intensive care unit level.

RESULTS

Overall, the median of hospital low-risk cesarean rates decreased across the measures, from NTSV-BC 30.7%, to Joint Commission linked 29.1%, and Society for Maternal Fetal Medicine hospital discharge 29.2% with a large decrease to Joint Commission hospital discharge 19.4% and Society for Maternal Fetal Medicine hospital discharge 18.1%. A similar trend was seen by neonatal intensive care unit level. For each of the measures, level II had the highest median low-risk cesarean rates (nulliparous. term, singleton, vertex birth certificate) 32.7%, Joint Commission linked (31.4%), Society for Maternal Fetal Medicine linked: 31.1%, Society for Maternal Fetal Medicine hospital discharge 19.3%), except for level III Joint Commission hospital discharge (20.0%). A comparison of the median number of low-risk births overall and by neonatal intensive care unit level showed a decreasing number across the linked and hospital discharge measures. Again, a wide gap in low-risk cesarean delivery rates was identified between linked measures and hospital discharge measures. However, this gap narrowed as hospital rates increased.

CONCLUSION

Quality monitoring of low-risk cesarean delivery rates measured by the nulliparous, term, singleton, vertex metric using the birth certificate was fairly accurate and provided timely assessment for use by Florida hospitals. The nulliparous, term, singleton, vertex birth certificate rates were comparable with low-risk metrics using the linked data source. Overall, metrics used within the same data source had similar rates, with the Society for Maternal-Fetal Medicine metric having the lowest rates. Across data sources, metrics using hospital discharge data only resulted in substantially underestimated rates because of the inclusion of multiparous women and should be interpreted with caution.

Quality and Safety Practices Among Academic Obstetrics and Gynecology Departments

The objective was to quantify resources devoted to quality and patient safety initiatives, to document the development and use of key performance indicator reports regarding patient outcomes and patient feedback, and to assess the culture of safety within academic obstetrics and gynecology departments. Chairs of academic obstetrics and gynecology departments were asked to complete a quality and safety assessment survey. Surveys were distributed to 138 departments, yielding 52 completed responses (37.7%). Five percent of departments reported including a patient representative on a quality committee. Most committee leaders (60.5%) and members (67.4%) received no compensation. Formal training was required in 28.8% of responding departments. Most departments monitored key performance metrics for inpatient outcomes (95.9%). Leaders scored their departments' culture of safety highly. Most departments provided no protected time to faculty devoted to quality efforts, generation of key performance indicators for inpatient activities was prevalent and integrating patient and community input remain unrealized opportunities.

Hospital-wide natural language processing summarising the health data of 1 million patients

Electronic health records (EHRs) represent a major repository of real world clinical trajectories, interventions and outcomes. While modern enterprise EHR's try to capture data in structured standardised formats, a significant bulk of the available information captured in the EHR is still recorded only in unstructured text format and can only be transformed into structured codes by manual processes. Recently, Natural Language Processing (NLP) algorithms have reached a level of performance suitable for large scale and accurate information extraction from clinical text. Here we describe the application of open-source named-entity-recognition and linkage (NER+L) methods (CogStack, MedCAT) to the entire text content of a large UK hospital trust (King's College Hospital, London). The resulting dataset contains 157M SNOMED concepts generated from 9.5M documents for 1.07M patients over a period of 9 years. We present a summary of prevalence and disease onset as well as a patient embedding that captures major comorbidity patterns at scale. NLP has the potential to transform the health data lifecycle, through large-scale automation of a traditionally manual task.

Not all phenotypes are created equal: covariates of success in e-phenotype specification

BACKGROUND

Electronic (e)-phenotype specification by noninformaticist investigators remains a challenge. Although validation of each patient returned by e-phenotype could ensure accuracy of cohort representation, this approach is not practical. Understanding the factors leading to successful e-phenotype specification may reveal generalizable strategies leading to better results.

MATERIALS AND METHODS

Noninformaticist experts (n = 21) were recruited to produce expert-mediated e-phenotypes using i2b2 assisted by a honest data-broker and a project coordinator. Patient- and visit-sets were reidentified and a random sample of 20 charts matching each e-phenotype was returned to experts for chart-validation. Attributes of the queries and expert characteristics were captured and related to chart-validation rates using generalized linear regression models.

RESULTS

E-phenotype validation rates varied according to experts' domains and query characteristics (mean = 61%, range 20-100%). Clinical domains that performed better included infectious, rheumatic, neonatal, and cancers, whereas other domains performed worse (psychiatric, GI, skin, and pulmonary). Match-rate was negatively impacted when specification of temporal constraints was required. In general, the increase in e-phenotype specificity contributed positively to match-rate.

DISCUSSIONS AND CONCLUSIONS

Clinical experts and informaticists experience a variety of challenges when building e-phenotypes, including the inability to differentiate clinical events from patient characteristics or appropriately configure temporal constraints; a lack of access to available and quality data; and difficulty in specifying routes of medication administration. Biomedical query mediation by informaticists and honest data-brokers in designing e-phenotypes cannot be overstated. Although tools such as i2b2 may be widely available to noninformaticists, successful utilization depends not on users' confidence, but rather on creating highly specific e-phenotypes.

A Provider-Focused Intervention to Promote Optimal Care of Pediatric Patients With Suspected Elbow Fracture

OBJECTIVE

Emergency department (ED) and urgent care (UC) physicians' accurate assessment of the neurovascular and musculoskeletal (NV/MSK) examination in pediatric patients with suspected elbow fracture is crucial to the early recognition of neurovascular compromise. Our objective was to determine the impact of computer-based simulation (CBS) and computerized clinical decision support systems (CCDSS) on ED and UC physicians' assessment of the NV/MSK examination of pediatric patients with elbow fracture as noted in their documentation.

METHODS

All ED UC physician participants received CBS training about management of pediatric patients with suspected elbow fracture. Participants were then randomized to receive CCDSS (intervention arm) when an eligible patient was seen or no further intervention (comparison arm.) Participants received feedback on the proportion of patients with discharge diagnosis of elbow fracturewith proper examination elements documented.

RESULTS

Twenty-eight ED and UC physicians were enrolled - 14 in each arm. Over the span of 16 weeks, 50 patients with a discharge diagnosis of elbow fracture were seen - 25 in each arm. Twenty-two of 25 (88%) patients seen by intervention arm participants had a complete NV/MSK examination documented. Six of 25 (24%) patients seen by comparison arm participants had a complete NV/MSK examination documented. Elements most commonly missed in the comparison arm included documentation of ulnar pulse as well as radial, median, and ulnar nerve motor functions.

CONCLUSIONS

Compared with single CBS training alone, repeated exposure to CCDSS after CBS training resulted in improved documentation of the NV/MSK status of pediatric patients with elbow fracture.

A System-Wide Population Health Value Approach to Reduce Hospitalization Among Chronic Kidney Disease Patients: an Observational Study

BACKGROUND

Chronic kidney disease (CKD) is a leading cause of healthcare morbidity, utilization, and expenditures nationally, and caring for late-stage CKD populations is complex. Improving health system efficiency could mitigate these outcomes and, in the COVID-19 era, reduce risks of viral exposure.

OBJECTIVE

As part of a system-wide transformation to improve healthcare value among populations with high healthcare utilization and morbidity, UCLA Health evaluated a new patient-centered approach that we hypothesized would reduce inpatient utilization for CKD patients.

DESIGN

For 18 months in 2015-2016 and 12 months in 2017, we conducted an interrupted time series regression analysis to evaluate the intervention's impact on inpatient utilization. We used internal electronic health records and claims data across six payers.

PARTICIPANTS

A total of 1442 stage 4-5 CKD patients at a large academic medical center.

INTERVENTION

Between October and December 2016, the organization implemented a Population Health Value CKD intervention for the CKD stages 4-5 population. A multispecialty leadership team risk stratified the population and identified improvement opportunities, redesigned multispecialty care coordination pathways across settings, and developed greater ambulatory infrastructure to support care needs.

MAIN MEASURES

Outcomes included utilization of hospitalizations, emergency department (ED) visits, inpatient bed days, and 30-day all-cause readmissions.

KEY RESULTS

During the 12 months following intervention implementation, the monthly estimated rate of decline for hospitalizations was 5.4% (95% CI: 3.4-7.4%), which was 3.4 percentage points faster than the 18-month pre-intervention decline of 2.0% (95% CI: 1.0-2.2%) per month (p = 0.004). Medicare CKD patients' monthly ED visit rate of decline was 3.0% (95% CI: 1.2-4.8%) after intervention, which was 2.6 percentage points faster than the pre-intervention decline of 0.4% (95% CI: - 0.8 to 1.6%) per month (p = 0.02).

CONCLUSIONS

By creating care pathways that link primary and specialty care teams across settings with increased ambulatory infrastructure, healthcare systems have potential to reduce inpatient healthcare utilization.

Private Hospital Workflow Optimization via Secure k-Means Clustering

Optimizing the workflow of a complex organization such as a hospital is a difficult task. An accurate option is to use a real-time locating system to track locations of both patients and staff. However, privacy regulations forbid hospital management to assess location data of their staff members. In this exploratory work, we propose a secure solution to analyze the joined location data of patients and staff, by means of an innovative cryptographic technique called Secure Multi-Party Computation, in which an additional entity that the staff members can trust, such as a labour union, takes care of the staff data. The hospital, owning location data of patients, and the labour union perform a two-party protocol, in which they securely cluster the staff members by means of the frequency of their patient facing times. We describe the secure solution in detail, and evaluate the performance of our proof-of-concept. This work thus demonstrates the feasibility of secure multi-party clustering in this setting.

Automating the Capture of Structured Pathology Data for Prostate Cancer Clinical Care and Research

PURPOSE

Cancer pathology findings are critical for many aspects of care but are often locked away as unstructured free text. Our objective was to develop a natural language processing (NLP) system to extract prostate pathology details from postoperative pathology reports and a parallel structured data entry process for use by urologists during routine documentation care and compare accuracy when compared with manual abstraction and concordance between NLP and clinician-entered approaches.

MATERIALS AND METHODS

From February 2016, clinicians used note templates with custom structured data elements (SDEs) during routine clinical care for men with prostate cancer. We also developed an NLP algorithm to parse radical prostatectomy pathology reports and extract structured data. We compared accuracy of clinician-entered SDEs and NLP-parsed data to manual abstraction as a gold standard and compared concordance (Cohen's κ) between approaches assuming no gold standard.

RESULTS

There were 523 patients with NLP-extracted data, 319 with SDE data, and 555 with manually abstracted data. For Gleason scores, NLP and clinician SDE accuracy was 95.6% and 95.8%, respectively, compared with manual abstraction, with concordance of 0.93 (95% CI, 0.89 to 0.98). For margin status, extracapsular extension, and seminal vesicle invasion, stage, and lymph node status, NLP accuracy was 94.8% to 100%, SDE accuracy was 87.7% to 100%, and concordance between NLP and SDE ranged from 0.92 to 1.0.

CONCLUSION

We show that a real-world deployment of an NLP algorithm to extract pathology data and structured data entry by clinicians during routine clinical care in a busy clinical practice can generate accurate data when compared with manual abstraction for some, but not all, components of a prostate pathology report.

Involving end-users in the design of an audit and feedback intervention in the emergency department setting - a mixed methods study

BACKGROUND

Long length of stays (LOS) in emergency departments (ED) negatively affect quality of care. Ordering of inappropriate diagnostic tests contributes to long LOS and reduces quality of care. One strategy to change practice patterns is to use performance feedback dashboards for physicians. While this strategy has proven to be successful in multiple settings, the most effective ways to deliver such interventions remain unknown. Involving end-users in the process is likely important for a successful design and implementation of a performance dashboard within a specific workplace culture. This mixed methods study aimed to develop design requirements for an ED performance dashboard and to understand the role of culture and social networks in the adoption process.

METHODS

We performed 13 semi-structured interviews with attending physicians in different roles within a single public ED in the U.S. to get an in-depth understanding of physicians' needs and concerns. Principles of human-centered design were used to translate these interviews into design requirements and to iteratively develop a front-end performance feedback dashboard. Pre- and post- surveys were used to evaluate the effect of the dashboard on physicians' motivation and to measure their perception of the usefulness of the dashboard. Data on the ED culture and underlying social network were collected. Outcomes were compared between physicians involved in the human-centered design process, those with exposure to the design process through the ED social network, and those with limited exposure.

RESULTS

Key design requirements obtained from the interviews were ease of access, drilldown functionality, customization, and a visual data display including monthly time-trends and blinded peer-comparisons. Identified barriers included concerns about unintended consequences and the veracity of underlying data. The surveys revealed that the ED culture and social network are associated with reported usefulness of the dashboard. Additionally, physicians' motivation was differentially affected by the dashboard based on their position in the social network.

CONCLUSIONS

This study demonstrates the feasibility of designing a performance feedback dashboard using a human-centered design approach in the ED setting. Additionally, we show preliminary evidence that the culture and underlying social network are of key importance for successful adoption of a dashboard.

Provider-Focused Intervention to Promote Comprehensive Screening for Adolescent Idiopathic Scoliosis by Primary Care Pediatricians

Screening can detect adolescent idiopathic scoliosis (AIS). The objective was to determine if computer-based simulation (CBS) and computerized clinical decision-support systems (CCDSS) would improve primary care providers' AIS screening exams as noted in their documentation. All participants received AIS screening CBS training. Participants were then randomized to receive either CCDSS when an eligible patient was seen (intervention arm) or no further intervention (comparison arm). Eligible patients' documentation was analyzed looking for a complete AIS screening exam. Over the span of 17 weeks, 1051 eligible patients were seen; 468 by providers in the intervention arm, 583 in the comparison arm. In all, 292/468 (62%) of eligible patients seen in the intervention arm and 0/583 (0%) in the comparison arm had a complete AIS screening exam documented. Compared with single CBS training alone, repeated exposure to CCDSS after CBS training resulted in improved documentation of the screening exam for AIS.

Use of an electronic health record data sharing system for identifying current contraceptive use within the WWAMI region Practice and Research Network

OBJECTIVE

To evaluate the ability of electronic health record (EHR) data extracted into a data-sharing system to accurately identify contraceptive use.

STUDY DESIGN

We compared rates of contraceptive use from electronic extraction of EHR data via a data-sharing system and manual abstraction of the EHR among 142 female patients ages 15-49 years from a family medicine clinic within a primary care practice-based research network (PBRN). Cohen's kappa coefficient measured agreement between electronic extraction and manual abstraction.

RESULTS

Manual abstraction identified 62% of women as contraceptive users, whereas electronic extraction identified only 27%. Long acting reversible (LARC) methods had 96% agreement (Cohen's kappa 0.78; confidence interval, 0.57-0.99) between electronic extraction and manual abstraction. EHR data extracted via a data-sharing system was unable to identify barrier or over-the-counter contraceptives.

CONCLUSIONS

Electronic extraction found substantially lower overall rates of contraceptive method use, but produced more comparable LARC method use rates when compared to manual abstraction among women in this study's primary care clinic.

IMPLICATIONS

Quality metrics related to contraceptive use that rely on EHR data in this study's data-sharing system likely under-estimated true contraceptive use.

PIE: A prior knowledge guided integrated likelihood estimation method for bias reduction in association studies using electronic health records data

Objectives

This study proposes a novelPrior knowledge guidedIntegrated likelihoodEstimation (PIE) method to correct bias in estimations of associations due to misclassification of electronic health record (EHR)-derived binary phenotypes, and evaluates the performance of the proposed method by comparing it to 2 methods in common practice.

Methods

We conducted simulation studies and data analysis of real EHR-derived data on diabetes from Kaiser Permanente Washington to compare the estimation bias of associations using the proposed method, the method ignoring phenotyping errors, the maximum likelihood method with misspecified sensitivity and specificity, and the maximum likelihood method with correctly specified sensitivity and specificity (gold standard). The proposed method effectively leverages available information on phenotyping accuracy to construct a prior distribution for sensitivity and specificity, and incorporates this prior information through the integrated likelihood for bias reduction.

Results

Our simulation studies and real data application demonstrated that the proposed method effectively reduces the estimation bias compared to the 2 current methods. It performed almost as well as the gold standard method when the prior had highest density around true sensitivity and specificity. The analysis of EHR data from Kaiser Permanente Washington showed that the estimated associations from PIE were very close to the estimates from the gold standard method and reduced bias by 60%–100% compared to the 2 commonly used methods in current practice for EHR data.

Conclusions

This study demonstrates that the proposed method can effectively reduce estimation bias caused by imperfect phenotyping in EHR-derived data by incorporating prior information through integrated likelihood.

Electronic approaches to making sense of the text in the adverse event reporting system

INTRODUCTION

Health care organizations working to eliminate preventable harm and to improve patient safety must have robust programs to collect and to analyze data on adverse events in order to use the information to affect improvement. Such adverse event reporting systems are based on frontline personnel reporting issues that arise in the course of their daily work. Limitations in how existing software systems handle these reports mean that use of this potentially rich information is resource intensive and prone to variable results.

AIM

The aim of this study was to develop an electronic approach to processing the text in medical event reports that would be reliable enough to be used to improve patient safety.

METHODS

At Connecticut Children's Medical Center, staff manually enter reports of adverse events into a web-based software tool. We evaluated the ability of 2 electronic methods-rule-based query and semi-supervised machine learning-to identify specific types of events ("use cases") versus a reference standard. Rule-based query was tested on 5 use cases and machine learning on a subset of 2 using 9164 events reported from February 2012-January 2014.

RESULTS

Machine learning found 93% of the weight-based errors and 92% of the errors in patient-identification. Rule-based query had accuracy of 99% or greater, high precision, and high recall for all use cases.

CONCLUSIONS

Electronic approaches to streamlining the use of adverse event reports are feasible to automate and valuable for categorizing this important data for use in improving patient safety.

Pain research using Veterans Health Administration electronic and administrative data sources

Health services researchers are using Veterans Health Administration (VHA) electronic health record (EHR) data sources to examine the prevalence, treatment, and outcomes of pain among Veterans in VHA care. Little guidance currently exists on using these data; thus, findings may vary depending on the methods, data sources, and definitions used. We sought to identify current practices in order to provide guidance to future pain researchers. We conducted an anonymous survey of VHA-affiliated researchers participating in a monthly national pain research teleconference. Thirty-two researchers (89%) responded: 75% conducted pain-focused research, 78% used pain intensity numeric rating screening scale (NRS) scores to identify pain, 41% used International Classification of Diseases-9th Revision codes, and 57% distinguished between chronic and acute pain using either NRS scores or pharmacy data. The NRS and pharmacy data were rated as the most valid pain data sources. Of respondents, 48% reported the EHR data sources were adequate for pain research, while 45% had published peer-reviewed articles based on the data. Despite limitations, VHA researchers are increasingly using EHR data for pain research, and several common methods were identified. More information on the performance characteristics of these data sources and definitions is needed.

Template for preparation of papers for IEEE sponsored conferences & symposia

To improve the access to medical information is necessary to design and implement integrated informatics techniques aimed to gather data from different and heterogeneous sources. This paper describes the technologies used to integrate data coming from the electronic medical record of the IRCCS Fondazione Maugeri (FSM) hospital of Pavia, Italy, and combines them with administrative, pharmacy drugs purchase coming from the local healthcare agency (ASL) of the Pavia area and environmental open data of the same region. The integration process is focused on data coming from a cohort of one thousand patients diagnosed with Type 2 Diabetes Mellitus (T2DM). Data analysis and temporal data mining techniques have been integrated to enhance the initial dataset allowing the possibility to stratify patients using further information coming from the mined data like behavioral patterns of prescription-related drug purchases and other frequent clinical temporal patterns, through the use of an intuitive dashboard controlled system.

A human factors approach to improving electronic performance measurement of venous thromboembolism prophylaxis

OBJECTIVE

In 2011, our institution developed a venous thromboembolism (VTE) prophylaxis order set to monitor prophylaxis management through physician-generated risk assessment orders. Prophylaxis rates obtained using the risk assessment orders were falsely low compared with chart review. Our goal was to redesign the order set to increase the percentage of VTE risk assessments ordered, both to improve care and to better reflect performance.

DESIGN

Quality Improvement Project.

SETTING

Veterans Health Administration.

PARTICIPANTS

Patients admitted to acute care and intensive care medical units.

INTERVENTIONS

Process analysis was used to identify systems failures limiting use of the original order set. The order set was redesigned using a human factors approach.

MAIN OUTCOME MEASURE

VTE risk assessment orders.

RESULTS

The order set was redesigned to reduce complexity and improve integration into provider workflow. The rate of risk assessment orders placed within 24 h increased from 48.6 to 80.4% (P < 0.001). There was no difference in the actual use of prophylaxis. However, for patients on prophylaxis, the rates of having a documented 'moderate' or 'high' risk assessment within 24 h increased from 66.7 to 95.7% (P < 0.001).

CONCLUSIONS

Using human factor principles to redesign an order set led to a significant increase in the percentage of patients with a risk assessment order placed within 24 h of admission. Although the risk assessments using the redesigned order set better reflected physician performance, it remained an imperfect measure for VTE prophylaxis. New technology used to measure human performance must be evaluated following implementation to assess accuracy.

Effects of Common Data Errors in Electronic Health Records on Emergency Department Operational Performance Metrics: A Monte Carlo Simulation

OBJECTIVES

The objective was to estimate how data errors in electronic health records (EHRs) can affect the accuracy of common emergency department (ED) operational performance metrics.

METHODS

Using a 3-month, 7,348-visit data set of electronic time stamps from a suburban academic ED as a baseline, Monte Carlo simulation was used to introduce four types of data errors (substitution, missing, random, and systematic bias) at three frequency levels (2, 4, and 7%). Three commonly used ED operational metrics (arrival to clinician evaluation, disposition decision to exit for admitted patients, and ED length of stay for admitted patients) were calculated and the proportion of ED visits that achieved each performance goal was determined.

RESULTS

Even small data errors have measurable effects on a clinical organization's ability to accurately determine whether it is meeting its operational performance goals. Systematic substitution errors, increased frequency of errors, and the use of shorter-duration metrics resulted in a lower proportion of ED visits reported as meeting the associated performance objectives. However, the presence of other error types mitigated somewhat the effect of the systematic substitution error. Longer time-duration metrics were found to be less sensitive to data errors than shorter time-duration metrics.

CONCLUSIONS

Infrequent and small-magnitude data errors in EHR time stamps can compromise a clinical organization's ability to determine accurately if it is meeting performance goals. By understanding the types and frequencies of data errors in an organization's EHR, organizational leaders can use data management best practices to better measure true performance and enhance operational decision-making.

Accessing primary care Big Data: the development of a software algorithm to explore the rich content of consultation records

OBJECTIVE

To develop a natural language processing software inference algorithm to classify the content of primary care consultations using electronic health record Big Data and subsequently test the algorithm's ability to estimate the prevalence and burden of childhood respiratory illness in primary care.

DESIGN

Algorithm development and validation study. To classify consultations, the algorithm is designed to interrogate clinical narrative entered as free text, diagnostic (Read) codes created and medications prescribed on the day of the consultation.

SETTING

Thirty-six consenting primary care practices from a mixed urban and semirural region of New Zealand. Three independent sets of 1200 child consultation records were randomly extracted from a data set of all general practitioner consultations in participating practices between 1 January 2008-31 December 2013 for children under 18 years of age (n=754,242). Each consultation record within these sets was independently classified by two expert clinicians as respiratory or non-respiratory, and subclassified according to respiratory diagnostic categories to create three 'gold standard' sets of classified records. These three gold standard record sets were used to train, test and validate the algorithm.

OUTCOME MEASURES

Sensitivity, specificity, positive predictive value and F-measure were calculated to illustrate the algorithm's ability to replicate judgements of expert clinicians within the 1200 record gold standard validation set.

RESULTS

The algorithm was able to identify respiratory consultations in the 1200 record validation set with a sensitivity of 0.72 (95% CI 0.67 to 0.78) and a specificity of 0.95 (95% CI 0.93 to 0.98). The positive predictive value of algorithm respiratory classification was 0.93 (95% CI 0.89 to 0.97). The positive predictive value of the algorithm classifying consultations as being related to specific respiratory diagnostic categories ranged from 0.68 (95% CI 0.40 to 1.00; other respiratory conditions) to 0.91 (95% CI 0.79 to 1.00; throat infections).

CONCLUSIONS

A software inference algorithm that uses primary care Big Data can accurately classify the content of clinical consultations. This algorithm will enable accurate estimation of the prevalence of childhood respiratory illness in primary care and resultant service utilisation. The methodology can also be applied to other areas of clinical care.

Analyzing complex patients' temporal histories: new frontiers in temporal data mining

In recent years, data coming from hospital information systems (HIS) and local healthcare organizations have started to be intensively used for research purposes. This rising amount of available data allows reconstructing the compete histories of the patients, which have a strong temporal component. This chapter introduces the major challenges faced by temporal data mining researchers in an era when huge quantities of complex clinical temporal data are becoming available. The analysis is focused on the peculiar features of this kind of data and describes the methodological and technological aspects that allow managing such complex framework. The chapter shows how heterogeneous data can be processed to derive a homogeneous representation. Starting from this representation, it illustrates different techniques for jointly analyze such kind of data. Finally, the technological strategies that allow creating a common data warehouse to gather data coming from different sources and with different formats are presented.

Variations in measurement of sexual activity based on EHR definitions

OBJECTIVE

The goal of this study was to compare the performance of 4 operational definitions of sexual activity by using data electronically abstracted from electronic health records (EHRs) and examine how documentation of Chlamydia screening and positivity vary according to definition of sexual activity.

METHODS

Extracts were created from EHRs of adolescent females 12 to 19 years old who had ≥1 visit to a primary care practice during 2011 at 4 US pediatric health care organizations. We created 4 definitions of sexual activity derived from electronically abstracted indicator variables. Percent sexually active, documentation of Chlamydia screening, and rate of positive Chlamydia test results per 1000 adolescent females according to the sexual activity definition were calculated.

RESULTS

The most commonly documented individual indicator of sexual activity was "patient report of being sexually active" (mean across 4 sites: 19.2%). The percentage of adolescent females classified as sexually active varied by site and increased as more indicator variables were included. As the definition of sexual activity expanded, the percentage of sexually active females who received at least 1 Chlamydia test decreased. Using a broader definition of sexual activity resulted in improved identification of adolescent females with Chlamydia infection. For each sexual activity definition and performance item, the difference was statistically significant (P < .0001).

CONCLUSIONS

Information about sexual activity may be gathered from a variety of data sources, and changing the configurations of these indicators results in differences in the percentage of adolescent females classified as sexually active, screened for Chlamydia infection, and Chlamydia infection rates.

Innovative uses of electronic health records and social media for public health surveillance

Electronic health records (EHRs) and social media have the potential to enrich public health surveillance of diabetes. Clinical and patient-facing data sources for diabetes surveillance are needed given its profound public health impact, opportunity for primary and secondary prevention, persistent disparities, and requirement for self-management. Initiatives to employ data from EHRs and social media for diabetes surveillance are in their infancy. With their transformative potential come practical limitations and ethical considerations. We explore applications of EHR and social media for diabetes surveillance, limitations to approaches, and steps for moving forward in this partnership between patients, health systems, and public health.

Caveats for the use of operational electronic health record data in comparative effectiveness research

The growing amount of data in operational electronic health record systems provides unprecedented opportunity for its reuse for many tasks, including comparative effectiveness research. However, there are many caveats to the use of such data. Electronic health record data from clinical settings may be inaccurate, incomplete, transformed in ways that undermine their meaning, unrecoverable for research, of unknown provenance, of insufficient granularity, and incompatible with research protocols. However, the quantity and real-world nature of these data provide impetus for their use, and we develop a list of caveats to inform would-be users of such data as well as provide an informatics roadmap that aims to insure this opportunity to augment comparative effectiveness research can be best leveraged.

The Analytic Information Warehouse (AIW): a platform for analytics using electronic health record data

OBJECTIVE

To create an analytics platform for specifying and detecting clinical phenotypes and other derived variables in electronic health record (EHR) data for quality improvement investigations.

MATERIALS AND METHODS

We have developed an architecture for an Analytic Information Warehouse (AIW). It supports transforming data represented in different physical schemas into a common data model, specifying derived variables in terms of the common model to enable their reuse, computing derived variables while enforcing invariants and ensuring correctness and consistency of data transformations, long-term curation of derived data, and export of derived data into standard analysis tools. It includes software that implements these features and a computing environment that enables secure high-performance access to and processing of large datasets extracted from EHRs.

RESULTS

We have implemented and deployed the architecture in production locally. The software is available as open source. We have used it as part of hospital operations in a project to reduce rates of hospital readmission within 30days. The project examined the association of over 100 derived variables representing disease and co-morbidity phenotypes with readmissions in 5years of data from our institution's clinical data warehouse and the UHC Clinical Database (CDB). The CDB contains administrative data from over 200 hospitals that are in academic medical centers or affiliated with such centers.

DISCUSSION AND CONCLUSION

A widely available platform for managing and detecting phenotypes in EHR data could accelerate the use of such data in quality improvement and comparative effectiveness studies.

The use and misuse of biomedical data: is bigger really better?

Very large biomedical research databases, containing electronic health records (EHR) and genomic data from millions of patients, have been heralded recently for their potential to accelerate scientific discovery and produce dramatic improvements in medical treatments. Research enabled by these databases may also lead to profound changes in law, regulation, social policy, and even litigation strategies. Yet, is "big data" necessarily better data? This paper makes an original contribution to the legal literature by focusing on what can go wrong in the process of biomedical database research and what precautions are necessary to avoid critical mistakes. We address three main reasons for approaching such research with care and being cautious in relying on its outcomes for purposes of public policy or litigation. First, the data contained in biomedical databases is surprisingly likely to be incorrect or incomplete. Second, systematic biases, arising from both the nature of the data and the preconceptions of investigators, are serious threats to the validity of research results, especially in answering causal questions. Third, data mining of biomedical databases makes it easier for individuals with political, social, or economic agendas to generate ostensibly scientific but misleading research findings for the purpose of manipulating public opinion and swaying policymakers. In short, this paper sheds much-needed light on the problems of credulous and uninformed acceptance of research results derived from biomedical databases. An understanding of the pitfalls of big data analysis is of critical importance to anyone who will rely on or dispute its outcomes, including lawyers, policymakers, and the public at large. The Article also recommends technical, methodological, and educational interventions to combat the dangers of database errors and abuses.