Developing a systematic approach to assessing data quality in secondary use of clinical data based on intended use

Challenges for Data Quality in the Clinical Data Life Cycle: Systematic Review

BACKGROUND

Electronic health record (EHR) data are anticipated to inform the development of health policy systems across countries and furnish valuable insights for the advancement of health and medical technology. As the current paradigm of clinical research is shifting toward data centricity, the utilization of health care data is increasingly emphasized.

OBJECTIVE

We aimed to review the literature on clinical data quality management and define a process for ensuring the quality management of clinical data, especially in the secondary utilization of data.

METHODS

A systematic review of PubMed articles from 2010 to October 2023 was conducted. A total of 82,346 articles were retrieved and screened based on the inclusion and exclusion criteria, narrowing the number of articles to 851 after title and abstract review. Articles focusing on clinical data quality management life cycles, assessment methods, and tools were selected.

RESULTS

We reviewed 105 papers describing the clinical data quality management process. This process is based on a 4-stage life cycle: planning, construction, operation, and utilization. The most frequently used dimensions were completeness, plausibility, concordance, security, currency, and interoperability.

CONCLUSIONS

Given the importance of the secondary use of EHR data, standardized quality control methods and automation are necessary. This study proposes a process to standardize data quality management and develop a data quality assessment system.

Assessing the harmonization of structured electronic health record data to reference terminologies and data completeness through data provenance

Introduction

(1) Assess the harmonization of structured electronic health record data (laboratory results and medications) to reference terminologies and characterize the severity of issues. (2) Identify issues of data completeness by comparing complementary data domains, stratifying by time, care setting, and provenance.

Methods

Queries were distributed to 3 Data Partners (DP). Using harmonization queries, we examined the top 200 laboratory results and medications by volume, identifying outliers and computing summary statistics. The completeness queries looked at 4 conditions of interest and related clinical concepts. Counts were generated for each condition, stratified by year, encounter type, and provenance. We analyzed trends over time within and across DPs.

Results

We found that the median number of codes associated with a given laboratory/medication name (and vice versa) generally met expectations, though there were DP-specific issues that resulted in outliers. In addition, there were drastic differences in the percentage of patients with a given concept depending on provenance.

Conclusions

The harmonization queries surfaced several mapping errors, as well as issues with overly specific codes and records with "null" codes. The completeness queries demonstrated having access to multiple types of data provenance provides more robust results compared with any single provenance type. Harmonization errors between source data and reference terminologies may not be widespread but do exist within CDMs, affecting tens of thousands or even millions of records. Provenance information can help identify potential completeness issues with EHR data, but only if it is represented in the CDM and then populated by DPs.

The Significance of Information Quality for the Secondary Use of the Information in the National Health Care Quality Registers in Finland

BACKGROUND

 The aim of the national health care quality registers is to monitor, assess, and improve the quality of care. The information utilized in quality registers must be of high quality to ensure that the information produced by the registers is reliable and useful. In Finland, one of the key sources of information for the quality registers is the national Kanta services.

OBJECTIVES

 The objective of the study was to increase understanding of the significance of information quality for the secondary use of the information in the national health care quality registers and to provide information on whether the information quality of the national Kanta services supports the information needs of the national quality registers, and how information quality should be developed.

METHODS

 The research data were collected by interviewing six experts responsible for national health care quality registers, and it was analyzed using theory-driven qualitative content analysis based on the DeLone and McLean model.

RESULTS

 Based on the results, the relevance of the information in the Kanta services met the information needs of the national quality registers. However, due to the limited amount of structured information and deficiencies in the completeness of the information, relevant information could not be fully utilized. Deficiencies in information quality posed challenges in information retrieval and hindered drawing conclusions in reporting. Challenges in information quality did not diminish the intention to use the information when information was considered relevant. Solutions to improve information quality included structuring, development of documentation practices, patient information systems and quality assurance, as well as collaboration among stakeholders.

CONCLUSION

 The Kanta services' information is relevant for the national health care quality registers, but developing the quality of the information, especially in terms of structures and completeness, is the key to fully enabling the secondary use of this information.

Investigation on the preferences for data quality assessment indicators of electronic health records: user-oriented perspective

Objectives

This study aims to investigate whether different types of electronic health record (EHR) users have distinct preferences for data quality assessment indicators (DQAI) and explore how these preferences can guide the enhancement of EHR systems and the optimization of related policies.

Materials and Methods

High-frequency indicators were identified by a systematic literature review to construct a DQAI system, which was assessed by a user-oriented investigation involving doctors, nurses, hospital supervisors, and clinical researchers. The entropy weight method and fuzzy comprehensive evaluation model were employed for the system comprehensive evaluation. Exploratory factor analysis was used to construct dimensions, and visualization analysis was utilized to explore preferences at both the indicator and dimension levels.

Results

Sixteen indicators were identified to construct the DQAI system and grouped into 2 dimensions: structural and relational. The DQAI system achieved a comprehensive evaluation score of 90.445, corresponding to a "very important" membership level (62.5%). Doctors and nurses exhibited a higher score mean (4.43-4.66 out of 5) than supervisors (3.73-4.55 out of 5). Researchers emphasized credibility, with a score mean of 4.79 out of 5.

Discussion

The findings reveal that different types of EHR users exhibit distinct preferences for the DQAI at both indicator and dimension levels. Doctors and nurses thought that all indicators were important, clinical researchers emphasized credibility, and supervisors focused mainly on accuracy. Indicators in the relational dimension were generally more valued than structural ones. Doctors and nurses prioritized indicators of relational dimension, while researchers and supervisors leaned towards indicators of structural dimension. These insights suggest that tailored approaches in EHR system development and policy-making could enhance EHR data quality.

Conclusion

This study underscores the importance of user-centered approaches in optimizing EHR systems, highlighting diverse user preferences at both indicator and dimension levels.

Development and initial validation of a data quality evaluation tool in obstetrics real-world data through HL7-FHIR interoperable Bayesian networks and expert rules

Background

The increasing prevalence of electronic health records (EHRs) in healthcare systems globally has underscored the importance of data quality for clinical decision-making and research, particularly in obstetrics. High-quality data is vital for an accurate representation of patient populations and to avoid erroneous healthcare decisions. However, existing studies have highlighted significant challenges in EHR data quality, necessitating innovative tools and methodologies for effective data quality assessment and improvement.

Objective

This article addresses the critical need for data quality evaluation in obstetrics by developing a novel tool. The tool utilizes Health Level 7 (HL7) Fast Healthcare Interoperable Resources (FHIR) standards in conjunction with Bayesian Networks and expert rules, offering a novel approach to assessing data quality in real-world obstetrics data.

Methods

A harmonized framework focusing on completeness, plausibility, and conformance underpins our methodology. We employed Bayesian networks for advanced probabilistic modeling, integrated outlier detection methods, and a rule-based system grounded in domain-specific knowledge. The development and validation of the tool were based on obstetrics data from 9 Portuguese hospitals, spanning the years 2019-2020.

Results

The developed tool demonstrated strong potential for identifying data quality issues in obstetrics EHRs. Bayesian networks used in the tool showed high performance for various features with area under the receiver operating characteristic curve (AUROC) between 75% and 97%. The tool's infrastructure and interoperable format as a FHIR Application Programming Interface (API) enables a possible deployment of a real-time data quality assessment in obstetrics settings. Our initial assessments show promised, even when compared with physicians' assessment of real records, the tool can reach AUROC of 88%, depending on the threshold defined.

Discussion

Our results also show that obstetrics clinical records are difficult to assess in terms of quality and assessments like ours could benefit from more categorical approaches of ranking between bad and good quality.

Conclusion

This study contributes significantly to the field of EHR data quality assessment, with a specific focus on obstetrics. The combination of HL7-FHIR interoperability, machine learning techniques, and expert knowledge presents a robust, adaptable solution to the challenges of healthcare data quality. Future research should explore tailored data quality evaluations for different healthcare contexts, as well as further validation of the tool capabilities, enhancing the tool's utility across diverse medical domains.

Systematic data quality assessment of electronic health record data to evaluate study-specific fitness: Report from the PRESERVE research study

Study-specific data quality testing is an essential part of minimizing analytic errors, particularly for studies making secondary use of clinical data. We applied a systematic and reproducible approach for study-specific data quality testing to the analysis plan for PRESERVE, a 15-site, EHR-based observational study of chronic kidney disease in children. This approach integrated widely adopted data quality concepts with healthcare-specific evaluation methods. We implemented two rounds of data quality assessment. The first produced high-level evaluation using aggregate results from a distributed query, focused on cohort identification and main analytic requirements. The second focused on extended testing of row-level data centralized for analysis. We systematized reporting and cataloguing of data quality issues, providing institutional teams with prioritized issues for resolution. We tracked improvements and documented anomalous data for consideration during analyses. The checks we developed identified 115 and 157 data quality issues in the two rounds, involving completeness, data model conformance, cross-variable concordance, consistency, and plausibility, extending traditional data quality approaches to address more complex stratification and temporal patterns. Resolution efforts focused on higher priority issues, given finite study resources. In many cases, institutional teams were able to correct data extraction errors or obtain additional data, avoiding exclusion of 2 institutions entirely and resolving 123 other gaps. Other results identified complexities in measures of kidney function, bearing on the study's outcome definition. Where limitations such as these are intrinsic to clinical data, the study team must account for them in conducting analyses. This study rigorously evaluated fitness of data for intended use. The framework is reusable and built on a strong theoretical underpinning. Significant data quality issues that would have otherwise delayed analyses or made data unusable were addressed. This study highlights the need for teams combining subject-matter and informatics expertise to address data quality when working with real world data.

The Preserving Kidney Function in Children With CKD (PRESERVE) Study: Rationale, Design, and Methods

Rationale & Objective

PRESERVE seeks to provide new knowledge to inform shared decision-making regarding blood pressure (BP) management for pediatric chronic kidney disease (CKD). PRESERVE will compare the effectiveness of alternative strategies for monitoring and treating hypertension on preserving kidney function; expand the National Patient-Centered Clinical Research Network (PCORnet) common data model by adding pediatric- and kidney-specific variables and linking electronic health record data to other kidney disease databases; and assess the lived experiences of patients related to BP management.

Study Design

Multicenter retrospective cohort study (clinical outcomes) and cross-sectional study (patient-reported outcomes [PROs]).

Setting & Participants

PRESERVE will include approximately 20,000 children between January 2009-December 2022 with mild-moderate CKD from 15 health care institutions that participate in 6 PCORnet Clinical Research Networks (PEDSnet, STAR, GPC, PaTH, CAPRiCORN, and OneFlorida+). The inclusion criteria were ≥1 nephrologist visit and ≥2 estimated glomerular filtration rate (eGFR) values in the range of 30 to <90 mL/min/1.73 m2 separated by ≥90 days without an intervening value ≥90 mL/min/1.73 m2 and no prior dialysis or kidney transplant.

Exposures

BP measurements (clinic-based and 24-hour ambulatory BP); urine protein; and antihypertensive treatment by therapeutic class.

Outcomes

The primary outcome is a composite event of a 50% reduction in eGFR, eGFR of <15 mL/min/1.73 m2, long-term dialysis or kidney transplant. Secondary outcomes include change in eGFR, adverse events, and PROs.

Analytical Approach

Longitudinal models for dichotomous (proportional hazards or accelerated failure time) and continuous (generalized linear mixed models) clinical outcomes; multivariable linear regression for PROs. We will evaluate heterogeneity of treatment effect by CKD etiology and degree of proteinuria and will examine variation in hypertension management and outcomes based on socio-demographics.

Limitations

Causal inference limited by observational analyses.

Conclusions

PRESERVE will leverage the PCORnet infrastructure to conduct large-scale observational studies that address BP management knowledge gaps for pediatric CKD, focusing on outcomes that are meaningful to patients.

Plain-Language Summary

Hypertension is a major modifiable contributor to loss of kidney function in chronic kidney disease (CKD). The purpose of PRESERVE is to provide evidence to inform shared decision-making regarding blood pressure management for children with CKD. PRESERVE is a consortium of 16 health care institutions in PCORnet, the National Patient-Centered Clinical Research Network, and includes electronic health record data for >19,000 children with CKD. PRESERVE will (1) expand the PCORnet infrastructure for research in pediatric CKD by adding kidney-specific variables and linking electronic health record data to other kidney disease databases; (2) compare the effectiveness of alternative strategies for monitoring and treating hypertension on preserving kidney function; and (3) assess the lived experiences of patients and caregivers related to blood pressure management.

Thought Leader Perspectives on the Benefits, Barriers, and Enablers for Routinely Collected Electronic Health Data to Support Professional Development: Qualitative Study

BACKGROUND

Hospitals routinely collect large amounts of administrative data such as length of stay, 28-day readmissions, and hospital-acquired complications; yet, these data are underused for continuing professional development (CPD). First, these clinical indicators are rarely reviewed outside of existing quality and safety reporting. Second, many medical specialists view their CPD requirements as time-consuming, having minimal impact on practice change and improving patient outcomes. There is an opportunity to build new user interfaces based on these data, designed to support individual and group reflection. Data-informed reflective practice has the potential to generate new insights about performance, bridging the gap between CPD and clinical practice.

OBJECTIVE

This study aims to understand why routinely collected administrative data have not yet become widely used to support reflective practice and lifelong learning.

METHODS

We conducted semistructured interviews (N=19) with thought leaders from a range of backgrounds, including clinicians, surgeons, chief medical officers, information and communications technology professionals, informaticians, researchers, and leaders from related industries. Interviews were thematically analyzed by 2 independent coders.

RESULTS

Respondents identified visibility of outcomes, peer comparison, group reflective discussions, and practice change as potential benefits. The key barriers included legacy technology, distrust with data quality, privacy, data misinterpretation, and team culture. Respondents suggested recruiting local champions for co-design, presenting data for understanding rather than information, coaching by specialty group leaders, and timely reflection linked to CPD as enablers to successful implementation.

CONCLUSIONS

Overall, there was consensus among thought leaders, bringing together insights from diverse backgrounds and medical jurisdictions. We found that clinicians are interested in repurposing administrative data for professional development despite concerns with underlying data quality, privacy, legacy technology, and visual presentation. They prefer group reflection led by supportive specialty group leaders, rather than individual reflection. Our findings provide novel insights into the specific benefits, barriers, and benefits of potential reflective practice interfaces based on these data sets. They can inform the design of new models of in-hospital reflection linked to the annual CPD planning-recording-reflection cycle.

Implementation of Gender Identity and Assigned Sex at Birth Data Collection in Electronic Health Records: Where Are We Now?

In 2015, the United States Department of Health and Human Services instantiated rules mandating the inclusion of sexual orientation and gender identity (SO/GI) data fields for systems certified under Stage 3 of the Meaningful Use of Electronic Health Records (EHR) program. To date, no published assessments have benchmarked implementation penetration and data quality. To establish a benchmark for a U.S. health system collection of gender identity and sex assigned at birth, we analyzed one urban academic health center's EHR data; specifically, the records of patients with unplanned hospital admissions during 2020 (N = 49,314). Approximately one-quarter of patient records included gender identity data, and one percent of them indicated a transgender or nonbinary (TGNB) status. Data quality checks suggested limited provider literacy around gender identity as well as limited provider and patient comfort levels with gender identity disclosures. Improvements are needed in both provider and patient literacy and comfort around gender identity in clinical settings. To include TGNB populations in informatics-based research, additional novel approaches, such as natural language processing, may be needed for more comprehensive and representative TGNB cohort discovery. Community and stakeholder engagement around gender identity data collection and health research will likely improve these implementation efforts.