A Survey of Clinicians' Views of the Utility of Large Language Models

OBJECTIVES

 Large language models (LLMs) like Generative pre-trained transformer (ChatGPT) are powerful algorithms that have been shown to produce human-like text from input data. Several potential clinical applications of this technology have been proposed and evaluated by biomedical informatics experts. However, few have surveyed health care providers for their opinions about whether the technology is fit for use.

METHODS

 We distributed a validated mixed-methods survey to gauge practicing clinicians' comfort with LLMs for a breadth of tasks in clinical practice, research, and education, which were selected from the literature.

RESULTS

 A total of 30 clinicians fully completed the survey. Of the 23 tasks, 16 were rated positively by more than 50% of the respondents. Based on our qualitative analysis, health care providers considered LLMs to have excellent synthesis skills and efficiency. However, our respondents had concerns that LLMs could generate false information and propagate training data bias.Our survey respondents were most comfortable with scenarios that allow LLMs to function in an assistive role, like a physician extender or trainee.

CONCLUSION

 In a mixed-methods survey of clinicians about LLM use, health care providers were encouraging of having LLMs in health care for many tasks, and especially in assistive roles. There is a need for continued human-centered development of both LLMs and artificial intelligence in general.

Selection, optimization and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse US populations

Polygenic risk scores (PRSs) have improved in predictive performance, but several challenges remain to be addressed before PRSs can be implemented in the clinic, including reduced predictive performance of PRSs in diverse populations, and the interpretation and communication of genetic results to both providers and patients. To address these challenges, the National Human Genome Research Institute-funded Electronic Medical Records and Genomics (eMERGE) Network has developed a framework and pipeline for return of a PRS-based genome-informed risk assessment to 25,000 diverse adults and children as part of a clinical study. From an initial list of 23 conditions, ten were selected for implementation based on PRS performance, medical actionability and potential clinical utility, including cardiometabolic diseases and cancer. Standardized metrics were considered in the selection process, with additional consideration given to strength of evidence in African and Hispanic populations. We then developed a pipeline for clinical PRS implementation (score transfer to a clinical laboratory, validation and verification of score performance), and used genetic ancestry to calibrate PRS mean and variance, utilizing genetically diverse data from 13,475 participants of the All of Us Research Program cohort to train and test model parameters. Finally, we created a framework for regulatory compliance and developed a PRS clinical report for return to providers and for inclusion in an additional genome-informed risk assessment. The initial experience from eMERGE can inform the approach needed to implement PRS-based testing in diverse clinical settings.

Planning for Actionable Precision Medicine

Genome-guided precision medicine applies consensus recommendations to the care of patients with particular genetic variants. As electronic health records begin to include patients' genomic data, recommendations will be formulated at an increasing rate. This study examined recommendations related to the current list of 73 actionable genes compiled by the American College of Medical Genetics and Genomics and found that conditions fall generally into five classes (cardiovascular, medication interactions, metabolic, neoplastic, and structural), with recommendations falling into seven categories (actions or circumstances to avoid, evaluation of relatives at risk, pregnancy management, prevention of primary manifestations, prevention of secondary complications, surveillance, and treatment of manifestations). This study represents a first step in facilitating automated, scalable clinical decision support and provides direction on formal representation of the contexts and actions for clinical recommendations derived from genome-informed learning health systems.

Semantically oriented EHR navigation with a patient specific knowledge base and a clinical context ontology

Widespread adoption of electronic health records (EHR) in the U.S. has been followed by unintended consequences, overexposing clinicians to widely reported EHR limitations. As an attempt to fixing the EHR, we propose the use of a clinical context ontology (CCO), applied to turn implicit contextual statements into formally represented data in the form of concept-relationship-concept tuples. These tuples form what we call a patient specific knowledge base (PSKB), a collection of formally defined tuples containing facts about the patient's care context. We report the process to create a CCO, which guides annotation of structured and narrative patient data to produce a PSKB. We also present an application of our PSKB using real patient data displayed on a semantically oriented patient summary to improve EHR navigation. Our approach can potentially save precious time spent by clinicians using today's EHRs, by showing a chronological view of the patient's record along with contextual statements needed for care decisions with minimum effort. We propose several other applications of a PSKB to improve multiple EHR functions to guide future research.

Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools

Objectives

To compare how clinical researchers generate data-driven hypotheses with a visual interactive analytic tool (VIADS, a visual interactive analysis tool for filtering and summarizing large datasets coded with hierarchical terminologies) or other tools.

Methods

We recruited clinical researchers and separated them into "experienced" and "inexperienced" groups. Participants were randomly assigned to a VIADS or control group within the groups. Each participant conducted a remote 2-hour study session for hypothesis generation with the same study facilitator on the same datasets by following a think-aloud protocol. Screen activities and audio were recorded, transcribed, coded, and analyzed. Hypotheses were evaluated by seven experts on their validity, significance, and feasibility. We conducted multilevel random effect modeling for statistical tests.

Results

Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid. The VIADS and control groups generated a similar number of hypotheses. The VIADS group took a significantly shorter time to generate one hypothesis (e.g., among inexperienced clinical researchers, 258 s versus 379 s, p = 0.046, power = 0.437, ICC = 0.15). The VIADS group received significantly lower ratings than the control group on feasibility and the combination rating of validity, significance, and feasibility.

Conclusion

The role of VIADS in hypothesis generation seems inconclusive. The VIADS group took a significantly shorter time to generate each hypothesis. However, the combined validity, significance, and feasibility ratings of their hypotheses were significantly lower. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development.

Beyond the override: Using evidence of previous drug tolerance to suppress drug allergy alerts; a retrospective study of opioid alerts

OBJECTIVE

Despite the extensive literature exploring alert fatigue, most studies have focused on describing the phenomenon, but not on fixing it. The authors aimed to identify data useful to avert clinically irrelevant alerts to inform future research on clinical decision support (CDS) design.

METHODS

We conducted a retrospective observational study of opioid drug allergy alert (DAA) overrides for the calendar year of 2019 at a large academic medical center, to identify data elements useful to find irrelevant alerts to be averted.

RESULTS

Overall, 227,815 DAAs were fired in 2019, with an override rate of 91 % (n = 208196). Opioids represented nearly two-thirds of these overrides (n = 129063; 62 %) and were the drug class with the highest override rate (96 %). On average, 29 opioid DAAs were overridden per patient. While most opioid alerts (97.1 %) are fired for a possible match (the drug class of the allergen matches the drug class of the prescribed drug), they are overridden significantly less frequently for definite match (exact match between allergen and prescribed drug) (88 % vs. 95.9 %, p < 0.001). When comparing the triggering drug with previously administered drugs, override rates were equally high for both definite match (95.9 %), no match (95.5 %), and possible match (95.1 %). Likewise, when comparing to home medications, overrides were excessively high for possible match (96.3 %), no match (96 %), and definite match (94.4 %).

CONCLUSION

We estimate that 74.5% of opioid DAAs (46.4% of all DAAs) at our institution could be relatively safely averted, since they either have a definite match for previous inpatient administrations suggesting drug tolerance or are fired as possible match with low risk of cross-sensitivity. Future research should focus on identifying other relevant data elements ideally with automated methods and use of emerging standards to empower CDS systems to suppress false-positive alerts while avoiding safety hazards.

Data-driven hypothesis generation among inexperienced clinical researchers: A comparison of secondary data analyses with visualization (VIADS) and other tools

Objectives

To compare how clinical researchers generate data-driven hypotheses with a visual interactive analytic tool (VIADS, a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies) or other tools.

Methods

We recruited clinical researchers and separated them into "experienced" and "inexperienced" groups. Participants were randomly assigned to a VIADS or control group within the groups. Each participant conducted a remote 2-hour study session for hypothesis generation with the same study facilitator on the same datasets by following a think-aloud protocol. Screen activities and audio were recorded, transcribed, coded, and analyzed. Hypotheses were evaluated by seven experts on their validity, significance, and feasibility. We conducted multilevel random effect modeling for statistical tests.

Results

Eighteen participants generated 227 hypotheses, of which 147 (65%) were valid. The VIADS and control groups generated a similar number of hypotheses. The VIADS group took a significantly shorter time to generate one hypothesis (e.g., among inexperienced clinical researchers, 258 seconds versus 379 seconds, p = 0.046, power = 0.437, ICC = 0.15). The VIADS group received significantly lower ratings than the control group on feasibility and the combination rating of validity, significance, and feasibility.

Conclusion

The role of VIADS in hypothesis generation seems inconclusive. The VIADS group took a significantly shorter time to generate each hypothesis. However, the combined validity, significance, and feasibility ratings of their hypotheses were significantly lower. Further characterization of hypotheses, including specifics on how they might be improved, could guide future tool development.

How do clinical researchers generate data-driven scientific hypotheses? Cognitive events using think-aloud protocol

Objectives

This study aims to identify the cognitive events related to information use (e.g., "Analyze data", "Seek connection") during hypothesis generation among clinical researchers. Specifically, we describe hypothesis generation using cognitive event counts and compare them between groups.

Methods

The participants used the same datasets, followed the same scripts, used VIADS (a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies) or other analytical tools (as control) to analyze the datasets, and came up with hypotheses while following the think-aloud protocol. Their screen activities and audio were recorded and then transcribed and coded for cognitive events.

Results

The VIADS group exhibited the lowest mean number of cognitive events per hypothesis and the smallest standard deviation. The experienced clinical researchers had approximately 10% more valid hypotheses than the inexperienced group. The VIADS users among the inexperienced clinical researchers exhibit a similar trend as the experienced clinical researchers in terms of the number of cognitive events and their respective percentages out of all the cognitive events. The highest percentages of cognitive events in hypothesis generation were "Using analysis results" (30%) and "Seeking connections" (23%).

Conclusion

VIADS helped inexperienced clinical researchers use fewer cognitive events to generate hypotheses than the control group. This suggests that VIADS may guide participants to be more structured during hypothesis generation compared with the control group. The results provide evidence to explain the shorter average time needed by the VIADS group in generating each hypothesis.

A metadata framework for computational phenotypes

With the burgeoning development of computational phenotypes, it is increasingly difficult to identify the right phenotype for the right tasks. This study uses a mixed-methods approach to develop and evaluate a novel metadata framework for retrieval of and reusing computational phenotypes. Twenty active phenotyping researchers from 2 large research networks, Electronic Medical Records and Genomics and Observational Health Data Sciences and Informatics, were recruited to suggest metadata elements. Once consensus was reached on 39 metadata elements, 47 new researchers were surveyed to evaluate the utility of the metadata framework. The survey consisted of 5-Likert multiple-choice questions and open-ended questions. Two more researchers were asked to use the metadata framework to annotate 8 type-2 diabetes mellitus phenotypes. More than 90% of the survey respondents rated metadata elements regarding phenotype definition and validation methods and metrics positively with a score of 4 or 5. Both researchers completed annotation of each phenotype within 60 min. Our thematic analysis of the narrative feedback indicates that the metadata framework was effective in capturing rich and explicit descriptions and enabling the search for phenotypes, compliance with data standards, and comprehensive validation metrics. Current limitations were its complexity for data collection and the entailed human costs.

Selection, optimization, and validation of ten chronic disease polygenic risk scores for clinical implementation in diverse populations

Polygenic risk scores (PRS) have improved in predictive performance supporting their use in clinical practice. Reduced predictive performance of PRS in diverse populations can exacerbate existing health disparities. The NHGRI-funded eMERGE Network is returning a PRS-based genome-informed risk assessment to 25,000 diverse adults and children. We assessed PRS performance, medical actionability, and potential clinical utility for 23 conditions. Standardized metrics were considered in the selection process with additional consideration given to strength of evidence in African and Hispanic populations. Ten conditions were selected with a range of high-risk thresholds: atrial fibrillation, breast cancer, chronic kidney disease, coronary heart disease, hypercholesterolemia, prostate cancer, asthma, type 1 diabetes, obesity, and type 2 diabetes. We developed a pipeline for clinical PRS implementation, used genetic ancestry to calibrate PRS mean and variance, created a framework for regulatory compliance, and developed a PRS clinical report. eMERGE's experience informs the infrastructure needed to implement PRS-based implementation in diverse clinical settings.

Development, validation, and usage of metrics to evaluate the quality of clinical research hypotheses

Objectives

Metrics and instruments can provide guidance for clinical researchers to assess their potential research projects at an early stage before significant investment. Furthermore, metrics can also provide structured criteria for peer reviewers to assess others' clinical research manuscripts or grant proposals. This study aimed to develop, test, validate, and use evaluation metrics and instruments to accurately, consistently, and conveniently assess the quality of scientific hypotheses for clinical research projects.

Materials and Methods

Metrics development went through iterative stages, including literature review, metrics and instrument development, internal and external testing and validation, and continuous revisions in each stage based on feedback. Furthermore, two experiments were conducted to determine brief and comprehensive versions of the instrument.

Results

The brief version of the instrument contained three dimensions: validity, significance, and feasibility. The comprehensive version of metrics included novelty, clinical relevance, potential benefits and risks, ethicality, testability, clarity, interestingness, and the three dimensions of the brief version. Each evaluation dimension included 2 to 5 subitems to evaluate the specific aspects of each dimension. For example, validity included clinical validity and scientific validity. The brief and comprehensive versions of the instruments included 12 and 39 subitems, respectively. Each subitem used a 5-point Likert scale.

Conclusion

The validated brief and comprehensive versions of metrics can provide standardized, consistent, and generic measurements for clinical research hypotheses, allow clinical researchers to prioritize their research ideas systematically, objectively, and consistently, and can be used as a tool for quality assessment during the peer review process.

Returning integrated genomic risk and clinical recommendations: The eMERGE study

PURPOSE

Assessing the risk of common, complex diseases requires consideration of clinical risk factors as well as monogenic and polygenic risks, which in turn may be reflected in family history. Returning risks to individuals and providers may influence preventive care or use of prophylactic therapies for those individuals at high genetic risk.

METHODS

To enable integrated genetic risk assessment, the eMERGE (electronic MEdical Records and GEnomics) network is enrolling 25,000 diverse individuals in a prospective cohort study across 10 sites. The network developed methods to return cross-ancestry polygenic risk scores, monogenic risks, family history, and clinical risk assessments via a genome-informed risk assessment (GIRA) report and will assess uptake of care recommendations after return of results.

RESULTS

GIRAs include summary care recommendations for 11 conditions, education pages, and clinical laboratory reports. The return of high-risk GIRA to individuals and providers includes guidelines for care and lifestyle recommendations. Assembling the GIRA required infrastructure and workflows for ingesting and presenting content from multiple sources. Recruitment began in February 2022.

CONCLUSION

Return of a novel report for communicating monogenic, polygenic, and family history-based risk factors will inform the benefits of integrated genetic risk assessment for routine health care.

Do electronic health record systems "dumb down" clinicians?

A panel sponsored by the American College of Medical Informatics (ACMI) at the 2021 AMIA Symposium addressed the provocative question: "Are Electronic Health Records dumbing down clinicians?" After reviewing electronic health record (EHR) development and evolution, the panel discussed how EHR use can impair care delivery. Both suboptimal functionality during EHR use and longer-term effects outside of EHR use can reduce clinicians' efficiencies, reasoning abilities, and knowledge. Panel members explored potential solutions to problems discussed. Progress will require significant engagement from clinician-users, educators, health systems, commercial vendors, regulators, and policy makers. Future EHR systems must become more user-focused and scalable and enable providers to work smarter to deliver improved care.

A visual analytic tool, VIADS, to assist the hypothesis generation process in clinical research—A usability study using mixed methods (Preprint)

BACKGROUND

Visualization can be a powerful tool for comprehending datasets, especially when they can be represented via hierarchical structures. Enhanced comprehension can facilitate the development of scientific hypotheses. However, the inclusion of excessive data can make a visualization overwhelming.

OBJECTIVE

We developed a Visual Interactive Analytic tool for filtering and summarizing large health Data Sets (VIADS) coded with hierarchical terminologies. In this study, we evaluated the usability of VIADS for visualizing data sets of patient diagnoses and procedures coded in the International Classification of Diseases, ninth revisions, clinical modification (ICD-9-CM).

METHODS

We used mixed methods in the study. A group of 12 clinical researchers participated in the generation of data-driven hypotheses using the same datasets and time frame (a 1-hour training session and a 2-hour study session), utilizing VIADS via the think-aloud protocol. The audio and screen activities were recorded remotely. A modified version of the System Usability Scale (SUS) survey and a brief survey with open-ended questions were administered after the study to assess the usability of VIADS and verify their intense usage experience of VIADS.

RESULTS

The range of SUS scores was 37.5 - 87.5. The mean SUS score for VIADS was 71.88 (out of a possible 100, standard deviation: 14.62 ), and the median SUS was 75. The participants unanimously agreed that VIADS offers new perspectives on data sets (100%), while 75% agreed that VIADS facilitates understanding, presentation, and interpretation of underlying datasets. The comments on the utility of VIADS were positive and aligned well with the design objectives of VIADS. The answers to the open-ended questions in the modified SUS provided specific suggestions regarding potential improvements in VIADS, and identified problems in usability were used to update the tool.

CONCLUSIONS

This usability study demonstrates that VIADS is a usable tool for analyzing secondary datasets with good average usability, SUS score, and favorable utility. Currently, VIADS accepts datasets with hierarchical codes and their corresponding frequencies. Consequently, only specific types of use cases are supported by the analytical results. Participants agreed, however, that VIADS provides new perspectives on datasets and is relatively easy to use. The functionalities mostly appreciated by participants were VIADS' ability to filter, summarize, compare, and visualize data.

CLINICALTRIAL

INTERNATIONAL REGISTERED REPORT

RR2-10.2196/39414.

A Systematic Approach to Configuring MetaMap for Optimal Performance

BACKGROUND

 MetaMap is a valuable tool for processing biomedical texts to identify concepts. Although MetaMap is highly configurative, configuration decisions are not straightforward.

OBJECTIVE

 To develop a systematic, data-driven methodology for configuring MetaMap for optimal performance.

METHODS

 MetaMap, the word2vec model, and the phrase model were used to build a pipeline. For unsupervised training, the phrase and word2vec models used abstracts related to clinical decision support as input. During testing, MetaMap was configured with the default option, one behavior option, and two behavior options. For each configuration, cosine and soft cosine similarity scores between identified entities and gold-standard terms were computed for 40 annotated abstracts (422 sentences). The similarity scores were used to calculate and compare the overall percentages of exact matches, similar matches, and missing gold-standard terms among the abstracts for each configuration. The results were manually spot-checked. The precision, recall, and F-measure (β =1) were calculated.

RESULTS

 The percentages of exact matches and missing gold-standard terms were 0.6-0.79 and 0.09-0.3 for one behavior option, and 0.56-0.8 and 0.09-0.3 for two behavior options, respectively. The percentages of exact matches and missing terms for soft cosine similarity scores exceeded those for cosine similarity scores. The average precision, recall, and F-measure were 0.59, 0.82, and 0.68 for exact matches, and 1.00, 0.53, and 0.69 for missing terms, respectively.

CONCLUSION

 We demonstrated a systematic approach that provides objective and accurate evidence guiding MetaMap configurations for optimizing performance. Combining objective evidence and the current practice of using principles, experience, and intuitions outperforms a single strategy in MetaMap configurations. Our methodology, reference codes, measurements, results, and workflow are valuable references for optimizing and configuring MetaMap.

A research agenda to support the development and implementation of genomics-based clinical informatics tools and resources

OBJECTIVE

The Genomic Medicine Working Group of the National Advisory Council for Human Genome Research virtually hosted its 13th genomic medicine meeting titled "Developing a Clinical Genomic Informatics Research Agenda". The meeting's goal was to articulate a research strategy to develop Genomics-based Clinical Informatics Tools and Resources (GCIT) to improve the detection, treatment, and reporting of genetic disorders in clinical settings.

MATERIALS AND METHODS

Experts from government agencies, the private sector, and academia in genomic medicine and clinical informatics were invited to address the meeting's goals. Invitees were also asked to complete a survey to assess important considerations needed to develop a genomic-based clinical informatics research strategy.

RESULTS

Outcomes from the meeting included identifying short-term research needs, such as designing and implementing standards-based interfaces between laboratory information systems and electronic health records, as well as long-term projects, such as identifying and addressing barriers related to the establishment and implementation of genomic data exchange systems that, in turn, the research community could help address.

DISCUSSION

Discussions centered on identifying gaps and barriers that impede the use of GCIT in genomic medicine. Emergent themes from the meeting included developing an implementation science framework, defining a value proposition for all stakeholders, fostering engagement with patients and partners to develop applications under patient control, promoting the use of relevant clinical workflows in research, and lowering related barriers to regulatory processes. Another key theme was recognizing pervasive biases in data and information systems, algorithms, access, value, and knowledge repositories and identifying ways to resolve them.

Development and validation of a trans-ancestry polygenic risk score for type 2 diabetes in diverse populations

BACKGROUND

Type 2 diabetes (T2D) is a worldwide scourge caused by both genetic and environmental risk factors that disproportionately afflicts communities of color. Leveraging existing large-scale genome-wide association studies (GWAS), polygenic risk scores (PRS) have shown promise to complement established clinical risk factors and intervention paradigms, and improve early diagnosis and prevention of T2D. However, to date, T2D PRS have been most widely developed and validated in individuals of European descent. Comprehensive assessment of T2D PRS in non-European populations is critical for equitable deployment of PRS to clinical practice that benefits global populations.

METHODS

We integrated T2D GWAS in European, African, and East Asian populations to construct a trans-ancestry T2D PRS using a newly developed Bayesian polygenic modeling method, and assessed the prediction accuracy of the PRS in the multi-ethnic Electronic Medical Records and Genomics (eMERGE) study (11,945 cases; 57,694 controls), four Black cohorts (5137 cases; 9657 controls), and the Taiwan Biobank (4570 cases; 84,996 controls). We additionally evaluated a post hoc ancestry adjustment method that can express the polygenic risk on the same scale across ancestrally diverse individuals and facilitate the clinical implementation of the PRS in prospective cohorts.

RESULTS

The trans-ancestry PRS was significantly associated with T2D status across the ancestral groups examined. The top 2% of the PRS distribution can identify individuals with an approximately 2.5-4.5-fold of increase in T2D risk, which corresponds to the increased risk of T2D for first-degree relatives. The post hoc ancestry adjustment method eliminated major distributional differences in the PRS across ancestries without compromising its predictive performance.

CONCLUSIONS

By integrating T2D GWAS from multiple populations, we developed and validated a trans-ancestry PRS, and demonstrated its potential as a meaningful index of risk among diverse patients in clinical settings. Our efforts represent the first step towards the implementation of the T2D PRS into routine healthcare.

The Roles of a Secondary Data Analytic Tool and Experience in Scientific Hypothesis Generation in Clinical Research: A Study Design

Background

Scientific hypothesis generation is a critical step in scientific research that determines the direction and impact of any investigation. Despite its vital role, we have limited knowledge of the process itself, thus hindering our ability to address some critical questions.

Objective

This study aims to answer the following questions: To what extent can secondary data analytics tools facilitate the generation of scientific hypotheses during clinical research? Are the processes similar in developing clinical diagnoses during clinical practice and developing scientific hypotheses for clinical research projects? Furthermore, this study explores the process of scientific hypothesis generation in the context of clinical research. It was designed to compare the role of VIADS, a visual interactive analysis tool for filtering and summarizing large data sets coded with hierarchical terminologies, and the experience levels of study participants during the scientific hypothesis generation process.

Methods

This manuscript introduces a study design. Experienced and inexperienced clinical researchers are being recruited since July 2021 to take part in this 2×2 factorial study, in which all participants use the same data sets during scientific hypothesis–generation sessions and follow predetermined scripts. The clinical researchers are separated into experienced or inexperienced groups based on predetermined criteria and are then randomly assigned into groups that use and do not use VIADS via block randomization. The study sessions, screen activities, and audio recordings of participants are captured. Participants use the think-aloud protocol during the study sessions. After each study session, every participant is given a follow-up survey, with participants using VIADS completing an additional modified System Usability Scale survey. A panel of clinical research experts will assess the scientific hypotheses generated by participants based on predeveloped metrics. All data will be anonymized, transcribed, aggregated, and analyzed.

Results

Data collection for this study began in July 2021. Recruitment uses a brief online survey. The preliminary results showed that study participants can generate a few to over a dozen scientific hypotheses during a 2-hour study session, regardless of whether they used VIADS or other analytics tools. A metric to more accurately, comprehensively, and consistently assess scientific hypotheses within a clinical research context has been developed.

Conclusions

The scientific hypothesis–generation process is an advanced cognitive activity and a complex process. Our results so far show that clinical researchers can quickly generate initial scientific hypotheses based on data sets and prior experience. However, refining these scientific hypotheses is a much more time-consuming activity. To uncover the fundamental mechanisms underlying the generation of scientific hypotheses, we need breakthroughs that can capture thinking processes more precisely.

International Registered Report Identifier (IRRID)

DERR1-10.2196/39414

The biomedical informatics short course at Woods Hole/Georgia: Training to support institutional change

The US National Library of Medicine’s Biomedical Informatics Short Course ran from 1992 to 2017, most of that time at the Marine Biological Laboratory in Woods Hole, Massachusetts. Its intention was to provide physicians, medical librarians and others engaged in health care with a basic understanding of the major topics in informatics so that they could return to their home institutions as “change agents”. Over the years, the course provided week-long, intense, morning-to-night experiences for some 1,350 students, consisting of lectures and hands-on project development, taught by many luminaries in the field, not the least of which was Donald A.B. Lindberg M.D., who spoke on topics ranging from bioinformatics to national policy.

Synergies between centralized and federated approaches to data quality: a report from the national COVID cohort collaborative

OBJECTIVE

In response to COVID-19, the informatics community united to aggregate as much clinical data as possible to characterize this new disease and reduce its impact through collaborative analytics. The National COVID Cohort Collaborative (N3C) is now the largest publicly available HIPAA limited dataset in US history with over 6.4 million patients and is a testament to a partnership of over 100 organizations.

MATERIALS AND METHODS

We developed a pipeline for ingesting, harmonizing, and centralizing data from 56 contributing data partners using 4 federated Common Data Models. N3C data quality (DQ) review involves both automated and manual procedures. In the process, several DQ heuristics were discovered in our centralized context, both within the pipeline and during downstream project-based analysis. Feedback to the sites led to many local and centralized DQ improvements.

RESULTS

Beyond well-recognized DQ findings, we discovered 15 heuristics relating to source Common Data Model conformance, demographics, COVID tests, conditions, encounters, measurements, observations, coding completeness, and fitness for use. Of 56 sites, 37 sites (66%) demonstrated issues through these heuristics. These 37 sites demonstrated improvement after receiving feedback.

DISCUSSION

We encountered site-to-site differences in DQ which would have been challenging to discover using federated checks alone. We have demonstrated that centralized DQ benchmarking reveals unique opportunities for DQ improvement that will support improved research analytics locally and in aggregate.

CONCLUSION

By combining rapid, continual assessment of DQ with a large volume of multisite data, it is possible to support more nuanced scientific questions with the scale and rigor that they require.

Mapping 2 Decades of Research in Health Services Research, Health Policy, and Health Economics Journals

OBJECTIVE

To identify major research topics and exhibit trends in these topics in 15 health services research, health policy, and health economics journals over 2 decades.

DATA SOURCES

The study sample of 35,159 abstracts (1999-2020) were collected from PubMed for 15 journals.

STUDY DESIGN

The study used a 3-phase approach for text analyses: (1) developing the corpus of 40,618 references from PubMed (excluding 5459 of those without abstract or author information); (2) preprocessing and generating the term list using natural language processing to eliminate irrelevant textual data and identify important terms and phrases; (3) analyzing the preprocessed text data using latent semantic analysis, topic analyses, and multiple correspondence analysis.

PRINCIPAL FINDINGS

Application of analyses generated 16 major research topics: (1) implementation/intervention science; (2) HIV and women's health; (3) outcomes research and quality; (4) veterans/military studies; (5) provider/primary-care interventions; (6) geriatrics and formal/informal care; (7) policies and health outcomes; (8) medication treatment/therapy; (9) patient interventions; (10) health insurance legislation and policies; (11) public health policies; (12) literature reviews; (13) cost-effectiveness and economic evaluation; (14) cancer care; (15) workforce issues; and (16) socioeconomic status and disparities. The 2-dimensional map revealed that some journals have stronger associations with specific topics. Findings were not consistent with previous studies based on user perceptions.

CONCLUSION

Findings of this study can be used by the stakeholders of health services research, policy, and economics to develop future research agendas, target journal submissions, and generate interdisciplinary solutions by examining overlapping journals for particular topics.

Racial Disparity in Clinical Alert Overrides

The existence of systemic racism in US health care is widely recognized, but the role that informatics plays has received little attention. Clinical guidelines, which can incorporate implicit racial bias or be adhered to in racially disparate ways, are often the basis for clinical alerting systems. It is also possible that clinicians might be discriminatory in their response to alerts (for example, by deciding whether to agree or override the alert). We sought to study whether alert logic in our hospital uses patient race as part of its criteria and if alert override rates show any racial disparities. We obtained data on 5,120,114 alert events at the University of Alabama at Birmingham (UAB) Hospital and examined override the rates and reasons with respect to patient race. We found override rates of 82.27% and 81.30% for Black or African American patients and White patients, respectively. Some differences by alert were statistically significant but generally small. Override patterns varied by clinician but reasons given were generally not disparate, suggesting that if racist behavior is present, it is not widely systemic. However, the great variability in individual clinician behavior suggests that deeper analysis is warranted to determine whether disparities are indeed racist in nature.

The Biomedical Informatics Short Course at Woods Hole/Georgia: Training to Support Institutional Change

The U.S. National Library of Medicine's Biomedical Informatics Short Course ran from 1992 to 2017, most of that time at the Marine Biological Laboratory in Woods Hole, Massachusetts. Its intention was to provide physicians, medical librarians and others engaged in health care with a basic understanding of the major topics in informatics so that they could return to their home institutions as "change agents". Over the years, the course provided week-long, intense, morning-to-night experiences for some 1,350 students, consisting of lectures and hands-on project development, taught by many luminaries in the field, not the least of which was Donald A.B. Lindberg M.D., who spoke on topics ranging from bioinformatics to national policy.

Developing real-world evidence from real-world data: Transforming raw data into analytical datasets

Development of evidence-based practice requires practice-based evidence, which can be acquired through analysis of real-world data from electronic health records (EHRs). The EHR contains volumes of information about patients-physical measurements, diagnoses, exposures, and markers of health behavior-that can be used to create algorithms for risk stratification or to gain insight into associations between exposures, interventions, and outcomes. But to transform real-world data into reliable real-world evidence, one must not only choose the correct analytical methods but also have an understanding of the quality, detail, provenance, and organization of the underlying source data and address the differences in these characteristics across sites when conducting analyses that span institutions. This manuscript explores the idiosyncrasies inherent in the capture, formatting, and standardization of EHR data and discusses the clinical domain and informatics competencies required to transform the raw clinical, real-world data into high-quality, fit-for-purpose analytical data sets used to generate real-world evidence.

Opportunity for Genotype-Guided Prescribing Among Adult Patients in 11 US Health Systems

The value of utilizing a multigene pharmacogenetic panel to tailor pharmacotherapy is contingent on the prevalence of prescribed medications with an actionable pharmacogenetic association. The Clinical Pharmacogenetics Implementation Consortium (CPIC) has categorized over 35 gene-drug pairs as "level A," for which there is sufficiently strong evidence to recommend that genetic information be used to guide drug prescribing. The opportunity to use genetic information to tailor pharmacotherapy among adult patients was determined by elucidating the exposure to CPIC level A drugs among 11 Implementing Genomics In Practice Network (IGNITE)-affiliated health systems across the US. Inpatient and/or outpatient electronic-prescribing data were collected between January 1, 2011 and December 31, 2016 for patients ≥ 18 years of age who had at least one medical encounter that was eligible for drug prescribing in a calendar year. A median of ~ 7.2 million adult patients was available for assessment of drug prescribing per year. From 2011 to 2016, the annual estimated prevalence of exposure to at least one CPIC level A drug prescribed to unique patients ranged between 15,719 (95% confidence interval (CI): 15,658-15,781) in 2011 to 17,335 (CI: 17,283-17,386) in 2016 per 100,000 patients. The estimated annual exposure to at least 2 drugs was above 7,200 per 100,000 patients in most years of the study, reaching an apex of 7,660 (CI: 7,632-7,687) per 100,000 patients in 2014. An estimated 4,748 per 100,000 prescribing events were potentially eligible for a genotype-guided intervention. Results from this study show that a significant portion of adults treated at medical institutions across the United States is exposed to medications for which genetic information, if available, should be used to guide prescribing.

Physicians' perceptions about narrative note sections format and content: A multi-specialty survey

OBJECTIVE

To assess physicians' perceptions about narrative note sections format and content commonly reported in visit notes to inform future research and EHR development.

METHODS

We conducted two online surveys with a multi-specialty panel of outpatient physicians from a large health system to collect their perceptions of the usefulness of three narrative formats and the relevance of content reported in the note sections history of present illness (HPI) and assessment and plan (AP). Survey questions were responded with a 7-point Likert scale and include two open-ended questions for comments on challenges and suggestions related to electronic clinical documentation.

RESULTS

Eighty-eight physicians completed the surveys. The most preferred format for HPI was story (i.e., coherent paragraph), followed by list without categories (i.e., non-categorized sentences) and list with categories (i.e., categorized sentences). The most preferred format for AP was list with categories, followed by story and list without categories. The most relevant type of content in HPI was temporal information and finding/condition. The most relevant type of content reported in AP was intervention and reasons and justifications. Challenges frequently mentioned include suboptimal note creation interfaces and bloated notes, and the most common suggestions for improvements are related to note entry facilitators and organizational improvements.

CONCLUSION

Physicians' input is extremely valuable to inform improvements to EHRs. More effective clinical documentation systems should include less intrusive, more intuitive and automated user interfaces for note creation, smarter autopoluation functionality and linkage between note content and data from other parts of the record.

Formal representation of patients' care context data: the path to improving the electronic health record

Objective

To develop a collection of concept-relationship-concept tuples to formally represent patients’ care context data to inform electronic health record (EHR) development.

Materials and Methods

We reviewed semantic relationships reported in the literature and developed a manual annotation schema. We used the initial schema to annotate sentences extracted from narrative note sections of cardiology, urology, and ear, nose, and throat (ENT) notes. We audio recorded ENT visits and annotated their parsed transcripts. We combined the results of each annotation into a consolidated set of concept-relationship-concept tuples. We then compared the tuples used within and across the multiple data sources.

Results

We annotated a total of 626 sentences. Starting with 8 relationships from the literature, we annotated 182 sentences from 8 inpatient consult notes (initial set of tuples = 43). Next, we annotated 232 sentences from 10 outpatient visit notes (enhanced set of tuples = 75). Then, we annotated 212 sentences from transcripts of 5 outpatient visits (final set of tuples = 82). The tuples from the visit transcripts covered 103 (74%) concepts documented in the notes of their respective visits. There were 20 (24%) tuples used across all data sources, 10 (12%) used only in inpatient notes, 15 (18%) used only in visit notes, and 7 (9%) used only in the visit transcripts.

Conclusions

We produced a robust set of 82 tuples useful to represent patients’ care context data. We propose several applications of our tuples to improve EHR navigation, data entry, learning health systems, and decision support.

A state-based approach to genomics for rare disease and population screening

PURPOSE

The Alabama Genomic Health Initiative (AGHI) is a state-funded effort to provide genomic testing. AGHI engages two distinct cohorts across the state of Alabama. One cohort includes children and adults with undiagnosed rare disease; a second includes an unselected adult population. Here we describe findings from the first 176 rare disease and 5369 population cohort AGHI participants.

METHODS

AGHI participants enroll in one of two arms of a research protocol that provides access to genomic testing results and biobank participation. Rare disease cohort participants receive genome sequencing to identify primary and secondary findings. Population cohort participants receive genotyping to identify pathogenic and likely pathogenic variants for actionable conditions.

RESULTS

Within the rare disease cohort, genome sequencing identified likely pathogenic or pathogenic variation in 20% of affected individuals. Within the population cohort, 1.5% of individuals received a positive genotyping result. The rate of genotyping results corroborated by reported personal or family history varied by gene.

CONCLUSIONS

AGHI demonstrates the ability to provide useful health information in two contexts: rare undiagnosed disease and population screening. This utility should motivate continued exploration of ways in which emerging genomic technologies might benefit broad populations.

Research informatics and the COVID-19 pandemic: Challenges, innovations, lessons learned, and recommendations

The recipients of NIH's Clinical and Translational Science Awards (CTSA) have worked for over a decade to build informatics infrastructure in support of clinical and translational research. This infrastructure has proved invaluable for supporting responses to the current COVID-19 pandemic through direct patient care, clinical decision support, training researchers and practitioners, as well as public health surveillance and clinical research to levels that could not have been accomplished without the years of ground-laying work by the CTSAs. In this paper, we provide a perspective on our COVID-19 work and present relevant results of a survey of CTSA sites to broaden our understanding of the key features of their informatics programs, the informatics-related challenges they have experienced under COVID-19, and some of the innovations and solutions they developed in response to the pandemic. Responses demonstrated increased reliance by healthcare providers and researchers on access to electronic health record (EHR) data, both for local needs and for sharing with other institutions and national consortia. The initial work of the CTSAs on data capture, standards, interchange, and sharing policies all contributed to solutions, best illustrated by the creation, in record time, of a national clinical data repository in the National COVID-19 Cohort Collaborative (N3C). The survey data support seven recommendations for areas of informatics and public health investment and further study to support clinical and translational research in the post-COVID-19 era.

The National COVID Cohort Collaborative (N3C): Rationale, design, infrastructure, and deployment

Objective

Coronavirus disease 2019 (COVID-19) poses societal challenges that require expeditious data and knowledge sharing. Though organizational clinical data are abundant, these are largely inaccessible to outside researchers. Statistical, machine learning, and causal analyses are most successful with large-scale data beyond what is available in any given organization. Here, we introduce the National COVID Cohort Collaborative (N3C), an open science community focused on analyzing patient-level data from many centers.

Materials and Methods

The Clinical and Translational Science Award Program and scientific community created N3C to overcome technical, regulatory, policy, and governance barriers to sharing and harmonizing individual-level clinical data. We developed solutions to extract, aggregate, and harmonize data across organizations and data models, and created a secure data enclave to enable efficient, transparent, and reproducible collaborative analytics.

Results

Organized in inclusive workstreams, we created legal agreements and governance for organizations and researchers; data extraction scripts to identify and ingest positive, negative, and possible COVID-19 cases; a data quality assurance and harmonization pipeline to create a single harmonized dataset; population of the secure data enclave with data, machine learning, and statistical analytics tools; dissemination mechanisms; and a synthetic data pilot to democratize data access.

Conclusions

The N3C has demonstrated that a multisite collaborative learning health network can overcome barriers to rapidly build a scalable infrastructure incorporating multiorganizational clinical data for COVID-19 analytics. We expect this effort to save lives by enabling rapid collaboration among clinicians, researchers, and data scientists to identify treatments and specialized care and thereby reduce the immediate and long-term impacts of COVID-19.

What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask

Coincident with the tsunami of COVID-19–related publications, there has been a surge of studies using real-world data, including those obtained from the electronic health record (EHR). Unfortunately, several of these high-profile publications were retracted because of concerns regarding the soundness and quality of the studies and the EHR data they purported to analyze. These retractions highlight that although a small community of EHR informatics experts can readily identify strengths and flaws in EHR-derived studies, many medical editorial teams and otherwise sophisticated medical readers lack the framework to fully critically appraise these studies. In addition, conventional statistical analyses cannot overcome the need for an understanding of the opportunities and limitations of EHR-derived studies. We distill here from the broader informatics literature six key considerations that are crucial for appraising studies utilizing EHR data: data completeness, data collection and handling (eg, transformation), data type (ie, codified, textual), robustness of methods against EHR variability (within and across institutions, countries, and time), transparency of data and analytic code, and the multidisciplinary approach. These considerations will inform researchers, clinicians, and other stakeholders as to the recommended best practices in reviewing manuscripts, grants, and other outputs from EHR-data derived studies, and thereby promote and foster rigor, quality, and reliability of this rapidly growing field.

Putting the "why" in "EHR": capturing and coding clinical cognition

Complaints about electronic health records, including information overload, note bloat, and alert fatigue, are frequent topics of discussion. Despite substantial effort by researchers and industry, complaints continue noting serious adverse effects on patient safety and clinician quality of life. I believe solutions are possible if we can add information to the record that explains the “why” of a patient’s care, such as relationships between symptoms, physical findings, diagnostic results, differential diagnoses, therapeutic plans, and goals. While this information may be present in clinical notes, I propose that we modify electronic health records to support explicit representation of this information using formal structure and controlled vocabularies. Such information could foster development of more situation-aware tools for data retrieval and synthesis. Informatics research is needed to understand what should be represented, how to capture it, and how to benefit those providing the information so that their workload is reduced.

PAGER-CoV: a comprehensive collection of pathways, annotated gene-lists and gene signatures for coronavirus disease studies

PAGER-CoV (http://discovery.informatics.uab.edu/PAGER-CoV/) is a new web-based database that can help biomedical researchers interpret coronavirus-related functional genomic study results in the context of curated knowledge of host viral infection, inflammatory response, organ damage, and tissue repair. The new database consists of 11 835 PAGs (Pathways, Annotated gene-lists, or Gene signatures) from 33 public data sources. Through the web user interface, users can search by a query gene or a query term and retrieve significantly matched PAGs with all the curated information. Users can navigate from a PAG of interest to other related PAGs through either shared PAG-to-PAG co-membership relationships or PAG-to-PAG regulatory relationships, totaling 19 996 993. Users can also retrieve enriched PAGs from an input list of COVID-19 functional study result genes, customize the search data sources, and export all results for subsequent offline data analysis. In a case study, we performed a gene set enrichment analysis (GSEA) of a COVID-19 RNA-seq data set from the Gene Expression Omnibus database. Compared with the results using the standard PAGER database, PAGER-CoV allows for more sensitive matching of known immune-related gene signatures. We expect PAGER-CoV to be invaluable for biomedical researchers to find molecular biology mechanisms and tailored therapeutics to treat COVID-19 patients.

Capturing Clinician Reasoning in Electronic Health Records: An Exploratory Study of Under-Treated Essential Hypertension

Monitoring response to antihypertensive medications is a frequent reason for outpatient visits. Blood pressure (BP) is often documented as elevated, but no change in medication occurs (Medication Non-adjustment or MNA). We studied the frequency of MNA, reasons for non-adjustment, how reasons (including reasons for patient nonadherence) were documented, and whether they could be represented in a clinical care context ontology. We examined 129 visit notes with MNA occurring in 80 cases (59%). We coded MNA as Conscious Maintenance (patient adherent but clinician continues therapy for stated reason), Nonadherence (clinician attributes BP elevation to patient nonadherence), and Finding Not Addressed (clinician does not indicate reasoning for MNA). We characterized Conscious Maintenance with 11 subcodes and Nonadherence with 6 subcodes. Our ontology successfully represented relationships between concepts and reasoning, supporting the feasibility of formal representation of clinical care contexts for patient care, decision support and research.

The anatomy of clinical documentation: an assessment and classification of narrative note sections format and content

Introduction. We systematically analyzed the most commonly used narrative note formats and content found in primary and specialty care visit notes to inform future research and electronic health record (EHR) development. Methods. We extracted data from the history of present illness (HPI) and impression and plan (IP) sections of 80 primary and specialty care visit notes. Two authors iteratively classified the format of the sections and compared the size of each section and the overall note size between primary and specialty care notes. We then annotated the content of these sections to develop a taxonomy of types of data communicated in the narrative note sections. Results. Both HPI and IP were significantly longer in primary care when compared to specialty care notes (HPI: n = 187 words, SD[130] vs. n = 119 words, SD [53]; p = 0.004 / IP: n = 270 words, SD [145] vs. n = 170 words, SD [101]; p < 0.001). Although we did not find a significant difference in the overall note size between the two groups, the proportion of HPI and IP content in relation to the total note size was significantly higher in primary care notes (40%, SD [13] vs. 28%, SD [11]; p < 0.001). We identified five combinations of format of HPI + IP sections respectively: (A) story + list with categories; (B) story + story; (C) list without categories + list with categories; (D) list with categories + list with categories; and (E) list with categories + story. HPI and IP content was significantly smaller in combination C compared to combination A (-172 words, [95% Conf. -326, -17.89]; p = 0.02). We identified seven taxa representing 45 different types of data: finding/condition documented (n = 14), intervention documented (n = 9), general descriptions and definitions (n = 7), temporal information (n = 6), reasons and justifications (n = 4), participants and settings (n = 4), and clinical documentation (n = 1). Conclusion. We identified commonly used narrative note section formats and developed a taxonomy of narrative note content to help researchers to tailor their efforts and design more efficient clinical documentation systems.

Prescribing Prevalence of Medications With Potential Genotype-Guided Dosing in Pediatric Patients

IMPORTANCE

Genotype-guided prescribing in pediatrics could prevent adverse drug reactions and improve therapeutic response. Clinical pharmacogenetic implementation guidelines are available for many medications commonly prescribed to children. Frequencies of medication prescription and actionable genotypes (genotypes where a prescribing change may be indicated) inform the potential value of pharmacogenetic implementation.

OBJECTIVE

To assess potential opportunities for genotype-guided prescribing in pediatric populations among multiple health systems by examining the prevalence of prescriptions for each drug with the highest level of evidence (Clinical Pharmacogenetics Implementation Consortium level A) and estimating the prevalence of potential actionable prescribing decisions.

DESIGN, SETTING, AND PARTICIPANTS

This serial cross-sectional study of prescribing prevalences in 16 health systems included electronic health records data from pediatric inpatient and outpatient encounters from January 1, 2011, to December 31, 2017. The health systems included academic medical centers with free-standing children's hospitals and community hospitals that were part of an adult health care system. Participants included approximately 2.9 million patients younger than 21 years observed per year. Data were analyzed from June 5, 2018, to April 14, 2020.

EXPOSURES

Prescription of 38 level A medications based on electronic health records.

MAIN OUTCOMES AND MEASURES

Annual prevalence of level A medication prescribing and estimated actionable exposures, calculated by combining estimated site-year prevalences across sites with each site weighted equally.

RESULTS

Data from approximately 2.9 million pediatric patients (median age, 8 [interquartile range, 2-16] years; 50.7% female, 62.3% White) were analyzed for a typical calendar year. The annual prescribing prevalence of at least 1 level A drug ranged from 7987 to 10 629 per 100 000 patients with increasing trends from 2011 to 2014. The most prescribed level A drug was the antiemetic ondansetron (annual prevalence of exposure, 8107 [95% CI, 8077-8137] per 100 000 children). Among commonly prescribed opioids, annual prevalence per 100 000 patients was 295 (95% CI, 273-317) for tramadol, 571 (95% CI, 557-586) for codeine, and 2116 (95% CI, 2097-2135) for oxycodone. The antidepressants citalopram, escitalopram, and amitriptyline were also commonly prescribed (annual prevalence, approximately 250 per 100 000 patients for each). Estimated prevalences of actionable exposures were highest for oxycodone and ondansetron (>300 per 100 000 patients annually). CYP2D6 and CYP2C19 substrates were more frequently prescribed than medications influenced by other genes.

CONCLUSIONS AND RELEVANCE

These findings suggest that opportunities for pharmacogenetic implementation among pediatric patients in the US are abundant. As expected, the greatest opportunity exists with implementing CYP2D6 and CYP2C19 pharmacogenetic guidance for commonly prescribed antiemetics, analgesics, and antidepressants.

A review of auditing techniques for the Unified Medical Language System

OBJECTIVE

The study sought to describe the literature related to the development of methods for auditing the Unified Medical Language System (UMLS), with particular attention to identifying errors and inconsistencies of attributes of the concepts in the UMLS Metathesaurus.

MATERIALS AND METHODS

We applied the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) approach by searching the MEDLINE database and Google Scholar for studies referencing the UMLS and any of several terms related to auditing, error detection, and quality assurance. A qualitative analysis and summarization of articles that met inclusion criteria were performed.

RESULTS

Eighty-three studies were reviewed in detail. We first categorized techniques based on various aspects including concepts, concept names, and synonymy (n = 37), semantic type assignments (n = 36), hierarchical relationships (n = 24), lateral relationships (n = 12), ontology enrichment (n = 8), and ontology alignment (n = 18). We also categorized the methods according to their level of automation (ie, automated systematic, automated heuristic, or manual) and the type of knowledge used (ie, intrinsic or extrinsic knowledge).

CONCLUSIONS

This study is a comprehensive review of the published methods for auditing the various conceptual aspects of the UMLS. Categorizing the auditing techniques according to the various aspects will enable the curators of the UMLS as well as researchers comprehensive easy access to this wealth of knowledge (eg, for auditing lateral relationships in the UMLS). We also reviewed ontology enrichment and alignment techniques due to their critical use of and impact on the UMLS.

Twilighted Homegrown Systems: The Experience of Six Traditional Electronic Health Record Developers in the Post-Meaningful Use Era

Objectives  This study aimed to understand if and how homegrown electronic health record (EHR) systems are used in the post–Meaningful Use (MU) era according to the experience of six traditional EHR developers.

Methods  We invited informatics leaders from a convenience sample of six health care organizations that have recently replaced their long used homegrown systems with commercial EHRs. Participants were asked to complete a written questionnaire with open-ended questions designed to explore if and how their homegrown system(s) is being used and maintained after adoption of a commercial EHR. We used snowball sampling to identify other potential respondents and institutions.

Results  Participants from all six organizations included in our initial sample completed the questionnaire and provided referrals to four other organizations; from these, two did not respond to our invitations and two had not yet replaced their system and were excluded. Two organizations (Columbia University and University of Alabama at Birmingham) still use their homegrown system for direct patient care and as a downtime system. Four organizations (Intermountain Healthcare, Partners Healthcare, Regenstrief Institute, and Vanderbilt University) kept their systems primarily to access historical data. All organizations reported the need to continue to develop or maintain local applications despite having adopted a commercial EHR. The most common applications developed include display and visualization tools and clinical decision support systems.

Conclusion  Homegrown EHR systems continue to be used for different purposes according to the experience of six traditional homegrown EHR developers. The annual cost to maintain these systems varies from $21,000 to over 1 million. The collective experience of these organizations indicates that commercial EHRs have not been able to provide all functionality needed for patient care and local applications are often developed for multiple purposes, which presents opportunities for future research and EHR development.

A Potential Answer to the Alert Override Riddle: Using Patient Attributes to Predict False Positive Alerts

Electronic health records (EHRs) use alerts to help prevent medical errors, yet clinicians override many of these alerts due to desensitization from constant exposure (alert fatigue). We hypothesize that a clinician might override an alert warning about the dangers of a treatment if the patient's health is so poor that the treatment is worth the risk or if a patient's health suggests the treatment is not needed. We used logistic regression with general estimating equations to determine if the Early Warning Score (EWS), a measurement used to predict critical care need, could be used to predict alert overrides. EWS was a significant predictor of overrides for three alerts. Although EWS could not predict overrides for all alert rules, these results suggest that EWS may be helpful for some alerts, but that additional EHR data will be needed for predicting override behavior to a useful degree.

Clinicians' reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis

Objective

To describe the literature exploring the use of electronic health record (EHR) systems to support creation and use of clinical documentation to guide future research.

Materials and Methods

We searched databases including MEDLINE, Scopus, and CINAHL from inception to April 20, 2018, for studies applying qualitative or mixed-methods examining EHR use to support creation and use of clinical documentation. A qualitative synthesis of included studies was undertaken.

Results

Twenty-three studies met the inclusion criteria and were reviewed in detail. We briefly reviewed 9 studies that did not meet the inclusion criteria but provided recommendations for EHR design. We identified 4 key themes: purposes of electronic clinical notes, clinicians' reasoning for note-entry and reading/retrieval, clinicians' strategies for note-entry, and clinicians' strategies for note-retrieval/reading. Five studies investigated note purposes and found that although patient care is the primary note purpose, non-clinical purposes have become more common. Clinicians' reasoning studies (n = 3) explored clinicians' judgement about what to document and represented clinicians' thought process in cognitive pathways. Note-entry studies (n = 6) revealed that what clinicians document is affected by EHR interfaces. Lastly, note-retrieval studies (n = 12) found that "assessment and plan" is the most read note section and what clinicians read is affected by external stimuli, care/information goals, and what they know about the patient.

Conclusion

Despite the widespread adoption of EHRs, their use to support note-entry and reading/retrieval is still understudied. Further research is needed to investigate approaches to capture and represent clinicians' reasoning and improve note-entry and retrieval/reading.

Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis

OBJECTIVE

The study sought to describe the literature describing clinical reasoning ontology (CRO)-based clinical decision support systems (CDSSs) and identify and classify the medical knowledge and reasoning concepts and their properties within these ontologies to guide future research.

METHODS

MEDLINE, Scopus, and Google Scholar were searched through January 30, 2019, for studies describing CRO-based CDSSs. Articles that explored the development or application of CROs or terminology were selected. Eligible articles were assessed for quality features of both CDSSs and CROs to determine the current practices. We then compiled concepts and properties used within the articles.

RESULTS

We included 38 CRO-based CDSSs for the analysis. Diversity of the purpose and scope of their ontologies was seen, with a variety of knowledge sources were used for ontology development. We found 126 unique medical knowledge concepts, 38 unique reasoning concepts, and 240 unique properties (137 relationships and 103 attributes). Although there is a great diversity among the terms used across CROs, there is a significant overlap based on their descriptions. Only 5 studies described high quality assessment.

CONCLUSION

We identified current practices used in CRO development and provided lists of medical knowledge concepts, reasoning concepts, and properties (relationships and attributes) used by CRO-based CDSSs. CRO developers reason that the inclusion of concepts used by clinicians' during medical decision making has the potential to improve CDSS performance. However, at present, few CROs have been used for CDSSs, and high-quality studies describing CROs are sparse. Further research is required in developing high-quality CDSSs based on CROs.

Health information technology as a learning health system: Call for a national monitoring system

After over half a century of computer application development in medicine, the US health system has gone digital with an enthusiastic confidence for rapid improvements in care outcomes, especially those of quality of care, safety, and productivity. The bad news is that evidence for the justification of the hype around health information technology (HIT) is conflicting, and the expected benefits of a digital health system have not yet materialized. We propose a national system for monitoring HIT impact based on the paradigm of the learning health system (LHS): learning from practical experience through high-quality, ongoing monitoring of care outcomes. Our proposal aims at leveraging current de facto standard research data repositories used to support large-scale clinical studies by incorporating data needed for more robust HIT assessments and application of rigorous research designs that are now feasible on a large scale.

Structured override reasons for drug-drug interaction alerts in electronic health records

Objective

The study sought to determine availability and use of structured override reasons for drug-drug interaction (DDI) alerts in electronic health records.

Materials and Methods

We collected data on DDI alerts and override reasons from 10 clinical sites across the United States using a variety of electronic health records. We used a multistage iterative card sort method to categorize the override reasons from all sites and identified best practices.

Results

Our methodology established 177 unique override reasons across the 10 sites. The number of coded override reasons at each site ranged from 3 to 100. Many sites offered override reasons not relevant to DDIs. Twelve categories of override reasons were identified. Three categories accounted for 78% of all overrides: “will monitor or take precautions,” “not clinically significant,” and “benefit outweighs risk.”

Discussion

We found wide variability in override reasons between sites and many opportunities to improve alerts. Some override reasons were irrelevant to DDIs. Many override reasons attested to a future action (eg, decreasing a dose or ordering monitoring tests), which requires an additional step after the alert is overridden, unless the alert is made actionable. Some override reasons deferred to another party, although override reasons often are not visible to other users. Many override reasons stated that the alert was inaccurate, suggesting that specificity of alerts could be improved.

Conclusions

Organizations should improve the options available to providers who choose to override DDI alerts. DDI alerting systems should be actionable and alerts should be tailored to the patient and drug pairs.

Unintended Consequences of Nationwide Electronic Health Record Adoption: Challenges and Opportunities in the Post-Meaningful Use Era

The US health system has recently achieved widespread adoption of electronic health record (EHR) systems, primarily driven by financial incentives provided by the Meaningful Use (MU) program. Although successful in promoting EHR adoption and use, the program, and other contributing factors, also produced important unintended consequences (UCs) with far-reaching implications for the US health system. Based on our own experiences from large health information technology (HIT) adoption projects and a collection of key studies in HIT evaluation, we discuss the most prominent UCs of MU: failed expectations, EHR market saturation, innovation vacuum, physician burnout, and data obfuscation. We identify challenges resulting from these UCs and provide recommendations for future research to empower the broader medical and informatics communities to realize the full potential of a now digitized health system. We believe that fixing these unanticipated effects will demand efforts from diverse players such as health care providers, administrators, HIT vendors, policy makers, informatics researchers, funding agencies, and outside developers; promotion of new business models; collaboration between academic medical centers and informatics research departments; and improved methods for evaluations of HIT.

Is Multiclass Automatic Text De-Identification Worth the Effort?

OBJECTIVES

Automatic de-identification to remove protected health information (PHI) from clinical text can use a "binary" model that replaces redacted text with a generic tag (e.g., "<PHI>"), or can use a "multiclass" model that retains more class information (e.g., "<Phone Number>"). Binary models are easier to develop, but result in text that is potentially less informative. We investigated whether building a multiclass de-identification is worth the extra effort.

METHODS

Using the 2014 i2b2 dataset, we compared the accuracy and impact on document readability of two models. In the first experiment, we generated one binary and two multiclass versions trained with the same machine-learning algorithm Conditional Random Field (CRF). Accuracy (recall, precision, f-score) and secondary metrics (e.g, training time, testing time, minimum memory required) were measured. In the second experiment, three reviewers accessed the readability of two redacted documents using the binary and multiclass methods. We estimated a pooled Kappa to estimate the inter-rater agreement.

RESULTS

The multiclass model did not demonstrate a clear accuracy advantage, with lower recall (-1.9%) and only slightly better precision (+0.6%), despite requiring additional computing resources. Three raters reached a very high agreement (Kappa = 0.975, 95% Confidence Interval (0.946, 1.00), p < 0.0001) that both binary and multiclass models have the same impact on document readability.

CONCLUSIONS

This study suggests that the development of more sophisticated classification of PHI may not be worth the effort in terms of both system accuracy and the usefulness of the output.

A visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS)

BACKGROUND

Vast volumes of data, coded through hierarchical terminologies (e.g., International Classification of Diseases, Tenth Revision-Clinical Modification [ICD10-CM], Medical Subject Headings [MeSH]), are generated routinely in electronic health record systems and medical literature databases. Although graphic representations can help to augment human understanding of such data sets, a graph with hundreds or thousands of nodes challenges human comprehension. To improve comprehension, new tools are needed to extract the overviews of such data sets. We aim to develop a visual interactive analytic tool for filtering and summarizing large health data sets coded with hierarchical terminologies (VIADS) as an online, and publicly accessible tool. The ultimate goals are to filter, summarize the health data sets, extract insights, compare and highlight the differences between various health data sets by using VIADS. The results generated from VIADS can be utilized as data-driven evidence to facilitate clinicians, clinical researchers, and health care administrators to make more informed clinical, research, and administrative decisions. We utilized the following tools and the development environments to develop VIADS: Django, Python, JavaScript, Vis.js, Graph.js, JQuery, Plotly, Chart.js, Unittest, R, and MySQL.

RESULTS

VIADS was developed successfully and the beta version is accessible publicly. In this paper, we introduce the architecture design, development, and functionalities of VIADS. VIADS includes six modules: user account management module, data sets validation module, data analytic module, data visualization module, terminology module, dashboard. Currently, VIADS supports health data sets coded by ICD-9, ICD-10, and MeSH. We also present the visualization improvement provided by VIADS in regard to interactive features (e.g., zoom in and out, customization of graph layout, expanded information of nodes, 3D plots) and efficient screen space usage.

CONCLUSIONS

VIADS meets the design objectives and can be used to filter, summarize, compare, highlight and visualize large health data sets that coded by hierarchical terminologies, such as ICD-9, ICD-10 and MeSH. Our further usability and utility studies will provide more details about how the end users are using VIADS to facilitate their clinical, research or health administrative decision making.