Using electronic health record audit log data for research: insights from early efforts

Does autotext usage decrease documentation time among resident physicians? A retrospective analysis of electronic health record usage data

Objective

Usage of autotext or "dotphrases" is ubiquitous among provider workflows in electronic health records (EHRs). Yet, little is known about the impact of these tools in inpatient settings and among resident physicians. We aimed to evaluate the association between autotext usage and documentation time among resident physicians in an academic medical center using the Cerner EHR.

Materials and Methods

The association between autotext executions and documentation time per patient seen for 705 resident physicians rotating at a large academic medical center from July 2021 to June 2023 was analyzed via linear regression after controlling for specialty, post-graduate year (PGY), provider gender and patient volume.

Results

There was no significant overall association between autotext executions per patient seen and documentation time per patient seen in specialties using Dynamic Documentation as their primary workflow (β=-0.1 min per autotext execution per patient seen, 95% CI -0.6 to 0.5 min, P =.79). However, there was increased documentation time among residents with no autotext usage compared to residents who used autotext, and this effect was mediated by use of personalized autotexts. Specialty, PGY, gender and patient volume were significant determinants of documentation time.

Discussion

Efforts to decrease documentation time among resident physicians should encourage autotext adoption but should not be focused on promotion of autotext usage alone. Further research should address the questions of identifying other determinants of documentation time, autotext design standards, and how autotext usage affects measures of note quality.

Conclusion

Autotext adoption decreases documentation time among resident physicians, but among those who adopt autotext, higher levels of usage show no benefit.

Optimizing Large Language Models for Discharge Prediction: Best Practices in Leveraging Electronic Health Record Audit Logs

Electronic Health Record (EHR) audit log data are increasingly utilized for clinical tasks, from workflow modeling to predictive analyses of discharge events, adverse kidney outcomes, and hospital readmissions. These data encapsulate user-EHR interactions, reflecting both healthcare professionals' behavior and patients' health statuses. To harness this temporal information effectively, this study explores the application of Large Language Models (LLMs) in leveraging audit log data for clinical prediction tasks, specifically focusing on discharge predictions. Utilizing a year's worth of EHR data from Vanderbilt University Medical Center, we fine-tuned LLMs with randomly selected 10,000 training examples. Our findings reveal that LLaMA-2 70B, with an AUROC of 0.80 [0.77-0.82], outperforms both GPT-4 128K in a zero-shot, with an AUROC of 0.68 [0.65-0.71], and DeBERTa, with an AUROC of 0.78 [0.75-0.82]. Among various serialization methods, the first-occurrence approach-wherein only the initial appearance of each event in a sequence is retained-shows superior performance. Furthermore, for the fine-tuned LLaMA-2 70B, logit outputs yield a higher AUROC of 0.80 [0.77-0.82] compared to text outputs, with an AUROC of 0.69 [0.67-0.72]. This study underscores the potential of fine-tuned LLMs, particularly when combined with strategic sequence serialization, in advancing clinical prediction tasks.

Addressing social and health needs in health care: Characterizing case managers' work to address patient-defined goals

OBJECTIVE

To test quantitative process measures characterizing the work of social needs case managers as they assisted patients with diverse health-related needs-spanning both medical and social domains.

STUDY SETTING AND DESIGN

The study analyzed secondary data on 7076 patients working with 147 case managers from the CommunityConnect social needs case management program in Contra Costa County, California from 2018 to 2021. The service-designed to be holistic with a focus on social determinants as root causes of health issues-helped patients navigate social services, health care, and mental health care.

DATA SOURCES AND ANALYTIC SAMPLE

We used cross-sectional analyses to quantitatively characterize electronic health records (EHRs) derived measures of case management intensity (goal updates), duration (days goal was open), and outcomes for 19 different categories of health and social goals. Mixed-effects regression models were used to examine how work process measures varied according to goal categories. Models nested goals within patients within case managers and adjusted for patient-level covariates.

PRINCIPAL FINDINGS

The most common goals were dental care (53%), food (40%), and housing (39%). In adjusted analyses, housing goals had significantly more case manager updates than any other type of goal with a marginal mean of 14.0 updates (95% CI: 13.4-14.7), were worked on for significantly longer (marginal mean of 417 days, 95% CI: 360-474) than any goal except dental care, and were least likely to be resolved. Utilities, insurance, and medication coordination goals were most likely to be resolved.

CONCLUSIONS

Case managers and patients repeatedly worked on goals over many months. Meeting housing needs and accessing dental care were issues that were not easily resolved and required extensive follow-up. One-time referral interventions may need follow-up systems to meaningfully support social and health needs.

Emerging Domains for Measuring Health Care Delivery With Electronic Health Record Metadata

This article aims to introduce emerging measurement domains made feasible through the electronic health record (EHR) use metadata, to inform the changing landscape of health care delivery. We reviewed emerging domains in which EHR metadata may be used to measure health care delivery, outlining a framework for evaluating measures based on desirability, feasibility, and viability. We argue that EHR use metadata may be leveraged to develop and operationalize novel measures in the domains of team structure and dynamics, workflows, and cognitive environment to provide a clearer understanding of modern health care delivery. Examples of measures feasible using metadata include quantification of teamwork and collaboration, patient continuity measures, workflow conformity measures, and attention switching. By enabling measures that can be used to inform the next generation of health care delivery, EHR metadata may be used to improve the quality of patient care and support clinician well-being. Careful attention is needed to ensure that these measures are desirable, feasible, and viable.

Adverse diagnostic events in hospitalised patients: a single-centre, retrospective cohort study

BACKGROUND

Adverse event surveillance approaches underestimate the prevalence of harmful diagnostic errors (DEs) related to hospital care.

METHODS

We conducted a single-centre, retrospective cohort study of a stratified sample of patients hospitalised on general medicine using four criteria: transfer to intensive care unit (ICU), death within 90 days, complex clinical events, and none of the aforementioned high-risk criteria. Cases in higher-risk subgroups were over-sampled in predefined percentages. Each case was reviewed by two adjudicators trained to judge the likelihood of DE using the Safer Dx instrument; characterise harm, preventability and severity; and identify associated process failures using the Diagnostic Error Evaluation and Research Taxonomy modified for acute care. Cases with discrepancies or uncertainty about DE or impact were reviewed by an expert panel. We used descriptive statistics to report population estimates of harmful, preventable and severely harmful DEs by demographic variables based on the weighted sample, and characteristics of harmful DEs. Multivariable models were used to adjust association of process failures with harmful DEs.

RESULTS

Of 9147 eligible cases, 675 were randomly sampled within each subgroup: 100% of ICU transfers, 38.5% of deaths within 90 days, 7% of cases with complex clinical events and 2.4% of cases without high-risk criteria. Based on the weighted sample, the population estimates of harmful, preventable and severely harmful DEs were 7.2% (95% CI 4.66 to 9.80), 6.1% (95% CI 3.79 to 8.50) and 1.1% (95% CI 0.55 to 1.68), respectively. Harmful DEs were frequently characterised as delays (61.9%). Severely harmful DEs were frequent in high-risk cases (55.1%). In multivariable models, process failures in assessment, diagnostic testing, subspecialty consultation, patient experience, and history were significantly associated with harmful DEs.

CONCLUSIONS

We estimate that a harmful DE occurred in 1 of every 14 patients hospitalised on general medicine, the majority of which were preventable. Our findings underscore the need for novel approaches for adverse DE surveillance.

Secure Messaging Use and Wrong-Patient Ordering Errors Among Inpatient Clinicians

Importance

Use of secure messaging for clinician-to-clinician communication has increased exponentially over the past decade, but its association with clinician work is poorly understood.

Objective

To investigate the association between secure messaging use and wrong-patient ordering errors.

Design, Setting, and Participants

This cohort study included inpatient attending physicians, trainee physicians, and advanced practice practitioners (APPs) from 14 academic and community hospitals. Secure messaging volume was assessed over a 3-month period (February 1 to April 30, 2023).

Exposure

Secure messaging volume per clinician-day, measured as the count of secure messages sent and received by a clinician on a given clinician-day.

Main Outcomes and Measures

Retract-and-reorder events were used to identify wrong-patient ordering errors, and the presence of any retract-and-reorder event on a clinician-day was the primary outcome. Multilevel logistic regression was used to examine the association between secure messaging volume and wrong-patient ordering errors after adjusting for clinician age, sex, patient load, order volume, and clinical service.

Results

A total of 3239 clinicians (median [IQR] age, 37 [32-46] years; 1791 female [55.3%]; 1680 attending physicians [51.2%], 560 trainee physicians [17.3%], and 999 APPs [30.8%]) with 75 546 clinician-days were included. Median secure messaging volume was 16 (IQR, 0-61) messages per day. Retract-and-reorder events were identified on 295 clinician-days (0.4%). Clinicians with secure messaging volume at the 75th percentile had a 10% higher odds of wrong-patient ordering errors compared with those at the 25th percentile (odds ratio [OR], 1.10; 95% CI, 1.01-1.20). After stratifying by clinician role, the association between secure messaging and wrong-patient ordering errors was observed only for attending physicians (OR, 1.20; 95% CI, 1.02-1.42) and APPs (OR, 1.18; 95% CI, 1.00-1.40).

Conclusions and Relevance

In this cohort study of inpatient clinicians, higher daily secure messaging was associated with increased odds of wrong-patient ordering errors. Although messaging may increase cognitive load and risk for wrong-patient ordering errors, these results do not provide conclusive evidence regarding the direct impact of secure messaging on errors, as increased messaging may also reflect greater care coordination, increased patient complexity, or communication of the presence of a wrong-patient ordering error.

Measuring cognitive effort using tabular transformer-based language models of electronic health record-based audit log action sequences

OBJECTIVES

To develop and validate a novel measure, action entropy, for assessing the cognitive effort associated with electronic health record (EHR)-based work activities.

MATERIALS AND METHODS

EHR-based audit logs of attending physicians and advanced practice providers (APPs) from four surgical intensive care units in 2019 were included. Neural language models (LMs) were trained and validated separately for attendings' and APPs' action sequences. Action entropy was calculated as the cross-entropy associated with the predicted probability of the next action, based on prior actions. To validate the measure, a matched pairs study was conducted to assess the difference in action entropy during known high cognitive effort scenarios, namely, attention switching between patients and to or from the EHR inbox.

RESULTS

Sixty-five clinicians performing 5 904 429 EHR-based audit log actions on 8956 unique patients were included. All attention switching scenarios were associated with a higher action entropy compared to non-switching scenarios (P < .001), except for the from-inbox switching scenario among APPs. The highest difference among attendings was for the from-inbox attention switching: Action entropy was 1.288 (95% CI, 1.256-1.320) standard deviations (SDs) higher for switching compared to non-switching scenarios. For APPs, the highest difference was for the to-inbox switching, where action entropy was 2.354 (95% CI, 2.311-2.397) SDs higher for switching compared to non-switching scenarios.

DISCUSSION

We developed a LM-based metric, action entropy, for assessing cognitive burden associated with EHR-based actions. The metric showed discriminant validity and statistical significance when evaluated against known situations of high cognitive effort (ie, attention switching). With additional validation, this metric can potentially be used as a screening tool for assessing behavioral action phenotypes that are associated with higher cognitive burden.

CONCLUSION

An LM-based action entropy metric-relying on sequences of EHR actions-offers opportunities for assessing cognitive effort in EHR-based workflows.

The Impact of Diagnosis-Specific Plan Templates on Admission Note Writing Time: A Quality Improvement Initiative

Background There are limited objective studies regarding the effectiveness of strategies to alleviate the documentation burden on resident physicians. Objective To develop and implement diagnosis-specific templates for the plan of care section of inpatient admission notes, aiming to reduce documentation time. Methods Twelve templates for the plan of care section of admission notes were written by the study authors, reviewed by attending physicians, and shared with the residents through the electronic health record (EHR) on September 23, 2022. EHR audit log data were collected to examine admission note writing times, supplemented by resident feedback on acceptability via an anonymous survey. Feasibility measures included time investment, experience with the EHR, and resident training. Results Between July 1, 2021 and June 30, 2023, 62 pediatric residents contributed 9840 admission notes. The templates were used in 557 admission notes. The mean total time spent on an admission note decreased from 97.9 minutes pre-intervention to 71.0 minutes post-intervention with the use of a template; an adjusted reduction of 23% (95% CI 16%-30%; P<.001). The mean attending time spent editing an admission note was unchanged. The survey results underscored wide acceptability of the templates among the residents. Feasibility data showed that the project required minimal time investment from the health care informatics team and minimal resident training. Conclusions Using templates in the care plan section of admission notes reduces the time residents spend writing admission notes.

The development and pilot of a novel mobile application to assess clinician perception of workload and work environment

BACKGROUND

Traditional measures of workload such as wRVUs may not be adequate to understand the impact of workload on key outcomes.

OBJECTIVE

The objective of this study was to develop a mobile application to assess, in near real time, clinicians' perception of workload and work environment.

DESIGNS, SETTINGS AND PARTICIPANTS

We developed the GrittyWork™ application (GW App) using the Chokshi and Mann process model for user-centered digital development. Study occured at a single academic medical center with hospitalist clinicians.

MAIN OUTCOME MEASURES AND MEASURES

Measures included the System Usability Scale (SUS), use measures from GW App, electronic health record (EHR) event log data and note counts, and qualitative interviews.

RESULTS

From October 28, 2022 to November 3, 2022, six hospitalist clinicians provided feedback on the early prototype of the GW App, and from February 28, 2023 to June 8, 2023, 30 hospitalist clinicians participated in the pilot while on clinical service. All 30 clinicians (100%) participated in the pilot submitting data for a total of 122 shifts. Participants reported working 10 ± 1 h per day (mean ± SD) and were responsible for an average of 11 ± 3 patients per day. The postpilot evaluation of the GW App showed a SUS score of 86 ± 11 and a participant preference toward mobile application-based surveys (73% of participants). Regarding workload measures, EHR event log data and notes data correlated with physician-reported workloads. Applying user-centered design techniques, we successfully developed a mobile application with high usability. These data can be paired with EHR event log data and outcomes to provide insights into the impact of workloads and work environments on outcomes.

Attention Among Health Care Professionals : A Scoping Review

BACKGROUND

The concept of attention can provide insight into the needs of clinicians and how health systems design can impact patient care quality and medical errors.

PURPOSE

To conduct a scoping review to 1) identify and characterize literature relevant to clinician attention; 2) compile metrics used to measure attention; and 3) create a framework of key concepts.

DATA SOURCES

Cumulated Index to Nursing and Allied Health Literature (CINAHL), Medline (PubMed), and Embase (Ovid) from 2001 to 26 February 2024.

STUDY SELECTION

English-language studies addressing health care worker attention in patient care. At least dual review and data abstraction.

DATA EXTRACTION

Article information, health care professional studied, practice environment, study design and intent, factor type related to attention, and metrics of attention used.

DATA SYNTHESIS

Of 6448 screened articles, 585 met inclusion criteria. Most studies were descriptive (n = 469) versus investigational (n = 116). More studies focused on barriers to attention (n = 387; 342 descriptive and 45 investigational) versus facilitators to improving attention (n = 198; 112 descriptive and 86 investigational). We developed a framework, grouping studies into 6 categories: 1) definitions of attention, 2) the clinical environment and its effect on attention, 3) personal factors affecting attention, 4) relationships between interventions or factors that affect attention and patient outcomes, 5) the effect of clinical alarms and alarm fatigue on attention, and 6) health information technology's effect on attention. Eighty-two metrics were used to measure attention.

LIMITATIONS

Does not synthesize answers to specific questions. Quality of studies was not assessed.

CONCLUSION

This overview may be a resource for researchers, quality improvement experts, and health system leaders to improve clinical environments. Future systematic reviews may synthesize evidence on metrics to measure attention and on the effectiveness of barriers or facilitators related to attention.

PRIMARY FUNDING SOURCE

None.

Experiences, barriers and expectations regarding current patient monitoring systems among ICU nurses in a University Hospital in Lebanon: a qualitative study

Purpose

The aim of the study is to assess the experiences, barriers, and expectations regarding current patient monitoring systems among intensive care unit nurses at one university hospital.

Methods

A qualitative exploratory study approach was adopted to test the research questions.

Results

Intensive care unit personnel placed a high value on practical criteria such as user friendliness and visualization while assessing the present monitoring system. Poor alarm handling was recognized as possible patient safety hazards. The necessity of high accessibility was highlighted once again for a prospective system; wireless, noninvasive, and interoperability of monitoring devices were requested; and smart phones for distant patient monitoring and alert management improvement were required.

Conclusion

Core comments from ICU personnel are included in this qualitative research on patient monitoring. All national healthcare involved parties must focus more on user-derived insights to ensure a speedy and effective introduction of digital health technologies in the ICU. The findings from the alarm control or mobile device studies might be utilized to train ICU personnel to use new technology, minimize alarm fatigue, increase medical device accessibility, and develop interoperability standards in critical care practice.

Creating Conversion Factors from EHR Event Log Data: A Comparison of Investigator-Derived and Vendor-Derived Metrics for Primary Care Physicians

Physicians spend a large amount of time with the electronic health record (EHR), which the majority believe contributes to their burnout. However, there are limitedstandardized measures of physician EHR time. Vendor-derived metrics are standardized but may underestimate real-world EHR experience. Investigator-derived metrics may be more reliable but not standardized, particularly with regard to timeout thresholds defining inactivity. This study aimed to enable standardized investigator-derived metrics using conversion factors between raw event log-derived metrics and Signal (Epic System's standardized metric) for primary care physicians. This was an observational, retrospective longitudinal study of EHR raw event logs and Signal data from a quaternary academic medical center and its community affiliates in California, over a 6-month period. The study evaluated 242 physicians over 1370 physician-months, comparing 53.7 million event logs to 6850 Signal metrics, in five different time based metrics. Results show that inactivity thresholds for event log metric derivation that most closely approximate Signal metrics ranged from 90 seconds (Visit Navigator) to 360 seconds ("Pajama time") depending on the metric. Based on this data, conversion factors for investigator-derived metrics across a wide range of inactivity thresholds, via comparison with Signal metrics, are provided which may allow researchers to consistently quantify EHR experience.

Identifying and Addressing Barriers to Implementing Core Electronic Health Record Use Metrics for Ambulatory Care: Virtual Consensus Conference Proceedings

Precise, reliable, valid metrics that are cost-effective and require reasonable implementation time and effort are needed to drive electronic health record (EHR) improvements and decrease EHR burden. Differences exist between research and vendor definitions of metrics. PROCESS:  We convened three stakeholder groups (health system informatics leaders, EHR vendor representatives, and researchers) in a virtual workshop series to achieve consensus on barriers, solutions, and next steps to implementing the core EHR use metrics in ambulatory care. CONCLUSION:  Actionable solutions identified to address core categories of EHR metric implementation challenges include: (1) maintaining broad stakeholder engagement, (2) reaching agreement on standardized measure definitions across vendors, (3) integrating clinician perspectives, and (4) addressing cognitive and EHR burden. Building upon the momentum of this workshop's outputs offers promise for overcoming barriers to implementing EHR use metrics.

Assessing Electronic Health Record (EHR) Use during a Major EHR Transition: An Innovative Mixed Methods Approach

BACKGROUND

Electronic health record (EHR) transitions are inherently disruptive to healthcare workers who must rapidly learn a new EHR and adapt to altered clinical workflows. Healthcare workers' perceptions of EHR usability and their EHR use patterns following transitions are poorly understood. The Department of Veterans Affairs (VA) is currently replacing its homegrown EHR with a commercial Cerner EHR, presenting a unique opportunity to examine EHR use trends and usability perceptions.

OBJECTIVE

To assess EHR usability and uptake up to 1-year post-transition at the first VA EHR transition site using a novel longitudinal, mixed methods approach.

DESIGN

A concurrent mixed methods strategy using EHR use metrics and qualitative interview data.

PARTICIPANTS

141 clinicians with data from select EHR use metrics in Cerner Lights On Network®. Interviews with 25 healthcare workers in various clinical and administrative roles.

APPROACH

We assessed changes in total EHR time, documentation time, and order time per patient post-transition. Interview transcripts (n = 90) were coded and analyzed for content specific to EHR usability.

KEY RESULTS

Total EHR time, documentation time, and order time all decreased precipitously within the first four months after go-live and demonstrated gradual improvements over 12 months. Interview participants expressed ongoing concerns with the EHR's usability and functionality up to a year after go-live such as tasks taking longer than the old system and inefficiencies related to inadequate training and inherent features of the new system. These sentiments did not seem to reflect the observed improvements in EHR use metrics.

CONCLUSIONS

The integration of quantitative and qualitative data yielded a complex picture of EHR usability. Participants described persistent challenges with EHR usability 1 year after go-live contrasting with observed improvements in EHR use metrics. Combining findings across methods can provide a clearer, contextualized understanding of EHR adoption and use patterns during EHR transitions.

Measuring the Cognitive Effort Associated with Task Switching in Routine EHR-based Tasks

OBJECTIVE

Clinical work involves performing overlapping, time-sensitive tasks that frequently require clinicians to switch their attention between multiple tasks. We developed a methodological approach using EHR-based audit logs to determine switch costs-the cognitive burden associated with task switching-and assessed its magnitude during routine EHR-based clinical tasks.

METHOD

Physician trainees (N=75) participated in a longitudinal study where they provided access to their EHR-based audit logs. Physicians' audit log actions were used to create a taxonomy of EHR tasks. These tasks were transformed into task sequences and the time spent on each task in a sequence was computed. Within these task sequences, instances of task switching (i.e., switching from one task to the next) and non-switching were identified. The primary outcome of interest was the time spent on a post-switch task. Using a mixed-effects regression model, we compared the durations of post-switch and non-switch tasks.

RESULTS

2,781,679 audit log events over 117,822 sessions from 75 physicians were analyzed. Physicians spent most time on chart review (Median (IQR)=5,439 (2,492-8,336) seconds), note review (1,936 (827-3,321) seconds), and navigating the EHR interface (1,048 (365.5-2,006) seconds) daily. Post task switch activity times were greater for documentation (Median increase=5 seconds), order entry (Median increase=3 seconds) and results review (Median increase=3 seconds). Mixed-effects regression showed that time spent on tasks were longer following a task switch (β=0.03; 95% CI_lower= 0.027, CI_upper=0.034), with greater post-swtich task times for imaging, order entry, note review, handoff, note entry, chart review and best practice advisory tasks.

DISCUSSION

Increased task switching time-an indicator of the cognitive burden associated with switching between tasks-is prevalent in routine EHR-based tasks. We discuss the cumulative impact of incremental switch costs have on overall EHR workload, wellness, and error rates. Relying on theoretical cognitive foundations, we suggest pragmatic design considerations for mitigating the effects of cognitive burden associated with task switching.