Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning

Optimizing Large Language Models for Discharge Prediction: Best Practices in Leveraging Electronic Health Record Audit Logs

Electronic Health Record (EHR) audit log data are increasingly utilized for clinical tasks, from workflow modeling to predictive analyses of discharge events, adverse kidney outcomes, and hospital readmissions. These data encapsulate user-EHR interactions, reflecting both healthcare professionals' behavior and patients' health statuses. To harness this temporal information effectively, this study explores the application of Large Language Models (LLMs) in leveraging audit log data for clinical prediction tasks, specifically focusing on discharge predictions. Utilizing a year's worth of EHR data from Vanderbilt University Medical Center, we fine-tuned LLMs with randomly selected 10,000 training examples. Our findings reveal that LLaMA-2 70B, with an AUROC of 0.80 [0.77-0.82], outperforms both GPT-4 128K in a zero-shot, with an AUROC of 0.68 [0.65-0.71], and DeBERTa, with an AUROC of 0.78 [0.75-0.82]. Among various serialization methods, the first-occurrence approach-wherein only the initial appearance of each event in a sequence is retained-shows superior performance. Furthermore, for the fine-tuned LLaMA-2 70B, logit outputs yield a higher AUROC of 0.80 [0.77-0.82] compared to text outputs, with an AUROC of 0.69 [0.67-0.72]. This study underscores the potential of fine-tuned LLMs, particularly when combined with strategic sequence serialization, in advancing clinical prediction tasks.

Emerging Domains for Measuring Health Care Delivery With Electronic Health Record Metadata

This article aims to introduce emerging measurement domains made feasible through the electronic health record (EHR) use metadata, to inform the changing landscape of health care delivery. We reviewed emerging domains in which EHR metadata may be used to measure health care delivery, outlining a framework for evaluating measures based on desirability, feasibility, and viability. We argue that EHR use metadata may be leveraged to develop and operationalize novel measures in the domains of team structure and dynamics, workflows, and cognitive environment to provide a clearer understanding of modern health care delivery. Examples of measures feasible using metadata include quantification of teamwork and collaboration, patient continuity measures, workflow conformity measures, and attention switching. By enabling measures that can be used to inform the next generation of health care delivery, EHR metadata may be used to improve the quality of patient care and support clinician well-being. Careful attention is needed to ensure that these measures are desirable, feasible, and viable.

Association Between Patient Portal Engagement and Weight Loss Outcomes in Patients After Bariatric Surgery: Longitudinal Observational Study Using Electronic Health Records

BACKGROUND

Bariatric surgery is an effective intervention for obesity, but comprehensive postoperative self-management is essential for optimal outcomes. While patient portals are generally seen as beneficial in engaging patients in health management, the link between their use and post-bariatric surgery weight loss remains unclear.

OBJECTIVE

This study aimed to investigate the association between patient portal engagement and postoperative BMI reduction among patients after bariatric surgery.

METHODS

This retrospective longitudinal study included patients who underwent Roux-en-Y gastric bypass or sleeve gastrectomy at Vanderbilt University Medical Center between January 2018 and March 2021. Patient portal engagement was measured during 4 stages: early (within 3 months after surgery), early midterm (3-6 months), late midterm (6-9 months), and late (9-12 months). Using generalized estimating equations, we estimated the associations between patients' portal engagements at these stages and the percentage of BMI reduction (%BMIR) at 3, 6, and 12 months after surgery. Covariates included duration since surgery, patient's age at the time of surgery, sex, race and ethnicity, type of bariatric surgery, severity of comorbid conditions, and socioeconomic disadvantage.

RESULTS

The study included 1415 patients, predominantly female (n=1145, 80.9%), with a racial composition of 76.9% (n=1088) White and 19.9% (n=282) Black. Overall, 805 (56.9%) patients underwent Roux-en-Y gastric bypass and 610 (43.1%) underwent sleeve gastrectomy. By 1 year after surgery, the median %BMIR was 31.5% (IQR 25.2%-36.8%), and the median number of active days on the patient portal was 54 (IQR 33-80). Early portal engagement was significantly associated with %BMIR at various postoperative times. Specifically, each additional 10 days of early portal engagement was associated with a 0.37% (95% CI -0.55% to -0.18%; P<.001) lower expected %BMIR at 3 months, a 1.11% (95% CI 0.82%-1.41%; P<.001) higher expected %BMIR at 6 months, and a 0.78% (95% CI 0.25%-1.31%; P=.004) higher expected %BMIR at 12 months. Furthermore, early midterm portal engagement was associated with a 0.36% (95% CI -0.69 to -0.03; P=.03) lower expected %BMIR at 6 months, but it was not significant at 12 months (P=.88). Late midterm and late portal engagement were not significantly associated with %BMIR at 12 months (P=.27 and P=.12, respectively). Furthermore, early engagement in various portal functions, such as messaging and accessing medical records, was significantly associated with a lower %BMIR at 3 months and a higher %BMIR at both 6 and 12 months (all P<.05).

CONCLUSIONS

Higher patient portal engagement within 3 months after surgery-suggestive of stronger adherence to postoperative instructions and improved communication with care teams-is associated with less favorable weight loss immediately after surgery but enhanced postoperative weight loss outcomes at 6 and 12 months. However, the limitations of retrospective data-driven studies highlight the need for future intervention-based studies to validate these associations and establish causality.

Digital Information Ecosystems in Modern Care Coordination and Patient Care Pathways and the Challenges and Opportunities for AI Solutions

The integration of digital technologies into health care has significantly enhanced the efficiency and effectiveness of care coordination. Our perspective paper explores the digital information ecosystems in modern care coordination, focusing on the processes of information generation, updating, transmission, and exchange along a patient's care pathway. We identify several challenges within this ecosystem, including interoperability issues, information silos, hard-to-map patient care journeys, increased workload on health care professionals, coordination and communication gaps, and compliance with privacy regulations. These challenges are often associated with inefficiencies and diminished care quality. We also examine how emerging artificial intelligence (AI) tools have the potential to enhance the management of patient information flow. Specifically, AI can boost interoperability across diverse health systems; optimize and monitor patient care pathways; improve information retrieval and care transitions; humanize health care by integrating patients' desired outcomes and patient-reported outcome measures; and optimize clinical workflows, resource allocation, and digital tool usability and user experiences. By strategically leveraging AI, health care systems can establish a more robust and responsive digital information ecosystem, improving care coordination and patient outcomes. This perspective underscores the importance of continued research and investment in AI technologies in patient care pathways. We advocate for a thoughtful integration of AI into health care practices to fully realize its potential in revolutionizing care coordination.

Improving Clinical Documentation with Artificial Intelligence: A Systematic Review

Clinicians dedicate significant time to clinical documentation, incurring opportunity cost. Artificial Intelligence (AI) tools promise to improve documentation quality and efficiency. This systematic review overviews peer-reviewed AI tools to understand how AI may reduce opportunity cost. PubMed, Embase, Scopus, and Web of Science databases were queried for original, English language research studies published during or before July 2024 that report a new development, application, and validation of an AI tool for improving clinical documentation. 129 studies were extracted from 673 candidate studies. AI tools improve documentation by structuring data, annotating notes, evaluating quality, identifying trends, and detecting errors. Other AI-enabled tools assist clinicians in real-time during office visits, but moderate accuracy precludes broad implementation. While a highly accurate end-to-end AI documentation assistant is not currently reported in peer-reviewed literature, existing techniques such as structuring data offer targeted improvements to clinical documentation workflows.

Improving Clinical Documentation with Artificial Intelligence: A Systematic Review

Clinicians dedicate significant time to clinical documentation, incurring opportunity cost. Artificial Intelligence (AI) tools promise to improve documentation quality and efficiency. This systematic review overviews peer-reviewed AI tools to understand how AI may reduce opportunity cost. PubMed, Embase, Scopus, and Web of Science databases were queried for original, English language research studies published during or before July 2024 that report a new development, application, and validation of an AI tool for improving clinical documentation. 129 studies were extracted from 673 candidate studies. AI tools improve documentation by structuring data, annotating notes, evaluating quality, identifying trends, and detecting errors. Other AI-enabled tools assist clinicians in real-time during office visits, but moderate accuracy precludes broad implementation. While a highly accurate end-to-end AI documentation assistant is not currently reported in peer-reviewed literature, existing techniques such as structuring data offer targeted improvements to clinical documentation workflows.

Learning from Non-Routine Events and Teamwork in Intensive Care Units: Challenges and Opportunities

Patients admitted to intensive care units (ICUs) have profound and complex illnesses, often fraught with uncertainties in diagnoses, treatments, and care decisions. Clinicians often deviate from best practices to handle ICUs' myriad complexities and uncertainties. Non-routine events (NREs), defined as any aspect of care perceived by clinicians as deviations from optimal care, are latent and frequent safety threats that, if left unchecked, can be precursors to adverse events. Proper identification and analysis of NREs that represent latent safety threats have been proposed as a feasible and more effective approach for performance improvement than traditional root cause analysis for patient safety events. However, NRE studies to date have yet to show the holistic picture of NREs in the contexts of teamwork and time-dependent tasks that are frequently associated with NREs. NREs, an upstream interventional area to understand root causes, team performance, and human-computer interaction, still needs to be expanded. This article presents concepts of NREs, and the use of real-world data (RWD) and informatics methodology to investigate NREs in contexts and discusses the opportunities and challenges to enhance NREs research in teamwork and time-dependent tasks.

Enhanced Patient Portal Engagement Associated with Improved Weight Loss Outcomes in Post-Bariatric Surgery Patients

Background

Bariatric surgery is an effective intervention for obesity, but it requires comprehensive postoperative self-management to achieve optimal outcomes. While patient portals are generally seen as beneficial in engaging patients in health management, the link between their use and post-bariatric surgery weight loss remains unclear.

Objective

This study investigated the association between patient portal engagement and postoperative body mass index (BMI) reduction among bariatric surgery patients.

Methods

This retrospective longitudinal study included patients who underwent Roux-en-Y gastric bypass (RYGB) or sleeve gastrectomy (SG) at Vanderbilt University Medical Center (VUMC) between January 2018 and March 2021. Using generalized estimating equations, we estimated the association between active days of postoperative patient portal use and the reduction of BMI percentage (%BMI) at 3, 6, and 12 months post-surgery. Covariates included duration since surgery, the patient's age at the time of surgery, gender, race and ethnicity, type of bariatric surgery, severity of comorbid conditions, and socioeconomic disadvantage.

Results

The study included 1,415 patients, mostly female (80.9%), with diverse racial and ethnic backgrounds. 805 (56.9%) patients underwent RYGB and 610 (43.1%) underwent SG. By one-year post-surgery, the mean (SD) %BMI reduction was 31.1% (8.3%), and the mean (SD) number of patient portal active days was 61.0 (41.2). A significantly positive association was observed between patient portal engagement and %BMI reduction, with variations revealed over time. Each 10-day increment of active portal use was associated with a 0.57% ([95% CI: 0.42- 0.72], P < .001) and 0.35% ([95% CI: 0.22- 0.49], P < .001) %BMI reduction at 3 and 6 months postoperatively. The association was not statistically significant at 12 months postoperatively (β=-0.07, [95% CI: -0.24- 0.09], P = .54). Various portal functions, including messaging, visits, my record, medical tools, billing, resources, and others, were positively associated with %BMI reduction at 3- and 6-months follow-ups.

Conclusions

Greater patient portal engagement, which may represent stronger adherence to postoperative instructions, better self-management of health, and enhanced communication with care teams, was associated with improved postoperative weight loss. Future investigations are needed to identify important portal features that contribute to the long-term success of weight loss management.

Differences in Health Professionals' Engagement With Electronic Health Records Based on Inpatient Race and Ethnicity

Importance

US health professionals devote a large amount of effort to engaging with patients' electronic health records (EHRs) to deliver care. It is unknown whether patients with different racial and ethnic backgrounds receive equal EHR engagement.

Objective

To investigate whether there are differences in the level of health professionals' EHR engagement for hospitalized patients according to race or ethnicity during inpatient care.

Design, Setting, and Participants

This cross-sectional study analyzed EHR access log data from 2 major medical institutions, Vanderbilt University Medical Center (VUMC) and Northwestern Medicine (NW Medicine), over a 3-year period from January 1, 2018, to December 31, 2020. The study included all adult patients (aged ≥18 years) who were discharged alive after hospitalization for at least 24 hours. The data were analyzed between August 15, 2022, and March 15, 2023.

Exposures

The actions of health professionals in each patient's EHR were based on EHR access log data. Covariates included patients' demographic information, socioeconomic characteristics, and comorbidities.

Main Outcomes and Measures

The primary outcome was the quantity of EHR engagement, as defined by the average number of EHR actions performed by health professionals within a patient's EHR per hour during the patient's hospital stay. Proportional odds logistic regression was applied based on outcome quartiles.

Results

A total of 243 416 adult patients were included from VUMC (mean [SD] age, 51.7 [19.2] years; 54.9% female and 45.1% male; 14.8% Black, 4.9% Hispanic, 77.7% White, and 2.6% other races and ethnicities) and NW Medicine (mean [SD] age, 52.8 [20.6] years; 65.2% female and 34.8% male; 11.7% Black, 12.1% Hispanic, 69.2% White, and 7.0% other races and ethnicities). When combining Black, Hispanic, or other race and ethnicity patients into 1 group, these patients were significantly less likely to receive a higher amount of EHR engagement compared with White patients (adjusted odds ratios, 0.86 [95% CI, 0.83-0.88; P < .001] for VUMC and 0.90 [95% CI, 0.88-0.92; P < .001] for NW Medicine). However, a reduction in this difference was observed from 2018 to 2020.

Conclusions and Relevance

In this cross-sectional study of inpatient EHR engagement, the findings highlight differences in how health professionals distribute their efforts to patients' EHRs, as well as a method to measure these differences. Further investigations are needed to determine whether and how EHR engagement differences are correlated with health care outcomes.

Inferring EHR Utilization Workflows through Audit Logs

Electronic health records (EHRs) usage and clinical workflows are intrinsically linked. To accommodate the complex care settings (e.g., emergency departments), EHR utilization workflows dynamically change in clinical practice, which in turn shapes the clinical workflows. Learning EHR workflows would provide an opportunity for healthcare organizations to enhance clinical workflows in the context of EHRs. However, very few studies investigated HER utilization workflows executed in clinical practice. We develop a network analysis framework and apply it to EHR audit logs to infer EHR workflows. We then measure the differences in the workflows between patient subgroups divided by races via differential network analysis. We apply our framework to trauma patients admitted to the emergency department, which is one of the clinical settings that need timely support from EHR utilizations. Our results show five core EHR workflows related to Narrator, Navigator, SmartTools, Chart Review, and ED workup activities in the ED. We find EHR workflows involving Narrator, SmartTools, and BPA are different when comparing patient subgroups.

Measuring the Cognitive Effort Associated with Task Switching in Routine EHR-based Tasks

OBJECTIVE

Clinical work involves performing overlapping, time-sensitive tasks that frequently require clinicians to switch their attention between multiple tasks. We developed a methodological approach using EHR-based audit logs to determine switch costs-the cognitive burden associated with task switching-and assessed its magnitude during routine EHR-based clinical tasks.

METHOD

Physician trainees (N=75) participated in a longitudinal study where they provided access to their EHR-based audit logs. Physicians' audit log actions were used to create a taxonomy of EHR tasks. These tasks were transformed into task sequences and the time spent on each task in a sequence was computed. Within these task sequences, instances of task switching (i.e., switching from one task to the next) and non-switching were identified. The primary outcome of interest was the time spent on a post-switch task. Using a mixed-effects regression model, we compared the durations of post-switch and non-switch tasks.

RESULTS

2,781,679 audit log events over 117,822 sessions from 75 physicians were analyzed. Physicians spent most time on chart review (Median (IQR)=5,439 (2,492-8,336) seconds), note review (1,936 (827-3,321) seconds), and navigating the EHR interface (1,048 (365.5-2,006) seconds) daily. Post task switch activity times were greater for documentation (Median increase=5 seconds), order entry (Median increase=3 seconds) and results review (Median increase=3 seconds). Mixed-effects regression showed that time spent on tasks were longer following a task switch (β=0.03; 95% CI_lower= 0.027, CI_upper=0.034), with greater post-swtich task times for imaging, order entry, note review, handoff, note entry, chart review and best practice advisory tasks.

DISCUSSION

Increased task switching time-an indicator of the cognitive burden associated with switching between tasks-is prevalent in routine EHR-based tasks. We discuss the cumulative impact of incremental switch costs have on overall EHR workload, wellness, and error rates. Relying on theoretical cognitive foundations, we suggest pragmatic design considerations for mitigating the effects of cognitive burden associated with task switching.

The use of machine learning and artificial intelligence within pediatric critical care

The field of pediatric critical care has been hampered in the era of precision medicine by our inability to accurately define and subclassify disease phenotypes. This has been caused by heterogeneity across age groups that further challenges the ability to perform randomized controlled trials in pediatrics. One approach to overcome these inherent challenges include the use of machine learning algorithms that can assist in generating more meaningful interpretations from clinical data. This review summarizes machine learning and artificial intelligence techniques that are currently in use for clinical data modeling with relevance to pediatric critical care. Focus has been placed on the differences between techniques and the role of each in the clinical arena. The various forms of clinical decision support that utilize machine learning are also described. We review the applications and limitations of machine learning techniques to empower clinicians to make informed decisions at the bedside. IMPACT: Critical care units generate large amounts of under-utilized data that can be processed through artificial intelligence. This review summarizes the machine learning and artificial intelligence techniques currently being used to process clinical data. The review highlights the applications and limitations of these techniques within a clinical context to aid providers in making more informed decisions at the bedside.

Using event logs to observe interactions with electronic health records: an updated scoping review shows increasing use of vendor-derived measures

OBJECTIVE

The aim of this article is to compare the aims, measures, methods, limitations, and scope of studies that employ vendor-derived and investigator-derived measures of electronic health record (EHR) use, and to assess measure consistency across studies.

MATERIALS AND METHODS

We searched PubMed for articles published between July 2019 and December 2021 that employed measures of EHR use derived from EHR event logs. We coded the aims, measures, methods, limitations, and scope of each article and compared articles employing vendor-derived and investigator-derived measures.

RESULTS

One hundred and two articles met inclusion criteria; 40 employed vendor-derived measures, 61 employed investigator-derived measures, and 1 employed both. Studies employing vendor-derived measures were more likely than those employing investigator-derived measures to observe EHR use only in ambulatory settings (83% vs 48%, P = .002) and only by physicians or advanced practice providers (100% vs 54% of studies, P < .001). Studies employing vendor-derived measures were also more likely to measure durations of EHR use (P < .001 for 6 different activities), but definitions of measures such as time outside scheduled hours varied widely. Eight articles reported measure validation. The reported limitations of vendor-derived measures included measure transparency and availability for certain clinical settings and roles.

DISCUSSION

Vendor-derived measures are increasingly used to study EHR use, but only by certain clinical roles. Although poorly validated and variously defined, both vendor- and investigator-derived measures of EHR time are widely reported.

CONCLUSION

The number of studies using event logs to observe EHR use continues to grow, but with inconsistent measure definitions and significant differences between studies that employ vendor-derived and investigator-derived measures.

Team is brain: leveraging EHR audit log data for new insights into acute care processes

OBJECTIVE

To determine whether novel measures of contextual factors from multi-site electronic health record (EHR) audit log data can explain variation in clinical process outcomes.

MATERIALS AND METHODS

We selected one widely-used process outcome: emergency department (ED)-based team time to deliver tissue plasminogen activator (tPA) to patients with acute ischemic stroke (AIS). We evaluated Epic audit log data (that tracks EHR user-interactions) for 3052 AIS patients aged 18+ who received tPA after presenting to an ED at three Northern California health systems (Stanford Health Care, UCSF Health, and Kaiser Permanente Northern California). Our primary outcome was door-to-needle time (DNT) and we assessed bivariate and multivariate relationships with six audit log-derived measures of treatment team busyness and prior team experience.

RESULTS

Prior team experience was consistently associated with shorter DNT; teams with greater prior experience specifically on AIS cases had shorter DNT (minutes) across all sites: (Site 1: -94.73, 95% CI: -129.53 to 59.92; Site 2: -80.93, 95% CI: -130.43 to 31.43; Site 3: -42.95, 95% CI: -62.73 to 23.17). Teams with greater prior experience across all types of cases also had shorter DNT at two sites: (Site 1: -6.96, 95% CI: -14.56 to 0.65; Site 2: -19.16, 95% CI: -36.15 to 2.16; Site 3: -11.07, 95% CI: -17.39 to 4.74). Team busyness was not consistently associated with DNT across study sites.

CONCLUSIONS

EHR audit log data offers a novel, scalable approach to measure key contextual factors relevant to clinical process outcomes across multiple sites. Audit log-based measures of team experience were associated with better process outcomes for AIS care, suggesting opportunities to study underlying mechanisms and improve care through deliberate training, team-building, and scheduling to maximize team experience.

Using electronic health record audit log data for research: insights from early efforts

Electronic health record audit logs capture a time-sequenced record of clinician activities while using the system. Audit log data therefore facilitate unobtrusive measurement at scale of clinical work activities and workflow as well as derivative, behavioral proxies (eg, teamwork). Given its considerable research potential, studies leveraging these data have burgeoned. As the field has matured, the challenges of using the data to answer significant research questions have come into focus. In this Perspective, we draw on our research experiences and insights from the broader audit log literature to advance audit log research. Specifically, we make 2 complementary recommendations that would facilitate substantial progress toward audit log-based measures that are: (1) transparent and validated, (2) standardized to allow for multisite studies, (3) sensitive to meaningful variability, (4) broader in scope to capture key aspects of clinical work including teamwork and coordination, and (5) linked to patient and clinical outcomes.

Characterizing the macrostructure of electronic health record work using raw audit logs: an unsupervised action embeddings approach

Raw audit logs provide a comprehensive record of clinicians' activities on an electronic health record (EHR) and have considerable potential for studying clinician behaviors. However, research using raw audit logs is limited because they lack context for clinical tasks, leading to difficulties in interpretation. We describe a novel unsupervised approach using the comparison and visualization of EHR action embeddings to learn context and structure from raw audit log activities. Using a dataset of 15 767 634 raw audit log actions performed by 88 intern physicians over 6 months of EHR use across inpatient and outpatient settings, we demonstrated that embeddings can be used to learn the situated context for EHR-based work activities, identify discrete clinical workflows, and discern activities typically performed across diverse contexts. Our approach represents an important methodological advance in raw audit log research, facilitating the future development of metrics and predictive models to measure clinician behaviors at the macroscale.

The Use of Artificial Intelligence and Machine Learning in Surgery: A Comprehensive Literature Review

Interest in the use of artificial intelligence (AI) and machine learning (ML) in medicine has grown exponentially over the last few years. With its ability to enhance speed, precision, and efficiency, AI has immense potential, especially in the field of surgery. This article aims to provide a comprehensive literature review of artificial intelligence as it applies to surgery and discuss practical examples, current applications, and challenges to the adoption of this technology. Furthermore, we elaborate on the utility of natural language processing and computer vision in improving surgical outcomes, research, and patient care.

Predicting next-day discharge via electronic health record access logs

OBJECTIVE

Hospital capacity management depends on accurate real-time estimates of hospital-wide discharges. Estimation by a clinician requires an excessively large amount of effort and, even when attempted, accuracy in forecasting next-day patient-level discharge is poor. This study aims to support next-day discharge predictions with machine learning by incorporating electronic health record (EHR) audit log data, a resource that captures EHR users' granular interactions with patients' records by communicating various semantics and has been neglected in outcome predictions.

MATERIALS AND METHODS

This study focused on the EHR data for all adults admitted to Vanderbilt University Medical Center in 2019. We learned multiple advanced models to assess the value that EHR audit log data adds to the daily prediction of discharge likelihood within 24 h and to compare different representation strategies. We applied Shapley additive explanations to identify the most influential types of user-EHR interactions for discharge prediction.

RESULTS

The data include 26 283 inpatient stays, 133 398 patient-day observations, and 819 types of user-EHR interactions. The model using the count of each type of interaction in the recent 24 h and other commonly used features, including demographics and admission diagnoses, achieved the highest area under the receiver operating characteristics (AUROC) curve of 0.921 (95% CI: 0.919-0.923). By contrast, the model lacking user-EHR interactions achieved a worse AUROC of 0.862 (0.860-0.865). In addition, 10 of the 20 (50%) most influential factors were user-EHR interaction features.

CONCLUSION

EHR audit log data contain rich information such that it can improve hospital-wide discharge predictions.

Characterizing Multitasking and Workflow Fragmentation in Electronic Health Records among Emergency Department Clinicians: Using Time-Motion Data to Understand Documentation Burden

BACKGROUND

The impact of electronic health records (EHRs) in the emergency department (ED) remains mixed. Dynamic and unpredictable, the ED is highly vulnerable to workflow interruptions.

OBJECTIVES

The aim of the study is to understand multitasking and task fragmentation in the clinical workflow among ED clinicians using clinical information systems (CIS) through time-motion study (TMS) data, and inform their applications to more robust and generalizable measures of CIS-related documentation burden.

METHODS

Using TMS data collected among 15 clinicians in the ED, we investigated the role of documentation burden, multitasking (i.e., performing physical and communication tasks concurrently), and workflow fragmentation in the ED. We focused on CIS-related tasks, including EHRs.

RESULTS

We captured 5,061 tasks and 877 communications in 741 locations within the ED. Of the 58.7 total hours observed, 44.7% were spent on CIS-related tasks; nearly all CIS-related tasks focused on data-viewing and data-entering. Over one-fifth of CIS-related task time was spent on multitasking. The mean average duration among multitasked CIS-related tasks was shorter than non-multitasked CIS-related tasks (20.7 s vs. 30.1 s). Clinicians experienced 1.4 ± 0.9 task switches/min, which increased by one-third when multitasking. Although multitasking was associated with a significant increase in the average duration among data-entering tasks, there was no significant effect on data-viewing tasks. When engaged in CIS-related task switches, clinicians were more likely to return to the same CIS-related task at higher proportions while multitasking versus not multitasking.

CONCLUSION

Multitasking and workflow fragmentation may play a significant role in EHR documentation among ED clinicians, particularly among data-entering tasks. Understanding where and when multitasking and workflow fragmentation occurs is a crucial step to assessing potentially burdensome clinician tasks and mitigating risks to patient safety. These findings may guide future research on developing more scalable and generalizable measures of CIS-related documentation burden that do not necessitate direct observation techniques (e.g., EHR log files).

Measuring Collaboration Through Concurrent Electronic Health Record Usage: Network Analysis Study

BACKGROUND

Collaboration is vital within health care institutions, and it allows for the effective use of collective health care worker (HCW) expertise. Human-computer interactions involving electronic health records (EHRs) have become pervasive and act as an avenue for quantifying these collaborations using statistical and network analysis methods.

OBJECTIVE

We aimed to measure HCW collaboration and its characteristics by analyzing concurrent EHR usage.

METHODS

By extracting concurrent EHR usage events from audit log data, we defined concurrent sessions. For each HCW, we established a metric called concurrent intensity, which was the proportion of EHR activities in concurrent sessions over all EHR activities. Statistical models were used to test the differences in the concurrent intensity between HCWs. For each patient visit, starting from admission to discharge, we measured concurrent EHR usage across all HCWs, which we called temporal patterns. Again, we applied statistical models to test the differences in temporal patterns of the admission, discharge, and intermediate days of hospital stay between weekdays and weekends. Network analysis was leveraged to measure collaborative relationships among HCWs. We surveyed experts to determine if they could distinguish collaborative relationships between high and low likelihood categories derived from concurrent EHR usage. Clustering was used to aggregate concurrent activities to describe concurrent sessions. We gathered 4 months of EHR audit log data from a large academic medical center's neonatal intensive care unit (NICU) to validate the effectiveness of our framework.

RESULTS

There was a significant difference (P<.001) in the concurrent intensity (proportion of concurrent activities: ranging from mean 0.07, 95% CI 0.06-0.08, to mean 0.36, 95% CI 0.18-0.54; proportion of time spent on concurrent activities: ranging from mean 0.32, 95% CI 0.20-0.44, to mean 0.76, 95% CI 0.51-1.00) between the top 13 HCW specialties who had the largest amount of time spent in EHRs. Temporal patterns between weekday and weekend periods were significantly different on admission (number of concurrent intervals per hour: 11.60 vs 0.54; P<.001) and discharge days (4.72 vs 1.54; P<.001), but not during intermediate days of hospital stay. Neonatal nurses, fellows, frontline providers, neonatologists, consultants, respiratory therapists, and ancillary and support staff had collaborative relationships. NICU professionals could distinguish high likelihood collaborative relationships from low ones at significant rates (3.54, 95% CI 3.31-4.37 vs 2.64, 95% CI 2.46-3.29; P<.001). We identified 50 clusters of concurrent activities. Over 87% of concurrent sessions could be described by a single cluster, with the remaining 13% of sessions comprising multiple clusters.

CONCLUSIONS

Leveraging concurrent EHR usage workflow through audit logs to analyze HCW collaboration may improve our understanding of collaborative patient care. HCW collaboration using EHRs could potentially influence the quality of patient care, discharge timeliness, and clinician workload, stress, or burnout.

Identifying Patterns in Administrative Tasks through Structural Topic Modeling: A Study of Task Definitions, Prevalence, and Shifts in a Mental Health Practice's Operations during the COVID-19 Pandemic

OBJECTIVE

This case study illustrates the use of natural language processing for identifying administrative task categories, prevalence and shifts necessitated by a major event (the COVID-19 pandemic) from user-generated data stored as free text in a task management system for a multi-site mental health practice with 40 clinicians and 13 administrative staff members.

METHODS

Structural topic modeling was applied on 7,079 task sequences from 13 administrative users of a HIPAA-compliant task management platform. Context was obtained through interviews with an expert panel.

RESULTS

10 task definitions spanning three major categories were identified, and their prevalence estimated. Significant shifts in task prevalence due to the pandemic were detected for tasks like billing inquiries to insurers, appointment cancellations, patient balances and new patient follow-up.

CONCLUSIONS

Structural topic modeling effectively detects task categories, prevalence, and shifts, providing opportunities for healthcare providers to reconsider staff roles and to optimize workflows and resource allocation.