1
|
Liu S, McCoy AB, Wright AP, Nelson SD, Huang SS, Ahmad HB, Carro SE, Franklin J, Brogan J, Wright A. Why do users override alerts? Utilizing large language model to summarize comments and optimize clinical decision support. J Am Med Inform Assoc 2024; 31:1388-1396. [PMID: 38452289 PMCID: PMC11105133 DOI: 10.1093/jamia/ocae041] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Revised: 02/06/2024] [Accepted: 02/21/2024] [Indexed: 03/09/2024] Open
Abstract
OBJECTIVES To evaluate the capability of using generative artificial intelligence (AI) in summarizing alert comments and to determine if the AI-generated summary could be used to improve clinical decision support (CDS) alerts. MATERIALS AND METHODS We extracted user comments to alerts generated from September 1, 2022 to September 1, 2023 at Vanderbilt University Medical Center. For a subset of 8 alerts, comment summaries were generated independently by 2 physicians and then separately by GPT-4. We surveyed 5 CDS experts to rate the human-generated and AI-generated summaries on a scale from 1 (strongly disagree) to 5 (strongly agree) for the 4 metrics: clarity, completeness, accuracy, and usefulness. RESULTS Five CDS experts participated in the survey. A total of 16 human-generated summaries and 8 AI-generated summaries were assessed. Among the top 8 rated summaries, five were generated by GPT-4. AI-generated summaries demonstrated high levels of clarity, accuracy, and usefulness, similar to the human-generated summaries. Moreover, AI-generated summaries exhibited significantly higher completeness and usefulness compared to the human-generated summaries (AI: 3.4 ± 1.2, human: 2.7 ± 1.2, P = .001). CONCLUSION End-user comments provide clinicians' immediate feedback to CDS alerts and can serve as a direct and valuable data resource for improving CDS delivery. Traditionally, these comments may not be considered in the CDS review process due to their unstructured nature, large volume, and the presence of redundant or irrelevant content. Our study demonstrates that GPT-4 is capable of distilling these comments into summaries characterized by high clarity, accuracy, and completeness. AI-generated summaries are equivalent and potentially better than human-generated summaries. These AI-generated summaries could provide CDS experts with a novel means of reviewing user comments to rapidly optimize CDS alerts both online and offline.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Aileen P Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Sean S Huang
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Hasan B Ahmad
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA 98195, United States
| | - Sabrina E Carro
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Jacob Franklin
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - James Brogan
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37212, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37212, United States
| |
Collapse
|
2
|
Liu S, McCoy AB, Peterson JF, Lasko TA, Sittig DF, Nelson SD, Andrews J, Patterson L, Cobb CM, Mulherin D, Morton CT, Wright A. Leveraging explainable artificial intelligence to optimize clinical decision support. J Am Med Inform Assoc 2024; 31:968-974. [PMID: 38383050 PMCID: PMC10990514 DOI: 10.1093/jamia/ocae019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 01/02/2024] [Accepted: 01/22/2024] [Indexed: 02/23/2024] Open
Abstract
OBJECTIVE To develop and evaluate a data-driven process to generate suggestions for improving alert criteria using explainable artificial intelligence (XAI) approaches. METHODS We extracted data on alerts generated from January 1, 2019 to December 31, 2020, at Vanderbilt University Medical Center. We developed machine learning models to predict user responses to alerts. We applied XAI techniques to generate global explanations and local explanations. We evaluated the generated suggestions by comparing with alert's historical change logs and stakeholder interviews. Suggestions that either matched (or partially matched) changes already made to the alert or were considered clinically correct were classified as helpful. RESULTS The final dataset included 2 991 823 firings with 2689 features. Among the 5 machine learning models, the LightGBM model achieved the highest Area under the ROC Curve: 0.919 [0.918, 0.920]. We identified 96 helpful suggestions. A total of 278 807 firings (9.3%) could have been eliminated. Some of the suggestions also revealed workflow and education issues. CONCLUSION We developed a data-driven process to generate suggestions for improving alert criteria using XAI techniques. Our approach could identify improvements regarding clinical decision support (CDS) that might be overlooked or delayed in manual reviews. It also unveils a secondary purpose for the XAI: to improve quality by discovering scenarios where CDS alerts are not accepted due to workflow, education, or staffing issues.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Josh F Peterson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Thomas A Lasko
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN 37212, United States
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, TX 77030, United States
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Jennifer Andrews
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Pathology, Microbiology and Immunology, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Lorraine Patterson
- HeathIT, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Cheryl M Cobb
- Department of Psychiatry and Behavioral Sciences, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - David Mulherin
- HeathIT, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Colleen T Morton
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN 37203, United States
- Department of Medicine, Vanderbilt University Medical Center, Nashville, TN 37203, United States
| |
Collapse
|
3
|
Sheehan KN, Cioci AL, Lucioni TM, Hernandez SM. Resident-Driven Clinical Decision Support Governance to Improve the Utility of Clinical Decision Support. Appl Clin Inform 2024; 15:335-341. [PMID: 38692282 PMCID: PMC11062759 DOI: 10.1055/s-0044-1786682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Accepted: 03/12/2024] [Indexed: 05/03/2024] Open
Abstract
OBJECTIVES This resident-driven quality improvement project aimed to better understand the known problem of a misaligned clinical decision support (CDS) strategy and improve CDS utilization. METHODS An internal survey was sent to all internal medicine (IM) residents to identify the most bothersome CDS alerts. Survey results were supported by electronic health record (EHR) data of CDS firing rates and response rates which were collected for each of the three most bothersome CDS tools. Changes to firing criteria were created to increase utilization and to better align with the five rights of CDS. Findings and proposed changes were presented to our institution's CDS Governance Committee. Changes were approved and implemented. Postintervention firing rates were then collected for 1 week. RESULTS Twenty nine residents participated in the CDS survey and identified sepsis alerts, lipid profile reminders, and telemetry renewals to be the most bothersome alerts. EHR data showed action rates for these CDS as low as 1%. We implemented changes to focus emergency department (ED)-based sepsis alerts to the right provider, better address the right information for lipid profile reminders, and select the right time in workflow for telemetry renewals to be most effective. With these changes we successfully eliminated ED-based sepsis CDS reminders for IM providers, saw a 97% reduction in firing rates for the lipid profile CDS, and noted a 55% reduction in firing rates for telemetry CDS. CONCLUSION This project highlighted that alert improvements spearheaded by resident teams can be completed successfully using robust CDS governance strategies and can effectively optimize interruptive alerts.
Collapse
Affiliation(s)
- Kristin N. Sheehan
- Department of Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Anthony L. Cioci
- Department of Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Tomas M. Lucioni
- Department of Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| | - Sean M. Hernandez
- Department of Internal Medicine, Wake Forest School of Medicine, Winston-Salem, North Carolina, United States
| |
Collapse
|
4
|
Musser RC, Senior R, Havrilesky LJ, Buuck J, Casarett DJ, Ibrahim S, Davidson BA. Randomized Comparison of Electronic Health Record Alert Types in Eliciting Responses about Prognosis in Gynecologic Oncology Patients. Appl Clin Inform 2024; 15:204-211. [PMID: 38232748 PMCID: PMC10937092 DOI: 10.1055/a-2247-9355] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2023] [Accepted: 01/16/2024] [Indexed: 01/19/2024] Open
Abstract
OBJECTIVES To compare the ability of different electronic health record alert types to elicit responses from users caring for cancer patients benefiting from goals of care (GOC) conversations. METHODS A validated question asking if the user would be surprised by the patient's 6-month mortality was built as an Epic BestPractice Advisory (BPA) alert in three versions-(1) Required on Open chart (pop-up BPA), (2) Required on Close chart (navigator BPA), and (3) Optional Persistent (Storyboard BPA)-randomized using patient medical record number. Meaningful responses were defined as "Yes" or "No," rather than deferral. Data were extracted over 6 months. RESULTS Alerts appeared for 685 patients during 1,786 outpatient encounters. Measuring encounters where a meaningful response was elicited, rates were highest for Required on Open (94.8% of encounters), compared with Required on Close (90.1%) and Optional Persistent (19.7%) (p < 0.001). Measuring individual alerts to which responses were given, they were most likely meaningful with Optional Persistent (98.3% of responses) and least likely with Required on Open (68.0%) (p < 0.001). Responses of "No," suggesting poor prognosis and prompting GOC, were more likely with Optional Persistent (13.6%) and Required on Open (10.3%) than with Required on Close (7.0%) (p = 0.028). CONCLUSION Required alerts had response rates almost five times higher than optional alerts. Timing of alerts affects rates of meaningful responses and possibly the response itself. The alert with the most meaningful responses was also associated with the most interruptions and deferral responses. Considering tradeoffs in these metrics is important in designing clinical decision support to maximize success.
Collapse
Affiliation(s)
- Robert Clayton Musser
- Department of Medicine, Duke University Health System, Durham, North Carolina, United States
- Duke Health Technology Solutions, Durham, North Carolina, United States
| | - Rashaud Senior
- Duke Health Technology Solutions, Durham, North Carolina, United States
- Duke Primary Care, Duke University Health System, Durham, North Carolina, United States
| | - Laura J. Havrilesky
- Division of Gynecologic Oncology, Department of Obstetrics and Gynecology, Duke University Health System, Durham, North Carolina, United States
| | - Jordan Buuck
- Duke Health Technology Solutions, Durham, North Carolina, United States
| | - David J. Casarett
- Section of Palliative Care, Department of Medicine, Duke University Health System, Durham, North Carolina, United States
| | - Salam Ibrahim
- Duke Health Performance Services, Duke University Health System, Durham, North Carolina, United States
| | - Brittany A. Davidson
- Division of Gynecologic Oncology, Department of Obstetrics and Gynecology, Duke University Health System, Durham, North Carolina, United States
| |
Collapse
|
5
|
Cross DA, Holmgren AJ, Apathy NC. The role of organizations in shaping physician use of electronic health records. Health Serv Res 2024; 59:e14203. [PMID: 37438938 PMCID: PMC10771898 DOI: 10.1111/1475-6773.14203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/14/2023] Open
Abstract
OBJECTIVE The aim of the study was to (1) characterize organizational differences in primary care physicians' electronic health record (EHR) behavior; (2) assess within-organization consistency in EHR behaviors; and (3) identify whether organizational consistency is associated with physician-level efficiency. DATA SOURCES EHR metadata capturing averaged weekly measures of EHR time and documentation composition from 75,124 US primary care physicians across 299 organizations between September 2020 and May 2021 were taken. EHR time measures include active time in orders, chart review, notes, messaging, time spent outside of scheduled hours, and total EHR time. Documentation composition measures include note length and percentage use of templated text or copy/paste. Efficiency is measured as the percent of visits with same-day note completion. STUDY DESIGN All analyses are cross-sectional. Across-organization differences in EHR use and documentation composition are presented via 90th-to-10th percentile ratios of means and SDs. Multilevel modeling with post-estimation variance partitioning assesses the extent of an organizational signature-the proportion of variation in our measures attributable to organizations (versus specialty and individual behaviors). We measured organizational internal consistency for each measure via organization-level SD, which we grouped into quartiles for regression. Association between internally consistent (i.e., low SD) organizational EHR use and physician-level efficiency was assessed with multi-variable OLS models. DATA COLLECTION Extraction from Epic's Signal platform used for measuring provider EHR efficiency. PRINCIPAL FINDINGS EHR time per visit for physicians at a 90th percentile organization is 1.94 times the average EHR time at a 10th percentile organization. There is little evidence, on average, of an organizational signature. However, physicians in organizations with high internal consistency in EHR use demonstrate increased efficiency. Physicians in organizations with the highest internal consistency (top quartile) have a 3.77 percentage point higher same-day visit closure rates compared with peers in bottom quartile organizations (95% confidence interval: 0.0142-0.0612). CONCLUSIONS Results suggest unrealized opportunities for organizations and policymakers to support consistency in how physicians engage in EHR-supported work.
Collapse
Affiliation(s)
- Dori A. Cross
- Division of Health Policy and ManagementUniversity of Minnesota School of Public HealthMinneapolisMinnesotaUSA
| | - A Jay Holmgren
- Department of MedicineUniversity of California San FranciscoSan FranciscoCaliforniaUSA
| | - Nate C. Apathy
- Center for Human Factors in Healthcare, MedStar Health Research InstituteHyattsvilleMarylandUSA
| |
Collapse
|
6
|
Fallon A, Haralambides K, Mazzillo J, Gleber C. Addressing Alert Fatigue by Replacing a Burdensome Interruptive Alert with Passive Clinical Decision Support. Appl Clin Inform 2024; 15:101-110. [PMID: 38086417 PMCID: PMC10830237 DOI: 10.1055/a-2226-8144] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2023] [Accepted: 12/11/2023] [Indexed: 02/02/2024] Open
Abstract
BACKGROUND Recognizing that alert fatigue poses risks to patient safety and clinician wellness, there is a growing emphasis on evaluation and governance of electronic health record clinical decision support (CDS). This is particularly critical for interruptive alerts to ensure that they achieve desired clinical outcomes while minimizing the burden on clinicians. This study describes an improvement effort to address a problematic interruptive alert intended to notify clinicians about patients needing coronavirus disease 2019 (COVID) precautions and how we collaborated with operational leaders to develop an alternative passive CDS system in acute care areas. OBJECTIVES Our dual aim was to reduce the alert burden by redesigning the CDS to adhere to best practices for decision support while also improving the percent of admitted patients with symptoms of possible COVID who had appropriate and timely infection precautions orders. METHODS Iterative changes to CDS design included adjustment to alert triggers and acknowledgment reasons and development of a noninterruptive rule-based order panel for acute care areas. Data on alert burden and appropriate precautions orders on symptomatic admitted patients were followed over time on run and attribute (p) and individuals-moving range control charts. RESULTS At baseline, the COVID alert fired on average 8,206 times per week with an alert per encounter rate of 0.36. After our interventions, the alerts per week decreased to 1,449 and alerts per encounter to 0.07 equating to an 80% reduction for both metrics. Concurrently, the percentage of symptomatic admitted patients with COVID precautions ordered increased from 23 to 61% with a reduction in the mean time between COVID test and precautions orders from 19.7 to -1.3 minutes. CONCLUSION CDS governance, partnering with operational stakeholders, and iterative design led to successful replacement of a frequently firing interruptive alert with less burdensome passive CDS that improved timely ordering of COVID precautions.
Collapse
Affiliation(s)
- Anne Fallon
- Division of Pediatric Hospital Medicine, Department of Pediatrics, University of Rochester Medical Center, Rochester, New York, United States
| | - Kristina Haralambides
- Department of Otolaryngology, University of Rochester Medical Center, Rochester, New York, United States
| | - Justin Mazzillo
- Department of Emergency Medicine, University of Rochester Medical Center, Rochester, New York, United States
| | - Conrad Gleber
- Division of Hospital Medicine, Department of Medicine, University of Rochester Medical Center, Rochester, New York, United States
| |
Collapse
|
7
|
Black KC, Snyder NA, Zhou M, Zhu Z, Uptegraft C, Chintalapani A, Orwoll B. An Electronic Health Record Alert for Inpatient Coronavirus Disease 2019 Vaccinations Increases Vaccination Ordering and Uncovers Workflow Inefficiencies. Appl Clin Inform 2024; 15:192-198. [PMID: 38253337 PMCID: PMC10917607 DOI: 10.1055/a-2250-6305] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Accepted: 01/19/2024] [Indexed: 01/24/2024] Open
Abstract
BACKGROUND Despite mortality benefits, only 19.9% of U.S. adults are fully vaccinated against the coronavirus disease 2019 (COVID-19). The inpatient setting is an opportune environment to update vaccinations, and inpatient electronic health record (EHR) alerts have been shown to increase vaccination rates. OBJECTIVE Our objective was to evaluate whether an EHR alert could increase COVID-19 vaccinations in eligible hospitalized adults by prompting providers to order the vaccine. METHODS This was a quasiexperimental pre-post-interventional design study at an academic and community hospital in the western United States between 1 January, 2021 and 31 October, 2021. Inclusion criteria were unvaccinated hospitalized adults. A soft-stop, interruptive EHR alert prompted providers to order COVID-19 vaccines for those with an expected discharge date within 48 hours and interest in vaccination. The outcome measured was the proportion of all eligible patients for whom vaccines were ordered and administered before and after alert implementation. RESULTS Vaccine ordering rates increased from 4.0 to 13.0% at the academic hospital (odds ratio [OR]: 4.01, 95% confidence interval [CI]: 3.39-4.74, p < 0.001) and from 7.4 to 11.6% at the community hospital (OR: 1.62, 95% CI: 1.23-2.13, p < 0.001) after alert implementation. Administration increased postalert from 3.6 to 12.7% at the academic hospital (OR: 3.21, 95% CI: 2.70-3.82, p < 0.001) but was unchanged at the community hospital, 6.7 to 6.7% (OR: 0.99, 95% CI: 0.73-1.37, p = 0.994). Further analysis revealed infrequent vaccine availability at the community hospital. CONCLUSION Vaccine ordering rates improved at both sites after alert implementation. Vaccine administration rates, however, only improved at the academic hospital, likely due in part to vaccine dispensation inefficiency at the community hospital. This study demonstrates the potential impact of complex workflow patterns on new EHR alert success and provides a rationale for subsequent qualitative workflow analysis with alert implementation.
Collapse
Affiliation(s)
| | | | - Mengyu Zhou
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
| | - Zhen Zhu
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
| | - Colby Uptegraft
- Health Informatics Directorate, Defense Health Agency, Falls Church, Virginia
| | - Ani Chintalapani
- Department of Medicine, Oregon Health and Science University, Portland, Oregon
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon
| | - Benjamin Orwoll
- Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon
- Department of Pediatrics, Oregon Health and Science University, Portland, Oregon
| |
Collapse
|
8
|
Moon J, Chladek JS, Wilson P, Chui MA. Clinical decision support systems in community pharmacies: a scoping review. J Am Med Inform Assoc 2023; 31:231-239. [PMID: 37875066 PMCID: PMC10746304 DOI: 10.1093/jamia/ocad208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Revised: 10/02/2023] [Accepted: 10/09/2023] [Indexed: 10/26/2023] Open
Abstract
OBJECTIVE Clinical decision support systems (CDSS) were implemented in community pharmacies over 40 years ago. However, unlike CDSS studies in other health settings, few studies have been undertaken to evaluate and improve their use in community pharmacies, where billions of prescriptions are filled every year. The aim of this scoping review is to summarize what research has been done surrounding CDSS in community pharmacies and call for rigorous research in this area. MATERIALS AND METHODS Six databases were searched using a combination of controlled vocabulary and keywords relating to community pharmacy and CDSS. After deduplicating the initial search results, 2 independent reviewers conducted title/abstract screening and full-text review. Then, the selected studies were synthesized in terms of investigational/clinical focuses. RESULTS The selected 21 studies investigated the perception of and response to CDSS alerts (n = 7), the impact of CDSS alerts (n = 7), and drug-drug interaction (DDI) alerts (n = 8). Three causes of the failures to prevent DDIs of clinical importance have been noted: the perception of and response to a high volume of DDI alerts, a suboptimal performance of CDSS, and a dearth of sociotechnical considerations for managing workload and workflow. Additionally, 7 studies emphasized the importance of utilizing CDSS for a specific clinical focus, ie, antibiotics, diabetes, opioids, and vaccinations. CONCLUSION Despite the range of topics dealt in the last 30 years, this scoping review confirms that research on CDSS in community pharmacies is limited and disjointed, lacking a comprehensive approach to highlight areas for improvement and ways to optimize CDSS utilization.
Collapse
Affiliation(s)
- Jukrin Moon
- Social and Administrative Sciences Division, School of Pharmacy, University of Wisconsin-Madison, Madison, WI, United States
- Sonderegger Research Center for Improved Medication Outcomes, University of Wisconsin-Madison, Madison, WI, United States
| | - Jason S Chladek
- Social and Administrative Sciences Division, School of Pharmacy, University of Wisconsin-Madison, Madison, WI, United States
- Sonderegger Research Center for Improved Medication Outcomes, University of Wisconsin-Madison, Madison, WI, United States
| | - Paije Wilson
- Ebling Library for the Health Sciences, University of Wisconsin-Madison, Madison, WI, United States
| | - Michelle A Chui
- Social and Administrative Sciences Division, School of Pharmacy, University of Wisconsin-Madison, Madison, WI, United States
- Sonderegger Research Center for Improved Medication Outcomes, University of Wisconsin-Madison, Madison, WI, United States
| |
Collapse
|
9
|
Ledger TS, Brooke-Cowden K, Coiera E. Post-implementation optimization of medication alerts in hospital computerized provider order entry systems: a scoping review. J Am Med Inform Assoc 2023; 30:2064-2071. [PMID: 37812769 PMCID: PMC10654862 DOI: 10.1093/jamia/ocad193] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 09/07/2023] [Accepted: 09/18/2023] [Indexed: 10/11/2023] Open
Abstract
OBJECTIVES A scoping review identified interventions for optimizing hospital medication alerts post-implementation, and characterized the methods used, the populations studied, and any effects of optimization. MATERIALS AND METHODS A structured search was undertaken in the MEDLINE and Embase databases, from inception to August 2023. Articles providing sufficient information to determine whether an intervention was conducted to optimize alerts were included in the analysis. Snowball analysis was conducted to identify additional studies. RESULTS Sixteen studies were identified. Most were based in the United States and used a wide range of clinical software. Many studies used inpatient cohorts and conducted more than one intervention during the trial period. Alert types studied included drug-drug interactions, drug dosage alerts, and drug allergy alerts. Six types of interventions were identified: alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression. The majority of interventions decreased alert quantity and enhanced alert acceptance. Alert quantity decreased with alert inactivation by 1%-25.3%, and with alert severity reclassification by 1%-16.5% in 6 of 7 studies. Alert severity reclassification increased alert acceptance by 4.2%-50.2% and was associated with a 100% acceptance rate for high-severity alerts when implemented. Clinical errors reported in 4 studies were seen to remain stable or decrease. DISCUSSION Post-implementation medication optimization interventions have positive effects for clinicians when applied in a variety of settings. Less well reported are the impacts of these interventions on the clinical care of patients, and how endpoints such as alert quantity contribute to changes in clinician and pharmacist perceptions of alert fatigue. CONCLUSION Well conducted alert optimization can reduce alert fatigue by reducing overall alert quantity, improving clinical acceptance, and enhancing clinical utility.
Collapse
Affiliation(s)
| | - Kalissa Brooke-Cowden
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| | - Enrico Coiera
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, NSW 2109, Australia
| |
Collapse
|
10
|
McCoy AB, Russo EM, Wright A. Clickbusters letter response. J Am Med Inform Assoc 2023; 30:1755. [PMID: 37535834 PMCID: PMC10531101 DOI: 10.1093/jamia/ocad150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 07/27/2023] [Indexed: 08/05/2023] Open
Affiliation(s)
- Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Elise M Russo
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
11
|
Lehmann CU, Subbian V. Advances in Clinical Decision Support Systems: Contributions from the 2022 Literature. Yearb Med Inform 2023; 32:179-183. [PMID: 38147860 PMCID: PMC10751149 DOI: 10.1055/s-0043-1768751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2023] Open
Abstract
OBJECTIVE To summarize significant research contributions published in 2022 in the field of clinical decision support (CDS) systems and select the best papers for the Decision Support section of the International Medical Informatics Association (IMIA) Yearbook 2023. METHODS A renewed search query for identifying CDS scholarship was developed using Medical Subject Headings (MeSH) terms and related keywords. The query was executed in PubMed in January 2023. The search results were reviewed in three stages by two reviewers: title-based triaging, followed by abstract screening, and then full text review. The resulting articles were sent for external review to identity best paper candidates. RESULTS A total of 1,939 articles related to CDS were retrieved. Of these, 11 articles were selected as candidates for best papers. The general themes of the final three best papers are (1) reducing documentation burden through in-line guidance for clinical notes, (2) clinician engagement for continuous improvement of CDS, and (3) mitigating healthcare-related carbon emissions using scalable and accessible CDS, respectively. CONCLUSION The field of clinical decision support remains highly active and dynamic, with innovative contributions to a range of clinical domains from primary to acute care. Interoperability issues, documentation burden, clinician acceptance, and the need for effective integration into existing healthcare workflows are among the prominent challenges and areas of interest faced by CDS implementation efforts.
Collapse
Affiliation(s)
- Christoph U. Lehmann
- Clinical Informatics Center, The University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Vignesh Subbian
- College of Engineering, The University of Arizona, Tucson, Arizona, USA
| | | |
Collapse
|
12
|
Garabedian PM, Rui A, Volk LA, Neville BA, Lipsitz SR, Healey MJ, Bates DW. A Multiyear Survey Evaluating Clinician Electronic Health Record Satisfaction. Appl Clin Inform 2023; 14:632-643. [PMID: 37586414 PMCID: PMC10431971 DOI: 10.1055/s-0043-1770900] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 05/12/2023] [Indexed: 08/18/2023] Open
Abstract
OBJECTIVES We assessed how clinician satisfaction with a vendor electronic health record (EHR) changed over time in the 4 years following the transition from a homegrown EHR system to identify areas for improvement. METHODS We conducted a multiyear survey of clinicians across a large health care system after transitioning to a vendor EHR. Eligible clinicians from the first institution to transition received a survey invitation by email in fall 2016 and then eligible clinicians systemwide received surveys in spring 2018 and spring 2019. The survey included items assessing ease/difficulty of completing tasks and items assessing perceptions of the EHR's value, usability, and impact. One item assessing overall satisfaction and one open-ended question were included. Frequencies and means were calculated, and comparison of means was performed between 2018 and 2019 on all clinicians. A multivariable generalized linear model was performed to predict the outcome of overall satisfaction. RESULTS Response rates for the surveys ranged from 14 to 19%. The mean response from 3 years of surveys for one institution, Brigham and Women's Hospital, increased for overall satisfaction between 2016 (2.85), 2018 (3.01), and 2019 (3.21, p < 0.001). We found no significant differences in mean response for overall satisfaction between all responders of the 2018 survey (3.14) and those of the 2019 survey (3.19). Systemwide, tasks rated the most difficult included "Monitoring patient medication adherence," "Identifying when a referral has not been completed," and "Making a list of patients based on clinical information (e.g., problem, medication)." Clinicians disagreed the most with "The EHR helps me focus on patient care rather than the computer" and "The EHR allows me to complete tasks efficiently." CONCLUSION Survey results indicate room for improvement in clinician satisfaction with the EHR. Usability of EHRs should continue to be an area of focus to ease clinician burden and improve clinician experience.
Collapse
Affiliation(s)
- Pamela M. Garabedian
- Clinical Quality and IS Analysis, Mass General Brigham, Inc., Somerville, Massachusetts, United States
| | - Angela Rui
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, United States
| | - Lynn A. Volk
- Clinical Quality and IS Analysis, Mass General Brigham, Inc., Somerville, Massachusetts, United States
| | - Bridget A. Neville
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, United States
| | - Stuart R. Lipsitz
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, United States
- Harvard Medical School, Harvard University, Ariadne Labs, Boston, Massachusetts, United States
| | - Michael J. Healey
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, United States
- Harvard Medical School, Boston, Massachusetts, United States
| | - David W. Bates
- Division of General Internal Medicine and Primary Care, Brigham and Women's Hospital, Boston, Massachusetts, United States
- Harvard Medical School, Boston, Massachusetts, United States
- Harvard School of Public Health, Harvard University, Boston, Massachusetts
| |
Collapse
|
13
|
Apathy NC, Rotenstein L, Bates DW, Holmgren AJ. Documentation dynamics: Note composition, burden, and physician efficiency. Health Serv Res 2023; 58:674-685. [PMID: 36342001 PMCID: PMC10154172 DOI: 10.1111/1475-6773.14097] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022] Open
Abstract
OBJECTIVE To analyze how physician clinical note length and composition relate to electronic health record (EHR)-based measures of burden and efficiency that have been tied to burnout. DATA SOURCES AND STUDY SETTING Secondary EHR use metadata capturing physician-level measures from 203,728 US-based ambulatory physicians using the Epic Systems EHR between September 2020 and May 2021. STUDY DESIGN In this cross-sectional study, we analyzed physician clinical note length and note composition (e.g., content from manual or templated text). Our primary outcomes were three time-based measures of EHR burden (time writing EHR notes, time in the EHR after-hours, and EHR time on unscheduled days), and one measure of efficiency (percent of visits closed in the same day). We used multivariate regression to estimate the relationship between our outcomes and note length and composition. DATA EXTRACTION Physician-week measures of EHR usage were extracted from Epic's Signal platform used for measuring provider EHR efficiency. We calculated physician-level averages for our measures of interest and assigned physicians to overall note length deciles and note composition deciles from six sources, including templated text, manual text, and copy/paste text. PRINCIPAL FINDINGS Physicians in the top decile of note length demonstrated greater burden and lower efficiency than the median physician, spending 39% more time in the EHR after hours (p < 0.001) and closing 5.6 percentage points fewer visits on the same day (p < 0.001). Copy/paste demonstrated a similar dose/response relationship, with top-decile copy/paste users closing 6.8 percentage points fewer visits on the same day (p < 0.001) and spending more time in the EHR after hours and on days off (both p < 0.001). Templated text (e.g., Epic's SmartTools) demonstrated a non-linear relationship with burden and efficiency, with very low and very high levels of use associated with increased EHR burden and decreased efficiency. CONCLUSIONS "Efficiency tools" like copy/paste and templated text meant to reduce documentation burden and increase provider efficiency may have limited efficacy.
Collapse
Affiliation(s)
- Nate C. Apathy
- National Center for Human Factors in HealthcareMedStar Health Research InstituteWashingtonDistrict of ColumbiaUSA
- Center for Biomedical InformaticsRegenstrief InstituteIndianapolisIndianaUSA
| | - Lisa Rotenstein
- Harvard Medical SchoolBostonMassachusettsUSA
- Population Health Brigham & Women's HospitalBostonMassachusettsUSA
| | - David W. Bates
- Harvard Medical SchoolBostonMassachusettsUSA
- Division of General Internal MedicineBrigham & Women's HospitalBostonMassachusettsUSA
- Present address:
Department of Health Policy and ManagementHarvard School of Public HealthBostonMAUSA
| | - A. Jay Holmgren
- Center for Clinical Informatics and Improvement Research, University of California – San Francisco School of MedicineSan FranciscoCaliforniaUSA
| |
Collapse
|
14
|
Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Using AI-generated suggestions from ChatGPT to optimize clinical decision support. J Am Med Inform Assoc 2023:7136722. [PMID: 37087108 DOI: 10.1093/jamia/ocad072] [Citation(s) in RCA: 49] [Impact Index Per Article: 49.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/28/2023] [Accepted: 04/11/2023] [Indexed: 04/24/2023] Open
Abstract
OBJECTIVE To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. METHODS We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. RESULTS Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. CONCLUSION AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Collapse
Affiliation(s)
- Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Aileen P Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Medicine, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Barron L Patterson
- Department of Pediatrics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Jonathan P Wanderer
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- Department of Anesthesiology, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Robert W Turer
- Department of Emergency Medicine, University of Texas Southwestern Medical Center, Dallas, Texas, USA
- Clinical Informatics Center, University of Texas Southwestern Medical Center, Dallas, Texas, USA
| | - Scott D Nelson
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Allison B McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Dean F Sittig
- School of Biomedical Informatics, University of Texas Health Science Center, Houston, Texas, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|
15
|
Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Assessing the Value of ChatGPT for Clinical Decision Support Optimization. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2023:2023.02.21.23286254. [PMID: 36865144 PMCID: PMC9980251 DOI: 10.1101/2023.02.21.23286254] [Citation(s) in RCA: 17] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/25/2023]
Abstract
Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.
Collapse
|
16
|
Reese TJ, Liu S, Steitz B, McCoy A, Russo E, Koh B, Ancker J, Wright A. Conceptualizing clinical decision support as complex interventions: a meta-analysis of comparative effectiveness trials. J Am Med Inform Assoc 2022; 29:1744-1756. [PMID: 35652167 PMCID: PMC9471719 DOI: 10.1093/jamia/ocac089] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Revised: 04/26/2022] [Accepted: 05/23/2022] [Indexed: 12/26/2022] Open
Abstract
OBJECTIVES Complex interventions with multiple components and behavior change strategies are increasingly implemented as a form of clinical decision support (CDS) using native electronic health record functionality. Objectives of this study were, therefore, to (1) identify the proportion of randomized controlled trials with CDS interventions that were complex, (2) describe common gaps in the reporting of complexity in CDS research, and (3) determine the impact of increased complexity on CDS effectiveness. MATERIALS AND METHODS To assess CDS complexity and identify reporting gaps for characterizing CDS interventions, we used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses reporting tool for complex interventions. We evaluated the effect of increased complexity using random-effects meta-analysis. RESULTS Most included studies evaluated a complex CDS intervention (76%). No studies described use of analytical frameworks or causal pathways. Two studies discussed use of theory but only one fully described the rationale and put it in context of a behavior change. A small but positive effect (standardized mean difference, 0.147; 95% CI, 0.039-0.255; P < .01) in favor of increasing intervention complexity was observed. DISCUSSION While most CDS studies should classify interventions as complex, opportunities persist for documenting and providing resources in a manner that would enable CDS interventions to be replicated and adapted. Unless reporting of the design, implementation, and evaluation of CDS interventions improves, only slight benefits can be expected. CONCLUSION Conceptualizing CDS as complex interventions may help convey the careful attention that is needed to ensure these interventions are contextually and theoretically informed.
Collapse
Affiliation(s)
- Thomas J Reese
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Siru Liu
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Bryan Steitz
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Allison McCoy
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Elise Russo
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Brian Koh
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Jessica Ancker
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| | - Adam Wright
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
| |
Collapse
|