Deployment of Real-time Natural Language Processing and Deep Learning Clinical Decision Support in the Electronic Health Record: Pipeline Implementation for an Opioid Misuse Screener in Hospitalized Adults

Leveraging natural language processing to elucidate real-world clinical decision-making paradigms: A proof of concept study

BACKGROUND

Understanding how clinicians arrive at decisions in actual practice settings is vital for advancing personalized, evidence-based care. However, systematic analysis of qualitative decision data poses challenges.

METHODS

We analyzed transcribed interviews with Hebrew-speaking clinicians on decision processes using natural language processing (NLP). Word frequency and characterized terminology use, while large language models (ChatGPT from OpenAI and Gemini by Google) identified potential cognitive paradigms.

RESULTS

Word frequency analysis of clinician interviews identified experience and knowledge as most influential on decision-making. NLP tentatively recognized heuristics-based reasoning grounded in past cases and intuition as dominant cognitive paradigms. Elements of shared decision-making through individualizing care with patients and families were also observed. Limited Hebrew clinical language resources required developing preliminary lexicons and dynamically adjusting stopwords. Findings also provided preliminary support for heuristics guiding clinical judgment while highlighting needs for broader sampling and enhanced analytical frameworks.

CONCLUSIONS

This study represents the first use of integrated qualitative and computational methods to systematically elucidate clinical decision-making. Findings supported experience-based heuristics guiding cognition. With methodological enhancements, similar analyses could transform global understanding of tailored care delivery. Standardizing interdisciplinary collaborations on developing NLP tools and analytical frameworks may advance equitable, evidence-based healthcare by elucidating real-world clinical reasoning processes across diverse populations and settings.

Comprehensive lifecycle quality control of medical data - automated monitoring and feedback mechanisms based on artificial intelligence

BackgroundDigital healthcare's advance has underscored an urgent requirement for solid medical record quality control, critical for data integrity, surpassing manual methods' inadequacies.ObjectiveThe goal was to develop an AI system to manage medical record quality control comprehensively, using advanced AI like reinforcement learning and NLP to boost management's precision and efficiency.MethodsThis AI system uses a closed-loop framework for real-time record review using natural language processing techniques and reinforcement learning, synchronized with the hospital information system. It features a data layer for monitoring, a service layer for AI analysis, and a presentation layer for user engagement. Its impact was evaluated by comparing quality metrics pre- and post-deployment.ResultsWith the AI system, quality control became fully operational, with review times per record plummeting from 4200 s to 2 s. The share of Grade A records rose from 89.43% to 99.21%, and the system markedly minimized formal and substantive record errors, enhancing completeness and accuracy. The implementation of the artificial intelligence-based medical record quality control system optimizes the quality control process, dynamically regulates the diagnostic behavior of medical staff, and promotes the standardization and normalization of clinical medical record writing.ConclusionsThe AI-driven system significantly upgraded the management of medical records in terms of efficiency and accuracy. It provides a scalable approach for hospitals to refine quality control, propelling healthcare towards heightened intelligence and automation, and foreshadowing AI's pivotal role in future healthcare quality management.

Clinical implementation of AI-based screening for risk for opioid use disorder in hospitalized adults

Adults with opioid use disorder (OUD) are at increased risk for opioid-related complications and repeated hospital admissions. Routine screening for patients at risk for an OUD to prevent complications is not standard practice in many hospitals, leading to missed opportunities for intervention. The adoption of electronic health records (EHRs) and advancements in artificial intelligence (AI) offer a scalable approach to systematically identify at-risk patients for evidence-based care. This pre-post quasi-experimental study evaluated whether an AI-driven OUD screener embedded in the EHR was non-inferior to usual care in identifying patients for addiction medicine consultations, aiming to provide a similarly effective but more scalable alternative to human-led ad hoc consultations. The AI screener used a convolutional neural network to analyze EHR notes in real time, identifying patients at risk and recommending consultations. The primary outcome was the proportion of patients who completed a consultation with an addiction medicine specialist, which included interventions such as outpatient treatment referral, management of complicated withdrawal, medication management for OUD and harm reduction services. The study period consisted of a 16-month pre-intervention phase followed by an 8-month post-intervention phase, during which the AI screener was implemented to support hospital providers in identifying patients for consultation. Consultations did not change between periods (1.35% versus 1.51%, P < 0.001 for non-inferiority). In secondary outcome analysis, the AI screener was associated with a reduction in 30-day readmissions (odds ratio: 0.53, 95% confidence interval: 0.30-0.91, P = 0.02) with an incremental cost of US$6,801 per readmission avoided, demonstrating its potential as a scalable, cost-effective solution for OUD care. ClinicalTrials.gov registration: NCT05745480 .

Identifying Patient-Reported Outcome Measure Documentation in Veterans Health Administration Chiropractic Clinic Notes: Natural Language Processing Analysis

Background

The use of patient-reported outcome measures (PROMs) is an expected component of high-quality, measurement-based chiropractic care. The largest health care system offering integrated chiropractic care is the Veterans Health Administration (VHA). Challenges limit monitoring PROM use as a care quality metric at a national scale in the VHA. Structured data are unavailable, with PROMs often embedded within clinic text notes as unstructured data requiring time-intensive, peer-conducted chart review for evaluation. Natural language processing (NLP) of clinic text notes is one promising solution to extracting care quality data from unstructured text.

Objective

This study aims to test NLP approaches to identify PROMs documented in VHA chiropractic text notes.

Methods

VHA chiropractic notes from October 1, 2017, to September 30, 2020, were obtained from the VHA Musculoskeletal Diagnosis/Complementary and Integrative Health Cohort. A rule-based NLP model built using medspaCy and spaCy was evaluated on text matching and note categorization tasks. SpaCy was used to build bag-of-words, convoluted neural networks, and ensemble models for note categorization. Performance metrics for each model and task included precision, recall, and F-measure. Cross-validation was used to validate performance metric estimates for the statistical and machine-learning models.

Results

Our sample included 377,213 visit notes from 56,628 patients. The rule-based model performance was good for soft-boundary text-matching (precision=81.1%, recall=96.7%, and F-measure=88.2%) and excellent for note categorization (precision=90.3%, recall=99.5%, and F-measure=94.7%). Cross-validation performance of the statistical and machine learning models for the note categorization task was very good overall, but lower than rule-based model performance. The overall prevalence of PROM documentation was low (17.0%).

Conclusions

We evaluated multiple NLP methods across a series of tasks, with optimal performance achieved using a rule-based method. By leveraging NLP approaches, we can overcome the challenges posed by unstructured clinical text notes to track documented PROM use. Overall documented use of PROMs in chiropractic notes was low and highlights a potential for quality improvement. This work represents a methodological advancement in the identification and monitoring of documented use of PROMs to ensure consistent, high-quality chiropractic care for veterans.

Outcomes and Cost-Effectiveness of an EHR-Embedded AI Screener for Identifying Hospitalized Adults at Risk for Opioid Use Disorder

Hospitalized adults with opioid use disorder (OUD) are at high risk for adverse events and rehospitalizations. This pre-post quasi-experimental study evaluated whether an AI-driven OUD screener embedded in the electronic health record (EHR) was non-inferior to usual care in identifying patients for Addiction Medicine consults, aiming to provide a similarly effective but more scalable alternative to human-led ad hoc consultations. The AI screener analyzed EHR notes in real-time with a convolutional neural network to identify patients at risk and recommend consultation. The primary outcome was the proportion of patients receiving consults, comparing a 16-month pre-intervention period to an 8-month post-intervention period with the AI screener. Consults did not change between periods (1.35% vs 1.51%, p < 0.001 for non-inferiority). The AI screener was associated with a reduction in 30-day readmissions (OR: 0.53, 95% CI: 0.30-0.91, p = 0.02) with an incremental cost of $6,801 per readmission avoided, demonstrating its potential as a scalable, cost-effective solution for OUD care.

ClinicalTrialsgov ID

NCT05745480.

Clinical Decision Support and Natural Language Processing in Medicine: Systematic Literature Review

BACKGROUND

Ensuring access to accurate and verified information is essential for effective patient treatment and diagnosis. Although health workers rely on the internet for clinical data, there is a need for a more streamlined approach.

OBJECTIVE

This systematic review aims to assess the current state of artificial intelligence (AI) and natural language processing (NLP) techniques in health care to identify their potential use in electronic health records and automated information searches.

METHODS

A search was conducted in the PubMed, Embase, ScienceDirect, Scopus, and Web of Science online databases for articles published between January 2000 and April 2023. The only inclusion criteria were (1) original research articles and studies on the application of AI-based medical clinical decision support using NLP techniques and (2) publications in English. A Critical Appraisal Skills Programme tool was used to assess the quality of the studies.

RESULTS

The search yielded 707 articles, from which 26 studies were included (24 original articles and 2 systematic reviews). Of the evaluated articles, 21 (81%) explained the use of NLP as a source of data collection, 18 (69%) used electronic health records as a data source, and a further 8 (31%) were based on clinical data. Only 5 (19%) of the articles showed the use of combined strategies for NLP to obtain clinical data. In total, 16 (62%) articles presented stand-alone data review algorithms. Other studies (n=9, 35%) showed that the clinical decision support system alternative was also a way of displaying the information obtained for immediate clinical use.

CONCLUSIONS

The use of NLP engines can effectively improve clinical decision systems' accuracy, while biphasic tools combining AI algorithms and human criteria may optimize clinical diagnosis and treatment flows.

TRIAL REGISTRATION

PROSPERO CRD42022373386; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=373386.

Year 2023 in Biomedical Natural Language Processing: a Tribute to Large Language Models and Generative AI

OBJECTIVES

This synopsis gives insights into scientific publications from 2023 in Natural Language Processing for the biomedical domain. We present the process we followed to identify candidates for NLP's best papers and the two best papers of this year. We also analyze the current trends in the 2023 publications.

METHODS

We queried two bibliographic databases (Medline and the ACL anthology) and refined the outputs through automatic scoring. We then manually shortlisted publications to review and selected candidate papers through an adjudication process. External reviewers assessed the interest of the 13 selected candidates. At last, the section editors chose the best NLP papers.

RESULTS

We collected 2,148 papers published in 2023, of which two were the best and selected as part of this NLP synopsis. Both address language models and propose solutions for data augmenta-tion, domain-specific model adaptation, and model distillation. Work is done on social media con-tent and electronic health records, using deep learning approaches such as ChatGPT and large lan-guage models.

CONCLUSION

Trends from 2023 cover classical NLP tasks (information extraction, text categoriza-tion, sentiment analysis), existing topics from several years (medical education), mainstream applications (Chatbots, generative approaches), and specific issues (cancer, COVID-19, mental health). Specifically for COVID-19, current researches deal with post-COVID-19 conditions, and they explore the understanding of how this pandemic has been managed and welcomed by populations. In addition, due to language models, a few works have been done to process languages other than English, especially using language portability approaches.

Bridging the Gap: Challenges and Strategies for the Implementation of Artificial Intelligence-based Clinical Decision Support Systems in Clinical Practice

OBJECTIVES

Despite the surge in development of artificial intelligence (AI) algorithms to support clinical decision-making, few of these algorithms are used in practice. We reviewed recent literature on clinical deployment of AI-based clinical decision support systems (AI-CDSS), and assessed the maturity of AI-CDSS implementation research. We also aimed to compare and contrast implementation of rule-based CDSS with implementation of AI-CDSS, and to give recommendations for future research in this area.

METHODS

We searched PubMed and Scopus for publications in 2022 and 2023 that focused on AI and/or CDSS, health care, and implementation research, and extracted: clinical setting; clinical task; translational research phase; study design; participants; implementation theory, model or framework used; and key findings.

RESULTS

We selected and described a total of 31 recent papers addressing implementation of AI-CDSS in clinical practice, categorised into four groups: (i) Implementation theories, frameworks, and models (4 papers); (ii) Stakeholder perspectives (22 papers); (iii) Implementation feasibility (three papers); and (iv) Technical infrastructure (2 papers). Stakeholders saw potential benefits of AI-CDSS, but emphasized the need for a strong evidence base and indicated that systems should fit into clinical workflows. There were clear similarities with rule-based CDSS, but also differences with respect to trust and transparency, knowledge, intellectual property, and regulation.

CONCLUSIONS

The field of AI-CDSS implementation research is still in its infancy. It can be strengthened by grounding studies in established theories, models and frameworks from implementation science, focusing on the perspectives of stakeholder groups other than healthcare professionals, conducting more real-world implementation feasibility studies, and through development of reusable technical infrastructure that facilitates rapid deployment of AI-CDSS in clinical practice.

Automated stratification of trauma injury severity across multiple body regions using multi-modal, multi-class machine learning models

OBJECTIVE

The timely stratification of trauma injury severity can enhance the quality of trauma care but it requires intense manual annotation from certified trauma coders. The objective of this study is to develop machine learning models for the stratification of trauma injury severity across various body regions using clinical text and structured electronic health records (EHRs) data.

MATERIALS AND METHODS

Our study utilized clinical documents and structured EHR variables linked with the trauma registry data to create 2 machine learning models with different approaches to representing text. The first one fuses concept unique identifiers (CUIs) extracted from free text with structured EHR variables, while the second one integrates free text with structured EHR variables. Temporal validation was undertaken to ensure the models' temporal generalizability. Additionally, analyses to assess the variable importance were conducted.

RESULTS

Both models demonstrated impressive performance in categorizing leg injuries, achieving high accuracy with macro-F1 scores of over 0.8. Additionally, they showed considerable accuracy, with macro-F1 scores exceeding or near 0.7, in assessing injuries in the areas of the chest and head. We showed in our variable importance analysis that the most important features in the model have strong face validity in determining clinically relevant trauma injuries.

DISCUSSION

The CUI-based model achieves comparable performance, if not higher, compared to the free-text-based model, with reduced complexity. Furthermore, integrating structured EHR data improves performance, particularly when the text modalities are insufficiently indicative.

CONCLUSIONS

Our multi-modal, multiclass models can provide accurate stratification of trauma injury severity and clinically relevant interpretations.

Use of artificial intelligence in critical care: opportunities and obstacles

BACKGROUND

Perhaps nowhere else in the healthcare system than in the intensive care unit environment are the challenges to create useful models with direct time-critical clinical applications more relevant and the obstacles to achieving those goals more massive. Machine learning-based artificial intelligence (AI) techniques to define states and predict future events are commonplace activities of modern life. However, their penetration into acute care medicine has been slow, stuttering and uneven. Major obstacles to widespread effective application of AI approaches to the real-time care of the critically ill patient exist and need to be addressed.

MAIN BODY

Clinical decision support systems (CDSSs) in acute and critical care environments support clinicians, not replace them at the bedside. As will be discussed in this review, the reasons are many and include the immaturity of AI-based systems to have situational awareness, the fundamental bias in many large databases that do not reflect the target population of patient being treated making fairness an important issue to address and technical barriers to the timely access to valid data and its display in a fashion useful for clinical workflow. The inherent "black-box" nature of many predictive algorithms and CDSS makes trustworthiness and acceptance by the medical community difficult. Logistically, collating and curating in real-time multidimensional data streams of various sources needed to inform the algorithms and ultimately display relevant clinical decisions support format that adapt to individual patient responses and signatures represent the efferent limb of these systems and is often ignored during initial validation efforts. Similarly, legal and commercial barriers to the access to many existing clinical databases limit studies to address fairness and generalizability of predictive models and management tools.

CONCLUSIONS

AI-based CDSS are evolving and are here to stay. It is our obligation to be good shepherds of their use and further development.

Comparison of the accuracy of 9 intraocular lens power calculation formulas after SMILE in Chinese myopic eyes

As of 2021, over 2.8 million small-incision lenticule extraction (SMILE) procedures have been performed in China. However, knowledge regarding the selection of intraocular lens (IOL) power calculation formula for post-SMILE cataract patients remains limited. This study included 52 eyes of 26 myopic patients from northern China who underwent SMILE at Tianjin Eye Hospital from September 2022 to February 2023 to investigate the suitability of multiple IOL calculation formulas in post-SMILE patients using a theoretical surgical model. We compared the postoperative results obtained from three artificial intelligence (AI)-based formulas and six conventional formulas provided by the American Society of Cataract and Refractive Surgery (ASCRS). These formulas were applied to calculate IOL power using both total keratometry (TK) and keratometry (K) values, and the results were compared to the preoperative results obtained from the Barrett Universal II (BUII) formula for the SMILE patients. Among the evaluated formulas, the results obtained from the Emmetropia Verifying Optical 2.0 Formula with TK (EVO-TK) (0.40 ± 0.29 D, range 0-1.23 D), Barrett True K with K formula (BTK-K, 0.41 ± 0.26 D, range 0.01-1.19 D), and Masket with K formula (Masket-K, 0.44 ± 0.33 D, range 0.02-1.39 D) demonstrated the closest proximity to BUII. Notably, the highest proportion of prediction errors within 0.5 D was observed with the BTK-K (71.15%), EVO-TK (69.23%), and Masket-K (67.31%), with the BTK-K showing a significantly higher proportion than the Masket-K (p < 0.001). Our research indicates that in post-SMILE patients, the EVO-TK, BTK-K, and Masket-K may yield more accurate calculation results. At their current stage in development, AI-based formulas do not demonstrate significant advantages over conventional formulas. However, the application of historical data can enhance the performance of these formulas.

Using Theories, Models, and Frameworks to Inform Implementation Cycles of Computerized Clinical Decision Support Systems in Tertiary Health Care Settings: Scoping Review

BACKGROUND

Computerized clinical decision support systems (CDSSs) are essential components of modern health system service delivery, particularly within acute care settings such as hospitals. Theories, models, and frameworks may assist in facilitating the implementation processes associated with CDSS innovation and its use within these care settings. These processes include context assessments to identify key determinants, implementation plans for adoption, promoting ongoing uptake, adherence, and long-term evaluation. However, there has been no prior review synthesizing the literature regarding the theories, models, and frameworks that have informed the implementation and adoption of CDSSs within hospitals.

OBJECTIVE

This scoping review aims to identify the theory, model, and framework approaches that have been used to facilitate the implementation and adoption of CDSSs in tertiary health care settings, including hospitals. The rationales reported for selecting these approaches, including the limitations and strengths, are described.

METHODS

A total of 5 electronic databases were searched (CINAHL via EBSCOhost, PubMed, Scopus, PsycINFO, and Embase) to identify studies that implemented or adopted a CDSS in a tertiary health care setting using an implementation theory, model, or framework. No date or language limits were applied. A narrative synthesis was conducted using full-text publications and abstracts. Implementation phases were classified according to the "Active Implementation Framework stages": exploration (feasibility and organizational readiness), installation (organizational preparation), initial implementation (initiating implementation, ie, training), full implementation (sustainment), and nontranslational effectiveness studies.

RESULTS

A total of 81 records (42 full text and 39 abstracts) were included. Full-text studies and abstracts are reported separately. For full-text studies, models (18/42, 43%), followed by determinants frameworks (14/42,33%), were most frequently used to guide adoption and evaluation strategies. Most studies (36/42, 86%) did not list the limitations associated with applying a specific theory, model, or framework.

CONCLUSIONS

Models and related quality improvement methods were most frequently used to inform CDSS adoption. Models were not typically combined with each other or with theory to inform full-cycle implementation strategies. The findings highlight a gap in the application of implementation methods including theories, models, and frameworks to facilitate full-cycle implementation strategies for hospital CDSSs.

DEPLOYR: a technical framework for deploying custom real-time machine learning models into the electronic medical record

OBJECTIVE

Heatlhcare institutions are establishing frameworks to govern and promote the implementation of accurate, actionable, and reliable machine learning models that integrate with clinical workflow. Such governance frameworks require an accompanying technical framework to deploy models in a resource efficient, safe and high-quality manner. Here we present DEPLOYR, a technical framework for enabling real-time deployment and monitoring of researcher-created models into a widely used electronic medical record system.

MATERIALS AND METHODS

We discuss core functionality and design decisions, including mechanisms to trigger inference based on actions within electronic medical record software, modules that collect real-time data to make inferences, mechanisms that close-the-loop by displaying inferences back to end-users within their workflow, monitoring modules that track performance of deployed models over time, silent deployment capabilities, and mechanisms to prospectively evaluate a deployed model's impact.

RESULTS

We demonstrate the use of DEPLOYR by silently deploying and prospectively evaluating 12 machine learning models trained using electronic medical record data that predict laboratory diagnostic results, triggered by clinician button-clicks in Stanford Health Care's electronic medical record.

DISCUSSION

Our study highlights the need and feasibility for such silent deployment, because prospectively measured performance varies from retrospective estimates. When possible, we recommend using prospectively estimated performance measures during silent trials to make final go decisions for model deployment.

CONCLUSION

Machine learning applications in healthcare are extensively researched, but successful translations to the bedside are rare. By describing DEPLOYR, we aim to inform machine learning deployment best practices and help bridge the model implementation gap.