Factors Influencing Clinician Trust in Predictive Clinical Decision Support Systems for In-Hospital Deterioration: Qualitative Descriptive Study

Comparing the influence of social risk factors on machine learning model performance across racial and ethnic groups in home healthcare

This study examined the impact of social risk factors on machine learning model performance for predicting hospitalization and emergency department visits in home healthcare. Using retrospective data from one U.S. home healthcare agency, four models were developed with unstructured social information documented in clinical notes. Performance was compared with and without social factors. A subgroup analyses was conducted by race and ethnicity to assess for fairness. LightGBM performed best overall. Social factors had a modest effect, but findings highlight the feasibility of integrating unstructured social information into machine learning models and the importance of fairness evaluation in home healthcare.

Clinical implementation of AI-based screening for risk for opioid use disorder in hospitalized adults

Adults with opioid use disorder (OUD) are at increased risk for opioid-related complications and repeated hospital admissions. Routine screening for patients at risk for an OUD to prevent complications is not standard practice in many hospitals, leading to missed opportunities for intervention. The adoption of electronic health records (EHRs) and advancements in artificial intelligence (AI) offer a scalable approach to systematically identify at-risk patients for evidence-based care. This pre-post quasi-experimental study evaluated whether an AI-driven OUD screener embedded in the EHR was non-inferior to usual care in identifying patients for addiction medicine consultations, aiming to provide a similarly effective but more scalable alternative to human-led ad hoc consultations. The AI screener used a convolutional neural network to analyze EHR notes in real time, identifying patients at risk and recommending consultations. The primary outcome was the proportion of patients who completed a consultation with an addiction medicine specialist, which included interventions such as outpatient treatment referral, management of complicated withdrawal, medication management for OUD and harm reduction services. The study period consisted of a 16-month pre-intervention phase followed by an 8-month post-intervention phase, during which the AI screener was implemented to support hospital providers in identifying patients for consultation. Consultations did not change between periods (1.35% versus 1.51%, P < 0.001 for non-inferiority). In secondary outcome analysis, the AI screener was associated with a reduction in 30-day readmissions (odds ratio: 0.53, 95% confidence interval: 0.30-0.91, P = 0.02) with an incremental cost of US$6,801 per readmission avoided, demonstrating its potential as a scalable, cost-effective solution for OUD care. ClinicalTrials.gov registration: NCT05745480 .

Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial

The COmmunicating Narrative Concerns Entered by RNs (CONCERN) early warning system (EWS) uses real-time nursing surveillance documentation patterns in its machine learning algorithm to identify deterioration risk. We conducted a 1-year, multisite, pragmatic trial with cluster-randomization of 74 clinical units (37 intervention; 37 usual care) across 2 health systems. Eligible adult hospital encounters were included. We tested if outcomes differed between patients whose care teams were and patients whose care teams were not informed by the CONCERN EWS. Coprimary outcomes were in-hospital mortality (examined as instantaneous risk) and length of stay. Secondary outcomes were cardiopulmonary arrest, sepsis, unanticipated intensive care unit transfers and 30-day hospital readmission. Among 60,893 hospital encounters (33,024 intervention; 27,869 usual care), intervention group encounters had 35.6% decreased instantaneous risk of death (adjusted hazard ratio (HR), 0.64; 95% confidence interval (CI), 0.53-0.78; P < 0.0001), 11.2% decreased length of stay (adjusted incidence rate ratio, 0.91; 95% CI, 0.90-0.93; P < 0.0001), 7.5% decreased instantaneous risk of sepsis (adjusted HR, 0.93; 95% CI, 0.86-0.99; P = 0.0317) and 24.9% increased instantaneous risk of unanticipated intensive care unit transfer (adjusted HR, 1.25; 95% CI, 1.09-1.43; P = 0.0011) compared with usual-care group encounters. No adverse events were reported. A machine learning-based EWS, modeled on nursing surveillance patterns, decreased inpatient deterioration risk with statistical significance. ClinicalTrials.gov registration: NCT03911687 .

Prioritizing Trust in Podiatrists' Preference for AI in Supportive Roles Over Diagnostic Roles in Health Care: Qualitative Interview and Focus Group Study

BACKGROUND

As artificial intelligence (AI) evolves, its roles have expanded from helping out with routine tasks to making complex decisions, once the exclusive domain of human experts. This shift is pronounced in health care, where AI aids in tasks ranging from image recognition in radiology to personalized treatment plans, demonstrating the potential to, at times, surpass human accuracy and efficiency. Despite AI's accuracy in some critical tasks, the adoption of AI in health care is a challenge, in part because of skepticism about being able to rely on AI decisions.

OBJECTIVE

This study aimed to identify and delve into more effective and acceptable ways of integrating AI into a broader spectrum of health care tasks.

METHODS

We included 2 qualitative phases to explore podiatrists' views on AI in health care. Initially, we interviewed 9 podiatrists (7 women and 2 men) with a mean age of 41 (SD 12) years and aimed to capture their sentiments regarding the use and role of AI in their work. Subsequently, a focus group with 5 podiatrists (4 women and 1 man) with a mean age of 54 (SD 10) years delved into AI's supportive and diagnostic roles on the basis of the interviews. All interviews were recorded, transcribed verbatim, and analyzed using Atlas.ti and QDA-Miner, using both thematic analysis for broad patterns and framework analysis for structured insights per established guidelines.

RESULTS

Our research unveiled 9 themes and 3 subthemes, clarifying podiatrists' nuanced views on AI in health care. Key overlapping insights in the 2 phases included a preference for using AI in supportive roles, such as triage, because of its efficiency and process optimization capabilities. There is a discernible hesitancy toward leveraging AI for diagnostic purposes, driven by concerns regarding its accuracy and the essential nature of human expertise. The need for transparency and explainability in AI systems emerged as a critical factor for fostering trust in both phases.

CONCLUSIONS

The findings highlight a complex view from podiatrists on AI, showing openness to its application in supportive roles while exercising caution with diagnostic use. This result is consistent with a careful introduction of AI into health care in roles, such as triage, in which there is initial trust, as opposed to roles that ask the AI for a complete diagnosis. Such strategic adoption can mitigate initial resistance, gradually building the confidence to explore AI's capabilities in more nuanced tasks, including diagnostics, where skepticism is currently more pronounced. Adopting AI stepwise could thus enhance trust and acceptance across a broader range of health care tasks, aligning technology integration with professional comfort and patient care standards.

Artificial Intelligence in Cardiac Surgery: Transforming Outcomes and Shaping the Future

Background: Artificial intelligence (AI) has emerged as a transformative technology in healthcare, with its integration into cardiac surgery offering significant advancements in precision, efficiency, and patient outcomes. However, a comprehensive understanding of AI's applications, benefits, challenges, and future directions in cardiac surgery is needed to inform its safe and effective implementation. Methods: A systematic review was conducted following PRISMA guidelines. Literature searches were performed in PubMed, Scopus, Cochrane Library, Google Scholar, and Web of Science, covering publications from January 2000 to November 2024. Studies focusing on AI applications in cardiac surgery, including risk stratification, surgical planning, intraoperative guidance, and postoperative management, were included. Data extraction and quality assessment were conducted using standardized tools, and findings were synthesized narratively. Results: A total of 121 studies were included in this review. AI demonstrated superior predictive capabilities in risk stratification, with machine learning models outperforming traditional scoring systems in mortality and complication prediction. Robotic-assisted systems enhanced surgical precision and minimized trauma, while computer vision and augmented cognition improved intraoperative guidance. Postoperative AI applications showed potential in predicting complications, supporting patient monitoring, and reducing healthcare costs. However, challenges such as data quality, validation, ethical considerations, and integration into clinical workflows remain significant barriers to widespread adoption. Conclusions: AI has the potential to revolutionize cardiac surgery by enhancing decision making, surgical accuracy, and patient outcomes. Addressing limitations related to data quality, bias, validation, and regulatory frameworks is essential for its safe and effective implementation. Future research should focus on interdisciplinary collaboration, robust testing, and the development of ethical and transparent AI systems to ensure equitable and sustainable advancements in cardiac surgery.

Bridging the Gap: A Mixed-Methods Study on Factors Influencing Breast Cancer Clinicians' Decisions to Use Clinical Prediction Models

Background. Clinical prediction models provide tailored risk estimates that can help guide decisions in breast cancer care. Despite their potential, few models are widely used in clinical practice. We aimed to identify the factors influencing breast cancer clinicians' decisions to adopt prediction models and assess their relative importance. Methods. We conducted a mixed-methods study, beginning with semi-structured interviews, followed by a nationwide online survey. Thematic analysis was used to qualitatively summarize the interviews and identify key factors. For the survey, we used descriptive analysis to characterize the sample and Mann-Whitney U and Kruskal-Wallis tests to explore differences in score (0 = not important to 10 = very important) distributions. Results. Interviews (N = 16) identified eight key factors influencing model use. Practical/methodological factors included accessibility, cost, understandability, objective accuracy, actionability, and clinical relevance. Perceptual factors included acceptability, subjective accuracy, and risk communication. In the survey (N = 146; 137 model users), clinicians ranked online accessibility (median score = 9 [interquartile range = 8-10]) as most important. Cost was also highly rated, with preferences for freely available models (9 [8-10]) and those with reimbursable tests (8 [8-10]). Formal regulatory approval (7 [5-8]) and direct integration with electronic health records (6 [3-8]) were considered less critical. Subgroup analysis revealed differences in score distributions; for example, clinicians from general hospitals prioritized inclusion of new biomarkers more than those in academic settings. Conclusions. Breast cancer clinicians' decisions to initiate use of prediction models are influenced by practical and perceptual factors, extending beyond technical metrics such as discrimination and calibration. Addressing these factors more holistically through collaborative efforts between model developers, clinicians, and communication and implementation experts, for instance, by developing clinician-friendly online tools that prioritize usability and local adaptability, could increase model uptake.

Highlights

Accessibility, cost, and practical considerations, such as ease of use and clinical utility, were prioritized slightly more than technical validation metrics, such as discrimination and calibration, when deciding to start using a clinical prediction model.Most breast cancer clinicians valued models with clear inputs (e.g., variable definitions, cutoffs) and outputs; few were interested in the exact model specifications.Perceptual or subjective factors, including perceived accuracy and peer acceptability, also influenced model adoption but were secondary to practical considerations.Sociodemographic variables, such as clinical specialization and hospital setting, influenced the importance of factors for model use.

End user experience of a widely used artificial intelligence based sepsis system

Objectives

Research on the Epic Sepsis System (ESS) has predominantly focused on technical accuracy, neglecting the user experience of healthcare professionals. Understanding these experiences is crucial for the design of Artificial Intelligence (AI) systems in clinical settings. This study aims to explore the socio-technical dynamics affecting ESS adoption and use, based on user perceptions and experiences.

Materials and Methods

Resident doctors and nurses with recent ESS interaction were interviewed using purposive sampling until data saturation. A content analysis was conducted using Dedoose software, with codes generated from Sittig and Singh's and Salwei and Carayon's frameworks, supplemented by inductive coding for emerging themes.

Results

Interviews with 10 healthcare providers revealed mixed but generally positive or neutral perceptions of the ESS. Key discussion points included its workflow integration and usability. Findings were organized into 2 main domains: workflow fit, and usability and utility, highlighting the system's seamless electronic health record integration and identifying design gaps.

Discussion

This study offers insights into clinicians' experiences with the ESS, emphasizing the socio-technical factors that influence its adoption and effective use. The positive reception was tempered by identified design issues, with clinician perceptions varying by their professional experience and frequency of ESS interaction.

Conclusion

The findings highlight the need for ongoing ESS refinement, emphasizing a balance between technological advancement and clinical practicality. This research contributes to the understanding of AI system adoption in healthcare, suggesting improvements for future clinical AI tools.

Implementation of Machine Learning Applications in Health Care Organizations: Systematic Review of Empirical Studies

BACKGROUND

There is a growing enthusiasm for machine learning (ML) among academics and health care practitioners. Despite the transformative potential of ML-based applications for patient care, their uptake and implementation in health care organizations are sporadic. Numerous challenges currently impede or delay the widespread implementation of ML in clinical practice, and limited knowledge is available regarding how these challenges have been addressed.

OBJECTIVE

This work aimed to (1) examine the characteristics of ML-based applications and the implementation process in clinical practice, using the Consolidated Framework for Implementation Research (CFIR) for theoretical guidance and (2) synthesize the strategies adopted by health care organizations to foster successful implementation of ML.

METHODS

A systematic literature review was conducted based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search was conducted in PubMed, Scopus, and Web of Science over a 10-year period (2013-2023). The search strategy was built around 4 blocks of keywords (artificial intelligence, implementation, health care, and study type). Only empirical studies documenting the implementation of ML applications in clinical settings were considered. The implementation process was investigated using a thematic analysis and coding procedure.

RESULTS

Thirty-four studies were selected for data synthesis. Selected papers were relatively recent, with only 9% (3/34) of records published before 2019. ML-based applications were implemented mostly within hospitals (29/34, 85%). In terms of clinical workflow, ML-based applications supported mostly prognosis (20/34, 59%) and diagnosis (10/34, 29%). The implementation efforts were analyzed using CFIR domains. As for the inner setting domain, access to knowledge and information (12/34, 35%), information technology infrastructure (11/34, 32%), and organizational culture (9/34, 26%) were among the most observed dimensions influencing the success of implementation. As for the ML innovation itself, factors deemed relevant were its design (15/34, 44%), the relative advantage with respect to existing clinical practice (14/34, 41%), and perceived complexity (14/34, 41%). As for the other domains (ie, processes, roles, and outer setting), stakeholder engagement (12/34, 35%), reflecting and evaluating practices (11/34, 32%), and the presence of implementation leaders (9/34, 26%) were the main factors identified as important.

CONCLUSIONS

This review sheds some light on the factors that are relevant and that should be accounted for in the implementation process of ML-based applications in health care. While the relevance of ML-specific dimensions, like trust, emerges clearly across several implementation domains, the evidence from this review highlighted that relevant implementation factors are not necessarily specific for ML but rather transversal for digital health technologies. More research is needed to further clarify the factors that are relevant to implementing ML-based applications at the organizational level and to support their uptake within health care organizations.

TRIAL REGISTRATION

PROSPERO 403873; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=403873.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/47971.

Exploring home healthcare clinicians' needs for using clinical decision support systems for early risk warning

OBJECTIVES

To explore home healthcare (HHC) clinicians' needs for Clinical Decision Support Systems (CDSS) information delivery for early risk warning within HHC workflows.

METHODS

Guided by the CDS "Five-Rights" framework, we conducted semi-structured interviews with multidisciplinary HHC clinicians from April 2023 to August 2023. We used deductive and inductive content analysis to investigate informants' responses regarding CDSS information delivery.

RESULTS

Interviews with thirteen HHC clinicians yielded 16 codes mapping to the CDS "Five-Rights" framework (right information, right person, right format, right channel, right time) and 11 codes for unintended consequences and training needs. Clinicians favored risk levels displayed in color-coded horizontal bars, concrete risk indicators in bullet points, and actionable instructions in the existing EHR system. They preferred non-intrusive risk alerts requiring mandatory confirmation. Clinicians anticipated risk information updates aligned with patient's condition severity and their visit pace. Additionally, they requested training to understand the CDSS's underlying logic, and raised concerns about information accuracy and data privacy.

DISCUSSION

While recognizing CDSS's value in enhancing early risk warning, clinicians highlighted concerns about increased workload, alert fatigue, and CDSS misuse. The top risk factors identified by machine learning algorithms, especially text features, can be ambiguous due to a lack of context. Future research should ensure that CDSS outputs align with clinical evidence and are explainable.

CONCLUSION

This study identified HHC clinicians' expectations, preferences, adaptations, and unintended uses of CDSS for early risk warning. Our findings endorse operationalizing the CDS "Five-Rights" framework to optimize CDSS information delivery and integration into HHC workflows.

Atrial fibrillation after cardiac surgery: identifying candidate predictors through a Delphi process

OBJECTIVES

This study was undertaken to identify potential predictors of atrial fibrillation after cardiac surgery (AFACS) through a modified Delphi process and expert consensus. These will supplement predictors identified through a systematic review and cohort study to inform the development of two AFACS prediction models as part of the PARADISE project (NCT05255224). Atrial fibrillation is a common complication after cardiac surgery. It is associated with worse postoperative outcomes. Reliable prediction of AFACS would enable risk stratification and targeted prevention. Systematic identification of candidate predictors is important to improve validity of AFACS prediction tools.

DESIGN

This study is a Delphi consensus exercise.

SETTING

This study was undertaken through remote participation.

PARTICIPANTS

The participants are an international multidisciplinary panel of experts selected through national research networks.

INTERVENTIONS

This is a two-stage consensus exercise consisting of generating a long list of variables, followed by refinement by voting and retaining variables selected by at least 40% of panel members.

RESULTS

The panel comprised 15 experts who participated in both stages, comprising cardiac intensive care physicians (n=3), cardiac anaesthetists (n=2), cardiac surgeons (n=1), cardiologists (n=4), cardiac pharmacists (n=1), critical care nurses (n=1), cardiac nurses (n=1) and patient representatives (n=2). Our Delphi process highlighted candidate AFACS predictors, including both patient factors and those related to the surgical intervention. We generated a final list of 72 candidate predictors. The final list comprised 3 demographic, 29 comorbidity, 4 vital sign, 13 intraoperative, 10 postoperative investigation and 13 postoperative intervention predictors.

CONCLUSIONS

A Delphi consensus exercise has the potential to highlight predictors beyond the scope of existing literature. This method proved effective in identifying a range of candidate AFACS predictors. Our findings will inform the development of future AFACS prediction tools as part of the larger PARADISE project.

TRIAL REGISTRATION NUMBER

NCT05255224.

Barriers and facilitators to implementing imaging-based diagnostic artificial intelligence-assisted decision-making software in hospitals in China: a qualitative study using the updated Consolidated Framework for Implementation Research

OBJECTIVES

To identify the barriers and facilitators to the successful implementation of imaging-based diagnostic artificial intelligence (AI)-assisted decision-making software in China, using the updated Consolidated Framework for Implementation Research (CFIR) as a theoretical basis to develop strategies that promote effective implementation.

DESIGN

This qualitative study involved semistructured interviews with key stakeholders from both clinical settings and industry. Interview guide development, coding, analysis and reporting of findings were thoroughly informed by the updated CFIR.

SETTING

Four healthcare institutions in Beijing and Shanghai and two vendors of AI-assisted decision-making software for lung nodules detection and diabetic retinopathy screening were selected based on purposive sampling.

PARTICIPANTS

A total of 23 healthcare practitioners, 6 hospital informatics specialists, 4 hospital administrators and 7 vendors of the selected AI-assisted decision-making software were included in the study.

RESULTS

Within the 5 CFIR domains, 10 constructs were identified as barriers, 8 as facilitators and 3 as both barriers and facilitators. Major barriers included unsatisfactory clinical performance (Innovation); lack of collaborative network between primary and tertiary hospitals, lack of information security measures and certification (outer setting); suboptimal data quality, misalignment between software functions and goals of healthcare institutions (inner setting); unmet clinical needs (individuals). Key facilitators were strong empirical evidence of effectiveness, improved clinical efficiency (innovation); national guidelines related to AI, deployment of AI software in peer hospitals (outer setting); integration of AI software into existing hospital systems (inner setting) and involvement of clinicians (implementation process).

CONCLUSIONS

The study findings contributed to the ongoing exploration of AI integration in healthcare from the perspective of China, emphasising the need for a comprehensive approach considering both innovation-specific factors and the broader organisational and contextual dynamics. As China and other developing countries continue to advance in adopting AI technologies, the derived insights could further inform healthcare practitioners, industry stakeholders and policy-makers, guiding policies and practices that promote the successful implementation of imaging-based diagnostic AI-assisted decision-making software in healthcare for optimal patient care.

Recommendations to promote fairness and inclusion in biomedical AI research and clinical use

OBJECTIVE

Understanding and quantifying biases when designing and implementing actionable approaches to increase fairness and inclusion is critical for artificial intelligence (AI) in biomedical applications.

METHODS

In this Special Communication, we discuss how bias is introduced at different stages of the development and use of AI applications in biomedical sciences and health care. We describe various AI applications and their implications for fairness and inclusion in sections on 1) Bias in Data Source Landscapes, 2) Algorithmic Fairness, 3) Uncertainty in AI Predictions, 4) Explainable AI for Fairness and Equity, and 5) Sociological/Ethnographic Issues in Data and Results Representation.

RESULTS

We provide recommendations to address biases when developing and using AI in clinical applications.

CONCLUSION

These recommendations can be applied to informatics research and practice to foster more equitable and inclusive health care systems and research discoveries.

Graph Artificial Intelligence in Medicine

In clinical artificial intelligence (AI), graph representation learning, mainly through graph neural networks and graph transformer architectures, stands out for its capability to capture intricate relationships and structures within clinical datasets. With diverse data-from patient records to imaging-graph AI models process data holistically by viewing modalities and entities within them as nodes interconnected by their relationships. Graph AI facilitates model transfer across clinical tasks, enabling models to generalize across patient populations without additional parameters and with minimal to no retraining. However, the importance of human-centered design and model interpretability in clinical decision-making cannot be overstated. Since graph AI models capture information through localized neural transformations defined on relational datasets, they offer both an opportunity and a challenge in elucidating model rationale. Knowledge graphs can enhance interpretability by aligning model-driven insights with medical knowledge. Emerging graph AI models integrate diverse data modalities through pretraining, facilitate interactive feedback loops, and foster human-AI collaboration, paving the way toward clinically meaningful predictions.

"Following the data": perceptions of and willingness to use clinical decision support tools to inform HIV care among Haitian clinicians

Background

Clinical decision support (CDS) tools can support HIV care, including through case tracking, treatment and medication monitoring, and promoting provider compliance with care guidelines. There has been limited research into the technical, organizational, and behavioral factors that impact perceptions of and willingness to use CDS tools at scale in resource-limited settings, including in Haiti.

Methods

Our sample included fifteen purposively chosen Haitian HIV program experts, including active clinicians and HIV program managers. Participants completed structured quantitative surveys and one-on-one qualitative semi-structured interviews.

Results

Study participants had high levels of familiarity and experience with CDS tools. The primary motivator for CDS tool use was a perceived benefit to quality of care, including improved provider time use, efficiency, and decision-making ability, and patient outcomes. Participants highlighted decision-making autonomy and how CDS tools could support provider decision making but should not supplant provider knowledge and experience. Participants highlighted the need for sufficient provider training/sensitization, inclusion of providers in the system design process, and prioritization of tool user-friendliness as key mechanisms to drive tool use and impact. Some participants noted that systemic issues, such as limited laboratory capacity, may reduce the usefulness of CDS alerts, particularly concerning differentiated care and priority viral load testing.

Conclusion

Respondents had largely positive perceptions of EMRs and CDS tools, particularly due to perceived improvements in quality of care. To improve tool use, stakeholders should prioritize tool user-friendliness and provider training. Addressing systemic health system issues is necessary to unlock the full potential of these tools.

Bridging the Gap: Challenges and Strategies for the Implementation of Artificial Intelligence-based Clinical Decision Support Systems in Clinical Practice

OBJECTIVES

Despite the surge in development of artificial intelligence (AI) algorithms to support clinical decision-making, few of these algorithms are used in practice. We reviewed recent literature on clinical deployment of AI-based clinical decision support systems (AI-CDSS), and assessed the maturity of AI-CDSS implementation research. We also aimed to compare and contrast implementation of rule-based CDSS with implementation of AI-CDSS, and to give recommendations for future research in this area.

METHODS

We searched PubMed and Scopus for publications in 2022 and 2023 that focused on AI and/or CDSS, health care, and implementation research, and extracted: clinical setting; clinical task; translational research phase; study design; participants; implementation theory, model or framework used; and key findings.

RESULTS

We selected and described a total of 31 recent papers addressing implementation of AI-CDSS in clinical practice, categorised into four groups: (i) Implementation theories, frameworks, and models (4 papers); (ii) Stakeholder perspectives (22 papers); (iii) Implementation feasibility (three papers); and (iv) Technical infrastructure (2 papers). Stakeholders saw potential benefits of AI-CDSS, but emphasized the need for a strong evidence base and indicated that systems should fit into clinical workflows. There were clear similarities with rule-based CDSS, but also differences with respect to trust and transparency, knowledge, intellectual property, and regulation.

CONCLUSIONS

The field of AI-CDSS implementation research is still in its infancy. It can be strengthened by grounding studies in established theories, models and frameworks from implementation science, focusing on the perspectives of stakeholder groups other than healthcare professionals, conducting more real-world implementation feasibility studies, and through development of reusable technical infrastructure that facilitates rapid deployment of AI-CDSS in clinical practice.

Multisite Pragmatic Cluster-Randomized Controlled Trial of the CONCERN Early Warning System

Importance

Late predictions of hospitalized patient deterioration, resulting from early warning systems (EWS) with limited data sources and/or a care team's lack of shared situational awareness, contribute to delays in clinical interventions. The COmmunicating Narrative Concerns Entered by RNs (CONCERN) Early Warning System (EWS) uses real-time nursing surveillance documentation patterns in its machine learning algorithm to identify patients' deterioration risk up to 42 hours earlier than other EWSs.

Objective

To test our a priori hypothesis that patients with care teams informed by the CONCERN EWS intervention have a lower mortality rate and shorter length of stay (LOS) than the patients with teams not informed by CONCERN EWS.

Design

One-year multisite, pragmatic controlled clinical trial with cluster-randomization of acute and intensive care units to intervention or usual-care groups.

Setting

Two large U.S. health systems.

Participants

Adult patients admitted to acute and intensive care units, excluding those on hospice/palliative/comfort care, or with Do Not Resuscitate/Do Not Intubate orders.

Intervention

The CONCERN EWS intervention calculates patient deterioration risk based on nurses' concern levels measured by surveillance documentation patterns, and it displays the categorical risk score (low, increased, high) in the electronic health record (EHR) for care team members.

Main Outcomes and Measures

Primary outcomes: in-hospital mortality, LOS; survival analysis was used. Secondary outcomes: cardiopulmonary arrest, sepsis, unanticipated ICU transfers, 30-day hospital readmission.

Results

A total of 60 893 hospital encounters (33 024 intervention and 27 869 usual-care) were included. Both groups had similar patient age, race, ethnicity, and illness severity distributions. Patients in the intervention group had a 35.6% decreased risk of death (adjusted hazard ratio [HR], 0.644; 95% confidence interval [CI], 0.532-0.778; P<.0001), 11.2% decreased LOS (adjusted incidence rate ratio, 0.914; 95% CI, 0.902-0.926; P<.0001), 7.5% decreased risk of sepsis (adjusted HR, 0.925; 95% CI, 0.861-0.993; P=.0317), and 24.9% increased risk of unanticipated ICU transfer (adjusted HR, 1.249; 95% CI, 1.093-1.426; P=.0011) compared with patients in the usual-care group.

Conclusions and Relevance

A hospital-wide EWS based on nursing surveillance patterns decreased in-hospital mortality, sepsis, and LOS when integrated into the care team's EHR workflow.

Trial Registration

ClinicalTrials.gov Identifier: NCT03911687.

Opportunities for incorporating intersectionality into biomedical informatics

Many approaches in biomedical informatics (BMI) rely on the ability to define, gather, and manipulate biomedical data to support health through a cyclical research-practice lifecycle. Researchers within this field are often fortunate to work closely with healthcare and public health systems to influence data generation and capture and have access to a vast amount of biomedical data. Many informaticists also have the expertise to engage with stakeholders, develop new methods and applications, and influence policy. However, research and policy that explicitly seeks to address the systemic drivers of health would more effectively support health. Intersectionality is a theoretical framework that can facilitate such research. It holds that individual human experiences reflect larger socio-structural level systems of privilege and oppression, and cannot be truly understood if these systems are examined in isolation. Intersectionality explicitly accounts for the interrelated nature of systems of privilege and oppression, providing a lens through which to examine and challenge inequities. In this paper, we propose intersectionality as an intervention into how we conduct BMI research. We begin by discussing intersectionality's history and core principles as they apply to BMI. We then elaborate on the potential for intersectionality to stimulate BMI research. Specifically, we posit that our efforts in BMI to improve health should address intersectionality's five key considerations: (1) systems of privilege and oppression that shape health; (2) the interrelated nature of upstream health drivers; (3) the nuances of health outcomes within groups; (4) the problematic and power-laden nature of categories that we assign to people in research and in society; and (5) research to inform and support social change.

Designing the User Interface of a Nitroglycerin Dose Titration Decision Support System: User-Centered Design Study

BACKGROUND

 Nurses adjust intravenous nitroglycerin infusions to provide acute relief for angina by manually increasing or decreasing the dosage. However, titration can pose challenges, as excessively high doses can lead to hypotension, and low doses may result in inadequate pain relief. Clinical decision support systems (CDSSs) that predict changes in blood pressure for nitroglycerin dose adjustments may assist nurses with titration.

OBJECTIVE

 This study aimed to design a user interface for a CDSS for nitroglycerin dose titration (Nitroglycerin Dose Titration Decision Support System [nitro DSS]).

METHODS

 A user-centered design (UCD) approach, consisting of an initial qualitative study with semistructured interviews to identify design specifications for prototype development, was used. This was followed by three iterative rounds of usability testing. Nurses with experience titrating nitroglycerin infusions in coronary care units participated.

RESULTS

 A total of 20 nurses participated, including 7 during the qualitative study and 15 during usability testing (2 nurses participated in both phases). Analysis of the qualitative data revealed four themes for the interface design to be (1) clear and consistent, (2) vigilant, (3) interoperable, and (4) reliable. The major elements of the final prototype included a feature for viewing the predicted and actual blood pressure over time to determine the reliability of the predictions, a drop-down option to report patient side effects, a feature to report reasons for not accepting the prediction, and a visual alert indicating any systolic blood pressure predictions below 90 mm Hg. Nurses' ratings on the questionnaires indicated excellent usability and acceptability of the final nitro DSS prototype.

CONCLUSION

 This study successfully applied a UCD approach to collaborate with nurses in developing a user interface for the nitro DSS that supports the clinical decision-making of nurses titrating nitroglycerin.

Exploring Algorithmic Explainability: Generating Explainable AI Insights for Personalized Clinical Decision Support Focused on Cannabis Intoxication in Young Adults

As an increasing number of states adopt more permissive cannabis regulations, the necessity of gaining a comprehensive understanding of cannabis's effects on young adults has grown exponentially, driven by its escalating prevalence of use. By leveraging popular eXplainable Artificial Intelligence (XAI) techniques such as SHAP (SHapley Additive exPlanations), rule-based explanations, intrinsically interpretable models, and counterfactual explanations, we undertake an exploratory but in-depth examination of the impact of cannabis use on individual behavioral patterns and physiological states. This study explores the possibility of facilitating algorithmic decision-making by combining interpretable artificial intelligence (XAI) techniques with sensor data, with the aim of providing researchers and clinicians with personalized analyses of cannabis intoxication behavior. SHAP analyzes the importance and quantifies the impact of specific factors such as environmental noise or heart rate, enabling clinicians to pinpoint influential behaviors and environmental conditions. SkopeRules simplify the understanding of cannabis use for a specific activity or environmental use. Decision trees provide a clear visualization of how factors interact to influence cannabis consumption. Counterfactual models help identify key changes in behaviors or conditions that may alter cannabis use outcomes, to guide effective individualized intervention strategies. This multidimensional analytical approach not only unveils changes in behavioral and physiological states after cannabis use, such as frequent fluctuations in activity states, nontraditional sleep patterns, and specific use habits at different times and places, but also highlights the significance of individual differences in responses to cannabis use. These insights carry profound implications for clinicians seeking to gain a deeper understanding of the diverse needs of their patients and for tailoring precisely targeted intervention strategies. Furthermore, our findings highlight the pivotal role that XAI technologies could play in enhancing the transparency and interpretability of Clinical Decision Support Systems (CDSS), with a particular focus on substance misuse treatment. This research significantly contributes to ongoing initiatives aimed at advancing clinical practices that aim to prevent and reduce cannabis-related harms to health, positioning XAI as a supportive tool for clinicians and researchers alike.

Decision discovery using clinical decision support system decision log data for supporting the nurse decision-making process

BACKGROUND

Decision-making in healthcare is increasingly complex; notably in hospital environments where the information density is high, e.g., emergency departments, oncology departments, and psychiatry departments. This study aims to discover decisions from logged data to improve the decision-making process.

METHODS

The Design Science Research Methodology (DSRM) was chosen to design an artifact (algorithm) for the discovery and visualization of decisions. The DSRM's different activities are explained, from the definition of the problem to the evaluation of the artifact. During the design and development activities, the algorithm itself is created. During the demonstration and evaluation activities, the algorithm was tested with an authentic synthetic dataset.

RESULTS

The results show the design and simulation of an algorithm for the discovery and visualization of decisions. A fuzzy classifier algorithm was adapted for (1) discovering decisions from a decision log and (2) visualizing the decisions using the Decision Model and Notation standard.

CONCLUSIONS

In this paper, we show that decisions can be discovered from a decision log and visualized for the improvement of the decision-making process of healthcare professionals or to support the periodic evaluation of protocols and guidelines.

User-Centered Design of a Machine Learning Dashboard for Prediction of Postoperative Complications

BACKGROUND

Machine learning models can help anesthesiology clinicians assess patients and make clinical and operational decisions, but well-designed human-computer interfaces are necessary for machine learning model predictions to result in clinician actions that help patients. Therefore, the goal of this study was to apply a user-centered design framework to create a user interface for displaying machine learning model predictions of postoperative complications to anesthesiology clinicians.

METHODS

Twenty-five anesthesiology clinicians (attending anesthesiologists, resident physicians, and certified registered nurse anesthetists) participated in a 3-phase study that included (phase 1) semistructured focus group interviews and a card sorting activity to characterize user workflows and needs; (phase 2) simulated patient evaluation incorporating a low-fidelity static prototype display interface followed by a semistructured interview; and (phase 3) simulated patient evaluation with concurrent think-aloud incorporating a high-fidelity prototype display interface in the electronic health record. In each phase, data analysis included open coding of session transcripts and thematic analysis.

RESULTS

During the needs assessment phase (phase 1), participants voiced that (a) identifying preventable risk related to modifiable risk factors is more important than nonpreventable risk, (b) comprehensive patient evaluation follows a systematic approach that relies heavily on the electronic health record, and (c) an easy-to-use display interface should have a simple layout that uses color and graphs to minimize time and energy spent reading it. When performing simulations using the low-fidelity prototype (phase 2), participants reported that (a) the machine learning predictions helped them to evaluate patient risk, (b) additional information about how to act on the risk estimate would be useful, and (c) correctable problems related to textual content existed. When performing simulations using the high-fidelity prototype (phase 3), usability problems predominantly related to the presentation of information and functionality. Despite the usability problems, participants rated the system highly on the System Usability Scale (mean score, 82.5; standard deviation, 10.5).

CONCLUSIONS

Incorporating user needs and preferences into the design of a machine learning dashboard results in a display interface that clinicians rate as highly usable. Because the system demonstrates usability, evaluation of the effects of implementation on both process and clinical outcomes is warranted.

Using artificial intelligence to promote equitable care for inpatients with language barriers and complex medical needs: clinical stakeholder perspectives

OBJECTIVES

Inpatients with language barriers and complex medical needs suffer disparities in quality of care, safety, and health outcomes. Although in-person interpreters are particularly beneficial for these patients, they are underused. We plan to use machine learning predictive analytics to reliably identify patients with language barriers and complex medical needs to prioritize them for in-person interpreters.

MATERIALS AND METHODS

This qualitative study used stakeholder engagement through semi-structured interviews to understand the perceived risks and benefits of artificial intelligence (AI) in this domain. Stakeholders included clinicians, interpreters, and personnel involved in caring for these patients or for organizing interpreters. Data were coded and analyzed using NVIVO software.

RESULTS

We completed 49 interviews. Key perceived risks included concerns about transparency, accuracy, redundancy, privacy, perceived stigmatization among patients, alert fatigue, and supply-demand issues. Key perceived benefits included increased awareness of in-person interpreters, improved standard of care and prioritization for interpreter utilization; a streamlined process for accessing interpreters, empowered clinicians, and potential to overcome clinician bias.

DISCUSSION

This is the first study that elicits stakeholder perspectives on the use of AI with the goal of improved clinical care for patients with language barriers. Perceived benefits and risks related to the use of AI in this domain, overlapped with known hazards and values of AI but some benefits were unique for addressing challenges with providing interpreter services to patients with language barriers.

CONCLUSION

Artificial intelligence to identify and prioritize patients for interpreter services has the potential to improve standard of care and address healthcare disparities among patients with language barriers.

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

Machine learning (ML)-based risk prediction models hold the potential to support the health-care setting in several ways; however, use of such models is scarce. We aimed to review health-care professional (HCP) and patient perceptions of ML risk prediction models in published literature, to inform future risk prediction model development. Following database and citation searches, we identified 41 articles suitable for inclusion. Article quality varied with qualitative studies performing strongest. Overall, perceptions of ML risk prediction models were positive. HCPs and patients considered that models have the potential to add benefit in the health-care setting. However, reservations remain; for example, concerns regarding data quality for model development and fears of unintended consequences following ML model use. We identified that public views regarding these models might be more negative than HCPs and that concerns (eg, extra demands on workload) were not always borne out in practice. Conclusions are tempered by the low number of patient and public studies, the absence of participant ethnic diversity, and variation in article quality. We identified gaps in knowledge (particularly views from under-represented groups) and optimum methods for model explanation and alerts, which require future research.

Trust in and Acceptance of Artificial Intelligence Applications in Medicine: Mixed Methods Study

BACKGROUND

Artificial intelligence (AI)-powered technologies are being increasingly used in almost all fields, including medicine. However, to successfully implement medical AI applications, ensuring trust and acceptance toward such technologies is crucial for their successful spread and timely adoption worldwide. Although AI applications in medicine provide advantages to the current health care system, there are also various associated challenges regarding, for instance, data privacy, accountability, and equity and fairness, which could hinder medical AI application implementation.

OBJECTIVE

The aim of this study was to identify factors related to trust in and acceptance of novel AI-powered medical technologies and to assess the relevance of those factors among relevant stakeholders.

METHODS

This study used a mixed methods design. First, a rapid review of the existing literature was conducted, aiming to identify various factors related to trust in and acceptance of novel AI applications in medicine. Next, an electronic survey including the rapid review-derived factors was disseminated among key stakeholder groups. Participants (N=22) were asked to assess on a 5-point Likert scale (1=irrelevant to 5=relevant) to what extent they thought the various factors (N=19) were relevant to trust in and acceptance of novel AI applications in medicine.

RESULTS

The rapid review (N=32 papers) yielded 110 factors related to trust and 77 factors related to acceptance toward AI technology in medicine. Closely related factors were assigned to 1 of the 19 overarching umbrella factors, which were further grouped into 4 categories: human-related (ie, the type of institution AI professionals originate from), technology-related (ie, the explainability and transparency of AI application processes and outcomes), ethical and legal (ie, data use transparency), and additional factors (ie, AI applications being environment friendly). The categorized 19 umbrella factors were presented as survey statements, which were evaluated by relevant stakeholders. Survey participants (N=22) represented researchers (n=18, 82%), technology providers (n=5, 23%), hospital staff (n=3, 14%), and policy makers (n=3, 14%). Of the 19 factors, 16 (84%) human-related, technology-related, ethical and legal, and additional factors were considered to be of high relevance to trust in and acceptance of novel AI applications in medicine. The patient's gender, age, and education level were found to be of low relevance (3/19, 16%).

CONCLUSIONS

The results of this study could help the implementers of medical AI applications to understand what drives trust and acceptance toward AI-powered technologies among key stakeholders in medicine. Consequently, this would allow the implementers to identify strategies that facilitate trust in and acceptance of medical AI applications among key stakeholders and potential users.

Engaging with artificial intelligence in mammography screening: Swedish breast radiologists' views on trust, information and expertise

Objectives

Lack of trust and transparency is stressed as a challenge for clinical implementation of artificial intelligence (AI). In breast cancer screening, AI-supported reading shows promising results but more research is needed on how medical experts, which are facing the integration of AI into their work, reason about trust and information needs. From a sociotechnical information practice perspective, we add to this knowledge by a Swedish case study. This study aims to: (1) clarify Swedish breast radiologists' views on trust, information and expertise pertaining to AI in mammography screening and (2) analytically address ideas about medical professionals' critical engagement with AI and motivations for trust in AI.

Method

An online survey was distributed to Swedish breast radiologists. Survey responses were analysed by descriptive statistical method, correlation analysis and qualitative content analysis. The results were used as foundation for analysing trust and information as parts of critical engagements with AI.

Results

Of the Swedish breast radiologists (n = 105), 47 answered the survey (response rate = 44.8%). 53.2% (n = 25) of the respondents would to a high/somewhat high degree trust AI assessments. To a great extent, additional information would support the respondents' trust evaluations. What type of critical engagement medical professionals are expected to perform on AI as decision support remains unclear.

Conclusions

There is a demand for enhanced information, explainability and transparency of AI-supported mammography. Further discussion and agreement are needed considering what the desired goals for trust in AI should be and how it relates to medical professionals' critical evaluation of AI-made claims in medical decision support.

Trust and stakeholder perspectives on the implementation of AI tools in clinical radiology

OBJECTIVES

To define requirements that condition trust in artificial intelligence (AI) as clinical decision support in radiology from the perspective of various stakeholders and to explore ways to fulfil these requirements.

METHODS

Semi-structured interviews were conducted with twenty-five respondents-nineteen directly involved in the development, implementation, or use of AI applications in radiology and six working with AI in other areas of healthcare. We designed the questions to explore three themes: development and use of AI, professional decision-making, and management and organizational procedures connected to AI. The transcribed interviews were analysed in an iterative coding process from open coding to theoretically informed thematic coding.

RESULTS

We identified four aspects of trust that relate to reliability, transparency, quality verification, and inter-organizational compatibility. These aspects fall under the categories of substantial and procedural requirements.

CONCLUSIONS

Development of appropriate levels of trust in AI in healthcare is complex and encompasses multiple dimensions of requirements. Various stakeholders will have to be involved in developing AI solutions for healthcare and radiology to fulfil these requirements.

CLINICAL RELEVANCE STATEMENT

For AI to achieve advances in radiology, it must be given the opportunity to support, rather than replace, human expertise. Support requires trust. Identification of aspects and conditions for trust allows developing AI implementation strategies that facilitate advancing the field.

KEY POINTS

• Dimensions of procedural and substantial demands that need to be fulfilled to foster appropriate levels of trust in AI in healthcare are conditioned on aspects related to reliability, transparency, quality verification, and inter-organizational compatibility. •Creating the conditions for trust to emerge requires the involvement of various stakeholders, who will have to compensate the problem's inherent complexity by finding and promoting well-defined solutions.

Automation in Contemporary Clinical Information Systems: a Survey of AI in Healthcare Settings

AIMS AND OBJECTIVES

To examine the nature and use of automation in contemporary clinical information systems by reviewing studies reporting the implementation and evaluation of artificial intelligence (AI) technologies in healthcare settings.

METHOD

PubMed/MEDLINE, Web of Science, EMBASE, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies reporting evaluation of AI in clinical settings from January 2021 to December 2022. We documented the clinical application areas and tasks supported, and the level of system autonomy. Reported effects on user experience, decision-making, care delivery and outcomes were summarised.

RESULTS

AI technologies are being applied in a wide variety of clinical areas. Most contemporary systems utilise deep learning, use routinely collected data, support diagnosis and triage, are assistive (requiring users to confirm or approve AI provided information or decisions), and are used by doctors in acute care settings in high-income nations. AI systems are integrated and used within existing clinical information systems including electronic medical records. There is limited support for One Health goals. Evaluation is largely based on quantitative methods measuring effects on decision-making.

CONCLUSION

AI systems are being implemented and evaluated in many clinical areas. There remain many opportunities to understand patterns of routine use and evaluate effects on decision-making, care delivery and patient outcomes using mixed-methods. Support for One Health including integrating data about environmental factors and social determinants needs further exploration.

Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department

This study aimed to develop a machine learning-based clinical decision support system for emergency departments based on the decision-making framework of physicians. We extracted 27 fixed and 93 observation features using data on vital signs, mental status, laboratory results, and electrocardiograms during emergency department stay. Outcomes included intubation, admission to the intensive care unit, inotrope or vasopressor administration, and in-hospital cardiac arrest. eXtreme gradient boosting algorithm was used to learn and predict each outcome. Specificity, sensitivity, precision, F1 score, area under the receiver operating characteristic curve (AUROC), and area under the precision-recall curve were assessed. We analyzed 303,345 patients with 4,787,121 input data, resampled into 24,148,958 1 h-units. The models displayed a discriminative ability to predict outcomes (AUROC > 0.9), and the model with lagging 6 and leading 0 displayed the highest value. The AUROC curve of in-hospital cardiac arrest had the smallest change, with increased lagging for all outcomes. With inotropic use, intubation, and intensive care unit admission, the range of AUROC curve change with the leading 6 was the highest according to different amounts of previous information (lagging). In this study, a human-centered approach to emulate the clinical decision-making process of emergency physicians has been adopted to enhance the use of the system. Machine learning-based clinical decision support systems customized according to clinical situations can help improve the quality of care.

Deployment of machine learning algorithms to predict sepsis: systematic review and application of the SALIENT clinical AI implementation framework

OBJECTIVE

To retrieve and appraise studies of deployed artificial intelligence (AI)-based sepsis prediction algorithms using systematic methods, identify implementation barriers, enablers, and key decisions and then map these to a novel end-to-end clinical AI implementation framework.

MATERIALS AND METHODS

Systematically review studies of clinically applied AI-based sepsis prediction algorithms in regard to methodological quality, deployment and evaluation methods, and outcomes. Identify contextual factors that influence implementation and map these factors to the SALIENT implementation framework.

RESULTS

The review identified 30 articles of algorithms applied in adult hospital settings, with 5 studies reporting significantly decreased mortality post-implementation. Eight groups of algorithms were identified, each sharing a common algorithm. We identified 14 barriers, 26 enablers, and 22 decision points which were able to be mapped to the 5 stages of the SALIENT implementation framework.

DISCUSSION

Empirical studies of deployed sepsis prediction algorithms demonstrate their potential for improving care and reducing mortality but reveal persisting gaps in existing implementation guidance. In the examined publications, key decision points reflecting real-word implementation experience could be mapped to the SALIENT framework and, as these decision points appear to be AI-task agnostic, this framework may also be applicable to non-sepsis algorithms. The mapping clarified where and when barriers, enablers, and key decisions arise within the end-to-end AI implementation process.

CONCLUSIONS

A systematic review of real-world implementation studies of sepsis prediction algorithms was used to validate an end-to-end staged implementation framework that has the ability to account for key factors that warrant attention in ensuring successful deployment, and which extends on previous AI implementation frameworks.

Prediction of inappropriate pre-hospital transfer of patients with suspected cardiovascular emergency diseases using machine learning: a retrospective observational study

BACKGROUND

This study aimed to develop a prediction model for transferring patients to an inappropriate hospital for suspected cardiovascular emergency diseases at the pre-hospital stage, using variables obtained from an integrated nationwide dataset, and to assess the performance of this model.

METHODS

We integrated three nationwide datasets and developed a two-step prediction model utilizing a machine learning algorithm. Ninety-eight clinical characteristics of patients identified at the pre-hospital stage and 13 hospital components were used as input data for the model. The primary endpoint of the model was the prediction of transfer to an inappropriate hospital.

RESULTS

A total of 94,256 transferred patients in the public pre-hospital care system matched the National Emergency Department Information System data of patients with a pre-hospital cardiovascular registry created in South Korea between July 2017 and December 2018. Of these, 1,770 (6.26%) patients failed to be transferred to a capable hospital. The area under the receiver operating characteristic curve of the final predictive model was 0.813 (0.800-0.825), and the area under the receiver precision-recall curve was 0.286 (0.265-0.308).

CONCLUSIONS

Our prediction model used machine learning to show favorable performance in transferring patients with suspected cardiovascular disease to a capable hospital. For our results to lead to changes in the pre-hospital care system, a digital platform for sharing real-time information should be developed.