Applying the RE-AIM Framework for the Evaluation of a Clinical Decision Support Tool for Pediatric Head Trauma: A Mixed-Methods Study

Identifying Facilitators and Barriers to Implementation of AI-Assisted Clinical Decision Support in an Electronic Health Record System

Recent advancements in computing have led to the development of artificial intelligence (AI) enabled healthcare technologies. AI-assisted clinical decision support (CDS) integrated into electronic health records (EHR) was demonstrated to have a significant potential to improve clinical care. With the rapid proliferation of AI-assisted CDS, came the realization that a lack of careful consideration of socio-technical issues surrounding the implementation and maintenance of these tools can result in unanticipated consequences, missed opportunities, and suboptimal uptake of these potentially useful technologies. The 48-h Discharge Prediction Tool (48DPT) is a new AI-assisted EHR CDS to facilitate discharge planning. This study aimed to methodologically assess the implementation of 48DPT and identify the barriers and facilitators of adoption and maintenance using the validated implementation science frameworks. The major dimensions of RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) and the constructs of the Consolidated Framework for Implementation Research (CFIR) frameworks have been used to analyze interviews of 24 key stakeholders using 48DPT. The systematic assessment of the 48DPT implementation allowed us to describe facilitators and barriers to implementation such as lack of awareness, lack of accuracy and trust, limited accessibility, and transparency. Based on our evaluation, the factors that are crucial for the successful implementation of AI-assisted EHR CDS were identified. Future implementation efforts of AI-assisted EHR CDS should engage the key clinical stakeholders in the AI tool development from the very inception of the project, support transparency and explainability of the AI models, provide ongoing education and onboarding of the clinical users, and obtain continuous input from clinical staff on the CDS performance.

Exploring the role of professional identity in the implementation of clinical decision support systems-a narrative review

BACKGROUND

Clinical decision support systems (CDSSs) have the potential to improve quality of care, patient safety, and efficiency because of their ability to perform medical tasks in a more data-driven, evidence-based, and semi-autonomous way. However, CDSSs may also affect the professional identity of health professionals. Some professionals might experience these systems as a threat to their professional identity, as CDSSs could partially substitute clinical competencies, autonomy, or control over the care process. Other professionals may experience an empowerment of the role in the medical system. The purpose of this study is to uncover the role of professional identity in CDSS implementation and to identify core human, technological, and organizational factors that may determine the effect of CDSSs on professional identity.

METHODS

We conducted a systematic literature review and included peer-reviewed empirical studies from two electronic databases (PubMed, Web of Science) that reported on key factors to CDSS implementation and were published between 2010 and 2023. Our explorative, inductive thematic analysis assessed the antecedents of professional identity-related mechanisms from the perspective of different health care professionals (i.e., physicians, residents, nurse practitioners, pharmacists).

RESULTS

One hundred thirty-one qualitative, quantitative, or mixed-method studies from over 60 journals were included in this review. The thematic analysis found three dimensions of professional identity-related mechanisms that influence CDSS implementation success: perceived threat or enhancement of professional control and autonomy, perceived threat or enhancement of professional skills and expertise, and perceived loss or gain of control over patient relationships. At the technological level, the most common issues were the system's ability to fit into existing clinical workflows and organizational structures, and its ability to meet user needs. At the organizational level, time pressure and tension, as well as internal communication and involvement of end users were most frequently reported. At the human level, individual attitudes and emotional responses, as well as familiarity with the system, most often influenced the CDSS implementation. Our results show that professional identity-related mechanisms are driven by these factors and influence CDSS implementation success. The perception of the change of professional identity is influenced by the user's professional status and expertise and is improved over the course of implementation.

CONCLUSION

This review highlights the need for health care managers to evaluate perceived professional identity threats to health care professionals across all implementation phases when introducing a CDSS and to consider their varying manifestations among different health care professionals. Moreover, it highlights the importance of innovation and change management approaches, such as involving health professionals in the design and implementation process to mitigate threat perceptions. We provide future areas of research for the evaluation of the professional identity construct within health care.

Factors influencing clinician and patient interaction with machine learning-based risk prediction models: a systematic review

Machine learning (ML)-based risk prediction models hold the potential to support the health-care setting in several ways; however, use of such models is scarce. We aimed to review health-care professional (HCP) and patient perceptions of ML risk prediction models in published literature, to inform future risk prediction model development. Following database and citation searches, we identified 41 articles suitable for inclusion. Article quality varied with qualitative studies performing strongest. Overall, perceptions of ML risk prediction models were positive. HCPs and patients considered that models have the potential to add benefit in the health-care setting. However, reservations remain; for example, concerns regarding data quality for model development and fears of unintended consequences following ML model use. We identified that public views regarding these models might be more negative than HCPs and that concerns (eg, extra demands on workload) were not always borne out in practice. Conclusions are tempered by the low number of patient and public studies, the absence of participant ethnic diversity, and variation in article quality. We identified gaps in knowledge (particularly views from under-represented groups) and optimum methods for model explanation and alerts, which require future research.

Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review

BACKGROUND

Successful implementation and utilization of Computerized Clinical Decision Support Systems (CDSS) in hospitals is complex and challenging. Implementation science, and in particular the Nonadoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework, may offer a systematic approach for identifying and addressing these challenges. This review aimed to identify, categorize, and describe barriers and facilitators to CDSS implementation in hospital settings and map them to the NASSS framework. Exploring the applicability of the NASSS framework to CDSS implementation was a secondary aim.

METHODS

Electronic database searches were conducted (21 July 2020; updated 5 April 2022) in Ovid MEDLINE, Embase, Scopus, PyscInfo, and CINAHL. Original research studies reporting on measured or perceived barriers and/or facilitators to implementation and adoption of CDSS in hospital settings, or attitudes of healthcare professionals towards CDSS were included. Articles with a primary focus on CDSS development were excluded. No language or date restrictions were applied. We used qualitative content analysis to identify determinants and organize them into higher-order themes, which were then reflexively mapped to the NASSS framework.

RESULTS

Forty-four publications were included. These comprised a range of study designs, geographic locations, participants, technology types, CDSS functions, and clinical contexts of implementation. A total of 227 individual barriers and 130 individual facilitators were identified across the included studies. The most commonly reported influences on implementation were fit of CDSS with workflows (19 studies), the usefulness of the CDSS output in practice (17 studies), CDSS technical dependencies and design (16 studies), trust of users in the CDSS input data and evidence base (15 studies), and the contextual fit of the CDSS with the user's role or clinical setting (14 studies). Most determinants could be appropriately categorized into domains of the NASSS framework with barriers and facilitators in the "Technology," "Organization," and "Adopters" domains most frequently reported. No determinants were assigned to the "Embedding and Adaptation Over Time" domain.

CONCLUSIONS

This review identified the most common determinants which could be targeted for modification to either remove barriers or facilitate the adoption and use of CDSS within hospitals. Greater adoption of implementation theory should be encouraged to support CDSS implementation.

Clinical decision support to Optimize Care of patients with Atrial Fibrillation or flutter in the Emergency department: protocol of a stepped-wedge cluster randomized pragmatic trial (O'CAFÉ trial)

BACKGROUND

Management of adults with atrial fibrillation (AF) or atrial flutter in the emergency department (ED) includes rate reduction, cardioversion, and stroke prevention. Different approaches to these components of care may lead to variation in frequency of hospitalization and stroke prevention actions, with significant implications for patient experience, cost of care, and risk of complications. Standardization using evidence-based recommendations could reduce variation in management, preventable hospitalizations, and stroke risk.

METHODS

We describe the rationale for our ED-based AF treatment recommendations. We also describe the development of an electronic clinical decision support system (CDSS) to deliver these recommendations to emergency physicians at the point of care. We implemented the CDSS at three pilot sites to assess feasibility and solicit user feedback. We will evaluate the impact of the CDSS on hospitalization and stroke prevention actions using a stepped-wedge cluster randomized pragmatic clinical trial across 13 community EDs in Northern California.

DISCUSSION

We hypothesize that the CDSS intervention will reduce hospitalization of adults with isolated AF or atrial flutter presenting to the ED and increase anticoagulation prescription in eligible patients at the time of ED discharge and within 30 days. If our hypotheses are confirmed, the treatment protocol and CDSS could be recommended to other EDs to improve management of adults with AF or atrial flutter.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05009225 .  Registered on 17 August 2021.

The role of organizational culture in health information technology implementations: A scoping review

Introduction

The exponential growth in health information technology (HIT) presents an immense opportunity for facilitating the data-to-knowledge-to-performance loop which supports learning health systems. This scoping review addresses the gap in knowledge around HIT implementation contextual factors such as organizational culture and provides a current state assessment.

Methods

A search of 13 databases guided by Arskey and O'Malley's framework identified content on HIT implementations and organizational culture. The Consolidated Framework for Implementation Research (CFIR) was used to assess culture and to develop review criteria. Culture stress, culture effort, implementation climate, learning climate, readiness for implementation, leadership engagement, and available resources were the constructs examined. Rayyan and Qualtrics were used for screening and data extraction.

Results

Fifty two studies included were mainly conducted in Academic Health Centers (n = 18, 35%) and at urban locations (n = 50, 96%). Interviews frequently used for data collection (n = 26, 50%) and guided by multiple frameworks (n = 34). Studies mostly focused on EHR implementations (n = 23, 44%) followed by clinical decision support (n = 9, 17%). About two-thirds (n = 34, 65%) reflected culture stress theme and 62% (21 of 34) acknowledged it as a barrier. Culture effort identified in 27 studies and was a facilitator in most (78%, 21 of 27). Leadership engagement theme in majority studies (71%, n = 37), with 35% (n = 13) noting it as a facilitator. Eighty percent (42 studies) noted available resources, 12 of which identified this as barrier to successful implementation.

Conclusions

It is vital to determine the culture and other CFIR inner setting constructs that are significant to HIT implementation as facilitators or barriers. This scoping review presents a limited number of empirical studies in this topic highlighting the need for additional research to quantify the effects of culture. This will help build evidence and best practices that facilitate HIT implementations and hence serve as a platform to support robust learning health systems.

Usability and Acceptability of Clinical Decision Support Based on the KIIDS-TBI Tool for Children with Mild Traumatic Brain Injuries and Intracranial Injuries

BACKGROUND

 The Kids Intracranial Injury Decision Support tool for Traumatic Brain Injury (KIIDS-TBI) tool is a validated risk prediction model for managing children with mild traumatic brain injuries (mTBI) and intracranial injuries. Electronic clinical decision support (CDS) may facilitate the clinical implementation of this evidence-based guidance.

OBJECTIVE

 Our objective was to evaluate the acceptability and usability of an electronic CDS tool for managing children with mTBI and intracranial injuries.

METHODS

 Emergency medicine and neurosurgery physicians (10 each) from 10 hospitals in the United States were recruited to participate in usability testing of a novel CDS prototype in a simulated electronic health record environment. Testing included a think-aloud protocol, an acceptability and usability survey, and a semi-structured interview. The prototype was updated twice during testing to reflect user feedback. Usability problems recorded in the videos were categorized using content analysis. Interview transcripts were analyzed using thematic analysis.

RESULTS

 Among the 20 participants, most worked at teaching hospitals (80%), freestanding children's hospitals (95%), and level-1 trauma centers (75%). During the two prototype updates, problems with clarity of terminology and navigating through the CDS interface were identified and corrected. Corresponding to these changes, the number of usability problems decreased from 35 in phase 1 to 8 in phase 3 and the number of mistakes made decreased from 18 (phase 1) to 2 (phase 3). Through the survey, participants found the tool easy to use (90%), useful for determining a patient's level of care (95%), and likely to improve resource use (90%) and patient safety (79%). Interview themes related to the CDS's ability to support evidence-based decision-making and improve clinical workflow proposed implementation strategies and potential pitfalls.

CONCLUSION

 After iterative evaluation and refinement, the KIIDS-TBI CDS tool was found to be highly usable and useful for aiding the management of children with mTBI and intracranial injuries.

Considerations for Designing EHR-Embedded Clinical Decision Support Systems for Antimicrobial Stewardship in Pediatric Emergency Departments

OBJECTIVE

This study was aimed to explore the intersection between organizational environment, workflow, and technology in pediatric emergency departments (EDs) and how these factors impact antibiotic prescribing decisions.

METHODS

Semistructured interviews with 17 providers (1 fellow and 16 attending faculty), and observations of 21 providers (1 physician assistant, 5 residents, 3 fellows, and 12 attendings) were conducted at three EDs in the United States. We analyzed interview transcripts and observation notes using thematic analysis.

RESULTS

Seven themes relating to antibiotic prescribing decisions emerged as follows: (1) professional judgement, (2) cognition as a critical individual resource, (3) decision support as a critical organizational resource, (4) patient management with imperfect information, (5) information-seeking as a primary task, (6) time management, and (7) broad process boundaries of antibiotic prescribing.

DISCUSSION

The emerging interrelated themes identified in this study can be used as a blueprint to design, implement, and evaluate clinical decision support (CDS) systems that support antibiotic prescribing in EDs. The process boundaries of antibiotic prescribing are broader than the current boundaries covered by existing CDS systems. Incongruities between process boundaries and CDS can under-support clinicians and lead to suboptimal decisions. We identified two incongruities: (1) the lack of acknowledgment that the process boundaries go beyond the physical boundaries of the ED and (2) the lack of integration of information sources (e.g., accessibility to prior cultures on an individual patient outside of the organization).

CONCLUSION

Significant opportunities exist to improve appropriateness of antibiotic prescribing by considering process boundaries in the design, implementation, and evaluation of CDS systems.

The Development and Implementation of a Cardiac Home Hospice Program: Results of a RE-AIM Analysis

BACKGROUND

Use of hospice has grown among patients with heart failure; however, gaps remain in the ability of agencies to tailor services to meet their needs.

AIM

This study describes the implementation of a cardiac home hospice program and insights for dissemination to other hospice programs.

DESIGN

We conducted a multimethod analysis structured around the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework.

SETTINGS/PARTICIPANTS

We used electronic medical records for our quantitative data source and interviews with hospice clinicians from a not-for-profit hospice agency (N = 32) for our qualitative data source.

RESULTS

Reach-A total of 1273 participants were enrolled in the cardiac home hospice program, of which 57% were female and 42% were black or Hispanic with a mean age was 86 years. Effectiveness-The cardiac home hospice program increased hospice enrollment among patients with heart failure from 7.9% to 9.5% over 1 year (2016-2017). Adoption-Institutional factors that supported the program included the acute need to support medically complex patients at the end of life and an engaged clinical champion. Implementation-Program implementation was supported by interdisciplinary teams who engaged in care coordination. Maintenance-The program has been maintained for over 3 years.

CONCLUSION

The cardiac home hospice program strengthened hospice clinicians' ability to confidently provide care for patients with heart failure, expanded awareness of their symptoms among clinicians, and was associated with increased enrollment of patients with heart failure over the study period. This RE-AIM evaluation provides lessons learned and strategies for future adoption, implementation, and maintenance of a cardiac home hospice program.

To Share is Human! Advancing Evidence into Practice through a National Repository of Interoperable Clinical Decision Support

BACKGROUND

Healthcare systems devote substantial resources to the development of clinical decision support (CDS) largely independently. The process of translating evidence-based practice into useful and effective CDS may be more efficient and less duplicative if healthcare systems shared knowledge about the translation, including workflow considerations, key assumptions made during the translation process, and technical details.

OBJECTIVE

Describe how a national repository of CDS can serve as a public resource for healthcare systems, academic researchers, and informaticists seeking to share and reuse CDS knowledge resources or "artifacts."

METHODS

In 2016, the Agency for Healthcare Research and Quality (AHRQ) launched CDS Connect as a public, web-based platform for authoring and sharing CDS knowledge artifacts. Researchers evaluated early use and impact of the platform by collecting user experiences of AHRQ-sponsored and community-led dissemination efforts and through quantitative/qualitative analysis of site metrics. Efforts are ongoing to quantify efficiencies gained by healthcare systems that leverage shared, interoperable CDS artifacts rather than developing similar CDS de novo and in isolation.

RESULTS

Federal agencies, academic institutions, and others have contributed over 50 entries to CDS Connect for sharing and dissemination. Analysis indicates shareable CDS resources reduce team sizes and the number of tasks and time required to design, develop, and deploy CDS. However, the platform needs further optimization to address sociotechnical challenges. Benefits of sharing include inspiring others to undertake similar CDS projects, identifying external collaborators, and improving CDS artifacts as a result of feedback. Organizations are adapting content available through the platform for continued research, innovation, and local implementations.

CONCLUSION

CDS Connect has provided a functional platform where CDS developers are actively sharing their work. CDS sharing may lead to improved implementation efficiency through numerous pathways, and further research is ongoing to quantify efficiencies gained.

Diffusing an Innovation: Clinician Perceptions of Continuous Predictive Analytics Monitoring in Intensive Care

BACKGROUND

The purpose of this article is to describe neonatal intensive care unit clinician perceptions of a continuous predictive analytics technology and how those perceptions influenced clinician adoption. Adopting and integrating new technology into care is notoriously slow and difficult; realizing expected gains remain a challenge.

METHODS

Semistructured interviews from a cross-section of neonatal physicians (n = 14) and nurses (n = 8) from a single U.S. medical center were collected 18 months following the conclusion of the predictive monitoring technology randomized control trial. Following qualitative descriptive analysis, innovation attributes from Diffusion of Innovation Theory-guided thematic development.

RESULTS

Results suggest that the combination of physical location as well as lack of integration into work flow or methods of using data in care decisionmaking may have delayed clinicians from routinely paying attention to the data. Once data were routinely collected, documented, and reported during patient rounds and patient handoffs, clinicians came to view data as another vital sign. Through clinicians' observation of senior physicians and nurses, and ongoing dialogue about data trends and patient status, clinicians learned how to integrate these data in care decision making (e.g., differential diagnosis) and came to value the technology as beneficial to care delivery.

DISCUSSION

The use of newly created predictive technologies that provide early warning of illness may require implementation strategies that acknowledge the risk-benefit of treatment clinicians must balance and take advantage of existing clinician training methods.