Investigating the Heart Pump Implant Decision Process: Opportunities for Decision Support Tools to Help

Information Seeking and Sensemaking in Emergency Medical Service through Simulation Video Review

Emergency medical services (EMS) providers often face significant challenges in their work, including collecting, integrating, and making sense of a variety of information. Despite their criticality, EMS work is one of the very few medical domains with limited technical support. To design and implement effective decision support, it is essential to examine and gain a holistic understanding of the fine-grained process of sensemaking in the field. To that end, we reviewed 25 video recordings of EMS simulations to understand the nuances of EMS sensemaking work, including 1) the types of information and situation that are collected and made sense of in the field; 2) the work practices and temporal patterns of EMS sensemaking work; and 3) the challenges in EMS sensemaking and decision-making process. Based on the results, we discuss implications for technology opportunities to support rapid information acquisition and sensemaking in time-critical, high-risk medical settings such as EMS.

Transitioning Cognitive Aids into Decision Support Platforms: Requirements and Design Guidelines

Digital cognitive aids have the potential to serve as clinical decision support platforms, triggering alerts about process delays and recommending interventions. In this mixed-methods study, we examined how a digital checklist for pediatric trauma resuscitation could trigger decision support alerts and recommendations. We identified two criteria that cognitive aids must satisfy to support these alerts: (1) context information must be entered in a timely, accurate, and standardized manner, and (2) task status must be accurately documented. Using co-design sessions and near-live simulations, we created two checklist features to satisfy these criteria: a form for entering the pre-hospital information and a progress slider for documenting the progression of a multi-step task. We evaluated these two features in the wild, contributing guidelines for designing these features on cognitive aids to support alerts and recommendations in time- and safety-critical scenarios.

Designing Flexible Longitudinal Regimens: Supporting Clinician Planning for Discontinuation of Psychiatric Drugs

Clinical decision support tools have typically focused on one-time support for diagnosis or prognosis, but have the ability to support providers in longitudinal planning of patient care regimens amidst infrastructural challenges. We explore an opportunity for technology support for discontinuing antidepressants, where clinical guidelines increasingly recommend gradual discontinuation over abruptly stopping to avoid withdrawal symptoms, but providers have varying levels of experience and diverse strategies for supporting patients through discontinuation. We conducted two studies with 12 providers, identifying providers’ needs in developing discontinuation plans and deriving design guidelines. We then iteratively designed and implemented AT Planner, instantiating the guidelines by projecting taper schedules and providing flexibility for adjustment. Provider feedback on AT Planner highlighted that discontinuation plans required balancing interpersonal and infrastructural constraints and surfaced the need for different technological support based on clinical experience. We discuss the benefits and challenges of incorporating flexibility and advice into clinical planning tools.

Inclusion of Clinicians in the Development and Evaluation of Clinical Artificial Intelligence Tools: A Systematic Literature Review

The application of machine learning (ML) and artificial intelligence (AI) in healthcare domains has received much attention in recent years, yet significant questions remain about how these new tools integrate into frontline user workflow, and how their design will impact implementation. Lack of acceptance among clinicians is a major barrier to the translation of healthcare innovations into clinical practice. In this systematic review, we examine when and how clinicians are consulted about their needs and desires for clinical AI tools. Forty-five articles met criteria for inclusion, of which 24 were considered design studies. The design studies used a variety of methods to solicit and gather user feedback, with interviews, surveys, and user evaluations. Our findings show that tool designers consult clinicians at various but inconsistent points during the design process, and most typically at later stages in the design cycle (82%, 19/24 design studies). We also observed a smaller amount of studies adopting a human-centered approach and where clinician input was solicited throughout the design process (22%, 5/24). A third (15/45) of all studies reported on clinician trust in clinical AI algorithms and tools. The surveyed articles did not universally report validation against the “gold standard” of clinical expertise or provide detailed descriptions of the algorithms or computational methods used in their work. To realize the full potential of AI tools within healthcare settings, our review suggests there are opportunities to more thoroughly integrate frontline users’ needs and feedback in the design process.

A Clinical Decision Support System for Sleep Staging Tasks With Explanations From Artificial Intelligence: User-Centered Design and Evaluation Study

Background

Despite the unprecedented performance of deep learning algorithms in clinical domains, full reviews of algorithmic predictions by human experts remain mandatory. Under these circumstances, artificial intelligence (AI) models are primarily designed as clinical decision support systems (CDSSs). However, from the perspective of clinical practitioners, the lack of clinical interpretability and user-centered interfaces hinders the adoption of these AI systems in practice.

Objective

This study aims to develop an AI-based CDSS for assisting polysomnographic technicians in reviewing AI-predicted sleep staging results. This study proposed and evaluated a CDSS that provides clinically sound explanations for AI predictions in a user-centered manner.

Methods

Our study is based on a user-centered design framework for developing explanations in a CDSS that identifies why explanations are needed, what information should be contained in explanations, and how explanations can be provided in the CDSS. We conducted user interviews, user observation sessions, and an iterative design process to identify three key aspects for designing explanations in the CDSS. After constructing the CDSS, the tool was evaluated to investigate how the CDSS explanations helped technicians. We measured the accuracy of sleep staging and interrater reliability with macro-F1 and Cohen κ scores to assess quantitative improvements after our tool was adopted. We assessed qualitative improvements through participant interviews that established how participants perceived and used the tool.

Results

The user study revealed that technicians desire explanations that are relevant to key electroencephalogram (EEG) patterns for sleep staging when assessing the correctness of AI predictions. Here, technicians wanted explanations that could be used to evaluate whether the AI models properly locate and use these patterns during prediction. On the basis of this, information that is closely related to sleep EEG patterns was formulated for the AI models. In the iterative design phase, we developed a different visualization strategy for each pattern based on how technicians interpreted the EEG recordings with these patterns during their workflows. Our evaluation study on 9 polysomnographic technicians quantitatively and qualitatively investigated the helpfulness of the tool. For technicians with <5 years of work experience, their quantitative sleep staging performance improved significantly from 56.75 to 60.59 with a P value of .05. Qualitatively, participants reported that the information provided effectively supported them, and they could develop notable adoption strategies for the tool.

Conclusions

Our findings indicate that formulating clinical explanations for automated predictions using the information in the AI with a user-centered design process is an effective strategy for developing a CDSS for sleep staging.

Why and How to Approach User Experience in Safety-Critical Domains: The Example of Health Care

OBJECTIVE

To highlight the importance of the personal experience of users who interact with technology in safety-critical domains and summarize three interaction concepts and the associated theories that provide the means for addressing user experience.

BACKGROUND

In health care, the dominant concepts of interaction are based on theories arising from classic cognitive psychology. These concepts focus mainly on safety and efficiency, with too little consideration being given to user experience.

METHOD

Users in complex socio-technical and safety-critical domains such as health care interact with many technological devices. Enhancing the user experience could improve the design of technology, enhance the well-being of staff, and contribute to modern safety management. We summarize concepts of "interaction" based on modern theories of human-computer interaction, which include the personal experience of users as an important construct.

RESULTS AND CONCLUSION

Activity theory, embodiment, and interaction as experience provide a theoretical foundation for considering user experience in safety-critical domains. Using an example from anesthesiology, we demonstrate how each theory provides a unique but complementary view on experience. Finally, the methodological possibilities for considering personal experience in design and evaluations vary among the theories.

APPLICATION

Considering user experience in health care and potentially other safety-critical domains can provide an additional means of optimizing interaction with technology, contributing to the well-being of staff, and improving safety.

Designing Interactive Alerts to Improve Recognition of Critical Events in Medical Emergencies

Vital sign values during medical emergencies can help clinicians recognize and treat patients with life-threatening injuries. Identifying abnormal vital signs, however, is frequently delayed and the values may not be documented at all. In this mixed-methods study, we designed and evaluated a two-phased visual alert approach for a digital checklist in trauma resuscitation that informs users about undocumented vital signs. Using an interrupted time series analysis, we compared documentation in the periods before (two years) and after (four months) the introduction of the alerts. We found that introducing alerts led to an increase in documentation throughout the post-intervention period, with clinicians documenting vital signs earlier. Interviews with users and video review of cases showed that alerts were ineffective when clinicians engaged less with the checklist or set the checklist down to perform another activity. From these findings, we discuss approaches to designing alerts for dynamic team-based settings.

Exploration and Initial Development of Text Classification Models to Identify Health Information Technology Usability-Related Patient Safety Event Reports

BACKGROUND

With the pervasive use of health information technology (HIT) there has been increased concern over the usability and safety of this technology. Identifying HIT usability and safety hazards, mitigating those hazards to prevent patient harm, and using this knowledge to improve future HIT systems are critical to advancing health care.

PURPOSE

The purpose of this work is to demonstrate the feasibility of a modeling approach to identify HIT usability-related patient safety events (PSEs) from the free-text of safety reports and the utility of such models for supporting patient safety analysts in their analysis of event data.

METHODS

We evaluated three feature representations (bag-of-words [BOWs], topic modeling, and document embeddings) to classify HIT usability-related PSE reports using 5,911 manually annotated reports. Model results were reviewed with patient safety analysts to gather feedback on their usefulness and integration into workflow.

RESULTS

The combination of term frequency-inverse document frequency BOWs and document embedding features modeled with support vector machine (SVM) with radial basis function (RBF) had the highest overall precision-recall area under the curve (AUC) and f1-score, 72 and 66%, respectively. Using only document embedding features achieved a similar precision-recall AUC and f1-score performance with the SVM RBF model, 70 and 66%, respectively. Models generally favored specificity and sensitivity over precision. Patient safety analysts found the model results to be useful and offered three suggestions on how it can be integrated into their workflow at the point of report entry, in a visual dashboard layer, and to support data retrievals.

CONCLUSION

Text mining and document embeddings can support identification of HIT usability-related PSE reports. The positive feedback received on the HIT usability model shows its potential utility in real-world applications.

Family caregiver-reported outcomes regarding decision-making for left ventricular assist device implantation

BACKGROUND

Family caregivers (FCGs) often participate in the decision for their loved one to receive a left ventricular assist device (LVAD). Little is known about the contribution of FCGs to this complex decision.

OBJECTIVES

To investigate family caregiver-reported outcomes related to decision-making for LVAD implantation and their experiences post-implantation.

METHODS

Descriptive thematic analysis was used to analyze longitudinal data. Thematic saturation was achieved.

RESULTS

Three key themes emerged from the data. The main theme in the pre-implantation period was: Not a decision. The two themes in the post-implantation period were: More satisfaction than regret and Unanticipated situational change.

CONCLUSIONS

Family caregiver-reported outcomes inform clinical practice and future research. FCGs of LVAD recipients did not see viable alternatives to LVAD implantation, were generally satisfied with post-implantation outcomes, and experienced unexpected life changes in the post-implantation period despite feeling prepared preoperatively. Education of both LVAD recipients and their FCGs must be optimized.

"It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation

We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.

Simulation as a preoperative planning approach in advanced heart failure patients. A retrospective clinical analysis

BACKGROUND

Modelling and simulation may become clinically applicable tools for detailed evaluation of the cardiovascular system and clinical decision-making to guide therapeutic intervention. Models based on pressure-volume relationship and zero-dimensional representation of the cardiovascular system may be a suitable choice given their simplicity and versatility. This approach has great potential for application in heart failure where the impact of left ventricular assist devices has played a significant role as a bridge to transplant and more recently as a long-term solution for non eligible candidates.

RESULTS

We sought to investigate the value of simulation in the context of three heart failure patients with a view to predict or guide further management. CARDIOSIM^© was the software used for this purpose. The study was based on retrospective analysis of haemodynamic data previously discussed at a multidisciplinary meeting. The outcome of the simulations addressed the value of a more quantitative approach in the clinical decision process.

CONCLUSIONS

Although previous experience, co-morbidities and the risk of potentially fatal complications play a role in clinical decision-making, patient-specific modelling may become a daily approach for selection and optimisation of device-based treatment for heart failure patients. Willingness to adopt this integrated approach may be the key to further progress.

A Visual Decision Support Tool for Appendectomy Care

Appendectomy is the most common abdominal surgical procedure performed in children in the United States. In order to assist care providers in creating treatment plans for the postoperative management of pediatric appendicitis, we have developed a predictive statistical model of outcomes on which we have built a prototype decision aid application. The model, trained on 3724 anonymized care records and evaluated on a separate set of 2205 cases from a tertiary care center, achieves 97.0% specificity, 25.1% true sensitivity, and 58.8% precision. We have also built an interactive decision support tool augmented with simple visualization techniques designed for clinicians to use in the course of making care decisions (e.g., discharge) and in patient/stakeholder communication. Its focus is on end-user ease of use and integration into existing clinician workflows, and is designed to evolve its predictions as more and better data become available.

Designing Contestability: Interaction Design, Machine Learning, and Mental Health

We describe the design of an automated assessment and training tool for psychotherapists to illustrate challenges with creating interactive machine learning (ML) systems, particularly in contexts where human life, livelihood, and wellbeing are at stake. We explore how existing theories of interaction design and machine learning apply to the psychotherapy context, and identify "contestability" as a new principle for designing systems that evaluate human behavior. Finally, we offer several strategies for making ML systems more accountable to human actors.