Visual Analytics for Pattern Discovery in Home Care. Clinical Relevance for Quality Improvement

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

BACKGROUND

 Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring.

OBJECTIVES

 In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes.

METHODS

 We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system.

RESULTS

 Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention.

CONCLUSION

 We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.

Using data visualization to detect patterns in whole‐person health data

Data visualization techniques are useful for examining large multidimensional data sets. In this exploratory data analysis (EDA) study, we applied a visualization pattern detection and testing process to deidentified data to discover patterns in whole-person health for adults 65 and older. Whole-person health examines a person's environmental, psychosocial, and physical health, as well as their health-related behaviors; and assesses their strengths, challenges, and needs. Strengths are defined as assets and capabilities in the face of short-or long-term stressors. We collected data using a mobile application that delivers a comprehensive whole-person assessment using a simplified version of a standardized instrument, the Omaha System. The visualization pattern detection process is iterative, includes various techniques, and requires visualization literacy. The data visualization techniques applied in this analysis included bubble charts, parallel coordinates line graphs, box plots, and alluvial flow diagrams. We discovered six patterns within the visualizations. We formulated and tested six hypotheses based on these six patterns, and all six hypotheses were supported. Adults 65 and older had more strengths than challenges and more challenges than needs (p < 0.001). Strengths and challenges were negatively correlated (p < 0.001). Unexpectedly, a subset of adults 65 and older who had many, but not all, strengths had significantly more needs (p = 0.04). The use of standardized terminology with its inherent data interrelationships was key to discovering patterns in whole-person health. This methodology may be used in future EDA research using new data sets.

Intergenerational Decision Making: The Role of Family Relationships in Medical Decision Making

A symposium held at the 42nd annual Society for Medical Decision Making conference on October 26, 2020, focused on intergenerational decision making. The symposium covered existing research and clinical experiences using formal presentations and moderated discussion and was attended by 43 people. Presentations focused on the roles of pediatric patients in decision making, caregiver decision making for a child with complex medical needs, caregiver involvement in advanced care planning, and the inclusion of spillover effects in economic evaluations. The moderated discussion, summarized in this article, highlighted existing resources and gaps in intergenerational decision making in four areas: decision aids, economic evaluation, participant perspectives, and measures. Intergenerational decision making is an understudied and poorly understood aspect of medical decision making that requires particular attention as our society ages and technological advances provide new innovations for life-sustaining measures across all stages of the lifespan.

Usability Testing of an Interactive Dashboard for Surgical Quality Improvement in a Large Congenital Heart Center

BACKGROUND

Interactive data visualization and dashboards can be an effective way to explore meaningful patterns in large clinical data sets and to inform quality improvement initiatives. However, these interactive dashboards may have usability issues that undermine their effectiveness. These usability issues can be attributed to mismatched mental models between the designers and the users. Unfortunately, very few evaluation studies in visual analytics have specifically examined such mismatches between these two groups.

OBJECTIVES

We aimed to evaluate the usability of an interactive surgical dashboard and to seek opportunities for improvement. We also aimed to provide empirical evidence to demonstrate the mismatched mental models between the designers and the users of the dashboard.

METHODS

An interactive dashboard was developed in a large congenital heart center. This dashboard provides real-time, interactive access to clinical outcomes data for the surgical program. A mixed-method, two-phase study was conducted to collect user feedback. A group of designers (N = 3) and a purposeful sample of users (N = 12) were recruited. The qualitative data were analyzed thematically. The dashboards were compared using the System Usability Scale (SUS) and qualitative data.

RESULTS

The participating users gave an average SUS score of 82.9 on the new dashboard and 63.5 on the existing dashboard (p = 0.006). The participants achieved high task accuracy when using the new dashboard. The qualitative analysis revealed three opportunities for improvement. The data analysis and triangulation provided empirical evidence to the mismatched mental models.

CONCLUSION

We conducted a mixed-method usability study on an interactive surgical dashboard and identified areas of improvements. Our study design can be an effective and efficient way to evaluate visual analytics systems in health care. We encourage researchers and practitioners to conduct user-centered evaluation and implement education plans to mitigate potential usability challenges and increase user satisfaction and adoption.

Decision-Centered Design of Patient Information Visualizations to Support Chronic Pain Care

BACKGROUND

For complex patients with chronic conditions, electronic health records (EHRs) contain large amounts of relevant historical patient data. To use this information effectively, clinicians may benefit from visual information displays that organize and help them make sense of information on past and current treatments, outcomes, and new treatment options. Unfortunately, few clinical decision support tools are designed to support clinical sensemaking.

OBJECTIVE

The objective of this study was to describe a decision-centered design process, and resultant interactive patient information displays, to support key clinical decision requirements in chronic noncancer pain care.

METHODS

To identify key clinical decision requirements, we conducted critical decision method interviews with 10 adult primary care clinicians. Next, to identify key information needs and decision support design seeds, we conducted a half-day multidisciplinary design workshop. Finally, we designed an interactive prototype to support the key clinical decision requirements and information needs uncovered during the previous research activities.

RESULTS

The resulting Chronic Pain Treatment Tracker prototype summarizes the current treatment plan, past treatment history, potential future treatments, and treatment options to be cautious about. Clinicians can access additional details about each treatment, current or past, through modal views. Additional decision support for potential future treatments and treatments to be cautious about is also provided through modal views.

CONCLUSION

This study designed the Chronic Pain Treatment Tracker, a novel approach to decision support that presents clinicians with the information they need in a structure that promotes quick uptake, understanding, and action.

Ontology-Based Interactive Visualization of Patient-Generated Research Questions

BACKGROUND

Crohn's disease and colitis are chronic conditions that affect every facet of patients' lives (e.g., social interaction, family, work, diet, and sleep). Thus, treatment consists largely of disease management. The University of North Carolina at Chapel Hill chapter of the Crohn's and Colitis Foundation-IBD Partners-has created an interactive website that, in addition to providing helpful information and disease management tools, provides a discussion forum for patients to talk about their experiences and suggest new lines of research into Crohn's disease and colitis.

OBJECTIVES

The primary objective of this work is to enable researchers to more effectively browse the forum content. Researchers wish to identify important/popular patient-suggested research topics, appreciate the full breadth of the research topics, and see connections between them, in order to more effectively prioritize research agendas.

METHODS

To help structure the forum content we have developed an ontology describing the major themes in the discussion forum. We have also created a prototype interactive visualization tool that leverages the ontology to help researchers identify common themes and related patient-generated research topics via linked views of (1) the ontology, (2) a research topic overview clustered by relevant ontology terms, and (3) a detailed view of the discussion forum content.

RESULTS

We discuss visualizations and interactions enabled by the visualization tool, provide an example scenario using the tool, and discuss limitations and future work based on feedback from potential users.

CONCLUSION

The integration of a user-community specific ontology with an interactive visualization tool is a promising approach for enabling researchers to more effectively study user-generated research questions.

Timely Data for Targeted Quality Improvement Interventions: Use of a Visual Analytics Dashboard for Bronchiolitis

BACKGROUND

Standard methods for obtaining data may delay quality improvement (QI) interventions including for bronchiolitis, a common cause of childhood hospitalization.

OBJECTIVE

To describe the use of a dashboard in the context of a multifaceted QI intervention aimed at reducing the use of chest radiographs, bronchodilators, antibiotics, steroids, and viral testing in patients with bronchiolitis.

METHODS

This QI initiative took place at Children's Minnesota, a large, not-for-profit children's health care organization. A multidisciplinary bronchiolitis workgroup developed a local clinical guideline and order-set. Delays in obtaining baseline data prompted a pediatric hospitalist and information technology specialist to modify a vendor's dashboard to display data related to bronchiolitis guideline metrics. Patients 2 months to 2 years old with a bronchiolitis emergency department (ED)/inpatient encounter in the period October 1, 2014 to April 30, 2018 were included. The primary outcome was a functioning dashboard; a process measure was the percentage of ED clinician logins. Outcome measures included the percent use of guideline metrics (e.g., bronchodilators) displayed on statistical process control charts (ED vs. inpatient). Balancing measures included length of stay, charge ratios, and hospital revisits.

RESULTS

A workgroup (formed October 2015) implemented a bronchiolitis order-set and guideline (February 2016) followed by a bronchiolitis dashboard (August 2016) consolidating disparate data sources loaded within 2 to 4 days of discharge. In total, 35% of ED clinicians logged in. Leaders used the dashboard to target and track interventions such as a bronchodilator order alert. There were improvements in most outcome metrics; however, timing did not suggest direct dashboard impact. ED balancing measures were lower after implementation.

CONCLUSION

We described use of a dashboard to support a multifaceted QI initiative for bronchiolitis. Leaders used the dashboard for targeted interventions but the dashboard did not directly impact the observed improvements. Future studies should assess reasons for low individual dashboard use.