Development of Performance Dashboards in Healthcare Sector: Key Practical Issues

Putting the pieces together: towards an integrative framework for healthcare performance

PURPOSE

In the past few decades, performance measuring systems have become important managerial tools for healthcare organizations. Healthcare performance metrics are a useful tool in understanding how healthcare organizations achieve their goals while satisfying the needs of their patients and conforming to national and international standards. Various efforts have been made to assess healthcare performance. Most of these measures are focused on a single perspective or developed by a single source to meet management and strategic objectives on time.

DESIGN/METHODOLOGY/APPROACH

We develop a review of the literature to shed light on the measures used to assess performance in the healthcare sector at various points in time, as well as to establish a thorough understanding of healthcare performance measurement.

FINDINGS

Developing real-time digital traceability of metrics and an integrative perspective that increases the actionability of information acquired is an attractive potential made possible by the introduction of new technologies and the digitization of data.

ORIGINALITY/VALUE

We conclude that a proper measurement system should be one to combine patient, physician, non-medical staff and system perspective, which will further facilitate the assessment of healthcare performance and the comparative function.

Dashboards to Support Implementation of the Quebec Alzheimer Plan: Evaluation Study With Regional and Professional Considerations

BACKGROUND

Health organizations face the critical task of executing and overseeing comprehensive health care. To address the challenges associated with this task, evidence-based dashboards have emerged as valuable tools. Since 2016, the regional health organizations of Quebec, Canada, have been responsible for ensuring implementation of the Quebec Alzheimer Plan (QAP), a provincial plan that aims to reinforce the capacity of primary care services to detect, diagnose, and treat persons with dementia. Despite the provincial scope of the QAP, the diverse material and human resources across regions introduce variability in the interest, utility, and specific needs associated with these dashboards.

OBJECTIVE

The aim of this study was to assess the interest and utility of dashboards to support the QAP implementation, as well as to determine the needs for improving these aspects according to the perspectives of various types of professionals involved across regions.

METHODS

An evaluative study using qualitative methods was conducted within a collaborative research approach involving different stakeholders, including the ministerial advisor and the four project managers responsible for supporting the implementation of the QAP, as well as researchers/scientific advisors. To support these organizations, we developed tailored, 2-page paper dashboards, detailing quantitative data on the prevalence of dementia, the use of health services by persons with dementia, and achievements and challenges of the QAP implementation in each organization's jurisdiction. We then conducted 23 focus groups with the managers and leading clinicians involved in the implementation of the QAP of each regional health organization. Real-time notes were taken using a structured observation grid. Content analysis was conducted according to different regions (organizations with university mandates or nearby organizations, labeled "university/peripheral"; organizations for which only part of the territory is in rural areas, labeled "mixed"; and organizations in remote or isolated areas, labeled "remote/isolated") and according to different types of participants (managers, leading clinicians, and other participants).

RESULTS

Participants from organizations in all regions expressed interest in these dashboards and found them useful in several ways. However, they highlighted the need for indicators on orphan patients and other health care providers. Differences between regions were observed, particularly in the interest in continuity of care in university/peripheral regions and the need for diagnostic tools adapted to the culture in remote/isolated regions.

CONCLUSIONS

These dashboards support the implementation of an Alzheimer Plan and contribute to the emergence of a learning health care system culture. This project allows each region to increase its monitoring capacity for the implementation of the QAP and facilitates reflection among individuals locally carrying out the implementation. The perspectives expressed will guide the preparation of the next iteration of the dashboards.

Developing public health surveillance dashboards: a scoping review on the design principles

BACKGROUND

Public Health Dashboards (PHDs) facilitate the monitoring and prediction of disease outbreaks by continuously monitoring the health status of the community. This study aimed to identify design principles and determinants for developing public health surveillance dashboards.

METHODOLOGY

This scoping review is based on Arksey and O'Malley's framework as included in JBI guidance. Four databases were used to review and present the proposed principles of designing PHDs: IEEE, PubMed, Web of Science, and Scopus. We considered articles published between January 1, 2010 and November 30, 2022. The final search of articles was done on November 30, 2022. Only articles in the English language were included. Qualitative synthesis and trend analysis were conducted.

RESULTS

Findings from sixty-seven articles out of 543 retrieved articles, which were eligible for analysis, indicate that most of the dashboards designed from 2020 onwards were at the national level for managing and monitoring COVID-19. Design principles for the public health dashboard were presented in five groups, i.e., considering aim and target users, appropriate content, interface, data analysis and presentation types, and infrastructure.

CONCLUSION

Effective and efficient use of dashboards in public health surveillance requires implementing design principles to improve the functionality of these systems in monitoring and decision-making. Considering user requirements, developing a robust infrastructure for improving data accessibility, developing, and applying Key Performance Indicators (KPIs) for data processing and reporting purposes, and designing interactive and intuitive interfaces are key for successful design and development.

A mixed methods evaluation assessing the feasibility of implementing a PrEP data dashboard in the Southeastern United States

BACKGROUND

Alabama is one of seven priority states for the National Ending the HIV Epidemic Initiative due to a disproportionate burden of rural infections. To reverse growing infection rates, the state must increase its focus on prevention efforts, including novel strategies. One such approach is to utilize dashboards that visualize real-time data on the pre-exposure prophylaxis (PrEP) care continuum to assist in prioritizing evidence-based preventative care for those most vulnerable for HIV infection.

METHODS

We conducted a mixed methods evaluation to ascertain stakeholders' perceptions on the acceptability, feasibility, appropriateness, and usability of a PrEP care continuum dashboard, as well as gain insight on ways to improve the activities necessary to sustain it. Clinicians, administrators, and data personnel from participating sites in Alabama completed surveys (n = 9) and participated in key informant interviews (n = 10) to better understand their experiences with the prototype data dashboard and to share feedback on how it can be modified to best fit their needs.

RESULTS

Surveys and interviews revealed that all participants find the pilot data dashboard to be an acceptable, feasible, and appropriate intervention for clinic use. Overall, stakeholders find the pilot dashboard to be usable and helpful in administrative efforts, such as report and grant writing; however, additional refining is needed in order to reduce burden and optimize usefulness. Participants voiced concerns about their site's abilities to sustain the dashboard, including the lack of systematized PrEP protocols and limited funds and staff time dedicated to PrEP data collection, cleaning, and upload.

CONCLUSION

Study participants from clinics providing HIV prevention services, including PrEP, in Alabama voiced interest in sustaining and refining a data dashboard that tracks clients across the PrEP care continuum. Despite viewing the platform itself as an acceptable, feasible, and appropriate intervention, participants agreed that efforts need to be focused on standardizing PrEP data collection protocols in order to ensure consistent, accurate data capture and that limited funds and staff time are barriers to the sustained implementation of the dashboard in practice.

Use of a Medication Safety Audit and Feedback Tool in the Emergency Department Is Affected by Prescribing Characteristics

BACKGROUND

The Enhancing Quality of Prescribing Practices for Older Veterans Discharged from the Emergency Department (EQUIPPED) program developed an audit and feedback health information technology (IT) solution with the intent to replace the in-person academic detailing service provided by the program. The EQUIPPED dashboard provides emergency department (ED) providers with a personalized view of their prescribing performance.

OBJECTIVES

Here, we analyze the association between ED provider characteristics and viewership of the EQUIPPED dashboard, adding insight into strategies for addressing barriers to initial use.

METHODS

We performed a retrospective analysis of EQUIPPED dashboard viewership among four Veterans Affairs (VA) EDs. We extracted quantitative data from user interaction logs to determine evidence of dashboard use. Provider characteristics and baseline potentially inappropriate medication (PIM) prescribing rate were extracted from the VA's Corporate Data Warehouse. Logistic regression was used to examine the association between dashboard use and provider characteristics.

RESULTS

A total of 82 providers were invited to receive audit and feedback via the EQUIPPED dashboard. Among invited providers, 40 (48.7%) had evidence of at least 1 dashboard view during the 1-year feedback period. Adjusted analyses suggest that providers with a higher baseline PIM prescribing rate were more likely to use the dashboard (odds ratio [OR]: 1.22; 95% confidence interval [CI]: 1.01-1.47). Furthermore, providers at ED site D were more likely to use the dashboard in comparison to the other sites (OR: 9.99; 95% CI: 1.72-58.04) and reportedly had the highest site-level baseline PIM rate.

CONCLUSION

Providers with lower PIM prescribing rates (i.e., <5%) receive communication from an integrated dashboard reminder system that they are "optimal prescribers" which may have discouraged initial attempts to view the dashboard. Site D had the highest baseline PIM rate, but further qualitative investigation is warranted to better understand why site D had the greatest users of the dashboard.

Digital health in smart cities: Rethinking the remote health monitoring architecture on combining edge, fog, and cloud

Purpose

Smart cities that support the execution of health services are more and more in evidence today. Here, it is mainstream to use IoT-based vital sign data to serve a multi-tier architecture. The state-of-the-art proposes the combination of edge, fog, and cloud computing to support critical health applications efficiently. However, to the best of our knowledge, initiatives typically present the architectures, not bringing adaptation and execution optimizations to address health demands fully.

Methods

This article introduces the VitalSense model, which provides a hierarchical multi-tier remote health monitoring architecture in smart cities by combining edge, fog, and cloud computing.

Results

Although using a traditional composition, our contributions appear in handling each infrastructure level. We explore adaptive data compression and homomorphic encryption at the edge, a multi-tier notification mechanism, low latency health traceability with data sharding, a Serverless execution engine to support multiple fog layers, and an offloading mechanism based on service and person computing priorities.

Conclusions

This article details the rationale behind these topics, describing VitalSense use cases for disruptive healthcare services and preliminary insights regarding prototype evaluation.

Characteristics and specifications of dashboards developed for the COVID-19 pandemic: a scoping review

Aim

The use of information-based solutions such as dashboards is on the rise for taking fact-based actions against the COVID-19 crisis. This scoping review aimed to comprehensively investigate COVID-19 dashboards from different technical perspectives.

Subject and methods

Three main bibliographic databases, PubMed, Web of Science, and Scopus, were searched on 28 August 2021 to retrieve relevant studies. Arksey and O'Malley's (Int J Soc Res Methodol 8(1):19-32, 2005) methodological framework and the enhanced version of this methodology developed by Levac et al. (Implement Sci 5(1):1-9, 2010) were adopted for conducting this review.

Results

In total, 26 articles were included. The COVID-19 dashboards mainly focused on the infected (n = 25), deceased (n = 17), and recovered cases (n = 13), as well as the performed test (n = 10). Most of the dashboards were interactive, with public accessibility targeting various user groups. While some dashboards were both informative and supportive (38%), most were mainly informative (92%). The dashboard data were generally analyzed using simple techniques (58%) and delivered through web-based applications (88%).

Conclusion

Dashboards can help immediately manage, analyze, and summarize a huge amount of information about a COVID-19 outbreak. The findings revealed that the developed COVID-19 dashboards share more or less analogous characteristics that could lay the groundwork for designing and developing dashboards for any other pandemic.

Analytical exploratory tool for healthcare professionals to monitor cancer patients' progress

Introduction

Cancer is a primary public concern in the European continent. Due to the large case numbers and survival rates, a significant population is living with cancer needs. Consequently, health professionals must deal with complex treatment decision-making processes. In this context, a large quantity of data is collected during cancer care delivery. Once collected, these data are complex for health professionals to access to support clinical decision-making and performance review. There is a need for innovative tools that make clinical data more accessible to support cancer health professionals in these activities.

Methods

Following a co-creation, an interactive approach thanks to the Interactive Process Mining paradigm, and data from a tertiary hospital, we developed an exploratory tool to present cancer patients' progress over time.

Results

This work aims to collect and report the process of developing an exploratory analytical Interactive Process Mining tool with clinical relevance for healthcare professionals for monitoring cancer patients' care processes in the context of the LifeChamps project together with a graphical and navigable Process Indicator in the context of prostate cancer patients.

Discussion

The tool presented includes Process Mining techniques to infer actual processes and present understandable results visually and navigable, looking for different types of patients, trajectories, and behaviors.

Triptychon: Usability evaluation and implementation of a web-based application for patients' lab and vital parameters

Background

A major challenge in healthcare is the interpretation of the constantly increasing amount of clinical data of interest to inpatients for diagnosis and therapy. It is vital to accurately structure and represent data from different sources to help clinicians make informed decisions.

Objective

We evaluated the usability of our tool 'Triptychon' - a three-part visualisation dashboard of essential patients' medical data provided by a direct overview of their hospitalisation information, laboratory, and vital parameters over time.

Methods

The study followed a cohort of 20 participants using the mixed-methods approach, including interviews and the usability questionnaires, Health Information Technology Usability Evaluation Scale (Health-ITUES), and User Experience Questionnaire (UEQ). The participant's interactions with the dashboard were also observed. A thematic analysis approach was applied to analyse qualitative data and the quantitative data's task completion time and success rates.

Results

The usability evaluation of the visualisation dashboard revealed issues relating to the terminology used in the user interface and colour coding in its left and middle panels. The Health-ITUES score was 3.72 (standard deviation (SD) = 1.0), and the UEQ score was 1.6 (SD = 0.74). The study demonstrated improvements in intuitive dashboard use and overall satisfaction with using the dashboard daily.

Conclusion

The Triptychon dashboard is a promising new tool for medical data presentation. We identified design and layout issues of the dashboard for improving its usability in routine clinical practice. According to users' feedback, the three panels on the dashboard provided a holistic view of a patient's hospital stay.

Requirements and challenges of hospital dashboards: a systematic literature review

Background

Today, the use of data in administrative and clinical processes is quite challenging due to the large volume of data, data collection from various sources, and lack of data structure. As a data management tool, dashboards play an important role in timely visual display of critical information on key performances.

Objectives

This systematic review aimed to identify functional and non-functional requirements, as well as challenges of using dashboards in hospitals.

Methods

In this systematic review, four databases, including the Web of Science, PubMed, EMBASE, and Scopus, were searched to find relevant articles from 2000 until May 30, 2020. The final search was conducted on May 30, 2020. Data collection was performed using a data extraction form and reviewing the content of relevant studies on the potentials and challenges of dashboard implementation.

Results

Fifty-four out of 1254 retrieved articles were selected for this study based on the inclusion and exclusion criteria. The functional requirements for dashboards included reporting, reminders, customization, tracking, alert creation, and assessment of performance indicators. On the other hand, the non-functional requirements included the dashboard speed, security, ease of use, installation on different devices (e.g., PCs and laptops), integration with other systems, web-based design, inclusion of a data warehouse, being up-to-data, and use of data visualization elements based on the user’s needs. Moreover, the identified challenges were categorized into four groups: data sources, dashboard content, dashboard design, implementation, and integration in other systems at the hospital level.

Conclusion

Dashboards, by providing information in an appropriate manner, can lead to the proper use of information by users. In order for a dashboard to be effective in clinical and managerial processes, particular attention must be paid to its capabilities, and the challenges of its implementation need to be addressed.

Using an Annual Diversity, Equity and Inclusion Dashboard to Accelerate Change in Academic Radiology Departments

Despite widespread interest in creating a more equitable and inclusive culture, a lack of workforce diversity persists in Radiology, in part due to a lack of universal and longitudinal metrics across institutions. In an attempt to establish benchmarks, a subset of the Society of Chairs of Academic Radiology Departments (SCARD) Diversity, Equity and Inclusion (DEI) Committee volunteered to design a DEI dashboard as a potential tool for academic radiology programs to use to document and track their progress. This freely-available, modular dashboard includes suggested (plus optional department-defined) DEI activities/parameters and suggested assessment criteria across three domains: faculty, residents & fellows, and medical students; it can be completed, in whole or in part, by departmental leaders annually. The suggested metrics and their associated rubrics were derived from the collective experiences of the five working group members, all of whom are chairs of academic radiology departments. The resulting dashboard was unanimously approved by the remaining 14 DEI committee members and endorsed by the SCARD board of directors.

Developing the VA Geriatric Scholars Programs' Clinical Dashboards Using the PDSA Framework for Quality Improvement

Background  Involving clinician end users in the development process of clinical dashboards is important to ensure that user needs are adequately met prior to releasing the dashboard for use. The challenge with following this approach is that clinician end users can undergo periodic turnover, meaning, the clinicians that played a role in the initial development process may not be the same individuals that use the dashboard in future.

Objectives  Here, we summarize our Plan, Do, Study, Act (PDSA)-guided clinical dashboard development process for the VA Geriatric Scholars Program (GSP) and the value of continuous, iterative development. We summarize dashboard adaptations that resulted from two PDSA cycles of improvement for the potentially inappropriate medication dashboard (PIMD), one of many Geriatric Scholars clinical dashboards. We also present the evaluative performance of the PIMD.

Methods  Evaluation of the PIMD was performed using the system usability scale (SUS) and through review of user interaction logs. Routine end users that were Geriatric Scholars and had evidence of 5 or more dashboard views were invited to complete an electronic form that contained the 10-item SUS.

Results  The proportion of Geriatric Scholars that utilized the PIMD increased for each iterative dashboard version that was produced as a byproduct from feedback (31.0% in 2017 to 60.2% in 2019). The overall usability of the PIMD among routine users was found to be above average (SUS score: 75.2 [95% CI 70.5–79.8]) in comparison to the recommended standard of acceptability (SUS score: 68)

Conclusion  The solicitation of feedback during dashboard orientations led to iterative adaptations of the PIMD that broadened its intended use. The presented PDSA-guided process to clinical dashboard development for the VA GSP can serve as a valuable framework for development teams seeking to produce well-adopted and usable health information technology (IT) innovations.

A Patient Outcomes-Driven Feedback Platform for Emergency Medicine Clinicians: Human-Centered Design and Usability Evaluation of Linking Outcomes Of Patients (LOOP)

Background

The availability of patient outcomes–based feedback is limited in episodic care environments such as the emergency department. Emergency medicine (EM) clinicians set care trajectories for a majority of hospitalized patients and provide definitive care to an even larger number of those discharged into the community. EM clinicians are often unaware of the short- and long-term health outcomes of patients and how their actions may have contributed. Despite large volumes of patients and data, outcomes-driven learning that targets individual clinician experiences is meager. Integrated electronic health record (EHR) systems provide opportunity, but they do not have readily available functionality intended for outcomes-based learning.

Objective

This study sought to unlock insights from routinely collected EHR data through the development of an individualizable patient outcomes feedback platform for EM clinicians. Here, we describe the iterative development of this platform, Linking Outcomes Of Patients (LOOP), under a human-centered design framework, including structured feedback obtained from its use.

Methods

This multimodal study consisting of human-centered design studios, surveys (24 physicians), interviews (11 physicians), and a LOOP application usability evaluation (12 EM physicians for ≥30 minutes each) was performed between August 2019 and February 2021. The study spanned 3 phases: (1) conceptual development under a human-centered design framework, (2) LOOP technical platform development, and (3) usability evaluation comparing pre- and post-LOOP feedback gathering practices in the EHR.

Results

An initial human-centered design studio and EM clinician surveys revealed common themes of disconnect between EM clinicians and their patients after the encounter. Fundamental postencounter outcomes of death (15/24, 63% respondents identified as useful), escalation of care (20/24, 83%), and return to ED (16/24, 67%) were determined high yield for demonstrating proof-of-concept in our LOOP application. The studio aided the design and development of LOOP, which integrated physicians throughout the design and content iteration. A final LOOP prototype enabled usability evaluation and iterative refinement prior to launch. Usability evaluation compared to status quo (ie, pre-LOOP) feedback gathering practices demonstrated a shift across all outcomes from “not easy” to “very easy” to obtain and from “not confident” to “very confident” in estimating outcomes after using LOOP. On a scale from 0 (unlikely) to 10 (most likely), the users were very likely (9.5) to recommend LOOP to a colleague.

Conclusions

This study demonstrates the potential for human-centered design of a patient outcomes–driven feedback platform for individual EM providers. We have outlined a framework for working alongside clinicians with a multidisciplined team to develop and test a tool that augments their clinical experience and enables closed-loop learning.

Dimensions of performance and related key performance indicators addressed in healthcare organisations: A literature review

INTRODUCTION

Performance measurement systems have become essential managerial tools for healthcare organisations in the last few decades. They allow hospital managers to pilot their institution and assess the development of the organisation in helping managers in decision-making and viewing the different impacts of these decisions. However, there is a need to investigate further the dimensions of performance those performance measurement systems address.

METHODS

A literature review was primarily conduced about performance measures in healthcare organisations. A comparative study was secondly made to identify the different performance dimensions that are present in the literature during the last decade. Forty-nine studies were considered and sixteen proposal frameworks were used to make the comparative analyses.

RESULTS

We classified dimensions depending on the frequency of mobilisation of their components in four categories: the stars, the first runners-up, the opportunists and the forgotten ones. For each of the dimensions presented in this classification, the main types of KPIs proposed in the theoretical frameworks are presented. A discussion on relevance and possible blind spots is then conducted.

CONCLUSION

Although they were a lot of proposal frameworks of KPI proposed in the last decades to assess healthcare organisations, some dimensions remain underrepresented. There is still a need to develop structure KPI and describe their links. To go further, the development of dashboards asks the question of the definition of KPI, the description of their interconnections and their temporality of driving, because static performance reporting systems are not able to completely satisfy healthcare manager's decision support needs.

Applications, features and key indicators for the development of Covid-19 dashboards: A systematic review study

Introduction

Interactive dashboards can collect data from various information sources and be used nationally and internationally. These information systems have played an important role in managing and controlling epidemic diseases, especially Covid-19. This study aimed to identify the applications, features, and key indicators of advanced dashboards in Covid-19.

Method

The present article is a systematic review study that searched the PubMed, Scopus, and ISI web of sciences databases in 2021 by combining the relevant keywords. After applying the inclusion and exclusion criteria and selecting articles, data collection was prepared using a data collection form. Data analysis was performed using the content analysis method.

Results

Out of 171 articles retrieved, 19 were included in the study for review by applying inclusion and exclusion criteria in the first stage. The most important data sources for the studied dashboards included general online, national, and hospital databases. Monitoring and tracking in the target community and resource management (hospital and public) are the most important issues in Covid-19 dashboards. The study showed that KPIs in 5 main categories of indicators related to hospital beds, clinical data in the hospital, diagnostic and therapeutic measures of hospitals, epidemiological data at the level community, and follow-up indicators of Covid-19 studies were worldwide.

Conclusion

Considering the technological advances at the world level and the large amount of data produced, one of the effective solutions for managing and controlling epidemic and pandemic conditions and diseases is the rapid development of interactive dashboards; Therefore, it is suggested that health officials and policymakers, in addition to developing and updating the existing dashboards in the field of Covid-19, developing the dashboard immediately in case of similar conditions.

Big Data Health Care Innovations: Performance Dashboarding as a Process of Collective Sensemaking

Big data is poised to revolutionize health care, and performance dashboards can be an important tool to manage big data innovations. Dashboards show the progress being made and provide critical management information about effectiveness and efficiency. However, performance dashboards are more than just a clear and straightforward representation of performance in the health care context. Instead, the development and maintenance of informative dashboards can be more productively viewed as an interactive and iterative process involving all stakeholders. We refer to this process as dashboarding and reflect on our learnings within a large European Union–funded project. Within this project, multiple big data applications in health care are being developed, piloted, and scaled up. In this paper, we discuss the ways in which we cope with the inherent sensitivities and tensions surrounding dashboarding in such a dynamic environment.

Enriching the Value of Patient Experience Feedback: Web-Based Dashboard Development Using Co-design and Heuristic Evaluation

Background

There is an abundance of patient experience data held within health care organizations, but stakeholders and staff are often unable to use the output in a meaningful and timely way to improve care delivery. Dashboards, which use visualized data to summarize key patient experience feedback, have the potential to address these issues.

Objective

The aim of this study is to develop a patient experience dashboard with an emphasis on Friends and Family Test (FFT) reporting, as per the national policy drive.

Methods

A 2-stage approach was used—participatory co-design involving 20 co-designers to develop a dashboard prototype, followed by iterative dashboard testing. Language analysis was performed on free-text patient experience data from the FFT, and the themes and sentiments generated were used to populate the dashboard with associated FFT metrics. Heuristic evaluation and usability testing were conducted to refine the dashboard and assess user satisfaction using the system usability score.

Results

The qualitative analysis from the co-design process informed the development of the dashboard prototype with key dashboard requirements and a significant preference for bubble chart display. The heuristic evaluation revealed that most cumulative scores had no usability problems (18/20, 90%), had cosmetic problems only (7/20, 35%), or had minor usability problems (5/20, 25%). The mean System Usability Scale score was 89.7 (SD 7.9), suggesting an excellent rating.

Conclusions

The growing capacity to collect and process patient experience data suggests that data visualization will be increasingly important in turning feedback into improvements to care. Through heuristic usability, we demonstrated that very large FFT data can be presented in a thematically driven, simple visual display without the loss of the nuances and still allow for the exploration of the original free-text comments. This study establishes guidance for optimizing the design of patient experience dashboards that health care providers find meaningful, which in turn drives patient-centered care.

Requirements for bespoke ICU Dashboard in response to the COVID-19 Pandemic

Background

Intensive care units (ICUs) around the world are in high demand due to patients with COVID-19 requiring hospitalization. As researchers at the University of Bristol, we were approached to develop a bespoke data visualization dashboard to assist two local ICUs during the pandemic that will centralize disparate data sources in the ICU to help reduce the cognitive load on busy ICU staff in the ever-evolving pandemic.

Objective

The aim of this study was to conduct interviews with ICU staff in University Hospitals Bristol and Weston National Health Service Foundation Trust to elicit requirements for a bespoke dashboard to monitor the high volume of patients, particularly during the COVID-19 pandemic.

Methods

We conducted six semistructured interviews with clinical staff to obtain an overview of their requirements for the dashboard and to ensure its ultimate suitability for end users. Interview questions aimed to understand the job roles undertaken in the ICU, potential uses of the dashboard, specific issues associated with managing COVID-19 patients, key data of interest, and any concerns about the introduction of a dashboard into the ICU.

Results

From our interviews, we found the following design requirements: (1) a flexible dashboard, where the functionality can be updated quickly and effectively to respond to emerging information about the management of this new disease; (2) a mobile dashboard, which allows staff to move around on wards with a dashboard, thus potentially replacing paper forms to enable detailed and consistent data entry; (3) a customizable and intuitive dashboard, where individual users would be able to customize the appearance of the dashboard to suit their role; (4) real-time data and trend analysis via informative data visualizations that help busy ICU staff to understand a patient’s clinical trajectory; and (5) the ability to manage tasks and staff, tracking both staff and patient movements, handovers, and task monitoring to ensure the highest quality of care.

Conclusions

The findings of this study confirm that digital solutions for ICU use would potentially reduce the cognitive load of ICU staff and reduce clinical errors at a time of notably high demand of intensive health care.

The experiences of 33 national COVID-19 dashboard teams during the first year of the pandemic in the World Health Organization European Region: A qualitative study

Background

Governments across the World Health Organization (WHO) European Region have prioritised dashboards for reporting COVID-19 data. The ubiquitous use of dashboards for public reporting is a novel phenomenon.

Objective

This study explores the development of COVID-19 dashboards during the first year of the pandemic and identifies common barriers, enablers and lessons from the experiences of teams responsible for their development.

Methods

We applied multiple methods to identify and recruit COVID-19 dashboard teams, using a purposive, quota sampling approach. Semi-structured group interviews were conducted from April to June 2021. Using elaborative coding and thematic analysis, we derived descriptive and explanatory themes from the interview data. A validation workshop was held with study participants in June 2021.

Results

Eighty informants participated, representing 33 national COVID-19 dashboard teams across the WHO European Region. Most dashboards were launched swiftly during the first months of the pandemic, February to May 2020. The urgency, intense workload, limited human resources, data and privacy constraints and public scrutiny were common challenges in the initial development stage. Themes related to barriers or enablers were identified, pertaining to the pre-pandemic context, pandemic itself, people and processes and software, data and users. Lessons emerged around the themes of simplicity, trust, partnership, software and data and change.

Conclusions

COVID-19 dashboards were developed in a learning-by-doing approach. The experiences of teams reveal that initial underpreparedness was offset by high-level political endorsement, the professionalism of teams, accelerated data improvements and immediate support with commercial software solutions. To leverage the full potential of dashboards for health data reporting, investments are needed at the team, national and pan-European levels.

CHDA CERTIFICATION EXAM SUCCESS FACTORS

This study explored possible success factors for passing the Certified Health Data Analyst Administration (CHDA) certification exam. According to the American Health Information Management Association (AHIMA), in 2019, only 10 percent of first-time test-takers passed the CHDA exam. Literature review offered insight into factors related to passing certification exams. Sources included existing, relevant peer-reviewed, and published literature since 1990 within 87 educational and health/medicine databases and 62 other articles and journal databases available at the University of South Dakota library. A correlational design was used in the study. Data was retrieved from AHIMA, cleaned, and data analysis was completed using binary logistic regression analysis. The CHDA study results indicate that candidates between ages 30 and 49 are less likely to pass the exam than those ages 50 and above, and those candidates with a master's degree are more likely to pass the exam than those with an associate or bachelor's degree. This new information will help improve the exam pass rates, provide a foundation for CHDA exam research, and add new knowledge in the HIM professional body of research.

Toward Better Data Dashboards for US Drug Value Assessments

OBJECTIVES

To explore the use of data dashboards to convey information about a drug's value, and reduce the need to collapse dimensions of value to a single measure.

METHODS

Review of the literature on US Drug Value Assessment Frameworks, and discussion of the value of data dashboards to improve the manner in which information on value is displayed.

RESULTS

The incremental cost per quality-adjusted life-year ratio is a useful starting point for conversation about a drug's value, but it cannot reflect all of the elements of value about which different audiences care deeply. Data dashboards for drug value assessments can draw from other contexts. Decision makers should be presented with well-designed value dashboards containing various metrics, including conventional cost per quality-adjusted life-year ratios as well as measures of a drug's impact on clinical and patient-centric outcomes, and on budgetary and distributional consequences, to convey a drug's value along different dimensions.

CONCLUSIONS

The advent of US drug value frameworks in health care has forced a concomitant effort to develop appropriate information displays. Researchers should formally test different formats and elements.

Emergency Department Quality Dashboard; a Systematic Review of Performance Indicators, Functionalities, and Challenges

Introduction:

Effective information management in the emergency department (ED) can improve the control and management of ED processes. Dashboards, known as data management tools, efficiently provide information and contribute greatly to control and management of ED. This study aimed to identify performance indicators quality dashboard functionalities, and analyze the challenges associated with dashboard implementation in the ED.

Methods:

This systematic review began with a search in four databases (Web of Science, PubMed, Embase, and Scopus) from 2000 to May 30, 2020, when the final search for papers was conducted. The data were collected using a data extraction form and the contents of the extracted papers were analyzed through ED performance indicators, dashboard functionalities, and implementation challenges.

Results:

Performance indicators reported in the reviewed papers were classified as the quality of care, patient flow, timeliness, costs, and resources. The main dashboard functionalities noted in the papers included reporting, customization, alert creation, resource management, and real-time information display. The dashboard implementation challenges included data sources, data quality, integration with other systems, adaptability of dashboard functionalities to user needs, and selection of appropriate performance indicators.

Conclusions:

Quality dashboards facilitate processes, communication, and situation awareness in the ED; hence, they can improve care provision in this department. To enhance the effectiveness and efficiency of ED dashboards, officials should set performance indicators and consider the conformity of dashboard functionalities with user needs. They should also integrate dashboards with other relevant systems at the departmental and hospital levels.

Exploring Changes to the Actionability of COVID-19 Dashboards Over the Course of 2020 in the Canadian Context: Descriptive Assessment and Expert Appraisal Study

BACKGROUND

Public web-based COVID-19 dashboards are in use worldwide to communicate pandemic-related information. Actionability of dashboards, as a predictor of their potential use for data-driven decision-making, was assessed in a global study during the early stages of the pandemic. It revealed a widespread lack of features needed to support actionability. In view of the inherently dynamic nature of dashboards and their unprecedented speed of creation, the evolution of dashboards and changes to their actionability merit exploration.

OBJECTIVE

We aimed to explore how COVID-19 dashboards evolved in the Canadian context during 2020 and whether the presence of actionability features changed over time.

METHODS

We conducted a descriptive assessment of a pan-Canadian sample of COVID-19 dashboards (N=26), followed by an appraisal of changes to their actionability by a panel of expert scorers (N=8). Scorers assessed the dashboards at two points in time, July and November 2020, using an assessment tool informed by communication theory and health care performance intelligence. Applying the nominal group technique, scorers were grouped in panels of three, and evaluated the presence of the seven defined features of highly actionable dashboards at each time point.

RESULTS

Improvements had been made to the dashboards over time. These predominantly involved data provision (specificity of geographic breakdowns, range of indicators reported, and explanations of data sources or calculations) and advancements enabled by the technologies employed (customization of time trends and interactive or visual chart elements). Further improvements in actionability were noted especially in features involving local-level data provision, time-trend reporting, and indicator management. No improvements were found in communicative elements (clarity of purpose and audience), while the use of storytelling techniques to narrate trends remained largely absent from the dashboards.

CONCLUSIONS

Improvements to COVID-19 dashboards in the Canadian context during 2020 were seen mostly in data availability and dashboard technology. Further improving the actionability of dashboards for public reporting will require attention to both technical and organizational aspects of dashboard development. Such efforts would include better skill-mixing across disciplines, continued investment in data standards, and clearer mandates for their developers to ensure accountability and the development of purpose-driven dashboards.

The usefulness of an ambulatory care pharmacy performance dashboard during the COVID-19 pandemic in a complex tertiary care system

Purpose

To describe the usefulness of an innovative “semi–real-time” pharmacy dashboard in managing workload during the unpredictable coronavirus disease 2019 (COVID-19) pandemic.

Summary

We created a pharmacy dashboard to monitor workload and key performance indicators during the dynamic COVID-19 crisis. The dashboard accessed the prescribing workload from our clinical information system and filled prescriptions from robotic prescription dispensing systems. The aggregated data was visualized using modern tools. The dashboard presents performance data in near real time and is updated every 15 minutes. After validation during the early weeks of the COVID-19 crisis, the dashboard provided reliable data and served as a great decision support aid in calculating the backlog of prescribed but unfilled prescriptions. It also aided in adjusting manpower, identifying prescribing and dispensing patterns, identifying trends, and diverting staff resources to appropriate locations. The dashboard has been useful in clearing the backlog in a timely manner, staff planning, and predicting the next coming surge so that we can proactively minimize accumulation of backlogged prescriptions.

Conclusion

Developing a dynamic, semi–real-time pharmacy dashboard during unstable circumstances such as those that have arisen during the COVID-19 pandemic can be very useful in ambulatory care pharmacy workload management.

Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards

Background

Since the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed.

Objective

The aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users (“why”), content and data (“what”), and analyses and displays (“how” they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards.

Methods

We conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question.

Results

A total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are “close to home”; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues.

Conclusions

COVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Big Data and Digital Solutions: Laying the Foundation for Cardiovascular Population Management CME

There are huge gaps in evidence-based cardiovascular care at the national, organizational, practice, and provider level that can be attributed to variation in provider attitudes, lack of incentives for positive change and care standardization, and observed uncertainty in clinical decision making. Big data analytics and digital application platforms-such as patient care dashboards, clinical decision support systems, mobile patient engagement applications, and key performance indicators-offer unique opportunities for value-based healthcare delivery and efficient cardiovascular population management. Successful implementation of big data solutions must include a multidisciplinary approach, including investment in big data platforms, harnessing technology to create novel digital applications, developing digital solutions that can inform the actions of clinical and policy decision makers and relevant stakeholders, and optimizing engagement strategies with the public and information-empowered patients.

Association Between Use of EHR-Generated Dashboards and Hospital Outcomes in 30-Day Heart Failure Readmissions

EXECUTIVE SUMMARY

This study examined whether usage of clinical data from the electronic health record (EHR) to create organizational- and unit-level performance dashboards and assess adherence to clinical practice guidelines is associated with hospital outcomes in risk-adjusted 30-day readmissions for patients with a principal diagnosis of heart failure (HF). It further assesses the association between the metrics of hospital financial health and the usage of those EHR-generated tools.A cross-sectional study design was used. The study used data from the 2016 American Hospital Association Annual Survey Information Technology Supplement, the Pennsylvania Health Care Cost Containment Council's (PHC4's) 2017 Hospital Performance Report, and the PHC4 General Acute Care Hospitals Financial Analysis Report for fiscal year 2017. Contingency tables, likelihood-ratio chi-square tests, and logistic regression were applied for data analysis.Usage of the EHR to assess adherence to clinical practice guidelines and create EHR-generated unit-level performance dashboards, rather than organizational performance dashboards, was more strongly associated with the hospitals' rating in risk-adjusted 30-day readmissions for HF patients. An increase in hospitals' operating margin was associated with greater odds of the usage of the discussed EHR tools, after controlling for hospital control/ownership and the total number of beds.Usage of some EHR-generated analytical tools may be more strongly associated with lower-than-expected 30-day hospital readmissions in HF patients than with other tools. Better hospital financial health is linked with greater uptake of the discussed analytical tools. Further research could help to determine which EHR-generated tools are more effective in improving hospital outcomes in HF.

Can publicly reported data be used to understand performance in an Australian rural hospital?

BACKGROUND

Despite agreement among policymakers, funders, consumers and researchers about the value of public reporting of health information, limited attention has been paid to how it can be used to understand the performance of rural hospitals.

OBJECTIVE

To determine whether publicly available information can be used to measure health service performance in a rural hospital.

METHOD

The study used performance data routinely reported for public consumption in Australia. Data across four domains, multiple measures and time periods were collected to examine access and equity; efficiency and sustainability; quality, safety and patient orientation; and employee engagement. Performance of the rural hospital was examined using a visualisation tool.

RESULTS

Visualisation of multiple measures of performance over time was achievable but required a high degree of health information management skills.

CONCLUSION AND IMPLICATIONS

Publicly reported data can be used to represent performance for a rural hospital. Timeliness, level of detail available and peer groupings of data limits optimal utility. Consumers, clinicians and health service managers wanting to understand the performance of rural hospitals will need to use significant health information management skills to gain a picture of performance. Further research in the applied use of publicly available performance data and relevant dashboards for rural hospitals is suggested.

Regulatory Sandboxes: A Cure for mHealth Pilotitis?

Mobile health (mHealth) and related digital health interventions in the past decade have not always scaled globally as anticipated earlier despite large investments by governments and philanthropic foundations. The implementation of digital health tools has suffered from 2 limitations: (1) the interventions commonly ignore the “law of amplification” that states that technology is most likely to succeed when it seeks to augment and not alter human behavior; and (2) end-user needs and clinical gaps are often poorly understood while designing solutions, contributing to a substantial decrease in usage, referred to as the “law of attrition” in eHealth. The COVID-19 pandemic has addressed the first of the 2 problems—technology solutions, such as telemedicine, that were struggling to find traction are now closely aligned with health-seeking behavior. The second problem (poorly designed solutions) persists, as demonstrated by a plethora of poorly designed epidemic prediction tools and digital contact-tracing apps, which were deployed at scale, around the world, with little validation. The pandemic has accelerated the Indian state’s desire to build the nation’s digital health ecosystem. We call for the inclusion of regulatory sandboxes, as successfully done in the fintech sector, to provide a real-world testing environment for mHealth solutions before deploying them at scale.

Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process

Background  With the increased usage of dashboard reporting systems to monitor and track patient panels by clinical users, developers must ensure that the information displays they produce are accurate and intuitive. When evaluating usability of a clinical dashboard among potential end users, developers oftentimes rely on methods such as questionnaires as opposed to other, more time-intensive strategies that incorporate direct observation.

Objectives  Prior to release of the potentially inappropriate medication (PIM) clinical dashboard, designed to facilitate completion of a quality improvement project by clinician scholars enrolled in the Veterans Affairs (VA) workforce development Geriatric Scholars Program (GSP), we evaluated the usability of the system. This article describes the process of usability testing a dashboard reporting system with clinicians using direct observation and think-aloud moderating techniques.

Methods  We developed a structured interview protocol that combines virtual observation, think-aloud moderating techniques, and retrospective questioning of the overall user experience, including use of the System Usability Scale (SUS). Thematic analysis was used to analyze field notes from the interviews of three GSP alumni.

Results  Our structured approach to usability testing identified specific functional problems with the dashboard reporting system that were missed by results from the SUS. Usability testing lead to overall improvements in the intuitive use of the system, increased data transparency, and clarification of the dashboard's purpose.

Conclusion  Reliance solely on questionnaires and surveys at the end stages of dashboard development can mask potential functional problems that will impede proper usage and lead to misinterpretation of results. A structured approach to usability testing in the developmental phase is an important tool for developers of clinician friendly systems for displaying easily digested information and tracking outcomes for the purpose of quality improvement.

Development, implementation and user experience of the Veterans Health Administration (VHA) dialysis dashboard

BACKGROUND

Adults with end-stage renal disease (ESRD) requiring chronic dialysis continue to suffer from poor health outcomes and represent a population rightfully targeted for quality improvement. Electronic dashboards are increasingly used in healthcare to facilitate quality measurement and improvement. However, detailed descriptions of the creation of healthcare dashboards are uncommonly available and formal inquiry into perceptions, satisfaction, and utility by clinical users has been rarely conducted, particularly in the context of dialysis care. Therefore, we characterized the development, implementation and user experience with Veterans Health Administration (VHA) dialysis dashboard.

METHODS

A clinical-quality dialysis dashboard was implemented, which displays clinical performance measures (CPMs) for Veterans with ESRD receiving chronic hemodialysis at all VHA facilities. Data on user experience and perceptions were collected via an e-mail questionnaire to dialysis medical directors and nurse managers at these facilities.

RESULTS

Since 2016 the dialysis dashboard reports monthly on CPMs for approximately 3000 Veterans receiving chronic hemodialysis across 70 VHA dialysis facilities. Of 141 dialysis medical directors and nurse managers, 61 completed the questionnaire. Sixty-six percent of respondents did not find the dashboard difficult to access, 64% agreed that it is easy to use, 59% agreed that its layout is good, and the majority agreed that presentation of data is clear (54%), accurate (56%), and up-to-date (54%). Forty-eight percent of respondents indicated that it helped them improve patient care while 12% did not. Respondents indicated that they used the dialysis dashboard for clinical reporting (71%), quality assessment/performance improvement (QAPI) (62%), and decision-making (23%).

CONCLUSIONS

Most users of the VHA dialysis dashboard found it accurate, up-to-date, easy to use, and helpful in improving patient care. It meets diverse user needs, including administrative reporting, clinical benchmarking and decision-making, and quality assurance and performance improvement (QAPI) activities. Moreover, the VHA dialysis dashboard affords national-, regional- and facility-level assessments of quality of care, guides and motivates best clinical practices, targets QAPI efforts, and informs and promotes population health management improvement efforts for Veterans receiving chronic hemodialysis.

Improving Health Care Management in Hospitals Through a Productivity Dashboard

Health information systems have been developed to help hospital managers steer daily operations, including key performance indicators (KPIs) for monitoring on a time-aggregated basis. Yet, current literature lacks in proposals of productivity dashboards to assist hospitals stakeholders. This research focuses on two related problems: (1) hospital organizations need access to productivity information to improve access to services; and (2) managers need productivity information to optimize resource allocation. This research consists in the development of dashboards to monitor information obtained from a hospital organization to support decision makers. To develop and evaluate the productivity dashboard, the Design Science Research (DSR) methodology was adopted. The dashboard was evaluated by stakeholders of a large Portuguese hospital who contributed to iteratively improving its design toward a useful decision support tool. Additionally, it was ascertained that monitoring productivity needs more study and that the dashboards on these themes are valuable assets at a monitoring level and subsequent decision-making process.

A qualitative study on measuring patient-centered care: Perspectives from clinician-scientists and quality improvement experts

BACKGROUND AND AIMS

Patient-centered care (PCC) benefits patients, health-care providers, and health-care systems by providing delivery of care that addresses patient values and needs while improving provider experiences, and by decreasing health-care expenditure. To improve PCC, health-care systems need to measure it. Recently, we developed a PCC framework that is evidence based and patient informed. The purpose of this study was to gather the perspective of clinician-scientists and quality improvement experts regarding the PCC domains included in the framework. Their perspectives were used to refine these domains, which ultimately will inform the development of PCC quality indicators.

METHODS

Participants were recruited via expert and snowball sampling. Semi-structured interviews were conducted with clinician-scientists and quality improvement experts from Canada, the United States, and the United Kingdom from October 2017 to January 2018. With the use of an interview guide developed using the PCC framework, interviews were audio recorded and transcribed for a thematic analysis using NVivo qualitative data analysis software. Inductive thematic analysis was used to identify themes and subthemes.

RESULTS

Sixteen semi-structured interviews were conducted, which included four clinician-scientists and 12 quality improvement experts. Twelve of the participants were from Canada, three from the United Kingdom, and one from the United States. From the thematic analysis, three major themes were identified: (a) measurability of PCC, (b) practical considerations for implementing measurement, and (c) policy and practice implications. Participants discussed barriers and recommendations to improve and increase the clarity of the PCC domains in health system reporting, resulting in several future directions to refine and target specific PCC domains.

CONCLUSION

Clinician-scientists and quality improvement experts provided key recommendations for the measurement of PCC. The perspectives of key stakeholders in PCC measurement will inform strategies for the implementation and uptake of patient-centered quality indicators in health-care systems. The views of these key experts can lay the foundation for the development of standardized measures of PCC, to ensure monitoring and improvement of PCC.

Automated analysis of free-text comments and dashboard representations in patient experience surveys: a multimethod co-design study

Background

Patient experience surveys (PESs) often include informative free-text comments, but with no way of systematically, efficiently and usefully analysing and reporting these. The National Cancer Patient Experience Survey (CPES), used to model the approach reported here, generates > 70,000 free-text comments annually.

Main aim

To improve the use and usefulness of PES free-text comments in driving health service changes that improve the patient experience.

Secondary aims

(1) To structure CPES free-text comments using rule-based information retrieval (IR) (‘text engineering’), drawing on health-care domain-specific gazetteers of terms, with in-built transferability to other surveys and conditions; (2) to display the results usefully for health-care professionals, in a digital toolkit dashboard display that drills down to the original free text; (3) to explore the usefulness of interdisciplinary mixed stakeholder co-design and consensus-forming approaches in technology development, ensuring that outputs have meaning for all; and (4) to explore the usefulness of Normalisation Process Theory (NPT) in structuring outputs for implementation and sustainability.

Design

A scoping review, rapid review and surveys with stakeholders in health care (patients, carers, health-care providers, commissioners, policy-makers and charities) explored clinical dashboard design/patient experience themes. The findings informed the rules for the draft rule-based IR [developed using half of the 2013 Wales CPES (WCPES) data set] and prototype toolkit dashboards summarising PES data. These were refined following mixed stakeholder, concept-mapping workshops and interviews, which were structured to enable consensus-forming ‘co-design’ work. IR validation used the second half of the WCPES, with comparison against its manual analysis; transferability was tested using further health-care data sets. A discrete choice experiment (DCE) explored which toolkit features were preferred by health-care professionals, with a simple cost–benefit analysis. Structured walk-throughs with NHS managers in Wessex, London and Leeds explored usability and general implementation into practice.

Key outcomes

A taxonomy of ranked PES themes, a checklist of key features recommended for digital clinical toolkits, rule-based IR validation and transferability scores, usability, and goal-oriented, cost–benefit and marketability results. The secondary outputs were a survey, scoping and rapid review findings, and concordance and discordance between stakeholders and methods.

Results

(1) The surveys, rapid review and workshops showed that stakeholders differed in their understandings of the patient experience and priorities for change, but that they reached consensus on a shortlist of 19 themes; six were considered to be core; (2) the scoping review and one survey explored the clinical toolkit design, emphasising that such toolkits should be quick and easy to use, and embedded in workflows; the workshop discussions, the DCE and the walk-throughs confirmed this and foregrounded other features to form the toolkit design checklist; and (3) the rule-based IR, developed using noun and verb phrases and lookup gazetteers, was 86% accurate on the WCPES, but needs modification to improve this and to be accurate with other data sets. The DCE and the walk-through suggest that the toolkit would be well accepted, with a favourable cost–benefit ratio, if implemented into practice with appropriate infrastructure support.

Limitations

Small participant numbers and sampling bias across component studies. The scoping review studies mostly used top-down approaches and focused on professional dashboards. The rapid review of themes had limited scope, with no second reviewer. The IR needs further refinement, especially for transferability. New governance restrictions further limit immediate use.

Conclusions

Using a multidisciplinary, mixed stakeholder, use of co-design, proof of concept was shown for an automated display of patient experience free-text comments in a way that could drive health-care improvements in real time. The approach is easily modified for transferable application.

Future work

Further exploration is needed of implementation into practice, transferable uses and technology development co-design approaches.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Designing a Performance Measurement System for Accountability, Quality Improvement, and Innovation

The purpose of this article is to detail a system for the design of performance measures that will be used to assess the achievement of a health care organization's strategic goals and its need for change. The article begins by emphasizing the importance of accountability and the need for the presence of a dynamic learning culture that is premised on a foundation of accountability, continuous improvement, learning, and innovation. This is followed by describing the importance of utilizing an interdisciplinary team with physician and patient involvement to guide the design and implementation of the performance measurement system. The goals of the system are then outlined and followed by a description of the process for the determination of the framework, scope, domains, measures, and reporting mechanisms for displaying the performance measures. Lastly, guidelines for the design of valid, reliable, and cost-effective performance measures are discussed with the aim of maximizing their utility by health care professionals, managers, and administrators.

Twelve tips to promote successful development of a learner performance dashboard within a medical education program

Easily accessible and interpretable performance data constitute critical feedback for learners that facilitate informed self-assessment and learning planning. To provide this feedback, there has been a proliferation of educational dashboards in recent years. An educational (learner) dashboard systematically delivers timely and continuous feedback on performance and can provide easily visualized and interpreted performance data. In this paper, we provide practical tips for developing a functional, user-friendly individual learner performance dashboard and literature review of dashboard development, assessment theory, and users' perspectives. Considering key design principles and maximizing current technological advances in data visualization techniques can increase dashboard utility and enhance the user experience. By bridging current technology with assessment strategies that support learning, educators can continue to improve the field of learning analytics and design of information management tools such as dashboards in support of improved learning outcomes.

Developing a hospital-wide quality and safety dashboard: a qualitative research study

Background

Several countries have national policies and programmes requiring hospitals to use quality and safety (QS) indicators. To present an overview of these indicators, hospital-wide QS (HWQS) dashboards are designed. There is little evidence how these dashboards are developed. The challenges faced to develop these dashboards in Dutch hospitals were retrospectively studied.

Methods

24 focus group interviews were conducted: 12 with hospital managers (n=25; 39.7%) and 12 support staff (n=38; 60.3%) in 12 of the largest Dutch hospitals. Open and axial codings were applied consecutively to analyse the data collected.

Results

A heuristic tool for the general development process for HWQS dashboards containing five phases was identified. In phase 1, hospitals make inventories to determine the available data and focus too much on quantitative data relevant for accountability. In phase 2, hospitals develop dashboard content by translating data into meaningful indicators for different users, which is not easy due to differing demands. In phase 3, hospitals search for layouts that depict the dashboard content suited for users with different cognitive abilities and analytical skills. In phase 4, hospitals try to integrate dashboards into organisational structures to ensure that data are systematically reviewed and acted on. In phase 5, hospitals want to improve the flexibility of their dashboards to make this adaptable under differing circumstances.

Conclusion

The literature on dashboards addresses the technical and content aspects of dashboards, but overlooks the organisational development process. This study shows how technical and organisational aspects are relevant in development processes.

The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review

Background

Intensive Care Units (ICUs) in the United States admit more than 5.7 million people each year. The ICU level of care helps people with life-threatening illness or injuries and involves close, constant attention by a team of specially-trained health care providers. Delay between condition onset and implementation of necessary interventions can dramatically impact the prognosis of patients with life-threatening diagnoses. Evidence supports a connection between information overload and medical errors. A tool that improves display and retrieval of key clinical information has great potential to benefit patient outcomes. The purpose of this review is to synthesize research on the use of visualization dashboards in health care.

Objective

The purpose of conducting this literature review is to synthesize previous research on the use of dashboards visualizing electronic health record information for health care providers. A review of the existing literature on this subject can be used to identify gaps in prior research and to inform further research efforts on this topic. Ultimately, this evidence can be used to guide the development, testing, and implementation of a new solution to optimize the visualization of clinical information, reduce clinician cognitive overload, and improve patient outcomes.

Methods

Articles were included if they addressed the development, testing, implementation, or use of a visualization dashboard solution in a health care setting. An initial search was conducted of literature on dashboards only in the intensive care unit setting, but there were not many articles found that met the inclusion criteria. A secondary follow-up search was conducted to broaden the results to any health care setting. The initial and follow-up searches returned a total of 17 articles that were analyzed for this literature review.

Results

Visualization dashboard solutions decrease time spent on data gathering, difficulty of data gathering process, cognitive load, time to task completion, errors, and improve situation awareness, compliance with evidence-based safety guidelines, usability, and navigation.

Conclusions

Researchers can build on the findings, strengths, and limitations of the work identified in this literature review to bolster development, testing, and implementation of novel visualization dashboard solutions. Due to the relatively few studies conducted in this area, there is plenty of room for researchers to test their solutions and add significantly to the field of knowledge on this subject.

A Human-Centered Platform for HIV Infection Reduction in New York: Development and Usage Analysis of the Ending the Epidemic (ETE) Dashboard

BACKGROUND

Dashboards have been increasingly used in clinic-based interventions, such as clinical performance improvement and monitoring risk of hospital readmissions, and are now gaining traction in population-based interventions, especially in disease assessment.

OBJECTIVE

We describe the design, development, and usage analysis of a geovisualization dashboard, the Ending the Epidemic (ETE) Dashboard. The ETE dashboard is a tool developed to track New York's progress towards achieving the goal of its ETE Initiative, to reduce new HIV infections from 3000 per year to 750 per year by the end of 2020.

METHODS

The ETE dashboard was adapted from an existing human-centered geovisualization platform, SanaViz, an Internet-enabled, interactive app incorporating principles of human-centered design and cognitive fit theory to enhance visual exploration of population health data. Usage evaluation of the ETE geovisualization dashboard was conducted using Google Analytics over a 4-week period from March 19 to April 18, 2016. The aim was to monitor user activity and analyze traffic on the ETE dashboard using evidence-based metrics that can provide adequate feedback to enhance its utilization. Usage was characterized based on three metrics: (1) number of unique visits to each page, (2) average time on each page in seconds, and (3) page bounce rate (ie, percentage of visits where user left the site immediately after viewing just a single page). Further analysis was also conducted by cross-tabulating specific usage metrics.

RESULTS

Of 860 sessions, 324 sessions were initiated by unique users (37.7%). The most common acquisition channels included direct source (353/860, 41.0%); followed by referral traffic (340/860, 39.5%) and organic search (134/860, 15.5%). Usage statistics indicate that for the 860 sessions initiated by both new and returning users, the average viewing time was 8 minutes, 51 seconds, and the bounce rate was 46%. These statistics reflect positive results given that prior literature estimates an average session duration of 10-20 seconds and a bounce rate of 40-60% for most websites. Key findings from our study showed that the highest bounce rates were seen for the "About" page of the ETE website (65%), which describes the ETE initiative. The lowest bounce rates were seen for the ETE blog page (17%), consistent with prior research on website usage metrics that indicate that blog pages are often the most frequently viewed pages on Web portals.

CONCLUSIONS

Our findings reveal the potential of Google Analytics as a tool to enhance user traffic and performance of the ETE geovisualization platform by using feedback from regular monitoring of key parameters including page bounce rates and average time on page. It also identifies the need for a follow-up usability assessment of the system.

Building and launching an online quality improvement information exchange for home visiting programs in Missouri

Continuous quality improvement initiatives (CQII) in home visiting programs have traditionally occurred within a local implementing agency (LIA), parent organization, or funding provision. In Missouri, certain LIAs participate in the Missouri Maternal, Infant, and Early Childhood Home Visiting program (MIECHV). Their CQII activities and the coordination of CQI efforts across agencies are limited to quarterly meetings to discuss barriers to service delivery and newsletters. Their designed CQI process does not include evaluation of program fidelity or assessment nor supports to assist with identifying and prioritizing areas where improvement is needed. Therefore, much of LIA CQII are often lost to the benefit of external agencies facing similar challenges. We developed a virtual environment, the Missouri MIECHV Gateway, for CQII activities. The Gateway promotes and supports quality improvement for LIAs while aligning stakeholders from seven home visiting LIAs. Development of the Gateway environment aims to complement the existing MIECHV CQI framework by: 1) adding CQI elements that are missing or ineffective, 2) adding elements for CQI identification and program evaluation, and 3) offering LIAs a network to share CQI experiences and collaborate at a distance. This web-based environment allows LIA personnel to identify program activities in need of quality improvement, and guides the planning, implementation, and evaluation of CQII. In addition, the Gateway standardizes quality improvement training, collates overlapping resources, and supports knowledge translation, thus aimed to improve capacity for measurable change in organizational initiatives. This interactive web-based portal provides the infrastructure to virtually connect and engage LIAs in CQI and stimulate sharing of ideas and best practices. This article describes the characteristics, development, build, and launch of this quality improvement practice exchange virtual environment and present results of three usability pilot tests and the site launch. Briefly, prior to deployment to 58 users, usability pilot testing of the site occurred in three stages, to three defined groups. Pilot testing results were overall positive, desirable, and vital to improving the site prior to the full-launch. The majority of reviewers stated they would access and use the learning materials (87%), use the site for completing CQII (80%), and reported that the site will benefit their work teams in addressing agency challenges (66%). The majority of reviewers also approved of the developed fidelity assessment: as, easy to use (79%), having a clear purpose (86%), providing value in self-identification of CQII (75%), and recommendations were appropriate (79%). The System Usability Scale (SUS) score increased (10%) between pilot groups 2 and 3, with a mean SUS score of 71.6, above the U.S. average of 68. The site launched to 60 invited users; the majority (67%) adopted and used the site. Site stability was remarkable (6 total minutes of downtime). The site averaged 29 page views per day.

Hospital performance dashboards: a literature review

Purpose

The purpose of this paper is to give a comprehensive and updated analysis of the available literature on hospital dashboards.

Design/methodology/approach

A search of the current literature was performed by searching electronic databases, including Google Scholar, EBSCO and Medline, as well as books.

Findings

In all, 48 manuscripts consisting of peer reviewed articles, conference proceedings, case reports and text books were included in this review.

Practical implications

Despite the numerous advantages of performance dashboards, several authors have mentioned a number of challenges. It was evident from the literature that any setting requires significant effort, especially to ensure the quality of data being collected. In fact, significant investment, both in terms of financial and human resources, is required to achieve an effective dashboard. Furthermore, most of the studies available in the literature were individual case reports or anecdotal accounts rather than empirical studies. Thus, further research is required to ascertain the effectiveness of performance dashboards. In view of these findings, each organisation should make its own decisions whether or not to adopt performance dashboards.

Originality/value

Most of the literature is fragmented as it reports the use of different types of dashboards, namely strategic, tactical and operational, as separate tools. This literature review contributes to knowledge as it brings together the different types of dashboards and the cascading effect of one dashboard onto another in order to achieve and retain organisational alignment with the overall strategic goals.