Examining Tensions That Affect the Evaluation of Technology in Health Care: Considerations for System Decision Makers From the Perspective of Industry and Evaluators

Timing, Indicators, and Approaches to Digital Patient Experience Evaluation: Umbrella Systematic Review

BACKGROUND

The increasing prevalence of DH applications has outpaced research and practice in digital health (DH) evaluations. Patient experience (PEx) was reported as one of the challenges facing the health system by the World Health Organization. To generate evidence on DH and promote the appropriate integration and use of technologies, a standard evaluation of PEx in DH is required.

OBJECTIVE

This study aims to systematically identify evaluation timing considerations (ie, when to measure), evaluation indicators (ie, what to measure), and evaluation approaches (ie, how to measure) with regard to digital PEx. The overall aim of this study is to generate an evaluation guide for further improving digital PEx evaluation.

METHODS

This is a 2-phase study parallel to our previous study. In phase 1, literature reviews related to PEx in DH were systematically searched from Scopus, PubMed, and Web of Science databases. Two independent raters conducted 2 rounds of paper screening, including title and abstract screening and full-text screening, and assessed the interrater reliability for 20% (round 1: 23/115 and round 2: 12/58) random samples using the Fleiss-Cohen coefficient (round 1: k1=0.88 and round 2: k2=0.80). When reaching interrater reliability (k>0.60), TW conducted the rest of the screening process, leaving any uncertainties for group discussions. Overall, 38% (45/119) of the articles were considered eligible for further thematic analysis. In phase 2, to check if there were any meaningful novel insights that would change our conclusions, we performed an updated literature search in which we collected 294 newly published reviews, of which 102 (34.7%) were identified as eligible articles. We considered them to have no important changes to our original results on the research objectives. Therefore, they were not integrated into the synthesis of this review and were used as supplementary materials.

RESULTS

Our review highlights 5 typical evaluation objectives that serve 5 stakeholder groups separately. We identified a set of key evaluation timing considerations and classified them into 3 categories: intervention maturity stages, timing of the evaluation, and timing of data collection. Information on evaluation indicators of digital PEx was identified and summarized into 3 categories (intervention outputs, patient outcomes, and health care system impact), 9 themes, and 22 subthemes. A set of evaluation theories, common study designs, data collection methods and instruments, and data analysis approaches was captured, which can be used or adapted to evaluate digital PEx.

CONCLUSIONS

Our findings enabled us to generate an evaluation guide to help DH intervention researchers, designers, developers, and program evaluators evaluate digital PEx. Finally, we propose 6 directions for encouraging further digital PEx evaluation research and practice to address the challenge of poor PEx.

Self-directed Technology-Based Therapeutic Methods for Adult Patients Receiving Mental Health Services: Systematic Review

BACKGROUND

Technological interventions used to treat illnesses and promote health are grouped under the umbrella term of digital therapeutics. The use of digital therapeutics is becoming increasingly common in mental health. Although many technologies are currently being implemented, research supporting their usability, efficacy, and risk requires further examination, especially for those interventions that can be used without support.

OBJECTIVE

This review aims to identify the evidence-based, self-directed, technology-based methods of care that can be used in adult patients after they are discharged from mental health services. The interventions reviewed are automated with no human input required (either at the patient's or at the technology's end), so the patients can implement them without any support.

METHODS

A systematic review was conducted according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and PROSPERO (International Prospective Register of Systematic Reviews) guidelines in 3 databases: PubMed, Web of Science, and OVID. The inclusion criteria were self-directed, automated, and technology-based interventions related to mental health, primarily for adults, having a solid evaluation process. The interventions had to be self-directed, in that the participants could use the technology without any external guidance.

RESULTS

We identified 36 papers that met the inclusion criteria: 26 randomized controlled trials, 9 nonrandomized controlled trial quantitative studies, and 1 qualitative study. The technologies used included websites, automated text messaging, phone apps, videos, computer software, and integrated voice response. There were 22 studies focused on internet-based cognitive behavioral therapies as a therapeutic paradigm compared with the waitlist, web-based human-delivered therapy, and other interventions. Among these studies, 14 used paradigms other than the internet-based cognitive behavioral therapy. Of the 8 studies comparing guided and unguided digital care, 3 showed no differences, 3 favored guided interventions, and 2 favored unguided interventions. The research also showed that dropout rates were as high as 80%, citing potential problems with the acceptability of the suggested technologies.

CONCLUSIONS

There is limited research on the efficacy and suitability of self-directed technology-based care options for mental health. Digital technologies have the potential to bridge the gap between ambulatory care and independent living. However, these interventions may need to be developed collaboratively with the users to encourage their acceptability and to avoid high dropout rates.

Catalyzing Digital Health Innovation in Ontario: The Role of an Academic Medical Centre

Overcoming barriers to health system innovation is an ongoing challenge in Canada. A total of 51 participants attended a digital health symposium in October 2017 to discuss the role of an academic medical centre (AMC) in advancing innovation. The conversation centred around (i) the current state of innovation in healthcare; (ii) the need for an innovation catalyst; and (iii) the roadmap for an AMC to drive change. AMCs can address the barriers to digital health innovation in Canada by providing a centralized network and infrastructure that supports innovation throughout its journey from "bench to bedside" as well as supporting educational reform.

Barriers to Creating Scalable Business Models for Digital Health Innovation in Public Systems: Qualitative Case Study

Background

Health systems are increasingly looking toward the private sector to provide digital solutions to address health care demands. Innovation in digital health is largely driven by small- and medium-sized enterprises (SMEs), yet these companies experience significant barriers to entry, especially in public health systems. Complex and fragmented care models, alongside a myriad of relevant stakeholders (eg, purchasers, providers, and producers of health care products), make developing value propositions for digital solutions highly challenging.

Objective

This study aims to identify areas for health system improvement to promote the integration of innovative digital health technologies developed by SMEs.

Methods

This paper qualitatively analyzes a series of case studies to identify health system barriers faced by SMEs developing digital health technologies in Canada and proposed solutions to encourage a more innovative ecosystem. The Women’s College Hospital Institute for Health System Solutions and Virtual Care established a consultation program for SMEs to help them increase their innovation capacity and take their ideas to market. The consultation involved the SME filling out an onboarding form and review of this information by an expert advisory committee using guided considerations, leading to a recommendation report provided to the SME. This paper reports on the characteristics of 25 SMEs who completed the program and qualitatively analyzed their recommendation reports to identify common barriers to digital health innovation.

Results

A total of 2 central themes were identified, each with 3 subthemes. First, a common barrier to system integration was the lack of formal evaluation, with SMEs having limited resources and opportunities to conduct such an evaluation. Second, the health system’s current structure does not create incentives for clinicians to use digital technologies, which threatens the sustainability of SMEs’ business models. SMEs faced significant challenges in engaging users and payers from the public system due to perverse economic incentives. Physicians are compensated by in-person visits, which actively works against the goals of many digital health solutions of keeping patients out of clinics and hospitals.

Conclusions

There is a significant disconnect between the economic incentives that drive clinical behaviors and the use of digital technologies that would benefit patients’ well-being. To encourage the use of digital health technologies, publicly funded health systems need to dedicate funding for the evaluation of digital solutions and streamlined pathways for clinical integration.

Identifying and Overcoming Policy-Level Barriers to the Implementation of Digital Health Innovation: Qualitative Study

Background

High-level policy barriers impede widespread adoption for even the most well-positioned innovations. Most of the work in this field assumes rather than analyzes the driving forces of health innovation. Objective: The aim of this study was to explore the challenges and opportunities experienced by health system stakeholders in the implementation of digital health innovation in Ontario.

Objective

The aim of this study was to explore the challenges and opportunities experienced by health system stakeholders in the implementation of digital health innovation in Ontario.

Methods

We completed semistructured interviews with 10 members of senior leadership across key organizations that are engaged in health care–related digital health activities. Data were analyzed using qualitative description.

Results

A total of 6 key policy priorities emerged, including the need for (1) a system-level definition of innovation, (2) a clear overarching mission, and (3) clearly defined organizational roles. Operationally, there is a need to (4) standardize processes, (5) shift the emphasis to change management, and (6) align funding structures.

Conclusions

These findings emphasize the critical role of the government in developing a vision and creating the foundation upon which innovation activities will be modeled.

Linking Quality Improvement and Health Information Technology through the QI-HIT Figure 8

The implementation of health information technology (HIT) is complex. A method for mitigating complexity is incrementalism. Incrementalism forms the foundation of both incremental software development models, like agile, and the Plan-Do-Study-Act cycles (PDSAs) of quality improvement (QI), yet we often fail to be incremental at the union of the disciplines. We propose a new model for HIT implementation that explicitly links incremental software development cycles with PDSAs, the QI-HIT Figure 8 (QIHIT-F8). We then detail a subsequent local HIT implementation where we demonstrated its use. The QIHIT-F8 requires a reprioritization of project management activities around tests of change, strong QI principles to detect these changes, and the presence of both baseline and prospective data about the chosen indicators. These conditions are most likely to be present when applied to indicators of high strategic importance to an organization.

Beyond "implementation": digital health innovation and service design

Digital tools have shown great potential to enhance health services’ capacity to achieve the goals of the triple aim (enhance patient experience, improve health outcomes, and control or reduce costs), but their actual impact remains variable. In this commentary, we suggest that shifting from a perspective focused on “implementing” new digital tools in health care settings toward one focused on “service design” will help teams execute more successful digital technology adoption projects. We present value proposition design (VPD) as a service design strategy requiring that stakeholders are brutally honest in determining the value of a new digital tool for their everyday work. Incorporating a perspective focused on how the value proposition of a technology is understood by each team member, and implications for their work routines, will help project teams to better understand how services can be reinvented during technology adoption initiatives. We present the simple heuristic [Tool+Team+Routine] as a reminder of the central considerations that make up a service design initiative, and present an illustrative case scenario of designing the use of a digital care coordination platform in an actual digital technology adoption project. We conclude by outlining two important challenges that need to be addressed to advance service design approaches to technology adoption in health care.

Agile research to complement agile development: a proposal for an mHealth research lifecycle

Mobile health (mHealth) technology is increasingly being used, but academic evaluations supporting its use are not keeping pace. This is partly due to the disconnect between the traditional pharmaceutical approach to product evaluation, with its incremental approach, and the flexible way in which mHealth products are developed. An important step to addressing these problems lies in establishing agile research methods that complement the agile development methodologies used to create modern digital health applications. We describe an mHealth research model that mirrors traditional clinical research methods in its attention to safety and efficacy, while also accommodating the rapid and iterative development and evaluation required to produce effective, evidence-based, and sustainable digital products. This approach consists of a project identification stage followed by four phases of clinical evaluation: Phase 1: User Experience Design, Development, & Alpha Testing; Phase 2: Beta testing; Phase 3: Clinical Trial Evaluation; and Phase 4: Post-Market Surveillance. These phases include sample gating questions and are adapted to accommodate the unique nature of digital product development.