An introduction to methodological challenges in the evaluation of eHealth research: Perspectives from the Health e-Technologies Initiative

Planning a holistic summative eHealth evaluation in an interdisciplinary and multi-national setting: a case study and propositions for guideline development

Background

Summative eHealth evaluations frequently lack quality, which affects the generalizability of the evidence, and its use in practice and further research. To guarantee quality, a number of activities are recommended in the guidelines for evaluation planning. This study aimed to examine a case of an eHealth evaluation planning in a multi-national and interdisciplinary setting and to provide recommendations for eHealth evaluation planning guidelines.

Methods

An empirical eHealth evaluation process was developed through a case study. The empirical process was compared with selected guidelines for eHealth evaluation planning using a pattern-matching technique.

Results

Planning in the interdisciplinary and multi-national team demanded extensive negotiation and alignment to support the future use of the evidence created. The evaluation planning guidelines did not provide specific strategies for different set-ups of the evaluation teams. Further, they did not address important aspects of quality evaluation, such as feasibility analysis of the outcome measures and data collection, monitoring of data quality, and consideration of the methods and measures employed in similar evaluations.

Conclusions

Activities to prevent quality problems need to be incorporated in the guidelines for evaluation planning. Additionally, evaluators could benefit from guidance in evaluation planning related to the different set-ups of the evaluation teams.

Can-Pain-a digital intervention to optimise cancer pain control in the community: development and feasibility testing

Purpose

To develop a novel digital intervention to optimise cancer pain control in the community. This paper describes intervention development, content/rationale and initial feasibility testing.

Methods

Determinants of suboptimal cancer pain management were characterised through two systematic reviews; patient, caregiver and healthcare professional (HCP) interviews (n = 39); and two HCP focus groups (n = 12). Intervention mapping was used to translate results into theory-based content, creating the app “Can-Pain”. Patients with/without a linked caregiver, their general practitioners and community palliative care nurses were recruited to feasibility test Can-Pain over 4 weeks.

Results

Patients on strong opioids described challenges balancing pain levels with opioid intake, side effects and activities and communicating about pain management problems with HCPs. Can-Pain addresses these challenges through educational resources, contemporaneous short-acting opioid tracking and weekly patient-reported outcome monitoring. Novel aspects of Can-Pain include the use of contemporaneous breakthrough analgesic reports as a surrogate measure of pain control and measuring the level at which pain becomes bothersome to the individual.

Patients were unwell due to advanced cancer, making recruitment to feasibility testing difficult. Two patients and one caregiver used Can-Pain for 4 weeks, sharing weekly reports with four HCPs. Can-Pain highlighted unrecognised problems, promoted shared understanding about symptoms between patients and HCPs and supported shared decision-making.

Conclusions

Preliminary testing suggests that Can-Pain is feasible and could promote patient-centred pain management. We will conduct further small-scale evaluations to inform a future randomised, stepped-wedge trial.

Trial registration

Qualitative research: ClinicalTrials.gov, reference NCT02341846

Feasibility study: NIHR CPMS database ID 34172

Electronic supplementary material

The online version of this article (10.1007/s00520-020-05510-0) contains supplementary material, which is available to authorized users.

A Practical Do-It-Yourself Recruitment Framework for Concurrent eHealth Clinical Trials: Identification of Efficient and Cost-Effective Methods for Decision Making (Part 2)

Background

The ability to successfully recruit participants for electronic health (eHealth) clinical trials is largely dependent on the use of efficient and effective recruitment strategies. Determining which types of recruitment strategies to use presents a challenge for many researchers.

Objective

The aim of this study was to present an analysis of the time-efficiency and cost-effectiveness of recruitment strategies for eHealth clinical trials, and it describes a framework for cost-effective trial recruitment.

Methods

Participants were recruited for one of 5 eHealth trials of interventions for common mental health conditions. A multipronged recruitment approach was used, including digital (eg, social media and Craigslist), research registry-based, print (eg, flyers and posters on public transportation), clinic-based (eg, a general internal medicine clinic within an academic medical center and a large nonprofit health care organization), a market research recruitment firm, and traditional media strategies (eg, newspaper and television coverage in response to press releases). The time costs and fees for each recruitment method were calculated, and the participant yield on recruitment costs was calculated by dividing the number of enrolled participants by the total cost for each method.

Results

A total of 777 participants were enrolled across all trials. Digital recruitment strategies yielded the largest number of participants across the 5 clinical trials and represented 34.0% (264/777) of the total enrolled participants. Registry-based recruitment strategies were in second place by enrolling 28.0% (217/777) of the total enrolled participants across trials. Research registry-based recruitment had a relatively high conversion rate from potential participants who contacted our center for being screened to be enrolled, and it was also the most cost-effective for enrolling participants in this set of clinical trials with a total cost per person enrolled at US $8.99.

Conclusions

On the basis of these results, a framework is proposed for participant recruitment. To make decisions on initiating and maintaining different types of recruitment strategies, the resources available and requirements of the research study (or studies) need to be carefully examined.

Young People, Adult Worries: Randomized Controlled Trial and Feasibility Study of the Internet-Based Self-Support Method "Feel the ViBe" for Adolescents and Young Adults Exposed to Family Violence

BACKGROUND

Adolescents and young adults (AYAs) are of special interest in a group of children exposed to family violence (FV). Past-year prevalence of exposure to FV is known to be highest in AYAs and has severe consequences. Peer support is an effective approach to behavior change and the Internet is considered suitable as a mode of delivery.

OBJECTIVE

The study aimed to evaluate both effectiveness and feasibility of a randomized controlled trial (RCT) and feasibility study of the Internet-based self-support method "Feel the ViBe" (FtV) using mixed-methods approach to fully understand the strengths and weaknesses of a new intervention.

METHODS

AYAs aged 12-25 years and exposed to FV were randomized in an intervention group (access to FtV + usual care) and a control group (minimally enhanced usual care) after they self-registered themselves. From June 2012 to July 2014, participants completed the Impact of Event Scale (IES) and Depression (DEP) and Anxiety (ANX) subscales of the Symptom CheckList-90-R (SCL-90) every 6 weeks. The Web Evaluation Questionnaire was completed after 12 weeks. Quantitative usage data were collected using Google analytics and content management system (CMS) logs and data files. A univariate analysis of variance (UNIANOVA) and mixed model analysis (intention-to-treat [ITT], complete case) were used to compare groups. Pre-post t tests were used to find within-group effects. Feasibility measures structurally address the findings. The CONsolidated Standards Of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth (CONSORT-EHEALTH) checklist was closely followed.

RESULTS

In total, 31 out of 46 participants in the intervention group and 26 out of 47 participants in the control group started FtV. Seventeen participants (intervention: n=8, control: n=9) completed all questionnaires. Mixed model analysis showed significant differences between groups on the SCL-90 DEP (P=.04) and ANX (P=.049) subscales between 6 and 12 weeks after participation started. UNIANOVA showed no significant differences. Pre-post paired sample t tests showed significant improvements after 12 weeks for the SCL-90 DEP (P=.03) and ANX (P=.046) subscales. Reported mean Web-based time per week was 2.83 with a session time of 36 min. FtV was rated a mean 7.47 (1-10 Likert scale) with a helpfulness score of 3.16 (1-5 Likert scale). All participants felt safe. Two-thirds of the intervention participants started regular health care.

CONCLUSIONS

No changes on the IES were found. SCL-90 DEP and ANX showed promising results; however, the calculated sample size was not reached (n=18). FtV functions best as a first step for adolescents and young adults in an early stage of change. FtV can be easily implemented without extensive resources and fits best in the field of public health care or national governmental care.

TRIAL REGISTRATION

Netherlands National Trial Register (NTR): NTR3692; http://www.trialregister.nl/trialreg/admin/ rctview.asp?TC=3692 (Archived by WebCite at http://www.webcitation.org/6qIeKyjA4).

Commentary: pediatric eHealth interventions: common challenges during development, implementation, and dissemination

OBJECTIVE

To provide an overview of common challenges that pediatric eHealth researchers may encounter when planning, developing, testing, and disseminating eHealth interventions along with proposed solutions for addressing these challenges.

METHODS

The article draws on the existing eHealth literature and the authors' collective experience in pediatric eHealth research.

RESULTS AND CONCLUSIONS

The challenges associated with eHealth interventions and their proposed solutions are multifaceted and cut across a number of areas from eHealth program development through dissemination. Collaboration with a range of individuals (e.g., multidisciplinary colleagues, commercial entities, primary stakeholders) is the key to eHealth intervention success. To ensure adequate resources for design, development, and planning for sustainability, a number of public and private sources of funding are available. A study design that addresses ethical concerns and security issues is critical to ensure scientific integrity and intervention dissemination. Table I summarizes key issues to consider during eHealth intervention development, testing, and dissemination.

Internet protocol television for personalized home-based health information: design-based research on a diabetes education system

Background

The use of Internet protocol television (IPTV) as a channel for consumer health information is a relatively under-explored area of medical Internet research. IPTV may afford new opportunities for health care service providers to provide health information and for consumers, patients, and caretakers to access health information. The technologies of Web 2.0 add a new and even less explored dimension to IPTV’s potential.

Objective

Our research explored an application of Web 2.0 integrated with IPTV for personalized home-based health information in diabetes education, particularly for people with diabetes who are not strong computer and Internet users, and thus may miss out on Web-based resources. We wanted to establish whether this system could enable diabetes educators to deliver personalized health information directly to people with diabetes in their homes; and whether this system could encourage people with diabetes who make little use of Web-based health information to build their health literacy via the interface of a home television screen and remote control.

Methods

This project was undertaken as design-based research in two stages. Stage 1 comprised a feasibility study into the technical work required to integrate an existing Web 2.0 platform with an existing IPTV system, populated with content and implemented for user trials in a laboratory setting. Stage 2 comprised an evaluation of the system by consumers and providers of diabetes information.

Results

The project succeeded in developing a Web 2.0 IPTV system for people with diabetes and low literacies and their diabetes educators. The performance of the system in the laboratory setting gave them the confidence to engage seriously in thinking about the actual and potential features and benefits of a more widely-implemented system. In their feedback they pointed out a range of critical usability and usefulness issues related to Web 2.0 affordances and learning fundamentals. They also described their experiences with the system in terms that bode well for its educational potential, and they suggested many constructive improvements to the system.

Conclusions

The integration of Web 2.0 and IPTV merits further technical development, business modeling, and health services and health outcomes research, as a solution to extend the reach and scale of home-based health care.

How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research

Background

eHealth interventions appear and change so quickly that they challenge the way we conduct research. By the time a randomized trial of a new intervention is published, technological improvements and clinical discoveries may make the intervention dated and unappealing. This and the spate of health-related apps and websites may lead consumers, patients, and caregivers to use interventions that lack evidence of efficacy.

Objective

This paper aims to offer strategies for increasing the speed and usefulness of eHealth research.

Methods

The paper describes two types of strategies based on the authors’ own research and the research literature: those that improve the efficiency of eHealth research, and those that improve its quality.

Results

Efficiency strategies include: (1) think small: conduct small studies that can target discrete but significant questions and thereby speed knowledge acquisition; (2) use efficient designs: use such methods as fractional-factorial and quasi-experimental designs and surrogate endpoints, and experimentally modify and evaluate interventions and delivery systems already in use; (3) study universals: focus on timeless behavioral, psychological, and cognitive principles and systems; (4) anticipate the next big thing: listen to voices outside normal practice and connect different perspectives for new insights; (5) improve information delivery systems: researchers should apply their communications expertise to enhance inter-researcher communication, which could synergistically accelerate progress and capitalize upon the availability of “big data”; and (6) develop models, including mediators and moderators: valid models are remarkably generative, and tests of moderation and mediation should elucidate boundary conditions of effects and treatment mechanisms. Quality strategies include: (1) continuous quality improvement: researchers need to borrow engineering practices such as the continuous enhancement of interventions to incorporate clinical and technological progress; (2) help consumers identify quality: consumers, clinicians, and others all need to easily identify quality, suggesting the need to efficiently and publicly index intervention quality; (3) reduce the costs of care: concern with health care costs can drive intervention adoption and use and lead to novel intervention effects (eg, reduced falls in the elderly); and (4) deeply understand users: a rigorous evaluation of the consumer’s needs is a key starting point for intervention development.

Conclusions

The challenges of distinguishing and distributing scientifically validated interventions are formidable. The strategies described are meant to spur discussion and further thinking, which are important, given the potential of eHealth interventions to help patients and families.

eHealth: extending, enhancing, and evolving health care

eHealth holds the promise of revolutionizing health care by improving its efficiency; extending and enhancing its reach; energizing and engaging its practitioners and their patients; and in the process, democratizing, decentralizing, and even partially demystifying the practice of medicine. In emerging and developing countries, the use of eHealth and smart health-care planning has the potential to expand access to necessary treatments and prevention services that can serve as underpinnings of rapid economic development. In developed countries, the application of eHealth promises to restructure the business model of health-care delivery, while at the same time improving and personalizing the quality of care received. This article reviews the past, present, and future of eHealth in an effort to illuminate the potential of its impact.

Promise of and potential for patient-facing technologies to enable meaningful use

Patients are using healthcare technologies for a variety of reasons. Recently, the Meaningful-Use rule was released by the Centers for Medicare and Medicaid Services, providing some initial guidance for patient-facing technologies. There needs to be more of an understanding of patients' needs and how these technologies can be utilized effectively. This article provides a framework for organizing patient-facing technologies into categories of meaningful use, and how these technologies can improve healthcare quality, safety, and population health. Barriers to achieving meaningful use of HIT and unintended consequences of patient-facing technologies are discussed. The success of healthcare reform is predicated on achieving improved health outcomes and reduced costs, which can be accomplished only by activating patients to become more engaged in their own care. Patient-facing technologies are likely to play a critical role in supporting patients to become more informed and activated and may also improve efficiencies. Further research is needed to identify the most useful and effective technologies for patients.

Nudging best practice: the HITECH act and behavioral medicine

In February 2009, the US Congress passed the Health Information Technology for Economic and Consumer Health (HITECH) Act in order to stimulate the "meaningful use" of health information technology within medical practice. Economists have noted that other sectors in the economy have demonstrated substantive productivity improvements from investments in information technology but that the health sector lags behind. The "meaningful use" stipulation of the HITECH Act focuses systems redesign within the health sector on user's behavior, a provision that opens a window of contribution from specialists in behavioral medicine. There are several ways for behavioral medicine to become involved in the redesign. One is to help craft a health services environment that optimizes communication between providers and patients, between primary care and specialist care providers, and between patients and their caregivers. Another is to help practitioners and policy-makers create new "decisional architectures" for "nudging" behavior in positive ways through better incentives, understandable instructions, healthy defaults, instructive feedback, back-ups for error, and structured decision-making. New funding opportunities in research, implementation, and training may facilitate the involvement of behavioral medicine-an involvement that is crucial for ensuring the success of reform efforts in the long run.