STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration

Interactive Toolkit for Classifying Digital Health Interventions, Services, and Applications Based on the WHO Framework

The rapidly advancing digital health requires a standardized approach to classifying Digital Health Interventions (DHIs) for better planning, monitoring, and resource distribution. The World Health Organisation (WHO) developed a Classification for Digital Health Interventions, Services, and Applications in Health (CDISAH) in response to this need. The purpose of this study was to develop an interactive toolkit based on WHO's CDISAH to enhance categorization, making it more interactive, user-friendly, and effective in classifying DHI services and applications, and demonstrate its practical implementation in the field of cardiac rehabilitation. We used a descriptive approach with a seven-step iterative process to create the toolkit. The process began with a review of best practices for converting framework into toolkit, followed by drafting an initial toolkit structure, which was refined through team discussions. The content was based on WHO CDISAH. Expert feedback was incorporated, and quality assurance was conducted through internal and external reviews. The toolkit's functionality and usability were evaluated through a use case including DHIs, services, and applications for cardiac rehabilitation. The toolkit for WHO CDISAH has a structured interface with clear definitions, practical examples, and intuitive navigation across three main axes: health system challenges, digital health interventions, and digital health applications and services. Pilot testing improved its usability and functionality for accurate classification, highlighting areas for refinement and identifying challenges and solutions for practical implementation. The developed toolkit provides a standardised, portable platform for classifying the multimodal DHIs that align with the framework presented by WHO.

LogMyScan: A pilot evaluation study of a mobile phone-based ultrasound logbook application

Introduction

We developed a mobile phone logbook application called LogMyScan. The study objectives were to evaluate a number of domains as per the technology acceptance model.

Methods

A prospective pilot cohort study was conducted using a version of LogMyScan and reported according to Statement on Reporting of Evaluation studies in Health Informatics. Prospective participants were directed to a website which provided information and links. Participants were recruited via email distribution lists and social media. A task list was provided, which included entering simulated scan data and navigating the application, prior to completing a questionnaire. Questions were aligned to technology acceptance model domains, and Cronbach's alpha was calculated to assess reliability.

Results

There were 49 participants and 7 were unable to complete the image upload and editing step. Levels of agreement for usability, educational and governance/security aspects were generally good. The desirability of existing and proposed features was high for images and video clip handling, feedback and reflection and less for motivational notifications and benchmarking. Comments identified some limitations and issues on the Android platform. Cronbach's alpha demonstrated good to excellent reliability: 0.87 (95% confidence interval, 0.79-0.91) for usability, educational and governance/security and 0.86 (95% confidence interval, 0.78-0.91) for desirability of existing and proposed features.

Conclusion

There was generally high perceived ease of use, usefulness, attitude towards use and behavioural intention to use LogMyScan. Further research is needed to evaluate LogMyScan for future iterations, allowing recruitment of a larger number of participants and evaluating user expectations prior to using, and at timed intervals after using.

Assessing the Usage and Usability of a Mental Health Advice Telephone Service in Uganda: Mixed Methods Study

BACKGROUND

Harnessing mobile health (mHealth) solutions could improve the delivery of mental health services and mitigate their impact in Uganda and similar low-resource settings. However, successful adoption requires that mHealth solutions have good usability. We have previously implemented a telephone service to provide mental health information and advice in English and Luganda, utilizing an automated interactive voice response (IVR) system linked to live agents, including mental health care workers and peer support workers.

OBJECTIVE

This study aims to assess the usage and usability of this mental health telephone service.

METHODS

We obtained usage data from the system's call logs over 18 months to study call volumes and trends. We then surveyed callers to gather their characteristics and assess usability using the Telehealth Usability Questionnaire. Additionally, call recordings were evaluated for conversation quality by 3 independent health care professionals, using the Telephone Nursing Dialogue Process, and correlations between quality and usability aspects were investigated.

RESULTS

Over 18 months, the system received 2863 meaningful calls (ie, calls that went past the welcome message) from 1125 unique telephone numbers. Of these, 1153 calls (40.27%) stopped at the prerecorded IVR information, while 1710 calls (59.73%) opted to speak to an agent. Among those who chose to speak with an agent, 1292 calls (75.56%) were answered, 393 calls (22.98%) went to voicemail and were returned in the following working days, and 25 calls (1.46%) were not answered. Usage was generally sustained over time, with spikes in call volume corresponding to marketing events. The survey (n=240) revealed that most callers were caregivers of patients with mental health issues (n=144, 60.0%) or members of the general public (n=46, 19.2%), while a few were patients with mental health issues (n=44, 18.3%). Additionally, the majority were male (n=143, 59.6%), spoke English (n=180, 75.0%), had postsecondary education (n=164, 68.3%), lived within 1 hour or less from Butabika Hospital (n=187, 77.9%), and were aged 25-44 years (n=160, 66.7%). The overall usability score for the system was 4.12 on a 5-point scale, significantly higher than the recommended target usability score of 4 (P=.006). The mean scores for usability components ranged from 3.66 for reliability to 4.41 for ease of use, with all components, except reliability, scoring higher than 4 or falling within its CI. Usability scores were higher for Luganda speakers compared with English speakers, but there was no association with other participant characteristics such as sex, distance from the hospital, age, marital status, duration of symptoms, or treatment status. The quality of call conversations (n=50) was rated at 4.35 out of 5 and showed a significant correlation with usability (Pearson r=0.34, P=.02).

CONCLUSIONS

We found sustained usage of the mental health telephone service, along with a positive user experience and high satisfaction across various user characteristics. mHealth solutions like this should be embraced and replicated to enhance the delivery of health services in Uganda and similar low-resource settings.

Usability, Engagement, and Report Usefulness of Chatbot-Based Family Health History Data Collection: Mixed Methods Analysis

BACKGROUND

Family health history (FHx) is an important predictor of a person's genetic risk but is not collected by many adults in the United States.

OBJECTIVE

This study aims to test and compare the usability, engagement, and report usefulness of 2 web-based methods to collect FHx.

METHODS

This mixed methods study compared FHx data collection using a flow-based chatbot (KIT; the curious interactive test) and a form-based method. KIT's design was optimized to reduce user burden. We recruited and randomized individuals from 2 crowdsourced platforms to 1 of the 2 FHx methods. All participants were asked to complete a questionnaire to assess the method's usability, the usefulness of a report summarizing their experience, user-desired chatbot enhancements, and general user experience. Engagement was studied using log data collected by the methods. We used qualitative findings from analyzing free-text comments to supplement the primary quantitative results.

RESULTS

Participants randomized to KIT reported higher usability than those randomized to the form, with a mean System Usability Scale score of 80.2 versus 61.9 (P<.001), respectively. The engagement analysis reflected design differences in the onboarding process. KIT users spent less time entering FHx information and reported more conditions than form users (mean 5.90 vs 7.97 min; P=.04; and mean 7.8 vs 10.1 conditions; P=.04). Both KIT and form users somewhat agreed that the report was useful (Likert scale ratings of 4.08 and 4.29, respectively). Among desired enhancements, personalization was the highest-rated feature (188/205, 91.7% rated medium- to high-priority). Qualitative analyses revealed positive and negative characteristics of both KIT and the form-based method. Among respondents randomized to KIT, most indicated it was easy to use and navigate and that they could respond to and understand user prompts. Negative comments addressed KIT's personality, conversational pace, and ability to manage errors. For KIT and form respondents, qualitative results revealed common themes, including a desire for more information about conditions and a mutual appreciation for the multiple-choice button response format. Respondents also said they wanted to report health information beyond KIT's prompts (eg, personal health history) and for KIT to provide more personalized responses.

CONCLUSIONS

We showed that KIT provided a usable way to collect FHx. We also identified design considerations to improve chatbot-based FHx data collection: First, the final report summarizing the FHx collection experience should be enhanced to provide more value for patients. Second, the onboarding chatbot prompt may impact data quality and should be carefully considered. Finally, we highlighted several areas that could be improved by moving from a flow-based chatbot to a large language model implementation strategy.

Advances in artificial intelligence for drug delivery and development: A comprehensive review

Artificial intelligence (AI) has emerged as a powerful tool to revolutionize the healthcare sector, including drug delivery and development. This review explores the current and future applications of AI in the pharmaceutical industry, focusing on drug delivery and development. It covers various aspects such as smart drug delivery networks, sensors, drug repurposing, statistical modeling, and simulation of biotechnological and biological systems. The integration of AI with nanotechnologies and nanomedicines is also examined. AI offers significant advancements in drug discovery by efficiently identifying compounds, validating drug targets, streamlining drug structures, and prioritizing response templates. Techniques like data mining, multitask learning, and high-throughput screening contribute to better drug discovery and development innovations. The review discusses AI applications in drug formulation and delivery, clinical trials, drug safety, and pharmacovigilance. It addresses regulatory considerations and challenges associated with AI in pharmaceuticals, including privacy, data security, and interpretability of AI models. The review concludes with future perspectives, highlighting emerging trends, addressing limitations and biases in AI models, and emphasizing the importance of collaboration and knowledge sharing. It provides a comprehensive overview of AI's potential to transform the pharmaceutical industry and improve patient care while identifying further research and development areas.

Evaluating health information systems-related errors using the human, organization, process, technology-fit (HOPT-fit) framework

Complex socio-technical health information systems (HIS) issues can create new error risks. Therefore, we evaluated the management of HIS-related errors using the proposed human, organization, process, and technology-fit framework to identify the lessons learned. Qualitative case study methodology through observation, interview, and document analysis was conducted at a 1000-bed Japanese specialist teaching hospital. Effective management of HIS-related errors was attributable to many socio-technical factors including continuous improvement, safety culture, strong management and leadership, effective communication, preventive and corrective mechanisms, an incident reporting system, and closed feedback loops. Enablers of medication errors include system sophistication and process factors like workarounds, variance, clinical workload, slips and mistakes, and miscommunication. The case management effectiveness in handling the HIS-related errors can guide other clinical settings. The potential of HIS to minimize errors can be achieved through continual, systematic, and structured evaluation. The case study validated the applicability of the proposed evaluation framework that can be applied flexibly according to study contexts to inform HIS stakeholders in decision-making. The comprehensive and specific measures of the proposed framework and approach can be a useful guide for evaluating complex HIS-related errors. Leaner and fitter socio-technical components of HIS can yield safer system use.

The Effect of an Electronic Medical Record-Based Clinical Decision Support System on Adherence to Clinical Protocols in Inflammatory Bowel Disease Care: Interrupted Time Series Study

Background

Clinical decision support systems (CDSSs) embedded in electronic medical records (EMRs), also called electronic health records, have the potential to improve the adoption of clinical guidelines. The University of Alberta Inflammatory Bowel Disease (IBD) Group developed a CDSS for patients with IBD who might be experiencing disease flare and deployed it within a clinical information system in 2 continuous time periods.

Objective

This study aims to evaluate the impact of the IBD CDSS on the adherence of health care providers (ie, physicians and nurses) to institutionally agreed clinical management protocols.

Methods

A 2-period interrupted time series (ITS) design, comparing adherence to a clinical flare management protocol during outpatient visits before and after the CDSS implementation, was used. Each interruption was initiated with user training and a memo with instructions for use. A group of 7 physicians, 1 nurse practitioner, and 4 nurses were invited to use the CDSS. In total, 31,726 flare encounters were extracted from the clinical information system database, and 9217 of them were manually screened for inclusion. Each data point in the ITS analysis corresponded to 1 month of individual patient encounters, with a total of 18 months of data (9 before and 9 after interruption) for each period. The study was designed in accordance with the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) guidelines for health informatics evaluations.

Results

Following manual screening, 623 flare encounters were confirmed and designated for ITS analysis. The CDSS was activated in 198 of 623 encounters, most commonly in cases where the primary visit reason was a suspected IBD flare. In Implementation Period 1, before-and-after analysis demonstrates an increase in documentation of clinical scores from 3.5% to 24.1% (P<.001), with a statistically significant level change in ITS analysis (P=.03). In Implementation Period 2, the before-and-after analysis showed further increases in the ordering of acute disease flare lab tests (47.6% to 65.8%; P<.001), including the biomarker fecal calprotectin (27.9% to 37.3%; P=.03) and stool culture testing (54.6% to 66.9%; P=.005); the latter is a test used to distinguish a flare from an infectious disease. There were no significant slope or level changes in ITS analyses in Implementation Period 2. The overall provider adoption rate was moderate at approximately 25%, with greater adoption by nurse providers (used in 30.5% of flare encounters) compared to physicians (used in 6.7% of flare encounters).

Conclusions

This is one of the first studies to investigate the implementation of a CDSS for IBD, designed with a leading EMR software (Epic Systems), providing initial evidence of an improvement over routine care. Several areas for future research were identified, notably the effect of CDSSs on outcomes and how to design a CDSS with greater utility for physicians. CDSSs for IBD should also be evaluated on a larger scale; this can be facilitated by regional and national centralized EMR systems.

Pilot Study of a Software Application to Identify Trauma Registry Inconsistencies

BACKGROUND

Trauma registries are essential to the functioning of modern trauma centers, and high-quality data are necessary to identify patient care issues, develop evidence-based practice, and more. However, institutional experience suggested existing methods to evaluate data quality were insufficient.

OBJECTIVE

This study aims to compare a new software application developed at our trauma center to our existing trauma registry platform on the ability to identify registry inconsistencies (i.e., potential data quality issues).

METHODS

We conducted a pilot retrospective cohort study of patients from September 2019 to August 2020 who underwent chart review during a Level I verification visit and had been audited several times for accuracy. Registry records were processed by both validation systems, and registry inconsistencies were recorded.

RESULTS

In registry data for 63 patients, the new software found 225 registry inconsistencies, and the registry systems found 153 inconsistencies. The most frequent inconsistencies identified by the new software were missing or unknown procedure start times, with 18/63 (28.6%) patients affected and prehospital supplemental oxygen being blank, with 29/53 (54.7%) patients with prehospital care affected. None of the 10 most common inconsistencies detected with the registry systems were true issues.

CONCLUSIONS

This study found the new software application identified 47% more inconsistencies than the standard registry systems, and none of the most frequent inconsistencies detected with the registry systems were true issues pertinent to institutional practice. Centers should consider additional methods to identify registry inconsistencies as existing processes appear insufficient.

Capturing the pulse: a state-of-the-art review on camera-based jugular vein assessment

Heart failure is associated with a rehospitalisation rate of up to 50% within six months. Elevated central venous pressure may serve as an early warning sign. While invasive procedures are used to measure central venous pressure for guiding treatment in hospital, this becomes impractical upon discharge. A non-invasive estimation technique exists, where the clinician visually inspects the pulsation of the jugular veins in the neck, but it is less reliable due to human limitations. Video and signal processing technologies may offer a high-fidelity alternative. This state-of-the-art review analyses existing literature on camera-based methods for jugular vein assessment. We summarize key design considerations and suggest avenues for future research. Our review highlights the neck as a rich imaging target beyond the jugular veins, capturing comprehensive cardiac signals, and outlines factors affecting signal quality and measurement accuracy. Addressing an often quoted limitation in the field, we also propose minimum reporting standards for future studies.

Development and Evaluation of Health Recommender Systems: Systematic Scoping Review and Evidence Mapping

BACKGROUND

Health recommender systems (HRSs) are information retrieval systems that provide users with relevant items according to the users' needs, which can motivate and engage users to change their behavior.

OBJECTIVE

This study aimed to identify the development and evaluation of HRSs and create an evidence map.

METHODS

A total of 6 databases were searched to identify HRSs reported in studies from inception up to June 30, 2022, followed by forward citation and grey literature searches. Titles, abstracts, and full texts were screened independently by 2 reviewers, with discrepancies resolved by a third reviewer, when necessary. Data extraction was performed by one reviewer and checked by a second reviewer. This review was conducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) statement.

RESULTS

A total of 51 studies were included for data extraction. Recommender systems were used across different health domains, such as general health promotion, lifestyle, and generic health service. A total of 23 studies had reported the use of a combination of recommender techniques, classified as hybrid recommender systems, which are the most commonly used recommender techniques in HRSs. In the HRS design stage, only 10 of 51 (19.6%) recommender systems considered personal preferences of end users in the design or development of the system; a total of 29 studies reported the user interface of HRSs, and most HRSs worked on users' mobile interfaces, usually a mobile app. Two categories of HRS evaluations were used, and evaluations of HRSs varied greatly; 62.7% (32/51) of the studies used the offline evaluations using computational methods (no user), and 33.3% (17/51) of the studies included end users in their HRS evaluation.

CONCLUSIONS

Through this scoping review, nonmedical professionals and policy makers can visualize and better understand HRSs for future studies. The health care professionals and the end users should be encouraged to participate in the future design and development of HRSs to optimize their utility and successful implementation. Detailed evaluations of HRSs in a user-centered approach are needed in future studies.

Moderated digital social therapy for young people with emerging mental health problems: A user-centered mixed-method design and usability study

Introduction

Over 25% of Dutch young people are psychologically unhealthy. Individual and societal consequences that follow from having mental health complaints at this age are substantial. Young people need care which is often unavailable. ENgage YOung people earlY (ENYOY) is a moderated digital social therapy-platform that aims to help youngsters with emerging mental health complaints. Comprehensive research is being conducted into the effects and to optimize and implement the ENYOY-platform throughout the Netherlands. The aim of this study is to explore the usability and user experience of the ENYOY-platform.

Methods

A user-centered mixed-method design was chosen. 26 young people aged 16-25 with emerging mental health complaints participated. Semi-structured interviews were conducted to explore usability, user-friendliness, impact, accessibility, inclusivity, and connection (Phase 1). Phase 2 assessed usability problems using the concurrent and retrospective Think Aloud-method. User experience and perceived helpfulness were assessed using a 10-point rating scale and semi-structured interviews (Phase 3). The Health Information Technology Usability Evaluation Scale (Health-ITUES; Phase 1) and System Usability Scale (SUS; Phase 2 and 3) were administered. Qualitative data was analyzed using thematic analysis. Task completion rate and time were tracked and usability problems were categorized using the Nielsen's rating scale (Phase 2).

Results

Adequate to high usability was found (Phase 1 Health-ITUES 4.0(0.34); Phase 2 SUS 69,5(13,70); Phase 3 SUS 71,6(5,63)). Findings from Phase 1 (N = 10) indicated that users viewed ENYOY as a user-friendly, safe, accessible, and inclusive initiative which helped them reduce their mental health complaints and improve quality of life. Phase 2 (N = 10) uncovered 18 usability problems of which 5 of major severity (e.g. troubles accessing the platform). Findings from Phase 3 (N = 6) suggested that users perceived the coaching calls the most helpful [9(0.71)] followed by the therapy content [6.25(1.41)]. Users liked the social networking aspect but rated it least helpful [6(2.1)] due to inactivity.

Conclusion

The ENYOY-platform has been found to have adequate to high usability and positive user experiences were reported. All findings will be transferred to the developmental team to improve the platform. Other evaluation methods and paring these with quantitative outcomes could provide additional insight in future research.

Usability, Feasibility, and Effect of a Biocueing Intervention in Addition to a Moderated Digital Social Therapy-Platform in Young People With Emerging Mental Health Problems: A Mixed-Method Approach

Introduction

To optimize treatment, it is of utmost importance to take into account the myriad of biological, social, and psychological changes that young people go through during adolescence which make them more vulnerable for developing mental health problems. Biocueing, a non-invasive method to transform physiological parameters into an observable signal, could strengthen stress- and emotion regulation by cueing physiologically unusual values in daily life. The aim of this study is to investigate the usability, feasibility, and exploratory effect of biocueing in addition to ENgage YOung people earlY (ENYOY), a moderated digital social therapy-platform, in young people with emerging mental health complaints.

Methods

A user-centered mixed-method design was used. A focus group was conducted to optimize the ENYOY-platform and biocueing intervention. Biocueing was operationalized by a smartwatch and the Sense-IT app. A within-subjects design was used; 10 days for all participants 'biofeedback off' (control), followed by 10 days 'biofeedback on' (experimental). Emotional awareness and perceived stress were measured using ecological momentary assessment. Eight individuals participated. User-friendliness, usability, and acceptance were assessed using a qualitative design.

Results

Findings from the focus group resulted in several adaptations of the biocueing intervention to the ENYOY-platform and vice versa. The average measurement compliance rate was 78.8%. Level-one findings showed different individual effects on perceived stress and emotional awareness. Level-two analyses showed no overall effects on perceived stress (B = -0.020, p = 0.562) and overall positive effects on emotional awareness (B = 0.030, p = 0.048) with small effect sizes (Improvement Rate Difference = 0.05-0.35). The intervention was found to be acceptable and showed moderate usability. Participants indicated they experienced improvements in reflection on feelings and changes in behavior, such as pausing and evaluating the situation.

Conclusion

These preliminary results show that biocueing could be a promising addition to digital treatment platforms and help young people become more emotionally aware. Improvements should be made regarding the usability and acceptability of the smartwatch, as well as more extensive integration of the biocueing intervention with a digital treatment platform. It would be relevant to gain a better understanding of which individuals would benefit most from an additional biocueing intervention.

Replication studies in the clinical decision support literature-frequency, fidelity, and impact

OBJECTIVE

To assess the frequency, fidelity, and impact of replication studies in the clinical decision support system (CDSS) literature.

MATERIALS AND METHODS

A PRISMA-compliant review identified CDSS replications across 28 health and biomedical informatics journals. Included articles were assessed for fidelity to the original study using 5 categories: Identical, Substitutable, In-class, Augmented, and Out-of-class; and 7 IMPISCO domains: Investigators (I), Method (M), Population (P), Intervention (I), Setting (S), Comparator (C), and Outcome (O). A fidelity score and heat map were generated using the ratings.

RESULTS

From 4063 publications matching search criteria for CDSS research, only 12/4063 (0.3%) were ultimately identified as replications. Six articles replicated but could not reproduce the results of the Han et al (2005) CPOE study showing mortality increase and, over time, changed from truth testing to generalizing this result. Other replications successfully tested variants of CDSS technology (2/12) or validated measurement instruments (4/12).

DISCUSSION

A replication rate of 3 in a thousand studies is low even by the low rates in other disciplines. Several new reporting methods were developed for this study, including the IMPISCO framework, fidelity scores, and fidelity heat maps. A reporting structure for clearly identifying replication research is also proposed.

CONCLUSION

There is an urgent need to better characterize which core CDSS principles require replication, identify past replication data, and conduct missing replication studies. Attention to replication should improve the efficiency and effectiveness of CDSS research and avoiding potentially harmful trial and error technology deployment.

Mobile Health for Smoking Cessation Among Disadvantaged Young Women During and After Pregnancy: User-Centered Design and Usability Study

Background

Smoking prevalence during and after pregnancy remains high among socioeconomically disadvantaged women. Mobile health (mHealth) apps with game and social support elements seem promising to support smoking cessation.

Objective

This study aims to describe the user-centered design and usability evaluation of Kindle, an mHealth app with game and social support elements, to support disadvantaged young women during and after pregnancy through the first stages of smoking cessation.

Methods

Disadvantaged women (n=9), members of their social networks (n=4), and nurses supporting these women (n=51) were informants throughout the iterative prototype development of Kindle according to the International Organization for Standardization 9241-11:2018. Specific phases included understanding the context of use through secondary analysis of qualitative interview data (phase 1), establishing the user and organizational requirements (phase 2), production of design solutions (phase 3), and usability inspection of the prototype through a heuristic evaluation (3 experts) along with user testing by a think aloud method (5 disadvantaged women and 5 nurses; phase 4). Usability problems were categorized according to the principles of the Healthcare Information and Management Systems Society.

Results

Phase 1 resulted in an understanding of the VoorZorg program and the needs of VoorZorg nurses and clients (eg, focus on early stages of change and building new supportive networks to aid clients in smoking cessation). In phase 2, we established requirements (n=22; eg, mHealth app, secure communication between nurses and clients, easy-to-use interfaces, inclusion of game elements, and tailoring at early stages of change in smoking cessation). Phase 3 resulted in a prototype of Kindle, combining the interface for nurses and clients, including the following functionalities: personal goal setting with earning points; secured chat function between nurses and other clients; and tips, diary, and profile creation. The heuristic evaluation and thinking aloud method in phase 4 revealed 78 usability problems in the interfaces. Most usability problems concerned simplicity (eg, unclear clickable button) and naturalness (eg, unclear icon).

Conclusions

The user-centered design and usability testing of the mHealth app Kindle yielded useful insights. The involvement of end users, specifically socioeconomically disadvantaged women during and after their pregnancy, resulted in a prototype that met their needs and requirements (eg, mHealth app, secure communication between nurses and clients, easy-to-use interfaces, inclusion of game elements, and tailoring to the early stages of change in smoking cessation) to achieve readiness for smoking cessation. Moreover, the usability evaluation by end users and experts revealed unique usability problems for this population. These insights allow for further optimization of Kindle and encourage future studies to engage disadvantaged populations in all phases of mHealth intervention design and usability testing.

Electronic Tools to Bridge the Language Gap in Health Care for People Who Have Migrated: Systematic Review

BACKGROUND

People who have migrated or with a language barrier may face significant hurdles in accessing health care. Some apps have been specifically developed to facilitate the dialogue between health care professionals and people who have migrated who have low-level language proficiency or to promote health among people who have migrated.

OBJECTIVE

We conducted a systematic review to investigate development, acceptability, and effectiveness of these types of apps.

METHODS

We conducted a search of PubMed, Scopus, and Embase databases. We included all study designs (qualitative, quantitative, mixed) reporting development, evaluation of efficacy, or acceptability of apps facilitating dialogue with a health professional or promoting health for people who have migrated, minorities, or tourists with a language barrier, using any outcome. Two researchers selected the studies independently. We collected general information about the app, information about health literacy and cultural adaptation, information about the development of the app, evidence on acceptability or efficacy, and information on app use. Data were collected by 2 researchers independently and results were reviewed to verify agreement and reported according to PRISMA (Preferred Reporting Items for Systematic Review and Meta-analysis).

RESULTS

Positive results for translation apps included better communication, but with possible limitations, and reduced consultation time. Positive results for health promotion apps included improved quality of life and better management of chronic illnesses.

CONCLUSIONS

Overall, the apps had good levels of acceptability, though only half had their efficacy evaluated. In those evaluations, the endpoints were mostly related to reported behavior change and knowledge improvement, which is common for evaluations of health promotion programs. In the future, as more health apps are created, it is essential that apps that claim to have a public health objective undergo a rigorous evaluation of their acceptability, efficacy, and actual use. Indicators of outcomes beyond changes in behavior and knowledge should be reported; change in health status or access to care should also be reported. This systematic review has helped us note the characteristics associated with improved acceptability and efficacy, which can be helpful for the development of future apps.

The impact of health information technology on prescribing errors in hospitals: a systematic review and behaviour change technique analysis

BACKGROUND

Health information technology (HIT) is known to reduce prescribing errors but may also cause new types of technology-generated errors (TGE) related to data entry, duplicate prescribing, and prescriber alert fatigue. It is unclear which component behaviour change techniques (BCTs) contribute to the effectiveness of prescribing HIT implementations and optimisation. This study aimed to (i) quantitatively assess the HIT that reduces prescribing errors in hospitals and (ii) identify the BCTs associated with effective interventions.

METHODS

Articles were identified using CINAHL, EMBASE, MEDLINE, and Web of Science to May 2020. Eligible studies compared prescribing HIT with paper-order entry and examined prescribing error rates. Studies were excluded if prescribing error rates could not be extracted, if HIT use was non-compulsory or designed for one class of medication. The Newcastle-Ottawa scale was used to assess study quality. The review was reported in accordance with the PRISMA and SWiM guidelines. Odds ratios (OR) with 95% confidence intervals (CI) were calculated across the studies. Descriptive statistics were used to summarise effect estimates. Two researchers examined studies for BCTs using a validated taxonomy. Effectiveness ratios (ER) were used to determine the potential impact of individual BCTs.

RESULTS

Thirty-five studies of variable risk of bias and limited intervention reporting were included. TGE were identified in 31 studies. Compared with paper-order entry, prescribing HIT of varying sophistication was associated with decreased rates of prescribing errors (median OR 0.24, IQR 0.03-0.57). Ten BCTs were present in at least two successful interventions and may be effective components of prescribing HIT implementation and optimisation including prescriber involvement in system design, clinical colleagues as trainers, modification of HIT in response to feedback, direct observation of prescriber workflow, monitoring of electronic orders to detect errors, and system alerts that prompt the prescriber.

CONCLUSIONS

Prescribing HIT is associated with a reduction in prescribing errors in a variety of hospital settings. Poor reporting of intervention delivery and content limited the BCT analysis. More detailed reporting may have identified additional effective intervention components. Effective BCTs may be considered in the design and development of prescribing HIT and in the reporting and evaluation of future studies in this area.

Students' perceptions on their use of an EHR: pilot questionnaire study

Introduction

Many clinical education programmes have not incorporated the use of the electronic health record (EHR) into their curriculum. It is important to incorporate technologies that will be used in real-world settings to better prepare students for clinical practice.

Objectives

To undertake a review of literature to identify a training evaluation framework; to conduct a self-completion survey, pretraining and post-training, to determine students’ perceptions on the benefit of using EHR training system.

Setting

Nursing School, University, North West England, UK; University Ethic Committee Approval Received.

Participants

Registered nurses undertaking a validated return to practice course; 24 participants for the first cohort who completed pretraining questionnaire and 23 for the second post-training cohort.

Results

The statistical results show that the students perceived that the training improved their capability in employing digital systems with statistically significant difference in the assessed preproficiency and post proficiency in the use of digital clinical systems (premedians and post medians are 2 and 5 on 10-point Likert scale, p=0.041). There was also an indication of an improvement in the knowledge of EHR systems although not statistically significant. Most students perceived it increased their knowledge on digital systems.

Conclusion

Students perceived an increase in proficiency with the EHR. There was evidence of improvement in confidence in the use of the EHR, but this confidence would be enhanced by additional use of the system. Some desire to increase confidence further and to develop knowledge of digital systems was expressed.

Recommendations for the Conduct and Reporting of Research Involving Flexible Electronic Health Record-Based Interventions

In the past 2 decades, the United States has seen widespread adoption of electronic health records (EHRs) and a transition from mostly locally developed EHRs to commercial systems. However, most research on quality improvement and safety interventions in EHRs is still conducted at a single site, in a single EHR. Although single-site studies are important early in the innovation lifecycle, multisite studies of EHR interventions are critical for generalizability. Because EHR software, configuration, and local context differ considerably across health care organizations, it can be difficult to implement a single, standardized intervention across multiple sites in a study. This article outlines key strengths, weaknesses, challenges, and opportunities for standardization of EHR interventions in multisite studies and describes flexible trial designs suitable for studying complex interventions, including EHR interventions. It also outlines key considerations for reporting on flexible trials of EHR interventions, including sharing details of the process for designing interventions and their content, details of outcomes being studied and approaches for pooling, and the importance of sharing code and configuration whenever possible.

Randomized Controlled Trials of Electronic Health Record Interventions: Design, Conduct, and Reporting Considerations

Electronic health record (EHR) systems can be configured to deliver novel EHR interventions that influence clinical decision making and to support efficient randomized controlled trials (RCTs) designed to evaluate the effectiveness, safety, and costs of those interventions. In designing RCTs of EHR interventions, one should carefully consider the unit of randomization (for example, patient, encounter, clinician, or clinical unit), balancing concerns about contamination of an intervention across randomization units within clusters (for example, patients within clinical units) against the superior control of measured and unmeasured confounders that comes with randomizing a larger number of units. One should also consider whether the key computational assessment components of the EHR intervention, such as a predictive algorithm used to target a subgroup for decision support, should occur before randomization (so that only 1 subgroup is randomized) or after randomization (including all subgroups). When these components are applied after randomization, one must consider expected heterogeneity in the effect of the differential decision support across subgroups, which has implications for overall impact potential, analytic approach, and sample size planning. Trials of EHR interventions should be reviewed by an institutional review board, but may not require patient-level informed consent when the interventions being tested can be considered minimal risk or quality improvement, and when clinical decision making is supported, rather than controlled, by an EHR intervention. Data and safety monitoring for RCTs of EHR interventions should be conducted to guide institutional pragmatic decision making about implementation and ensure that continuing randomization remains justified. Reporting should follow the CONSORT (Consolidated Standards of Reporting Trials) Statement, with extensions for pragmatic trials and cluster RCTs when applicable, and should include detailed materials to enhance reproducibility.

Safety concerns with consumer-facing mobile health applications and their consequences: a scoping review

OBJECTIVE

To summarize the research literature about safety concerns with consumer-facing health apps and their consequences.

MATERIALS AND METHODS

We searched bibliographic databases including PubMed, Web of Science, Scopus, and Cochrane libraries from January 2013 to May 2019 for articles about health apps. Descriptive information about safety concerns and consequences were extracted and classified into natural categories. The review was conducted in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) statement.

RESULTS

Of the 74 studies identified, the majority were reviews of a single or a group of similar apps (n = 66, 89%), nearly half related to disease management (n = 34, 46%). A total of 80 safety concerns were identified, 67 related to the quality of information presented including incorrect or incomplete information, variation in content, and incorrect or inappropriate response to consumer needs. The remaining 13 related to app functionality including gaps in features, lack of validation for user input, delayed processing, failure to respond to health dangers, and faulty alarms. Of the 52 reports of actual or potential consequences, 5 had potential for patient harm. We also identified 66 reports about gaps in app development, including the lack of expert involvement, poor evidence base, and poor validation.

CONCLUSIONS

Safety of apps is an emerging public health issue. The available evidence shows that apps pose clinical risks to consumers. Involvement of consumers, regulators, and healthcare professionals in development and testing can improve quality. Additionally, mandatory reporting of safety concerns is needed to improve outcomes.

A web application to involve patients in the medication reconciliation process: a user-centered usability and usefulness study

Objective

Medication reconciliation (MedRec) can improve patient safety by resolving medication discrepancies. Because information technology (IT) and patient engagement are promising approaches to optimizing MedRec, the SEAMPAT project aims to develop a MedRec IT platform based on two applications: the "patient app" and the "MedRec app." This study evaluates three dimensions of the usability (efficiency, satisfaction, and effectiveness) and usefulness of the patient app.

Methods

We performed a four-month user-centered observational study. Quantitative and qualitative data were collected. Participants completed the system usability scale (SUS) questionnaire and a second questionnaire on usefulness. Effectiveness was assessed by measuring the completeness of the medication list generated by the patient application and its correctness (ie medication discrepancies between the patient list and the best possible medication history). Qualitative data were collected from semi-structured interviews, observations and comments, and questions raised by patients.

Results

Forty-two patients completed the study. Sixty-nine percent of patients considered the patient app to be acceptable (SUS Score ≥ 70) and usefulness was high. The medication list was complete for a quarter of the patients (7/28) and there was a discrepancy for 21.7% of medications (21/97). The qualitative data enabled the identification of several barriers (related to functional and non-functional aspects) to the optimization of usability and usefulness.

Conclusions

Our findings highlight the importance and value of user-centered usability testing of a patient application implemented in "real-world" conditions. To achieve adoption and sustained use by patients, the app should meet patients' needs while also efficiently improving the quality of MedRec.

Does health informatics have a replication crisis?

Objective

Many research fields, including psychology and basic medical sciences, struggle with poor reproducibility of reported studies. Biomedical and health informatics is unlikely to be immune to these challenges. This paper explores replication in informatics and the unique challenges the discipline faces.

Methods

Narrative review of recent literature on research replication challenges.

Results

While there is growing interest in re-analysis of existing data, experimental replication studies appear uncommon in informatics. Context effects are a particular challenge as they make ensuring replication fidelity difficult, and the same intervention will never quite reproduce the same result in different settings. Replication studies take many forms, trading-off testing validity of past findings against testing generalizability. Exact and partial replication designs emphasize testing validity while quasi and conceptual studies test generalizability of an underlying model or hypothesis with different methods or in a different setting.

Conclusions

The cost of poor replication is a weakening in the quality of published research and the evidence-based foundation of health informatics. The benefits of replication include increased rigor in research, and the development of evaluation methods that distinguish the impact of context and the nonreproducibility of research. Taking replication seriously is essential if biomedical and health informatics is to be an evidence-based discipline.

A review of measurement practice in studies of clinical decision support systems 1998-2017

OBJECTIVE

To assess measurement practice in clinical decision support evaluation studies.

MATERIALS AND METHODS

We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt's typology to categorize the studies.

RESULTS

There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse.

DISCUSSION

Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere.

CONCLUSION

Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

Alerts for community pharmacist-provided medication therapy management: recommendations from a heuristic evaluation

BACKGROUND

Medication therapy management (MTM) is a service, most commonly provided by pharmacists, intended to identify and resolve medication therapy problems (MTPs) to enhance patient care. MTM is typically documented by the community pharmacist in an MTM vendor's web-based platform. These platforms often include integrated alerts to assist the pharmacist with assessing MTPs. In order to maximize the usability and usefulness of alerts to the end users (e.g., community pharmacists), MTM alert design should follow principles from human factors science. Therefore, the objectives of this study were to 1) evaluate the extent to which alerts for community pharmacist-delivered MTM align with established human factors principles, and 2) identify areas of opportunity and recommendations to improve MTM alert design.

METHODS

Five categories of MTM alerts submitted by community pharmacists were evaluated: 1) indication, 2) effectiveness; 3) safety; 4) adherence; and 5) cost-containment. This heuristic evaluation was guided by the Instrument for Evaluating Human-Factors Principles in Medication-Related Decision Support Alerts (I-MeDeSA) which we adapted and contained 32 heuristics. For each MTM alert, four analysts' individual ratings were summed and a mean score on the modified I-MeDeSA computed. For each heuristic, we also computed the percent of analyst ratings indicating alignment with the heuristic. We did this for all alerts evaluated to produce an "overall" summary of analysts' ratings for a given heuristic, and we also computed this separately for each alert category. Our results focus on heuristics where ≤50% of analysts' ratings indicated the alerts aligned with the heuristic.

RESULTS

I-MeDeSA scores across the five alert categories were similar. Heuristics pertaining to visibility and color were generally met. Opportunities for improvement across all MTM alert categories pertained to the principles of alert prioritization; text-based information; alarm philosophy; and corrective actions.

CONCLUSIONS

MTM alerts have several opportunities for improvement related to human factors principles, resulting in MTM alert design recommendations. Enhancements to MTM alert design may increase the effectiveness of MTM delivery by community pharmacists and result in improved patient outcomes.

Cultivating eHealth evaluation leadership

This article discusses eHealth evaluation leadership in terms of the need, key characteristics, and means to cultivate this leadership. It is well known that health leaders with technical informatics skills and prior eHealth adoption experience are more likely to commit to a long-term eHealth vision, are highly motivated, and can navigate through complex adoption issues. However, much less attention is paid to leadership when it comes to electronic health evaluation. The aim here is to engage the eHealth and broader health leadership communities in a dialogue about eHealth evaluation leadership in terms of its importance, the issues involved, and ways to build capacity in Canada and abroad.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications

OBJECTIVES

This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance.

METHOD

A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems.

RESULTS

There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed.

CONCLUSION

Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

Bridging the Gap Between Academic Research and Pragmatic Needs in Usability: A Hybrid Approach to Usability Evaluation of Health Care Information Systems

BACKGROUND

Technology is increasingly embedded into the full spectrum of health care. This movement has benefited from the application of software development practices such as usability testing and agile development processes. These practices are frequently applied in both commercial or operational and academic settings. However, the relative importance placed on rapid iteration, validity, reproducibility, generalizability, and efficiency differs between the 2 settings and the needs and objectives of academic versus pragmatic usability evaluations.

OBJECTIVE

This paper explores how usability evaluation typically varies on key dimensions in pragmatic versus academic settings that impact the rapidity, validity, and reproducibility of findings and proposes a hybrid approach aimed at satisfying both pragmatic and academic objectives.

METHODS

We outline the characteristics of pragmatic versus academically oriented usability testing in health care, describe the tensions and gaps resulting from differing contexts and goals, and present a model of this hybrid process along with 2 case studies of digital development projects in which we demonstrate this integrated approach to usability evaluation.

RESULTS

The case studies presented illustrate design choices characteristic of our hybrid approach to usability evaluation.

CONCLUSIONS

Designed to leverage the strengths of both pragmatically and academically focused usability studies, a hybrid approach allows new development projects to efficiently iterate and optimize from usability data as well as preserves the ability of these projects to produce deeper insights via thorough qualitative analysis to inform further tool development and usability research by way of academically focused dissemination.

The GUIDES checklist: development of a tool to improve the successful use of guideline-based computerised clinical decision support

Background

Computerised decision support (CDS) based on trustworthy clinical guidelines is a key component of a learning healthcare system. Research shows that the effectiveness of CDS is mixed.

Multifaceted context, system, recommendation and implementation factors may potentially affect the success of CDS interventions. This paper describes the development of a checklist that is intended to support professionals to implement CDS successfully.

Methods

We developed the checklist through an iterative process that involved a systematic review of evidence and frameworks, a synthesis of the success factors identified in the review, feedback from an international expert panel that evaluated the checklist in relation to a list of desirable framework attributes, consultations with patients and healthcare consumers and pilot testing of the checklist.

Results

We screened 5347 papers and selected 71 papers with relevant information on success factors for guideline-based CDS. From the selected papers, we developed a 16-factor checklist that is divided in four domains, i.e. the CDS context, content, system and implementation domains. The panel of experts evaluated the checklist positively as an instrument that could support people implementing guideline-based CDS across a wide range of settings globally. Patients and healthcare consumers identified guideline-based CDS as an important quality improvement intervention and perceived the GUIDES checklist as a suitable and useful strategy.

Conclusions

The GUIDES checklist can support professionals in considering the factors that affect the success of CDS interventions. It may facilitate a deeper and more accurate understanding of the factors shaping CDS effectiveness. Relying on a structured approach may prevent that important factors are missed.

Electronic supplementary material

The online version of this article (10.1186/s13012-018-0772-3) contains supplementary material, which is available to authorized users.

Evidence-based Health Informatics: How Do We Know What We Know?

Background: Health IT is expected to have a positive impact on the quality and efficiency of health care. But reports on negative impact and patient harm continue to emerge. The obligation of health informatics is to make sure that health IT solutions provide as much benefit with as few negative side effects as possible. To achieve this, health informatics as a discipline must be able to learn, both from its successes as well as from its failures.

Objectives: To present motivation, vision, and history of evidence-based health informatics, and to discuss achievements, challenges, and needs for action.

Methods: Reflections on scientific literature and on own experiences.

Results: Eight challenges on the way towards evidence-based health informatics are identified and discussed: quality of studies; publication bias; reporting quality; availability of publications; systematic reviews and meta-analysis; training of health IT evaluation experts; translation of evidence into health practice; and post-market surveil-lance. Identified needs for action comprise: establish health IT study registers; increase the quality of publications; develop a taxonomy for health IT systems; improve indexing of published health IT evaluation papers; move from meta-analysis to meta-summaries; include health IT evaluation competencies in curricula; develop evidence-based implementation frameworks; and establish post-marketing surveillance for health IT.

Conclusions: There has been some progress, but evidence-based health informatics is still in its infancy. Building evidence in health informatics is our obligation if we consider medical informatics a scientific discipline.

Problems with health information technology and their effects on care delivery and patient outcomes: a systematic review

Objective

To systematically review studies reporting problems with information technology (IT) in health care and their effects on care delivery and patient outcomes.

Materials and methods

We searched bibliographic databases including Scopus, PubMed, and Science Citation Index Expanded from January 2004 to December 2015 for studies reporting problems with IT and their effects. A framework called the information value chain, which connects technology use to final outcome, was used to assess how IT problems affect user interaction, information receipt, decision-making, care processes, and patient outcomes. The review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement.

Results

Of the 34 studies identified, the majority ( n  = 14, 41%) were analyses of incidents reported from 6 countries. There were 7 descriptive studies, 9 ethnographic studies, and 4 case reports. The types of IT problems were similar to those described in earlier classifications of safety problems associated with health IT. The frequency, scale, and severity of IT problems were not adequately captured within these studies. Use errors and poor user interfaces interfered with the receipt of information and led to errors of commission when making decisions. Clinical errors involving medications were well characterized. Issues with system functionality, including poor user interfaces and fragmented displays, delayed care delivery. Issues with system access, system configuration, and software updates also delayed care. In 18 studies (53%), IT problems were linked to patient harm and death. Near-miss events were reported in 10 studies (29%).

Discussion and conclusion

The research evidence describing problems with health IT remains largely qualitative, and many opportunities remain to systematically study and quantify risks and benefits with regard to patient safety. The information value chain, when used in conjunction with existing classifications for health IT safety problems, can enhance measurement and should facilitate identification of the most significant risks to patient safety.

Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis

Objectives: To set the scientific context and then suggest principles for an evidence-based approach to secondary uses of clinical data, covering both evaluation of the secondary uses of data and evaluation of health systems and services based upon secondary uses of data. Method: Working Group review of selected literature and policy approaches. Results: We present important considerations in the evaluation of secondary uses of clinical data from the angles of governance and trust, theory, semantics, and policy. We make the case for a multi-level and multi-factorial approach to the evaluation of secondary uses of clinical data and describe a methodological framework for best practice. We emphasise the importance of evaluating the governance of secondary uses of health data in maintaining trust, which is essential for such uses. We also offer examples of the re-use of routine health data to demonstrate how it can support evaluation of clinical performance and optimize health IT system design. Conclusions: Great expectations are resting upon "Big Data" and innovative analytics. However, to build and maintain public trust, improve data reliability, and assure the validity of analytic inferences, there must be independent and transparent evaluation. A mature and evidence-based approach needs not merely data science, but must be guided by the broader concerns of applied health informatics.

Improving Evaluation to Address the Unintended Consequences of Health Information Technology:. a Position Paper from the Working Group on Technology Assessment & Quality Development

BACKGROUND AND OBJECTIVES

With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT.

METHODS

Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation.

CONCLUSION

Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a proactive and evidence-based approach to detecting and addressing the unintended effects of health IT.

Health information systems evaluation frameworks: A systematic review

BACKGROUND

Evaluation of health information systems (HISs) is complicated because of the complex nature of the health care domain. Various studies have proposed different frameworks to reduce the complexity in the assessment of these systems. The aim of these frameworks is to provide a set of guidelines for the evaluation of the adequacy of health care information systems.

OBJECTIVE

This paper aims to analyse studies on the evaluation of HISs by applying a content, context and process (CCP) framework to address the 'who', 'what', 'how', 'when', and 'why' of the evaluation processes used. This will allow for a better understanding of the relative strengths and weaknesses of various HISs evaluation frameworks, and will pave the way for developing a more complete framework for HISs.

METHOD

A systematic literature review on HIS evaluation studies was undertaken to identify the currently available HIS evaluation frameworks. Five academic databases were selected to conduct this systematic literature review.

RESULTS

Most of the studies only address some, but not all, of the five main questions, i.e. the who, what, how, when, why, and that there was a lack of consensus in the way these questions were addressed. The critical role of context was also largely neglected in these studies.

CONCLUSIONS

Evaluation of HISs is complex. The health care domain is highly context sensitive and in order to have a complete assessment of HISs, consideration of contextual factors is necessary. Specifically, to have the right set of criteria to measure the 'what', the answer to the 'who' of the evaluation is necessary.

Improving the effectiveness of drug safety alerts to increase adherence to the guideline for gastrointestinal prophylaxis

OBJECTIVE

Gastrointestinal bleedings are the most frequently occurring reason for medication-related hospital admissions, which are potentially preventable. We implemented a clinical decision support system that recommends to prescribe gastrointestinal prophylaxis in patients with an increased risk according to the Dutch guideline. Our primary objective was to determine whether the implementation resulted in improved compliance with this guideline for gastrointestinal prophylaxis. A secondary objective was to determine whether implementation resulted in a reduction of the number of drug safety alerts.

MATERIALS AND METHODS

This intervention study was performed at the Spaarne Gasthuis, a teaching hospital, using Epic as hospital information system. We selected prescriptions with an indication for gastrointestinal prophylaxis according to the guideline, in the three months before and after implementation of the clinical decision support in November 2014. We analyzed whether gastrointestinal prophylaxis was prescribed more frequently after implementation using the Pearson's Chi-square test and the change in the number of drug safety alerts.

RESULTS

Before implementation in 84.0% of the included 2064 prescriptions gastrointestinal prophylaxis was co-prescribed. After implementation this percentage increased to 94.5% of the 2269 prescriptions (p<0.001). The number of drug safety alerts decreased by 78.2% from 980 to 217 alerts.

CONCLUSION

The introduction of a clinical decision support system for gastrointestinal prophylaxis improved adherence to the Dutch guideline. This was most likely due to a reduction in the number of irrelevant drug safety alerts.

Developing the FARSEEING Taxonomy of Technologies: Classification and description of technology use (including ICT) in falls prevention studies

BACKGROUND

Recent Cochrane reviews on falls and fall prevention have shown that it is possible to prevent falls in older adults living in the community and in care facilities. Technologies aimed at fall detection, assessment, prediction and prevention are emerging, yet there has been no consistency in describing or reporting on interventions using technologies. With the growth of eHealth and data driven interventions, a common language and classification is required.

OBJECTIVE

The FARSEEING Taxonomy of Technologies was developed as a tool for those in the field of biomedical informatics to classify and characterise components of studies and interventions.

METHODS

The Taxonomy Development Group (TDG) comprised experts from across Europe. Through face-to-face meetings and contributions via email, five domains were developed, modified and agreed: Approach; Base; Components of outcome measures; Descriptors of technologies; and Evaluation. Each domain included sub-domains and categories with accompanying definitions. The classification system was tested against published papers and further amendments undertaken, including development of an online tool. Six papers were classified by the TDG with levels of consensus recorded.

RESULTS

Testing the taxonomy with papers highlighted difficulties in definitions across international healthcare systems, together with differences of TDG members' backgrounds. Definitions were clarified and amended accordingly, but some difficulties remained. The taxonomy and manual were large documents leading to a lengthy classification process. The development of the online application enabled a much simpler classification process, as categories and definitions appeared only when relevant. Overall consensus for the classified papers was 70.66%. Consensus scores increased as modifications were made to the taxonomy.

CONCLUSION

The FARSEEING Taxonomy of Technologies presents a common language, which should now be adopted in the field of biomedical informatics. In developing the taxonomy as an online tool, it has become possible to continue to develop and modify the classification system to incorporate new technologies and interventions.

Key influences in the design and implementation of mental health information systems in Ghana and South Africa

INTRODUCTION

Strengthening of mental health information systems (MHIS) is essential to monitor and evaluate mental health services in low and middle-income countries. While research exists assessing wider health management information systems, there is limited published evidence exploring the design and implementation of MHIS in these settings. This paper aims to identify and assess the key factors affecting the design and implementation of MHIS, as perceived by the key stakeholders in Ghana and South Africa.

METHODS

We report findings from the Mental Health and Poverty Project, a 5-year research programme implemented within four African countries. The MHIS strengthening in South Africa and Ghana included two related components: intervention and research. The intervention component aimed to strengthen MHIS in the two countries, and the research component aimed to document interventions in each country, including the key influences. Data were collected using semi structured interviews with key stakeholders and reviews of key documents and secondary data from the improved MHIS. We analyzed the qualitative data using a framework approach.

RESULTS

Key components of the MHIS intervention involved the introduction of a redesigned patient registration form, entry into computers for analysis every 2 months by clinical managerial staff, and utilization of data in hospital management meetings in three psychiatric hospitals in Ghana; and the introduction of a new set of mental health indicators and related forms and tally sheets at primary care clinics and district hospitals in five districts in the KwaZulu-Natal and Northern Cape provinces in South Africa. Overall, the key stakeholders perceived the MHIS strengthening as an effective intervention in both countries with an enhanced set of indicators in South Africa and introduction of a computerized system in Ghana.

DISCUSSION

Influences on the design and implementation of MHIS interventions in Ghana and South Africa relate to resources, working approaches (including degree of consultations during the design stage and communication during implementation stage) and the low priority of mental health. Although the influencing factors represent similar categories, more influences were identified on MHIS implementation, compared with the design stage. Different influences appear to be related within, and across, the MHIS design and implementation and may reinforce or negate each other thus leading to the multiplier or minimization effects. The wider context, similar to other studies, is important in ensuring the success of such interventions.

CONCLUSION

Future MHIS strengthening interventions can consider three policy implications which emerged from our analysis and experience: enhancing consultations during the intervention design, better consideration of implementation challenges during design, and better recognition of relations between different influences.

What's Past is Prologue: A Scoping Review of Recent Public Health and Global Health Informatics Literature

OBJECTIVE

To categorize and describe the public health informatics (PHI) and global health informatics (GHI) literature between 2012 and 2014.

METHODS

We conducted a semi-systematic review of articles published between January 2012 and September 2014 where information and communications technologies (ICT) was a primary subject of the study or a main component of the study methodology. Additional inclusion and exclusion criteria were used to filter PHI and GHI articles from the larger biomedical informatics domain. Articles were identified using MEDLINE as well as personal bibliographies from members of the American Medical Informatics Association PHI and GHI working groups.

RESULTS

A total of 85 PHI articles and 282 GHI articles were identified. While systems in PHI continue to support surveillance activities, we identified a shift towards support for prevention, environmental health, and public health care services. Furthermore, articles from the U.S. reveal a shift towards PHI applications at state and local levels. GHI articles focused on telemedicine, mHealth and eHealth applications. The development of adequate infrastructure to support ICT remains a challenge, although we identified a small but growing set of articles that measure the impact of ICT on clinical outcomes.

DISCUSSION

There is evidence of growth with respect to both implementation of information systems within the public health enterprise as well as a widening of scope within each informatics discipline. Yet the articles also illuminate the need for more primary research studies on what works and what does not as both searches yielded small numbers of primary, empirical articles.

CONCLUSION

While the body of knowledge around PHI and GHI continues to mature, additional studies of higher quality are needed to generate the robust evidence base needed to support continued investment in ICT by governmental health agencies.

A case study evaluation of a Critical Care Information System adoption using the socio-technical and fit approach

BACKGROUND

Clinical information systems have long been used in intensive care units but reports on their adoption and benefits are limited. This study evaluated a Critical Care Information System implementation.

METHODS

A case study summative evaluation was conducted, employing observation, interview, and document analysis in operating theatres and 16-bed adult intensive care units in a 400-bed Malaysian tertiary referral centre from the perspectives of users (nurses and physicians), management, and information technology staff. System implementation, factors influencing adoption, fit between these factors, and the impact of the Critical Care Information System were evaluated after eight months of operation.

RESULTS

Positive influences on system adoption were associated with technical factors, including system ease of use, usefulness, and information relevancy; human factors, particularly user attitude; and organisational factors, namely clinical process-technology alignment and champions. Organisational factors such as planning, project management, training, technology support, turnover rate, clinical workload, and communication were barriers to system implementation and use. Recommendations to improve the current system problems were discussed. Most nursing staff positively perceived the system's reduction of documentation and data access time, giving them more time with patients. System acceptance varied among doctors. System use also had positive impacts on timesaving, data quality, and clinical workflow.

CONCLUSIONS

Critical Care Information Systems is crucial and has great potentials in enhancing and delivering critical care. However, the case study findings showed that the system faced complex challenges and was underutilised despite its potential. The role of socio-technical factors and their fit in realizing the potential of Critical Care Information Systems requires continuous, in-depth evaluation and stakeholder understanding and acknowledgement. The comprehensive and specific evaluation measures of the Human-Organisation-Technology Fit framework can flexibly evaluate Critical Care Information Systems.

The impact of clinical leadership on health information technology adoption: systematic review

PURPOSE

To conduct a systematic review to examine evidence of associations between clinical leadership and successful information technology (IT) adoption in healthcare organisations.

METHODS

We searched Medline, Embase, Cinahl, and Business Source Premier for articles published between January 2000 to May 2013 with keywords and subject terms related to: (1) the setting--healthcare provider organisations; (2) the technology--health information technology; (3) the process--adoption; and (4) the intervention--leadership. We identified 3121 unique citations, of which 32 met our criteria and were included in the review. Data extracted from the included studies were assessed in light of two frameworks: Bassellier et al.'s IT competence framework; and Avgar et al.'s health IT adoption framework.

RESULTS

The results demonstrate important associations between the attributes of clinical leaders and IT adoption. Clinical leaders who have technical informatics skills and prior experience with IT project management are likely to develop a vision that comprises a long-term commitment to the use of IT. Leaders who possess such a vision believe in the value of IT, are motivated to adopt it, and can maintain confidence and stability through the adversities that IT adoptions often entail. This leads to proactive leadership behaviours and partnerships with IT professionals that are associated with successful organisational and clinical outcomes.

CONCLUSIONS

This review provides evidence that clinical leaders can positively contribute to successful IT adoption in healthcare organisations. Clinical leaders who aim for improvements in the processes and quality of care should cultivate the necessary IT competencies, establish mutual partnerships with IT professionals, and execute proactive IT behaviours to achieve successful IT adoption.