Helping patient educators meet health literacy needs: End-user testing and iterative development of an innovative health literacy editing tool

Objective

The Sydney Health Literacy Lab (SHeLL) Editor is an online text-editing tool that provides real-time assessment and feedback on written health information (assesses grade reading score, complex language, passive voice). This study aimed to explore how the design could be further enhanced to help health information providers interpret and act on automated feedback.

Methods

The prototype was iteratively refined across four rounds of user-testing with health services staff (N = 20). Participants took part in online interviews and a brief follow-up survey using validated usability scales (System Usability Scale, Technology Acceptance Model). After each round, Yardley's (2021) optimisation criteria guided which changes would be implemented.

Results

Participants rated the Editor as having adequate usability (M = 82.8 out of 100, SD = 13.5). Most modifications sought to reduce information overload (e.g. simplifying instructions for new users) or make feedback motivating and actionable (e.g. using frequent incremental feedback to highlight changes to the text altered assessment scores).

Conclusion

terative user-testing was critical to balancing academic values and the practical needs of the Editor's target users. The final version emphasises actionable real-time feedback and not just assessment.

Innovation

The Editor is a new tool that will help health information providers apply health literacy principles to written text.

Can we achieve better trial recruitment by presenting patient information through multimedia? Meta-analysis of 'studies within a trial' (SWATs)

BACKGROUND

People need high-quality information to make decisions about research participation. Providing information in written format alone is conventional but may not be the most effective and acceptable approach. We developed a structure for the presentation of information using multimedia which included generic and trial-specific content. Our aim was to embed 'Studies Within A Trial' (SWATs) across multiple ongoing trials to test whether multimedia presentation of patient information led to better rates of recruitment.

METHODS

Five trials included a SWAT and randomised their participants to receive a multimedia presentation alongside standard information, or standard written information alone. We collected data on trial recruitment, acceptance and retention and analysed the pooled results using random effects meta-analysis, with the primary outcome defined as the proportion of participants randomised following an invitation to take part.

RESULTS

Five SWATs provided data on the primary outcome of proportion of participants randomised. Multimedia alongside written information results in little or no difference in recruitment rates (pooled odds ratio = 0.96, 95% CI: 0.79 to 1.17, p-value = 0.671, I2 = 0%). There was no effect on any other outcomes.

CONCLUSIONS

Multimedia alongside written information did not improve trial recruitment rates.

TRIAL REGISTRATION

ISRCTN71952900, ISRCTN 06710391, ISRCTN 17160087, ISRCTN05926847, ISRCTN62869767.

Digital encounter decision aids linked to clinical practice guidelines: results from user testing SHARE-IT decision aids in primary care

BACKGROUND

Encounter decision aids (EDAs) are tools that can support shared decision making (SDM), up to the clinical encounter. However, adoption of these tools has been limited, as they are hard to produce, to keep up-to-date, and are not available for many decisions. The MAGIC Evidence Ecosystem Foundation has created a new generation of decision aids that are generically produced along digitally structured guidelines and evidence summaries, in an electronic authoring and publication platform (MAGICapp). We explored general practitioners' (GPs) and patients' experiences with five selected decision aids linked to BMJ Rapid Recommendations in primary care.

METHODS

We applied a qualitative user testing design to evaluate user experiences for both GPs and patients. We translated five EDAs relevant to primary care, and observed the clinical encounters of 11 GPs when they used the EDA with their patients. We conducted a semi-structured interview with each patient after the consultation and a think-aloud interview with each GPs after multiple consultations. We used the Qualitative Analysis Guide (QUAGOL) for data analysis.

RESULTS

Direct observations and user testing analysis of 31 clinical encounters showed an overall positive experience. The EDAs created better involvement in decision making and resulted in meaningful insights for patients and clinicians. The design and its interactive, multilayered structure made the tool enjoyable and well-organized. Difficult terminology, scales and numbers hindered understanding of certain information, which was sometimes perceived as too specialized or even intimidating. GPs thought the EDA was not suitable for every patient. They perceived a learning curve was required and the need for time investment was a concern. The EDAs were considered trustworthy as they were provided by a credible source.

CONCLUSIONS

This study showed that EDAs can be useful tools in primary care by supporting actual shared decision making and enhancing patient involvement. The graphical approach and clear representation help patients better understand their options. To overcome barriers such as health literacy and GPs attitudes, effort is still needed to make the EDAs as accessible, intuitive and inclusive as possible through use of plain language, uniform design, rapid access and training.

TRIAL REGISTRATION

The study protocol was approved by the The Research Ethics Committee UZ/KU Leuven (Belgium) on 31-10-2019 with reference number MP011977.

Sensor-based evaluation of a Urine Trap toilet in a shared bathroom

Urine diversion in a No-Mix Toilet is a promising approach for sustainable fertilizers and reduction of the nutrient load for wastewater treatment; however, user adoption remains a challenge. This study evaluates the Urine Trap, a passive No-Mix toilet design based on the teapot effect, wherein the urine stream inlet is invisible to the user and therefore it does not impact the user experience for increased adoption. This study evaluated the nutrient separation performance of a Urine Trap flush toilet in a bathroom shared by women in two sites in India. Over three different testing periods, 841 uses of this squat plate were recorded in 50 days. Analytical measurements found 36 % separation efficiency for total nitrogen (TN). While effective, the Urine Trap under test by users did not yield a 70-80 % TN separation efficiency observed under engineering characterization. High temporal resolution data from sensors on waste collection tanks, the opening of the bathroom door, and cleansing water flow were used to gain insights into hygiene practices. The data showed a frequent habit of wetting the squat plate during physiological excretion, a hygienic practice that eases cleaning but degrades the teapot separation effect of the Urine Trap design. By using sensors, we demonstrate a method to non-invasively gain quantitative insights into hygiene practices to inform sanitation technologies deployment strategies for improved outcomes.

Evaluating the effectiveness of automatic image captioning for web accessibility

The web has become a fundamental tool for carrying out many activities spanning from education to work and private life. For this reason, it must be accessible to every user regardless of any form of impairment or disability. Images on the web are a primary means for communicating information, and specific HTML elements were defined to enrich images with textual descriptions, which can be read aloud by screen readers or rendered by braille displays. A relevant problem is that adding a text describing each image published on a website is a demanding task requiring a non-negligible effort for web developers. Several tools based on machine learning have emerged, which can automatically return descriptions for the images. In this work, we evaluate the correctness of their outputs by comparing the generated descriptions with human-defined references. More specifically, we selected 60 images from Wikipedia and their corresponding descriptions as defined by Wikipedia contributors. We then generated the corresponding descriptions employing four state of the art tools (Azure Computer Vision Engine, Amazon Rekognition, Cloudsight, and Auto Alt-Text for Google Chrome) and asked 76 computer science students to blindly evaluate the perceived correctness of the descriptions without being aware of their source. The results show that the descriptions available in Wikipedia are still perceived as the best ones. However, some tools generate good results for specific categories of images, and they can represent proper candidates for the automated and massive addition of image descriptions to websites, helping to increase the accessibility level of the web drastically.

Making BRCA1 genetic test reports easier to understand through user-centered design: A randomized trial

PURPOSE

Our objective was to apply a user-centered design process to identify phrases, graphics, and ways of communicating numerical risks that could be used to help patients understand their cancer risk and next steps on receiving BRCA1 genetic test results (positive, negative, and variants of uncertain significance).

METHODS

The first phase of the study, a user-centered design process, consisted of 4 rounds of interviews (N = 42, including 13 health care professionals and 16 patients having undergone BRCA testing). The second was a randomized, between-participants experimental study of 456 United Kingdom residents that compared the resulting reports to reports used in a United Kingdom national genomic laboratory hub. Outcomes were subjective and objective comprehension, communication efficacy, actionability, and perceived risk.

RESULTS

Subjective comprehension, communication efficacy, and actionability were all higher for the user-centered reports, with no difference in perceived risk. Comprehension of participants viewing user-centered reports was significantly better on 2 items, directionally (but not significantly) better on 6 items, and directionally (but not significantly) worse on 2 items.

CONCLUSION

Our results imply that user-centered design is a promising approach for developing materials about complex genetic risks. We suggest wordings that are likely to lead to improved comprehension when communicating BRCA-associated risks.

User testing of a Scottish Intercollegiate Guideline Network public guideline for the parents of children with autism

Background

The Scottish Intercollegiate Guidelines Network (SIGN) is the leading national clinical guideline producer in Scotland. Improved design and dissemination of guidelines produced for the public can empower people to take an active role in self-management and shared decision-making. The public version of the guideline examined covered getting assessed and diagnosed with autism, and approaches that can help. The aim of this study was to test a public version of a guideline for the parents of children and young people with autism, implement improvements, and identify what works in making it usable and accessible.

Methods

We recruited mothers from across Scotland. User testing involved formal ‘think aloud’ semi-structured interviews that guided users through the booklet. Interviews took place individually and were recorded and transcribed. Key findings were identified and themed using the honeycomb user experience model.

Results

Fourteen user-testing interviews were conducted. Facilitators for usability and desirability of the guideline included the chunking of text, consistent use of colour and boxes to highlight important information. Simple language, written in a tone of partnership, helped to engage mothers. Value arose from the guidelines ability to explain the process of diagnosis and make mothers feel empowered in their relationships with healthcare professionals. There was a lack of consensus on the usefulness of rating the strength of evidence and recommendations.

Conclusion

There was a marked similarity between what was important to the mothers and what has been found to be important to other groups. The involvement of service users and carers in the guidelines development was key to its credibility. One size does not fit all in presenting evidence-based recommendations to the public and it is a challenge to provide sufficient information while avoiding information overload. Recommendations and evidence levels are suitable for use in public versions, but these should be kept as simple as possible.

User testing to examine patient understanding of pharmacy generated medication labels

OBJECTIVE

To examine patients' and carers' understanding of pharmacy generated medication labels.

METHODS

A user testing questionnaire was conducted individually for 80 participants at 2 hospitals through a face-to face semi-structured interview. Pharmacy generated medication labels from different locations were grouped based on components into 4 different variations. Participants were asked to read and demonstrate understanding of the dose and frequency from 1 of the 4 variations for 4 prescription medications. Twenty participants for each variation were recruited so that demographic characteristics matched between variations.

RESULTS

Overall, only 45% of participants were able to correctly understand the dose and frequency presented on all the pharmacy labels presented on medications. Medication labels with standardised timing performed better than other variations with 91% of participants able to determine the correct frequency. The use of numeric figures was understood by 80-90% of participants compared to the use of capitalised text (65-70%). Pharmacy generated medication labels that proposed one step were better understood than instructions that incorporated several steps.

CONCLUSION/PRACTICE IMPLICATIONS

The study supports the use of simple, clear and explicit written instructions along with the use of numeric figures in pharmacy generated medication labels to achieve higher understandability in patients.

Field testing of a household-scale onsite blackwater treatment system in Coimbatore, India

4.2 billion people live without access to safely managed sanitation services. This report describes the field testing of an onsite prototype system designed to treat blackwater from a single flush toilet and reuse of the treated effluent for flushing. The system passes wastewater through a solid-liquid separator followed by settling tanks and granular activated carbon columns into an electrochemical reactor that oxidizes chloride salts from urine to generate chlorine to remove pathogens. The objectives of the study were to verify the functionality of the system (previously demonstrated in the laboratory) under realistic use conditions, to identify maintenance requirements, and to make a preliminary assessment of the system's user acceptability. The prototype was installed in a women's workplace and residential toilet block in Coimbatore, India, and tested over a period of 10 months. The treated water met stringent disinfection threshold for both E. coli and helminth eggs and produced a clear, colorless effluent that met or nearly met local and international discharge standards for non-sewered sanitation systems. The effluent had an average chemical oxygen demand of 81 mg/L, total suspended solids of 11 mg/L, and reduction of total nitrogen by 65%. These tests determined the recommended service lifetimes and maintenance intervals for key system components including the electrochemical cell, granular activated carbon columns, and solid-liquid separator. User feedback regarding the use of treated blackwater as flush water was positive. These findings will inform the design and implementation of next-generation systems currently under development.

Field testing of a household-scale onsite blackwater treatment system in South Africa

Innovations that enable cost-effective and resource-conserving treatment of human waste are required for the 4.2 billion people in the world who currently lack safe and reliable sanitation services. Onsite treatment and reuse of blackwater is one strategy towards this end, greatly reducing the need to transport wastewater over long distances either via sewers or trucks. Here, we report on the field testing of a prototype onsite blackwater treatment system conducted over a period of 8 months. The system was connected to a women's toilet in a public communal ablution block located in an informal settlement near Durban, South Africa. Liquid waste was treated by separation and diversion of large solids, settling of suspended solids, and filtration through activated carbon prior to disinfection by electrochemical oxidation. System performance was monitored daily by measurement of chemical and physical water quality parameters onsite and confirmed by periodic detailed analysis of chemical and biological parameters at an offsite lab. Daily monitoring of system performance indicated that the effluent had minimal color and turbidity (maximum 90 Pt/Co units and 6.48 NTU, respectively), and consistent evolution of chlorine as blackwater passed through the system. Weekly offsite analysis confirmed that the system consistently inactivated pathogens (E. coli and coliforms) and reduced chemical oxygen demand and total suspended solids to meet ISO 30500 category B standards. Significant reductions in total nitrogen load were also observed, though these reductions often fell short of the 70% reduction required by ISO 30500. No significant reduction in total phosphorus was observed. Maintenance requirements were identified, and the resilience of the system to restart following a prolonged shutdown was demonstrated, but significant improvements are required in the design of the solid/liquid separation mechanism for application of this system in a wiping culture.

Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach

BACKGROUND

Claims about what we need to do to improve our health are everywhere. Most interventions simply tell people what to do, and do not empower them to critically assess health information. Our objective was to design mass media resources to enable the public to critically appraise the trustworthiness of claims about the benefits and harms of treatments and make informed health choices.

METHODS

Research was conducted between 2013 and 2016 across multiple iterative phases. Participants included researchers, journalists, parents, other members of the public. First, we developed a list of 32 key concepts that people need to understand to be able to assess the trustworthiness of claims about treatment effects. Next, we used a human-centred design approach, to generate ideas for resources for teaching the key concepts, and developed and user-tested prototypes through qualitative interviews. We addressed identified problems and repeated this process until we had a product that was deemed relevant and desirable by our target audience, and feasible to implement.

RESULTS

We generated over 160 ideas, mostly radio-based. After prototyping some of these, we found that a podcast produced collaboratively by health researchers and journalists was the most promising approach. We developed eight episodes of the Informed Health Choices podcast, a song on critical thinking about treatments and a reminder checklist. Early versions of the podcast were reportedly too long, boring and confusing. We shortened the episodes, included one key concept per episode, and changed to story-telling with skits. The final version of the podcast was found to be useful, understandable, credible and desirable.

CONCLUSION

We found many problems with various prototypes of mass media resources. Using a human-centred design approach, we overcame those problems. We have developed a guide to help others prepare similar podcasts.

Field testing of a prototype mechanical dry toilet flush

A prototype of a non-fluid based mechanical toilet flush was tested in a semi-public, institutional setting and in selected peri-urban households in eThekwini municipality, Republic of South Africa. The mechanism's functionality and users' perception of the flush were assessed. User perception varied depending on background: Users accustomed to porcelain water flush toilets were open to, yet reserved about the idea of using a waterless flush in their homes. Those who commonly use Urine Diversion Dehydration Toilets were far more receptive. The user-centred field trials were complemented by a controlled laboratory experiment, using synthetic urine, -faeces, and -menstrual blood, to systematically assess the efficiency of three swipe materials to clean the rotating bowl of the flush. A silicone rubber with oil-bleed-effect was found to be the best performing material for the swipe. Lubrication of the bowl prior to use further reduced fouling. A mechanical waterless flush that does not require consumables, like plastic wrappers, is a novelty and could - implemented in existing dry toilet systems - improve acceptance and thus the success of waterless sanitation.

User testing of an adaptation of fishbone diagrams to depict results of systematic reviews

Background

Summary of findings tables in systematic reviews are highly informative but require epidemiological training to be interpreted correctly. The usage of fishbone diagrams as graphical displays could offer researchers an effective approach to simplify content for readers with limited epidemiological training. In this paper we demonstrate how fishbone diagrams can be applied to systematic reviews and present the results of an initial user testing.

Methods

Findings from two systematic reviews were graphically depicted in the form of the fishbone diagram. To test the utility of fishbone diagrams compared with summary of findings tables, we developed and pilot-tested an online survey using Qualtrics. Respondents were randomized to the fishbone diagram or a summary of findings table presenting the same body of evidence. They answered questions in both open-ended and closed-answer formats; all responses were anonymous. Measures of interest focused on first and second impressions, the ability to find and interpret critical information, as well as user experience with both displays. We asked respondents about the perceived utility of fishbone diagrams compared to summary of findings tables. We analyzed quantitative data by conducting t-tests and comparing descriptive statistics.

Results

Based on real world systematic reviews, we provide two different fishbone diagrams to show how they might be used to display complex information in a clear and succinct manner. User testing on 77 students with basic epidemiological training revealed that participants preferred summary of findings tables over fishbone diagrams. Significantly more participants liked the summary of findings table than the fishbone diagram (71.8% vs. 44.8%; p < .01); significantly more participants found the fishbone diagram confusing (63.2% vs. 35.9%, p < .05) or indicated that it was difficult to find information (65.8% vs. 45%; p < .01). However, more than half of the participants in both groups were unable to find critical information and answer three respective questions correctly (52.6% in the fishbone group; 51.3% in the summary of findings group).

Conclusions

Fishbone diagrams are compact visualizations that, theoretically, may prove useful for summarizing the findings of systematic reviews. Initial user testing, however, did not support the utility of such graphical displays.

Electronic supplementary material

The online version of this article (10.1186/s12874-017-0452-z) contains supplementary material, which is available to authorized users.

Evaluating the Use of Plain Language in a Cancer Clinical Trial Website/App

Medically complex titles and descriptions found on clinical trial websites and online applications present a barrier to comprehension for users from the general public. In this study, we examine the effectiveness of plain language trial descriptions for user comprehension of basic trial details. Two hundred seventeen volunteers recruited from patient waiting areas completed 441 user tests of ten plain language trial descriptions. The majority of volunteers adequately comprehended the cancer type and basic inclusion/exclusion criteria from plain language trial descriptions. Difficulty comprehending the treatment being studied was seen in seven of ten descriptions tested. Revision and retesting of the seven trial descriptions showed continued user challenges in comprehending the treatment being studied. Plain language clinical trial descriptions integrated into a website/app allowed users to understand basic inclusion/exclusion criteria. Despite plain language used, discerning the treatment being studied may be difficult for some users. Integration of plain language descriptions into clinical trial online applications can help users understand trial basics. Further research regarding effective use of plain language to communicate the treatment being studied is needed.

Evaluation of a novel multi-articulated endoscope: proof of concept through a virtual simulation

PURPOSE

In endoscopic surgery such as video-assisted thoracoscopic surgery and laparoscopic surgery, providing the surgeon a good view of the target is important. Rigid endoscope has for years been the go-to tool for this purpose, but it has certain limitations like the inability to work around obstacles. To improve on current tools, a novel multi-articulated endoscope (MAE) is currently under development. To investigate its feasibility and possible value, we performed a user test using virtual prototype of the MAE with the intent to show that it outperforms the conventional endoscope while bringing minimal additional burden to the operator.

METHODS

To evaluate the prototype, we built a virtual model of the MAE and a rigid oblique-viewing endoscope. Through a comparative user study we evaluate the ability of each device to visualize certain targets placed inside the virtual chest cavity by the angle between the visual axis of the scope and the normal of the plane of the target, while accounting for the usability of each endoscope by recording the time taken for each task. In addition, we collected a questionnaire from each participant to obtain feedback.

RESULTS

The angles obtained using the MAE were smaller on average ([Formula: see text]), indicating that better visualization can be achieved through the proposed method. A nonsignificant difference in mean time taken for each task in favor of the rigid endoscope was also found ([Formula: see text]).

CONCLUSIONS

We have demonstrated that better visualization for endoscopic surgery can be achieved through our novel MAE. The scope may bring about a paradigm shift in the field of minimally invasive surgery by providing more freedom in viewpoint selection, enabling surgeons to perform more elaborate procedures in minimally invasive settings.

Preparation, validation and user-testing of pictogram-based patient information leaflets for hemodialysis patients

Background: Patient information leaflets are universally-accepted resources to educate the patients/users about their medications, disease and lifestyle modification. Objectives: The objective of the study was to prepare, validate and perform user-testing of pictogram-based patient information leaflets (P-PILs) among hemodialysis (HD) patients. Methods: The P-PILs are prepared by referring to the primary, secondary and tertiary resources. The content and pictograms of the leaflet have been validated by an expert committee consisting of three nephrologists and two academic pharmacists. The Baker Able Leaflet Design has been applied to develop the layout and design of the P-PILs. Results: Quasi-experimental pre- and post-test design without control group was conducted on 81 HD patients for user-testing of P-PILs. The mean Baker Able Leaflet Design assessment score for English version of the leaflet was 28, and 26 for Kannada version. The overall user-testing knowledge assessment mean scores were observed to have significantly improved from 44.25 to 69.62 with p value <0.001. Conclusion: The overall user opinion of content and legibility of the leaflets was good. Pictogram-based patient information leaflets can be considered an effective educational tool for HD patients.

With how many users should you test a medical infusion pump? Sampling strategies for usability tests on high-risk systems

Usability testing is recognized as an effective means to improve the usability of medical devices and prevent harm for patients and users. Effectiveness of problem discovery in usability testing strongly depends on size and representativeness of the sample. We introduce the late control strategy, which is to continuously monitor effectiveness of a study towards a preset target. A statistical model, the LNB(zt) model, is presented, supporting the late control strategy. We report on a case study, where a prototype medical infusion pump underwent a usability test with 34 users. On the data obtained in this study, the LNB(zt) model is evaluated and compared against earlier prediction models. The LNB(zt) model fits the data much better than previously suggested approaches and improves prediction. We measure the effectiveness of problem identification, and observe that it is lower than is suggested by much of the literature. Larger sample sizes seem to be in order. In addition, the testing process showed high levels of uncertainty and volatility at small to moderate sample sizes, partly due to users' individual differences. In reaction, we propose the idiosyncrasy score as a means to obtain representative samples. Statistical programs are provided to assist practitioners and researchers in applying the late control strategy.