Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach

What is the effect of the Informed Health Choices secondary school intervention on the ability of students in Rwanda to think critically about health choices after one-year follow-up? A cluster-randomized trial

AIM

The aim of this study was to evaluate the effects of the Informed Health Choices secondary school intervention on the ability of students in Rwanda to think critically and make informed health choices after 1 year.

METHODS

This was a two-arm cluster-randomized trial conducted in 84 lower secondary schools from 10 districts representing five provinces of Rwanda. We used stratified randomization to allocate schools 1:1 to the intervention or control arm. One class in each intervention school had ten 40-min lessons taught by a trained teacher in addition to the usual curriculum. Control schools followed the usual curriculum. The primary outcome was a passing score (≥ 9 out of 18 questions answered correctly) for students on the Critical Thinking about Health Test completed 1 year after the intervention. We conducted an intention to treat analysis using generalized linear mixed models, accounting for the cluster design using random intercepts.

RESULTS

After 1 year, 35 of 42 teachers (83.3%) and 1181 of 1556 students (75.9%) in the control arm completed the test. In the intervention arm, 35 of 42 teachers (83.3%) and 1238 of 1572 students (78.8%) completed the test. The proportion of students who had a passing score in the intervention arm was 625/1238 (50.5%) compared to 230/1181 (19.5%) in the control arm (adjusted odds ratio 7.6 [95% CI: 4.6-12.6], p < 0.0001). The adjusted difference in the proportion of students with a passing score was 32.2% (95% CI 24.5-39.8%).

CONCLUSIONS

The IHC secondary school intervention was effective after 1 year. However, the size of the effect was smaller than immediately after the intervention (adjusted difference 32.2% vs 37.2%) due to decay in the proportion of students in intervention schools with a passing score (50.5% vs 58.2%).

TRIAL REGISTRATION

Pan African Clinical Trial Registry (PCTR), trial identifier: PACTR202203880375077. Registered on February 15, 2022.

Teaching critical thinking about health information and choices in secondary schools: human-centred design of digital resources

Background

Learning to thinking critically about health information and choices can protect people from unnecessary suffering, harm, and resource waste. Earlier work revealed that children can learn these skills, but printing costs and curricula compatibility remain important barriers to school implementation. We aimed to develop a set of digital learning resources for students to think critically about health that were suitable for use in Kenyan, Rwandan, and Ugandan secondary schools.

Methods

We conducted work in two phases collaborating with teachers, students, schools, and national curriculum development offices using a human-centred design approach. First, we conducted context analyses and an overview of teaching strategies, prioritised content and collected examples. Next, we developed lessons and guidance iteratively, informed by data from user-testing, individual and group interviews, and school pilots.

Results

Final resources include online lesson plans, teachers' guide, and extra resources, with lesson plans in two modes, for use in a classroom equipped with a blackboard/flip-chart and a projector. The resources are accessible offline for use when electricity or Internet is lacking. Teachers preferred the projector mode, as it provided structure and a focal point for class attention. Feedback was largely positive, with teachers and students appreciating the learning and experiencing it as relevant. Four main challenges included time to teach lessons; incorrect comprehension; identifying suitable examples; and technical, logistical, and behavioural challenges with a student-computer mode that we piloted. We resolved challenges by simplifying and combining lessons; increasing opportunities for review and assessment; developing teacher training materials, creating a searchable set of examples; and deactivating the student-computer mode.

Conclusion

Using a human-centred design approach, we created digital resources for teaching secondary school students to think critically about health actions and for training teachers. Be smart about your health resources are open access and can be translated or adapted to other settings.

Key concepts for informed health choices: Where's the evidence?

Background

The Informed Health Choices (IHC) Key Concepts is a framework that provides a basis for developing educational resources and evaluating people's ability to think critically about health actions. We developed the original Key Concepts framework by reviewing texts and checklists for the public, journalists, and health professionals and collecting structured feedback from an international advisory group. We revised the original 2015 framework yearly from 2016 to 2018 based on feedback and experience using the framework. The objectives of this paper are to describe the development of the framework since 2018 and summarise their basis.

Methods

For the 2019 version, we responded to feedback on the 2018 version. For the current 2022 version, in addition to responding to feedback on the 2019 version, we reviewed the evidence base for each of the concepts. Whenever possible, we referenced systematic reviews that provide a basis for a concept. We screened all Cochrane methodology reviews and searched Epistemonikos, PubMed, and Google Scholar for methodology reviews and meta-epidemiological studies.

Results

The original framework included 32 concepts in six groups. The 2019 version and the current 2022 version include 49 concepts in the same three main groups that we have used since 2016. There are now 10 subgroups or higher-level concepts. For each concept, there is an explanation including one or more examples, the basis for the concept, and implications. Over 600 references are cited that support the concepts, and over half of the references are systematic reviews.

Conclusions

There is a large body of evidence that supports the IHC key concepts and we have received few suggestions for changes since 2019.

Strategies for communicating scientific evidence on healthcare to managers and the population: a scoping review

BACKGROUND

Health evidence needs to be communicated and disseminated in a manner that is clearly understood by decision-makers. As an inherent component of health knowledge translation, communicating results of scientific studies, effects of interventions and health risk estimates, in addition to understanding key concepts of clinical epidemiology and interpreting evidence, represent a set of essential instruments to reduce the gap between science and practice. The advancement of digital and social media has reshaped the concept of health communication, introducing new, direct and powerful communication platforms and gateways between researchers and the public. The objective of this scoping review was to identify strategies for communicating scientific evidence in healthcare to managers and/or population.

METHODS

We searched Cochrane Library, Embase®, MEDLINE® and other six electronic databases, in addition to grey literature, relevant websites from related organizations for studies, documents or reports published from 2000, addressing any strategy for communicating scientific evidence on healthcare to managers and/or population.

RESULTS

Our search identified 24 598 unique records, of which 80 met the inclusion criteria and addressed 78 strategies. Most strategies focused on risk and benefit communication in health, were presented by textual format and had been implemented and somehow evaluated. Among the strategies evaluated and appearing to yield some benefit are (i) risk/benefit communication: natural frequencies instead of percentages, absolute risk instead relative risk and number needed to treat, numerical instead nominal communication, mortality instead survival; negative or loss content appear to be more effective than positive or gain content; (ii) evidence synthesis: plain languages summaries to communicate the results of Cochrane reviews to the community were perceived as more reliable, easier to find and understand, and better to support decisions than the original summaries; (iii) teaching/learning: the Informed Health Choices resources seem to be effective for improving critical thinking skills.

CONCLUSION

Our findings contribute to both the knowledge translation process by identifying communication strategies with potential for immediate implementation and to future research by recognizing the need to evaluate the clinical and social impact of other strategies to support evidence-informed policies. Trial registration protocol is prospectively available in MedArxiv (doi.org/10.1101/2021.11.04.21265922).

Development of a video-based evidence synthesis knowledge translation resource: Drawing on a user-centred design approach

Objectives

We aimed to develop a video animation knowledge translation (KT) resource to explain the purpose, use and importance of evidence synthesis to the public regarding healthcare decision-making.

Methods

We drew on a user-centred design approach to develop a spoken animated video (SAV) by conducting two cycles of idea generation, prototyping, user testing, analysis, and refinement. Six researchers identified the initial key messages of the SAV and informed the first draft of the storyboard and script. Seven members of the public provided input on this draft and the key messages through think-aloud interviews, which we used to develop an SAV prototype. Seven additional members of the public participated in think-aloud interviews while watching the video prototype. All members of the public also completed a questionnaire on perceived usefulness, desirability, clarity and credibility. We subsequently synthesised all data to develop the final SAV.

Results

Researchers identified the initial key messages as 1) the importance of evidence synthesis, 2) what an evidence synthesis is and 3) how evidence synthesis can impact healthcare decision-making. Members of the public rated the initial video prototype as 9/10 for usefulness, 8/10 for desirability, 8/10 for clarity and 9/10 for credibility. Using their guidance and feedback, we produced a three-and-a-half-minute video animation. The video was uploaded on YouTube, has since been translated into two languages, and viewed over 12,000 times to date.

Conclusions

Drawing on user-centred design methods provided a structured and transparent approach to the development of our SAV. Involving members of the public enhanced the credibility and usefulness of the resource. Future work could explore involving the public from the outset to identify key messages in developing KT resources explaining methodological topics. This study describes the systematic development of a KT resource with limited resources and provides transferrable learnings for others wishing to do similar.

Development of the informed health choices resources in four countries to teach primary school children to assess claims about treatment effects: a qualitative study employing a user-centred approach

Background

People of all ages are flooded with health claims about treatment effects (benefits and harms of treatments). Many of these are not reliable, and many people lack skills to assess their reliability. Primary school is the ideal time to begin to teach these skills, to lay a foundation for continued learning and enable children to make well-informed health choices, as they grow older. However, these skills are rarely being taught and yet there are no rigorously developed and evaluated resources for teaching these skills.

Objectives

To develop the Informed Health Choices (IHC) resources (for learning and teaching people to assess claims about the effects of treatments) for primary school children and teachers.

Methods

We prototyped, piloted, and user-tested resources in four settings that included Uganda, Kenya, Rwanda, and Norway. We employed a user-centred approach to designing IHC resources which entailed multiple iterative cycles of development (determining content scope, generating ideas, prototyping, testing, analysing and refining) based on continuous close collaboration with teachers and children.

Results

We identified 24 Key Concepts that are important for children to learn. We developed a comic book and a separate exercise book to introduce and explain the Key Concepts to the children, combining lessons with exercises and classroom activities. We developed a teachers' guide to supplement the resources for children.

Conclusion

By employing a user-centred approach to designing resources to teach primary children to think critically about treatment claims and choices, we developed learning resources that end users experienced as useful, easy to use and well-suited to use in diverse classroom settings.

Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about treatment effects: one-year follow up of a randomised trial

INTRODUCTION

Earlier, we designed and evaluated an educational mass media intervention for improving people's ability to think more critically and to assess the trustworthiness of claims (assertions) about the benefits and harms (effects) of treatments. The overall aims of this follow-up study were to evaluate the impact of our intervention 1 year after it was administered, and to assess retention of learning and behaviour regarding claims about treatments.

METHODS

We randomly allocated consenting parents to listen to either the Informed Health Choices podcast (intervention) or typical public service announcements about health issues (control) over 7-10 weeks. Each intervention episode explained how the trustworthiness of treatment claims can be assessed by using relevant key concepts of evidence-informed decision-making. Participants listened to two episodes per week, delivered by research assistants. We evaluated outcomes immediately, and a year after the intervention. Primary outcomes were mean score and the proportion with a score indicating a basic ability to apply the key concepts (> 11 out of 18 correct answers) on a tool measuring people's ability to critically appraise the trustworthiness of treatment claims. Skills decay/retention was estimated by calculating the relative difference between the follow-up and initial results in the intervention group, adjusting for chance. Statistical analyses were performed using R (R Core Team, Vienna, Austria; version 3.4.3).

RESULTS

After 1 year, the mean score for parents in the intervention group was 58.9% correct answers, compared to 52.6% in the control (adjusted mean difference of 6.7% (95% CI 3.3% to 10.1%)). In the intervention group, 47.2% of 267 parents had a score indicating a basic ability to assess treatment claims compared to 39.5% of 256 parents in the control (adjusted difference of 9.8% more parents (95% CI 0.9% to 18.9%). These represent relative reductions of 29% in the mean scores and 33% in the proportion of parents with a score indicating a basic ability to assess the trustworthiness of claims about treatment effects.

CONCLUSIONS

Although listening to the Informed Health Choices podcast initially led to a large improvement in the ability of parents to assess claims about the effects of treatments, our findings show that these skills decreased substantially over 1 year. More active practice could address the substantial skills decay observed over 1 year.

TRIAL REGISTRATION

Pan African Clinical Trial Registry (www.pactr.org), PACTR201606001676150. Registered on 12 June 2016.

Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona

Introduction: The Informed Health Choices (IHC) project has developed learning resources to teach primary school children (10 to 12-year-olds) to assess treatment claims and make informed health choices. The aim of our study is to explore both the students' and teachers' experience when using these resources in the context of Barcelona (Spain). Methods: During the 2019-2020 school year, we will conduct a pilot study with 4 th and 5 th-year primary school students (9 to 11-year-olds) from three schools in Barcelona. The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the students. The data collection will include: 1) initial assessment of the resources by the teachers, 2) non-participatory observations during the lessons, 3) semi-structured interviews with the students after a lesson, 4) assessment of the lessons by the teachers, 5) treatment claim assessment by the students, and 6) final assessment of the resources by the teachers. We will use ad hoc questionnaires and guides to register the data. We will perform a quantitative and qualitative analysis of the data to explore understandability, desirability, suitability, usefulness, facilitators and barriers of the resources. The most relevant results will be discussed and some recommendations on how to use, how to adapt (if needed), and how to implement the IHC resources to this context will be agreed. We will publish the study results in a peer-reviewed journal and in several Internet resources (web pages, electronic bulletins, and social media), and we will present them to the different users of interest in conferences, workshops, and meetings. Ethical considerations: The study protocol has obtained an approval exemption from the Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain).

Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona

Introduction: The Informed Health Choices (IHC) project has developed learning resources to teach primary school children (10 to 12-year-olds) to assess treatment claims and make informed health choices. The aim of our study is to explore both the students' and teachers' experience when using these resources in the context of Barcelona (Spain). Methods: During the 2019-2020 school year, we will conduct a pilot study with 4 th and 5 th-year primary school students (9 to 11-year-olds) from three schools in Barcelona. The intervention in the schools will include: 1) assessment of the IHC resources by the teachers before the lessons, 2) non-participatory observations during the lessons, 3) semi-structured interviews with the students after a lesson, 4) assessment of the lessons by the teachers after a lesson, 5) treatment claim assessment by the students at the end of the lessons, and 6) assessment of the IHC resources by the teachers at the end of the lessons. We will use ad hoc questionnaires and guides to register the data. We will perform a quantitative and qualitative analysis of the data to explore understandability, desirability, suitability, usefulness, facilitators and barriers of the resources. The most relevant results will be discussed and some recommendations on how to use, how to adapt (if needed), and how to implement the IHC resources to this context will be agreed. The findings of the contextualization activities could inform the design of a cluster-randomised trial, to determine the effectiveness of the IHC resources in this context prior to scaling-up its use. Ethical considerations: The study protocol has obtained an approval exemption from the Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain).

Learning to make informed health choices: Protocol for a pilot study in schools in Barcelona

Introduction: The Informed Health Choices (IHC) project has developed learning resources to teach primary school children (10 to 12-year-olds) to assess treatment claims and make informed health choices. The aim of our study is to explore both the students' and teachers' experience when using these resources in the context of Barcelona (Spain). Methods: During the 2019-2020 school year, we will conduct a pilot study with 4 th and 5 th-year primary school students (9 to 11-year-olds) from three schools in Barcelona. The intervention in the schools will include: 1) a workshop with the teachers, and 2) lessons to the students. The data collection will include: 1) assessment of the IHC resources by the teachers before the lessons, 2) non-participatory observations during the lessons, 3) semi-structured interviews with the students after a lesson, 4) assessment of the lessons by the teachers after a lesson, 5) treatment claim assessment by the students at the end of the lessons, and 6) assessment of the IHC resources by the teachers at the end of the lessons. We will use ad hoc questionnaires and guides to register the data. We will perform a quantitative and qualitative analysis of the data to explore understandability, desirability, suitability, usefulness, facilitators and barriers of the resources. The most relevant results will be discussed and some recommendations on how to use, how to adapt (if needed), and how to implement the IHC resources to this context will be agreed. The findings of the contextualization activities could inform the design of a cluster-randomised trial, to determine the effectiveness of the IHC resources in this context prior to scaling-up its use. Ethical considerations: The study protocol has obtained an approval exemption from the Ethics Committee of the Hospital de la Santa Creu i Sant Pau (Barcelona, Spain).

Informed health choices intervention to teach primary school children in low-income countries to assess claims about treatment effects: process evaluation

BACKGROUND

We developed the informed health choices (IHC) primary school resources to teach children how to assess the trustworthiness of claims about the effects of treatments. We evaluated these resources in a randomised trial in Uganda. This paper describes the process evaluation that we conducted alongside this trial.

OBJECTIVES

To identify factors affecting the implementation, impact and scaling up of the intervention; and potential adverse and beneficial effects of the intervention.

METHODS

All 85 teachers in the 60 schools in the intervention arm of the trial completed a questionnaire after each lesson and at the end of the term. We conducted structured classroom observations at all 60 schools. For interviews and focus groups, we purposively selected six schools. We interviewed district education officers, teachers, head teachers, children and their parents. We used a framework analysis approach to analyse the data.

RESULTS

Most of the participants liked the IHC resources and felt that the content was important. This motivated the teachers and contributed to positive attitudes. Although some teachers started out lacking confidence, many found that the children's enthusiasm for the lessons made them more confident. Nearly everyone interviewed thought that the children learnt something important and many thought that it improved their decision-making. The main barrier to scaling up use of the IHC resources that participants identified was the need to incorporate the lessons into the national curriculum.

CONCLUSION

The mostly positive findings reflect the trial results, which showed large effects on the children's and the teachers' critical appraisal skills. The main limitations of this evaluation are that the investigators were responsible for both developing and evaluating the intervention.