Development of the informed health choices resources in four countries to teach primary school children to assess claims about treatment effects: a qualitative study employing a user-centred approach

Background

People of all ages are flooded with health claims about treatment effects (benefits and harms of treatments). Many of these are not reliable, and many people lack skills to assess their reliability. Primary school is the ideal time to begin to teach these skills, to lay a foundation for continued learning and enable children to make well-informed health choices, as they grow older. However, these skills are rarely being taught and yet there are no rigorously developed and evaluated resources for teaching these skills.

Objectives

To develop the Informed Health Choices (IHC) resources (for learning and teaching people to assess claims about the effects of treatments) for primary school children and teachers.

Methods

We prototyped, piloted, and user-tested resources in four settings that included Uganda, Kenya, Rwanda, and Norway. We employed a user-centred approach to designing IHC resources which entailed multiple iterative cycles of development (determining content scope, generating ideas, prototyping, testing, analysing and refining) based on continuous close collaboration with teachers and children.

Results

We identified 24 Key Concepts that are important for children to learn. We developed a comic book and a separate exercise book to introduce and explain the Key Concepts to the children, combining lessons with exercises and classroom activities. We developed a teachers' guide to supplement the resources for children.

Conclusion

By employing a user-centred approach to designing resources to teach primary children to think critically about treatment claims and choices, we developed learning resources that end users experienced as useful, easy to use and well-suited to use in diverse classroom settings.

Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about treatment effects: one-year follow up of a randomised trial

INTRODUCTION

Earlier, we designed and evaluated an educational mass media intervention for improving people's ability to think more critically and to assess the trustworthiness of claims (assertions) about the benefits and harms (effects) of treatments. The overall aims of this follow-up study were to evaluate the impact of our intervention 1 year after it was administered, and to assess retention of learning and behaviour regarding claims about treatments.

METHODS

We randomly allocated consenting parents to listen to either the Informed Health Choices podcast (intervention) or typical public service announcements about health issues (control) over 7-10 weeks. Each intervention episode explained how the trustworthiness of treatment claims can be assessed by using relevant key concepts of evidence-informed decision-making. Participants listened to two episodes per week, delivered by research assistants. We evaluated outcomes immediately, and a year after the intervention. Primary outcomes were mean score and the proportion with a score indicating a basic ability to apply the key concepts (> 11 out of 18 correct answers) on a tool measuring people's ability to critically appraise the trustworthiness of treatment claims. Skills decay/retention was estimated by calculating the relative difference between the follow-up and initial results in the intervention group, adjusting for chance. Statistical analyses were performed using R (R Core Team, Vienna, Austria; version 3.4.3).

RESULTS

After 1 year, the mean score for parents in the intervention group was 58.9% correct answers, compared to 52.6% in the control (adjusted mean difference of 6.7% (95% CI 3.3% to 10.1%)). In the intervention group, 47.2% of 267 parents had a score indicating a basic ability to assess treatment claims compared to 39.5% of 256 parents in the control (adjusted difference of 9.8% more parents (95% CI 0.9% to 18.9%). These represent relative reductions of 29% in the mean scores and 33% in the proportion of parents with a score indicating a basic ability to assess the trustworthiness of claims about treatment effects.

CONCLUSIONS

Although listening to the Informed Health Choices podcast initially led to a large improvement in the ability of parents to assess claims about the effects of treatments, our findings show that these skills decreased substantially over 1 year. More active practice could address the substantial skills decay observed over 1 year.

TRIAL REGISTRATION

Pan African Clinical Trial Registry (www.pactr.org), PACTR201606001676150. Registered on 12 June 2016.

Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects, 1-year follow-up: a cluster-randomised trial

INTRODUCTION

We evaluated an intervention designed to teach 10- to 12-year-old primary school children to assess claims about the effects of treatments (any action intended to maintain or improve health). We report outcomes measured 1 year after the intervention.

METHODS

In this cluster-randomised trial, we included primary schools in the central region of Uganda that taught year 5 children (aged 10 to 12 years). We randomly allocated a representative sample of eligible schools to either an intervention or control group. Intervention schools received the Informed Health Choices primary school resources (textbooks, exercise books and a teachers' guide). The primary outcomes, measured at the end of the school term and again after 1 year, were the mean score on a test with two multiple-choice questions for each of the 12 concepts and the proportion of children with passing scores.

RESULTS

We assessed 2960 schools for eligibility; 2029 were eligible, and a random sample of 170 were invited to recruitment meetings. After recruitment meetings, 120 eligible schools consented and were randomly assigned to either the intervention group (n = 60 schools; 76 teachers and 6383 children) or the control group (n = 60 schools; 67 teachers and 4430 children). After 1 year, the mean score in the multiple-choice test for the intervention schools was 68.7% compared with 53.0% for the control schools (adjusted mean difference 16.7%; 95% CI, 13.9 to 19.5; P < 0.00001). In the intervention schools, 3160 (80.1%) of 3943 children who completed the test after 1 year achieved a predetermined passing score (≥ 13 of 24 correct answers) compared with 1464 (51.5%) of 2844 children in the control schools (adjusted difference, 39.5%; 95% CI, 29.9 to 47.5).

CONCLUSION

Use of the learning resources led to a large improvement in the ability of children to assess claims, which was sustained for at least 1 year.

TRIAL REGISTRATION

Pan African Clinical Trial Registry (www.pactr.org), PACTR201606001679337. Registered on 13 June 2016.

Development of mass media resources to improve the ability of parents of primary school children in Uganda to assess the trustworthiness of claims about the effects of treatments: a human-centred design approach

BACKGROUND

Claims about what we need to do to improve our health are everywhere. Most interventions simply tell people what to do, and do not empower them to critically assess health information. Our objective was to design mass media resources to enable the public to critically appraise the trustworthiness of claims about the benefits and harms of treatments and make informed health choices.

METHODS

Research was conducted between 2013 and 2016 across multiple iterative phases. Participants included researchers, journalists, parents, other members of the public. First, we developed a list of 32 key concepts that people need to understand to be able to assess the trustworthiness of claims about treatment effects. Next, we used a human-centred design approach, to generate ideas for resources for teaching the key concepts, and developed and user-tested prototypes through qualitative interviews. We addressed identified problems and repeated this process until we had a product that was deemed relevant and desirable by our target audience, and feasible to implement.

RESULTS

We generated over 160 ideas, mostly radio-based. After prototyping some of these, we found that a podcast produced collaboratively by health researchers and journalists was the most promising approach. We developed eight episodes of the Informed Health Choices podcast, a song on critical thinking about treatments and a reminder checklist. Early versions of the podcast were reportedly too long, boring and confusing. We shortened the episodes, included one key concept per episode, and changed to story-telling with skits. The final version of the podcast was found to be useful, understandable, credible and desirable.

CONCLUSION

We found many problems with various prototypes of mass media resources. Using a human-centred design approach, we overcame those problems. We have developed a guide to help others prepare similar podcasts.

Informed Health Choices media intervention for improving people's ability to critically appraise the trustworthiness of claims about treatment effects: a mixed-methods process evaluation of a randomised trial in Uganda

We developed the Informed Health Choices podcast to improve people's ability to assess claims about the effects of treatments. We evaluated the effects of the podcast in a randomised trial.

OBJECTIVES

We conducted this process evaluation to assess the fidelity of the intervention, identify factors that affected the implementation and impact of the intervention and could affect scaling up, and identify potential adverse and beneficial effects.

SETTING

The study was conducted in central Uganda in rural, periurban and urban settings.

PARTICIPANTS

We collected data on parents who were in the intervention arm of the Informed Health Choices study that evaluated an intervention to improve parents' ability to assess treatment effects.

PROCEDURES

We conducted 84 semistructured interviews during the intervention, 19 in-depth interviews shortly after, two focus group discussions with parents, one focus group discussion with research assistants and two in-depth interviews with the principal investigators. We used framework analysis to manage qualitative data, assessed the certainty of the findings using the GRADE-CERQual (Grading of Recommendations, Assessment, Development and Evaluations-Confidence in the Evidence from Reviews of Qualitative Research) approach, and organised findings in a logic model.

OUTCOMES

Proportion of participants listening to all episodes; factors influencing the implementation of the podcast; ways to scale up and any adverse and beneficial effects.

RESULTS

All participants who completed the study listened to the podcast as intended, perhaps because of the explanatory design and recruitment of parents with a positive attitude. This was also likely facilitated by the podcast being delivered by research assistants, and providing the participants with MP3 players. The podcast was reportedly clear, understandable, credible and entertaining, which motivated them to listen and eased implementation. No additional adverse effects were reported.

CONCLUSIONS

Participants experienced the podcast positively and were motivated to engage with it. These findings help to explain the short-term effectiveness of the intervention, but not the decrease in effectiveness over the following year.

Validación de un cuestionario para medir la habilidad de la población general para evaluar afirmaciones acerca de tratamientos médicos

Introducción

Todos los días, las personas se enfrentan a afirmaciones acerca de tratamientos en medios de comunicación, redes sociales o por viva voz.

Objetivo

Validar un cuestionario en español para medir las habilidades de un individuo para evaluar afirmaciones acerca de tratamientos.

Método

Veintidós preguntas de opción múltiple de la base de datos Claim Evaluation Tools fueron traducidas y aplicadas a 172 niños y 268 adultos. Mediante un modelo Rasch se exploró el ajuste promedio e individual por reactivo, el potencial comportamiento diferencial del reactivo (basado en el género, edad y modo de aplicación), la multidimensionalidad y la independencia local.

Resultados

El ajuste promedio por reactivo fue apropiado. Cuatro preguntas de opción múltiple mostraron pobre ajuste. La fiabilidad del cuestionario fue satisfactoria, con un índice de separación de 0.7. Las preguntas de opción múltiple fueron unidimensionales, y no hubo dependencia específica.

Conclusión

Se obtuvo un conjunto de 18 preguntas de opción múltiple con ajuste satisfactorio. El cuestionario es el primero disponible y validado en español para medir las habilidades de los individuos para evaluar afirmaciones acerca de tratamientos.

Introduction

Every day, people are faced with claims about treatment effects through mass media, social media, or by word of mouth.

Objective

To validate a Spanish-language questionnaire to measure the ability of an individual to assess claims about treatments effects.

Method

A set of 22 multiple choice questions taken from the claim evaluation tools database were translated and applied to 172 children and 268 adults. Using a Rasch model, overall and individual item-person fit was explored, as well as the potential item differential functioning (based on gender, age and mode of administration), multidimensionality and local independence.

Results

Overall item-person fit was appropriate. Four multiple-choice questions showed a poor fit. Reliability of the questionnaire was satisfactory with a person separation index of 0.7. Multiple-choice questions were unidimensional, and there was no specific dependency.

Conclusion

A set of 18 multiple-choice questions with satisfactory fit was obtained. This is the first available questionnaire validated in Spanish to measure individuals’ ability to assess claims about treatment effects.

Evaluating people's ability to assess treatment claims: Validating a test in Mandarin from Claim Evaluation Tools database

OBJECTIVE

To describe the psychometric testing using Rasch analysis of a test in Mandarin developed from the Claim Evaluation Tools database.

METHODS

We translated selected MCQs from the IHC Claim Evaluation Tools database to Mandarin and created a test including 24 MCQs covering 11 key concepts. We used purposeful sampling and surveyed children and adults in the Lanzhou area. In total 389 responses were entered into the analysis. We evaluated the psychometric properties of the test using Rasch analysis and the RUMM2030 software, testing for internal construct validity (multidimensionality), invariance of the items (item-person interaction), and item bias (differential item functioning).

RESULTS

Overall, the psychometric properties of the test were found to be satisfactory. Based on findings from the Rasch analysis, we deleted three MCQs with suboptimal fit.

CONCLUSIONS

The resulting test includes 21 MCQs and can be used in school and other teaching settings, in randomized trials evaluating outcomes of educational interventions, or in cross-sectional studies in Mandarin-speaking populations in China.

The James Lind Initiative: books, websites and databases to promote critical thinking about treatment claims, 2003 to 2018

BACKGROUND

The James Lind Initiative (JLI) was a work programme inaugurated by Iain Chalmers and Patricia Atkinson to press for better research for better health care. It ran between 2003 and 2018, when Iain Chalmers retired. During the 15 years of its existence, the JLI developed three strands of work in collaboration with the authors of this paper, and with others.

WORK THEMES

The first work strand involved developing a process for use by patients, carers and clinicians to identify shared priorities for research - the James Lind Alliance. The second strand was a series of articles, meetings, prizes and other developments to raise awareness of the massive amounts of avoidable waste in research, and of ways of reducing it. The third strand involved using a variety of approaches to promote better public and professional understanding of the importance of research in clinical practice and public health. JLI work on the first two themes has been addressed in previously published reports. This paper summarises JLI involvement during the 15 years of its existence in giving talks, convening workshops, writing books, and creating websites and databases to promote critical thinking about treatment claims.

CONCLUSION

During its 15-year life, the James Lind Initiative worked collaboratively with others to create free teaching and learning resources to help children and adults learn how to recognise untrustworthy claims about the effects of treatments. These resources have been translated in more than twenty languages, but much more could be done to support their uptake and wider use.

Key Concepts for assessing claims about treatment effects and making well-informed treatment choices

Background: The Informed Health Choices (IHC) Key Concepts are standards for judgement, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (evidence) used to support claims, and for making treatment choices. The list of concepts provides a framework, or starting point, for teachers, journalists and other intermediaries for identifying and developing resources (such as longer explanations, examples, games and interactive applications) to help people to understand and apply the concepts. The first version of the list was published in 2015 and has been updated yearly since then. We report here the changes that have been made from when the list was first published up to the current (2018) version. Methods: We developed the IHC Key Concepts by searching the literature and checklists written for the public, journalists, and health professionals; and by considering concepts related to assessing the certainty of evidence about the effects of treatments. We have revised the Key Concepts yearly, based on feedback and suggestions; and learning from using the IHC Key Concepts, other relevant frameworks, and adaptation of the IHC Key Concepts to other types of interventions besides treatments. Results: We have made many changes since the Key Concepts were first published in 2015. There are now 44 Key Concepts compared to the original 32; the concepts have been reorganised from six to three groups; we have added higher-level concepts in each of those groups; we have added short titles; and we have made changes to many of the concepts. Conclusions: We will continue to revise the IHC Key Concepts in response to feedback. Although we and others have found them helpful since they were first published, we anticipate that there are still ways in which they can be further improved. We welcome suggestions for how to do this.

Key Concepts for assessing claims about treatment effects and making well-informed treatment choices

Background: The Informed Health Choices (IHC) Key Concepts are standards for judgement, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (evidence) used to support claims, and for making treatment choices. The list of concepts provides a framework, or starting point, for teachers, journalists and other intermediaries for identifying and developing resources (such as longer explanations, examples, games and interactive applications) to help people to understand and apply the concepts. The first version of the list was published in 2015 and has been updated yearly since then. We report here the changes that have been made from when the list was first published up to the current (2018) version. Methods: We developed the IHC Key Concepts by searching the literature and checklists written for the public, journalists, and health professionals; and by considering concepts related to assessing the certainty of evidence about the effects of treatments. We have revised the Key Concepts yearly, based on feedback and suggestions; and learning from using the IHC Key Concepts, other relevant frameworks, and adaptation of the IHC Key Concepts to other types of interventions besides treatments. Results: We have made many changes since the Key Concepts were first published in 2015. There are now 44 Key Concepts compared to the original 32; the concepts have been reorganised from six to three groups; we have added higher-level concepts in each of those groups; we have added short titles; and we have made changes to many of the concepts. Conclusions: The IHC Key Concepts have proven useful in designing learning resources, evaluating them, and organising them. We will continue to revise the IHC Key Concepts in response to feedback. We welcome suggestions for how to do this.

Key Concepts for assessing claims about treatment effects and making well-informed treatment choices

Background: The Informed Health Choices (IHC) Key Concepts are standards for judgement, or principles for evaluating the trustworthiness of treatment claims and treatment comparisons (evidence) used to support claims, and for making treatment choices. The list of concepts provides a framework, or starting point, for teachers, journalists and other intermediaries for identifying and developing resources (such as longer explanations, examples, games and interactive applications) to help people to understand and apply the concepts. The first version of the list was published in 2015 and has been updated yearly since then. We report here the changes that have been made from when the list was first published up to the current (2018) version. Methods: We developed the IHC Key Concepts by searching the literature and checklists written for the public, journalists, and health professionals; and by considering concepts related to assessing the certainty of evidence about the effects of treatments. We have revised the Key Concepts yearly, based on feedback and suggestions; and learning from using the IHC Key Concepts, other relevant frameworks, and adaptation of the IHC Key Concepts to other types of interventions besides treatments. Results: We have made many changes since the Key Concepts were first published in 2015. There are now 44 Key Concepts compared to the original 32; the concepts have been reorganised from six to three groups; we have added higher-level concepts in each of those groups; we have added short titles; and we have made changes to many of the concepts. Conclusions: The IHC Key Concepts have proven useful in designing learning resources, evaluating them, and organising them. We will continue to revise the IHC Key Concepts in response to feedback. We welcome suggestions for how to do this.

Effect of requiring a general practitioner at scenes of serious injury: A systematic review

BACKGROUND

In Norway, each municipality is responsible for providing first line emergency healthcare, and it is mandatory to have a primary care physician/general practitioner on call continuously. This mandate ensures that a physician can assist patients and ambulance personnel at the site of severe injuries or illnesses. The compulsory presence of the general practitioner at the scene could affect different parts of patient treatment, and it might save resources by obviating resources from secondary healthcare, like pre-hospital anaesthesiologists and other specialized resources. This systematic review aimed to examine how survival, time spent at the scene, the choice of transport destination, assessment of urgency, the number of admissions, and the number of cancellations of specialized pre-hospital resources were affected by the presence of a general practitioner at the scene of a suspected severe injury.

METHODS

We searched for published and planned systematic reviews and primary studies in the Cochrane Library, Medline, Embase, OpenGrey, GreyLit and trial registries. The search was completed in December 2017. Two individuals independently screened the references and assessed the eligibility of all potentially relevant studies.

RESULTS

The search for systematic reviews and primary studies identified 5981 articles. However, no studies met the pre-defined inclusion criteria.

CONCLUSION

No studies met our inclusion criteria; consequently, it remains uncertain how the presence of a general practitioner at the injury scene might affect the selected outcomes.

Key Concepts for Informed Health Choices: a framework for helping people learn how to assess treatment claims and make informed choices

Many claims about the effects of treatments, though well intentioned, are wrong. Indeed, they are sometimes deliberately misleading to serve interests other than the well-being of patients and the public. People need to know how to spot unreliable treatment claims so that they can protect themselves and others from harm. The ability to assess the trustworthiness of treatment claims is often lacking. Acquiring this ability depends on being familiar with, and correctly applying, some key concepts, for example, that' association is not the same as causation.' The Informed Health Choices (IHC) Project has identified 36 such concepts and shown that people can be taught to use them in decision making. A randomised trial in Uganda, for example, showed that primary school children with poor reading skills could be taught to apply 12 of the IHC Key Concepts. The list of IHC Key Concepts has proven to be effective in providing a framework for developing and evaluating IHC resources to help children to think critically about treatment claims. The list also provides a framework for retrieving, coding and organising other teaching and learning materials for learners of any age. It should help teachers, researchers, clinicians, and patients to structure critical thinking about the trustworthiness of claims about treatment effects.

The plain language Glossary of Evaluation Terms for Informed Treatment choices (GET-IT) at www.getitglossary.org

Well-informed choices about how to improve health outcomes depend on access to reliable information, including research evidence. Many people (the public, health professionals and policymakers) have problems understanding some of the terminology used in describing evaluations of treatments. In this article, we describe the development of a glossary to provide plain language explanations of commonly used terms. We identified terms for inclusion in the glossary by screening multiple sources, including 14 other glossaries. Using an iterative process that included feedback and user testing, we designed an online glossary that is freely available at http://getitglossary.org. The glossary includes 242 commonly used terms. For each entry there is a suggested preferred term (plain language term), a short definition that can be accessed as a 'pop-up' within text, synonyms, a longer explanation and links to related terms. We also flag 11 terms that we advise against using because they are ambiguous or misinterpreted, with explanations of why we advise against using them. The glossary can be incorporated into documents, providing support for evidence-informed health-care decisions, and embedded in other websites serving a wide variety of functions.

Establishing a library of resources to help people understand key concepts in assessing treatment claims-The "Critical thinking and Appraisal Resource Library" (CARL)

BACKGROUND

People are frequently confronted with untrustworthy claims about the effects of treatments. Uncritical acceptance of these claims can lead to poor, and sometimes dangerous, treatment decisions, and wasted time and money. Resources to help people learn to think critically about treatment claims are scarce, and they are widely scattered. Furthermore, very few learning-resources have been assessed to see if they improve knowledge and behavior.

OBJECTIVES

Our objectives were to develop the Critical thinking and Appraisal Resource Library (CARL). This library was to be in the form of a database containing learning resources for those who are responsible for encouraging critical thinking about treatment claims, and was to be made available online. We wished to include resources for groups we identified as 'intermediaries' of knowledge, i.e. teachers of schoolchildren, undergraduates and graduates, for example those teaching evidence-based medicine, or those communicating treatment claims to the public. In selecting resources, we wished to draw particular attention to those resources that had been formally evaluated, for example, by the creators of the resource or independent research groups.

METHODS

CARL was populated with learning-resources identified from a variety of sources-two previously developed but unmaintained inventories; systematic reviews of learning-interventions; online and database searches; and recommendations by members of the project group and its advisors. The learning-resources in CARL were organised by 'Key Concepts' needed to judge the trustworthiness of treatment claims, and were made available online by the James Lind Initiative in Testing Treatments interactive (TTi) English (www.testingtreatments.org/category/learning-resources).TTi English also incorporated the database of Key Concepts and the Claim Evaluation Tools developed through the Informed Healthcare Choices (IHC) project (informedhealthchoices.org).

RESULTS

We have created a database of resources called CARL, which currently contains over 500 open-access learning-resources in a variety of formats: text, audio, video, webpages, cartoons, and lesson materials. These are aimed primarily at 'Intermediaries', that is, 'teachers', 'communicators', 'advisors', 'researchers', as well as for independent 'learners'. The resources included in CARL are currently accessible at www.testingtreatments.org/category/learning-resources.

CONCLUSIONS

We hope that ready access to CARL will help to promote the critical thinking about treatment claims, needed to help improve healthcare choices.

Effects of the Informed Health Choices podcast on the ability of parents of primary school children in Uganda to assess claims about treatment effects: a randomised controlled trial

BACKGROUND

As part of the Informed Health Choices project, we developed a podcast called The Health Choices Programme to help improve the ability of people to assess claims about the benefits and harms of treatments. We aimed to evaluate the effects of the podcast on the ability of parents of primary school children in Uganda to assess claims about the effects of treatments.

METHODS

We did this randomised controlled trial in central Uganda. We recruited parents of children aged 10-12 years who were in their fifth year of school at 35 schools that were participating in a linked trial of the Informed Health Choices primary school resources. The parents were randomly allocated (1:1), via a web-based random number generator with block sizes of four and six, to listen to either the Informed Health Choices podcast (intervention group) or typical public service announcements about health issues (control group). Randomisation was stratified by parents' highest level of formal education attained (primary school, secondary school, or tertiary education) and the allocation of their children's school in the trial of the primary school resources (intervention vs control). The primary outcome, measured after listening to the entire podcast, was the mean score and the proportion of parents with passing scores on a test with two multiple choice questions for each of nine key concepts essential to assessing claims about treatments (18 questions in total). We did intention-to-treat analyses. This trial is registered with the Pan African Clinical Trial Registry, number PACTR201606001676150.

FINDINGS

We recruited parents between July 21, 2016, and Oct 7, 2016. We randomly assigned 675 parents to the podcast group (n=334) or the public service announcement group (n=341); 561 (83%) participants completed follow-up. The mean score for parents in the podcast group was 67·8% (SD 19·6) compared with 52·4% (17·6) in the control group (adjusted mean difference 15·5%, 95% CI 12·5-18·6; p<0·0001). In the podcast group, 203 (71%) of 288 parents had a predetermined passing score (≥11 of 18 correct answers) compared with 103 (38%) of 273 parents in the control group (adjusted difference 34%, 95% CI 26-41; p<0·0001). No adverse events were reported.

INTERPRETATION

Listening to the Informed Health Choices podcast led to a large improvement in the ability of parents to assess claims about the effects of treatments. Future studies should assess the long-term effects of use of the podcast, the effects on actual health choices and outcomes, and how transferable our findings are to other countries.

FUNDING

Research Council of Norway.

Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial

BACKGROUND

Claims about what improves or harms our health are ubiquitous. People need to be able to assess the reliability of these claims. We aimed to evaluate an intervention designed to teach primary school children to assess claims about the effects of treatments (ie, any action intended to maintain or improve health).

METHODS

In this cluster-randomised controlled trial, we included primary schools in the central region of Uganda that taught year-5 children (aged 10-12 years). We excluded international schools, special needs schools for children with auditory and visual impairments, schools that had participated in user-testing and piloting of the resources, infant and nursery schools, adult education schools, and schools that were difficult for us to access in terms of travel time. We randomly allocated a representative sample of eligible schools to either an intervention or control group. Intervention schools received the Informed Health Choices primary school resources (textbooks, exercise books, and a teachers' guide). Teachers attended a 2 day introductory workshop and gave nine 80 min lessons during one school term. The lessons addressed 12 concepts essential to assessing claims about treatment effects and making informed health choices. We did not intervene in the control schools. The primary outcome, measured at the end of the school term, was the mean score on a test with two multiple-choice questions for each of the 12 concepts and the proportion of children with passing scores on the same test. This trial is registered with the Pan African Clinical Trial Registry, number PACTR201606001679337.

FINDINGS

Between April 11, 2016, and June 8, 2016, 2960 schools were assessed for eligibility; 2029 were eligible, and a random sample of 170 were invited to recruitment meetings. After recruitment meetings, 120 eligible schools consented and were randomly assigned to either the intervention group (n=60, 76 teachers and 6383 children) or control group (n=60, 67 teachers and 4430 children). The mean score in the multiple-choice test for the intervention schools was 62·4% (SD 18·8) compared with 43·1% (15·2) for the control schools (adjusted mean difference 20·0%, 95% CI 17·3-22·7; p<0·00001). In the intervention schools, 3967 (69%) of 5753 children achieved a predetermined passing score (≥13 of 24 correct answers) compared with 1186 (27%) of 4430 children in the control schools (adjusted difference 50%, 95% CI 44-55). The intervention was effective for children with different levels of reading skills, but was more effective for children with better reading skills.

INTERPRETATION

The use of the Informed Health Choices primary school learning resources, after an introductory workshop for the teachers, led to a large improvement in the ability of children to assess claims about the effects of treatments. The results show that it is possible to teach primary school children to think critically in schools with large student to teacher ratios and few resources. Future studies should address how to scale up use of the resources, long-term effects, including effects on actual health choices, transferability to other countries, and how to build on this programme with additional primary and secondary school learning resources.

FUNDING

Research Council of Norway.

Measuring ability to assess claims about treatment effects: a latent trait analysis of items from the 'Claim Evaluation Tools' database using Rasch modelling

BACKGROUND

The Claim Evaluation Tools database contains multiple-choice items for measuring people's ability to apply the key concepts they need to know to be able to assess treatment claims. We assessed items from the database using Rasch analysis to develop an outcome measure to be used in two randomised trials in Uganda. Rasch analysis is a form of psychometric testing relying on Item Response Theory. It is a dynamic way of developing outcome measures that are valid and reliable.

OBJECTIVES

To assess the validity, reliability and responsiveness of 88 items addressing 22 key concepts using Rasch analysis.

PARTICIPANTS

We administrated four sets of multiple-choice items in English to 1114 people in Uganda and Norway, of which 685 were children and 429 were adults (including 171 health professionals). We scored all items dichotomously. We explored summary and individual fit statistics using the RUMM2030 analysis package. We used SPSS to perform distractor analysis.

RESULTS

Most items conformed well to the Rasch model, but some items needed revision. Overall, the four item sets had satisfactory reliability. We did not identify significant response dependence between any pairs of items and, overall, the magnitude of multidimensionality in the data was acceptable. The items had a high level of difficulty.

CONCLUSION

Most of the items conformed well to the Rasch model's expectations. Following revision of some items, we concluded that most of the items were suitable for use in an outcome measure for evaluating the ability of children or adults to assess treatment claims.

Does the use of the Informed Healthcare Choices (IHC) primary school resources improve the ability of grade-5 children in Uganda to assess the trustworthiness of claims about the effects of treatments: protocol for a cluster-randomised trial

BACKGROUND

The ability to appraise claims about the benefits and harms of treatments is crucial for informed health care decision-making. This research aims to enable children in East African primary schools (the clusters) to acquire and retain skills that can help them make informed health care choices by improving their ability to obtain, process and understand health information. The trial will evaluate (at the individual participant level) whether specially designed learning resources can teach children some of the key concepts relevant to appraising claims about the benefits and harms of health care interventions (treatments).

METHODS

This is a two-arm, cluster-randomised trial with stratified random allocation. We will recruit 120 primary schools (the clusters) between April and May 2016 in the central region of Uganda. We will stratify participating schools by geographical setting (rural, semi-urban, or urban) and ownership (public or private). The Informed Healthcare Choices (IHC) primary school resources consist of a textbook and a teachers' guide. Each of the students in the intervention arm will receive a textbook and attend nine lessons delivered by their teachers during a school term, with each lesson lasting 80 min. The lessons cover 12 key concepts that are relevant to assessing claims about treatments and making informed health care choices. The second arm will carry on with the current primary school curriculum. We have designed the Claim Evaluation Tools to measure people's ability to apply key concepts related to assessing claims about the effects of treatments and making informed health care choices. The Claim Evaluation Tools use multiple choice questions addressing each of the 12 concepts covered by the IHC school resources. Using the Claim Evaluation Tools we will measure two primary outcomes: (1) the proportion of children who 'pass', based on an absolute standard and (2) their average scores.

DISCUSSION

As far as we are aware this is the first randomised trial to assess whether key concepts needed to judge claims about the effects of treatment can be taught to primary school children. Whatever the results, they will be relevant to learning how to promote critical thinking about treatment claims. Trial status: the recruitment of study participants was ongoing at the time of manuscript submission.

TRIAL REGISTRATION

Pan African Clinical Trial Registry, trial identifier: PACTR201606001679337 . Registered on 13 June 2016.

Measuring ability to assess claims about treatment effects: the development of the 'Claim Evaluation Tools'

OBJECTIVES

To describe the development of the Claim Evaluation Tools, a set of flexible items to measure people's ability to assess claims about treatment effects.

SETTING

Methodologists and members of the community (including children) in Uganda, Rwanda, Kenya, Norway, the UK and Australia.

PARTICIPANTS

In the iterative development of the items, we used purposeful sampling of people with training in research methodology, such as teachers of evidence-based medicine, as well as patients and members of the public from low-income and high-income countries. Development consisted of 4 processes: (1) determining the scope of the Claim Evaluation Tools and development of items; (2) expert item review and feedback (n=63); (3) cognitive interviews with children and adult end-users (n=109); and (4) piloting and administrative tests (n=956).

RESULTS

The Claim Evaluation Tools database currently includes a battery of multiple-choice items. Each item begins with a scenario which is intended to be relevant across contexts, and which can be used for children (from age 10  and above), adult members of the public and health professionals. People with expertise in research methods judged the items to have face validity, and end-users judged them relevant and acceptable in their settings. In response to feedback from methodologists and end-users, we simplified some text, explained terms where needed, and redesigned formats and instructions.

CONCLUSIONS

The Claim Evaluation Tools database is a flexible resource from which researchers, teachers and others can design measurement instruments to meet their own requirements. These evaluation tools are being managed and made freely available for non-commercial use (on request) through Testing Treatments interactive (testingtreatments.org).

TRIAL REGISTRATION NUMBERS

PACTR201606001679337 and PACTR201606001676150; Pre-results.

[Patient volume and quality in surgery for abdominal aortic aneurysm]

BACKGROUND Patient volume is assumed to affect quality, whereby complex procedures are best performed by those who perform them frequently. We have conducted a systematic review of the research on the association between patient volume and quality of vascular surgery. In this article we describe the outcomes for abdominal aortic aneurysm surgery.MATERIAL AND METHOD We undertook systematic searches in relevant databases. We searched for systematic reviews, and randomised and observational studies. The search was concluded in December 2015. We have summarised the results descriptively and assessed the overall quality of the evidence.RESULTS Forty-six observational studies fulfilled our inclusion criteria. We found a possible association for both hospital and surgeon volume. Higher patient volume may possibly be associated with lower 30-day mortality and lower hospital mortality for both open and endovascular surgery. Although the association appears to apply to both elective and acute hospitalisations, there is greater uncertainty with regard to the most ill patients. For hospital volume there may also be fewer complications for open and endovascular surgery, as well as for all surgery assessed as a whole. We considered the evidence base to be medium to very low quality.INTERPRETATION We found a possible correlation between patient volume and quality indicators such as mortality and complications. It may be advantageous to allocate planned procedures to institutions and surgeons with high volume, while this is less certain with regard to acute hospitalisations.

Can an educational podcast improve the ability of parents of primary school children to assess the reliability of claims made about the benefits and harms of treatments: study protocol for a randomised controlled trial

Background

Claims made about the effects of treatments are very common in the media and in the population more generally. The ability of individuals to understand and assess such claims can affect their decisions and health outcomes. Many people in both low- and high-income countries have inadequate aptitude to assess information about the effects of treatments. As part of the Informed Healthcare Choices project, we have prepared a series of podcast episodes to help improve people’s ability to assess claims made about treatment effects. We will evaluate the effect of the Informed Healthcare Choices podcast on people’s ability to assess claims made about the benefits and harms of treatments. Our study population will be parents of primary school children in schools with limited educational and financial resources in Uganda.

Methods

This will be a two-arm, parallel-group, individual-randomised trial. We will randomly allocate consenting participants who meet the inclusion criteria for the trial to either listen to nine episodes of the Informed Healthcare Choices podcast (intervention) or to listen to nine typical public service announcements about health issues (control). Each podcast includes a story about a treatment claim, a message about one key concept that we believe is important for people to be able to understand to assess treatment claims, an explanation of how that concept applies to the claim, and a second example illustrating the concept.

We designed the Claim Evaluation Tools to measure people’s ability to apply key concepts related to assessing claims made about the effects of treatments and making informed health care choices. The Claim Evaluation Tools that we will use include multiple-choice questions addressing each of the nine concepts covered by the podcast. Using the Claim Evaluation Tools, we will measure two primary outcomes: (1) the proportion that ‘pass’, based on an absolute standard and (2) the average score.

Discussion

As far as we are aware this is the first randomised trial to assess the use of mass media to promote understanding of the key concepts needed to judge claims made about the effects of treatments.

Trial registration

Pan African Clinical Trials Registry, PACTR201606001676150. Registered on 12 June 2016. http://www.pactr.org/ATMWeb/appmanager/atm/atmregistry?dar=true&tNo=PACTR201606001676150.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-016-1745-y) contains supplementary material, which is available to authorized users.

Interventions and assessment tools addressing key concepts people need to know to appraise claims about treatment effects: a systematic mapping review

BACKGROUND

People's ability to appraise claims about treatment effects is crucial for informed decision-making. Our objective was to systematically map this area of research in order to (a) provide an overview of interventions targeting key concepts that people need to understand to assess treatment claims and (b) to identify assessment tools used to evaluate people's understanding of these concepts. The findings of this review provide a starting point for decisions about which key concepts to address when developing new interventions, and which assessment tools should be considered.

METHODS

We conducted a systematic mapping review of interventions and assessment tools addressing key concepts important for people to be able to assess treatment claims. A systematic literature search was done by a reserach librarian in relevant databases. Judgement about inclusion of studies and data collection was done by at least two researchers. We included all quantitative study designs targeting one or more of the key concepts, and targeting patients, healthy members of the public, and health professionals. The studies were divided into four categories: risk communication and decision aids, evidence-based medicine and critical appraisal, understanding of controlled trials, and science education. Findings were summarised descriptively.

RESULTS

We included 415 studies, of which the interventions and assessment tools we identified included only a handful of the key concepts. The most common key concepts in interventions were "Treatments usually have beneficial and harmful effects," "Treatment comparisons should be fair," "Compare like with like," and "Single studies can be misleading." A variety of assessment tools were identified, but only four assessment tools included 10 or more key concepts.

CONCLUSIONS

There is great potential for developing learning and assessment tools targeting key concepts that people need to understand to assess claims about treatment effects. There is currently no instrument covering assessment of all these key concepts.

A case study of the organisation of research knowledge by systematic reviews within the mental health care field

BACKGROUND

Our aim was to explore how systematic reviews of the effects of complex interventions within the field of organisation of mental health care organised research knowledge.

METHODS

We looked up references included in 14 mental health care reviews in the database Epistemonikos to examine overlap between reviews. We classified topic concordance between study reports and reviews as similar, narrower, broader or other topic to indicate how well the research knowledge was organised.

FINDINGS

We examined 182 comparisons between the theme of study reports and reviews. In 100 (55%) of the 182 comparisons, the review had a broader theme, 18 (10%) narrower, 34 (19%) similar and in 30 (16%), the review's theme was classified as 'other'. The content analysis indicated that there existed (inconsistent) overlapping of study reports between different topics.

CONCLUSIONS

Within the field organisation of mental health care, there exists some unclear and inconsistent organisation of current knowledge. This may diminish the validity and reliability of systematic reviews. It is important that review authors take care in defining the review question precisely, conduct thorough literature searches, consider each study's hypothesis, disciplinary tradition and context, and if necessary, examine which other reviews have included a study in question.

Key concepts that people need to understand to assess claims about treatment effects

OBJECTIVE

People are confronted with claims about the effects of treatments and health policies daily. Our objective was to develop a list of concepts that may be important for people to understand when assessing claims about treatment effects.

METHODS

An initial list of concepts was generated by the project team by identifying key concepts in literature and tools written for the general public, journalists, and health professionals, and consideration of concepts related to assessing the certainty of evidence for treatment effects. We invited key researchers, journalists, teachers and others with expertise in health literacy and teaching or communicating evidence-based health care to patients to act as the project's advisory group.

RESULTS

Twenty-nine members of the advisory group provided feedback on the list of concepts and judged the list to be sufficiently complete and organised appropriately. The list includes 32 concepts divided into six groups: (i) Recognising the need for systematic reviews of fair tests, (ii) Judging whether a comparison of treatments is fair comparison, (iii) Understanding the role of chance, (iv) Considering all the relevant fair comparisons, (v) Understanding the results of fair comparisons of treatments, (vi) Judging whether fair comparisons of treatments are relevant.

CONCLUSION

The concept list provides a starting point for developing and evaluating resources to improve people's ability to assess treatment effects. The concepts are considered to be universally relevant, and include considerations that can help people assess claims about the effects of treatments, including claims that are found in mass media reports, in advertisements and in personal communication.

Development of a complex intervention to improve health literacy skills

BACKGROUND

Providing insight into the developmental processes involved in building interventions is an important way to ensure methodological transparency and inform future research efforts. The objective of this study was to describe the development of a web portal designed to improve health literacy skills among the public.

METHODS

The web portal was tailored to address three key barriers to obtaining information, using the conceptual frameworks of shared decision-making and evidence-based practice and based on explicit criteria for selecting the content and form of the intervention.

RESULTS

The web portal targeted the general public and took the form of structured sets of tools. Content included: an introduction to research methods, help on how to find evidence-based health information efficiently based on the steps of evidence-based practice, an introduction to critical appraisal, information about patient participation rights in decision-making, and a decision aid for consultations.

CONCLUSIONS

The web portal was designed in a systematic and transparent way and address key barriers to obtaining and acting upon reliable health information. The web portal provides open access to the tools and can be used independently by health care users, or during consultations with health professionals.

Pasienten som medvirker og kunnskapshåndterer

<p>En forutsetning for en kunnskapsbasert helsetjeneste er at pasienter tas med i beslutninger om egen og andres helse. Dette forutsetter tilgang på pålitelig kunnskap og kompetanse i å ta den i bruk. Den sentrale arenaen for pasientmedvirkning er møtet mellom pasient og helsearbeider, men andre viktige områder for medvirkning omfatter beslutninger på system- og politisk nivå. Tiltak for å fremme pasienters tilgang til pålitelig helseinformasjon kan deles i tre kategorier. Den første dreier seg om at forskere, helsepersonell og andre fagfolk kvalitetssikrer, oppsummerer og formidler den beste tilgjengelige kunnskapen. Den andre relaterer hovedsakelig til nettbasert informasjon, der informasjonen blir filtrert enten manuelt av eksperter eller ved bruk av automatiske søkemotorer. Tiltak i den tredje kategorien er opplæring av pasienter, for eksempel i form av kurs, eller i bruk av sjekklister og andre ressurser, for å styrke deres kompetanse i kritisk vurdering av forskning.</p><p>Austvoll-Dahlgren A, Johansen M. <strong>The patient as participant and knowledge manager</strong>. <em>Nor J Epidemiol</em> 2013; <em>23</em> (2): 225-230.</p><p><strong>ENGLISH SUMMARY</strong></p><p>A prerequisite for an evidence based health care is that patients take part in decisions about their own and others' health care. This requires access to reliable information and competence in applying it. The central arena for patient participation is the meeting between patient and health professional, but other important areas of involvement include decisions at the system and policy levels. Measures to promote patient access to reliable health information can be divided into three categories. The first entails that researchers, health personnel and other professionals quality assess, summarise and communicate the best available evidence. The second relates mainly to online information and how information is filtered either manually by experts or by automated search engines. The third types of measures are those where patients are trained, for example in the form of courses, or by the use of checklists and other resources, to enhance their skills in critical appraisal of research.</p>

Evaluation of a web portal for improving public access to evidence-based health information and health literacy skills: a pragmatic trial

Background

Using the conceptual framework of shared decision-making and evidence-based practice, a web portal was developed to serve as a generic (non disease-specific) tailored intervention to improve the lay public's health literacy skills.

Objective

To evaluate the effects of the web portal compared to no intervention in a real-life setting.

Methods

A pragmatic randomised controlled parallel trial using simple randomisation of 96 parents who had children aged <4 years. Parents were allocated to receive either access to the portal or no intervention, and assigned three tasks to perform over a three-week period. These included a searching task, a critical appraisal task, and reporting on perceptions about participation. Data were collected from March through June 2011.

Results

Use of the web portal was found to improve attitudes towards searching for health information. This variable was identified as the most important predictor of intention to search in both samples. Participants considered the web portal to have good usability, usefulness, and credibility. The intervention group showed slight increases in the use of evidence-based information, critical appraisal skills, and participation compared to the group receiving no intervention, but these differences were not statistically significant.

Conclusion

Despite the fact that the study was underpowered, we found that the web portal may have a positive effect on attitudes towards searching for health information. Furthermore, participants considered the web portal to be a relevant tool. It is important to continue experimenting with web-based resources in order to increase user participation in health care decision-making.

Trial Registration

ClinicalTrials.gov NCT01266798

Public health nurses' barriers and facilitators to the use of research in consultations about childhood vaccinations

The aim of this study was to describe sources of information, as well as barriers and facilitators to the use of research during consultations by public health nurses concerning childhood vaccinations. The study was conducted using semi-structured focus group interviews in a grounded theory approach. Overall 16 public health nurses participated into three focus groups conducted in 2008. We found that the public health nurses' most important sources of information were the National guidelines and other information issued by the National Institute of Public Health. Although they argued that research was important for being able to base practice on solid information, for their own professional development and for meeting parents' demands, they were reluctant to search for such information themselves. This was explained by beliefs about their own role, limited critical appraisal skills and perceived capacity. We conclude with that insight into how knowledge is produced and how to obtain such information is not only a necessity for good quality health care and professional development, but is also a way to address challenges such as time, overload of information and the ability to answer questions parents may have. More emphasis should be given to empowering public health nurses so they can find and critically appraise research, and this should be an integrated part of practice.

Pharmaceutical policies: effects of cap and co-payment on rational drug use

BACKGROUND

Growing expenditures on prescription drugs represent a major challenge to many health systems. Cap and co-payment (direct cost-share) policies are intended as an incentive to deter unnecessary or marginal utilisation, and to reduce third-party payer expenditures by shifting parts of the financial burden from the insurer to patients, thus increasing their financial responsibility for prescription drugs. Direct patient drug payment policies include caps (maximum number of prescriptions or drugs that are reimbursed), fixed co-payments (patients pay a fixed amount per prescription or drug), coinsurance (patients pay a percent of the price), ceilings (patients pay the full price or part of the cost up to a ceiling, after which drugs are free or available at reduced cost), and tier co-payments (differential co-payments usually assigned to generic and brand drugs).

OBJECTIVES

To determine the effects of cap and co-payment (cost-sharing) policies on drug use, healthcare utilisation, health outcomes and costs (expenditures).

SEARCH STRATEGY

We searched the following databases and web sites: Effective Practice and Organisation of Care Group Register (date of last search: 6 September 07), Cochrane Central Register of Controlled Trials (27 August 07), MEDLINE (29 August 07), EMBASE (29 August 07), NHS EED (27 August 07), ISI Web of Science (09 January 07), CSA Worldwide Political Science Abstracts (21 October 03), EconLit (23 October 03), SIGLE (12 November 03), INRUD (21 November 03), PAIS International (23 March 04), International Political Science Abstracts (09 January 04), PubMed (25 February 04), NTIS (03 March 04), IPA (22 April 04), OECD Publications & Documents (30 August 05), SourceOECD (30 August 05), World Bank Documents & Reports (30 August 05), World Bank e-Library (04 May 05), JOLIS (22 February 06), Global Jolis (22 February 06), WHOLIS(22 February 06), WHO web site browsed (25 August 05).

SELECTION CRITERIA

We defined policies in this review as laws, rules, or financial or administrative orders made by governments, non-government organisations or private insurers. We included randomised controlled trials, non-randomised controlled trials, interrupted time series analyses, repeated measures studies and controlled before-after studies of cap or co-payment policies for a large jurisdiction or system of care. To be included, a study had to include an objective measure of at least one of the following outcomes: drug use, healthcare utilisation, health outcomes or costs (expenditures).

DATA COLLECTION AND ANALYSIS

Two authors independently extracted data and assessed study limitations. We undertook quantitative analysis of time series data for studies with sufficient data.

MAIN RESULTS

We included 30 evaluations (in 21 studies). Of these, 11 evaluated fixed co-payment, six evaluated coinsurance with a ceiling, four evaluated caps, three evaluated fixed co-payment with a ceiling, three evaluated tier co-payment, one evaluated ceiling, one evaluated fixed co-payment and coinsurance with a ceiling, and one evaluated a fixed co-payment with a cap. Most of the included evaluations were observational studies and the quality of the evidence was found to be generally low to moderate. Introducing or increasing direct co-payments reduced drug use and saved plan drug expenditures across studies. Patients responded through drug discontinuation or by cost-sharing. Investigators found reductions for life-sustaining drugs or drugs that are important in treating chronic conditions as well as other drugs. Few studies reported on the effects on health and healthcare utilisation. One study found adverse effects on health through increased healthcare utilisation when a cap was introduced in a vulnerable population. No statistically significant change in use of healthcare services was found in other studies when a cap was introduced on a drug considered over-prescribed in a vulnerable population, or following a shift from a two-tier to a three-tier system with increased co-payments for tier-1 drugs in a general population.

AUTHORS' CONCLUSIONS

We found a diversity of cap and co-payment policies. Poor reporting of the intensity of interventions and differences in setting, populations and interventions made it difficult to make comparisons across studies. Cap and co-payment polices can reduce drug use and save plan drug expenditures. However, although insufficient data on health outcomes were available, substantial reductions in the use of life-sustaining drugs or drugs that are important in treating chronic conditions may have adverse effects on health, and as a result increase the use of healthcare services and overall expenditures. Direct payments are less likely to cause harm if only non-essential drugs are included or exemptions are built in to ensure that patients receive needed medical care.

A televised, web-based randomised trial of an herbal remedy (valerian) for insomnia

BACKGROUND

This trial was conducted as part of a project that aims to enhance public understanding and use of research in decisions about healthcare by enabling viewers to participate in research and to follow the process, through television reports and on the web. Valerian is an herbal over-the-counter drug that is widely used for insomnia. Systematic reviews have found inconsistent and inconclusive results about its effects.

METHODS

Participants were recruited through a weekly nationally televised health program in Norway. Enrolment and data collection were over the Internet. 405 participants who were 18 to 75 years old and had insomnia completed a two week diary-keeping run-in period without treatment and were randomised and mailed valerian or placebo tablets for two weeks. All participants and investigators were blind to treatment until after the analysis was completed.

FINDINGS

For the primary outcome of a minimally important improvement in self-reported sleep quality (> or = 0.5 units on a 7 point scale), the difference between the valerian group (29%) and the placebo group (21%) was not statistically significant (difference 7.5%; 95% CI-0.9 to 15.9; p = 0.08). On the global self-assessment question at the end of the treatment period 5.5% (95% CI 0.2 to 10.8) more participants in the valerian group perceived their sleep as better or much better (p = 0.04). There were similar trends favouring the valerian group for night awakenings (difference = 6.0%, 95% CI-0.5 to 12.5) and sleep duration (difference = 7.5%, 95% CI-1.0 to 16.1). There were no serious adverse events and no important or statistically significant differences in minor adverse events.

INTERPRETATION

Based on this and previous studies, valerian appears to be safe, but with modest beneficial effects at most on insomnia compared to placebo. The combined use of television and the Internet in randomised trials offers opportunities to answer questions about the effects of health care interventions and to improve public understanding and use of randomised trials.

TRIAL REGISTRATION

Controlled-Trials.com ISRCTN72748991.

Pharmaceutical policies: effects of financial incentives for prescribers

BACKGROUND

Pharmaceuticals, while central to medical therapy, pose a significant burden to health care budgets. Therefore regulations to control prescribing costs and improve quality of care are implemented increasingly. These include the use of financial incentives for prescribers, namely increased financial accountability using budgets and performance based payments.

OBJECTIVES

To determine the effects on drug use, healthcare utilisation, health outcomes and costs (expenditures) of policies, that intend to affect prescribers by means of financial incentives.

SEARCH STRATEGY

We searched the following databases and web sites: Effective Practice and Organisation of Care Group Register (August 2003), Cochrane Central Register of Controlled Trials (October 2003), MEDLINE (October 2005), EMBASE (October 2005), and other databases.

SELECTION CRITERIA

Policies were defined as laws, rules, financial and administrative orders made by governments, non-government organisations or private insurers. One of the following outcomes had to be reported: drug use, healthcare utilisation, health outcomes, and costs. The study had to be a randomised or non-randomised controlled trial, interrupted time series analysis, repeated measures study or controlled before-after study evaluating financial incentives for prescribers introduced for a jurisdiction or healthcare system.

DATA COLLECTION AND ANALYSIS

Two review authors independently extracted data and assessed study limitations.

MAIN RESULTS

Thirteen evaluations of budgetary policies and none of performance based payments met our inclusion criteria. Ten studies evaluated general practice fundholding in the UK, one the Irish Indicative Drug Target Savings Scheme (IDTSS) and two evaluated German drug budgets for physicians in private practice. The interrupted time series analyses had some limitations. All the controlled before-after studies (all from the UK) had serious limitations. Drug expenditure (per item and per patient) and prescribed drug volume decreased with budgets in all three countries. Evidence indicated increased use of generic drugs in the UK and Ireland, but was inconclusive on the use of new and expensive drugs. We found no clear evidence of increased health care utilisation and no studies reporting effects on health. Administration costs were not reported. No studies on the effects of performance-based payments or other policies met our inclusion criteria.

AUTHORS' CONCLUSIONS

Based on the evidence in this review from three Western European countries, drug budgets for physicians in private practice can limit drug expenditure by limiting the volume of prescribed drugs, increasing the use of generic drugs or both. Since the majority of studies included were found to have serious limitations, these results should be interpreted with care.