Can automated content analysis be used to assess and improve the use of evidence in mental health policy? A systematic review

BACKGROUND

This review assesses the utility of applying an automated content analysis method to the field of mental health policy development. We considered the possibility of using the Wordscores algorithm to assess research and policy texts in ways that facilitate the uptake of research into mental health policy.

METHODS

The PRISMA framework and the McMaster appraisal tools were used to systematically review and report on the strengths and limitations of the Wordscores algorithm. Nine electronic databases were searched for peer-reviewed journal articles published between 2003 and 2016. Inclusion criteria were (1) articles had to be published in public health, political science, social science or health services disciplines; (2) articles had to be research articles or opinion pieces that used Wordscores; and (3) articles had to discuss both strengths and limitations of using Wordscores for content analysis.

RESULTS

The literature search returned 118 results. Twelve articles met the inclusion criteria. These articles explored a range of policy questions and appraised different aspects of the Wordscores method.

DISCUSSION

Following synthesis of the material, we identified the following as potential strengths of Wordscores: (1) the Wordscores algorithm can be used at all stages of policy development; (2) it is valid and reliable; (3) it can be used to determine the alignment of health policy drafts with research evidence; (4) it enables existing policies to be revised in the light of research; and (5) it can determine whether changes in policy over time were supported by the evidence. Potential limitations identified were (1) decreased accuracy with short documents, (2) words constitute the unit of analysis and (3) expertise is needed to choose 'reference texts'.

CONCLUSIONS

Automated content analysis may be useful in assessing and improving the use of evidence in mental health policies. Wordscores is an automated content analysis option for comparing policy and research texts that could be used by both researchers and policymakers.

An appraisal of practice guidelines for smoking cessation in people with severe mental illness

OBJECTIVE

To review the quality of current smoking cessation guidelines that include recommendations for people with severe mental illness.

METHODS

A systematic search of scientific databases, central government health authority websites, psychiatry peak bodies, guideline clearing houses and Google was undertaken for relevant smoking cessation guidelines. Three reviewers independently assessed guideline quality using the AGREE II (Appraisal of Guidelines for REsearch and Evaluation II) instrument. Two reviewers extracted recommendations specific to smokers with severe mental illness.

RESULTS

Thirteen guidelines met the inclusion criteria. Seven guidelines scored ⩾60% in at least four domains. Median scores for 'Editorial independence', 'Rigour of development', 'Stakeholder Involvement' and 'Applicability' were less than 60%. The highest median scores were for 'Scope and purpose' (87%, 69-96%) and 'Clarity of presentation' (87%, 56-98%). 'Editorial independence' (33.3%, 0-86%) and 'Rigour of development' (54%, 11-92%) had the lowest median domain scores. The guidelines varied greatly in their recommendations but the majority recommended nicotine replacement therapy, bupropion or varenicline as first-line pharmacotherapy, along with behavioural support.

CONCLUSION

Many guidelines did not adequately report their methods or the competing interests of the authors. Future guidelines development may benefit from more specifically addressing AGREE II criteria and the needs of smokers with severe mental illness.

How do we define the policy impact of public health research? A systematic review

Background

In order to understand and measure the policy impact of research we need a definition of research impact that is suited to the task. This article systematically reviewed both peer-reviewed and grey literature for definitions of research impact to develop a definition of research impact that can be used to investigate how public health research influences policy.

Method

Keyword searches of the electronic databases Web of Science, ProQuest, PubMed, EMBASE, CINAHL, Informit, PsycINFO, The Cochrane Database of Systematic Reviews and Google Scholar were conducted between August 2015 and April 2016. Keywords included ‘definition’ and ‘policy’ and ‘research impact’ or ‘research evidence’. The search terms ‘health’, public health’ or ‘mental health’ and ‘knowledge transfer’ or ‘research translation’ were used to focus the search on relevant health discipline approaches. Studies included in the review described processes, theories or frameworks associated with public health, health services or mental health policy.

Results

We identified 108 definitions in 83 publications. The key findings were that literature on research impact is growing, but only 23% of peer-reviewed publications on the topic explicitly defined the term and that the majority (76%) of definitions were derived from research organisations and funding institutions. We identified four main types of definition, namely (1) definitions that conceptualise research impacts in terms of positive changes or effects that evidence can bring about when transferred into policies (example Research Excellence Framework definition), (2) definitions that interpret research impacts as measurable outcomes (Research Councils UK), and (3) bibliometric and (4) use-based definitions. We identified four constructs underpinning these definitions that related to concepts of contribution, change, avenues and levels of impact.

Conclusion

The dominance of bureaucratic definitions, the tendency to discuss but not define the concept of research impact, and the heterogeneity of definitions confirm the need for conceptual clarity in this area. We propose a working definition of research impact that can be used in a range of health policy contexts.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-017-0247-z) contains supplementary material, which is available to authorized users.

Use of videos to support teaching and learning of clinical skills in nursing education: A review

Information and communications technology is influencing the delivery of education in tertiary institutions. In particular, the increased use of videos for teaching and learning clinical skills in nursing may be a promising direction to pursue, yet we need to better document the current research in this area of inquiry. The aim of this paper was to explore and document the current areas of research into the use of videos to support teaching and learning of clinical skills in nursing education. The four main areas of current and future research are effectiveness, efficiency, usage, and quality of videos as teaching and learning materials. While there is a clear need for additional research in the area, the use of videos seems to be a promising, relevant, and increasingly used instructional strategy that could enhance the quality of clinical skills education.