Impact of clinical and health services research projects on decision-making: a qualitative study

Contribution of Arab world in transplant research: A PubMed-based bibliometric analysis

BACKGROUND

The Arab world consists of 22 countries situated in the Middle East and North Africa, tied together by linguistic and cultural bonds. Over numerous past decades, this region tended to fall behind regarding biomedical research. Organ, tissue, and cell transplantation are life-saving and life-improving treatments for various diseases. Given the positive correlation between research activity and the improvement of the clinical frame of transplantation, this article intended to examine the regional activity of transplantation research between 2005 and 2019.

METHODS

Using the PubMed database, the number of transplantation-related articles published by each country, and regarding 26 different procedures, was assessed. The results were normalized with respect to each country's average population & average Gross Domestic Product (GDP). In addition, the co-occurrence of keywords and the co-authorships were analyzed by VOS Viewer.

RESULTS

Arab countries contributed to 1.25% of total transplantation publications. The number of transplantation publications had an inflection in 2013, with the last six years alone contributing to 65.27% of the overall regional transplantation-related publications. Kuwait and Lebanon ranked first in terms of publications per million persons, while Egypt ranked first in terms of publications per national GDP. Stem cell, bone & kidney transplantations had the highest number of Arab transplantation-related publications. Low levels of collaboration between authors and organizations were observed, besides a modest but increasing trend towards experimental work on animals and newer therapies.

CONCLUSION

Despite the increase in transplant-related research activity in recent years, the Arab world still lags behind in this field compared to the worldwide contribution.

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

BACKGROUND

This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.

OBJECTIVES

(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.

DATA SOURCES

We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.

REVIEW METHODS

This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.

RESULTS

The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.

DISCUSSION

The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish^® (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.

LIMITATIONS

There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.

CONCLUSIONS

Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.

FUNDING

The National Institute for Health Research HTA programme.

ISRIA statement: ten-point guidelines for an effective process of research impact assessment

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school's learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders' needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school's faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

'Knowledge for better health' revisited - the increasing significance of health research systems: a review by departing Editors-in-Chief

How can nations organise research investments to obtain the best bundle of knowledge and the maximum level of improved health, spread as equitably as possible? This question was the central focus of a major initiative from WHO led by Prof Tikki Pang, which resulted in a range of developments, including the publication of a conceptual framework for national health research systems - Knowledge for better health - in 2003, and in the founding of the journal Health Research Policy and Systems (HARPS). As Editors-in-Chief of the journal since 2006, we mark our retirement by tracking both the progress of the journal and the development of national health research systems. HARPS has maintained its focus on a range of central themes that are key components of a national health research system in any country. These include building capacity to conduct and use health research, identifying appropriate priorities, securing funds and allocating them accountably, producing scientifically valid research outputs, promoting the use of research in polices and practice in order to improve health, and monitoring and evaluating the health research system. Some of the themes covered in HARPS are now receiving increased attention and, for example, with the assessment of research impact and development of knowledge translation platforms, the journal has covered their progress throughout that expansion of interest. In addition, there is increasing recognition of new imperatives, including the importance of promoting gender equality in health research if benefits are to be maximised. In this Editorial, we outline some of the diverse and developing perspectives considered within each theme, as well as considering how they are held together by the growing desire to build effective health research systems in all countries.From 2003 until mid-June 2017, HARPS published 590 articles on the above and related themes, with authors being located in 76 countries. We present quantitative data tracing the journal's growth and the increasing external recognition of its role. We thank the many colleagues who have kindly contributed to the journal's success, and finish on an exciting note by welcoming the new Editors-in-Chief who will take HARPS forward.

INFLUENCE OF HEALTH TECHNOLOGY ASSESSMENT AND ITS MEASUREMENT

Objectives: The aim of this study was to obtain information on methods used to measure health technology assessment (HTA) influence, decisions that were influenced, and outcomes linked to HTA.

Methods: Electronic databases were used to locate studies in which HTA influence had been demonstrated. Inclusion criteria were studies that reliably reported consideration by decision makers of HTA findings; comparative studies of technology use before and after HTA; and details of changes in policy, health outcomes, or research that could be credibly linked to an HTA.

Results: Fifty-one studies were selected for review. Settings were national (24), regional (12), both national and regional (3) hospitals (9), and multinational (3). The most common approach to appraisal of influence was review of policy or administrative decisions following HTA recommendations (51 percent). Eighteen studies (35 percent) reported interview or survey findings, thirteen (26 percent) reviewed administrative data, and six considered the influence of primary studies. Of 142 decisions informed by HTA, the most common types were on routine clinical practice (67 percent of studies), coverage (63 percent), and program operation (37 percent). The most frequent indications of HTA influence were on decisions related to resource allocation (59 percent), change in practice pattern (31 percent), and incorporation of HTA details in reference material (18 percent). Few publications assessed the contribution of HTA to changing patient outcomes.

Conclusions: The literature on HTA influence remains limited, with little on longer term effects on practice and outcomes. The reviewed publications indicated how HTA is being used in different settings and approaches to measuring its influence that might be more widely applied, such as surveys and monitoring administrative data.

A mixed methods study of the factors that influence whether intervention research has policy and practice impacts: perceptions of Australian researchers

OBJECTIVES

To investigate researchers' perceptions about the factors that influenced the policy and practice impacts (or lack of impact) of one of their own funded intervention research studies.

DESIGN

Mixed method, cross-sectional study.

SETTING

Intervention research conducted in Australia and funded by Australia's National Health and Medical Research Council between 2003 and 2007.

PARTICIPANTS

The chief investigators from 50 funded intervention research studies were interviewed to determine if their study had achieved policy and practice impacts, how and why these impacts had (or had not) occurred and the approach to dissemination they had employed.

RESULTS

We found that statistically significant intervention effects and publication of results influenced whether there were policy and practice impacts, along with factors related to the nature of the intervention itself, the researchers' experience and connections, their dissemination and translation efforts, and the postresearch context.

CONCLUSIONS

This study indicates that sophisticated approaches to intervention development, dissemination actions and translational efforts are actually widespread among experienced researches, and can achieve policy and practice impacts. However, it was the links between the intervention results, further dissemination actions by researchers and a variety of postresearch contextual factors that ultimately determined whether a study had policy and practice impacts. Given the complicated interplay between the various factors, there appears to be no simple formula for determining which intervention studies should be funded in order to achieve optimal policy and practice impacts.

Four centuries on from Bacon: progress in building health research systems to improve health systems?

In 1627, Francis Bacon’s New Atlantis described a utopian society in which an embryonic research system contributed to meeting the needs of the society. In this editorial, we use some of the aspirations described in New Atlantis to provide a context within which to consider recent progress in building health research systems to improve health systems and population health. In particular, we reflect on efforts to build research capacity, link research to policy, identify the wider impacts made by the science, and generally build fully functioning research systems to address the needs identified.

In 2014, Health Research Policy and Systems has continued to publish one-off papers and article collections covering a range of these issues in both high income countries and low- and middle-income countries. Analysis of these contributions, in the context of some earlier ones, is brought together to identify achievements, challenges and possible ways forward. We show how 2014 is likely to be a pivotal year in the development of ways to assess the impact of health research on policies, practice, health systems, population health, and economic benefits.

We demonstrate how the increasing focus on health research systems will contribute to realising the hopes expressed in the World Health Report, 2013, namely that all nations would take a systematic approach to evaluating the outputs and applications resulting from their research investment.

Understanding factors associated with the translation of cardiovascular research: a multinational case study approach

Background

Funders of health research increasingly seek to understand how best to allocate resources in order to achieve maximum value from their funding. We built an international consortium and developed a multinational case study approach to assess benefits arising from health research. We used that to facilitate analysis of factors in the production of research that might be associated with translating research findings into wider impacts, and the complexities involved.

Methods

We built on the Payback Framework and expanded its application through conducting co-ordinated case studies on the payback from cardiovascular and stroke research in Australia, Canada and the United Kingdom. We selected a stratified random sample of projects from leading medical research funders. We devised a series of innovative steps to: minimize the effect of researcher bias; rate the level of impacts identified in the case studies; and interrogate case study narratives to identify factors that correlated with achieving high or low levels of impact.

Results

Twenty-nine detailed case studies produced many and diverse impacts. Over the 15 to 20 years examined, basic biomedical research has a greater impact than clinical research in terms of academic impacts such as knowledge production and research capacity building. Clinical research has greater levels of wider impact on health policies, practice, and generating health gains. There was no correlation between knowledge production and wider impacts. We identified various factors associated with high impact. Interaction between researchers and practitioners and the public is associated with achieving high academic impact and translation into wider impacts, as is basic research conducted with a clinical focus. Strategic thinking by clinical researchers, in terms of thinking through pathways by which research could potentially be translated into practice, is associated with high wider impact. Finally, we identified the complexity of factors behind research translation that can arise in a single case.

Conclusions

We can systematically assess research impacts and use the findings to promote translation. Research funders can justify funding research of diverse types, but they should not assume academic impacts are proxies for wider impacts. They should encourage researchers to consider pathways towards impact and engage potential research users in research processes.