Conducting retrospective impact analysis to inform a medical research charity's funding strategies: the case of Asthma UK

What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors

The discussion on the 'societal impact of university research' has emerged within the recent decade with different utilisation from the British REF impact stories to the EU Framework Programmes' criteria. The fundamental idea is that (university) research should contribute to addressing societal challenges, such as climate change, and not just pursue 'scientific excellence'. Several approaches have been developed, from case studies of the 'productive interactions' that co-create knowledge with societal stakeholders to the so-called 'altmetrics' tracking online dissemination of research outputs. Nonetheless, these experiences do not always point in the same direction, and the notion of 'societal impact' seems still unclearly defined. Given the growing policy pressure on universities to address societal challenges, this paper reviews the scientific literature on the so-called 'societal impact' of university research. By querying three scientific archives (Web of Science, Scopus and ScienceDirect), 135 relevant scientific publications are selected and discussed, looking for conceptualisations, monitoring methods and success factors. The review maps a value-laden discussion arguing for longer-term, multi-dimensional perspectives on university research impacts beyond bibliometric indicators. Findings highlight a progressive shift from 'attribution', i.e., looking for causal relationships between research and societal changes, to 'contribution', acknowledging researchers' efforts to engage with societal challenges.

Increased use of knowledge translation strategies is associated with greater research impact on public health policy and practice: an analysis of trials of nutrition, physical activity, sexual health, tobacco, alcohol and substance use interventions

BACKGROUND

Greater use of knowledge translation (KT) strategies is recommended to improve the research impact of public health trials. The purpose of this study was to describe (1) the research impact of setting-based public health intervention trials on public health policy and practice; (2) the association between characteristics of trials and their research impact on public health policy and practice; and (3) the association between the use of KT strategies and research impacts on public health policy and practice.

METHODS

We conducted a survey of authors of intervention trials targeting nutrition, physical activity, sexual health, tobacco, alcohol or substance use. We assessed the use of KT strategies aligned to domains of the Knowledge-To-Action Framework. We defined "research impact" on health policy and practice as any one or more of the following: citation in policy documents or announcements, government reports, training materials, guidelines, textbooks or court rulings; or endorsement by a (non)governmental organization; use in policy or practice decision-making; or use in the development of a commercial resource or service.

RESULTS

Of the included trials, the authors reported that 65% had one or more research impacts. The most frequently reported research impact was citation in a policy document or announcement (46%). There were no significant associations between the effectiveness of the intervention, trial risk of bias, setting or health risk and trial impact. However, for every one unit increase in the total KT score (range 0-8), reflecting greater total KT activity, the odds of a health policy or practice research impact increased by approximately 30% (OR = 1.30, 95% CI: 1.02, 1.66; p = 0.031). Post hoc examination of KT domain scores suggests that KT actions focused on providing tailored support to facilitate program implementation and greater use of research products and tools to disseminate findings to end-users may be most influential in achieving impact.

CONCLUSIONS

Trials of public health interventions frequently have public health impacts, and the use of more comprehensive KT strategies may facilitate greater research impact.

A systems approach for optimizing implementation to impact: meeting report and proceedings of the 2019 In the Trenches: Implementation to Impact International Summit

Background

The In the Trenches series of cutting-edge knowledge sharing events on impact for front-line experts and practitioners provides an engagement platform for diverse stakeholders across government, research funding organizations, industry, and academia to share emerging knowledge and practical experiences. The second event of the series In the Trenches: Implementation to Impact International Summit was held in Banff, Alberta, Canada, on June 7–8, 2019. The overarching vision for the Summit was to create an engagement platform for addressing key challenges and finding practical solutions to move from implementation (i.e. putting findings into effect) to impact (i.e. creating benefits to society and the economy).

Processes and proceedings

The Summit used diverse approaches to facilitate active engagement and knowledge sharing between 80 delegates across sectors and jurisdictions. Summit sessions mostly consisted of short talks and moderated panels grouped into eight thematic sessions. Each presentation included a summary of Key Messages, along with a summary of the Actionable Insights which concluded each session. The presentations and discussions are analysed, synthesized and described in this proceedings paper using a systems approach. This demonstrates how the Summit focused on each of the necessary functions (and associated components) that should be undertaken, and combined, for effective research and innovation: stewardship and governance, securing finance, creating capacity, and producing and using research. The approach also identifies relevant challenges.

Conclusions

There is increased interest globally in the benefits that can accrue from adopting a systems approach to research and innovation. Various organizations in Canada and internationally have made considerable progress on Implementation to Impact, often as a result of well-planned initiatives. The Summit highlights the value of 1) collaboration between researchers and potential users, and 2) the adoption by funders of approaches involving an increasing range of responsibilities and activities. The Summit website (https://inthetrenchessummit.com/) will be periodically updated with new resources and information about future In the Trenches events.

Research and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015)

OBJECTIVES

Health technology assessment aims to inform and support healthcare decision making, and trials are part of that process. The purpose of this study was to measure the impact of a sample of trials in a meaningful but robust fashion.

METHODS

All randomized controlled trials funded and published by the UK National Institute of Health Research in the Health Technology Assessment journal series and other peer-reviewed journals were identified for 2006 to 2015. Citation analysis was performed for all trials, and quantitative content analysis was undertaken on a purposive sample to determine whether impact could be categorized as "instrumental" (ie, having a clear influence on key research and policy publications).

RESULTS

The search identified 133 relevant trials. The citation rate per trial was 102.97. Of the 133 trials, 129 (98%) were cited in 1 or more systematic reviews or meta-analyses (mean per trial = 7.18, range = 0-44). Where they were cited, the trials were used in some form of synthesis 63% of the time. Ninety-one of the 133 (68%) trials were found to be cited in 1 or more guidance or policy document (mean per trial = 2.75, range = 0-26) and had an instrumental influence 41% of the time. The publication of these trials' results in journals other than the Health Technology Assessment journal appears to enhance the discoverability of the trial data. Altmetric.com proved to be very useful in identifying unique policy and guidance documents.

CONCLUSION

These trials have impressive citation rates, and a sizeable proportion are certainly being used in key publications in a genuinely instrumental manner.

Research funding impact and priority setting - advancing universal access and quality healthcare research in Malaysia

BACKGROUND

Health Research Priority Setting (HRPS) in the Ministry of Health (MOH) Malaysia was initiated more than a decade ago to drive effort toward research for informed decision and policy-making. This study assessed the impact of funded prioritised research and identified research gaps to inform future priority setting initiatives for universal access and quality healthcare in Malaysia.

METHODS

Research impact of universal access and quality healthcare projects funded by the National Institutes of Health Malaysia were assessed based on the modified Payback Framework, addressing categories of informing policy, knowledge production, and benefits to health and health sector. For the HRPS process, the Child Health and Nutrition Research Initiative methodology was adapted and adopted, with the incorporation of stakeholder values using weights and monetary allocation survey. Workshop discussions and interviews with stakeholders and research groups were conducted to identify research gaps, with the use of conceptual frameworks to guide the search.

RESULTS

Seventeen ongoing and 50 completed projects were identified for research funding impact analysis. Overall, research fund allocation differed from stakeholders' expectation. For research impact, 48 out of 50 completed projects (96.0%) contributed to some form of policy-making efforts. Almost all completed projects resulted in outputs that contributed to knowledge production and were expected to lead to health and health sector benefits. The HRPS process led to the identification of research priority areas that stemmed from ongoing and new issues identified for universal access and quality healthcare.

CONCLUSION

The concerted efforts of evaluation of research funding impact, prioritisation, dissemination and policy-maker involvement were valuable for optimal health research resource utilisation in a resource constrained developing country. Embedding impact evaluation into a priority setting process and funding research based on national needs could facilitate health research investment to reach its potential.

Development and validation of a questionnaire to measure research impact

Although questionnaires are widely used in research impact assessment, their metric properties are not well known. Our aim is to test the internal consistency and content validity of an instrument designed to measure the perceived impacts of a wide range of research projects. To do so, we designed a questionnaire to be completed by principal investigators in a variety of disciplines (arts and humanities, social sciences, health sciences, and information and communication technologies). The impacts perceived and their associated characteristics were also assessed. This easy-to-use questionnaire demonstrated good internal consistency and acceptable content validity. However, its metric properties were more powerful in areas such as knowledge production, capacity building and informing policy and practice, in which the researchers had a degree of control and influence. In general, the research projects represented an stimulus for the production of knowledge and the development of research skills. Behavioural aspects such as engagement with potential users or mission-oriented projects (targeted to practical applications) were associated with higher social benefits. Considering the difficulties in assessing a wide array of research topics, and potential differences in the understanding of the concept of ‘research impact’, an analysis of the context can help to focus on research needs. Analyzing the metric properties of questionnaires can open up new possibilities for validating instruments used to measure research impact. Further to the methodological utility of the current exercise, we see a practical applicability to specific contexts where multiple discipline research impact is requires.

The impact generated by publicly and charity-funded research in the United Kingdom: a systematic literature review

Objective

To identify, synthesise and critically assess the empirical evidence of the impact generated by publicly and charity-funded health research in the United Kingdom.

Methods

We conducted a systematic literature review of the empirical evidence published in English in peer-reviewed journals between 2006 and 2017. Studies meeting the inclusion criteria were selected and their findings were analysed using the Payback Framework and categorised into five main dimensions, namely knowledge, benefits to future research and research use, benefits from informing policy and product development, health and health sector benefits, and broader economic benefits. We assessed the studies for risk of selection, reporting and funding bias.

Results

Thirteen studies met the inclusion criteria. The majority of the studies (10 out of 13) assessed impact at multiple domains including the main five key themes of the Payback Framework. All of them showed a positive impact of funded research on outcomes. Of those studies, one (8%), six (46%) and six (46%) presented a low, moderate and high risk of bias, respectively.

Conclusions

Empirical evidence on the impact of publicly and charity-funded research is still limited and subject to funding and selection bias. More work is needed to establish the causal effects of funded research on academic outcomes, policy, practice and the broader economy.

Electronic supplementary material

The online version of this article (10.1186/s12961-019-0425-2) contains supplementary material, which is available to authorized users.

Does citation matter? Research citation in policy documents as an indicator of research impact - an Australian obesity policy case-study

Background

Citation of research in policy documents has been suggested as an indicator of the potential longer-term impacts of research. We investigated the use of research citations in childhood obesity prevention policy documents from New South Wales (NSW), Australia, considering the feasibility and value of using research citation as a proxy measure of research impact.

Methods

We examined childhood obesity policy documents produced between 2000 and 2015, extracting childhood obesity-related references and coding these according to reference type, geographical origin and type of research. A content analysis of the policy documents examined where and how research was cited in the documents and the context of citation for individual research publications.

Results

Over a quarter (28%) of the policy documents (n = 86) were not publicly available, almost two-thirds (63%) contained references, half (47%) cited obesity-related research and over a third (41%) of those containing references used unorthodox referencing styles, making reference extraction laborious. No patterns, in terms of the types of documents more likely to cite research, were observed and the number of obesity research publications cited per document was highly variable. In total, 263 peer-reviewed and 94 non-peer-reviewed obesity research publications were cited. Research was most commonly cited to support a policy argument or choice of solution. However, it was not always possible to determine how or why individual publications were cited or whether the cited research itself had influenced the policy process. Content analysis identified circumstances where research was mentioned or considered, but not directly cited.

Conclusions

Citation of research in policy documents in this case did not always provide evidence that the cited research had influenced the policy process, only that it was accessible and relevant to the content of the policy document. Research citation across these public health policy documents varied greatly and is unlikely to be an accurate reflection of actual research use by the policy agencies involved. The links between citation and impact may be more easily drawn in specific policy areas or types of documents (e.g. clinical guidelines), where research appraisal feeds directly into policy recommendations.

Measuring research impact: a large cancer research funding programme in Australia

BACKGROUND

Measuring research impact is of critical interest to philanthropic and government funding agencies interested in ensuring that the research they fund is both scientifically excellent and has meaningful impact into health and other outcomes. The Beat Cancer Project (BCP) is a AUD $34 m cancer research funding scheme that commenced in 2011. It was initiated by an Australian charity (Cancer Council SA), and supported by the South Australian Government and the state's major universities.

METHODS

This study applied Buxton and Hanney's Payback Framework to assess research impact generated from the BCP after 3 years of funding. Data sources were an audit of peer-reviewed publications from January 2011 to September 2014 from Web of Knowledge and a self-report survey of investigators awarded BCP research funding during its first 3 years of implementation (2011-2013). Of the 104 surveys, 92 (88%) were completed.

RESULTS

The BCP performed well across all five categories of the Payback Framework. In terms of knowledge production, 1257 peer-reviewed publications were generated and the mean impact factor of publishing journals increased annually. There were many benefits to future research with 21 respondents (23%) reporting career advancement, and 110 higher degrees obtained or expected (including 84 PhDs). Overall, 52% of funded projects generated tools for future research. The funded research attracted substantial further income yielding a very high rate of leverage. For every AUD $1 that the cancer charity invested, the BCP gained an additional AUD $6.06. Five projects (5%) had informed policy and 5 (5%) informed product development, with an additional 31 (34%) and 35 (38%) projects, respectively, anticipating doing so. In terms of health and sector and broader economic benefits, 8 (9%) projects had influenced practice or behaviour of health staff and 32 (34%) would reportedly to do so in the future.

CONCLUSIONS

Research impact was a priority of charity and government funders and led to a deliberate funding strategy. Emphasising research impact while maintaining rigorous, competitive processes can achieve the joint objectives of excellence in research, yielding good research impact and a high rate of leverage for philanthropic and public investment, as indicated by these early results.

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

BACKGROUND

This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.

OBJECTIVES

(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.

DATA SOURCES

We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.

REVIEW METHODS

This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.

RESULTS

The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.

DISCUSSION

The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish^® (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.

LIMITATIONS

There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.

CONCLUSIONS

Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.

FUNDING

The National Institute for Health Research HTA programme.

Measurement of the translation and impact from a childhood obesity trial programme: rationale and protocol for a research impact assessment

Background

There is growing recognition amongst health and medical research funders and researchers that translation of research into policy and practice needs to increase and that more transparency is needed on how impacts are realised. Several approaches are advocated for achieving this, including co-production of research or academic-practitioner research. The Population Health Unit (PHU) within the Hunter New England Local Health District in regional Australia, as an early adopter of this model, has been working to increase the likelihood that its research is translated into community health benefits. With the New South Wales Ministry of Health, the PHU responded to the burden of child overweight and obesity by combining service delivery with research expertise. The ‘Good for Kids, Good for Life’ (Good for Kids) dissemination trial was developed and implemented in seven community settings in the Hunter region of Australia between 2006 and 2010. This study aims to undertake a retrospective impact assessment to measure the research translation and impact of Good for Kids.

Methods

The method will be based upon the application of the Framework to Assess the Impact from Translational health research (FAIT), comprising three core elements, namely quantified metrics, economic assessment and a narrative of the process by which the research in question translates and generates impact.

Discussion

Increasingly, funders are interested both in the outcomes resulting from investments in health research and in the expected return on their investments. FAIT was developed specifically for this purpose and its use is anticipated to provide transparency to the pathway to translation and potentially drive increased investment in translational research programmes such as Good for Kids.

The impact on healthcare, policy and practice from 36 multi-project research programmes: findings from two reviews

BACKGROUND

We sought to analyse the impacts found, and the methods used, in a series of assessments of programmes and portfolios of health research consisting of multiple projects.

METHODS

We analysed a sample of 36 impact studies of multi-project research programmes, selected from a wider sample of impact studies included in two narrative systematic reviews published in 2007 and 2016. We included impact studies in which the individual projects in a programme had been assessed for wider impact, especially on policy or practice, and where findings had been described in such a way that allowed them to be collated and compared.

RESULTS

Included programmes were highly diverse in terms of location (11 different countries plus two multi-country ones), number of component projects (8 to 178), nature of the programme, research field, mode of funding, time between completion and impact assessment, methods used to assess impact, and level of impact identified. Thirty-one studies reported on policy impact, 17 on clinician behaviour or informing clinical practice, three on a combined category such as policy and clinician impact, and 12 on wider elements of impact (health gain, patient benefit, improved care or other benefits to the healthcare system). In those multi-programme projects that assessed the respective categories, the percentage of projects that reported some impact was policy 35% (range 5-100%), practice 32% (10-69%), combined category 64% (60-67%), and health gain/health services 27% (6-48%). Variations in levels of impact achieved partly reflected differences in the types of programme, levels of collaboration with users, and methods and timing of impact assessment. Most commonly, principal investigators were surveyed; some studies involved desk research and some interviews with investigators and/or stakeholders. Most studies used a conceptual framework such as the Payback Framework. One study attempted to assess the monetary value of a research programme's health gain.

CONCLUSION

The widespread impact reported for some multi-project programmes, including needs-led and collaborative ones, could potentially be used to promote further research funding. Moves towards greater standardisation of assessment methods could address existing inconsistencies and better inform strategic decisions about research investment; however, unresolved issues about such moves remain.

An assessment of health research impact in Iran

Background

In recent years, Iran has made significant developments in the field of health sciences. However, the question is whether this considerable increase has affected public health. The research budget has always been negligible and unsustainable in developing countries. Hence, using the Payback Framework, we conducted this study to evaluate the impact of health research in Iran.

Methods

By using a cross-sectional method and two-stage stratified cluster sampling, the projects were randomly selected from six medical universities. A questionnaire was designed according to the Payback Framework and completed by the principle investigators of the randomly selected projects.

Results

The response rate was 70.4%. Ten point twenty-four percent (10.24%) of the studies had been ordered by a knowledge user organization. The average number of articles published in journals per project was 0.96, and half of the studies had no articles published in Scopus. The results of 12% of the studies had been used in systematic review articles and the same proportion had been utilized in clinical or public health guidelines. The results of 5.3% of the studies had been implemented in the Health Ministry’s policymaking. 62% of the studies were expected to affect health directly, 38% of them had been implemented, and among the latter 60% had achieved the expected results. Concerning the economic impacts, the most common expected impact was the reduction of ‘days of work missed because of illness or disability’ and impact on personal and health system costs. About 36% of these studies had been implemented, and 61% had achieved the expected impact.

Conclusion

In most aspects, the status of research impact needs improvement. A comparison of Iran’s ranking of knowledge creation and knowledge impact in the Global Innovation Index confirms these findings. The most important problems identified were, not conducting research based on national needs, and the lack of implementation of research results.

Electronic supplementary material

The online version of this article (doi:10.1186/s12961-016-0129-9) contains supplementary material, which is available to authorized users.

Research impact: a narrative review

Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.

Tracing the indirect societal impacts of biomedical research: development and piloting of a technique based on citations

There is growing interest in assessing the societal impacts of research such as informing health policies and clinical practice, and contributing to improved health. Bibliometric approaches have long been used to assess knowledge outputs, but can they also help evaluate societal impacts? We aimed to see how far the societal impacts could be traced by identifying key research articles in the psychiatry/neuroscience area and exploring their societal impact through analysing several generations of citing papers. Informed by a literature review of citation categorisation, we developed a prototype template to qualitatively assess a reference’s importance to the citing paper and tested it on 96 papers. We refined the template for a pilot study to assess the importance of citations, including self-cites, to four key research articles. We then similarly assessed citations to those citing papers for which the key article was Central i.e. it was very important to the message of the citing article. We applied a filter of three or more citation occasions in order to focus on the citing articles where the reference was most likely to be Central. We found the reference was Central for 4.4 % of citing research articles overall and ten times more frequently if the article contained three or more citation occasions. We created a citation stream of influence for each key paper across up to five generations of citations. We searched the Web of Science for citations to all Central papers and identified societal impacts, including international clinical guidelines citing papers across the generations.

Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool

Background

There is a growing emphasis on the importance of research having demonstrable public benefit. Measurements of the impacts of research are therefore needed. We applied a modified impact assessment process that builds on best practice to 5 years (2003–2007) of intervention research funded by Australia’s National Health and Medical Research Council to determine if these studies had post-research real-world policy and practice impacts.

Methods

We used a mixed method sequential methodology whereby chief investigators of eligible intervention studies who completed two surveys and an interview were included in our final sample (n = 50), on which we conducted post-research impact assessments. Data from the surveys and interviews were triangulated with additional information obtained from documentary analysis to develop comprehensive case studies. These case studies were then summarized and the reported impacts were scored by an expert panel using criteria for four impact dimensions: corroboration; attribution, reach, and importance.

Results

Nineteen (38%) of the cases in our final sample were found to have had policy and practice impacts, with an even distribution of high, medium, and low impact scores. While the tool facilitated a rigorous and explicit criterion-based assessment of post-research impacts, it was not always possible to obtain evidence using documentary analysis to corroborate the impacts reported in chief investigator interviews.

Conclusions

While policy and practice is ideally informed by reviews of evidence, some intervention research can and does have real world impacts that can be attributed to single studies. We recommend impact assessments apply explicit criteria to consider the corroboration, attribution, reach, and importance of reported impacts on policy and practice. Impact assessments should also allow sufficient time between impact data collection and completion of the original research and include mechanisms to obtain end-user input to corroborate claims and reduce biases that result from seeking information from researchers only.

How long does biomedical research take? Studying the time taken between biomedical and health research and its translation into products, policy, and practice

BACKGROUND

The time taken, or 'time lags', between biomedical/health research and its translation into health improvements is receiving growing attention. Reducing time lags should increase rates of return to such research. However, ways to measure time lags are under-developed, with little attention on where time lags arise within overall timelines. The process marker model has been proposed as a better way forward than the current focus on an increasingly complex series of translation 'gaps'. Starting from that model, we aimed to develop better methods to measure and understand time lags and develop ways to identify policy options and produce recommendations for future studies.

METHODS

Following reviews of the literature on time lags and of relevant policy documents, we developed a new approach to conduct case studies of time lags. We built on the process marker model, including developing a matrix with a series of overlapping tracks to allow us to present and measure elements within any overall time lag. We identified a reduced number of key markers or calibration points and tested our new approach in seven case studies of research leading to interventions in cardiovascular disease and mental health. Finally, we analysed the data to address our study's key aims.

RESULTS

The literature review illustrated the lack of agreement on starting points for measuring time lags. We mapped points from policy documents onto our matrix and thus highlighted key areas of concern, for example around delays before new therapies become widely available. Our seven completed case studies demonstrate we have made considerable progress in developing methods to measure and understand time lags. The matrix of overlapping tracks of activity in the research and implementation processes facilitated analysis of time lags along each track, and at the cross-over points where the next track started. We identified some factors that speed up translation through the actions of companies, researchers, funders, policymakers, and regulators. Recommendations for further work are built on progress made, limitations identified and revised terminology.

CONCLUSIONS

Our advances identify complexities, provide a firm basis for further methodological work along and between tracks, and begin to indicate potential ways of reducing lags.

Evaluation of the impact of National Breast Cancer Foundation-funded research

OBJECTIVE

To evaluate the impact of the National Breast Cancer Foundation's (NBCF's) research investment.

DESIGN AND PARTICIPANTS

Surveys based on the Payback Framework were sent to chief investigators involved in research funded by the NBCF during 1995-2012; a bibliometric analysis of NBCF-funded publications in 2006-2010 was conducted; and a purposive, stratified sample of case studies was obtained.

MAIN OUTCOME MEASURES

Research impact on knowledge production, the research system, informing policy, product development and broader health and economic benefits.

RESULTS

Of 242 surveys sent, 153 (63%) were returned. The average impact of journals in which NBCF publications appeared was double that of world publications. Seventy surveys (46%) reported career progression, and 185 higher degrees were obtained or expected, including 121 PhDs. One hundred and one grants (66%) produced tools that built capacity across the research system, and research teams leveraged an additional $1.40 in funding for every dollar invested. Fifteen applied grants and one basic grant impacted on policy. Ten basic and four applied grants led to the development of drugs, prognostic tools or diagnostic technologies. Twenty applied and two basic grants led to changes in practice and behaviour of health care staff, consumers and the public, with further impacts anticipated. Case studies provided illustrations of high impact.

CONCLUSIONS

NBCF's strategy of investing in a mixed portfolio of research areas and mechanisms encouraged a broad range of impacts across all Payback categories. The impacts from basic research tended to focus on knowledge production and drug development; while applied research generated greater impacts within the other Payback categories. The funding of shared infrastructure stimulated impact across the research system.

Understanding factors associated with the translation of cardiovascular research: a multinational case study approach

Background

Funders of health research increasingly seek to understand how best to allocate resources in order to achieve maximum value from their funding. We built an international consortium and developed a multinational case study approach to assess benefits arising from health research. We used that to facilitate analysis of factors in the production of research that might be associated with translating research findings into wider impacts, and the complexities involved.

Methods

We built on the Payback Framework and expanded its application through conducting co-ordinated case studies on the payback from cardiovascular and stroke research in Australia, Canada and the United Kingdom. We selected a stratified random sample of projects from leading medical research funders. We devised a series of innovative steps to: minimize the effect of researcher bias; rate the level of impacts identified in the case studies; and interrogate case study narratives to identify factors that correlated with achieving high or low levels of impact.

Results

Twenty-nine detailed case studies produced many and diverse impacts. Over the 15 to 20 years examined, basic biomedical research has a greater impact than clinical research in terms of academic impacts such as knowledge production and research capacity building. Clinical research has greater levels of wider impact on health policies, practice, and generating health gains. There was no correlation between knowledge production and wider impacts. We identified various factors associated with high impact. Interaction between researchers and practitioners and the public is associated with achieving high academic impact and translation into wider impacts, as is basic research conducted with a clinical focus. Strategic thinking by clinical researchers, in terms of thinking through pathways by which research could potentially be translated into practice, is associated with high wider impact. Finally, we identified the complexity of factors behind research translation that can arise in a single case.

Conclusions

We can systematically assess research impacts and use the findings to promote translation. Research funders can justify funding research of diverse types, but they should not assume academic impacts are proxies for wider impacts. They should encourage researchers to consider pathways towards impact and engage potential research users in research processes.

Engagement in research: an innovative three-stage review of the benefits for health-care performance

Background

There is a widely held assumption that research engagement improves health-care performance at various levels, but little direct empirical evidence.

Objectives

To conduct a theoretically and empirically grounded synthesis to map and explore plausible mechanisms through which research engagement might improve health services performance. A review of the effects on patients of their health-care practitioner's or institution's participation in clinical trials was published after submission of the proposal for this review. It identified only 13 relevant papers and, overall, suggested that the evidence that research engagement improves health-care performance was less strong than some thought. We aimed to meet the need for a wider review.

Methods

An hourglass review was developed, consisting of three stages: (1) a planning and mapping stage; (2) a focused review concentrating on the core question of whether or not research engagement improves health care; and (3) a wider (but less systematic) review of papers identified during the two earlier stages. Studies were included inthe focused review if the concept of ‘engagementinresearch’ was an input and some measure of ‘performance’ an output. The search strategy covered the period 1990 to March 2012. MEDLINE, EMBASE, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science and other relevant databases were searched. A total of 10,239 papers were identified through the database searches, and 159 from other sources. A further relevance and quality check on 473 papers was undertaken, and identified 33 papers for inclusion in the review. A standard meta-analysis was not possible on the heterogeneous mix of papers in the focused review. Therefore an explanatory matrix was developed to help characterise the circumstances in which research engagement might improve health-care performance and the mechanisms that might be at work, identifying two main dimensions along which to categorise the studies:the degree of intentionalityandthe scope of the impact.

Results

Of the 33 papers in the focused review, 28 were positive (of which six were positive/mixed) in relation to the question of whether or not research engagement improves health-care performance. Five papers were negative (of which two were negative/mixed). Seven out of 28 positive papers reported some improvement in health outcomes. For the rest, the improved care took the form of improved processes of care. Nine positive papers were at a clinician level and 19 at an institutional level. The wider review demonstrated, for example, how collaborative and action research can encourage some progress along the pathway from research engagement towards improved health-care performance. There is also evidence that organisations in which the research function is fully integrated into the organisational structure out-perform other organisations that pay less formal heed to research and its outputs. The focused and wider reviews identified the diversity in the mechanisms through which research engagement might improve health care: there are many circumstances and mechanisms at work, more than one mechanism is often operative, and the evidence available for each one is limited.

Limitations

To address the complexities of this evidence synthesis of research we needed to spend significant time mapping the literature, and narrowed the research question to make it feasible. We excluded many potentially relevant papers (though we partially addressed this by conducting a wider additional synthesis). Studies assessing the impact made on clinician behaviour by small, locally conducted pieces of research could be difficult to interpret without full knowledge of the context.

Conclusions

Drawing on the focused and wider reviews, it is suggested that when clinicians and health-care organisations engage in research there is the likelihood of a positive impact on health-care performance. Organisations that have deliberately integrated the research function into organisational structures demonstrate how research engagement can, among other factors, contribute to improved health-care performance. Further explorations are required of research networks and schemes to promote the engagement of clinicians and managers in research. Detailed observational research focusing on research engagement within organisations would build up an understanding of mechanisms.

Study registration

PROSPERO: CRD42012001990.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.