Assessing research impact: Australian Research Council criteria and the case of Family Wellbeing research

Evidencing the impacts of health research: Insights from trials reported in the 2018 Australian Engagement and Impact Assessment

ISSUE ADDRESSED

While definitions of impact may vary, they often refer to the wider benefits of research evidenced beyond academia. We evaluated case studies featuring randomised trials from the 2018 Engagement and Impact Assessment to better understand how the impacts of health research are evidenced and assessed within Australia.

METHODS

We collated and evaluated 'high' scoring case studies submitted by higher education institutions with a focus on randomised trials across all areas of health research. A qualitative coding system was used for manual content analysis to assess the key characteristics of trials reported, subsequent impacts and the methods used to evidence impacts.

RESULTS

A total of 14 case studies were identified citing 35 clinical trials. The majority of interventions were behavioural with a focus on mental, behavioural or neurodevelopmental disorders. Most trials were phase III, focused on the treatment of the indication and were funded by industry. Contribution to clinical guidelines was the highest cited research impact. While there was evidence of researchers seeking to maximise trial impact, case studies lacked details on the role of trial participants and other beneficiaries in generating impact.

CONCLUSIONS

The impacts of health research can be improved through a better understanding of the priorities and agendas of funders, providing evidence of tangible impact rather than information that is contextual or predictive, and through the early development of impact strategies involving both researchers and beneficiaries. SO WHAT?: Large-scale impact exercises intended for a broad range of disciplines may not be reflective of the depth and scope of health sciences research including trials.

Producing knowledge together: a participatory approach to synthesising research across a large-scale collaboration in Aboriginal and Torres Strait Islander health

BACKGROUND

Despite that stakeholder participation in evidence synthesis could result in more useful outcomes, there are few examples of processes that actively involve them in synthesis work. Techniques are needed that engage diverse stakeholders as equal partners in knowledge co-production. The aims of this paper are to describe an innovative participatory process of synthesising a large body of academic research products and compare the findings of the participatory process against two traditional approaches to synthesis: a rapid review and a structured review.

METHODS

First, a rapid synthesis of all research outputs (n = 86) was conducted by researchers with in-depth knowledge of the collaboration's research. Second, a team of researchers and service providers conducted a structured synthesis of seventy-eight peer-reviewed articles and reports generated by the collaboration. Fifty-five publications were brought forward for further synthesis in part three, a facilitated participatory synthesis. Finally, we explored the value added by the participatory method by comparing findings generated across the three synthesis approaches.

RESULTS

Twelve researchers and 11 service providers/policy partners-8 self-identified as Aboriginal and/or Torres Strait Islander-participated in two facilitated workshops (totalling 4 h). Workshop activities engaged participants in reviewing publication summaries, identifying key findings, and evoked review, discussion and refinement. The process explicitly linked experiential knowledge to citations of academic research, clearly connecting the two knowledge types. In comparing the findings generated across all three methods we found mostly consistencies; the few discrepancies did not contradict but gave deeper insights into statements created by the other methods. The participatory synthesis generated the most, detailed, and unique findings, and contextual insights about the relevance of the key messages for practice.

CONCLUSION

The participatory synthesis engaged stakeholders with diverse backgrounds and skillsets in synthesising a large body of evidence in a relatively short time. The participatory approach produced findings comparable to traditional synthesis methods while extending knowledge and identifying lessons most relevant for the participants who, ultimately, are the end users of the research. This process will interest other large-scale research collaborations seeking to engage stakeholders in evidence synthesis.

Evaluation of crop research institutes under data and resource constraints: An alternative approach

The evaluation of crop research institutes in the developing world under limited data availability has not been assessed in the past due to resource constraints. The paper assesses the social benefits of rice research taking the case of a research institute from India following a new approach. The area coverage of the varieties was estimated to be 3.4 million ha and the gain in production was 6.2 million tonnes per year in India. The additional return obtained due to the adoption of these varieties was about ₹ 14,621 million (US$ 232 million) per year at constant 2014-5 prices. The return per rupee investment in the institute's research and extension was ₹ 17. This approach is recommended for the impact evaluation of other crop research institutes in India and the developing world under resource constraints.

What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors

The discussion on the 'societal impact of university research' has emerged within the recent decade with different utilisation from the British REF impact stories to the EU Framework Programmes' criteria. The fundamental idea is that (university) research should contribute to addressing societal challenges, such as climate change, and not just pursue 'scientific excellence'. Several approaches have been developed, from case studies of the 'productive interactions' that co-create knowledge with societal stakeholders to the so-called 'altmetrics' tracking online dissemination of research outputs. Nonetheless, these experiences do not always point in the same direction, and the notion of 'societal impact' seems still unclearly defined. Given the growing policy pressure on universities to address societal challenges, this paper reviews the scientific literature on the so-called 'societal impact' of university research. By querying three scientific archives (Web of Science, Scopus and ScienceDirect), 135 relevant scientific publications are selected and discussed, looking for conceptualisations, monitoring methods and success factors. The review maps a value-laden discussion arguing for longer-term, multi-dimensional perspectives on university research impacts beyond bibliometric indicators. Findings highlight a progressive shift from 'attribution', i.e., looking for causal relationships between research and societal changes, to 'contribution', acknowledging researchers' efforts to engage with societal challenges.

How can impact strategies be developed that better support universities to address twenty-first-century challenges?

To better address twenty-first-century challenges, research institutions often develop and publish research impact strategies, but as a tool, impact strategies are poorly understood. This study provides the first formal analysis of impact strategies from the UK, Canada, Australia, Denmark, New Zealand and Hong Kong, China, and from independent research institutes. Two types of strategy emerged. First, ‘achieving impact’ strategies tended to be bottom-up and co-productive, with a strong emphasis on partnerships and engagement, but they were more likely to target specific beneficiaries with structured implementation plans, use boundary organisations to co-produce research and impact, and recognise impact with less reliance on extrinsic incentives. Second, ‘enabling impact’ strategies were more top-down and incentive-driven, developed to build impact capacity and culture across an institution, faculty or centre, with a strong focus on partnerships and engagement, and they invested in dedicated impact teams and academic impact roles, supported by extrinsic incentives including promotion criteria. This typology offers a new way to categorise, analyse and understand research impact strategies, alongside insights that may be used by practitioners to guide the design of future strategies, considering the limitations of top-down, incentive-driven approaches versus more bottom-up, co-productive approaches.

What funders are doing to assess the impact of their investments in health and biomedical research

As pressures to maximize research funding grow, biomedical research funders are increasingly tasked with demonstrating the long-term and real-world impacts of their funded research investments. Over the past three decades, research impact assessments (RIA) have emerged as an important tool for analysing the impacts of research by incorporating logic models, frameworks and indicators to track measures of knowledge production, capacity-building, development of research products, adoption of research into clinical guidelines and policies, and the realization of health, economic and social benefits. While there are currently several models for RIA within the literature, less attention has been paid to how funders can practically select and implement a RIA model to demonstrate the impacts of their own research portfolios. In this paper, a literature review was performed to understand (1) which research funders have performed RIAs of their research portfolios to date; (2) how funders have designed their assessments, including the models and tools they have used; (3) what challenges to and facilitators of success have funders found when adopting the RIA model to their own portfolio; and (4) who participates in the assessments. Forty-four papers from both published and grey literature were found to meet the review criteria and were examined in detail. There is a growing culture of RIA among funders, and included papers spanned a diverse set of funders from 10 countries or regions. Over half of funders (59.1%) used a framework to conduct their assessment, and a variety of methods for collecting impact data were reported. Issues of methodological rigour were observed across studies in the review, and this was related to numerous challenges funders faced in designing timely RIAs with quality impact data. Over a third of articles (36.4%) included input from stakeholders, yet only one article reported surveying patients and members of the public as part of the assessment. To advance RIA among funders, we offer several recommendations for increasing the methodological rigour of RIAs and suggestions for future research, and call for a careful reflection of the voices needed in an impact assessment to ensure that RIAs are having a meaningful impact on patients and the public.

"We are not stray leaves blowing about in the wind": exploring the impact of Family Wellbeing empowerment research, 1998-2021

BACKGROUND

An Aboriginal-developed empowerment and social and emotional wellbeing program, known as Family Wellbeing (FWB), has been found to strengthen the protective factors that help Indigenous Australians to deal with the legacy of colonisation and intergenerational trauma. This article reviews the research that has accompanied the implementation of the program, over a 23 year period. The aim is to assess the long-term impact of FWB research and identify the key enablers of research impact and the limitations of the impact assessment exercise. This will inform more comprehensive monitoring of research impact into the future.

METHODS

To assess impact, the study took an implementation science approach, incorporating theory of change and service utilisation frameworks, to create a logic model underpinned by Indigenous research principles. A research impact narrative was developed based on mixed methods analysis of publicly available data on: 1) FWB program participation; 2) research program funding; 3) program outcome evaluation (nine studies); and 4) accounts of research utilisation (seven studies).

RESULTS

Starting from a need for research on empowerment identified by research users, an investment of $2.3 million in research activities over 23 years produced a range of research outputs that evidenced social and emotional wellbeing benefits arising from participation in the FWB program. Accounts of research utilisation confirmed the role of research outputs in educating participants about the program, and thus, facilitating more demand (and funding acquisition) for FWB. Overall research contributed to 5,405 recorded participants accessing the intervention. The key enablers of research impact were; 1) the research was user- and community-driven; 2) a long-term mutually beneficial partnership between research users and researchers; 3) the creation of a body of knowledge that demonstrated the impact of the FWB intervention via different research methods; 4) the universality of the FWB approach which led to widespread application.

CONCLUSIONS

The FWB research impact exercise reinforced the view that assessing research impact is best approached as a "wicked problem" for which there are no easy fixes. It requires flexible, open-ended, collaborative learning-by-doing approaches to build the evidence base over time. Steps and approaches that research groups might take to build the research impact knowledge base within their disciplines are discussed.

Evaluating the Implementation of a Mental Health Referral Service "Connect to Wellbeing": A Quality Improvement Approach

There is increasing demand for mental health services to be accessible to diverse populations in flexible, yet, cost-effective ways. This article presents the findings from a study that evaluated the process of implementing Connect to Wellbeing (CTW), a new mental health intake, assessment and referral service in regional Australia, to determine how well it improved access to services, and to identify potential measures that could be used to evaluate value for money. The study used a hybrid study design to conduct a process evaluation to better understand: the process of implementing CTW; and the barriers and factors enabling implementation of CTW. In addition, to better understand how to measure the cost-effectiveness of such services, the hybrid study design included an assessment of potential outcome measures suitable for ascertaining both the effectiveness of CTW in client health outcomes, and conducting a value for money analysis. The process evaluation found evidence that by improving processes, and removing waitlists CTW had created an opportunity to broadened the scope and type of psychological services offered which improved accessibility. The assessment of potential outcome measures provided insight into suitable measures for future evaluation into service effectiveness, client health outcomes and value for money.

The Language of Sustainable Tourism as a Proxy Indicator of Quality

Sustainable tourism (ST) has recently become the mainstream of the tourism industry and, accordingly, has influenced contemporary tourism research. However, ST is not just theories about indications and contraindications of global travel, but also a specific language that needs mastering to take sustainability work forward. In other words, what research receives recognition depends on the proficiency in how the articulation in research proposals and within assessment under the heading of “research impact”. The aim of this paper is to investigate how tourism research gains recognition within research evaluation, by investigating the national research appraisal in the United Kingdom (Research Excellence Framework). By using content analysis, we disentangle the rhetorical choices and narrative constructions within researchers’ impact claims. Our findings suggest that researchers adopt a rhetorical style that implies causality and promotes good outcomes facilitating ST. However, the structure of the assessment format enforces an articulation of sustainable research impact without stating the methodological limitations of that such claim. Therefore, the rhetorical choices of ST researchers merely represent a proxy indicator of the claimed impact. We conclude that the lack of rigor in accounting for the impact of ST research may inadvertently restrict attaining ST.