Approaches to enabling rapid evaluation of innovations in health and social care: a scoping review of evidence from high-income countries

OBJECTIVE

The COVID-19 pandemic increased the demand for rapid evaluation of innovation in health and social care. Assessment of rapid methodologies is lacking although challenges in ensuring rigour and effective use of resources are known. We mapped reports of rapid evaluations of health and social care innovations, categorised different approaches to rapid evaluation, explored comparative benefits of rapid evaluation, and identified knowledge gaps.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE and Health Management Information Consortium (HMIC) databases were searched through 13 September 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included publications reporting primary research or methods for rapid evaluation of interventions or services in health and social care in high-income countries.

DATA EXTRACTION AND SYNTHESIS

Two reviewers developed and piloted a data extraction form. One reviewer extracted data, a second reviewer checked 10% of the studies; disagreements and uncertainty were resolved through consensus. We used narrative synthesis to map different approaches to conducting rapid evaluation.

RESULTS

We identified 16 759 records and included 162 which met inclusion criteria.We identified four main approaches for rapid evaluation: (1) Using methodology designed specifically for rapid evaluation; (2) Increasing rapidity by doing less or using less time-intensive methodology; (3) Using alternative technologies and/or data to increase speed of existing evaluation method; (4) Adapting part of non-rapid evaluation.The COVID-19 pandemic resulted in an increase in publications and some limited changes in identified methods. We found little research comparing rapid and non-rapid evaluation.

CONCLUSIONS

We found a lack of clarity about what 'rapid evaluation' means but identified some useful preliminary categories. There is a need for clarity and consistency about what constitutes rapid evaluation; consistent terminology in reporting evaluations as rapid; development of specific methodologies for making evaluation more rapid; and assessment of advantages and disadvantages of rapid methodology in terms of rigour, cost and impact.

Possibilities and constraints of rapid online ethnography: Lessons from a rapid assessment of COVID-19 policy for people who use drugs

During the COVID-19 Pandemic, health care provision changed rapidly and funding became available to assess pandemic-related policy change. Research activities, however, were limited to contactless, online delivery. It was clear early on that some elements of online rapid ethnography were feasible and effective, while others would not approach traditional ethnographic depth. We conducted an online Rapid Assessment, Response, and Evaluation (RARE) project from August 2020 to September 2021 to understand how COVID-19 policy impacted people who use drugs. Our interdisciplinary research team conducted online ethnographic interviews and focus groups with 45 providers and community stakeholders, and 19 clients from rural and urban areas throughout Arizona. In addition, 26 webinars, online trainings, and virtual conferences focused on opioid policy and medication for opioid use disorders (MOUD) were opportunities to observe conversations among providers and program representatives about how best to implement policy changes, how to reach people in recovery, and what aspects of the changes should carry forward into better all-around opioid services in the future. Our RARE project was successful in collecting a range of providers' perspectives on both rural and urban implementation of take-home MOUDs as well as a wide view of national conversations, but client perspectives were limited to those who were not impacted by the policies and continued to attend in-person daily clinic visits. We describe challenges to online rapid ethnography and how online research may have allowed for an in-depth, but incomplete picture of how policy changes during COVID-19 policy affected people with opioid use disorders.