Rapid evaluation for health and social care innovations: challenges for "quick wins" using interrupted time series

Economic evaluation of a medical ambulatory care service using a single group interrupted time-series design

RATIONALE

Increasing demand for hospital services can lead to overcrowding and delays in treatment, poorer outcomes and a high cost-burden. The medical ambulatory care service (MACS) provides out of hospital patient care, including diagnostic and therapeutic interventions for patients that require urgent attention, but which can be safely administered in the ambulatory environment. The programme is yet to be rigorously evaluated.

AIMS/OBJECTIVES

The aim of this study is to evaluate the impact of the MACS programme on emergency department (ED) presentations, hospital admissions, length-of-stay and health service costs from a health system perspective.

METHOD

We used a single group interrupted time series methodology with a multiple baseline approach to analyse the impact of the MACS clinic on ED presentations, hospital admissions, length-of-stay and cost outcomes for general practitioners (GP)-referred, ED-referred and ward-referred patients under two counterfactual scenarios: an increasing trend in health utilization based on preperiod predictions or a stabilization of utilization rates.

RESULTS

The time trend of hospital utilization differed after attending MACS for all three referral groups. The time trend for the GP-referred group declined significantly by 0.36 ED presentations per 100 patients per 30 days [95% confidence interval (CI): -0.52 to -0.2], while inpatient length of stay declined significantly by 1.56 and 3.70 days, respectively, per 100 ED-referred and ward-referred patients per 30 days (95% CI: -2.51 to -0.57 and -5.71 to -1.69, respectively). Under two different counterfactual scenarios, the predicted net savings for MACS across three patient groups were $78,685 (95% CI: $54,807-$102,563) and $547,639 (95% CI: $503,990-$591,287) per 100 patients over 18 months.

CONCLUSION

MACS was found to be cost-effective for GP and ward-referred groups, but the expected impact for ED-referred patients is sensitive to assumptions. Expansion of the service for GP-referred patients is expected to reduce hospitalizations the most and generate the largest net cost savings.

Approaches to enabling rapid evaluation of innovations in health and social care: a scoping review of evidence from high-income countries

OBJECTIVE

The COVID-19 pandemic increased the demand for rapid evaluation of innovation in health and social care. Assessment of rapid methodologies is lacking although challenges in ensuring rigour and effective use of resources are known. We mapped reports of rapid evaluations of health and social care innovations, categorised different approaches to rapid evaluation, explored comparative benefits of rapid evaluation, and identified knowledge gaps.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE and Health Management Information Consortium (HMIC) databases were searched through 13 September 2022.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

We included publications reporting primary research or methods for rapid evaluation of interventions or services in health and social care in high-income countries.

DATA EXTRACTION AND SYNTHESIS

Two reviewers developed and piloted a data extraction form. One reviewer extracted data, a second reviewer checked 10% of the studies; disagreements and uncertainty were resolved through consensus. We used narrative synthesis to map different approaches to conducting rapid evaluation.

RESULTS

We identified 16 759 records and included 162 which met inclusion criteria.We identified four main approaches for rapid evaluation: (1) Using methodology designed specifically for rapid evaluation; (2) Increasing rapidity by doing less or using less time-intensive methodology; (3) Using alternative technologies and/or data to increase speed of existing evaluation method; (4) Adapting part of non-rapid evaluation.The COVID-19 pandemic resulted in an increase in publications and some limited changes in identified methods. We found little research comparing rapid and non-rapid evaluation.

CONCLUSIONS

We found a lack of clarity about what 'rapid evaluation' means but identified some useful preliminary categories. There is a need for clarity and consistency about what constitutes rapid evaluation; consistent terminology in reporting evaluations as rapid; development of specific methodologies for making evaluation more rapid; and assessment of advantages and disadvantages of rapid methodology in terms of rigour, cost and impact.

Effectiveness of a population-based integrated care model in reducing hospital activity: an interrupted time series analysis

Objectives

First impact assessment analysis of an integrated care model (ICM) to reduce hospital activity in the London Borough of Hillingdon, UK.

Methods

We evaluated a population-based ICM consisting of multiple interventions based on self-management, multidisciplinary teams, case management and discharge management. The sample included 331 330 registered Hillingdon residents (at the time of data extraction) between October 2018 and July 2020. Longitudinal data was extracted from the Whole Systems Integrated Care database. Interrupted time series Poisson and Negative binomial regressions were used to examine changes in non-elective hospital admissions (NEL admissions), accident and emergency visits (A&E) and length of stay (LoS) at the hospital. Multiple imputations were used to replace missing data. Subgroup analysis of various groups with and without long-term conditions (LTC) was also conducted using the same models.

Results

In the whole registered population of Hillingdon at the time of data collection, gradual decline over time in NEL admissions (RR 0.91, 95% CI 0.90 to 0.92), A&E visits (RR 0.94, 95% CI 0.93 to 0.95) and LoS (RR 0.93, 95% CI 0.92 to 0.94) following an immediate increase during the first months of implementation in the three outcomes was observed. Subgroup analysis across different groups, including those with and without LTCs, showed similar effects. Sensitivity analysis did not show a notable change compared with the original analysis.

Conclusion

The Hillingdon ICM showed effectiveness in reducing NEL admissions, A&E visits and LoS. However, further investigations and analyses could confirm the results of this study and rule out the potential effects of some confounding events, such as the emergence of COVID-19 pandemic.

Using rapid evaluation methods to assess service delivery changes: Lessons learned for evaluation practice during the COVID-19 pandemic

The COVID-19 pandemic required large-scale service delivery changes for government, and provided the opportunity for evaluators to step up and support decision makers to understand the impact of these changes. Rapid evaluation methods (REM) provide a pragmatic approach for generating timely information for evidence-based policy and decision-making. Grounded in developmental and utilisation-focused evaluation theory, REM incorporates a team-based, mixed methods design, executed over a 6-8-week period. Customised rubrics were used to rigorously assess effectiveness and scalability of practice changes to inform COVID-19 response planning. REM is an alternative approach to full-scale evaluation models frequently implemented to assess policies and programs. Adapted use of REM suggests that meaningful insights can be gained through use of smaller scale evaluations. This article shares lessons learned from a novel rapid evaluation method applied in the context of the COVID-19 pandemic. The rapid evaluation approach was implemented to provide real-time insights and evaluative conclusions for 15 program and practice adaptations across Victorian health and human service settings. The article shares insights about the practical applicability of balancing rigour and timeliness when implementing a rapid evaluation, and strengths and limitations of working within a fast-paced evaluation framework. Findings can inform evaluative practice in resource and time-limited settings.

Models of provider care in long-term care: A rapid scoping review

INTRODUCTION

One of the current challenges in long-term care homes (LTCH) is to identify the optimal model of care, which may include specialty physicians, nursing staff, person support workers, among others. There is currently no consensus on the complement or scope of care delivered by these providers, nor is there a repository of studies that evaluate the various models of care. We conducted a rapid scoping review to identify and map what care provider models and interventions in LTCH have been evaluated to improve quality of life, quality of care, and health outcomes of residents.

METHODS

We conducted this review over 10-weeks of English language, peer-reviewed studies published from 2010 onward. Search strategies for databases (e.g., MEDLINE) were run on July 9, 2020. Studies that evaluated models of provider care (e.g., direct patient care), or interventions delivered to facility, staff, and residents of LTCH were included. Study selection was performed independently, in duplicate. Mapping was performed by two reviewers, and data were extracted by one reviewer, with partial verification by a second reviewer.

RESULTS

A total of 7,574 citations were screened based on the title/abstract, 836 were reviewed at full text, and 366 studies were included. Studies were classified according to two main categories: healthcare service delivery (n = 92) and implementation strategies (n = 274). The condition/ focus of the intervention was used to further classify the interventions into subcategories. The complex nature of the interventions may have led to a study being classified in more than one category/subcategory.

CONCLUSION

Many healthcare service interventions have been evaluated in the literature in the last decade. Well represented interventions (e.g., dementia care, exercise/mobility, optimal/appropriate medication) may present opportunities for future systematic reviews. Areas with less research (e.g., hearing care, vision care, foot care) have the potential to have an impact on balance, falls, subsequent acute care hospitalization.