Cost analysis in implementation studies of evidence-based practices for mental health and substance use disorders: a systematic review

Implementation science and entrepreneurship: Two complementary frameworks for advancing pharmacy practice

BACKGROUND

Academic leaders and clinicians have promoted implementation science (IS) and entrepreneurship (EN) as frameworks for transforming pharmacy education and practice. Although distinct, IS and EN share many of the same processes and goals in promoting solutions to problems in healthcare.

OBJECTIVES

To distinguish the similarities and differences between IS and EN. Major components of each method will be evaluated for relative strengths and limitations in advancing pharmacy practice interventions.

METHODS

Basic components of IS and EN were compared. The Consolidated Framework for Implementation Research Intervention (CFIR) was used to represent the elements of IS and the resource-based theory of competitive advantage was used to operationalize key constructs in entrepreneurship. The five CFIR domains matched well with the five constructs of the resource-based theory of competitive advantage, allowing five common components for comparisons: (1) intervention, (2) cost benefit, (3) market viability, (4) organizational setting, and (5) characteristics of the individuals and inputs.

RESULTS

IS and EN are two scholarly approaches that differ primarily in how they frame research questions. IS frameworks prioritize the study of the implementation of evidence-based interventions and their theoretical contributions. They focus predominantly on quantifying the impact of the intervention and less so on the needs of community partners, market viability of intervention, and return on investment. EN is more open to interventions that lack strong objective evidence for effectiveness, relying more on what makes an intervention financially sustainable and profitable in markets.

CONCLUSION

IS and EN both examine the role of the intervention, organization, market viability, inputs, and financial return on the successful adoption and sustainability of interventions. Either framework can be used to advance the adoption and dissemination of pharmacy practice interventions, as long as researchers understand the limitations of each framework.

Development of an instrument (Cost-IS) to estimate costs of implementation strategies for digital health solutions: a modified e-Delphi study

BACKGROUND

It is important to determine the relative value of health innovations when allocating limited healthcare resources. Implementation strategies require and consume healthcare resources yet are often excluded from published economic evaluations. This paper reports on the development of a pragmatic implementation costing instrument to assist with the planning, delivery, and evaluation of digital health implementation strategies.

METHODS

A modified e-Delphi process was adopted to develop an implementation costing instrument. Purposive sampling was used to recruit a panel of experts in implementation science, health economic evaluations and/or digital health from the academic, government, clinical or health service sectors. In each round, participants were sent an electronic questionnaire and a prototype of the implementation costing instrument. The prototype in the initial round was informed from a literature review and qualitative interview findings. The prototype was updated iteratively between rounds in response to the panel's feedback. In subsequent rounds, participants also received the anonymous results of items that did not reach consensus in the previous round. Termination occurred once consensus was reached on integral questions (those pertaining specifically to the instrument design) or when three rounds were completed, to prevent sample fatigue. Consensus was defined as at least 75% of experts in agreement for any item.

RESULTS

Consensus was reached on the core components and design of the instrument from a panel of twelve experts in implementation science, health economic evaluations and/or digital health. Areas where consensus was not reached included users' level of implementation science knowledge, specificity of the tool to digital health and accessibility via digital formats.

CONCLUSIONS

Cost-IS is a pragmatic data collection instrument designed to estimate the costs of implementation strategies for digital health solutions. Further piloting of Cost-IS is required to establish its feasibility and generalisability.

Development and usability testing of a tool to estimate the budget impact of implementing a smoking cessation intervention for cancer patients

PURPOSE

Budget impact analyses (BIAs) aim to estimate costs of evidence-based programs in specific contexts, an important component of implementation decision making. We developed a BIA tool focused on the Smokefree Support Study, a cost-effective smoking cessation program for patients entering cancer care and refined the tool through usability testing.

METHODS

The BIA tool was structured using data from the Smokefree Support Study cost-effectiveness study. We refined the tool via usability testing to improve functionality and gain insight into the tool's potential for informing adoption decisions. We recruited participants from sites participating in the NCI-funded Cancer Center Cessation Initiative.

RESULTS

The final BIA tool allowed users generate context-specific cost estimates. Usability testing interviews informed changes to improve the BIA tool's usability and also illustrated users' natural inclination toward adaptation, helped identify the target audience for the tool, and underscored that cost results should be contextualized with other decision criteria to support program adoption.

CONCLUSION

We developed a BIA tool with which users can generate context-specific cost estimates of the Smokefree Support Study program. The breadth of usability feedback provided by participants and perspectives on using the BIA tool underscore the importance of involving end users in the development of tools and products.

Challenges and recommendations for collecting and quantifying implementation costs in practice: a qualitative interview study

BACKGROUND

The cost of implementation is typically not accounted for in published economic evaluations, which determine the relative value for money of health innovations and are important for allocating scarce resources. Despite key papers outlining relevant implementation costs, they continue to be under reported in the literature and often not considered in practice. This study sought to understand and outline current practices for capturing the costs associated with implementation efforts, with examples from the digital health setting.

METHODS

A qualitative study of semi-structured interviews with purposefully sampled experts in implementation science, health economics and/or digital health was conducted. The interview guide was informed by a literature review and was pilot tested. Interviews were digitally recorded and transcribed. A hybrid inductive/deductive framework analysis was conducted using thematic analysis to elicit key concepts related to the research question.

RESULTS

Interviews were conducted with sixteen participants with specialist expertise in implementation science (n = 8), health economics (n = 6), and/or digital health (n = 8). Five participants were experienced in more than one field. Four key themes were elicited from the data: difficulty identifying and collecting implementation cost data; variation in approaches for collecting implementation cost data; the value of implementation costs; and collaboration enables implementation costing. Broadly, while interviewees recognised implementation costs as important, only some costs were considered in practice likely due to the perceived ill-defined boundaries and inconsistencies in terminology. A variety of methods were used to collect and estimate implementation costs; the most frequent approach was staff time tracking. Multidisciplinary collaboration facilitated this process, but the burden of collecting the necessary data was also highlighted.

CONCLUSIONS

In current practice, standardised methods are not commonly used for data collection or estimation of implementation costs. Improved data collection through standardised practices may support greater transparency and confidence in implementation cost estimates. Although participants had industry exposure, most were also academic researchers and findings may not be representative of non-academic industry settings.

A Mixed Methods Examination of Session Planning Among Public Mental Health Therapists

Session planning is a core activity for implementing evidence-based practices (EBPs), yet it is unknown whether public mental health settings provide the support for therapists to session plan. This two-part study conducted in collaboration with EBP leaders in Philadelphia's public mental health system deployed mixed methods to examine therapists' session planning practices and preferences. In Study 1, 61 public mental health therapists completed an online survey to identify session planning barriers and facilitators, current practices, and desired planning supports. In Study 2, nine therapists who ranked a session planning tool as a top choice support in Study 1 participated in two focus groups to elaborate on their survey responses and provide feedback on three session planning tool prototypes. Study 1 survey respondents cited multi-level barriers and facilitators to session planning. In both closed- and open-ended responses, analyzed descriptively and via content analysis respectively, therapists described wanting more time, lower caseloads, financial incentives for session planning, and additional clinical resources and guidance from trainings, peers, and supervisors to support session planning. Study 2 focus group participants, whose responses were analyzed using content analysis, reiterated the need for these multi-level supports and expressed the need for a "one-stop" database of session planning tools that would be free, easily searchable, and modifiable for varied clinical needs. All three session planning tool prototypes reviewed were acceptable; two were also considered feasible and appropriate. This investigation of an under-studied aspect of the EBP implementation process reveals the need for multi-level session planning supports.

The intersection of health and housing: Analysis of the research portfolios of the National Institutes of Health, Centers for Disease Control and Prevention, and U.S. Department of Housing and Urban Development

BACKGROUND

Housing is a major social determinant of health that affects health status and outcomes across the lifespan.

OBJECTIVES

An interagency portfolio analysis assessed the level of funding invested in "health and housing research" from fiscal years (FY) 2016-2020 across the National Institutes of Health (NIH), the United States Department of Housing and Urban Development (HUD), and the Centers for Disease Control and Prevention (CDC) to characterize the existing health and housing portfolio and identify potential areas for additional research and collaboration.

METHODS/RESULTS

We identified NIH, HUD, and CDC research projects that were relevant to both health and housing and characterized them by housing theme, health topic, population, and study design. We organized the assessment of the individual housing themes by four overarching housing-to-health pathways. From FY 2016-2020, NIH, HUD, and CDC funded 565 health and housing projects combined. The Neighborhood pathway was most common, followed by studies of the Safety and Quality pathway. Studies of the Affordability and Stability pathways were least common. Health topics such as substance use, mental health, and cardiovascular disease were most often studied. Most studies were observational (66%); only a little over one fourth (27%) were intervention studies.

DISCUSSION

This review of the research grant portfolios of three major federal funders of health and housing research in the United States describes the diversity and substantial investment in research at the intersection between housing and health. Analysis of the combined portfolio points to gaps in studies on causal pathways linking housing to health outcomes. The findings highlight the need for research to better understand the causal pathways from housing to health and prevention intervention research, including rigorous evaluation of housing interventions and policies to improve health and well-being.

Establishing a Protocol for Determining the Costs of an Integrated Set of Evidence-based Practices Aimed at Reducing Opioid Overdose Deaths

OBJECTIVES

In the midst of the opioid overdose crisis, local jurisdictions face a choice of public health interventions. A significant barrier when considering evidence-based practices (EBPs) is the lack of information regarding their implementation cost. This protocol paper provides the methodological foundation for the economic cost evaluations of community-wide strategies on the scale of a national study. It can serve as a resource for other communities, local policymakers, and stakeholders as they consider implementing possible public health strategies in their unique settings.

METHODS

We present a protocol that details (1) the process of identifying, reviewing, and analyzing individual strategies for study-funded and non-study-funded costs; (2) prospective costing tool designation, and; (3) data collection. To do this, we set up working groups with community stakeholders, reviewed financial invoices, and surveyed individuals with detailed knowledge of their community implementation.

DISCUSSION

There were 3 main challenges/limitations. The first was the lack of a standard structure for documenting nonfunded costs associated with each strategy. The second was the need for timely implementation of cost data. The third was generalizability because our study designed its strategies for selected communities due to their high opioid overdose mortality rates. Future steps include more tailored questions to ask during the categorization/filter process and establishing realistic expectations for organizations regarding documenting.

CONCLUSIONS

Data collected will provide a critical methodological foundation for costing large community-based EBP strategies and provide clarity for stakeholders on the cost of implementing EBP strategies to reduce opioid overdose deaths.

The Costs of Implementing a Conversation Aid for Uterine Fibroids in Multiple Health Care Settings

BACKGROUND

Health care organizations considering adopting a conversation aid (CA), a type of patient decision aid innovation, need information about the costs of implementation.

OBJECTIVES

The aims of this study were to: (1) calculate the costs of introducing a CA in a study of supported implementation in 5 gynecologic settings that manage individuals diagnosed with uterine fibroids and (2) estimate the potential costs of future clinical implementation efforts in hypothetical settings.

RESEARCH DESIGN

We used time-driven activity-based costing to estimate the costs of CA implementation at multiple steps: integration with an electronic health record, preimplementation, implementation, and sustainability. We then estimated costs for 2 disparate hypothetical implementation scenarios.

SUBJECTS AND DATA COLLECTION

We conducted semistructured interviews with participants and examined internal documentation.

RESULTS

We interviewed 41 individuals, analyzed 51 documents and 100 emails. Overall total implementation costs over ∼36 months of activities varied significantly across the 5 settings, ranging from $14,157 to $69,134. Factors influencing costs included size/complexity of the setting, urban/rural location, practice culture, and capacity to automate patient identification. Initial investments were substantial, comprising mostly personnel time. Settings that embedded CA use into standard workflows and automated identification of appropriate patients had the lowest initial investment and sustainability costs. Our estimates of the costs of sustaining implementation were much lower than initial investments and mostly attributable to CA subscription fees.

CONCLUSION

Initiation and implementation of the interventions require significant personnel effort. Ongoing costs to maintain use are much lower and are a small fraction of overall organizational operating costs.

The Price of Practice Change: Assessing the Cost of Integrating Research Findings Into Clinical Practice

BACKGROUND

Clinicians, health care administrators, and implementation scientists know that it takes intentional effort, resources, and implementation strategies to integrate research findings into routine clinical practice. An oft-cited concern for those considering whether and how to implement an evidence-based program is how much it will cost to implement the change. Yet information about the cost of implementation is not often available to health care decision-makers. Teams that received Implementation Award funding from PCORI are conducting implementation projects to promote the uptake of evidence-based practices in health care settings. As part of their implementation efforts, a number of teams have examined the costs of implementation. In this Topical Collection, 5 teams will report their findings on implementation costs and discuss their methods for data collection and analysis.

DISCUSSION

The teams' costing efforts provide specific information about the costs sites can expect to incur in promoting the uptake of specific evidence-based programs. In addition, the papers illuminate 3 key features of the teams' approaches to measuring the cost of implementation: (1) the use of specific micro-costing methods with time-driven activity-based costing serving as the most popular method; (2) different ways to categorize and organize costs, including a site-based and non-site-based framework; and (3) cost collection challenges experienced by the teams.

CONCLUSION

The cost of implementation is a critical consideration for organizations seeking to improve practice in accordance with research findings. This Topical Collection describes detailed approaches to providing this type of cost information and highlights insights to be gained from a rigorous focus on implementation cost.

Cost components of school-based oral health-promoting programs: A systematic review protocol

INTRODUCTION

Oral health-promoting school programs play a crucial role in achieving universal coverage of oral health care, by addressing oral diseases and promoting the well-being and quality of life of children and adolescents. However, a lack of studies has evaluated the costs associated with implementing these programs, which hinders decision-makers in adopting them on a large scale. This review aims to assess the cost components involved in school-based oral health-promoting programs.

METHODS

This review will include studies that have conducted either partial or full economic evaluations, focusing on describing the cost components of oral health-promoting programs implemented in primary schools involving students aged 6 to 14. A systematic search was conducted across multiple databases: MEDLINE, The Cochrane Library, the Virtual Health Library, the NHS Economic Evaluation Database, Web of Science, Scopus, and EMBASE. Additionally, gray literature was searched using the Health Technology Assessment Database. Two independent reviewers will screen the titles and abstracts, followed by a full-text review based on predefined inclusion criteria. Data extraction and critical appraisal evaluation will also be carried out independently by two reviewers. In case of disagreements, the reviewer team will resolve them through discussion.

DISCUSSION

The systematic review resulting from this protocol aims to provide evidence regarding the cost components and necessary resources for implementing and maintaining oral health-promoting school programs. This information can assist decision-makers in adopting these programs on a larger scale and effectively addressing oral diseases among children and adolescents.

PROTOCOL REGISTRATION

CRD42022363743.

Cost of Implementing an Evidence-Based Intervention to Support Safer Use of Antipsychotics in Youth

To estimate the cost of implementing a clinical program designed to support safer use of antipsychotics in children and adolescents (youth) age 3-17 years at the time of initiating an antipsychotic medication. We calculate the costs of implementing a psychiatric consultation and navigation program for youth prescribed antipsychotic medications across 4 health systems, which included an electronic health record (EHR) decision support tool, consultation with a child and adolescent psychiatrist, and up to 6 months of behavioral health care navigation, as well as telemental health for patients (n = 348). Cost data were collected for both start-up and ongoing intervention phases and are estimated over a 1-year period. Data sources included study records and time-in-motion reports, analyzed from a health system perspective. Costs included both labor and nonlabor costs (2019 US dollars). The average total start-up and ongoing costs per health system were $34,007 and $185,174, respectively. The average total cost per patient was $2,128. The highest average ongoing labor cost components were telemental health ($901 per patient), followed by child and adolescent psychiatrist consultation ($659), and the lowest cost component was primary care/behavioral health provider time to review/respond to the EHR decision support tool and case consultation ($24). For health systems considering programs to promote safer and targeted use of antipsychotics among youth, this study provides estimates of the full start-up and ongoing costs of an EHR decision support tool, psychiatric consultation service, and psychotherapeutic services for patients and families.Trial registration: Clinicaltrials.gov, NCT03448575.

Costs of implementing a multi-site facilitation intervention to increase access to medication treatment for opioid use disorder

BACKGROUND

The United States has been grappling with the opioid epidemic, which has resulted in over 75,000 opioid-related deaths between April 2020 and 2021. Evidence-based pharmaceutical interventions (buprenorphine, methadone, and naltrexone) are available to reduce opioid-related overdoses and deaths. However, adoption of these medications for opioid use disorder has been stifled due to individual- and system-level barriers. External facilitation is an evidence-based implementation intervention that has been used to increase access to medication for opioid use disorder (MOUD), but the implementation costs of external facilitation have not been assessed. We sought to measure the facility-level direct costs of implementing an external facilitation intervention for MOUD to provide decision makers with estimates of the resources needed to implement this evidence-based program.

METHODS

We performed a cost analysis of the pre-implementation and implementation phases, including an itemization of external facilitation team and local site labor costs. We used labor estimates from the Bureau of Labor and Statistics, and sensitivity analyses were performed using labor estimates from the Veterans Health Administration (VHA) Financial Management System general ledger data.

RESULTS

The average total costs for implementing an external facilitation intervention for MOUD per site was $18,847 (SD 6717) and ranged between $11,320 and $31,592. This translates to approximately $48 per patient with OUD. Sites with more encounters and participants with higher salaries in attendance had higher costs. This was driven mostly by the labor involved in planning and implementation activities. The average total cost of the pre-implementation and implementation activities were $1031 and $17,816 per site, respectively. In the sensitivity analysis, costs for VHA were higher than BLS estimates likely due to higher wages.

CONCLUSIONS

Implementing external facilitation to increase MOUD prescribing may be affordable depending on the payer's budget constraints. Our study reported that there were variations in the time invested at each phase of implementation and the number and type of participants involved with implementing an external facilitation intervention. Participant composition played an important role in total implementation costs, and decision makers will need to identify the most efficient and optimal number of stakeholders to involve in their implementation plans.

Models, frameworks and theories in the implementation of programs targeted to reduce formal coercion in mental health settings: a systematic review

Introduction

Implementation models, frameworks and theories (hereafter tools) provide researchers and clinicians with an approach to understand the processes and mechanisms for the successful implementation of healthcare innovations. Previous research in mental health settings has revealed, that the implementation of coercion reduction programs presents a number of challenges. However, there is a lack of systematized knowledge of whether the advantages of implementation science have been utilized in this field of research. This systematic review aims to gain a better understanding of which tools have been used by studies when implementing programs aiming to reduce formal coercion in mental health settings, and what implementation outcomes they have reported.

Methods

A systematic search was conducted using PubMed, CINAHL, PsycINFO, Cochrane, Scopus, and Web of Science. A manual search was used to supplement database searches. Quality appraisal of included studies was undertaken using MMAT-Mixed Methods Appraisal Tool. A descriptive and narrative synthesis was formed based on extracted data. Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) guidelines were followed in this review.

Results

We identified 5,295 references after duplicates were removed. Four additional references were found with a manual search. In total eight studies reported in nine papers were included in the review. Coercion reduction programs that were implemented included those that were holistic, and/or used professional judgement, staff training and sensory modulation interventions. Eight different implementation tools were identified from the included studies. None of them reported all eight implementation outcomes sought from the papers. The most frequently reported outcomes were acceptability (4/8 studies) and adaptation (3/8). With regards to implementation costs, no data were provided by any of the studies. The quality of the studies was assessed to be overall quite low.

Discussion

Systematic implementation tools are seldom used when efforts are being made to embed interventions to reduce coercive measures in routine mental health care. More high-quality studies are needed in the research area that also involves perspectives of service users and carers. In addition, based on our review, it is unclear what the costs and resources are needed to implement complex interventions with the guidance of an implementation tool.

Systematic review registration

[Prospero], identifier [CRD42021284959].

Programmatic Costs of Project ImPACT for Children with Autism: A Time-Driven Activity Based Costing Study

Programmatic cost assessment of clinical interventions can inform future dissemination and implementation efforts. We conducted a randomized trial of Project ImPACT (Improving Parents As Communication Teachers) in which community early intervention (EI) providers coached caregivers in techniques to improve young children's social communication skills. We estimated implementation and intervention costs while demonstrating an application of Time-Driven Activity-Based Costing (TDABC). We defined Project ImPACT implementation and intervention as processes that can be broken down successively into a set of procedures. We created process maps for both implementation and intervention delivery. We determined resource use and costs, per unit procedure in the first year of the program, from a payer perspective. We estimated total implementation cost per clinician and per site, intervention cost per child, and provided estimates of total hours spent and associated costs for implementation strategies, intervention activities and their detailed procedures. Total implementation cost was $43,509 per clinic and $14,503 per clinician. Clinician time (60%) and coach time (12%) were the most expensive personnel resources. Implementation coordination and monitoring (47%), ongoing consultation (26%) and clinician training (19%) comprised most of the implementation cost, followed by fidelity assessment (7%), and stakeholder engagement (1%). Per-child intervention costs were $2619 and $9650, respectively, at a dose of one hour per week and four hours per week Project ImPACT. Clinician and clinic leader time accounted for 98% of per child intervention costs. Highest cost intervention activity was ImPACT delivery to parents (89%) followed by assessment for child's ImPACT eligibility (10%). The findings can be used to inform funding and policy decision-making to enhance early intervention options for young children with autism. Uncompensated time costs of clinicians are large which raises practical and ethical concerns and should be considered in planning of implementation initiatives. In program budgeting, decisionmakers should anticipate resource needs for coordination and monitoring activities. TDABC may encourage researchers to assess costs more systematically, relying on process mapping and gathering prospective data on resource use and costs concurrently with their collection of other trial data.

Implementation costs of hospital-based computerised decision support systems: a systematic review

BACKGROUND

The importance of accurately costing implementation strategies is increasingly recognised within the field of implementation science. However, there is a lack of methodological guidance for costing implementation, particularly within digital health settings. This study reports on a systematic review of costing analyses conducted alongside implementation of hospital-based computerised decision support systems.

METHODS

PubMed, Embase, Scopus and CINAHL databases were searched between January 2010 and August 2021. Two reviewers independently screened and selected original research studies that were conducted in a hospital setting, examined the implementation of a computerised decision support systems and reported implementation costs. The Expert Recommendations for Implementing Change Framework was used to identify and categorise implementation strategies into clusters. A previously published costing framework was applied to describe the methods used to measure and value implementation costs. The reporting quality of included studies was assessed using the Consolidated Health Economic Evaluation Reporting Standards checklist.

RESULTS

Titles and abstracts of 1836 articles were screened, with nine articles eligible for inclusion in the review. Implementation costs were most frequently reported under the 'evaluative and iterative strategies' cluster, followed by 'provide interactive assistance'. Labour was the largest implementation-related cost in the included papers, irrespective of implementation strategy. Other reported costs included consumables, durable assets and physical space, which was mostly associated with stakeholder training. The methods used to cost implementation were often unclear. There was variation across studies in the overall quality of reporting.

CONCLUSIONS

A relatively small number of papers have described computerised decision support systems implementation costs, and the methods used to measure and value these costs were not well reported. Priorities for future research should include establishing consistent terminology and appropriate methods for estimating and reporting on implementation costs.

TRIAL REGISTRATION

The review protocol is registered with PROSPERO (ID: CRD42021272948).

Implementing brief and low-intensity psychological interventions for children and young people with internalizing disorders: a rapid realist review

INTRODUCTION

Many children fail to receive the mental health treatments they need, despite strong evidence demonstrating efficacy of brief and low-intensity psychological interventions. This review identifies the barriers and facilitators to their implementation.

SOURCES OF DATA

PsycInfo, EMBASE and Medline were searched and a systematic approach to data extraction using Normalization Process Theory highlighted key mechanisms and contextual factors.

AREAS OF AGREEMENT

Ten interventions from 9 papers, including 371 young people, were included. Studies identified organizational demands, lack of implementation strategy and stigma as barriers to implementation, and clear training and plans for implementation as facilitators.

AREAS OF CONTROVERSY

No standardized implementation outcomes were used across papers so meta-analysis was not possible.

GROWING POINTS

Barriers and facilitators have been clearly identified across different settings.

AREAS TIMELY FOR DEVELOPING RESEARCH

Longitudinal studies can identify methods and processes for enhancing long-term implementation and considers ways to monitor and evaluate uptake into routine practice.

Implementing trauma-focused cognitive behavioral therapy in Philadelphia: A 10-year evaluation

Background: In 2012, Philadelphia's Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) developed an initiative to implement an evidence-based treatment for posttraumatic stress disorder (PTSD), trauma-focused cognitive behavioral therapy (TF-CBT), across the city's behavioral health system. This report evaluates the initiative's 10-year implementation and effectiveness outcomes. Method: The Exploration, Preparation, Implementation, and Sustainment framework guided our implementation evaluation. The implementation outcomes include adoption, reach, and sustainment; these were obtained during regular evaluation data collection from publicly funded behavioral health agencies participating in the TF-CBT initiative. We analyze effectiveness outcomes (i.e., changes in PTSD symptoms) from a subset of patients receiving TF-CBT, which were collected in 6-month intervals by our research team between 2013 and 2021. Results: From 2012 to 2021, DBHIDS trained 478 clinicians in TF-CBT across 20 behavioral health agencies. During this time, 23,401 youths were screened for potentially traumatic events and PTSD symptoms, and 7,550 youths received TF-CBT. Through the TF-CBT initiative, the city expanded the network of TF-CBT providers from 3 to 20 agencies. DBHIDS sustained this network by maintaining the participation of 16 behavioral health agencies over the course of a decade. The subset of 202 youths who were evaluated to assess TF-CBT effectiveness was drawn from 94 therapists and 20 agencies across Philadelphia. All participating youths completed a baseline assessment, and 151 (75%) completed at least one follow-up assessment. Linear mixed-effects models accounting for observations nested within participants and nested within clinicians found that treatment significantly reduced PTSD symptoms. Conclusion: Between 2012 and 2021, DBHIDS successfully implemented and sustained TF-CBT across the city's behavioral health system. Adoption, reach, and sustainment of TF-CBT were high. Despite the considerable adverse experiences faced by youths seeking treatment in Philadelphia's behavioral health system, TF-CBT was effective. Future directions to improve TF-CBT implementation in the next iteration of the initiative are described.

Using decision analysis to support implementation planning in research and practice

BACKGROUND

The process of implementing evidence-based interventions, programs, and policies is difficult and complex. Planning for implementation is critical and likely plays a key role in the long-term impact and sustainability of interventions in practice. However, implementation planning is also difficult. Implementors must choose what to implement and how best to implement it, and each choice has costs and consequences to consider. As a step towards supporting structured and organized implementation planning, we advocate for increased use of decision analysis.

MAIN TEXT

When applied to implementation planning, decision analysis guides users to explicitly define the problem of interest, outline different plans (e.g., interventions/actions, implementation strategies, timelines), and assess the potential outcomes under each alternative in their context. We ground our discussion of decision analysis in the PROACTIVE framework, which guides teams through key steps in decision analyses. This framework includes three phases: (1) definition of the decision problems and overall objectives with purposeful stakeholder engagement, (2) identification and comparison of different alternatives, and (3) synthesis of information on each alternative, incorporating uncertainty. We present three examples to illustrate the breadth of relevant decision analysis approaches to implementation planning.

CONCLUSION

To further the use of decision analysis for implementation planning, we suggest areas for future research and practice: embrace model thinking; build the business case for decision analysis; identify when, how, and for whom decision analysis is more or less useful; improve reporting and transparency of cost data; and increase collaborative opportunities and training.

Scoping review of costs of implementation strategies in community, public health and healthcare settings

OBJECTIVES

To identify existing evidence concerning the cost of dissemination and implementation (D&I) strategies in community, public health and health service research, mapped with the 'Expert Recommendations for Implementing Change' (ERIC) taxonomy.

DESIGN

Scoping review.

DATA SOURCES

MEDLINE, EMBASE, CINAHL, PsycINFO, Scopus and the Cochrane Library were searched to identify any English language reports that had been published between January 2008 and December 2019 concerning the cost of D&I strategies.

DATA EXTRACTION

We matched the strategies identified in each article using ERIC taxonomies; further classified them into five areas (eg, dissemination, implementation, integration, capacity building and scale-up); and extracted the corresponding costs (total costs and cots per action target and per evidence-based programme (EBP) participant). We also recorded the reported level of costing methodology used for cost assessment of D&I strategies.

RESULTS

Of the 6445 articles identified, 52 studies were eligible for data extraction. Lack of D&I strategy cost data was the predominant reason (55% of the excluded studies) for study exclusion. Predominant topic, setting, country and research design in the included studies were mental health (19%), primary care settings (44%), the US (35%) and observational (42%). Thirty-five (67%) studies used multicomponent D&I strategies (ranging from two to five discrete strategies). The most frequently applied strategies were Conduct ongoing training (50%) and Conduct educational meetings (23%). Adoption (42%) and reach (27%) were the two most frequently assessed outcomes. The overall costs of Conduct ongoing training ranged from $199 to $105 772 ($1-$13 973 per action target and $0.02-$412 per EBP participant); whereas the cost of Conduct educational meetings ranged from $987 to $1.1-$2.9 million/year ($33-$54 869 per action target and $0.2-$146 per EBP participant). The wide range of costs was due to the varying scales of the studies, intended audiences/diseases and the complexities of the strategy components. Most studies presented limited information on costing methodology, making interpretation difficult.

CONCLUSIONS

The quantity of published D&I strategy cost analyses is increasing, yet guidance on conducting and reporting of D&I strategy cost analysis is necessary to facilitate and promote the application of comparative economic evaluation in the field of D&I research.

Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers

Background

Economic evaluations frequently are utilized to compare the value of different interventions in medicine and health in concrete terms. Implementation science also would benefit from the incorporation of economic evaluations, but such studies are rare in the literature. The National Cancer Institute has supported a special collection of articles focusing on economic evaluations in implementation science. Even when interventions are supported by substantial evidence, they are implemented infrequently in the field. Implementation costs are important determinants for whether organizational decision-makers choose to adopt an intervention and whether the implementation process is successful. Economic evaluations, such as cost-effectiveness analyses, can help organizational decision-makers choose between implementation approaches for evidence-based interventions by accounting for costs and succinctly presenting cost/benefit tradeoffs.

Main text

This manuscript presents a discussion of important considerations for incorporating economic evaluations into implementation science. First, the distinction between intervention and implementation costs is presented, along with an explanation of why the comprehensive representation of implementation costs is elusive. Then, the manuscript describes how economic evaluations in implementation science may differ from those in medicine and health intervention studies, especially in terms of determining the perspectives and outcomes of interest. Finally, referencing a scale-up trial of an evidence-based behavioral health intervention, concrete case examples of how cost data can be collected and used in economic evaluations targeting implementation, rather than clinical outcomes, are described.

Conclusions

By gaining a greater understanding of the costs and economic impact associated with different implementation approaches, organizational decision-makers will have better transparency for future replication and scale-up. The use of economic evaluations can help to advance this understanding and provide researchers, purveyors or third-party intermediaries, and organizational decision-makers with essential information to facilitate implementation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00295-1.

Design and Implementation of Online Intelligent Mental Health Testing Platform

In order to solve the problems of high misevaluation rate and low work efficiency in the process of mental health intelligent evaluation, a method of mental health intelligent evaluation system oriented to the decision tree algorithm is proposed. First, the current research status of mental health intelligent evaluation was analyzed and the framework of mental health intelligent evaluation system was constructed. Then, the mental health intelligent evaluation data were collected and the decision tree algorithm was used to analyze and classify the mental health intelligent evaluation data to obtain the mental health intelligent evaluation results. Finally, specific simulation experiments are used to analyze the feasibility and superiority of the mental health intelligent evaluation system. The experimental results show that the recall rate of each system increases with the increasing number of iterations, and the system has the highest recall rate. Also, it is stable after the number of iterations reaches 20, with good recall and adaptive scheduling performance. The recall rate of comparison system 1 and comparison system 2 fluctuates greatly, and the recall rate is lower than that of the system in this paper. It is proved that the method of the mental health intelligent evaluation system of the decision tree algorithm can effectively solve the problem and improve the accuracy of the mental health intelligent evaluation. The efficiency of mental health intelligent evaluation is improved, and the system stability is better, which can meet the actual requirements of current mental health intelligent evaluation.

Cost data in implementation science: categories and approaches to costing

A lack of cost information has been cited as a barrier to implementation and a limitation of implementation research. This paper explains how implementation researchers might optimize their measurement and inclusion of costs, building on traditional economic evaluations comparing costs and effectiveness of health interventions. The objective of all economic evaluation is to inform decision-making for resource allocation and to measure costs that reflect opportunity costs—the value of resource inputs in their next best alternative use, which generally vary by decision-maker perspective(s) and time horizon(s). Analyses that examine different perspectives or time horizons must consider cost estimation accuracy, because over longer time horizons, all costs are variable; however, with shorter time horizons and narrower perspectives, one must differentiate the fixed and variable costs, with fixed costs generally excluded from the evaluation. This paper defines relevant costs, identifies sources of cost data, and discusses cost relevance to potential decision-makers contemplating or implementing evidence-based interventions. Costs may come from the healthcare sector, informal healthcare sector, patient, participant or caregiver, and other sectors such as housing, criminal justice, social services, and education. Finally, we define and consider the relevance of costs by phase of implementation and time horizon, including pre-implementation and planning, implementation, intervention, downstream, and adaptation, and through replication, sustainment, de-implementation, or spread.

Applying a scoping review approach for identifying effective implementation strategies in oral health settings

Dental service providers have limited capacity to identify strategies to implement evidence-based practices (EBPs). We developed a rigorous yet parsimonious scoping review approach to identify, select, and rate implementation strategies based on an oral health system context. From 153 strategies identified, we selected the top 11 strategies, which had a moderate level of support of evidence and where managers were the main actors. The main actions were to educate, remind, structure, and influence. Targets included dentists, dental hygienists, and assistants and managers from a large prepaid dental care delivery system. This approach responds to calls for rapid and innovative methods to implement EBPs in oral health.

Implementation science issues in understanding, collecting, and using cost estimates: a multi-stakeholder perspective

Understanding the resources needed to achieve desired implementation and effectiveness outcomes is essential to implementing and sustaining evidence-based practices (EBPs). Despite this frequent observation, cost and economic measurement and reporting are rare, but becoming more frequent in implementation science, and when present is seldom reported from the perspective of multiple stakeholders (e.g., the organization, supervisory team), including those who will ultimately implement and sustain EBPs.Incorporating a multi-level framework is useful for understanding and integrating the perspectives and priorities of the diverse set of stakeholders involved in implementation. Stakeholders across levels, from patients to delivery staff to health systems, experience different economic impacts (costs, benefit, and value) related to EBP implementation and have different perspectives on these issues. Economic theory can aid in understanding multi-level perspectives and approaches to addressing potential conflict across perspectives.This paper provides examples of key cost components especially important to different types of stakeholders. It provides specific guidance and recommendations for cost assessment activities that address the concerns of various stakeholder groups, identifies areas of agreement and conflict in priorities, and outlines theoretically informed approaches to understanding conflicts among stakeholder groups and processes to address them. Involving stakeholders throughout the implementation process and presenting economic information in ways that are clear and meaningful to different stakeholder groups can aid in maximizing benefits within the context of limited resources. We posit that such approaches are vital to advancing economic evaluation in implementation science. Finally, we identify directions for future research and application.Considering a range of stakeholders is critical to informing economic evaluation that will support appropriate decisions about resource allocation across contexts to inform decisions about successful adoption, implementation, and sustainment. Not all perspectives need to be addressed in a given project but identifying and understanding perspectives of multiple groups of key stakeholders including patients and direct implementation staff not often explicitly considered in traditional economic evaluation are needed in implementation research.