Comparing rating paradigms for evidence-based program registers in behavioral health: evidentiary criteria and implications for assessing programs

Best practice portals in health promotion and disease prevention: approaches, definitions, and intervention evaluation criteria

Introduction

The evaluation of practices is a valuable source of evidence in the context of an evidence-based approach to public health. Best practice portals (BPPs) are promising tools for facilitating access to recommended programmes, monitoring and improving the quality of interventions. There are several such portals in Europe, but there is little work in the scientific literature on the subject. The study aimed to identify and characterise BPPs in health promotion and disease prevention and analyse the approaches, definitions, and criteria for evaluating interventions.

Methods

To identify portals, websites of public health institutions and organisations, the PubMed database and grey literature were searched. The material consisted of elements of each portal's design, information available on their websites, and collected publications. The study applied a qualitative analysis with a descriptive approach and covered a detailed description of the four selected portals.

Results

Among the analysed BPPs, three were from the European region, and one was from Canada (pioneer in developing best practice tools). The dates of launching the portals ranged from the year 2003 to 2016. The number of interventions collected in the databases ranged from 120 to 337. Portals were useful, well-designed, and developed tools. BPPs differed in terms of their objectives and roles, adopted standards and criteria for assessing practices, and other operational factors. In each portal, interventions underwent a rigorous and multilevel assessment process conducted by independent experts in the field and based on intervention evaluation criteria. Generally, the analysed catalogues described similar issues, e.g., Selection of the issue addressed by the practice, Description of a particular element of the practice, Theoretical foundation, or Evaluation/Effectiveness. However, we identified both similarities and differences in the adopted terms (names of criteria) and their definitions. It was shown that sometimes the same criterion had different names depending on the catalogue. On the other hand, criteria with identical or similar names could be defined differently within the detailed thematic scope.

Conclusion

The similarities and differences presented in this work can serve as a valuable starting point for designing such tools to support practice-based and evidence-based decision-making in health promotion and disease prevention.

[Evidence Register Green List Prevention: Analysis of the listed effectiveness-tested programmes]

BACKGROUND

In the areas of prevention and health promotion, there is a large number of measures for children and adolescents. One way of facilitating evidence-based action for those involved in these taks is by making available online evidence registers with customised, effectiveness-tested measures. The Green List Prevention is such a register and offers an overview of evidence-based programmes in Germany, currently with a focus on psychosocial health.

OBJECTIVE

The aims of this study were (a) to analyse the characteristics of the available and evaluated programmes on the psychosocial health of children and adolescents, (b) to identify priorities and underrepresented areas of the Green List Prevention and (c) to optimise the search functions of the register.

METHOD

The characteristic features were recorded on the basis of the existing upper categories of the register entries which were differentiated into subcategories in an inductive procedure by at least two persons. In addition, deductive categories were added for relevant aspects concerning content and implementation. The upper and lower categories formed were operationalized with characteristic values. All entries were analyzed by using a data sheet and were descriptively evaluated.

RESULTS

The 102 programmes listed (as of 2/2024) addressed not only the primary target group of children and youth, but also secondary target groups (mainly teachers and guardians). Social and life skills programmes as well as trainings for guardians represented a focus. Behavioral prevention programmes on the topics of violence (including bullying) (63.7%), addiction (46.1%) and/or mental health (35.3%) were frequently represented, whereas nutrition and/or physical activity (4.9%) were hardly represented. Most of the programmes (88.2%) could be assigned to the eligibility criteria of the statutory health insurers (§20a SGB V). Potentials digital implementation forms and further implementation aspects were identified.

CONCLUSION

The Green List Prevention bundles a large number of different measures and that there is potential for expansion. Processing knowledge about effective measures in a user-friendly manner can be optimised through expanded search functions, so that resource-conserving, evidence-based action can be facilitated.

Open Science Standards at Journals that Inform Evidence-Based Policy

Evidence-based policy uses intervention research to inform consequential decisions about resource allocation. Research findings are often published in peer-reviewed journals. Because detrimental research practices associated with closed science are common, journal articles report more false-positives and exaggerated effect sizes than would be desirable. Journal implementation of standards that promote open science-such as the transparency and openness promotion (TOP) guidelines-could reduce detrimental research practices and improve the trustworthiness of research evidence on intervention effectiveness. We evaluated TOP implementation at 339 peer-reviewed journals that have been used to identify evidence-based interventions for policymaking and programmatic decisions. Each of ten open science standards in TOP was not implemented in most journals' policies (instructions to authors), procedures (manuscript submission systems), or practices (published articles). Journals implementing at least one standard typically encouraged, but did not require, an open science practice. We discuss why and how journals could improve implementation of open science standards to safeguard evidence-based policy.

Organizational change and evidence-based practices in support services for people with intellectual and developmental disabilities

The impact on support services for persons with intellectual and developmental disabilities of the socioeconomic movements and theoretical reformulations of the last decades has generated the necessity, in order to guarantee their sustainability, to carry out processes of profound change in their organizational culture, intervening in the elements that compose it. Among them are professional practices as the best way to intervene in culture, with the use of comparative analysis between an organization's current practices and those expected with culture change. In this line, the organizational self-assessment tool "Organizational Effectiveness and Efficiency Scale" (OEES) is applied in a study with 24 organizations, which uses a collaborative assessment approach in the service of a set of evidence-based practices identified as standards in key aspects that guide culture change, specifically, a person-centered approach, participative structures, use of information systems and data management, implementation of quality systems and participative and transformational leadership. The results obtained show that a large majority of organizations have significant discrepancies between their current practices and evidence-based practices. The descriptive analysis allows affirming the usefulness of the scale for an organizational diagnosis and identification of strategies to guide transformational change.

Normalization of Prevention Principles and Practices to Reduce Substance Use Disorders Through an Integrated Dissemination and Implementation Framework

Major research breakthroughs over the past 30 years in the field of substance use prevention have served to: (1) enhance understanding of pharmacological effects on the central and peripheral nervous systems and the health and social consequences of use of psychoactive substances, particularly for children and adolescents; (2) delineate the processes that increase vulnerability to or protect from initiation of substance use and progression to substance use disorders (SUDs) and, based on this understanding, (3) develop effective strategies and practices to prevent the initiation and escalation of substance use. The challenge we now face as a field is to "normalize" what we have learned from this research so that it is incorporated into the work of those involved in supporting, planning, and delivering prevention programming to populations around the world, is integrated into health and social service systems, and helps to shape public policies. But we wish to go further, to incorporate these effective prevention practices into everyday life and the mind-sets of the public, particularly parents and educators. This paper reviews the advances that have been made in the field of prevention and presents a framework and recommendations to achieve these objectives generated during several meetings of prevention and implementation science researchers sponsored by the International Consortium of Universities for Drug Demand Reduction (ICUDDR) that guides a roadmap to achieve "normalization."

Health promotion and disease prevention registries in the EU: a cross country comparison

BACKGROUND

Health promotion and disease prevention programme registries (HPPRs), also called 'best practice portals', serve as entry points and practical repositories that provide decision-makers with easy access to (evidence-based) practices. However, there is limited knowledge of differences or overlaps of howe current national HPPRs in Europe function, the context and circumstances in which these HPPRs were developed, and the mechanisms utilised by each HPPR for the assessment, classification and quality improvement of the included practices. This study prepared an overview of different approaches in several national HPPRs and the EU Best Practice Portal (EU BPP) as well as identified commonalities and differences among the core characteristics of the HPPRs.

METHODS

We conducted a descriptive comparison - that focused on six European countries with existing or recently developed/implemented national HPPR and the EU BPP -to create a comparative overview. We used coding mechanisms to identify commonalities and differences; we performed data management, collection and building consensus during EuroHealthNet Thematic Working Group meetings.

RESULTS

All HPPRs offer a broad range of health promotion and disease-prevention practices and serve to support practitioners, policymakers and researchers in selecting practices. Almost all HPPRs have an assessment process in place or planned, requiring the application of assessment criteria that differ among the HPPRs. While all HPPRs collect and share recommendable practices, others have implemented further measures to improve the quality of the submitted practices. Different dissemination tools and strategies are employed to promote the use of the HPPRs, including social media, newsletters and publications as well as capacity building workshops for practice owners or technical options to connect citizens/patients with local practices.

CONCLUSIONS

Collaboration between HPPRs (at national and EU level) is appreciated, especially regarding the use consistent terminology to avoid misinterpretation, facilitate cross-country comparison and enable discussions on the adaption of assessment criteria by national HPPRs. Greater efforts are needed to promote the actual implementation and transfer of practices at the national level to address public health challenges with proven and effective practices.

A Scoping Review of the Influence of Evidence-Based Program Resources (EBPR) Websites for Behavioral Health

Evidence-based program resources (EBPR) websites evaluate behavioral health programs, practices or policies (i.e., interventions) according to a predetermined set of research criteria and standards, usually resulting in a summary rating of the strength of an intervention's evidence base. This study is a mixed-methods analysis of the peer-reviewed academic literature relating to the influence of EBPRs on clinical practice and policy in the behavioral health field. Using an existing framework for a scoping review, we searched for research articles in PubMed, Web of Science, SCOPUS, and ProQuest that were published between January 2002 and March 2022, referenced an EBPR or multiple EBPRs, and presented data showing the influence of one or more EBPRs on behavioral health. A total of 210 articles met the inclusion criteria and were classified into five distinct categories of influence, the most important of which was showing the direct impact of one or more EBPRs on behavioral health (8.1% of articles), defined as documenting observable changes in interventions or organizations that are at least partly due to information obtained from EBPR(s). These included impacts at the state legislative and policy-making level, at the community intervention level, provider agency level, and individual practitioner level. The majority of influences identified in the study were indirect demonstrations of how EBPRs are used in various ways. However, more studies are needed to learn about the direct impact of information from EBPRs on the behavioral health field, including impact on clinician practice and treatment outcomes for consumers.

The influence of evidence-based program registry websites for dissemination of evidence-based interventions in behavioral healthcare

PURPOSE

Evidence-based program registries (EBPRs) are web-based databases of evaluation studies that summarize the available evidence for the effectiveness of behavioral healthcare programs, including programs addressing substance misuse, mental health, child welfare, or offender rehabilitation. The study determined the extent to which visitors to selected EBPRs accomplished the objectives of their visits and how often those visits resulted in the adoption of new or improved evidence-based interventions (EBIs).

METHOD

A follow-up telephone survey was conducted with 216 visitors to a convenience sample of six EBPRs an average of six months after the visitors' incident visit to the EBPR.

RESULTS

The most frequent objective was to identify evidence-based programs/services, curricula or assessments, followed by finding resources to implement or improve the preceding and writing a grant proposal including to comply with funding requirements; 71% of such objectives were achieved across the full set of objectives. Implementation of an EBI was completely achieved for 31% of relevant objectives and some progress on EBI implementation occurred for 19% of relevant objectives.

CONCLUSIONS

This is the first study to document the usage of EBPRs as a modality to increase the utilization of EBIs in the actual practice of behavioral healthcare. The results support the continued use of web-based EBPRs for disseminating information on evidence-based interventions for behavioral healthcare.

Evidence Clearinghouses as Tools to Advance Health Equity: What We Know from a Systematic Scan

Evidence clearinghouses evaluate and summarize literature to help decision-makers prioritize and invest in evidence-informed interventions. Clearinghouses and related practice-oriented tools are continuously evolving; however, it is unclear the extent to which these tools assess and summarize evidence describing an intervention's impact on health equity. We conducted a systematic scan to explore how clearinghouses communicated an intervention's equity impact and reviewed their underlying methods and how they defined and operationalized health equity. In 2021, we identified 18 clearinghouses that were US-focused, web-based registries of interventions that assigned an intervention effectiveness rating for improving community health and the social determinants of health. We reviewed each clearinghouse's website and collected publicly available information about their health equity impact review, review methods, and health equity definitions and values. We conducted a comparative analysis among select clearinghouses using qualitative methods. Among the 18 clearinghouses, fewer than half (only seven) summarized an intervention's potential impact on health equity. Overall, those seven clearinghouses defined and operationalized equity differently, and most lacked transparency in their review methods. Clearinghouses used one or more approaches to communicate findings from their review: summarize study findings on differential impact for subpopulations, curate interventions that reduce health disparities, and/or assign a disparity/equity rating to each intervention. Evidence clearinghouses can enhance equity-focused methods and be transparent in their underlying values to better support the uptake and implementation of evidence-informed interventions to advance health equity. However, clearinghouses are unable to do so without underlying equity-focused empirical evidence.

Discrepancies in Ratings of Behavioral Healthcare Interventions Among Evidence-Based Program Resources Websites

Decision makers in the behavioral health disciplines could benefit from tools to assist them in identifying and implementing evidence-based interventions. One tool is an evidence-based program resources website (EBPR). Prior studies documented that when multiple EBPRs rate an intervention, they may disagree. Prior research concerning the reason for such conflicts is sparse. The present study examines how EBPRs rate interventions and the sources of disagreement between EBPRs when rating the same intervention. This study hypothesizes that EBPRs may disagree about intervention ratings because they either use different rating paradigms or they use different studies as evidence of intervention effectiveness (or both). This study identified 15 EBPRs for inclusion. One author (M.J.L.E.) coded the EBPRs for which "tiers of evidence" each EBPR used to classify behavioral health interventions and which criteria they used when rating interventions. The author then computed one Jaccard index of similarity for the criteria shared between each pair of EBPRs that co-rated interventions, and one for the studies used by EBPR rating pairs when rating the same program. The authors used a combination of chi-square, correlation, and binary logistic regression analyses to analyze the data. There was a statistically significant negative correlation between the number of Cochrane Risk of Bias criteria shared between 2 EBPRs and the likelihood of those 2 EBPRs agreeing on an intervention rating (r = -.12, P ≤ .01). There was no relationship between the number of studies evaluated by 2 EBPRs and the likelihood of those EBPRs agreeing on an intervention rating. The major reason for disagreements between EBPRs when rating the same intervention in this study was due to differences in the rating criteria used by the EBPRs. The studies used by the EBPRs to rate programs does not appear to have an impact.

State Department and Provider Agency Utilization of Evidence-Based Program Registries in Behavioral Healthcare and Child Welfare

Evidence-based program registries (EBPRs) are web-based compilations of behavioral healthcare programs/interventions that rely on research-based criteria to rate program efficacy or effectiveness for support of programmatic decision-making. The objective was to determine the extent to which behavioral health decision-makers access EBPRs and to understand whether and exactly how they use the information obtained from EPBRs. Single State Authorities (SSAs) and service provider agencies in the areas of behavioral health and child welfare were recruited nationally. Senior staff (n = 375) responsible for the selection and implementation of programs and/or policies were interviewed by telephone concerning their visits (if any) to 28 relevant EBPRs, the types of information they were seeking, whether they found it, and how they may have used that information to effect changes in their organizations. At least one EBPR was visited by 80% of the respondents, with a median of three different registers being visited. Most visitors (55%) found all the information they were seeking; those who did not desired more guidance or tools for individual program implementation or were unable to locate the program or practice that they were seeking. Most visitors (65%) related using the information obtained to make changes in their organizations, in particular to select, start or change a program, or to support the adoption or improvement of evidence-based clinical practices. EBPRs were shown to be important resources for dissemination of research-based program effectiveness data, leading to increased use of evidence-based practices in the field, but the study also identified needs for greater awareness of EBPRs generally and for more attention to implementation of specific recommended programs and practices.

How Consistently Do 13 Clearinghouses Identify Social and Behavioral Development Programs as "Evidence-Based"?

Clearinghouses develop scientific criteria that they then use to vet existing research studies on a program to reach a verdict about how evidence-based it is. This verdict is then recorded on a website in hopes that stakeholders in science, public policy, the media, and even the general public, will consult it. This paper (1) compares the causal design and analysis preferences of 13 clearinghouses that assess the effectiveness of social and behavioral development programs, (2) estimates how consistently these clearinghouses rank the same program, and then (3) uses case studies to probe why their conclusions differ. Most clearinghouses place their highest value on randomized control trials, but they differ in how they treat program implementation, quasi-experiments, and whether their highest program ratings require effects of a given size that independently replicate or that temporally persist. Of the 2525 social and behavioral development programs sampled over clearinghouses, 82% (n = 2069) were rated by a single clearinghouse. Of the 297 programs rated by two clearinghouses, agreement about program effectiveness was obtained for about 55% (n = 164), but the clearinghouses agreed much more on program ineffectiveness than effectiveness. Most of the inconsistency is due to clearinghouses' differences in requiring independently replicated and/or temporally sustained effects. Without scientific consensus about matters like these, "evidence-based" will remain more of an aspiration than achievement in the social and behavioral sciences.

Visitors' Assessment and Utilization of Evidence-Based Program Resources (EBPR) Websites

Background: Evidence-based program resources (EBPR) websites for behavioral health are a potentially useful tool to assist decision-makers and practitioners in deciding which behavioral health interventions to implement. EBPR websites apply rigorous research standards to assess the effectiveness of behavioral healthcare programs, models, and clinical practices. Method: Visitors to a convenience sample of six EBPR websites (N=369, excluding students) were recruited for telephone interviews primarily by means of a pop-up invitation on the sites. Results: The visitors view the EBPR sites as important sources of information to support the identification and adoption of evidence-based programs/practices (EBPs) in behavioral healthcare, which aligns with the primary mission of EBPRs. For repeat visitors, there was some indication that the information obtained helped effect certain changes in their agencies' programs and policies. However, increased or improved guidance on EBP implementation was also requested. Conclusion: EBPR websites should be better publicized to the behavioral healthcare field.

Variations in Definitions of Evidence-Based Interventions for Behavioral Health in Eight Selected U.S. States

BACKGROUND

U.S. state legislatures fill a vital role in supporting the use of evidence-based interventions (EBIs) through statutes and regulations (mandates).

OBJECTIVE

The study determined the terms used by selected states to describe EBIs and how those terms are defined in mandates.

RESEARCH METHODS

The mandates of eight purposely selected states were accessed and coded using the Westlaw Legal Research Database.

RESULTS

Considerable variation was found in the terms used by states to describe EBIs. Although "evidence-based" was the most frequently utilized term (60% of mandates), an additional 29 alternative terms appeared with varying frequencies. Most terms were simply mentioned, with no further definition or elaboration. When terms were further defined or elaborated, the majority were defined using numerous and different types of external sources or references. Three approaches were found in the mandates defining EBIs: "single definition," "hierarchies of evidence levels," and "best available evidence"; the states differed considerably in the approaches used in their mandates.

CONCLUSIONS

The variations in EBI-related terminology across states and within states, coupled with a lack of elaboration on the meaning of important terms and the predominant use of external rather than internal guidelines, may be a source of confusion for behavioral health provider agencies that seek direction about what constitutes an EBI. Prior studies indicate that many agencies may lack staff with the technical ability to adequately evaluate what constitutes an EBI. Thus, lack of clear guidance from official state government mandates may impede the implementation of EBIs within states.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are "evidence-based," clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as "evidence-based." In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions-an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed "TOP Guidelines for Clearinghouses" includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving "evidence-based" designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

References to Evidence-based Program Registry (EBPR) websites for behavioral health in U.S. state government statutes and regulations

BACKGROUND AND AIM

U.S. state governments have the responsibility to regulate and license behavioral healthcare interventions, such as for addiction and mental illness, with increasing emphasis on implementing evidence-based programs (EBPs). A serious obstacle to this is lack of clarity or agreement about what constitutes "evidence-based." The study's purpose was to determine the extent to which and in what contexts web-based Evidence-based Program Registries (EBPRs) are referenced in state government statutes and regulations ("mandates") concerning behavioral healthcare. Examples are: What Works Clearinghouse; National Register of Evidence-based Programs and Practices; Cochrane Database of Systematic Reviews.

METHODS

The study employed the Westlaw Legal Research Database to search for 30 known EBPR websites relevant to behavioral healthcare within the statutes and regulations of all 50 states.

RESULTS

There was low prevalence of EBPR references in state statutes and regulations pertaining to behavioral healthcare; 20 states had a total of 33 mandates that referenced an EBPR. These mandates usually do not rely on an EBPR as the sole acceptable source for classifying a program or practice as "evidence-based." Instead, EBPRs were named in conjunction with internal state or external sources of information about putative program effectiveness, which may be less valid than EBPRs, to determine what is "evidence-based."

CONCLUSION

Greater awareness of scientifically - based EBPRs and greater understanding of their advantages need to be fostered among state legislators and regulators charged with making policy to increase or improve the use of evidence-based programs and practices in behavioral healthcare in the U.S.

In Search of the Common Elements of Clinical Supervision: A Systematic Review

The importance of clinical supervision for supporting effective implementation of evidence-based treatments (EBTs) is widely accepted; however, very little is known about which supervision practice elements contribute to implementation effectiveness. This systematic review aimed to generate a taxonomy of empirically-supported supervision practice elements that have been used in treatment trials and shown to independently predict improved EBT implementation. Supervision practice elements were identified using a two-phase, empirically-validated distillation process. In Phase I, a systematic review identified supervision protocols that had evidence of effectiveness based on (a) inclusion in one or more EBT trials, and (b) independent association with improved EBT implementation in one or more secondary studies. In Phase II, a hybrid deductive-inductive coding process was applied to the supervision protocols to characterize the nature and frequency of supervision practice elements across EBTs. Twenty-one of the 876 identified articles assessed the associations of supervision protocols with implementation or clinical outcomes, representing 13 separate studies. Coding and distillation of the supervision protocols resulted in a taxonomy of 21 supervision practice elements. The most frequently used elements were: reviewing supervisees' practice (92%; n = 12), clinical suggestions (85%; n = 11), behavioral rehearsal (77%; n = 10), elicitation (77%; n = 10), and fidelity assessment (77%; n = 10). This review identified supervision practice elements that could be targets for future research testing which elements are necessary and sufficient to support effective EBT implementation. Discrepancies between supervision practice elements observed in trials as compared to routine practice highlights the importance of research addressing supervision-focused implementation strategies.

Utilization of Evidence-based Intervention Criteria in U.S. Federal Grant Funding Announcements for Behavioral Healthcare

Recent U.S. federal government policy has required or recommended the use of evidence-based interventions (EBIs), so that it is important to determine the extent to which this priority is reflected in actual federal solicitations for intervention funding, particularly for behavioral healthcare interventions. Understanding how well such policies are incorporated in federal opportunity announcements (FOAs) for grant funding could improve compliance with policy and increase the societal use of evidence-based interventions for behavioral healthcare. FOAs for discretionary grants (n = 243) in fiscal year 2021 were obtained from the Grants.gov website for 44 federal departments, agencies and sub-agencies that were likely to fund interventions in behavioral health-related areas. FOAs for block/formula grants to states that included behavioral healthcare (n = 17) were obtained from the SAM.gov website. Across both discretionary and block grants, EBIs were required in 60% and recommended in 21% of these FOAs for funding. Numerous different terms were used to signify EBIs by the FOAs, with the greatest variation occurring among the block grants. Lack of adequate elaboration or definition of alternative EBI terms prominently characterized FOAs issued by the Department of Health and Human Services, although less so for those issued by the Departments of Justice and Education. Overall, 43% of FOAs referenced evidence-based program registers on the web, which are scientifically credible sources of EBIs. Otherwise, most of the remaining elaborations of EBI terms in these FOAs were quite brief, often idiosyncratic, and not scientifically vetted. The FOAs generally adhered to federal policy requiring or encouraging the use of EBIs for funding requests. However, an overall pattern showing lack or inadequate elaboration of terms signifying EBIs makes it difficult for applicants to comply with federal policies regarding use of EBIs for behavioral healthcare.

Clearinghouse Standards of Evidence on the Transparency, Openness, and Reproducibility of Intervention Evaluations

Clearinghouses are influential repositories of information on the effectiveness of social interventions. To identify which interventions are “evidence-based,” clearinghouses review intervention evaluations using published standards of evidence that focus primarily on internal validity and causal inferences. Open science practices can improve trust in evidence from evaluations on the effectiveness of social interventions. Including open science practices in clearinghouse standards of evidence is one of many efforts that could increase confidence in designations of interventions as “evidence-based.” In this study, we examined the policies, procedures, and practices of 10 federal evidence clearinghouses that review preventive interventions—an important and influential subset of all evidence clearinghouses. We found that seven consider at least one open science practice when evaluating interventions: replication (6 of 10 clearinghouses), public availability of results (6), investigator conflicts of interest (3), design and analysis transparency (3), study registration (2), and protocol sharing (1). We did not identify any policies, procedures, or practices related to analysis plan registration, data sharing, code sharing, material sharing, and citation standards. We provide a framework with specific recommendations to help federal and other evidence clearinghouses implement the Transparency and Openness Promotion (TOP) Guidelines. Our proposed “TOP Guidelines for Clearinghouses” includes reporting whether evaluations used open science practices, incorporating open science practices in their standards for receiving “evidence-based” designations, and verifying that evaluations used open science practices. Doing so could increase the trustworthiness of evidence used for policy making and support improvements throughout the evidence ecosystem.

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions

Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention "works." This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. We conducted a descriptive analysis of 851 RCTs published from 2010 to 2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically proven and scalable interventions. We used Blueprints' evaluation criteria that correspond to a subset of SPR's standards of evidence. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

The Role of Clearinghouses in Promoting Transparent Research: A Methodological Study of Transparency Practices for Preventive Interventions

Transparency of research methods is vital to science, though incentives are variable, with only some journals and funders adopting transparency policies. Clearinghouses are also important stakeholders; however, to date none have implemented formal procedures that facilitate transparent research. Using data from the longest standing clearinghouse, we examine transparency practices for preventive interventions to explore the role of online clearinghouses in incentivizing researchers to make their research more transparent. We conducted a descriptive analysis of 88 evaluation reports reviewed in 2018-2019 by Blueprints for Healthy Youth Development, when the clearinghouse began checking for trial registrations, and expanded on these efforts by applying broader transparency standards to interventions eligible for an endorsement on the Blueprints website during the study period. Reports were recent, with 84% published between 2010 and 2019. We found that few reports had data, code, or research materials that were publicly available. Meanwhile, 40% had protocols that were registered, but only 8% were registered prospectively, while one-quarter were registered before conducting analyses. About one-third included details in a registered protocol describing the treatment contrast and planned inclusions, and less than 5% had a registered statistical analysis plan (e.g., planned analytical methods, pre-specified covariates). Confirmatory research was distinguished from exploratory work in roughly 40% of reports. Reports published more recently (after 2015) had higher rates of transparency. Preventive intervention research needs to be more transparent. Since clearinghouses rely on robust findings to make well-informed decisions and researchers are incentivized to meet clearinghouse standards, clearinghouses should consider policies that encourage transparency to improve the credibility of evidence-based interventions.

State behavioral health agency website references to evidence-based program registers

PURPOSE

Evidence-based program registers (EBPRs) are important tools for facilitating the use of evidence-based practices or programs (EBPs) by state statutory agencies responsible for behavioral healthcare, broadly defined as substance misuse, mental health, HIV/AIDS prevention, child welfare, and offender rehabilitation. There are currently no data on the purposes for which such state agencies reference EBPRs on their official websites.

METHOD

A webscraping method was used to identify and classify relevant "hits", defined as a state behavioral health webpage with single or multiple references to a study EBPR. A total of 778 hits (unique combinations of webpage and register) were coded. Up to three codes were applied to each hit for the "reasons for the EBPR reference" (EBPR use) dimension, one code was applied to each hit for the "purpose of the EBPR reference" and "intended audience of the webpage containing the hit" dimensions, and up to two codes were applied to each hit for the "funding mentions" dimension.

RESULTS

Three EBPRs out of 28 accounted for 73.6% of the hits. The most frequent reason for referencing EBPRs were as a resource for selecting EBPs or validating existing programs and practices. The references tended to appear in reports from the state, in training materials, or guidelines. The references tended to address broad groups of behavioral healthcare professionals. EBPRs were frequently referenced in the context of federal block grants or other federal funding.

CONCLUSIONS

Increasing state agencies' awareness and use of the entire range of existing EBPRs may improve implementation of EBPs nationally.

Making Evidence-Based Interventions Relevant for Users: A Comparison of Requirements for Dissemination Readiness Across Program Registries

This study compares prevention program registries in current use on their level of support for users seeking to implement evidence-based programs. Despite the importance of registries as intermediaries between researchers and the public, and although previous studies have examined how registries define their standards for methodological soundness and evidence of efficacy, little research has focused on the degree to which registries consider programs' dissemination readiness. The result is that registry users are uncertain whether listed programs and their necessary support materials are even available for implementation. This study evaluates 11 publicly and privately funded prevention registries that review the evidence base of programs seeking to improve child health and prosocial outcomes on the degree to which they use dissemination readiness as an evidentiary criterion for rating programs, and the extent and type of information they provide about dissemination readiness to support real-world implementation. The results show wide variability, with few having standards about dissemination readiness or making evidence-based information about interventions easily accessible to users. Findings indicate the need for registries to (1) do more to assess dissemination readiness before including programs on their website and (2) offer more complete information on dissemination readiness and implementation support to users.

Scaling up Evidence-Based Interventions in US Public Systems to Prevent Behavioral Health Problems: Challenges and Opportunities

A number of programs, policies, and practices have been tested using rigorous scientific methods and shown to prevent behavioral health problems (Catalano et al., Lancet 379:1653-1664, 2012; National Research Council and Institute of Medicine, 2009). Yet these evidence-based interventions (EBIs) are not widely used in public systems, and they have limited reach (Glasgow et al., American Journal of Public Health 102:1274-1281, 2012; National Research Council and Institute of Medicine 2009; Prinz and Sanders, Clinical Psychology Review 27:739-749, 2007). To address this challenge and improve public health and well-being at a population level, the Society for Prevention Research (SPR) formed the Mapping Advances in Prevention Science (MAPS) IV Translation Research Task Force, which considered ways to scale up EBIs in five public systems: behavioral health, child welfare, education, juvenile justice, and public health. After reviewing other efforts to scale up EBIs in public systems, a common set of factors were identified as affecting scale-up in all five systems. The most important factor was the degree to which these systems enacted public policies (i.e., statutes, regulations, and guidance) requiring or recommending EBIs and provided public funds for EBIs. Across systems, other facilitators of scale-up were creating EBIs that are ready for scale-up, public awareness of and support for EBIs, community engagement and capacity to implement EBIs, leadership support for EBIs, a skilled workforce capable of delivering EBIs, and data monitoring and evaluation capacity. It was concluded that the following actions are needed to significantly increase EBI scale-up in public systems: (1) provide more public policies and funding to support the creation, testing, and scaling up of EBIs; (2) develop and evaluate specific frameworks that address systems level barriers impeding EBI scale-up; and (3) promote public support for EBIs, community capacity to implement EBIs at scale, and partnerships between community stakeholders, policy makers, practitioners, and scientists within and across systems.

Similarities and differences in program registers: A case study

Researchers, clinicians, and other professionals are increasingly in need of cost-effective, evidence-based programs and practices. However, these individuals may lack the time and, for some, the required expertise to search for and identify such interventions. To address this concern, several online registers that list or categorize programs according to their empirical evidence of effectiveness have been established. Although these registers are designed to simplify the task of selecting evidence-based interventions, the use of distinct review processes and standards by each register creates discrepancies in final program classifications, which can pose a challenge for users. The present case study highlights three programs that have been evaluated by more than one register and have received similar or different classifications. Reasons for inconsistencies are discussed, and several recommendations for evaluating organizations and register users are provided to enhance the functionality and ease of use of online program registers.

Innovation and Integrity: Desiderata and Future Directions for Prevention and Intervention Science

This article summarizes essential implications of the papers within this special issue and discusses directions for future prevention and intervention research on conceptual issues, methodological and transfer-related challenges and opportunities. We identify a need to move from programs to principles in intervention research and encourage the implementation of research on potential mechanisms underlying intervention effectiveness. In addition, current methodological issues in intervention research are highlighted, including advancements in methodology and statistical procedures, extended outcome assessments, replication studies, and a thorough examination of potential biases. We further discuss transfer-related issues, for example the need for more research on the flexibility and adaptability of programs and intervention approaches as well as more general problems in knowledge translation reasoning the need for enhanced communication between practitioners, policy makers, and researchers. Finally, we briefly touch on the need to discuss the relation between single intervention programs, the mental health system, and changes of contextual conditions at the macro level.

Reviewing the Reviews: Examining Similarities and Differences Between Federally Funded Evidence Reviews

BACKGROUND

The federal government's emphasis on supporting the implementation of evidence-based programs has fueled a need to conduct and assess rigorous evaluations of programs. Through partnerships with researchers, policy makers, and practitioners, evidence reviews-projects that identify, assess, and summarize existing research in a given area-play an important role in supporting the quality of these evaluations and how the findings are used. These reviews encourage the use of sound scientific principles to identify, select, and implement evidence-based programs. The goals and standards of each review determine its conclusions about whether a given evaluation is of high quality or a program is effective. It can be difficult for decision makers to synthesize the body of evidence when faced with results from multiple program evaluations.

SAMPLE

This study examined 14 federally funded evidence reviews to identify commonalities and differences in their assessments of evidence of effectiveness.

FINDINGS

There were both similarities and significant differences across the reviews. In general, the evidence reviews agreed on the broad critical elements to consider when assessing evaluation quality, such as research design, low attrition, and baseline equivalence. The similarities suggest that, despite differences in topic and the availability of existing research, reviews typically favor evaluations that limit potential bias in their estimates of program effects. However, the way in which some of the elements were assessed, such as what constituted acceptable amounts of attrition, differed. Further, and more substantially, the reviews showed greater variation in how they conceptualized "effectiveness."

Evidence and Evidence-Based Practices: Are We There Yet?

The purpose of this article is to move the field of intellectual and closely related developmental disabilities (IDD) towards a better understanding of evidence and evidence-based practices. To that end, we discuss (a) different perspectives on and levels of evidence, (b) commonly used evidence-gathering strategies, (c) standards to evaluate evidence, (d) the distinction between internal and external validity, and (e) guidelines for establishing evidence-based practices. We also describe how the conceptualization and use of evidence and evidence-based practices are changing to accommodate recent trends in the field.

Evaluation in the field of social services for minors: Measuring the efficacy of interventions in the Italian service for health protection and promotion

This article presents the availment of a new Methodology for the efficacy evaluation of interventions in the field of social science: the Method of Computerized Textual Data Analysis (M.A.D.I.T.). In the beginning, we present some elements of the international and Italian legislation referred to the efficacy evaluation and about the child protection. Subsequently this work describes the process of efficacy evaluation of an intervention of minor protection delivered by a public Italian Service, the Minor and Family Service, MiFa. The MADIT Methodology is applied to the efficacy evaluation and it is interested in discursive repertoires, defined as "a linguistically intended mode of construction of finite reality". The aim of the research is to show, through the description of every step of the implementation of the Methodology based on text analysis, how is possible to notice if there are progress in the direction of the objective of intervention of child protection. The results describes how from a starting situation of "first appearance of psychiatric career" referred to the minor, the work of the psychologist of the Service MiFa has enabled to produce a shifting in the direction of objective of the intervention, that was "developing the competence of the minor to identify objectives". Through this work, we show how a rigorous methodology for assessing effectiveness may contribute to improve the quality of service of Minor Protection and may also be suitable for new fields of social science.

Implementing Effective Substance Abuse Treatments in General Medical Settings: Mapping the Research Terrain

The National Institute on Alcohol Abuse and Alcoholism (NIAAA), National Institute on Drug Abuse (NIDA), and Veterans Health Administration (VHA) share an interest in promoting high quality, rigorous health services research to improve the availability and utilization of evidence-based treatment for substance use disorders (SUD). Recent and continuing changes in the healthcare policy and funding environments prioritize the integration of evidence-based substance abuse treatments into primary care and general medical settings. This area is a prime candidate for implementation research. Recent and ongoing implementation projects funded by these agencies are reviewed. Research in five areas is highlighted: screening and brief intervention for risky drinking; screening and brief intervention for tobacco use; uptake of FDA-approved addiction pharmacotherapies; safe opioid prescribing; and disease management. Gaps in the portfolios, and priorities for future research, are described.

An overview of evidence-based program registers (EBPRs) for behavioral health

Evaluations of behavioral health interventions have identified many that are potentially effective. However, clinicians and other decision makers typically lack the time and ability to effectively search and synthesize the relevant research literature. In response to this opportunity, and to increasing policy and funding pressures for the use of evidence-based practices, a number of “what works” websites have emerged to assist decision makers in selecting interventions with the highest probability of benefit. However, these registers as a whole are not well understood. This article, which represents phase one of a concurrent mixed methods study, presents a review of the scopes, structures, dissemination strategies, uses, and challenges faced by evidence-based registers in the behavioral health disciplines. The major findings of this study show that in general, registers of evidence-based practices are able, to a degree, to identify the most effective practices meet this need to a degree. However, much needs to be done to improve the ability of the registers to fully realize their purpose.