Fidelity of supported employment programs and employment outcomes

Assessing the implementation fidelity, feasibility, and sustainability of community-based house improvement for malaria control in southern Malawi: a mixed-methods study

BACKGROUND

Despite significant success in the fight against malaria over the past two decades, malaria control programmes rely on only two insecticidal methods: indoor residual spraying and insecticidal-treated nets. House improvement (HI) can complement these interventions by reducing human-mosquito contact, thereby reinforcing the gains in disease reduction. This study assessed the implementation fidelity, which is the assessment of how closely an intervention aligns with its intended design, feasibility, and sustainability of community-led HI in southern Malawi.

METHODS

The study, conducted in 22 villages (2730 households), employed a mixed-methods approach. Implementation fidelity was assessed using a modified framework, with longitudinal surveys collecting data on HI coverage indicators. Quantitative analysis, employing descriptive statistics, evaluated the adherence to HI implementation. Qualitative data came from in-depth interviews, key informant interviews, and focus groups involving project beneficiaries and implementers. Qualitative data were analysed using content analysis guided by the implementation fidelity model to explore facilitators, challenges, and factors affecting intervention feasibility.

RESULTS

The results show that HI was implemented as planned. There was good adherence to the intended community-led HI design; however, the adherence could have been higher but gradually declined over time. In terms of intervention implementation, 74% of houses had attempted to have eaves closed in 2016-17 and 2017-18, compared to 70% in 2018-19. In 2016-17, 42% of houses had all four sides of the eaves closed, compared to 33% in 2018-19. Approximately 72% of houses were screened with gauze wire in 2016-17, compared to 57% in 2018-19. High costs, supply shortages, labour demands, volunteers' poor living conditions and adverse weather were reported to hinder the ideal HI implementation. Overall, the community described community-led HI as feasible and could be sustained by addressing these socioeconomic and contextual challenges.

CONCLUSION

Our study found that although HI was initially implemented as planned, its fidelity declined over time. Using trained volunteers facilitated the fidelity and feasibility of implementing the intervention. A combination of rigorous community education, consistent training, information, education and communication, and intervention modifications may be necessary to address the challenges and enhance the intervention's fidelity, feasibility, and sustainability.

Perspectives on implementing Individual Placement and Support (IPS) within primary health care settings for adults living in British Columbia, Canada

BACKGROUND

Individual placement and support (IPS) is an evidence-based practice (EBP) designed to help people with severe mental illness re-enter the labour market. Implementing an IPS program within a new context (e.g., primary health care setting) to support populations that are complex and multi-barriered presents a set of unique challenges and considerations. This paper provides community-based perspectives that identify implementation strengths and challenges and highlights potential strategies aimed at addressing emergent barriers.

METHODS

A case study was conducted across three community health centres in British Columbia (BC), Canada, where a novel IPS program was embedded within primary care services. Data collection consisted of open-ended surveys and focus groups with service providers directly involved in program implementation and their associated clinical and managerial support teams (n = 15). Using the updated Consolidated Framework for Implementation Research (CFIR) as a guide, we performed deductive thematic analysis to identify key areas impacting IPS implementation.

RESULTS

Integration with existing health care systems and primary health care teams and support from leadership across all levels were identified as both key facilitators and barriers to implementation. Facilitators and barriers were identified across all domains, with those within innovation and process most easily addressed. Four cross-cutting themes emerged for promoting more integrated and sustainable program implementation: investing in pre-implementation activities, supporting a dynamic and flexible program, building from community experiences, and developing a system for shared knowledge.

CONCLUSIONS

Implementing an IPS program embedded within primary health care settings is complex and requires extensive planning and consultation with community-based service providers and decision-makers to achieve full integration. Future practice and policy decisions aimed at supporting employment and well-being should be made in collaboration with communities.

Effort-based decision-making as a determinant of supported employment outcomes in psychotic disorders

Schizophrenia is associated with a heavy economic burden in the United States that is partly due to the high rates of chronic unemployment. Individual Placement and Support (IPS) is an evidenced-based type of supported employment that can improve job obtainment and work outcomes in psychotic disorders. Outcomes vary widely and a persistent challenge for IPS is low levels of engagement in the initial job search phase. Past studies have focused on interview-based motivation deficits as a key determinant of poor treatment engagement and work outcomes in schizophrenia. New validated performance-based measures of motivation, including effort-based decision-making (EBDM) tasks, may explain supported employment outcomes and provide insights into individual differences in IPS outcomes. This study investigated the degree to which IPS engagement (i.e., number of sessions attended during the first four months of service delivery) was related to baseline interview-based motivation deficits and performance on three EBDM tasks - two tasks of physical effort and one of cognitive effort (i.e., Balloon Task, Effort Expenditure for Rewards Task, Deck Choice Effort Task) - in a sample (N = 47) of people with a psychotic disorder. Results indicated that the level of EBDM performance, specifically on the Balloon Task, predicted IPS engagement, accounting for an additional 17 % of the variance above and beyond interview-based motivation deficits (total R2 = 24 %). Overall, these findings suggest that addressing motivational deficits in effort-based decision-making may be beneficial to IPS engagement, which in turn may improve the trajectory of work outcomes.

The effects of supported employment interventions in populations of people with conditions other than severe mental health: a systematic review

AIM

To assess the effectiveness of supported employment interventions for improving competitive employment in populations of people with conditions other than only severe mental illness.

BACKGROUND

Supported employment interventions have been extensively tested in severe mental illness populations. These approaches may be beneficial outside of these populations.

METHODS

We searched PubMed, Embase, CINAHL, PsycInfo, Web of Science, Scopus, JSTOR, PEDro, OTSeeker, and NIOSHTIC for trials including unemployed people with any condition and including severe mental illness if combined with other co-morbidities or other specific circumstances (e.g., homelessness). We excluded trials where inclusion was based on severe mental illness alone. Two reviewers independently assessed risk of bias (RoB v2.0) and four reviewers extracted data. We assessed rates of competitive employment as compared to traditional vocational rehabilitation or waiting list/services as usual.

FINDINGS

Ten randomised controlled trials (913 participants) were included. Supported employment was more effective than control interventions for improving competitive employment in seven trials: in people with affective disorders [risk ratio (RR) 10.61 (1.49, 75.38)]; mental disorders and justice involvement [RR 4.44 (1.36,14.46)]; veterans with posttraumatic stress disorder (PTSD) [RR 2.73 (1.64, 4.54)]; formerly incarcerated veterans [RR 2.17 (1.09, 4.33)]; people receiving methadone treatment [RR 11.5 (1.62, 81.8)]; veterans with spinal cord injury at 12 months [RR 2.46 (1.16, 5.22)] and at 24 months [RR 2.81 (1.98, 7.37)]; and young people not in employment, education, or training [RR 5.90 (1.91-18.19)]. Three trials did not show significant benefits from supported employment: populations of workers with musculoskeletal injuries [RR 1.38 (1.00, 1.89)]; substance abuse [RR 1.85 (0.65, 5.41)]; and formerly homeless people with mental illness [RR 1.55 (0.76, 3.15)]. Supported employment interventions may be beneficial to people from more diverse populations than those with severe mental illness alone. Defining competitive employment and increasing (and standardising) measurement of non-vocational outcomes may help to improve research in this area.

Adherence of HIV clinics to guidelines for the delivery of TB screening among people living with HIV/AIDS in Ghana

Background

Tuberculosis screening of people living with HIV (PLHIV) – an intervention to reduce the burden of TB among PLHIV – is being implemented at HIV clinics in Ghana since 2007, but TB screening coverage remains low. Facility adherence to intervention guidelines may be a factor but is missing in implementation science literature. This study assesses the level of HIV clinic adherence to the guidelines and related facility characteristics in selected district hospitals in Ghana.

Methods

This cross-sectional study was conducted in all 27 district hospitals with HIV clinics, X-ray and geneXpert machines in Ghana. These hospitals are in 27 districts representing about 27% of the 100 district hospitals with HIV clinics in Ghana. A data collection tool with 18-items (maximum score of 29) was developed from the TB/HIV collaborative guidelines to assess facility adherence to four interrelated components of the TB screening programme as stated in the guidelines: intensive TB case-finding among PLHIV (ITCF), Isoniazid preventive therapy initiation (IPT), TB infection control (TIC), and programme review meetings (PRM). Data were collected through record review and interviews with 27 key informants from each hospital. Adherence scores per component were summed to determine an overall adherence score per facility and summarized using medians and converted to proportions. Facility characteristics were assessed and compared across facilities with high (above median) versus low (below median) overall adherence scores, using nonparametric test statistics.

Results

From the 27 key interviews and facility records reviewed, the median adherence scores for ITCF, IPT, TIC, and PRM components were 85.7% (IQR: 85.5–100.0), 0% (IQR: 0–66.7), 33.3% (IQR: 33.3–50.0), and 90.0% (IQR: 70.0–90.0), respectively. The overall median adherence score was 62.1% (IQR: 58.6–65.1), and 17 clinics (63%) with overall adherence score above the median were categorized as high adherence. Compared to low adherence facilities, high adherence facilities had statistically significant lower PLHIV clinic attendees per month (256 (IQR: 60–904) vs. 900 (IQR: 609–2622); p = 0.042), and lower HIV provider workloads (28.6 (IQR: 8.6–113) vs. 90 (IQR: 66.7–263.5); p = 0.046), and most had screening guidelines (76%, p < 0.01) and questionnaire (80%, p < 0.01) available on-site.

Conclusion

PRM had highest score while the IPT component had the lowest score. Almost a third of the facilities implemented the TB screening programme activities with a high level of adherence to the guidelines. We suggest to ensure adherence to all four components, reducing staff workloads and making TB screening questionnaires and guidelines available on-site would increase facility adherence to the intervention and ultimately achieve intervention targets.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-021-07121-9.

Strategies to enhance routine physical activity in care home residents: the REACH research programme including a cluster feasibility RCT

Background

Care home residents are mainly inactive, leading to increased dependency and low mood. Although exercise classes may increase activity, a more sustainable model is to engage staff and residents in increasing routine activity.

Objectives

The objectives were to develop and preliminarily test strategies to enhance the routine physical activity of care home residents to improve their physical, psychological and social well-being through five overlapping workstreams.

Design

This trial had a mixed-methods research design to develop and test the feasibility of undertaking an evaluative study consisting of gaining an understanding of the opportunities for and barriers to enhancing physical activity in care homes (workstream 1); testing physical activity assessment instruments (workstream 2); developing an intervention through a process of intervention mapping (workstream 3); refining the provisional intervention in the care home setting and clarifying outcome measurement (workstream 4); and undertaking a cluster randomised feasibility trial of the intervention [introduced via three facilitated workshops at baseline (with physiotherapist input), 2 weeks (with artist input) and 2 months], with embedded process and health economic evaluations (workstream 5).

Setting

The trial was set in 12 residential care homes differing in size, location, ownership and provision in Yorkshire, UK.

Participants

The participants were elderly residents, carers, managers and staff of care homes.

Intervention

The intervention was MoveMore, designed for the whole home, to encourage and support the movement of residents in their daily routines.

Main outcome measures

The main outcome measures related to the feasibility and acceptability of implementing a full-scale trial in terms of recruitment and retention of care homes and residents, intervention delivery, completion and reporting of baseline data and outcomes (including hours of accelerometer wear, hours of sedentary behaviour and hours and type of physical activity), and safety and cost data (workstream 5).

Results

Workstream 1 – through a detailed understanding of life in a care home, a needs assessment was produced, and barriers to and facilitators of activity were identified. Key factors included ethos of care; organisation, management and delivery of care; use of space; and the residents’ daily routines. Workstream 2 – 22 (73.3%) out of 30 residents who wore a hip accelerometer had valid data (≥ 8 hours on ≥ 4 days of the week). Workstream 3 – practical mechanisms for increasing physical activity were developed, informed by an advisory group of stakeholders and outputs from workstreams 1 and 2, framed by the process of intervention mapping. Workstream 4 – action groups were convened in four care homes to refine the intervention, leading to further development of implementation strategies. The intervention, MoveMore, is a whole-home intervention involving engagement with a stakeholder group to implement a cyclical process of change to encourage and support the movement of residents in their daily routines. Workstream 5 – 12 care homes and 153 residents were recruited to the cluster randomised feasibility trial. Recruitment in the care homes varied (40–89%). Five care homes were randomised to the intervention and seven were randomised to usual care. Predetermined progression criteria were recruitment of care homes and residents (green); intervention delivery (amber); and data collection and follow-up – 52% of residents provided usable accelerometer data at 9 months (red), > 75% of residents had reported outcomes at 9 months (green, but self-reported resident outcomes were red), 26% loss of residents to follow-up at 9 months [just missing green criterion (no greater than 25%)] and safety concerns (green).

Limitations

Observations of residents’ movements were not conducted in private spaces. Working with care home residents to identify appropriate outcome measures was challenging. Take-up of the intervention was suboptimal in some sites. It was not possible to make a reliably informed decision on the most appropriate physical activity end point(s) for future use in a definitive trial.

Conclusions

A whole-home intervention was developed that was owned and delivered by staff and was informed by residents and staff. The feasibility of conducting a cluster randomised controlled trial was successfully tested: the target numbers of care homes and residents were recruited, demonstrating that it is possible to recruit care home residents to a cluster randomised trial, although this process was time-consuming and resource heavy. A large data set was collected, which provided a comprehensive picture of the environment, residents and staff in care homes. Extensive quantitative and qualitative work comprehensively explored a neglected area of health and social care research. Completion of ethnographic work in a range of settings enabled the production of an in-depth picture of life in care homes that will be helpful for other researchers considering organisational change in this setting.

Future work

The content and delivery of the intervention requires optimisation and the outcome measurement requires further refinement prior to undertaking a full trial evaluation. Consideration could be given to a recommended, simplified, core outcome set, which would facilitate data collection in this population.

Trial registration

Current Controlled Trials ISRCTN16076575.

Funding

This project was funded by the National Institute for Health Research (NIHR) Programme Grant for Applied Research programme and will be published in full in Programme Grant for Applied Research; Vol. 9, No. 9. See the NIHR Journals Library website for further project information.

Teaching emotional and social competencies: Efficacy of a work readiness program designed for vulnerable youth

BACKGROUND: As emotional and social competency training proliferates within a work readiness context, concerns remain regarding their efficacy. Data on these programs tends to be scarce and outcome objectives are often poorly defined. OBJECTIVE: Authors developed and tested a work readiness emotional and social competency program specifically designed for at-risk young adults, tailored with best practices in mind. METHOD: 84 clients of a community organization that provides employment support to young adults with disabilities (48 men and 36 women) with a mean age of 28.17 years (SD = 11.64) completed measures of emotional intelligence and alexithymia on either side of the 4-week intervention. RESULTS: Men’s interpersonal scores and women’s adaptability scores showed significant improvement across the intervention. In addition, women’s scores in both identifying and describing feelings improved significantly, as did men’s scores in describing feelings. CONCLUSIONS: Within the context of work readiness, participants in an intervention to improve emotional and social competencies can see key improvements to competencies linked to occupational attainment.

A structured mixed method process evaluation of a randomized controlled trial of Individual Placement and Support (IPS)

Background

Individual Placement and Support (IPS) is an evidence-based work rehabilitation program helping people with moderate to severe mental illness to obtain ordinary employment. Although IPS has proven superior to other work rehabilitation programs, in many studies, the majority of the participants remain unemployed. Structured process evaluations of IPS that use mixed methods are scarce, although they could identify implementation aspects that may enhance its effect. The aim of the current study is to assess reach, fidelity, and identify barriers and facilitators to implement IPS.

Methods

The process evaluation was conducted alongside a randomized controlled trial including six IPS centers, comparing IPS with treatment as usual in a population of patients in treatment for moderate to severe mental illness. Mixed methods were used in the process evaluation, including focus group interviews with service providers, individual interviews and survey data from participants, and fidelity reviews using the validated IPS Fidelity Scale.

Results

The intervention reached the intended target group. All centers reached fair to good fidelity according to the IPS Fidelity Scale within the project period (range 97–109, SD 8.1) (see Table 5). Certain fidelity items indicated implementation issues related to employer contact, community-based services, and integration with health services. Survey data showed that less than half of the participants regarded their illness as a barrier for participating in IPS and that freedom of disclosure was important. Participant interviews gave further insight into the role of the IPS specialist, emphasizing their availability and consistent job focus.

Conclusions

Indications of implementation challenges across centers during the first year suggest special attention should be given to these aspects in an early phase to ensure higher fidelity from the start and thus enhance the effectiveness of IPS. The IPS specialist played an important role for participants and was described as positive, pushing in a positive way, and encouraging. More knowledge on the characteristics of successful IPS specialists could further enhance the effectiveness of the intervention.

Trial registration

The study was registered on clinicaltrials.gov prior to the inclusion period (reg.no: NCT01964092, registered 17/07/2013).

Supplementary information

Supplementary information accompanies this paper at 10.1186/s43058-020-00083-9.

30 Years After the Americans with Disabilities Act: Perspectives on Employment for Persons with Spinal Cord Injury

The Americans with Disabilities Act, passed in 1990, represented landmark legislation and led to significant improvements in accessibility, such as prohibiting discrimination based on disability in public life, including employment. Now 30 years later, however, employment rates for persons with disabilities, including spinal cord injury, remain low. This article discusses why employment is so important for persons with spinal cord injury and challenges that remain. Presented are previously unpublished employment data from a nationally representative US sample. Finally, the state of the art of vocational rehabilitation, including models proven to facilitate this critical rehabilitation outcome, is discussed.

The methods and baseline characteristics of a VA randomized controlled study evaluating supported employment provided in primary care patient aligned care teams

BACKGROUND

This article describes the design and baseline sample of a single-site trial comparing Individual Placement and Support (IPS) supported employment delivered within a Veterans Health Administration (VHA) primary care Patient Aligned Care Team (PACT) to treatment-as-usual vocational rehabilitation (TAU-VR) that includes transitional work.

METHODS

Unemployed U.S. military veterans receiving care in a VHA PACT who were seeking competitive work, otherwise eligible for vocational rehabilitation, and diagnosed with a mental health condition other than a psychotic or bipolar I disorder were prospectively randomized to receive either IPS or TAU-VR. Employment outcomes and measures of quality of life, self-esteem, and community reintegration are being collected for 12 months.

RESULTS

The participant sample (n = 119) is comprised of 17.6% female, 73.1% African-Americans, and 1.7% Hispanic. Average age is 38.2 (SD ± 8.41) years; 80.7% served in the military since 2001; 78% are receiving or applying for U.S. Department of Veterans Affairs (VA) service-connected disability; 26.9% have not held a competitive job in the past 3 years; and the average length of pre-randomization unemployment is 1.4 (SD ± 2.3) years.

CONCLUSIONS

Unique design features include evaluating the efficacy of evidenced-based IPS within the primary care setting, having broad diagnostic eligibility, and defining the primary outcome criterion as "steady employment", i.e. holding a competitive job for ≥26 weeks of the 12-month follow-up period. The findings illustrate the characteristics of a primary care veteran sample in need of employment services.

TRIAL REGISTRATION

www.clinicaltrials.gov Identifier: NCT02400736.

The Value Equation: Three complementary propositions for reconciling fidelity and adaptation in evidence-based practice implementation

BACKGROUND

There has long been debate about the balance between fidelity to evidence-based interventions (EBIs) and the need for adaptation for specific contexts or particular patients. The debate is relevant to virtually all clinical areas. This paper synthesises arguments from both fidelity and adaptation perspectives to provide a comprehensive understanding of the challenges involved, and proposes a theoretical and practical approach for how fidelity and adaptation can optimally be managed.

DISCUSSION

There are convincing arguments in support of both fidelity and adaptations, representing the perspectives of intervention developers and internal validity on the one hand and users and external validity on the other. Instead of characterizing fidelity and adaptation as mutually exclusive, we propose that they may better be conceptualized as complimentary, representing two synergistic perspectives that can increase the relevance of research, and provide a practical way to approach the goal of optimizing patient outcomes. The theoretical approach proposed, the "Value Equation," provides a method for reconciling the fidelity and adaptation debate by putting it in relation to the value (V) that is produced. The equation involves three terms: intervention (IN), context (C), and implementation strategies (IS). Fidelity and adaptation determine how these terms are balanced and, in turn, the end product - the value it produces for patients, providers, organizations, and systems. The Value Equation summarizes three central propositions: 1) The end product of implementation efforts should emphasize overall value rather than only the intervention effects, 2) implementation strategies can be construed as a method to create fit between EBIs and context, and 3) transparency is vital; not only for the intervention but for all of the four terms of the equation. There are merits to arguments for both fidelity and adaptation. We propose a theoretical approach, a Value Equation, to reconciling the fidelity and adaptation debate. Although there are complexities in the equation and the propositions, we suggest that the Value Equation be used in developing and testing hypotheses that can help implementation science move toward a more granular understanding of the roles of fidelity and adaptation in the implementation process, and ultimately sustainability of practices that provide value to stakeholders.

Crisis resolution teams for people experiencing mental health crises: the CORE mixed-methods research programme including two RCTs

Background

Crisis resolution teams (CRTs) seek to avert hospital admissions by providing intensive home treatment for people experiencing a mental health crisis. The CRT model has not been highly specified. CRT care is often experienced as ending abruptly and relapse rates following CRT discharge are high.

Aims

The aims of CORE (Crisis resolution team Optimisation and RElapse prevention) workstream 1 were to specify a model of best practice for CRTs, develop a measure to assess adherence to this model and evaluate service improvement resources to help CRTs implement the model with high fidelity. The aim of CORE workstream 2 was to evaluate a peer-provided self-management programme aimed at reducing relapse following CRT support.

Methods

Workstream 1 was based on a systematic review, national CRT manager survey and stakeholder qualitative interviews to develop a CRT fidelity scale through a concept mapping process with stakeholders (n = 68). This was piloted in CRTs nationwide (n = 75). A CRT service improvement programme (SIP) was then developed and evaluated in a cluster randomised trial: 15 CRTs received the SIP over 1 year; 10 teams acted as controls. The primary outcome was service user satisfaction. Secondary outcomes included CRT model fidelity, catchment area inpatient admission rates and staff well-being. Workstream 2 was a peer-provided self-management programme that was developed through an iterative process of systematic literature reviewing, stakeholder consultation and preliminary testing. This intervention was evaluated in a randomised controlled trial: 221 participants recruited from CRTs received the intervention and 220 did not. The primary outcome was re-admission to acute care at 1 year of follow-up. Secondary outcomes included time to re-admission and number of days in acute care over 1 year of follow-up and symptoms and personal recovery measured at 4 and 18 months’ follow-up.

Results

Workstream 1 – a 39-item CRT fidelity scale demonstrated acceptability, face validity and promising inter-rater reliability. CRT implementation in England was highly variable. The SIP trial did not produce a positive result for patient satisfaction [median Client Satisfaction Questionnaire score of 28 in both groups at follow-up; coefficient 0.97, 95% confidence interval (CI) –1.02 to 2.97]. The programme achieved modest increases in model fidelity. Intervention teams achieved lower inpatient admission rates and less inpatient bed use. Qualitative evaluation suggested that the programme was generally well received. Workstream 2 – the trial yielded a statistically significant result for the primary outcome, in which rates of re-admission to acute care over 1 year of follow-up were lower in the intervention group than in the control group (odds ratio 0.66, 95% CI 0.43 to 0.99; p = 0.044). Time to re-admission was lower and satisfaction with care was greater in the intervention group at 4 months’ follow-up. There were no other significant differences between groups in the secondary outcomes.

Limitations

Limitations in workstream 1 included uncertainty regarding the representativeness of the sample for the primary outcome and lack of blinding for assessment. In workstream 2, the limitations included the complexity of the intervention, preventing clarity about which were effective elements.

Conclusions

The CRT SIP did not achieve all its aims but showed potential promise as a means to increase CRT model fidelity and reduce inpatient service use. The peer-provided self-management intervention is an effective means to reduce relapse rates for people leaving CRT care.

Study registration

The randomised controlled trials were registered as Current Controlled Trials ISRCTN47185233 and ISRCTN01027104. The systematic reviews were registered as PROSPERO CRD42013006415 and CRD42017043048.

Funding

The National Institute for Health Research Programme Grants for Applied Research programme.

Employment Interventions in Health Settings: A Systematic Review and Synthesis

PURPOSE

Employment is a key social determinant of health. People who are unemployed typically have worse health than those employed. Illness and disability can result in unemployment and be a barrier to regaining employment. We combined a systematic review and knowledge synthesis to identify both studies of employment interventions in health care settings and common characteristics of successful interventions.

METHODS

We searched the peer-reviewed literature (1995-2017), and titles and abstracts were screened for inclusion and exclusion criteria by 2 independent reviewers. We extracted data on the study setting, participants, intervention, methods, and findings. We also conducted a narrative synthesis and iteratively developed a conceptual model to inform future primary care interventions.

RESULTS

Of 6,729 unique citations, 88 articles met our criteria. Most articles (89%) focused on people with mental illness. The majority of articles (74%) tested interventions that succeeded in helping participants gain employment. We identified 5 key features of successful interventions: (1) a multidisciplinary team that communicates regularly and collaborates, (2) a comprehensive package of services, (3) one-on-one and tailored components, (4) a holistic view of health and social needs, and (5) prospective engagement with employers.

CONCLUSIONS

Our findings can inform new interventions that focus on employment as a social determinant of health. Although hiring a dedicated employment specialist may not be feasible for most primary care organizations, pathways using existing resources with links to external agencies can be created. As precarious work becomes more common, helping patients engage in safe and productive employment could improve health, access to health care, and well-being.

Individual placement and support: from research to practice

Vocational rehabilitation for people with severe mental health problems is poorly developed in the UK. Although there is a clear evidence base indicating the effectiveness of approaches to helping people with severe mental health problems gain and retain employment there is generally a lack of awareness of this evidence. As a result there has been a lack of implementation within routine clinical practice of the most effective approaches to improving employment outcomes for such individuals.

Meeting patient needs trumps adherence. A cross-sectional study of adherence and adaptations when national guidelines are used in practice

RATIONALE, AIMS AND OBJECTIVES

In the evidence-to-practice pathway, guidelines are developed to provide a practical summary of evidence and stimulate change. However, when guidelines are used in practice, adherence to the recommendations in guidelines is limited, and adaptations are common. Thus, we need more detailed knowledge about adherence and adaptations when guidelines are used in practice to understand the end of the evidence-to-practice pathway. Subsequently, the aim is to examine adherence to and adaptations of recommendations in the Swedish National Guidelines for Methods of Preventing Disease.

MATERIAL AND METHODS

A questionnaire was sent to healthcare professionals and managers in Stockholm between January and March 2014. Adherence to the recommendations was compared between practice settings, and the frequency of different adaptations and reasons for adaptations was analysed.

RESULTS

Partial adherence to the guidelines was found. The adherence was significantly greater within primary care than at the hospitals (P < .001). Modifications formed the most common category of adaptations (55%) and included mainly prioritization of specific patient groups and increased patient customization. The most common reason for adaptations (25%) was to meet the patients' specific needs and capabilities.

CONCLUSIONS

This study provides insight into adherence and adaptation when guidelines are used in practice. Work with lifestyle habits was partially done in accordance with the guidelines. Lack of time and lack of resources were not the most common reasons for adaptations. Rather, the findings suggest that when patient needs and capabilities contrast with guideline recommendations, patient needs trump adherence to guidelines.

Significance of scientific evidence in organizing care processes

Purpose - The purpose of this paper is to analyze how staff and managers in health and social care organizations use scientific evidence when making decisions about the organization of care practices. Design/methodology/approach - Document analysis and repeated interviews (2008-2010) with staff (n=39) and managers (n=26) in health and social care organizations. The respondents were involved in a randomized controlled study about testing a continuum of care model for older people. Findings - Scientific evidence had no practical function in the social care organization, while it was a prioritized source of information in the health care organization. This meant that the decision making regarding care practices was different in these organizations. Social care tended to rely on ad hoc practice-based information and political decisions when organizing care, while health care to some extent also relied in an unreflected manner on the scientific knowledge. Originality/value - The study illustrates several difficulties that might occur when managers and staff try to consider scientific evidence when making complicated decisions about care practices.

A qualitative study of cognitive behavioural therapy in multiple sclerosis: experiences of psychotherapists

Purpose: To investigate how psychotherapists experience using individual, face-to-face cognitive behavioural therapy (CBT) aimed at alleviating depressive symptoms in persons with multiple sclerosis (MS).

Method: Semi-structured interviews with three psychotherapists were conducted after CBT with 12 participants with MS, and analysed using qualitative content analysis.

Results: Two main themes emerged: Trusting their expertise as psychotherapists whilst lacking MS-specific knowledge, and The process of exploring the participants’ readiness for CBT with modifications of content and delivery. The psychotherapists perceived it difficult to know whether a symptom was attributable to depression or to MS, and for some participants the CBT needed to be adapted to a more concrete content.

Conclusions: Psychotherapists may need more MS-specific knowledge and an insight into the individual’s functioning. The content of CBT in terms of concrete home assignments and behavioural activation needs to be individualised.

Implementing Evidence-Based Employment Services in New Zealand for Young Adults with Psychosis: Progress during the First Five Years

This practice evaluation demonstrates how occupational therapists can play a key role in providing vocational services for young people experiencing a first episode of psychosis. By implementing the evidence-based Individual Placement and Support model, occupational therapists can expand their traditional roles to provide services that integrate functional recovery with clinical recovery. This New Zealand programme achieved vocational outcomes of between 49% and 59% over a 4-year period. The lessons learned from these experiences add to the body of international knowledge on how best to implement evidence-based employment services for young people with mental health problems.

Client, Contextual and Program Elements Influencing Supported Employment: A Literature Review

Supported employment is an evidence-based practice with a well-established research base. Most studies track such outcomes as employment rates, time to employment and wages earned. Few studies address client and contextual factors that impact outcomes or consider program elements beyond those that comprise the individual placement and support model. This paper reviews existing literature to shed light on the following questions: (1) What impact do labour market trends have on the effectiveness of SE? (2) How lasting are the effects of SE and what factors influence longevity of SE effects? (3) What levels and types of employment are targeted by SE? (4) What are the characteristics of people who benefit from SE? (5) What is the role of peer support in SE? and (6) What are the barriers to effective SE implementation? Research findings are synthesized and suggestions for service enhancements are offered so that the model can continue to evolve.

Unearthing the Theoretical Underpinnings of "Green Care" in Mental Health and Substance Misuse Care: Theoretical Underpinnings and Contemporary Clinical Examples

The concept of "Green Care" can increasingly be found in mental health and addictions literature and some Psychiatric/Mental Health Nurses are practicing in facilities that base their approach on some or all of the underpinning theoretical elements of Green Care. However, Green Care is not yet widely considered to be part of mainstream psychiatry. Unearthing and articulating its theoretical underpinnings and clinical applications may further advance its legitimacy. Accordingly, this article identifies four principal theoretical elements of Green Care: Connectedness; Contact with Nature; Benefits of Exercise; and Occupation/Work as Therapeutic, each of which is explored and articulated by drawing on relevant literature.

Supported Employment for the Reintegration of Disability Pensioners with Mental Illnesses: A Randomized Controlled Trial

Work is beneficial for the recovery from mental illness. Although the approach of individual placement and support (IPS) has been shown to be effective in Europe, it has not yet been widely implemented in European health care systems. The aim of this randomized controlled trial was to assess the effectiveness of IPS for disability pensioners with mental illnesses new on disability benefits in Switzerland. In the study at hand, 250 participants were randomly assigned to either the control or the intervention group. The participants in the intervention group received job coaching according to IPS during 2 years. The control group received no structured support. Both groups were interviewed at baseline and followed up every 6 months (baseline, 6, 12, 16, 18, 24 months) for 2 years. Primary outcome was to obtain a job in the competitive employment. IPS was more effective for the reintegration into the competitive employment market for disability pensioners than the control condition. Thirty-two percent of the participants of the intervention group and 12% of the control group obtained new jobs in the competitive employment. IPS is also effective for the reintegration into competitive employment of people with mental illness receiving disability pensions.

Supporting Recovery Orientated Services for People With Severe Mental Illness

There is now wide agreement that people with severe mental illness can be adequately treated and cared for in the community, provided back-up hospital care is available when needed. Another important development has been the recognition that clinical treatment and care is insufficient for recovery and restoration of role functioning following illness onset, and must be supplemented by evidence-based practices in psychiatric rehabilitation. This article describes how allied health professionals can lead recovery oriented approaches that incorporate evidence-based forms of psychiatric rehabilitation. Family psychoeducation and supported employment are provided as examples of such evidence-based practices that require wider implementation.

Time utilisation trends of supported employment services by persons with mental disability in South Africa

BACKGROUND

This paper reports on the second phase of a two-phased study that was undertaken to determine the feasibility of supported employment (SE) as a strategy with which to facilitate the employment of persons with disability in competitive work contexts. The study population comprised people with mental disabilities receiving SE in the Western Cape Province, South Africa.

OBJECTIVE

To describe the components of SE utilised by persons with mental disability (i.e. psychiatric or intellectual disability) in terms of type and time utilisation patterns over 12 months.

METHODS

Criterion sampling, a form of purposive sampling, was used to identify 29 study participants - 19 with intellectual disability and 10 with psychiatric disability. Data collection commenced for each participant when a work placement had been identified and preparation for such ensued. Data was collected prospectively for a period of 12 months. SE service components utilised by participants were captured using a data capture sheet that was developed for this purpose.

RESULTS

Time utilisation indicated a steep downwards trend for both cohorts. The decrease in utilisation of SE service components over a period of one year was more pronounced in the psychiatric disability (PD) cohort, who utilized almost half the total SE services in the first month.

CONCLUSIONS

SE services can be considered as a viable option for return to work in resource-constrained environments. Providers of SE services will need to modify approaches in order to meet contextual realities.

Novel Programs, International Adoptions, or Contextual Adaptations? Meta-Analytical Results From German and Swedish Intervention Research

One of the major dilemmas in intervention and implementation research is adaptation versus adherence. High fidelity to an intervention protocol is essential for internal validity. At the same time, it has been argued that adaptation is necessary for improving the adoption and use of interventions by, for example, improving the match between an intervention and its cultural context, thus improving external validity. This study explores the origins of intervention programs (i.e., novel programs, programs adopted from other contexts with or without adaptation) in two meta-analytic intervention data sets from two European countries and compares the effect sizes of the outcomes of the interventions evaluated. Results are based on two samples of studies evaluating German child and youth preventative interventions (k = 158), and Swedish evaluations of a variety of psychological and social interventions (k = 139). The studies were categorized as novel programs, international adoption and contextual adaptation, with a total of six subcategories. In the German sample, after statistically controlling for some crucial methodological aspects, novel programs were significantly more effective than adopted programs. In the Swedish sample, a trend was found suggesting that adopted programs were less effective than adapted and novel programs. If these results are generalizable and unbiased, they favor novel and adapted programs over adopted programs with no adaptation and indicate that adoption of transported programs should not be done without considering adaptation.

Issues of rigor and feasibility when observing the quality of program implementation: a case study

Program evaluators have paid little attention in the literature to the manner in which measuring the quality of implementation with observations requires tradeoffs between rigor (reliability and validity) and program evaluation feasibility. We present a case example of how we addressed rigor in light of feasibility concerns when developing and conducting observations for measuring the quality of implementation of a small education professional development program. We discuss the results of meta-evaluative analyses of the reliability of the quality observations, and we present conclusions about conducting observations in a rigorous and feasible manner. The results show that the feasibility constraints that we faced did not notably reduce the rigor of our methods.

Health and well-being of homeless veterans participating in transitional and supported employment: Six-month outcomes

Supported employment, specifically individual placement and support (IPS), improves competitive employment (CE) rates for individuals with serious mental illness, but has not shown greater improvement in non-vocational outcomes than other rehabilitation approaches. The Department of Veterans Affairs offers two types of vocational services, IPS and transitional work experience (TWE), but no study has compared the effectiveness of these approaches. This secondary analysis of data from a study of homeless veterans compared 6 mo improvement in diverse outcomes for five employment patterns: never worked, worked only in TWE, worked in TWE followed by CE, worked in CE without IPS, and worked in CE with IPS referral. Veterans referred to IPS were more likely to be competitively employed. Those who worked in CE (whether following TWE or with or without IPS referral) showed the greatest increase in days worked, employment income, and total income and the greatest decrease in public support income when compared with those who worked only in TWE or not at all. Veterans in TWE showed the greatest increase in residential treatment days, but there were no other differences in non-vocational outcomes between groups. There are multiple paths to CE, but few differences in non-vocational outcomes across employment experiences.

Ethnography, fidelity, and the evidence that anthropology adds: supplementing the fidelity process in a clinical trial of supported employment

This discussion considers the role and findings of ethnographic research within a clinical trial of supported employment for veterans with spinal cord injury. Contributing to qualitative evaluation research and to debates over anthropological evidence vis-à-vis clinical trials, we demonstrate how enactors of a randomized controlled trial can simultaneously attend to both the trial's evidentiary and procedural requirements and to the lived experiences and needs of patients and clinicians. Three major findings are described: (1) contextual information essential to fidelity efforts within the trial; (2) the role of human interrelationships and idiosyncratic networks in the trial's success; and (3) a mapping of the power and authority structures relevant to the staff's ability to perform the protocol. We emphasize strengths of anthropological ethnography in clinical trials that include the provision of complementary, qualitative data, the capture of otherwise unmeasured parts of the trial, and the realization of important information for the translation of the clinical findings into new settings.

A review and retrospective analysis of mental health services provided after the September 11 attacks

OBJECTIVE

Various mental health services were provided in New York City in the response to the September 11 attacks. This article describes these services and the subsequent research in disaster psychiatry to identify recommended practices for future disasters.

METHOD

A PubMed search identified 198 articles since the 9/11 attacks that included "New York City," and either "9/11" or "disaster," and either "psychiatry" or "mental health." The abstracts of all of these articles were reviewed to identify articles that described clinical services and those articles were then reviewed in detail. Additional reports on 9/11-related services were identified through book chapters.

RESULTS

Acute services included debriefings and other single-time outreach interventions. Postacute phase interventions included community outreach with the goal of addressing normative psychological sequelae, multisession interventions based on evidence-based treatments, and hospital-based psychiatric services provided in the context of general health screenings for response workers.

CONCLUSIONS

Many programs integrated medical and nonmedical services. The professional literature in the decade since this disaster has shifted recommended practices away from psychological debriefings in the acute phase to the implementation of psychological first aid (PFA). Many of the acute 9/11-related services actually closely resembled PFA. In the postacute phase, resources need to be made available for more resource-intensive psychiatric treatment for the fraction of survivors who develop psychiatric illness.

Advances in fidelity measurement for mental health services research: four measures

Mental health intervention research requires clear and accurate specification of treatment conditions in intervention studies. Measures are increasingly available for community-based interventions for persons with serious mental illnesses. Measures must go beyond structural features to assess critical processes in interventions. They must also balance effectiveness, or adequate coverage of active treatment elements, with efficiency, or the degree to which measures may be used cost-effectively. The context of their use is changing with the emergence of new frameworks for implementation research and quality improvement. To illustrate a range of approaches, this article describes four recently developed fidelity measures: Cognitive Therapy for Psychosis Adherence Scale, Strengths Model Fidelity Scale, Illness Management and Recovery Program Fidelity Scale, and Tool for Measurement of ACT. The fidelity measures assess interventions in a range of treatment contexts from dyads to teams. Each measure focuses assessment resources on critical elements. Each has demonstrated coverage of its target intervention and satisfactory psychometric properties and is related to outcomes. Measures have been used for training, quality improvement, or certification. They assess domains and have uses beyond their nominal position in implementation and quality frameworks. This review of recent fidelity measures indicates that process components in community-based interventions can be effectively assessed. Omission of elements assessing potentially critical active treatment components poses risk to both research and practice until there is evidence to demonstrate that they are nonessential. Further development of fidelity measurement theory and approaches should proceed in conjunction with development of theory and methods in implementation science.

Fidelity and moderating factors in complex interventions: a case study of a continuum of care program for frail elderly people in health and social care

Background

Prior studies measuring fidelity of complex interventions have mainly evaluated adherence, and not taken factors affecting adherence into consideration. A need for studies that clarify the concept of fidelity and the function of factors moderating fidelity has been emphasized. The aim of the study was to systematically evaluate implementation fidelity and possible factors influencing fidelity of a complex care continuum intervention for frail elderly people.

Methods

The intervention was a systematization of the collaboration between a nurse with geriatric expertise situated at the emergency department, the hospital ward staff, and a multi-professional team with a case manager in the municipal care services for older people. Implementation was evaluated between September 2008 and May 2010 with observations of work practices, stakeholder interviews, and document analysis according to a modified version of The Conceptual Framework for Implementation Fidelity.

Results

A total of 16 of the 18 intervention components were to a great extent delivered as planned, while some new components were added to the model. No changes in the frequency or duration of the 18 components were observed, but the dose of the added components varied over time. Changes in fidelity were caused in a complex, interrelated fashion by all the moderating factors in the framework, i.e., context, staff and participant responsiveness, facilitation, recruitment, and complexity.

Discussion

The Conceptual Framework for Implementation Fidelity was empirically useful and included comprehensive measures of factors affecting fidelity. Future studies should focus on developing the framework with regard to how to investigate relationships between the moderating factors and fidelity over time.

Trial registration

ClinicalTrials.gov, NCT01260493.

The Importance of Service Integration in Developing Effective Employment Services for People with Severe Mental Health Conditions

Introduction:

This report examines the importance of integrating mental health services with supported employment services for people with severe mental health conditions and psychiatric disabilities. Integrated mental health and employment services are considered a key principle of evidence-based practices in supported employment for this group.

Method:

The importance of this principle was tested by examining both the service practices and the employment outcomes attained by 270 clients of a segregated employment service in Christchurch, New Zealand. Service practices were examined using an established scale to assess the fidelity of evidence-based practices in supported employment. Employment outcomes were benchmarked to a range of employment outcome variables reported in national and international studies of specialised supported employment services.

Results:

Despite only a fair overall fidelity score of 64/75, 73.7% of clients commenced competitive employment within a mean of 86 days from programme entry. The mean hours worked by those employed was 22 hours per week, and 41.5% of those who commenced employment worked for 26 weeks or more.

Conclusion:

These promising results confirm that very good implementation of evidence-based practices other than service integration practices can compensate for the disadvantages of service segregation.

The TMACT: a new tool for measuring fidelity to assertive community treatment

BACKGROUND

Fidelity assessment is important for implementation of evidence-based practices (EBPs), including assertive community treatment (ACT).

OBJECTIVES

The TMACT, an enhanced fidelity tool, was developed and pilot-tested to better assess critical ACT structures and processes.

DESIGN

Ten ACT teams were administered the TMACT and the long-standing ACT fidelity measure, the Dartmouth Assertive Community Treatment Scale (DACTS), at baseline, 6, 12, and 18 months.

RESULTS

Overall, fidelity scores for all 10 teams were relatively high. Six teams showed improvement, concluding with high TMACT scores at 18 months. Four teams with significantly lower total scores had experienced turnover and organizational barriers. TMACT ratings were higher in core ACT practices than in recovery practices and EBPs. TMACT scores rose steadily but were significantly lower than DACTS scores, which remained unchanged.

CONCLUSIONS

The TMACT sets higher performance standards through enhanced assessment of recovery-orientation, EBPs, and teamwork and is more sensitive to change than the DACTS.

Employment services as an early intervention for young people with mental illness

AIM

To examine the service characteristics and effectiveness of a segregated employment service assisting young clients with mental illness in New Zealand.

METHODS

The service assisted both youth and adults with severe mental illness to find and keep competitive employment. A retrospective case study method was used to examine service effectiveness with respect to employment outcomes attained by 49 clients aged 16–25 years over a 2-year period (2005–2007). These results were compared with recent national and international benchmarks.

RESULTS

As a service segregated from public mental health services, there were no formal arrangements with local mental health teams, limiting coordination of services and reducing fidelity to evidence-based practices in supported employment. Despite an inability to collaborate closely with local community mental health services and a contract not specifically targeting youth, the service was high performing on a range of employment outcome variables.

CONCLUSIONS

Subject to some study design and benchmarking limitations, these results support the continuing use of evidence-based practices in supported employment and supported education as important early interventions for young people with mental illnesses.

The 2009 schizophrenia PORT psychosocial treatment recommendations and summary statements

The Schizophrenia Patient Outcomes Research Team (PORT) psychosocial treatment recommendations provide a comprehensive summary of current evidence-based psychosocial treatment interventions for persons with schizophrenia. There have been 2 previous sets of psychosocial treatment recommendations (Lehman AF, Steinwachs DM. Translating research into practice: the Schizophrenia Patient Outcomes Research Team (PORT) treatment recommendations. Schizophr Bull. 1998;24:1-10 and Lehman AF, Kreyenbuhl J, Buchanan RW, et al. The Schizophrenia Patient Outcomes Research Team (PORT): updated treatment recommendations 2003. Schizophr Bull. 2004;30:193-217). This article reports the third set of PORT recommendations that includes updated reviews in 7 areas as well as adding 5 new areas of review. Members of the psychosocial Evidence Review Group conducted reviews of the literature in each intervention area and drafted the recommendation or summary statement with supporting discussion. A Psychosocial Advisory Committee was consulted in all aspects of the review, and an expert panel commented on draft recommendations and summary statements. Our review process produced 8 treatment recommendations in the following areas: assertive community treatment, supported employment, cognitive behavioral therapy, family-based services, token economy, skills training, psychosocial interventions for alcohol and substance use disorders, and psychosocial interventions for weight management. Reviews of treatments focused on medication adherence, cognitive remediation, psychosocial treatments for recent onset schizophrenia, and peer support and peer-delivered services indicated that none of these treatment areas yet have enough evidence to merit a treatment recommendation, though each is an emerging area of interest. This update of PORT psychosocial treatment recommendations underscores both the expansion of knowledge regarding psychosocial treatments for persons with schizophrenia at the same time as the limitations in their implementation in clinical practice settings.

Creation of a measure to assess knowledge of the individual placement and support model

OBJECTIVE

This article describes the development and psychometric evaluation of the IPS-Q, a quiz assessing practitioner knowledge of the Individual Placement and Support (IPS) model.

METHODS

Using multiple resources, including the IPS Fidelity Scale and publications about the IPS model, a 30-item multiple-choice quiz covering fundamental IPS principles and key components was developed. Ratings by eight IPS experts supported its content validity. To examine the IPS-Q's psychometric properties, 107 practitioners from high-fidelity IPS programs and 59 practitioners from non-IPS vocational programs were administered the IPS-Q via a web-based survey tool.

RESULTS

The measure displayed good internal consistency (Cronbach's alpha = .89) and a difficulty level consistent with that of typical job knowledge tests. The IPS-Q did not correlate with demographic variables that were considered extraneous to knowledge of IPS and did correlate with self-reported supported employment familiarity. However, contrary to our hypothesis, the quiz was not significantly correlated with exposure to supported employment. The IPS-Q significantly discriminated between practitioners working in high-fidelity IPS programs and non-IPS vocational programs. Furthermore, supervisors from high-fidelity IPS programs scored significantly higher on the measure than vocational workers from high-fidelity IPS programs.

CONCLUSIONS

The IPS-Q is a valid measure of IPS knowledge and might prove useful for assessment, research, and training purposes. Practitioners sometimes assume that they understand a program model when in fact they have misconceptions. The IPS-Q provides an objective way to gauge practitioner knowledge.

Reviewing the Theory and Practice of Occupational Therapy in Mental Health Rehabilitation

The delineation of evidence-based practices in supported employment for people with schizophrenia now represents a paradigm shift in the theory and practice of mental health rehabilitation. The principles and methods of traditional vocational rehabilitation and traditional mental health rehabilitation are giving way to evidence-based practices in supported employment, which are consistently proving two to three times more effective at producing competitive employment outcomes.

These practices include close coordination with optimal forms of mental health treatment and care and highly individualised forms of intensive supported employment. There is a focus on the vocational services being provided, whereas the traditional and currently prevailing approach follows a more gradual and stepwise process with less of an individual focus, and where individual characteristics are considered important predictors of vocational rehabilitation success.

This paradigm shift now challenges occupational therapists working in mental health rehabilitation to revise their theory and practice critically in order to support the implementation of evidence-based practices in supported employment for people with schizophrenia and related psychotic disorders. This article discusses how occupational therapists can adapt to this paradigm shift and revitalise their theory and practice in mental health rehabilitation.

Implementing evidence-based practices for people with schizophrenia

Over the last decade, a consensus has emerged regarding a set of evidence-based practices for schizophrenia that address symptom management and psychosocial functioning. Yet, surveys suggest that the great majority of the population of individuals with schizophrenia do not receive evidence-based care. In this article, we review the empirical literature on implementation of evidence-based practices for schizophrenia patients. We first examine lessons learned from implementation studies in general medicine. We then summarize the implementation literature specific to schizophrenia, including medication practices, psychosocial interventions, information technology, and state- and federal-level interventions. We conclude with recommendations for future directions.

Individualization and quality improvement: two new scales to complement measurement of program fidelity

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.

The State Health Authority Yardstick (SHAY)

State mental health authorities have a leadership role in implementing evidence-based practices (EBPs), but few instruments are available to assess the impact of this role. We describe the development of the State Mental Health Authority Yardstick (SHAY), a behaviorally anchored instrument designed to assess state-level facilitating conditions associated with successful implementation of EBPs in community mental health centers. The SHAY assesses the SMHA role in seven domains: Planning, Financing, Training, Leadership, Policies and Regulations, Quality Improvement, and Stakeholders. Preliminary evidence from the National Evidence-Based Practices Project partially supports the construct and criterion-oriented validity of this instrument for rating state-level activities supporting or blocking the implementation of evidence-based practices.