Principles for assessment of patient outcomes in mental health care

Assessing the Perception of Family and Caregivers' Experience with Mental Health and Substance Use Services

Standardized client feedback surveys encourage a culture of continuous quality improvement, allow for comparison of results over time and across similar types of service providers, and encourage use of evidence-based practices. Recognizing the importance of family and other caregivers in supporting people accessing services for mental health and substance use challenges, a standardized perception-of-care tool (the Ontario Perception of Care Tool for Mental Health and Addictions, OPOC-MHA) was adapted to collect feedback specific to the caregiver experience with these services. A collaborative process engaged a broad range of mental health and/or addiction providers, family advisory networks, and family members and caregivers to identify themes, specific items, and implementation approaches. The final version of the tool evolved through an iterative process of pilot testing and stakeholder feedback. Family member and caregiver perceptions of care will identify service areas in need of improvement, contribute to quality improvement initiatives, and facilitate the comparison of findings over time.

Improving community-based mental health care for children: translating knowledge into action

There is urgent need for improvement in community-based mental health care for children and families. Multiple studies have documented serious limitations in the effectiveness of "usual care." Fortunately, many empirically-supported strategies to improve care have been developed, and thus there is now a great deal of knowledge available to address this significant public health problem. The goal of this selective review is to highlight and synthesize that empirically-supported knowledge to stimulate and facilitate the needed translation of knowledge into action. The review provides a sound foundation for constructing improved services by consolidating descriptive data on the status quo in children's mental health care, as well as evidence for an array of promising strategies to improve (a) Service access and engagement; (b) Delivery of evidence-based practices; and (c) Outcome accountability. A multi-level framework is used to highlight recommended care improvement targets.

The relationship between symptomatic changes and perceived improvement among patients treated in Brazilian community mental health services

Perceived improvement has been proposed as an important patient-reported outcome measure in mental health services evaluation. One advantage of this measure is that it can indicate whether other outcome measures, as pre-post differences in symptoms, correspond to a noticeable impact in patients' lives, as assessed by themselves. This study investigated the association between observer and self-reported symptomatic changes and perceived improvement by patients treated in two Brazilian outpatient mental health services. Significant and positive correlations were found between perceived improvement scores and both pre-post differences, obtained in observer-reported and patient-reported symptom scores. Nevertheless, scores of perceived improvement showed to be more correlated to patient-reported than to the observer-rated symptomatic change score. In addition, a greater correlation was found between perceived improvement scores and post-treatment symptom scores, compared to pre-treatment. These results suggest that an improvement in symptom severity, measured by pre-post differences scores, corresponds to the patient perception that he is actually better than before. However, the correlations found were moderate, suggesting that other factors may also be related to perceived improvement and must be investigated in further studies.

A good quality of life under the influence of methadone: a qualitative study among opiate-dependent individuals

BACKGROUND

Attention from researchers and health care workers to the quality of life (QoL) of opiate users is growing, but most studies are quantitative, giving limited attention to the consumer's perspective. No information is available on how opiate-dependent individuals themselves perceive QoL and what they see as the important components that contribute to a good QoL.

OBJECTIVES

This qualitative study aims to expand our knowledge concerning opiate-dependent individuals' perceptions of a good QoL and the impact of methadone on components of a good QoL.

METHODS

In-depth interviews were conducted with 25 opiate-dependent individuals aged between 26 and 46 years old who started a methadone maintenance treatment at least 5 years ago. Purposive sampling was used to recruit participants with different socio-demographic characteristics and drug use profiles. The interviews were audio-tape recorded, transcribed verbatim and analysed thematically.

RESULTS

Thematic analyses revealed five key themes contributing to a good QoL for opiate-dependent individuals: (1) having social relationships, (2) holding an occupation, (3) feeling good about one's self, (4) being independent and (5) having a meaningful life. Opiate-dependent individuals valued methadone's ability to help them function normally, overcome their psychological problems and dependence on illicit opiates, and support them in achieving certain life goals. On the other hand, stigmatisation, discrimination, dependence on methadone and the drug's paralysing effects on their emotions were mentioned as common negative consequences.

CONCLUSIONS

The findings of this study highlight the importance of supporting opiate-dependent individuals in their daily life by means of practical, social and environmental support (alongside pharmacological treatment) in order to improve their QoL. This study further illustrates the ambivalent influence of methadone on opiate-dependent individuals' QoL, and demonstrates how something commonly perceived as a 'good' can also be a 'bad' for some people. Efforts should be made to limit the negative consequences of methadone on opiate-dependent individuals' QoL, while increasing its potential benefits.

Factor structure of the Multnomah Community Ability Scale--longitudinal analysis

Community mental health services benefit from measuring clinical outcomes relevant to a community-based context in contrast to medically modeled outcomes. The Multnomah Community Ability Scale (MCAS) addresses broad dimensions of community functioning and was developed for clinical and evaluation purposes. We assessed the structural consistency and fit of the scale as a measure of community functioning through confirmatory factor analysis using a longitudinal sample of individuals (n=408) with severe and persistent mental illness receiving services from community mental health programs. None of the previously hypothesised factor solutions achieved a good fit and a high degree of invariance over time was observed. Through exploratory factor analysis, the possibility of alternative solutions was explored. After exclusion of two of the 17 items, four models--including four-, three-, two- and one-factor solutions--were tested for fit and invariance with no improvement. We discuss our findings of poor fit under the assumption that the MCAS should psychometrically behave as a scale. Alternative interpretations for the tool and suggestions for the use of its items as an index that measures aspects of disability are proposed.

Reliability and validity of the BASIS-24 Mental Health Survey for Whites, African-Americans, and Latinos

Increasing racial and ethnic diversity calls for mental health assessment instruments that are appropriate, reliable, and valid for the wide range of cultures that comprise the current US population. However, most assessment instruments have not been tested on diverse samples. This study assessed psychometric properties and sensitivity to change of the revised Behavior and Symptom Identification Scale (BASIS-24) among the three largest race/ethnicity groups in the USA: Whites, African-Americans, and Latinos. BASIS-24 assessments were obtained for 2436 inpatients and 2975 outpatients treated at one of 27 mental health and/or substance abuse programs. Confirmatory factor analysis and several psychometric tests supported the factor structure, reliability, concurrent validity, and sensitivity of the instrument within each race/ethnicity group, although discriminant validity may be weaker for African-Americans and Latinos than for Whites. Further research is needed to test and validate assessment instruments with other race/ethnicity groups.

The Revised Behavior and Symptom Identification Scale (BASIS-R)

BACKGROUND

To assess outcomes of health services, providers need brief, responsive, reliable, and valid measures that can be implemented in clinical settings with minimal cost and burden. The Behavior and Symptom Identification Scale (BASIS-32) is a self-report measure developed in 1984 to assess mental health treatment outcomes. During the past 3 years, multiple methods were used to revise the instrument to improve reliability, validity, and applicability to diverse groups of mental health service recipients.

OBJECTIVE

The objective of this study was to field test the revised instrument, make further changes based on analysis of the field test data, and assess reliability and validity of the final version (BASIS-24).

METHODS

A field test was implemented at 27 treatment sites across the United States. A total of 2656 inpatients and 3222 outpatients participated. Factor analytic methods, classic test theory, and item response theory modeling were used to select the most discriminating, nonredundant items for inclusion in the final version of the instrument and to assess its reliability and validity. Item response theory modeling was used to score the instrument.

RESULTS

The final instrument includes 24 items assessing 6 domains: depression/ functioning, interpersonal relationships, self-harm, emotional lability, psychosis, and substance abuse. Test-retest and internal consistency reliability were acceptable. Tests of construct and discriminant validity supported the instrument's ability to discriminate groups expected to differ in mental health status, and its correlation with other measures of mental health.

CONCLUSIONS

Analyses of the BASIS-24 supported its reliability and validity for assessing mental health status from the patient's perspective.

Using the effect size to model change in preference values from descriptive health status

OBJECTIVES

This pilot study describes a modelling approach to translate group-level changes in health status into changes in preference values, by using the effect size (ES) to summarize group-level improvement.

METHODS

ESs are the standardized mean difference between treatment groups in standard deviation (SD) units. Vignettes depicting varying severity in SD decrements on the SF-12 mental health summary scale, with corresponding symptom severity profiles, were valued by a convenience sample of general practitioners (n = 42) using the rating scale (RS) and time trade-off methods. Translation factors between ES differences and change in preference value were developed for five mental disorders, such that ES from published meta-analyses could be transformed into predicted changes in preference values.

RESULTS

An ES difference in health status was associated with an average 0.171-0.204 difference in preference value using the RS, and 0.104-0.158 using the time trade off.

CONCLUSIONS

This observed relationship may be particular to the specific versions of the measures employed in the present study. With further development using different raters and preference measures, this approach may expand the evidence base available for modelling preference change for economic analyses from existing data.

Cost-effectiveness of current and optimal treatment for schizophrenia

BACKGROUND

This paper is part of a project to identify the proportion of the burden of each mental disorder averted by current and optimal interventions, and the cost-effectiveness of both.

AIMS

To use epidemiological data on schizophrenia to model the cost-effectiveness of current and optimal treatment.

METHOD

Calculate the burden of schizophrenia in the years lived with disability (YLD) component of disability-adjusted life-years lost, the proportion averted by current interventions, the proportion that could be averted by optimal treatment and the cost-effectiveness of both.

RESULTS

Current interventions avert some 13% of the burden, whereas 22% could be averted by optimal treatment. Current interventions cost about AUS 200,000 dollars per YLD averted, whereas optimal treatment at a similar cost could increase the number of YLDs averted by two-thirds. Even so, the majority of the burden of schizophrenia remains unavertable.

CONCLUSIONS

Optimal treatment is affordable within the present budget and should be implemented.

Assessing patients' perception of self-care agency in psychiatric care

Self-care agency is necessary for psychiatric patients to maintain a level of health that will enable them to have a satisfying personal life as well as to participate in their communities. This article discusses some theoretical and operational dimensions of self-care agency in patients treated for psychiatric illness, with an emphasis on self-care agency in mood disorder. Tools for measuring self-care agency based on Orem's power components are available, but tools specific to psychiatric patients' strengths and deficits, such as in the area of medication adherence and symptom management, need to be developed.

Case-mix adjustment of adolescent mental health treatment outcomes

Case-mix adjustment methods are needed to account for differences between providers when the youth they treat have characteristics that adversely affect treatment success. This study explores variables for adjusting mental health treatment outcomes for adolescents and the differential effects of case-mix adjustment on providers' performance. Linear regression modeling was used to identify case-mix variables for five outcomes. Predictive equations for each outcome were developed for models based on intake clinical data alone, clinical data plus administrative data, and clinical data plus data describing youth history and family environment. Variance explained by intake clinical data alone did not increase appreciably with the addition of administrative data or data describing youth history and family environment. Adjusting outcomes changed the relative performance of certain individual providers substantially, but had a more moderate impact on the overall interpretation of providers' performance.

What outcomes to measure in routine mental health services, and how to assess them: a systematic review

OBJECTIVE

Routine outcome assessment in adult mental health services involves the on-going assessment of patient-level outcomes. Use of outcomes to inform treatment is widely recommended, but seldom implemented. The goals of this review were (i) to identify principles that have been proposed for implementing routine outcome assessment, (ii) to identify the full range of outcome domains that have been proposed for assessment, and (iii) to synthesize proposals for specific outcome domains into emergent categories.

METHOD

A systematic review of published and unpublished research was undertaken, using electronic databases, research registers, conference proceedings, expert informants and the World Wide Web. For goal (i) studies were included that proposed principles for implementing routine outcome assessment. For goal (ii) studies were included that identified at least two patient-level outcome domains for patients using adult mental health services and made some reference to a broader literature base.

RESULTS

Six thousand four hundred publications matched initial search criteria. Seven distinct sets of principles for choosing patient-level outcomes were located, which showed a fair degree of consensus. Sixteen outcome domain proposals were identified, which were synthesized into seven emergent categories: wellbeing, cognition/emotion, behaviour, physical health, interpersonal, society and services.

CONCLUSIONS

The findings from this review were used to develop a four-step method for adult mental health services wishing to implement routine outcome assessment.

Challenges encountered in measuring outcome for a rural psychiatric residential program

Evaluation of outcome measures can provide policymakers with valuable information on the effectiveness of psychiatric rehabilitation. Two specific challenges in collecting outcome measures for psychiatric rehabilitation programs are heterogeneity of outcomes and difficulty with follow-up. These two challenges were illustrated in the process of evaluating Rose Hill Center, a rural residential psychiatric rehabilitation program. The original design was to conduct interviews with former residents and family members and verify healthcare utilization. The difficulty of locating people and their reluctance to participate conspired to lower the follow-up rates. The design was modified to improve the follow-up rate but decrease the details of specific outcomes. The results showed a high but biased follow-up rate, with more information obtained for people who graduated from the program. The residents with planned discharge showed excellent outcomes in terms of living situation, working situation, healthcare utilization, and low severity of current symptoms. High compliance with medication spoke to the program's philosophy of including the residents in the decision-making process. The dual challenges of heterogeneity of outcomes and difficulty in follow-up that limit efforts to document the value of psychiatric rehabilitation are discussed.

A framework for selecting performance measures for opioid treatment programs

As a result of new federal regulations released in early 2001 that move the monitoring and evaluation of opioid treatment programs from a government regulation to an accreditation model, program staff members are now being challenged to develop performance measurement systems that improve care and service. Using measurement selection criteria is the first step in developing a performance measurement system as a component of an overall quality management (QM) strategy. Opioid treatment programs can "leapfrog" the development of such systems by using lessons learned from the healthcare quality industry. This article reviews performance measurement definitions, proposes performance measurement selection criteria, and makes a business case for Internet automation and accessibility. Performance measurement sets that are appropriate for opioid treatment programs are proposed, followed by a discussion on how performance measurement can be used within a comprehensive QM program. It is hoped that through development, adoption, and implementation of such a performance measurement program, treatment for clients and their families will continuously improve.

The use of patient-level outcomes to inform treatment

OBJECTIVE

The assessment of outcome for any purpose is not undertaken routinely in European mental health services. This paper discusses the merits of using outcome data to inform the planning of mental health care for individual patients, and provides practical advice to support the implementation of this new approach to working.

METHOD

The use of outcomes in North America and Europe is briefly reviewed. A conceptual basis is proposed for routine outcome assessment--the ongoing measurement and use of outcome data to inform decisions about whether to continue, change or curtail treatment. A cognitive psychology model is developed which indicates that the routine use of outcomes will improve mental health care. Perceived problems with routine outcome assessment are discussed, and principles for implementation are identified.

RESULTS

Outcomes are used mainly for generating local-level (rather than patient-level) data in North America, and rarely used in Europe. The use of outcome data routinely may facilitate reflective clinical practice, a model of decision-making which leads to a higher quality of clinical care than automated problem-solving. One issue relates to the use of standardised assessments designed for research purposes in clinical settings, and this is being addressed through the development of a new generation of outcome measures which are explicitly designed for clinical use. However, most clinicians remain unconvinced of the benefits of routine outcome assessment, and relevant research is currently underway across Europe which will address this concern. Scientific principles to maximise quality and pragmatic principles to maximise the chances of successful implementation are identified.

CONCLUSIONS

The routine use of outcomes will become increasingly prominent in European mental health services. This provides clinicians with an opportunity to improve the quality of clinical care offered to patients.

CASIG: a consumer-centered assessment for planning individualized treatment and evaluating program outcomes

This paper reports the psychometric characteristics of a measure that assesses the treatment outcomes of individuals with serious and persistent mental illness. Unlike other outcome measures, this one is designed to be embedded in the clinical process of planning and evaluating treatment. It collects individualized data, structures how the data are used to plan and evaluate a client's treatment, and produces aggregate information relevant for research and program purposes. Two parallel versions were developed: one for the client's self-report, and one for an informant's report. The self-report measure was administered by peer-interviewers to 244 community interviewees, and by inpatient peer-interviewers to 93 inpatient interviewees. The community interviewees also completed the BASIS-32 and SF-36. Informants for 103 of the community interviewees completed the informant version of the measure, and the CCAR. Inpatient staff completed the informant version for 161 inpatient residents without regard for matching the 93 inpatient interviewees. The two versions had acceptable internal consistency, test--retest, and interrater reliabilities. Correlations of the community interviewees' and informants' results with the BASIS-32, SF-36, and CCAR provided evidence of convergent and discriminant validity, as did contrasts between community and inpatients interviewees. The usefulness of the instrument for clinical, program and research purposes is discussed, with emphasis on the characteristics that enhance its value in clinical practice --- assessment of meaningful outcomes, operationalization of client empowerment, comprehensiveness, easy administration, and continuity across time and provider. Also discussed is a computer-based program to summarize and present the results in a rapid, clinically meaningful manner.

Use of consumer ratings for quality improvement in behavioral health insurance plans

BACKGROUND

The Consumer Assessment of Behavioral Healthcare Services (CABHS) survey collects consumers' reports about their health care plans and treatment. The use of the CABHS to identify opportunities for improvement, with specific attention to how organizations have used the survey information for quality improvement, is described.

METHODS

In 1998 and 1999, data were collected from five groups of adult patients in commercial health plans and five groups of adult patients in public assistance health plans with services received through four organizations (one of three managed behavioral health care organizations or a health system). Patients who received behavioral health care services during the previous year were mailed the CABHS survey. Non-respondents were contacted by telephone to complete the survey.

RESULTS

Response rates ranged from 49% to 65% for commercial patient groups and from 36% to 51% for public assistance patients. Promptly getting treatment from clinicians and aspects of care most influenced by health plan policies and operations, such as access to treatment and plan administrative services, received the least positive responses, whereas questions about communication received the most positive responses. In addition, questions about access- and plan-related aspects of quality showed the most interplan variability. Three of the organizations in this study focused quality improvement efforts on access to treatment.

DISCUSSION

Surveys such as the CABHS can identify aspects of the plan and treatment that are improvement priorities. Use of these data is likely to extend beyond the behavioral health plan to consumers, purchasers, regulators, and policymakers, particularly because the National Committee for Quality Assurance is encouraging behavioral health plans to use a similar survey for accreditation purposes.

Toward a level playing field: predictive factors for the outcomes of mental health treatment for adolescents

OBJECTIVE

To understand better the effectiveness of routine treatment for emotional and behavioral problems experienced by adolescents, methods are needed to control for between-provider differences in the distribution of factors that adversely affect treatment success. Such methods are necessary to fairly compare providers' outcomes and to aid clinicians in identifying adolescents for whom routine care may need to be altered. As a preliminary step toward developing a model to adjust treatment outcomes to account for predictive factors, findings from studies of treated samples of adolescents were reviewed to identify the factors that influence the likelihood of treatment success for this population.

METHOD

Medline and PSYCInfo databases were searched for studies of treated adolescents that reported the association between expert-nominated predictive factors and outcomes. Thirty-four studies met inclusion criteria.

RESULTS

Significant predictors identified in these studies include diagnosis, baseline severity of symptoms and functional impairment, family dysfunction, and previous treatment. Several expert-nominated factors have not been adequately studied in treated samples.

CONCLUSIONS

Much basic work is needed before a convincing body of empirical evidence can explain predictive factors for adolescent mental health treatment outcomes. Future efforts should determine a reduced set of predictive factors that can be measured with minimal burden to providers.

What happens when capitated behavioral health comes to town? The transition from the Fort Bragg demonstration to a capitated managed behavioral health contract

Capitated managed care contracts for behavioral health services are becoming more prevalent across the country in both public and private sectors. This study followed the transition from a demonstration project for child mental health services to a capitated managed behavioral health care contract with a for-profit managed care company. The focus of the study was on the impact--at both the service system and the individual consumer level--pertaining to the start-up and maintenance of a capitated managed behavioral health program. A case study using multiple methods and multiple sources of information incorporated a program fidelity framework that examined micro to macro levels of program implementation. The findings of this study include the following: access to services decreased, the lengths of stay and average daily census in the more intensive levels of treatment declined, difficult-to-treat children were shifted to the public sector, and ratings of service system performance and coordination fell.

Developing outpatient mental health services for managed care

One of the characteristics of the best outpatient treaters is their capacity to manage their own anxieties so as not to interfere with their patients' progress. This does not mean "sitting on," shutting out, or otherwise ignoring those signals to prevent their intruding into the therapeutic space. On the contrary, this requires that a clinician be acutely aware of his or her internal dialogue and be able to listen to use it to facilitate the treatment. This has meant simultaneously listening to oneself while paying complete attention to one's patient. This paradox is part of the nature of this work and remains one of its important features. Today, those who choose to treat psychiatric patients in this setting are presented with many more such paradoxes, including the necessity of being aware of (1) populations needing to be served while providing the best care for individual patients; (2) limited resources; and (3) balancing ethical, fiscal, legal, and clinical concerns. This article demonstrates that the successful development of outpatient services in a managed care environment requires attention to these same tensions on a larger scale. It also shows that understanding the evolution of managed care and the myriad competing interests involved are necessary to be able to care successfully for psychiatric patients.

Mission unfulfilled: potholes on the road to mental health parity

Managed care holds the promise of facilitating parity between general medical care and alcohol, drug, and mental health care by reducing expenditures, even while expanding benefits. Limitations in our knowledge of variations in needs and treatment standards for substance use and psychiatric illnesses make such disorders an easy target for management. Costs for behavioral health care services have been reduced at a faster pace than has been the case for general medical care costs. The most severely ill face the potential burdens of managed care as access and intensity of care become more uniform across patient populations.