Managing what you measure: creating outcome-driven systems of care for youth with serious emotional disturbances

Development of a measure to assess the implementation of children's systems of care: the Systems of Care Implementation Survey (SOCIS)

The children's system of care framework has been extensively implemented in the U.S. Since its inception in 1993, the Comprehensive Community Mental Health Services for Children and Their Families Program has invested in excess of $1 billion supporting the development of systems of care in 164 grantee sites across the country. Despite these efforts to implement children's systems of care nationally, little is known about the extent to which the principles and values actually have been put into practice outside of the funded grantee sites. This paper describes the development of the Systems of Care Implementation Survey, a measure designed specifically for the first ever study assessing the level of implementation of factors contributing to effective children's systems of care in a nationally representative sample of counties throughout the U.S.

Determinates of youth and parent satisfaction in usual care psychotherapy

OBJECTIVE

Client satisfaction with mental health services is used commonly as an indicator of the quality of care, but there is minimal research on the construct of client satisfaction in youth services, and the extent to which satisfaction is related to improvements in clinical functioning versus other determinants. We examined the relationship between parent and youth satisfaction with youth services, and tested for significant determinants of satisfaction across three major domains: (1) change in youth clinical functioning; (2) youth/family service entry characteristics; (3) treatment/therapist characteristics.

METHOD

The participants were 143 youths receiving community-based outpatient care. Youths and parents were interviewed at service entry and six months later using well-established measures of clinical functioning and service satisfaction.

RESULTS

Youths and parents reported generally high satisfaction, but the correlation between them was low. Despite testing for many potential predictors of satisfaction, very few significant effects were found. In regression analyses of significant predictors of satisfaction, higher youth satisfaction was significantly associated with Caucasian ethnicity and more positive youth expectations about treatment. Higher parent satisfaction was associated with lower caregiver strain at service entry, increased number of sessions, and improvement in youth-reported functional impairment.

CONCLUSIONS

Client satisfaction remains a rather elusive construct, but it is not necessarily a strong indicator of service effectiveness.

Practice parameter on child and adolescent mental health care in community systems of care

This parameter presents overarching principles and practices for child and adolescent mental health care in community systems of care. Community systems of care are defined broadly as comprising the wide array of child-serving agencies, programs, and practitioners (both public and private), in addition to natural community supports such as religious and consumer organizations. Recommended principles and practices are derived from the system-of-care approach to service delivery. Based on the principles of the Child and Adolescent Service System Program, this approach has had a major influence on community systems of care through extensive federally funded projects and initiatives. The system-of-care model emphasizes that care should be tailored to the individual needs and strengths of the child and family and provided in the most community-based and least restrictive setting that meets their needs. Families are included as partners in the clinical process and are also involved in program development and evaluation. Services are coordinated and integrated into a comprehensive care plan. This model can be practiced even in the absence of formal systems of care or protocols, with the individual clinician promoting interagency coordination and child and family collaboration. This parameter is written for a broad audience of mental health professionals, with special emphasis on the roles of child and adolescent psychiatrists in community systems of care.

Utilizing program evaluation as a strategy to promote community change: evaluation of a comprehensive, community-based, family violence initiative

This paper describes the authors' work in a community that received Federal funding for an integrated system of care to reduce the impact and incidence of exposure to violence for children less than six years of age. The paper includes a review of the conceptual framework that guided the work of the authors and provides a brief overview of the issue of family violence, the impact of this violence on young children, and the Federal response to this issue. In addition, a description of the Initiative and the community in which it was based is provided along with some aspects of the evaluation plan. Finally, the authors discuss how their work with this Initiative depicts an approach to facilitating change within communities.

Developing and maintaining collaboration in systems of care for children and youths with emotional and behavioral disabilities and their families

Many communities have implemented systems of care in an effort to better coordinate and integrate mental health and other social services for children and youths, while simultaneously managing existing funding sources more effectively. Systems of care represent a fundamentally different way of delivering mental health services and accordingly require new approaches for both developing and sustaining collaboration. This article examines obstacles to collaboration and addresses key factors required to build and sustain collaboration.

Assessing the effectiveness of care for youth with severe emotional disturbances: is there agreement between popular outcome measures?

Limited information exists regarding how some popular measures for assessing the effectiveness of services to children with severe emotional disturbance interrelate when used as part of ongoing outcome accountability systems. This article examines the interrelationships--at intake and over time--between the Child Behavior Checklist (CBCL), the Youth Self Report (YSR), the Child and Adolescent Functional Assessment Scale (CAFAS), and Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) clinical diagnoses across six California care systems. At intake, there were mild to moderate relationships between the clinical diagnosis, the CAFAS, the CBCL, and the YSR. These relationships were not as strong as those found in other studies conducted in more controlled research environments. There was limited agreement among the CAFAS, the CBCL, and the YSR regarding the outcomes of the youths served. These findings raise questions regarding translating measures from research settings to clinical environments and reconciling differences between outcome measures.

The Connecticut Mental Health Center Patient Profile Project: application of a service needs index

This article describes a quality assurance effort aimed at defining the characteristics of the patient population of the Connecticut Mental Health Center, a statefunded agency that provides comprehensive clinical and rehabilitative services to persons with mental illness. Also described is how this information guided management decisions in both caseload distribution and clinical service development. This "Patient Profile Project" was informed by research principles which view evaluation as continual, rather than terminal activity that involves key stakeholders from all levels within the mental health system of care and makes maximum use of data in ongoing performance improvement initiatives. The service-need index that the project produced represents our first efforts to accurately capture service need and use it in clinical decision making. This review of the Connecticut Mental Health Center Patient Profile Project illustrates the utility of a continuous evaluation system in promoting improvements in a large mental health treatment system.

The Child and Adolescent Functional Assessment Scale (CAFAS): review and current status

Measures of impairment in psychological and behavioral functioning have a long history in the field of children's mental health, and appear particularly useful in eligibility determination, treatment planning, and outcome evaluation of services for children and adolescents with serious emotional disturbance (SED). One recently developed multidimensional measure of functional impairment-the Child and Adolescent Functional Assessment Scale (CAFAS; K. Hodges, 1989, 1997)-has enjoyed widespread use nationwide. It has been adopted as a tool for making treatment eligibility decisions and documenting outcomes on a statewide level in more than 20 states and on a local level in dozens of research and demonstration projects. In this paper, the technical merits of the CAFAS are closely examined, with the conclusion that empirical evidence is lacking to support its valid use in making the types of treatment decisions for which it is currently being employed across the nation. Furthermore, there appears to be little concern among mental health researchers, practitioners, administrators, and state legislators about these apparent limitations of the CAFAS. The potential benefits of establishing objective and valid level-of-need criteria, using the CAFAS are numerous and the interest in doing so is clear; however, the psychometric limitations of the scale identified in this review need to be addressed before its full potential can be realized.