Implementing routine outcome measures in child and adolescent mental health services: from present to future practice

Measuring what matters to patients: Using goal content to inform measure choice and development

INTRODUCTION

Personalised care requires personalised outcomes and ways of feeding back clinically useful information to clinicians and practitioners, but it is not clear how to best personalise outcome measurement and feedback using existing standardised outcome measures.

METHOD

The constant comparison method of grounded theory was used to compare goal themes derived from goals set at the outset of therapy for 180 children aged between 4 and 17 years, visiting eight child and adolescent mental health services, to existing standardised outcome measures used as part of common national datasets.

RESULTS

In all, 20 out of 27 goal themes corresponded to items on at least one commonly used outcome measure.

DISCUSSION

Consideration of goal themes helped to identify potential relevant outcome measures. However, there were several goal themes that were not captured by items on standardised outcome measures. These seemed to be related to existential factors such as understanding, thinking about oneself and future planning.

CONCLUSION

This presents a powerful framework for how clinicians can use goals to help select a standardised outcome measure (where this is helpful) in addition to the use of a goal-based outcome measure and personalise choices. There may be areas not captured by standardised outcome measures that may be important for children and young people and which may only be currently captured in goal measurement. There is an indication that we may not be measuring what is important to children and young people. We may need to develop or look for new measures that capture these areas.

A Qualitative Exploration of Patient and Clinician Views on Patient Reported Outcome Measures in Child Mental Health and Diabetes Services

Patient Reported Outcome Measures (PROMs) are increasingly being recommended for use in both mental and physical health services. The present study is a qualitative exploration of the views of young people, mothers, and clinicians on PROMs. Semi-structured interviews were conducted with a sample of n = 10 participants (6 young people, 4 clinicians) from mental health services and n = 14 participants (4 young people, 7 mothers, 3 clinicians) from a diabetes service. For different reasons, young people, mothers, and clinicians saw feedback from PROMs as having the potential to alter the scope of clinical discussions.

Complementary primary mental health programs for young people in Australia: Access to Allied Psychological Services (ATAPS) and headspace

OBJECTIVE

Access to Allied Psychological Services (ATAPS) was introduced in 2001 by the Australian Government to provide evidence-based psychological interventions for people with high prevalence disorders. headspace, Australia's National Youth Mental Health Foundation, was established in 2006 to promote and facilitate improvements in the mental health, social wellbeing and economic participation of young people aged 12-25 years. Both programs provided free or low cost psychological services. This paper aims to describe the uptake of psychological services by people aged 12-25 years via ATAPS and headspace, the characteristics of these clients, the types of services received and preliminary client outcomes achieved.

METHODS

Data from 1 July 2009 to 30 June 2012 were sourced from the respective national web-based minimum datasets used for routine data collection in ATAPS and headspace.

RESULTS

In total, 20,156 and 17,337 young people accessed two or more psychological services via ATAPS and headspace, respectively, in the 3-year analysis period. There were notable differences between the clients of, and the services delivered by, the programs. ATAPS clients were less likely to be male (31 vs 39%) and to reside in major cities (51 vs 62%) than headspace clients; ATAPS clients were also older (18-21 vs 15-17 years modal age group). There was some variation in the number and types of psychological sessions that young people received via the programs but the majority received at least one session of cognitive behavioural therapy. Based on limited available outcome data, both programs appear to have produced improvements in clients' mental health; specifically, psychological distress as assessed by the Kessler-10 (K-10) was reduced.

CONCLUSIONS

ATAPS and headspace have delivered free or low-cost psychological services to 12-25 year olds with somewhat different characteristics. Both programs have had promising effects on mental health. ATAPS and headspace have operated in a complementary fashion to fill a service gap for young people.

Patient reported outcome measures in child and adolescent mental health services: associations between clinician demographic characteristics, attitudes and efficacy

BACKGROUND

Policy recommends using patient reported outcome measures (PROMs), yet their use is persistently low. Our aim was to examine the association between PROM use and clinician demographic characteristics, attitudes and efficacy.

METHOD

A sample of N = 109 clinicians completed an online survey.

RESULTS

Clinicians who reported higher levels of use of cognitive behaviour or humanistic approaches had higher levels of PROM use than clinicians who reported lower levels of use of these approaches. Clinicians who reported having received training had higher levels of self-efficacy regarding PROMs than clinicians who reported not having received training, but the effects of training on PROM attitudes and use were not significant. Still, clinicians with more positive attitudes or self-efficacy regarding PROMs had higher levels of PROM use than clinicians with less positive attitudes or self-efficacy regarding PROMs.

CONCLUSION

Clinicians should be supported to have the knowledge, skills and confidence to effectively use PROMs in their clinical practice.

Considering harm and safety in youth mental health: a call for attention and action

The possibility of harm from mental health provision, and in particular harm from youth mental health provision, has been largely overlooked. We contend that if we continue to assume youth mental health services can do no harm, and all that is needed is more services, we continue to risk the possibility that the safety of children and young people is unintentionally compromised. We propose a three level framework for considering harm from youth mental health provision (1. ineffective engagement, 2. ineffective practice and 3. adverse events) and suggest how this framework could be used to support quality improvement in services.

Using Routine Outcome Measures to Provide Feedback at the Service Agency Level

The use of routine outcome measurement (ROM) to assess service effectiveness has been on the rise in mental health settings. However, there is a scarcity of information on the use of ROM data to provide feedback to teams of service providers. In this paper we review the existing literature to identify the principles that can guide the use of ROM data as feedback with the aim to improve quality of service provision in mental health settings. We present a pilot trial of 12 agencies participating in group feedback sessions. The guiding principles and core processes, the procedure and implementation in a pilot trial, lessons learned and future directions are discussed. Based on this experience we conclude that using ROM to implement group feedback among mental health stakeholders is feasible and can generate discussions and directions for improvement.

Using Patient Reported Outcome Measures to Improve Service Effectiveness (UPROMISE): Training clinicians to Use Outcome Measures in Child Mental Health

Patient reported outcome measures (PROMs) are prevalent in child mental health services. In this point of view, we discuss our experience of training clinicians to use PROMs and to interpret and discuss feedback from measures. Findings from pre-post observational data from clinicians who attended either a 1- or 3-day training course showed that clinicians in both courses had more positive attitudes and higher levels of self-efficacy regarding administering measures and using feedback after training. We hope that this special issue will lead the way for future research on training clinicians to use outcome measures so that PROMs may be a source of clinically useful practice based evidence.

Horses for courses? A qualitative exploration of goals formulated in mental health settings by young people, parents, and clinicians

INTRODUCTION

This research sought to explore and categorise goals set by children and young people, parents/caregivers and jointly by a combination of children/young people, parents/caregivers and/or clinicians within mental health settings across the United Kingdom.

METHOD

Using a dataset of 441 goals formed at the outset of 180 treatment episodes (2007-2010) from UK child mental health services using the Goal-Based Outcomes tool, a grounded theory approach was taken, which built on previous research into child-rated goals to develop frameworks for parent and joint goal data which were then compared with the child goal data.

RESULTS

A total of 19 subthemes and four overarching themes were identified for parent goals. A total of 19 subthemes in five overarching themes were identified for joint goals. These were compared with 25 subthemes and three overarching themes for child goals. A comparison of subthemes between parent, child and joint goals demonstrated many consistencies, but also differences. Most commonly rated goals from children focused on coping with specific difficulties, personal growth and independence. Parent goals focused mainly on managing specific difficulties, parent-specific goals and improving self or life. Jointly negotiated goals focused on parent-specific goals, self-confidence and understanding, hopes for the future and managing specific problems.

CONCLUSION

The results suggest that goals may capture areas not captured by other normed outcome measures. In particular, goals may capture higher order, underlying factors, such as confidence, resilience, coping, and parenting factors that may not be explored by other measures. The differences across perspectives also link to existing literature suggesting a different focus on treatment based on perspectives and highlights the potential importance when jointly agreeing goals of ensuring the voice of the child/young person is heard and included in goal setting.

Using a Brief Parent-Report Measure to Track Outcomes for Children and Teens with Internalizing Disorders

The Pediatric Symptom Checklist (PSC) is a widely-used, parent-completed measure of children's emotional and behavioral functioning. Previous research has shown that the PSC and its subscales are generally responsive to patient progress over the course of psychiatric treatment. In this naturalistic study, we examined the performance and utility of the five-item PSC Internalizing Subscale (PSC-IS) as an assessment of routine treatment in outpatient pediatric psychiatry. Parents and clinicians of 1,593 patients aged 17 or younger completed standardized measures at intake and three-month follow-up appointments. Comparisons between PSC-IS scores and clinician-reported diagnoses, internalizing symptoms, and overall functioning showed acceptable levels of agreement. Change scores on the PSC-IS were also larger among patients with internalizing diagnoses than those with non-internalizing diagnoses. As a brief measure of internalizing symptoms, the PSC may be particularly useful to mental health clinicians treating youth with depression and anxiety as a quality assurance or treatment outcome measure.

A systematic review of mental health outcome measures for young people aged 12 to 25 years

BACKGROUND

Mental health outcome measures are used to monitor the quality and effectiveness of mental health services. There is also a growing expectation for implementation of routine measurement and measures being used by clinicians as a feedback monitoring system to improve client outcomes. The recent focus in Australia and elsewhere targeting mental health services to young people aged 12-25 years has meant that outcome measures relevant to this age range are now needed. This is a shift from the traditional divide of child and adolescent services versus adult services with a transitioning age at 18 years. This systematic review is the first to examine mental health outcome measures that are appropriate for the 12 to 25 year age range.

METHODS

MEDLINE and PsychINFO databases were systematically searched to identify studies using mental health outcome measures with young people aged 12 to 25 years. The search strategy complied with the relevant sections of the PRISMA statement.

RESULTS

A total of 184 published articles were identified, covering 29 different outcome measures. The measures were organised into domains that consisted of eight measures of cognition and emotion, nine functioning measures, six quality of life measures, and six multidimensional mental health measures. No measures were designed specifically for young people aged 12 to 25 years and only two had been used by clinicians as a feedback monitoring system. Five measures had been used across the whole 12 to 25 year age range, in a range of mental health settings and were deemed most appropriate for this age group.

CONCLUSIONS

With changes to mental health service systems that increasingly focus on early intervention in adolescence and young adulthood, there is a need for outcome measures designed specifically for those aged 12 to 25 years. In particular, multidimensional measures that are clinically meaningful need to be developed to ensure quality and effectiveness in youth mental health. Additionally, outcome measures can be clinically useful when designed to be used within routine feedback monitoring systems.

Innovations in Practice: Piloting electronic session-by-session monitoring in Child and Adolescent Mental Health Services: a preliminary study

BACKGROUND

Recent UK initiatives have advocated the use of session-by-session outcome measurement in CAMHS. However, little is known about the feasibility of this approach.

METHOD

The PROMPT study (Patient Reported Outcome Monitoring Progress Tracker) piloted an iPad administered brief session-by-session measure (S × S) related to the Strengths and Difficulties Questionnaire impact supplement in three CAMHS teams. We report adherence to electronic S × S monitoring and a preliminary analysis of sensitivity to change.

RESULTS

Adherence to S × S was 57%, which is higher than the completion rates for the standard set of outcome measures usually completed by clinicians and young people. S × S showed some sensitivity to change.

CONCLUSIONS

Session-by-session monitoring in CAMHS is worthy of further pursuit.

Interpreting standardized and idiographic outcome measures in CAMHS: what does change mean and how does it relate to functioning and experience?

BACKGROUND

Patient Reported Outcome Measures (PROMs) are increasingly being used in Child and Adolescent Mental Health Services (CAMHS). The aim of this research was to explore change in standardized and idiographic outcome measures in CAMHS using naturalistic, routinely collected data.

METHOD

We explored change in psychosocial difficulties and impact on daily life as evaluated by a broad standardized measure, the Strengths and Difficulties Questionnaire (SDQ) (Goodman, 1997, J. Child Psychol. Psychiatry, 38, 581) and progress toward goals as evaluated by a personalized idiographic measure, the Goal Based Outcomes tool (GBO) (Law, 2011, Goals and Goal Based Outcomes (GBOs): some useful information. London: CAMHS Press) in a sample of N = 137 CAMHS attenders.

RESULTS

Psychosocial difficulties and impact on daily life showed less change over the course of treatment than progress toward goals in this study. Change in psychosocial difficulties and impact on daily life were also less strongly associated with change in clinician-reported functioning and satisfaction with care at time two than change in progress toward goals.

CONCLUSIONS

Findings of the present research may support previous studies in which service users and clinicians report that idiographic measures are more capable than standardized measures of capturing relevant change for individuals.

Are clinician ratings useful in evaluating outcomes in Child and Adolescent Mental Health Services (CAMHS)? A study of a continuous series of 1446 cases from an inner city CAMHS

RATIONALE, AIMS AND OBJECTIVES

Routine outcome evaluation in Child and Adolescent Mental Health Services is an essential part of effective service delivery but it has been hard for services to obtain client-rated outcomes on more than 50% of cases. Clinician-rated outcomes are examined whether this would provide a valid and reliable way of contributing to addressing this difficulty.

METHOD

This paper will evaluate the pragmatic utility, reliability and validity of a method of measuring clinical outcomes using clinician ratings using an adapted form of the Clinical Global Impressions scale with additional items based on the Every Child Matters framework on a continuous case series of 1446 cases.

RESULTS

A rating of clinical outcome was obtained on 93% of cases. Approximately 55% of cases were rated as improved and about 30% as showing no change with about 5% reported as being worse. Test-retest reliability was acceptable (Pearson r = 0.94; P < 0.001) and criterion validity, comparing clinician and parent ratings, showed a significant correlation of 0.42 on severity of problem (Kendall's tau; t = 2.321, P = 0.02) and 0.36 on the degree of improvement (t = 2.637, P = 0.008). Rates of clinical improvement in studies of usual care suggested similar rates to those reported in this study.

CONCLUSIONS

Clinician ratings were obtained for a high proportion of cases and the burden on clinicians was extremely low with negative outcomes similar to known rates of usual care. This high coverage may add value to the evaluation of service outcomes.

The use of routine outcome monitoring in child semi-residential psychiatry: predicting parents' completion rates

BACKGROUND

Parents' perspectives on their children's treatment process and outcomes are valuable to treatment development and improvement. Participants' engagement in Routine Outcome Monitoring (ROM) has, however, been difficult and may particularly be so in specialized settings, such as semi-residential psychiatry. In this paper, the use of a web-based ROM system implemented in a child semi-residential psychiatric setting is described and predictors associated with low completion rates of questionnaires by parents are identified.

METHODS

Parents and the multidisciplinary team of 46 children admitted to semi-residential psychiatric treatment participated in this study and completed a battery of questionnaires in three month intervals.

RESULTS

The overall completion rate of both parents during ROM assessment was 77 % compared to 83 % of all clinicians involved. Completion of questionnaires by parents was higher around first assessments and declined after a year treatment. For eight clients at least one of the parents stopped filling out questionnaires during ROM measuring. Logistic multilevel analyses revealed initial treatment factors associated with a low completion of questionnaires by parents during ROM: high comorbidity of the child on DSM Axis I, single parenthood, a higher parental educational level and having a weaker therapeutic alliance regarding goal setting.

CONCLUSIONS

The findings in this paper demonstrate relatively high completion of questionnaires by clinicians and parents when using ROM in child semi-residential psychiatry. Strong administrative and electronic support undoubtedly contributed to this result. Clinicians are encouraged to motivate parents to mutually invest in ROM, and to take into account factors indicating a possible lower completion of questionnaires by parents.

Uses and abuses of patient reported outcome measures (PROMs): potential iatrogenic impact of PROMs implementation and how it can be mitigated

Having been a national advocate for the use of patient reported outcome measures (PROMs) in Child and Adolescent Mental Health Services (CAMHS) in the UK for the last decade, I have become increasingly concerned that unless the potential iatrogenic impact of widespread policy requirement for use of PROMs (Department of Health, Children and Young People’s Health Outcomes Strategy, 2012) is recognised and addressed their real potential benefits (Sapyta et al., J Clin Psychol 61(2):145–153, 2005) may never be realized. Drawing on examples from PROMs implementation in CAMHS in the UK (Wolpert et al., J Ment Health 21(2):165–173, 2012a; Child Adolesc Mental Health 17(3):129–130, 2012b). I suggest key ways forward if PROMs are to support best clinical practice rather than undermine it.

The validity of the Child Health Utility instrument (CHU9D) as a routine outcome measure for use in child and adolescent mental health services

BACKGROUND

Few cost-utility studies of child and adolescent mental health services (CAMHS) use quality adjusted life years (a combination of utility weights and time in health state) as the outcome to enable comparison across disparate programs and modalities. Part of the solution to this problem involves embedding preference-based health-related quality of life (PBHRQOL) utility instruments, which generate utility weights, in clinical practice and research. The Child Health Utility (CHU9D) is a generic PBHRQOL instrument developed specifically for use in young people. The purpose of this study was to assess the suitability of the CHU9D as a routine outcome measure in CAMHS clinical practice.

METHODS

Two hundred caregivers of children receiving community mental health services completed the CHU9D alongside a standardised child and adolescent mental health measure (the Strengths and Difficulties Questionnaire - SDQ) during a telephone interview. We investigated face validity, practicality, internal consistency, and convergent validity of the CHU9D. In addition, we compared the utility weights obtained in this group with utility weights from other studies of child and adolescent mental health populations.

RESULTS

Participants found the CHU9D easy and quick to complete. It demonstrated acceptable internal consistency, and correlated moderately with the SDQ. It was able to discriminate between children in the abnormal range and those in the non-clinical/borderline range as measured by the SDQ. Three CHU9D items without corollaries in the SDQ (sleep, schoolwork, daily routine) were found to be significant predictors of the SDQ total score and may be useful clinical metrics. The mean utility weight of this sample was comparable with clinical subsamples from other CHU9D studies, but was significantly higher than mean utility weights noted in other child and adolescent mental health samples.

CONCLUSIONS

Initial validation suggests further investigation of the CHU9D as a routine outcome measure in CAMHS is warranted. Further investigation should explore test-retest reliability, sensitivity to change, concordance between caregiver and child-completed forms, and the calibration of the utility weights. Differences between utility weights generated by the CHU9D and other utility instruments in this population should be further examined by administering a range of PBHRQOL instruments concurrently in a mental health group.

Cutting a long story short? The clinical relevance of asking parents, nurses, and young children themselves to identify children's mental health problems by one or two questions

BACKGROUND AND AIMS

Assessing young children's mental health is a crucial and challenging task. The aim of the study was to evaluate the clinical relevance of asking parents, nurses, and young children themselves to identify children's mental health problems by only one or two questions.

METHODS

In regular health check-ups of 4- to 9-year-old children (n = 2682), parents and public health nurses assessed by one question whether the child had any emotional or behavioral difficulties. The child completed a self-evaluation enquiry on his/her emotional well-being. A stratified proportion of the participating parents were invited to a diagnostic interview.

RESULTS

Sensitivities were fairly good for the parents' (68%), nurses' (65%), and their combined (79%) one-question screens. Difficulties identified by parents and nurses were major risks (OR 10-14) for any child psychiatric disorders (P < 0.001). The child's self-evaluation was related to 2-fold to 3-fold risks (P < 0.05) for any psychiatric diagnosis, for any emotional diagnosis, and for negative situational factors.

CONCLUSION

The one-question screen for parents and public health nurses together quite adequately identified the young children with mental health problems. The child's self-evaluation provided relevant and complementary information on his/her mental health and especially emotional problems.

A qualitative process evaluation of electronic session-by-session outcome measurement in child and adolescent mental health services

BACKGROUND

Regular monitoring of patient progress is important to assess the clinical effectiveness of an intervention. Recently, initiatives within UK child and adolescent mental health services (CAMHS) have advocated the use of session-by-session monitoring to continually evaluate the patient's outcome throughout the course of the intervention. However, the feasibility and acceptability of such regular monitoring is unknown.

METHOD

Semi-structured qualitative interviews were conducted with clinicians (n = 10), administrative staff (n = 8) and families (n = 15) who participated in a feasibility study of an electronic session-by-session outcome monitoring tool, (SxS), which is based on the Strengths and Difficulties Questionnaire (SDQ). This study took place in three CAMHS clinics in Nottinghamshire. The interview transcripts were thematically analysed.

RESULTS

We found clinicians accepted the need to complete outcome measures, particularly valuing those completed by the patient. However, there were some difficulties with engaging clinicians in this practice and in the training offered. Generally, patients were supportive of completing SxS in the waiting room prior to the clinic session and assistance with the process from administrative staff was seen to be a key factor. Clinicians and families found the feedback reports created from SxS to be helpful for tracking progress, facilitating communication and engagement, and as a point of reflection. The use of technology was considered positively, although some technological difficulties hindered the completion of SxS. Clinicians and families appreciated the brevity of SxS, but some were concerned that a short questionnaire could not adequately encapsulate the complexity of the patient's issues.

CONCLUSIONS

The findings show the need for appropriate infrastructure, mandatory training, and support to enable an effective system of session-by-session monitoring. Our findings indicate that clinicians, administrative staff and young people and their parents/carers would support regular monitoring if the system is easy to implement, with a standard 'clinic-wide' adoption of the procedure, and the resulting data are clinically useful.

The use of routine outcome measures in two child and adolescent mental health services: a completed audit cycle

BACKGROUND

Routine outcome measurement (ROM) is important for assessing the clinical effectiveness of health services and for monitoring patient outcomes. Within Child and Adolescent Mental Health Services (CAMHS) in the UK the adoption of ROM in CAMHS has been supported by both national and local initiatives (such as government strategies, local commissioning policy, and research).

METHODS

With the aim of assessing how these policies and initiatives may have influenced the uptake of ROM within two different CAMHS we report the findings of two case-note audits: a baseline audit conducted in January 2011 and a re-audit conducted two years later in December 2012-February 2013.

RESULTS

The findings show an increase in both the single and repeated use of outcome measures from the time of the original audit, with repeated use (baseline and follow-up) of the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) scale increasing from 10% to 50% of cases. Re-audited case-notes contained more combined use of different outcome measures, with greater consensus on which measures to use. Outcome measures that were applicable across a wide range of clinical conditions were more likely to be used than symptom-specific measures, and measures that were completed by the clinician were found more often than measures completed by the service user.

CONCLUSIONS

The findings show a substantial improvement in the use of outcome measures within CAMHS. These increases in use were found across different service organisations which were subject to different types of local service priorities and drivers.