From 'reckless' to 'mindful' in the use of outcome data to inform service-level performance management: perspectives from child mental health

The extent to which child- and parent-report Revised Children's Anxiety and Depression Scale, short Mood and Feeling Questionnaire, Strength and Difficulty Questionnaire and child-report KIDSCREEN identify the same young people as at risk of mental health conditions

BACKGROUND

We rely heavily on cut-off points of brief measures of psychological distress in research and clinical practice to identify those at risk of mental health conditions; however, few studies have compared the performance of different scales.

AIM

To determine the extent to which the child- and parent-report Strength and Difficulty Questionnaire (SDQ), Revised Children's Anxiety and Depression Scale (RCADS), short Mood and Feeling Questionnaire (sMFQ) and child-report KIDSCREEN correlated and identified the same respondents above cut-off points and at risk of mental health conditions.

METHOD

A cross-sectional survey was conducted among 231 children aged 11-16 years and 289 parents who completed all the above measures administered via a mobile app, MyJournE, including the SDQ, RCADS and sMFQ.

RESULTS

The psychopathology measures identified similar proportions of young people as above the cut-off point and at risk of depression (child report 14.7% RCADS, 19.9% sMFQ, parent report 8.7% RCADS, 12.1% sMFQ), anxiety (child report 24.7% RCADS, 26.0% SDQ-Emotional subscale, parent report 20.1% RCADS, 26% SDQ-Emotional subscale) and child-report internalising problems (26.8% RCADS, 29.9% SDQ). Despite strong correlations between measures (child report 0.77-0.84 and parent report 0.70-0.80 between the SDQ, sMFQ and RCADS) and expected directions of correlation with KIDSCREEN and SDQ subscales, kappa values indicate moderate to substantial agreement between measures. Measures did not consistently identify the same children; half (n = 36, 46%) of those on child report and a third (n = 30, 37%) on parent report, scoring above the cut-off point for the SDQ-Emotional subscale, RCADS total or sMFQ, scored above the cut-off point on all of them. Only half (n = 46, 54%) of the children scored above the cut-off point on child report by the SDQ-Internalising and RCADS total scales.

CONCLUSION

This study highlights the risk of using a screening test to 'rule out' potential psychopathology. Screening tests should not be used diagnostically and are best used together with broad assessment.

Notes from the youth mental health field: Using movement towards goals as a potential indicator of service change and quality improvement

The aim of this paper is to report our notes from the field on using movement toward goals at an aggregate level as an inference of service effectiveness. Analysis of routinely collected data from UK youth mental health services was conducted (N = 8,172, age M = 13.8, 67% female, 32% male) to explore the impact of including goal-based outcome data in combined calculations of standardized measures based on the principles of reliable change ("measurable change"). Due to the broad nature of standardized measures, inferred validity becomes diluted in any team or service level aggregate analysis. To make inferences that are closer to the person's interpretation of their difficulties, we argue that Idiographic Patient Reported Outcome Measures (I-PROMs) counterbalance these limitations. This is supported by our findings. The measurable change metric is the first step towards enabling national analysis of aggregated I-PROMs. I-PROMs, supplemented by standardized measures should be used to consider service evaluation.

Monitoring and Measurement in Child and Adolescent Mental Health: It's about More than Just Symptoms

Routine outcome monitoring (ROM) provides information to practitioners and others providing healthcare support to demonstrate the impact of interventions and for service evaluation [...].

Measuring psychological outcomes in paediatric settings: Making outcomes meaningful using client-defined perspectives

There is a growing drive to develop and implement patient-reported outcome measures within paediatric health services, particularly for young people living with chronic health conditions; however, there is little consensus on how best to do this in meaningful ways within psychological services working alongside medical teams. This reflective commentary considers some of the challenges of collecting psychological outcome measures in paediatric services and considers alternative approaches to making outcome measurement meaningful. All measures have their limitations; however, they become meaningless if they are not used in meaningful and considered ways with young people. Client-defined outcome measurement, such as goal-based outcome measures, alongside other types of measurement, can capture outcomes of meaning to young people living with chronic health conditions, and can enable them to feed into a shared decision-making process.

Outcomes of inpatient psychiatric treatment for adolescents: A multiple perspectives evaluation

Adolescent inpatient psychiatric treatment was evaluated from the multiple perspectives of clinicians, young people and parents using standardised measures and goal-based outcomes (GBOs). The sample included cases ( N = 128) discharged from a London adolescent unit between April 2009 and December 2015. Measures were completed at admission and discharge, and change in ratings was analysed to assess treatment outcomes. Ratings of clinicians and young people on the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA) were compared. Adolescents demonstrated significant improvement across all measures from admission to discharge. Correlation between clinicians' and adolescents' HoNOSCA ratings was weak at admission ( r = .25) but stronger at discharge ( r = .63). Standardised effect sizes were larger for GBOs ( d = 1.73 and 3.16 for adolescent and clinician-rated goals, respectively) compared to all standardised measures ( d = 0.31-0.93). Improvement was observed across all measures of functioning and symptoms following inpatient treatment. Clinicians and young people developed better shared understanding of the problems from admission to discharge. GBOs are more sensitive to change compared to standardised measures and may be meaningfully adopted by inpatient units for routine outcome monitoring.

Innovations in Practice: Using clinician-rated outcomes to support improved service decision making in a Child and Adolescent Mental Health Service

BACKGROUND

Clinical outcomes are now routinely collected in most services. However, there is a need to make full use of the information collected in order to improve the use of limited Child and Adolescent Mental Health Service (CAMHS) resources. This paper describes a method of improving service decision making by making the interpretation of outcomes data accessible to frontline staff.

METHOD

Clinician ratings of child outcomes for 2876 cases were routinely collected between 2009 and 2013 in a single CAMHS service. These outcomes were assessed against seven 'point of entry' and 'treatment' variables.

RESULTS

Older children, referred by social care and presenting with multiple problems were associated with worse treatment outcomes than other service users. These findings were used to create an accessible Service Evaluation Chart of outcomes, to support discussion and decision making within the service. This process enabled the identification of a group who had particularly poor outcomes and resulted in consideration about service changes to try to improve outcomes for this specific group.

CONCLUSIONS

Providing clinicians with accessible information about outcomes promotes the likelihood of outcomes being used in service decision making.

Goal formulation and tracking in child mental health settings: when is it more likely and is it associated with satisfaction with care?

Goal formulation and tracking may support preference-based care. Little is known about the likelihood of goal formulation and tracking and associations with care satisfaction. Logistic and Poisson stepwise regressions were performed on clinical data for N = 3757 children from 32 services in the UK (M age = 11; SDage = 3.75; most common clinician-reported presenting problem was emotional problems = 55.6%). Regarding the likelihood of goal formulation, it was more likely for pre-schoolers, those with learning difficulties or those with both hyperactivity disorder and conduct disorder. Regarding the association between goal formulation and tracking and satisfaction with care, parents of children with goals information were more likely to report complete satisfaction by scoring at the maximum of the scale. Findings of the present research suggest that goal formulation and tracking may be an important part of patient satisfaction with care. Clinicians should be encouraged to consider goal formulation and tracking when it is clinically meaningful as a means of promoting collaborative practice.

Service-level variation, patient-level factors, and treatment outcome in those seen by child mental health services

Service comparison is a policy priority but is not without controversy. This paper aims to investigate the amount of service-level variation in outcomes in child mental health, whether it differed when examining outcomes unadjusted vs. adjusted for expected change over time, and which patient-level characteristics were associated with the difference observed between services. Multilevel regressions were used on N = 3256 young people (53% male, mean age 11.33 years) from 13 child mental health services. Outcome was measured using the parent-reported Strengths and Difficulties Questionnaire. The results showed there was 4-5% service-level variation in outcomes. Findings were broadly consistent across unadjusted vs. adjusted outcomes. Young people with autism or infrequent case characteristics (e.g., substance misuse) had greater risk of poor outcomes. Comparison of services with high proportions of young people with autism or infrequent case characteristics requiring specialist input needs particular caution as these young people may be at greater risk of poor outcomes.

Does parent-child agreement vary based on presenting problems? Results from a UK clinical sample

BACKGROUND

Discrepancies are often found between child and parent reports of child psychopathology, nevertheless the role of the child's presenting difficulties in relation to these is underexplored. This study investigates whether parent-child agreement on the conduct and emotional scales of the Strengths and Difficulties Questionnaire (SDQ) varied as a result of certain child characteristics, including the child's presenting problems to clinical services, age and gender.

METHODS

The UK-based sample consisted of 16,754 clinical records of children aged 11-17, the majority of which were female (57%) and White (76%). The dataset was provided by the Child Outcomes Research Consortium , which collects outcome measures from child services across the UK. Clinicians reported the child's presenting difficulties, and parents and children completed the SDQ.

RESULTS

Using correlation analysis, the main findings indicated that agreement varied as a result of the child's difficulties for reports of conduct problems, and this seemed to be related to the presence or absence of externalising difficulties in the child's presentation. This was not the case for reports of emotional difficulties. In addition, agreement was higher when reporting problems not consistent with the child's presentation; for instance, agreement on conduct problems was greater for children presenting with internalising problems. Lastly, the children's age and gender did not seem to have an impact on agreement.

CONCLUSIONS

These findings demonstrate that certain child presenting difficulties, and in particular conduct problems, may be related to informant agreement and need to be considered in clinical practice and research. Trial Registration This study was observational and as such did not require trial registration.

Measuring what matters to patients: Using goal content to inform measure choice and development

INTRODUCTION

Personalised care requires personalised outcomes and ways of feeding back clinically useful information to clinicians and practitioners, but it is not clear how to best personalise outcome measurement and feedback using existing standardised outcome measures.

METHOD

The constant comparison method of grounded theory was used to compare goal themes derived from goals set at the outset of therapy for 180 children aged between 4 and 17 years, visiting eight child and adolescent mental health services, to existing standardised outcome measures used as part of common national datasets.

RESULTS

In all, 20 out of 27 goal themes corresponded to items on at least one commonly used outcome measure.

DISCUSSION

Consideration of goal themes helped to identify potential relevant outcome measures. However, there were several goal themes that were not captured by items on standardised outcome measures. These seemed to be related to existential factors such as understanding, thinking about oneself and future planning.

CONCLUSION

This presents a powerful framework for how clinicians can use goals to help select a standardised outcome measure (where this is helpful) in addition to the use of a goal-based outcome measure and personalise choices. There may be areas not captured by standardised outcome measures that may be important for children and young people and which may only be currently captured in goal measurement. There is an indication that we may not be measuring what is important to children and young people. We may need to develop or look for new measures that capture these areas.

Learning from a Learning Collaboration: The CORC Approach to Combining Research, Evaluation and Practice in Child Mental Health

This paper outlines the experience of the Child Outcomes Research Consortium—formerly known as the CAMHS Outcomes Research Consortium; the named changed in 2014 in recognition of the widening scope of the work of the collaboration; a learning collaboration of service providers, funders, service user groups and researchers across the UK and beyond, jointly committed to collecting and using routinely collected outcome data to improve and enhance service provision and improve understanding of how best to help young people with mental health issues and their families.

IMPRoving Outcomes for children exposed to domestic ViolencE (IMPROVE): an evidence synthesis

Background

Exposure to domestic violence and abuse (DVA) during childhood and adolescence increases the risk of negative outcomes across the lifespan.

Objectives

To synthesise evidence on the clinical effectiveness, cost-effectiveness and acceptability of interventions for children exposed to DVA, with the aim of making recommendations for further research.

Design

(1) A systematic review of controlled trials of interventions; (2) a systematic review of qualitative studies of participant and professional experience of interventions; (3) a network meta-analysis (NMA) of controlled trials and cost-effectiveness analysis; (4) an overview of current UK provision of interventions; and (5) consultations with young people, parents, service providers and commissioners.

Settings

North America (11), the Netherlands (1) and Israel (1) for the systematic review of controlled trials of interventions; the USA (4) and the UK (1) for the systematic review of qualitative studies of participant and professional experience of interventions; and the UK for the overview of current UK provision of interventions and consultations with young people, parents, service providers and commissioners.

Participants

A total of 1345 children for the systematic review of controlled trials of interventions; 100 children, 202 parents and 39 professionals for the systematic review of qualitative studies of participant and professional experience of interventions; and 16 young people, six parents and 20 service providers and commissioners for the consultation with young people, parents, service providers and commissioners.

Interventions

Psychotherapeutic, advocacy, parenting skills and advocacy, psychoeducation, psychoeducation and advocacy, guided self-help.

Main outcome measures

Internalising symptoms and externalising behaviour, mood, depression symptoms and diagnosis, post-traumatic stress disorder symptoms and self-esteem for the systematic review of controlled trials of interventions and NMA; views about and experience of interventions for the systematic review of qualitative studies of participant and professional experience of interventions and consultations.

Data sources

MEDLINE, Cumulative Index to Nursing and Allied Health Literature, PsycINFO, EMBASE, Cochrane Central Register of Controlled Trials, Science Citation Index, Applied Social Sciences Index and Abstracts, International Bibliography of the Social Sciences, Social Services Abstracts, Social Care Online, Sociological Abstracts, Social Science Citation Index, World Health Organization trials portal and clinicaltrials.gov.

Review methods

A narrative review; a NMA and incremental cost-effectiveness analysis; and a qualitative synthesis.

Results

The evidence base on targeted interventions was small, with limited settings and types of interventions; children were mostly < 14 years of age, and there was an absence of comparative studies. The interventions evaluated in trials were mostly psychotherapeutic and psychoeducational interventions delivered to the non-abusive parent and child, usually based on the child’s exposure to DVA (not specific clinical or broader social needs). Qualitative studies largely focused on psychoeducational interventions, some of which included the abusive parent. The evidence for clinical effectiveness was as follows: 11 trials reported improvements in behavioural or mental health outcomes, with modest effect sizes but significant heterogeneity and high or unclear risk of bias. Psychoeducational group-based interventions delivered to the child were found to be more effective for improving mental health outcomes than other types of intervention. Interventions delivered to (non-abusive) parents and to children were most likely to be effective for improving behavioural outcomes. However, there is a large degree of uncertainty around comparisons, particularly with regard to mental health outcomes. In terms of evidence of cost-effectiveness, there were no economic studies of interventions. Cost-effectiveness was modelled on the basis of the NMA, estimating differences between types of interventions. The outcomes measured in trials were largely confined to children’s mental health and behavioural symptoms and disorders, although stakeholders’ concepts of success were broader, suggesting that a broader range of outcomes should be measured in trials. Group-based psychoeducational interventions delivered to children and non-abusive parents in parallel were largely acceptable to all stakeholders. There is limited evidence for the acceptability of other types of intervention. In terms of the UK evidence base and service delivery landscape, there were no UK-based trials, few qualitative studies and little widespread service evaluation. Most programmes are group-based psychoeducational interventions. However, the funding crisis in the DVA sector is significantly undermining programme delivery.

Conclusions

The evidence base regarding the acceptability, clinical effectiveness and cost-effectiveness of interventions to improve outcomes for children exposed to DVA is underdeveloped. There is an urgent need for more high-quality studies, particularly trials, that are designed to produce actionable, generalisable findings that can be implemented in real-world settings and that can inform decisions about which interventions to commission and scale. We suggest that there is a need to pause the development of new interventions and to focus on the systematic evaluation of existing programmes. With regard to the UK, we have identified three types of programme that could be justifiably prioritised for further study: psycho-education delivered to mothers and children, or children alone; parent skills training in combination with advocacy: and interventions involving the abusive parent/caregiver. We also suggest that there is need for key stakeholders to come together to explicitly identify and address the structural, practical and cultural barriers that may have hampered the development of the UK evidence base to date.

Future work recommendations

There is a need for well-designed, well-conducted and well-reported UK-based randomised controlled trials with cost-effectiveness analyses and nested qualitative studies. Development of consensus in the field about core outcome data sets is required. There is a need for further exploration of the acceptability and effectiveness of interventions for specific groups of children and young people (i.e. based on ethnicity, age, trauma exposure and clinical profile). There is also a need for an investigation of the context in which interventions are delivered, including organisational setting and the broader community context, and the evaluation of qualities, qualifications and disciplines of personnel delivering interventions. We recommend prioritisation of psychoeducational interventions and parent skills training delivered in combination with advocacy in the next phase of trials, and exploratory trials of interventions that engage both the abusive and the non-abusive parent.

Study registration

This study is registered as PROSPERO CRD42013004348 and PROSPERO CRD420130043489.

Funding

The National Institute for Health Research Public Health Research programme.

Considering harm and safety in youth mental health: a call for attention and action

The possibility of harm from mental health provision, and in particular harm from youth mental health provision, has been largely overlooked. We contend that if we continue to assume youth mental health services can do no harm, and all that is needed is more services, we continue to risk the possibility that the safety of children and young people is unintentionally compromised. We propose a three level framework for considering harm from youth mental health provision (1. ineffective engagement, 2. ineffective practice and 3. adverse events) and suggest how this framework could be used to support quality improvement in services.

Using Routine Outcome Measures to Provide Feedback at the Service Agency Level

The use of routine outcome measurement (ROM) to assess service effectiveness has been on the rise in mental health settings. However, there is a scarcity of information on the use of ROM data to provide feedback to teams of service providers. In this paper we review the existing literature to identify the principles that can guide the use of ROM data as feedback with the aim to improve quality of service provision in mental health settings. We present a pilot trial of 12 agencies participating in group feedback sessions. The guiding principles and core processes, the procedure and implementation in a pilot trial, lessons learned and future directions are discussed. Based on this experience we conclude that using ROM to implement group feedback among mental health stakeholders is feasible and can generate discussions and directions for improvement.

Horses for courses? A qualitative exploration of goals formulated in mental health settings by young people, parents, and clinicians

INTRODUCTION

This research sought to explore and categorise goals set by children and young people, parents/caregivers and jointly by a combination of children/young people, parents/caregivers and/or clinicians within mental health settings across the United Kingdom.

METHOD

Using a dataset of 441 goals formed at the outset of 180 treatment episodes (2007-2010) from UK child mental health services using the Goal-Based Outcomes tool, a grounded theory approach was taken, which built on previous research into child-rated goals to develop frameworks for parent and joint goal data which were then compared with the child goal data.

RESULTS

A total of 19 subthemes and four overarching themes were identified for parent goals. A total of 19 subthemes in five overarching themes were identified for joint goals. These were compared with 25 subthemes and three overarching themes for child goals. A comparison of subthemes between parent, child and joint goals demonstrated many consistencies, but also differences. Most commonly rated goals from children focused on coping with specific difficulties, personal growth and independence. Parent goals focused mainly on managing specific difficulties, parent-specific goals and improving self or life. Jointly negotiated goals focused on parent-specific goals, self-confidence and understanding, hopes for the future and managing specific problems.

CONCLUSION

The results suggest that goals may capture areas not captured by other normed outcome measures. In particular, goals may capture higher order, underlying factors, such as confidence, resilience, coping, and parenting factors that may not be explored by other measures. The differences across perspectives also link to existing literature suggesting a different focus on treatment based on perspectives and highlights the potential importance when jointly agreeing goals of ensuring the voice of the child/young person is heard and included in goal setting.

A systematic review of mental health outcome measures for young people aged 12 to 25 years

BACKGROUND

Mental health outcome measures are used to monitor the quality and effectiveness of mental health services. There is also a growing expectation for implementation of routine measurement and measures being used by clinicians as a feedback monitoring system to improve client outcomes. The recent focus in Australia and elsewhere targeting mental health services to young people aged 12-25 years has meant that outcome measures relevant to this age range are now needed. This is a shift from the traditional divide of child and adolescent services versus adult services with a transitioning age at 18 years. This systematic review is the first to examine mental health outcome measures that are appropriate for the 12 to 25 year age range.

METHODS

MEDLINE and PsychINFO databases were systematically searched to identify studies using mental health outcome measures with young people aged 12 to 25 years. The search strategy complied with the relevant sections of the PRISMA statement.

RESULTS

A total of 184 published articles were identified, covering 29 different outcome measures. The measures were organised into domains that consisted of eight measures of cognition and emotion, nine functioning measures, six quality of life measures, and six multidimensional mental health measures. No measures were designed specifically for young people aged 12 to 25 years and only two had been used by clinicians as a feedback monitoring system. Five measures had been used across the whole 12 to 25 year age range, in a range of mental health settings and were deemed most appropriate for this age group.

CONCLUSIONS

With changes to mental health service systems that increasingly focus on early intervention in adolescence and young adulthood, there is a need for outcome measures designed specifically for those aged 12 to 25 years. In particular, multidimensional measures that are clinically meaningful need to be developed to ensure quality and effectiveness in youth mental health. Additionally, outcome measures can be clinically useful when designed to be used within routine feedback monitoring systems.

Interpreting standardized and idiographic outcome measures in CAMHS: what does change mean and how does it relate to functioning and experience?

BACKGROUND

Patient Reported Outcome Measures (PROMs) are increasingly being used in Child and Adolescent Mental Health Services (CAMHS). The aim of this research was to explore change in standardized and idiographic outcome measures in CAMHS using naturalistic, routinely collected data.

METHOD

We explored change in psychosocial difficulties and impact on daily life as evaluated by a broad standardized measure, the Strengths and Difficulties Questionnaire (SDQ) (Goodman, 1997, J. Child Psychol. Psychiatry, 38, 581) and progress toward goals as evaluated by a personalized idiographic measure, the Goal Based Outcomes tool (GBO) (Law, 2011, Goals and Goal Based Outcomes (GBOs): some useful information. London: CAMHS Press) in a sample of N = 137 CAMHS attenders.

RESULTS

Psychosocial difficulties and impact on daily life showed less change over the course of treatment than progress toward goals in this study. Change in psychosocial difficulties and impact on daily life were also less strongly associated with change in clinician-reported functioning and satisfaction with care at time two than change in progress toward goals.

CONCLUSIONS

Findings of the present research may support previous studies in which service users and clinicians report that idiographic measures are more capable than standardized measures of capturing relevant change for individuals.

Comparison of indices of clinically meaningful change in child and adolescent mental health services: difference scores, reliable change, crossing clinical thresholds and 'added value' - an exploration using parent rated scores on the SDQ

BACKGROUND

Establishing what constitutes clinically significant change is important both for reviewing the function of services and for reflecting on individual clinical practice. A range of methods for assessing change exist, but it remains unclear which are best to use and under which circumstances.

METHOD

This paper reviews four indices of change [difference scores (DS), crossing clinical threshold (CCT), reliable change index (RCI) and added value scores (AVS)] drawing on outcome data for 9764 young people from child and adolescent mental health services across England.

RESULTS

Looking at DS, the t-test for time one to time two scores indicated a significant difference between baseline and follow up scores, with a standardised effect size of d = 0.40. AVS analysis resulted in a smaller effect size of 0.12. Analysis of those crossing the clinical threshold showed 21.2% of cases were classified as recovered, while 5.5% were classified as deteriorated. RCI identified 16.5% of cases as showing reliable improvement and 2.3% of cases as showing reliable deterioration. Across RCI and CCT 80.5% of the pairings were exact (i.e., identified in the same category using each method).

CONCLUSIONS

Findings indicate that the level of agreement across approaches is at least moderate; however, the estimated extent of change varied to some extent based on the index used. Each index may be appropriate for different contexts: CCT and RCI may be best suited to use for individual case review; whereas DS and AVS may be more appropriate for case-mix adjusted national reporting.

Measurement Issues: Review of four patient reported outcome measures: SDQ, RCADS, C/ORS and GBO - their strengths and limitations for clinical use and service evaluation

BACKGROUND

There is an international drive for routine use of Patient Reported Outcome Measures (PROMs) across all health services including in relation to Child and Adolescent Mental Health Services (CAMHS). A number of reviews have summarized the validity and reliability of well-being and mental health measures for children but there are fewer attempts to consider utility for routine use.

METHOD

This review considers four child self-report measures: the Strengths and Difficulties Questionnaire (SDQ), the Revised Child Anxiety and Depression Scale (RCADS), (Child) Outcomes Rating Scale (C/ORS) and Goals Based Outcomes (GBOs). It explores the strengths and limitations of each and considers how they can be used to support both clinical practice and service evaluation.

RESULTS

There is evidence for the clinical utility of RCADS, C/ORS and GBOs, although the utility of the SDQ as a feedback measure remains unclear. For service evaluation, the SDQ has the greatest evidence for norms making it useful for comparison and there is evidence that the RCADS may be the most sensitive to change of the measures reviewed; C/ORS has issues around ceiling effect, data error and data manipulation. More research is required around GBOs before their use for service evaluation can be determined.

CONCLUSIONS

In summary, these different measures may be viewed as complementary tools and determining the best way to make use of them severally and individually in clinical and community settings is a current focus for child mental health practitioners.