Measurement Issues: Large effect sizes do not mean most people get better - clinical significance and the importance of individual results

Ethnic differences in treatment outcome for children and young people accessing mental health support

Children and Young People (CYP) from minoritized ethnic backgrounds experience structural inequalities in Children and Young People's Mental Health Settings (CYPMHS). This mixed methods study explores whether CYP's ethnicity is associated with their treatment outcomes (operationalised as 'measurable change') from CYPMHS. A multilevel multi-nominal regression analysis, controlling for age, gender, referral source, presenting difficulty, case closure reason, suggests that CYP from Asian backgrounds (OR = 0.82, CI [0.70, 0.96]) and Mixed-race (odds ratio (OR) = 0.80; 95% CI [0.69, 0.92]) are less likely to report measurable improvement in mental health difficulties compared to White British CYP. Three themes from a thematic analysis of semi-structured interviews with 15 CYP from minoritized ethnic backgrounds focused on views and experiences of ending mental health support are also presented. CYP view personalised support and the right therapist as conducive to good endings and valued a range of outcomes pertaining to empowerment. Experiences of stigma and inequalities may begin to explain the less positive outcomes experienced by Asian and Mixed-race CYP found in the regression analysis. The implications of these findings and future areas of research are suggested.

PEERS® for Preschoolers preliminary outcomes and predictors of treatment response

PEERS® for Preschoolers (P4P) is a social skills group program for young autistic children and their caregivers, which provides everyday tools for interacting and communicating with others. Twenty-two caregiver-child dyads participated and completed pre-treatment, post-treatment, and follow-up measures (4-16 weeks after). Using single-subject analyses to examine social skills, 60% demonstrated post-treatment improvement, and 53.85% demonstrated follow-up improvement. Regarding a secondary outcome of behavioral difficulties, 33.33% demonstrated post-treatment reduction, and 7.69% demonstrated follow-up reduction. Using regressions, autistic traits predicted outcomes; fewer social communication difficulties predicted both greater social skills and fewer behavioral difficulties at post-treatment, while fewer repetitive behaviors predicted fewer post-treatment and follow-up behavioral difficulties. These results preliminarily demonstrate the benefits of P4P and how autistic traits may impact P4P outcomes.

How much is enough? Considering minimally important change in youth mental health outcomes

To make decisions in mental health care, service users, clinicians, and administrators need to make sense of research findings. Unfortunately, study results are often presented as raw questionnaire scores at different time points and regression coefficients, which are difficult to interpret with regards to their clinical meaning. Other commonly reported treatment outcome indicators in clinical trials or meta-analyses do not convey whether a given change score would make a noticeable difference to service users. There is an urgent need to improve the interpretability and relevance of outcome indicators in youth mental health (aged 12-24 years), in which shared decision making and person-centred care are cornerstones of an ongoing global transformation of care. In this Personal View, we make a case for considering minimally important change (MIC) as a meaningful, accessible, and user-centred outcome indicator. We discuss what the MIC represents, how it is calculated, and how it can be implemented in dialogues between clinician and researcher, and between youth and clinician. We outline how use of the MIC could enhance reporting in clinical trials, meta-analyses, clinical practice guidelines, and measurement-based care. Finally, we identify current methodological challenges around estimating the MIC and areas for future research. Efforts to select outcome domains and valid measurement instruments that resonate with youth, families, and clinicians have increased in the past 5 years. In this context, now is the time to define demarcations of changes in outcome scores that are clinically relevant, and meaningful to youth and families. Through the use of MIC, youth-centred outcome measurement, analysis, and reporting would support youth-centred therapeutic decision making.

Adapting the multilevel model for estimation of the reliable change index (RCI) with multiple timepoints and multiple sources of error

OBJECTIVE

One of the primary tools in the assessment of individual-level patient outcomes is Jacobson and Truax, (1991's) Reliable Change Index (RCI). Recent efforts to optimize the RCI have revolved around three issues: (a) extending the RCI beyond two timepoints, (b) estimating the RCI using scale scores from item response theory or factor analysis and (c) estimation of person- and time-specific standard errors of measurement.

METHOD

We present an adaptation of a two-stage procedure, a measurement error-corrected multilevel model, as a tool for RCI estimation (with accompanying Statistical Analysis System syntax). Using DASS-21 data from a community-based mental health center (N = 379), we illustrate the potential for the model as unifying framework for simultaneously addressing all three limitations in modeling individual-level RCI estimates.

RESULTS

Compared to the optimal-fitting RCI model (moderated nonlinear factor analysis scoring with measurement error correction), an RCI model that uses DASS-21 total scores produced errors in RCI inferences in 50.8% of patients; this was largely driven by overestimation of the proportion of patients with statistically significant improvement.

CONCLUSION

Estimation of the RCI can now be enhanced by the use of latent variables, person- and time-specific measurement errors, and multiple timepoints.

Online Support and Intervention (OSI) for child anxiety: a case series within routine clinical practice

BACKGROUND

Online treatments for child anxiety offer a potentially cost-effective and non-stigmatizing means to widen access to evidence-based treatments and meet the increasing demand on services; however, uptake in routine clinical practice remains a challenge. This study conducted an initial evaluation of the clinical effectiveness, feasibility and acceptability of OSI (Online Support and Intervention for child anxiety) within clinical practice. OSI is a co-designed online therapist-supported, parent-led CBT treatment for pre-adolescent children with anxiety problems.

METHOD

This case series was part of routine service evaluation in a clinic in England where families were offered OSI to treat a primary anxiety difficulty among 7- to 12-year-old children; 24 families were offered OSI, and 23 took it up. Measures of anxiety symptomatology, functional impairment and progress towards therapeutic goals were taken at pre-treatment, post-treatment and 4-week follow-up. Treatment satisfaction and engagement were also measured throughout the intervention.

RESULTS

Mean anxiety symptoms significantly improved to below the clinical cut-off post-treatment, with further reduction at follow-up. Functional impairment also significantly improved and significant progress was made towards treatment goals. The majority of children showed reliable change in anxiety symptoms and reliable recovery by follow-up, and were discharged without needing further treatment for anxiety. Uptake, adherence and engagement in OSI were excellent, and parents reported high levels of satisfaction with the treatment.

CONCLUSIONS

We have provided initial evidence that OSI is feasible, acceptable to families, and appears to be associated with good outcomes within routine clinical practice.

Are We Comparing Apples with Oranges? Assessing Improvement Across Symptoms, Functioning, and Goal Progress for Adolescent Anxiety and Depression

Strategies for comparing routinely collected outcome data across services or systems include focusing on a common indicator (e.g., symptom change) or aggregating results from different measures or outcomes into a comparable core metric. The implications of either approach for judging treatment success are not fully understood. This study drew on naturalistic outcome data from 1641 adolescents with moderate or severe anxiety and/or depression symptoms who received routine specialist care across 60 mental health services in England. The study compared rates of meaningful improvement between the domains of internalizing symptoms, functioning, and progress towards self-defined goals. Consistent cross-domain improvement was observed in only 15.6% of cases. Close to one in four (24.0%) young people with reliably improved symptoms reported no reliable improvement in functioning. Inversely, one in three (34.8%) young people reported meaningful goal progress but no reliable symptom improvement. Monitoring systems that focus exclusively on symptom change risk over- or under-estimating actual impact, while aggregating different outcomes into a single metric can mask informative differences in the number and type of outcomes showing improvement. A move towards harmonized outcome measurement approaches across multiple domains is needed to ensure fair and meaningful comparisons.

BED-online: Acceptance and efficacy of an internet-based treatment for binge-eating disorder: A randomized clinical trial including waitlist conditions

Objective

Internet‐based guided self‐help (GSH) programs increase accessibility and utilization of evidence‐based treatments in binge‐eating disorder (BED). We evaluated acceptance and short as well as long‐term efficacy of our 8‐session internet‐based GSH program in a randomized clinical trial with an immediate treatment group, and two waitlist control groups, which differed with respect to whether patients received positive expectation induction during waiting or not.

Method

Sixty‐three patients (87% female, mean age 37.2 years) followed the eight‐session guided cognitive‐behavioural internet‐based program and three booster sessions in a randomized clinical trial design including an immediate treatment and two waitlist control conditions. Outcomes were treatment acceptance, number of weekly binge‐eating episodes, eating disorder pathology, depressiveness, and level of psychosocial functioning.

Results

Treatment satisfaction was high, even though 27% of all patients dropped out during the active treatment and 9.5% during the follow‐up period of 6 months. The treatment, in contrast to the waiting conditions, led to a significant reduction of weekly binge‐eating episodes from 3.4 to 1.7 with no apparent rebound effect during follow‐up. All other outcomes improved as well during active treatment. Email‐based positive expectation induction during waiting period prior to the treatment did not have an additional beneficial effect on the temporal course and thus treatment success, of binge episodes in this study.

Conclusion

This short internet‐based program was clearly accepted and highly effective regarding core features of BED. Dropout rates were higher in the active and lower in the follow‐up period. Positive expectations did not have an impact on treatment effects.

The present internet‐based guided self‐help program adds to the existing research regarding online treatment of binge‐eating disorder and is currently one of the two existing validated programs available in German language. It is based on an established cognitive‐behavioural treatment approach, shows high acceptance by patients and high efficacy after eight guided online sessions, thereby representing the shortest duration of currently evaluated treatments

During the internet‐based therapy, the number of weekly binge‐eating episodes, depressive symptoms, eating disorder pathology as well as impairments in psychosocial functioning all significantly decreased. These positive effects were maintained during follow‐up (6 months). Abstainer rate (no binge‐eating during last month) continued to increase during follow‐up with booster sessions

An email‐based pre‐treatment positive expectation induction did not alter the temporal course and thus treatment success, of binge episodes

Putting the Patient Back in Clinical Significance: Moderated Nonlinear Factor Analysis for Estimating Clinically Significant Change in Treatment for Posttraumatic Stress Disorder

The present study introduced a modernized approach to Jacobson and Truax's (1991) methods of estimating treatment effects on individual-level (a) movement from the clinical to the normative range and (b) reliable change on posttraumatic stress disorder (PTSD) severity. Participants were 450 trauma-exposed women (M age = 39.2 years, SD = 8.9, range: 18-65 years) who presented to seven geographically diverse community mental health and substance use treatment centers. Data from 53 of these women, none of whom met the criteria for full or subthreshold PTSD, were used to establish the normative range. Using moderated nonlinear factor analysis (MNLFA) scale scoring, which weights symptoms by their clinical relevance, a significantly larger proportion of participants moved into the normative range for PTSD severity scores and/or exhibited reliable changes after treatment compared to the same individuals' movement when using symptom counts. Further, approximately 24% of the participants showed discrepant judgments on reliable change indices (RCI) between MNLFA scores and symptom counts, likely due to the false assumption that the standard error of measurement is equal for all levels of underlying PTSD severity when estimating RCIs with symptom counts. An MNLFA approach to estimating underlying PTSD severity can provide clinically meaningful information about individual-level change without the de facto assumption that PTSD symptoms have equivalent weight. Study implications are discussed with regard to a joint emphasis on (a) measurement models that highlight differential symptom weighting and (b) treatment-arm differences in individual-level outcomes rather than the current overemphasis of treatment-arm differences on group-averaged trajectories.

A Critical Review of Effective Child Mass Trauma Interventions: What We Know and Do Not Know from the Evidence

Over the last 20 years, numerous interventions have been developed and evaluated for use with children exposed to mass trauma with six publications reporting meta-analyses of randomized controlled trials of child mass trauma interventions using inactive controls to examine intervention effects on posttraumatic stress, depression, anxiety, and functional impairment. The current report reviews the results of these meta-analytic studies to examine the status of the evidence for child mass trauma mental health interventions and to evaluate potential moderators of intervention effect and implications for practice. The meta-analyses reviewed for the current report revealed a small to medium overall effect of interventions on posttraumatic stress, a non-statistically significant to small overall effect on depression, a non-statistically significant overall effect on anxiety, and a small overall effect on functional impairment. The subgroup analyses suggest that interventions should be matched to the populations being served and to the context. Additional research is needed to tailor future interventions to further address outcomes other than posttraumatic stress including depression, anxiety, and functional impairment.

Systematic Review and Meta-analysis: Outcomes of Routine Specialist Mental Health Care for Young People With Depression and/or Anxiety

OBJECTIVE

Depression and anxiety are the most prevalent mental health problems in youth, yet almost nothing is known about what outcomes are to be expected at the individual level following routine treatment. This paper sets out to address this gap by undertaking a systematic review of outcomes following treatment as usual (TAU) with a particular focus on individual-level outcomes.

METHOD

MEDLINE, Embase and PsycInfo were searched for articles published between 1980 and January 2019 that assessed TAU outcomes for youth depression and anxiety accessing specialist mental health care. Meta-analysis considered change at both group-level pre-post effect size (ES) and individual-level recovery, reliable change, and reliable recovery. Temporal analysis considered stability of primary and secondary outcomes over time. Subgroup analysis considered the moderating effect of informant; presenting problem; study design; study year; mean age of youth; use of medication; intervention dosage and type of treatment offered on outcomes. A protocol was preregistered on PROSPERO (CRD42017063914).

RESULTS

Initial screening of 6,350 publications resulted in 38 that met the inclusion criteria, and that were subsequently included in meta-analyses. This resulted in a final full pooled sample of 11,739 young people (61% of whom were female, mean age 13.8 years). The pre-post ES (Hedges' g) at first/final outcome (13/26 weeks) was -0.74/-0.87. The individual-level change on measures of self-report was 38% reliable improvement, 44% no reliable change, and 6% reliable deterioration. Outcomes varied according to moderators, informant, problem type and dosage.

CONCLUSION

Poor data quantity and quality are limitations, but this is the first study that indicates likely rates of reliable improvement for those accessing TAU. We propose the need for improved reporting of both individual-level metrics and details of TAU to enable greater understanding of likely current outcomes from routine care for youths with depression and anxiety in order to allow the potential for further improvement of impact.

Evaluation of reliable improvement rates in depression and anxiety at the end of treatment in adolescents

BACKGROUND

Literature has focused on effect sizes rather than individual-level improvement rates to determine how effectively services address burgeoning numbers of adolescents with anxiety and depression.AimsTo consider how many adolescents report reliable improvement in anxiety, depression and comorbid depression and anxiety by end of treatment.

METHOD

The primary outcome was reliable improvement (i.e. change greater than likely the result of measurement error) in self-reported anxiety and depression for N = 4464 adolescents (mean age 14.5 years, s.d. = 1.9; 75% female; 61% White) seen in specialist mental health services in England.

RESULTS

In total, 53% of those with anxiety, 44% with depression, and 35% with comorbid depression and anxiety showed reliable improvement.

CONCLUSIONS

Improvement rates were higher than previously reported, but lower than generally used in advice to the public. There may be a need to set more realistic expectations, including with young people who seek help.Declaration of interestAll authors were involved in the programme of service transformation that this report draws on. M.W. led the outcomes and evaluation group that agreed the approach to measurement used in the initiative.