Identifying thresholds for clinically meaningful change among clients of drug and alcohol services using the Australian Treatment Outcomes Profile

AIMS

The Austraian Treatment Outcomes Profile (ATOP) is a brief clinical outcomes tool used widely in the Australian alcohol and other drugs treatment sector to monitor clients' substance use, health, wellbeing and clinical risk factors. It has demonstrated reliability and validity, and has recommended clinical cut-offs for assessing single-occasion client-rated health scores. This study determined clinically meaningful change thresholds for ATOP substance use and health and wellbeing variables for use by clinicians in monitoring client progress, and for quality improvement and service evaluation.

DESIGN, SETTING AND PARTICIPANTS

A framework for assessing clinically meaningful changes scores was developed by (1) calculating statistically reliable change thresholds using data-driven techniques with a reference sample of clinical ATOP data and (2) conducting a multi-disciplinary subject matter expert group to review the utility and validity of data-derived clinically meaningful change. The study was conducted within Outpatient Alcohol and Other Drug treatment services in New South Wales, Australia. The reference sample comprised 6100 ATOPs from clients at entry to public outpatient Alcohol and Other Drug treatment services; the subject matter expert group comprised 29 key stakeholders from the specialist alcohol and other drug treatment sector.

MEASUREMENTS AND FINDINGS

We used the Reliable Change Index method to calculate clinically meaningful change thresholds for ATOP variables. For substance use variables, a change of 30% in days of use in the last 28 (minimum 4 days) was the threshold for clinically meaningful change for substance use; for health and wellbeing variables, a change of 2 or more points in psychological health, physical health or quality of life scores (measured on 0-10 scales) was the minimum clinically meaningful change.

CONCLUSIONS

Clinically meaningful change thresholds have been proposed for Australian Treatment Outcomes Profile substance use and health and wellbeing items, based on statistical reliability and subject matter expert assessment. These will be used in the development of an outcomes metric for assessing change and assigning meaning in aggregated data for evaluation of services.

TEST-RETEST RELIABILITY AND RELIABLE CHANGE INDEX OF MOBILE APPLICATION NEUROCOGNITIVE TEST AMONG MIDDLE AND HIGH SCHOOL ATHLETES

CONTEXT

A mobile application neurocognitive assessment has been used in place of equipment intensive computerized neurocognitive assessment protocol. A previous study reported high to very high test-retest reliability of neurocognitive assessment using the mobile application in healthy adults, but no studies have examined test-retest reliability, reliable change indices (RCIs), and sex effect in middle school and high school populations when conducted 1 year apart.

OBJECTIVE

The purpose of this study was to examine the test-retest reliability and RCIs of baseline data collected at 2-time points approximately 1 year apart using a mobile application neurocognitive rest in middle school and high school athletes. The secondary purpose of the study was to investigate the sex difference in neurocognitive measures.

DESIGN

Cross-sectional study.

SETTING

Institutional.

PATIENTS OR OTHER PARTICIPANTS

172 middle school and high school healthy student-athletes (mean age=13.78±1.59 years old).

MAIN OUTCOME MEASURE(S)

Mobile application neurocognitive rest scores (reaction time, impulse control, inspection, and memory).

RESULTS

The result from the study demonstrated that neurocognitive measures had low test-retest reliability across a 1-year time period in middle and high school settings. Upon retesting, reaction time and inspection time improved significantly in both middle and high school athletes, and impulse control showed significant improvement in middle school athletes. More athletes in middle school showed more RCI improvements compared to high school athletes. While both males and females demonstrated improvements in neurocognitive measures throughout adolescence, males outperformed females on reaction time and impulse control.

CONCLUSIONS

Findings from the study indicate unacceptably low test-retest reliability of a mobile application neurocognitive test most likely due to cognitive development occurring throughout adolescence. Additionally, significant RCIs were noted. These naturally occurring improvements due to cognitive development could mask the post-concussion deficits. The findings warrant consideration of age and sex on the neurocognitive performance of middle and high school athletes.

Cognitive training via a mobile application to reduce some forms of body dissatisfaction in young females at high-risk for body image disorders: A randomized controlled trial

Body dissatisfaction has been related to Body Image Disorders (BIDs) such as Eating Disorders (EDs) and Body Dysmorphic Disorder (BDD). This study investigates the efficacy of a mHealth app in reducing body dissatisfaction, BDD/ED symptoms and related features. Ninety-five women who were identified as high-risk of developing BIDs (using the Structured Clinical Interview for DSM-5) were randomized into: immediate-use App (iApp group; n = 47) and delayed-use App (dApp group; n = 48). The iApp group started using the app at baseline for 16 days (T1). The dApp group started using the app at T1 for 16 days. Participants completed questionnaires at baseline (T0), 16 days from baseline (T1), and 32 days from baseline (T2). Repeated measure Multivariate Analysis of Variance (MANOVA) showed Group (iApp vs. dApp) × Time (T0 vs. T1) interactions indicating decrease in BDD symptoms and body dissatisfaction related to EDs in the iApp group at T1. The Reliable Change Index indicated changes on extreme body dissatisfaction/BDD symptoms for 34.74% of participants. Although preliminary, these findings highlight that a mHeatlh app might reduce BDD symptoms and body dissatisfaction related to EDs in women at high-risk for BIDs. Effects on ED symptoms and associated features seem more limited.

Utility of VOMS, SCAT3, and ImPACT Baseline Evaluations for Acute Concussion Identification in Collegiate Athletes: Findings From the NCAA-DoD Concussion Assessment, Research and Education (CARE) Consortium

BACKGROUND

The Vestibular/Ocular-Motor Screening (VOMS) is a valuable component of acute (<72 hours) sports-related concussion (SRC) assessments and is increasingly used with the Immediate Post-concussion Assessment and Cognitive Testing (ImPACT) instrument and the third edition of the Sport Concussion Assessment Tool (SCAT3). Research has suggested that VOMS acute postinjury scores are useful in identifying acute concussion. However, the utility of preseason baseline measurements to improve diagnostic accuracy remains ambiguous. To this end, there is a need to determine how reliable VOMS baseline assessments are across years and whether incorporating individuals' baseline performance improves diagnostic yield for acute concussions.

PURPOSE

To analyze VOMS, SCAT3, and ImPACT to evaluate the test-retest reliability of consecutive-year preseason baseline assessments to directly compare the diagnostic utility of these tools when incorporating baseline assessments versus using postinjury data alone to identify acute SRC.

STUDY DESIGN

Cohort study (diagnosis); Level of evidence, 2.

METHODS

Preseason and postinjury VOMS, SCAT3, ImPACT Post-Concussion Symptom Scale (PCSS), and ImPACT composite scores were analyzed for 3958 preseason (47.7% female) and 496 acute (≤48 hours) SRC (37.5% female) collegiate athlete evaluations in the National Collegiate Athletic Association-Department of Defense Concussion Assessment Research and Education Consortium. Descriptive statistics, Kolmogorov-Smirnov significance, and Cohen d effect size were calculated. Consecutive-year baseline reliability was evaluated for a subset of 447 athlete encounters using Pearson r, Cohen κ, Cohen d, and 2-way mixed intraclass correlation coefficients (ICCs). Wilcoxon signed rank tests were used to determine the statistical significance between population performances, and the 90% reliable change index (RCI) was calculated from the test-retest results. Preseason to postinjury change scores were then calculated from each tool's RCI. Finally, receiver operating characteristic (ROC) curve analyses were conducted, and DeLong method was used to compare the area under the curve (AUC) of raw postinjury scores versus change scores from preseason baseline assessments. Potential effects of sex, medical history (learning disorders or attention-deficit/hyperactivity disorder), and outlier data were also explored.

RESULTS

Effect sizes were large, and overall predictive utilities were clinically useful for postinjury VOMS Total (d = 2.44; AUC = 0.85), the SCAT3 Symptom Evaluation total severity score (d = 1.74; AUC = 0.82), and the ImPACT PCSS total severity score (d = 1.67; AUC = 0.80). Comparatively, effect sizes were small and predictive utilities were poor for Standardized Assessment of Concussion (SAC), modified Balance Error Scoring System (mBESS), and all ImPACT composites (d = 0.11-0.46; AUC = 0.48-0.59). Preseason baseline test-retest reliability was poor to moderate (r = 0.23-0.52; κ = 0.32-0.36; ICC = 0.36-0.68) for all assessments except ImPACT Visual Motion Sensitivity (r = 0.73; ICC = 0.85). Incorporating baseline scores for VOMS Total, SCAT3 (Symptom Evaluation, SAC, mBESS), ImPACT PCSS, or ImPACT composites did not significantly improve AUCs.

CONCLUSION

VOMS Total and symptom severity (SCAT3, PCSS) total scores had large effect sizes and clinically useful AUCs for identifying acute concussion. However, all tools demonstrated high within-patient test-retest variability, resulting in poor reliability. The findings in this sample of collegiate athletes suggest that incorporating baseline assessments does not significantly increase diagnostic yield for acute concussion.

Mapping Female Patients' Judgments of Satisfaction to Hypothetical Changes in Depression Symptom Severity

Within mental health, approaches to determine whether a patient experienced "meaningful" change from treatment have predominantly involved imposing thresholds on three types of metrics derived from assessments of symptom severity: end score (posttreatment score), absolute change (pre- minus posttreatment score), and proportion of change. However, none of these approaches have considered input from the consumer. This study examined correspondences between various reductions from pre- to posttreatment symptom severity levels and patients' judgments of satisfaction with change. Former or currently depressed patients were asked to provide judgments of their satisfaction reflected in vignettes that used descriptions from the Hamilton Rating Scale for Depression. Judgments from 108 female participants were fit using four metrics: end score, absolute change, proportion of change, and the combination of end score and absolute change. Akaike information criteria (AICs) and Akaike weights were used to determine the best-fitting model. Cutoffs were calculated for the five levels of satisfaction with change. Proportion of change best accounted for variation in the patients' ratings. For "slightly … ," "somewhat … ," "moderately … ," and "very … ," the proportions of reduction that corresponded with each of these ratings of satisfaction were, respectively: 17%, 39%, 62%, and 84%. Our a priori level of satisfaction (between "somewhat" and "moderately") corresponded to a 50% reduction in pretreatment severity. This study may provide services some insight into their female patients' satisfaction with change from treatment for depression using only the proportion of reduction from pretreatment severity. A similar procedure could be applied to other diagnostic groups, as well as other constructs that attend to the patient's perspective.

[Contract Focused Group Therapy - Results of a Second Evaluation]

Contract Focused Group Therapy - Results of a Second Evaluation Contract focused group therapy (AFoG) can be understood as a successful transformation of systemic solution focused mindsets into a group therapy concept for children and adolescents. The current evaluation study intents to investigate the effectiveness of AFoG. The sample size includes 72 children who attended AFoG and their parents. A pre-post-design was used to measure the perceived exposure of the child's problem behavior via caregiver report (CBCL 4-18) and self-report (YSR 11-18) at the beginning and three months after group therapy. Based on t-tests, the hypothesis that children and parents report significantly lower symptoms after finishing AFoG than at the beginning of the group therapy can be confirmed. Especially children and adolescents who were assessed as highly displaying several behavioral problems (t-values between 71-80), show a clear decrease in the perceived symptom load in both the self-report and the caregiver report. Moreover, the clinical significance of the decrease were examined by using the Reliable Change Index. In this sense, a positive effect is recorded. Further research inclusive a control group is needed to support the effectiveness of AFoG.

Approaches to denote treatment outcome: Clinical significance and clinical global impression compared

OBJECTIVES

The authors of a previous study proposed a statistically based approach to denote treatment outcome, translating pretest and posttest scores into clinically relevant categories, such as recovery and reliable improvement. We assessed the convergent validity of the Jacobson-Truax (JT) approach, using T-score based cutoff values, with ratings by an independent evaluator.

METHODS

Pretest and retest scores on the Brief Symptom Inventory (BSI) and clinical global impression improvement (CGI-I) ratings were collected repeatedly through routine outcome monitoring from 5,900 outpatients with common mental disorders. Data were collected in everyday practice in a large mental health care provider.

RESULTS

Continuous pretest-to-retest BSI change scores had a stronger association with CGI-I than the categorical variable based on JT. However, JT categorization and improvement according to CGI converged substantially with association indices (Somers' D) ranging from D = .50 to .56. Discordance was predominantly due to a more positive outcome according to JT than on CGI-I ratings.

CONCLUSION

Converting continuous outcome variables into clinically meaningful categories comes at the price of somewhat diminished concurrent validity with CGI-I. Nevertheless, support was found for the proposed threshold values for reliable change and recovery, and the outcome denoted in these terms corresponded with CGI improvement for most patients.

Severity of Substance Use Disorder: Utility as an Outcome in Clinical Settings

BACKGROUND

Some authors have pointed out the usefulness of the levels of substance use disorder (SUD) as a treatment outcome. However, in order to use this variable as an outcome measure, its impact needs to be addressed within a clinical context. The aim of this study was to analyze the sensitivity of SUD levels as a measure for detecting reliable changes and to make a comparison between the changes in SUD levels detected when using the number of criteria fulfilled and when using the reliable change index (RCI).

METHODS

The sample consisted of 206 (106 in follow-up) patients diagnosed with abuse/dependence on alcohol and cocaine, according to DSM-IV criteria. The Substance Dependence Severity Scale for DSM-5 was used to determine current alcohol use disorder (AUD) and cocaine use disorder (CUD). Number of DSM-5 criteria fulfilled and RCI were used to determine the change in SUD levels.

RESULTS

No association was found between adherence to/abandonment of treatment and AUD severity levels (χ²  = 7.029, p = 0.071) or CUD severity levels (χ²  = 2.044, p = 0.413). Statistical significant differences for levels of AUD (z = -3.870, p = 0.000) and CUD (z = -5.382, p = 0.000) were found between baseline assessment and follow-up. According to the number of DSM-5 criteria and RCI, the Kappa coefficient for the change in patient status (improved, worsened, or no change) was k = 0.61 for alcohol patients and k = 0.64 for cocaine patients. The "mild" category showed the greatest inconsistency between both procedures.

CONCLUSIONS

Levels of SUD are sensitive to the impact of treatment as measured by the difference between the baseline assessment and 3-month follow-up. However, conclusions differ according to whether the DSM-5 criteria or the RCI is applied.

A measurement-based pilot study in positive psychiatry

INTRODUCTION

Primary prevention of mental disorders is a major issue in positive psychiatry. Adjustment disorder is one of the very few discrete mental disorders linked to an etiological factor, namely psychosocial stressors given rise to a maladaptive reaction with a course of symptoms vanishing with the removal of the stressor. We have focused on a measurement-based method to prevent the development of an adjustment disorder.

AIM

The aim of this study has been to analyze from an ongoing Worklife Barometer Survey in which the World Health Organization Well-Being Scale (WHO-5) has been applied to prevent distress leading to an adjustment disorder.

METHODS

Persons identified with a decrease of 15 points in their repeatedly WHO-5 ratings over three months were through a brief psychological intervention by experienced psychologists. The Reliable Change Index (RCI) was used to determine the clinically meaningful change in the WHO-5 ratings.

RESULTS

Within the group who received the psychological intervention (N = 1338), 35% of the persons were identified by the RCI analysis to have developed a clinically reliable change in the WHO-5 at the time of the intervention. The remaining 65% of the persons obtained changes in the WHO-5 which might be considered as spontaneous fluctuations. In the month after the intervention, the persons with a clinically reliable change in the WHO-5 were restored.

CONCLUSION

In this measurement-based pilot study, the repeatedly WHO-5 ratings identified a group of persons with a clinically reliable change in WHO-5 and a clinically significant improvement after a brief psychological intervention.

Influence of the postoperative inflammatory response on cognitive decline in elderly patients undergoing on-pump cardiac surgery: a controlled, prospective observational study

Background

The role of non-infective inflammatory response (IR) in the aetiology of postoperative cognitive dysfunction (POCD) is still controversial. The aim of this controlled, prospective observational study was to assess the possible relationship between the grade of IR, defined by procalcitonin (PCT) changes, and development of POCD related to cardiac surgery.

Methods

Forty-two patients, who were ≥ 60 years of age and scheduled for elective cardiac surgery, were separated into the low inflammatory (LIR) and high inflammatory (HIR) response groups based on their PCT levels measured on the first postoperative day. A matched normative control group of 32 subjects was recruited from primary care practice. The PCT and C-reactive protein (CRP) levels were monitored daily during the first five postoperative days. The cognitive function and mood state were preoperatively tested with a set of five neurocognitive tests and two mood inventories and at the seventh postoperative day. The Reliable Change Index modified for practice (RCIp) using data from normative controls was applied to determine the significant decline in test performance.

Results

The LIR (n = 20) and HIR (n = 22) groups differed significantly in the PCT (p < 0.001) but not in the CRP time courses. The incidence of POCD at the first postoperative week was 35.7% in the cohort. The LIR and HIR groups did not vary in the RCIp Z scores of neurocognitive tests and frequencies of POCD (7 vs 8 cases, respectively, p > 0.05). Additionally, there was no difference in the mood states, anxiety levels and perioperative parameters known to influence the development of POCD.

Conclusions

In this study, the magnitude of the non-infective inflammatory response generated by on-pump cardiac surgery did not influence the development of POCD in the early postoperative period in elderly patients.

Normative Data on the Beck Depression Inventory--Second Edition (BDI-II) in College Students

OBJECTIVE

This study was conducted to provide normative data on the Beck Depression Inventory--Second Edition (BDI-II) in college students.

METHOD

Data were obtained from 15,233 college students drawn from 17 universities in the United States, weighted to match the gender and race/ethnicity of enrollment in degree-granting institutions.

RESULTS

Descriptive statistics, point prevalence of individuals exceeding cutoff scores, and mean differences by gender and race/ethnicity were provided. Because the distribution of BDI-II scores was not normal, percentile ranks for raw scores were provided for the total sample and separately by gender and race/ethnicity for the total sample and by race/ethnicity for men and women. Normative data were used to calculate the Reliable Change Index on the BDI-II for college students.

CONCLUSION

Because the distribution of BDI-II scores demonstrated significant skewness and non-normal kurtosis, percentile ranks are important to consider in interpreting scores on the measure, in addition to descriptive statistics.

Neurocognitive test performance and symptom reporting in cheerleaders with concussions

OBJECTIVE

To evaluate neurocognitive test results and symptom reporting after sports-related concussion in a group of female cheerleaders.

STUDY DESIGN

Junior and senior high school female cheerleaders (n = 138) underwent preparticipation baseline testing and repeated the ImPACT (Immediate Postconcussion Assessment and Cognitive Testing) evaluation within 7 days of concussive injury (range, 0-7 days; mean, 3.9 days). Postinjury neurocognitive and symptom scores were compared with preinjury (baseline) scores. "Abnormal" test performance was determined statistically using Reliable Change Index scores and self-reported symptoms. Main outcome variables included the composite scores indices from the ImPACT test, as well as symptoms reported by participants. Preinjury baseline and postinjury test results were compared using MANOVA.

RESULTS

As a group, cheerleaders with concussion evaluated within 7 days of injury performed poorly on the ImPACT test battery relative to their own baseline (F = 6.5; P = .00). In addition, 61% of the cheerleaders with concussions reported an increase in symptoms compared with baseline. The groups did not differ significantly by position on the squad (F = 0.37; P = .96). Of the group of cheerleaders who did not report increased symptoms at the time of postinjury evaluation, 37% had at least 1 abnormal ImPACT composite score result, suggesting some residual cognitive decline compared with baseline.

CONCLUSION

The diagnosis and management of concussion in cheerleaders should not consist solely of self-reported symptoms. Neurocognitive test results represent an important component of the evaluation process and may identify athletes with residual neurocognitive deficits who report being clinically asymptomatic.