Reliable Change on Memory Tests is Common in Healthy Children and Adolescents

Test-Retest Reliability and Reliable Change on the NIH Toolbox Cognition Battery

OBJECTIVE

Researchers and practitioners can detect cognitive improvement or decline within a single examinee by applying a reliable change methodology. This study examined reliable change through test-retest data from the English-language National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) normative sample.

METHOD

Participants included adults (n = 138; age: M ± SD = 54.8 ± 20.0, range: 18-85; 51.4% men; 68.1% White) who completed test-retest assessments about a week apart on five fluid cognition tests, providing raw scores, age-adjusted standard scores (SS), and demographic-adjusted T-scores (T).

RESULTS

The Fluid Cognition Composite (SS: ICC = 0.87; T-score: ICC = 0.84) and the five fluid cognition tests had good test-retest reliability (SS: ICC range = 0.66-0.85; T-score: ICC range = 0.64-0.86). The lower and upper bounds of 70%, 80%, and 90% confidence intervals (CIs) were calculated around change scores, which serve as cutoffs for determining reliable change. Using T-scores, 90% CI, and adjustment for practice effects, 32.3% declined on one or more tests, 9.7% declined on two or more tests, 36.6% improved on one or more tests, and 5.4% improved on two or more tests.

CONCLUSIONS

It was common for participants to show reliable change on at least one test score, but not two or more test scores. Per an 80% CI, test-retest difference scores beyond these cutoffs would indicate reliable change: Dimensional Change Card Sort (SS ≥ 14/T ≥ 10), Flanker (SS ≥ 12/T ≥ 8), List Sorting (SS ≥ 14/T ≥ 10), Picture Sequence Memory (SS ≥ 19/T ≥ 13), Pattern Comparison (SS ≥ 11/T ≥ 8), and Fluid Cognition Composite (SS ≥ 10/T ≥ 7). The reliable change cutoffs could be applied in research or practice to detect within-person change in fluid cognition at the individual level.

Recent Advances in Neuropsychological Test Interpretation for Clinical Practice

Much attention in the field of clinical neuropsychology has focused on adapting to the modern healthcare environment by advancing telehealth and promoting technological innovation in assessment. Perhaps as important (but less discussed) are advances in the development and interpretation of normative neuropsychological test data. These techniques can yield improvement in diagnostic decision-making and treatment planning with little additional cost. Brooks and colleagues (Can Psychol 50: 196-209, 2009) eloquently summarized best practices in normative data creation and interpretation, providing a practical overview of norm development, measurement error, the base rates of low scores, and methods for assessing change. Since the publication of this seminal work, there have been several important advances in research on development and interpretation of normative neuropsychological test data, which may be less familiar to the practicing clinician. Specifically, we provide a review of the literature on regression-based normed scores, item response theory, multivariate base rates, summary/factor scores, cognitive intraindividual variability, and measuring change over time. For each topic, we include (1) an overview of the method, (2) a rapid review of the recent literature, (3) a relevant case example, and (4) a discussion of limitations and controversies. Our goal was to provide a primer for use of normative neuropsychological test data in neuropsychological practice.

Consensus-Based Evaluation of Outcome Measures in Pediatric Stroke Care: A Toolkit

Following a pediatric stroke, outcome measures selected for monitoring functional recovery and development vary widely. We sought to develop a toolkit of outcome measures that are currently available to clinicians, possess strong psychometric properties, and are feasible for use within clinical settings. A multidisciplinary group of clinicians and scientists from the International Pediatric Stroke Organization comprehensively reviewed the quality of measures in multiple domains described in pediatric stroke populations including global performance, motor and cognitive function, language, quality of life, and behavior and adaptive functioning. The quality of each measure was evaluated using guidelines focused on responsiveness and sensitivity, reliability, validity, feasibility, and predictive utility. A total of 48 outcome measures were included and were rated by experts based on the available evidence within the literature supporting the strengths of their psychometric properties and practical use. Only three measures were found to be validated for use in pediatric stroke: the Pediatric Stroke Outcome Measure, the Pediatric Stroke Recurrence and Recovery Questionnaire, and the Pediatric Stroke Quality of Life Measure. However, multiple additional measures were deemed to have good psychometric properties and acceptable utility for assessing pediatric stroke outcomes. Strengths and weaknesses of commonly used measures including feasibility are highlighted to guide evidence-based and practicable outcome measure selection. Improving the coherence of outcome assessment will facilitate comparison of studies and enhance research and clinical care in children with stroke. Further work is urgently needed to close the gap and validate measures across all clinically significant domains in the pediatric stroke population.

Reliable cognitive change following unilateral deep brain stimulation in essential tremor

Objective: This retrospective analysis assessed regression-based reliable change (RC) of cognition in a sample of essential tremor (ET) patients who underwent unilateral deep brain stimulation of the ventral intermediate nucleus of the thalamus (VIM-DBS).Method: Thirty patients (mean age at pre-evaluation = 70.4 ± 6.3 years) underwent neuropsychological evaluation pre- and post-unilateral VIM-DBS placement (mean time between pre and post-evaluation = 13.1 ± 4.0 months). Paired samples t-tests and RC analyses were employed.Results: No significant within-group differences were observed when cognitive scores were compared between evaluations. The vast majority of patients demonstrated stability across pre-and post-surgical evaluations (i.e. 29 out of 30); however, those with high-risk co-morbid medical conditions may be vulnerable to post-surgical cognitive decline as indicated by RC measures.Conclusions: The use of regression-based RC indices to assess individual cognitive changes between pre and post-surgical evaluations control for systematic and measurement errors that can occur over repeated evaluations, and may be able to identify cognitive changes that evade detection in traditional within-group comparisons.

Effects of Rhythmic Interventions on Cognitive Abilities in Parkinson's Disease

OBJECTIVE

The objective of this study is to compare 2 different rhythmic, high-intensive interventions, that is, rhythmic speech-language therapy (rSLT) versus rhythmic balance-mobility training (rBMT), against a no-therapy (NT) condition in patients with Parkinson's disease and against healthy controls (HCs) with regard to the change in or enhancement of cognitive abilities.

METHODS

The 4 groups (rSLT: N = 16; rBMT: N = 10; NT: N = 18; and HC: N = 17) were matched for age, sex, and educational level and were tested in 6 cognitive domains: working memory, executive function, visuo-construction, episodic memory, attention, and word retrieval. Assessments took place at baseline, at 4 weeks (T1), and at 6 months (T2). Rhythmic interventions were provided 3 times per week for 4 weeks in total. To analyze true intervention effects between groups and across time, statistical analyses included reliable change index. Intergroup differences were assessed with multivariate assessment of variance, while differences within groups were assessed with 95% confidence intervals of mean difference.

RESULTS

The rSLT improved working memory and word retrieval (p < 0.05), possibly a beneficial transfer effect of the training method per se. In contrast, the NT group worsened in phonemic and semantic shifting (p < 0.01). Observed improvements in flexibility and in episodic memory in the HC may be linked to training effects of retesting.

CONCLUSIONS

Rhythmic cues are resistant to neurodegeneration and have a strong motivating factor. As thus, these may facilitate high-intensive and demanding training. Although both trainings were superior to NT, the improvement of cognitive abilities depends on the specific training method. Further, therapy may be more effective when delivered by a therapist rather than by an impersonal computer program.

The Pair Test: A computerised measure of learning and memory

There is increasing interest in the assessment of learning and memory in typically developing children as well as in children with neurodevelopmental disorders. However, neuropsychological assessments have been hampered by the dearth of standardised tests that enable direct comparison between distinct memory processes or between types of stimulus materials. We developed a tablet-based paired-associate learning paradigm, the Pair Test, based on neurocognitive models of learning and memory. The aims are to (i) establish the utility of this novel memory tool for use with children across a wide age range, and (ii) examine test validity, reliability and reproducibility of the construct. The convergent validity of the test was found to be adequate, and higher test reliability was shown for the Pair Test compared to standardised measures. Moderate test-retest reproducibility was shown, despite a long time interval between sessions (14 months). Moreover, the Pair Test is able to capture developmental changes in memory, and can therefore chart the developmental trajectory of memory and learning functions across childhood and adolescence. Finally, we used this novel instrument to acquire normative data from 130 typically developing children, aged 8-18 years. Age-stratified normative data are provided for learning, delayed recall and delayed recognition, for measures of verbal and non-verbal memory. The Pair Test thus provides measures of learning and memory accounting for encoding, consolidation and retrieval processes. As such, the standardised test results can be used to determine the status of learning and memory in healthy children, and also to identify deficits in paediatric patients at risk of damage to the neural network underlying mnemonic functions.

Projected Retained Ability Score (PRAS): A New Methodology for Quantifying Absolute Change in Norm-Based Psychological Test Scores Over Time

A limitation of norm-based ability test scores is that they can only be used to evaluate relative change (compared with change in the norm sample), as opposed to absolute (raw) change in performance from Time 1 to Time 2. To address this limitation, a novel method (Projected Retained Ability Score [PRAS]) was developed to characterize absolute change in norm-based ability test scores. The PRAS method was applied to Differential Ability Scales®–Second Edition (DAS-II) General Conceptual Ability (GCA) scores in three cases of children with the neurodegenerative condition mucopolysaccharidosis type II (MPS II) who were assessed at two visits, 16 to 23 months apart. Although all three cases showed declines in norm-based GCA scores, the PRAS method revealed differences in absolute change in performance. The PRAS method allows for differentiation of slower-than-average improvement or stabilization versus deterioration of cognitive ability when norm-based scores decline from Time 1 to Time 2.

Multivariate Base Rates of Low Scores on Tests of Learning and Memory Among Latino Adult Populations

OBJECTIVE

To determine the prevalence of low scores for two neuropsychological tests with five total scores that evaluate learning and memory functions.

METHOD

N = 5402 healthy adults from 11 countries in Latin America and the commonwealth of Puerto Rico were administered the Rey-Osterrieth Complex Figure (ROCF) and the Hopkins Verbal Learning Test (HVLT-R). Two-thirds of the participants were women, and the average age was 53.5 ± 20.0 years. Z-scores were calculated for ROCF Copy and Memory scores and HVLT-R Total Recall, Delayed Recall, and Recognition scores, adjusting for age, age2, sex, education, and interaction variables if significant for the given country. Each Z-score was converted to a percentile for each of the five subtest scores. Each participant was categorized based on his/her number of low scoring tests in specific percentile cutoff groups (25th, 16th, 10th, 5th, and 2nd).

RESULTS

Between 57.3% (El Salvador) and 64.6% (Bolivia) of the sample scored below the 25th percentile on at least one of the five scores. Between 27.1% (El Salvador) and 33.9% (Puerto Rico) scored below the 10th percentile on at least one of the five subtests. Between 5.9% (Chile, El Salvador, Peru) and 10.3% (Argentina) scored below the 2nd percentile on at least one of the five scores.

CONCLUSIONS

Results are consistent with other studies that found that low scores are common when multiple neuropsychological outcomes are evaluated in healthy individuals. Clinicians should consider the higher probability of low scores when evaluating learning and memory using various sets of scores to reduce false-positive diagnoses of cognitive deficits.

Multivariate base rates for the assessment of executive functioning among children and adolescents

This study provides observed base rates of low executive functioning test scores among healthy children and adolescents, stratifies those base rates by narrow intellectual functioning and age groupings, and provides normative classification ranges to aid the interpretation of performances on the Delis-Kaplan Executive Function System (D-KEFS) in clinical practice and research. Participants included 875 children and adolescents between 8 and 19 years old from the D-KEFS normative sample (48% male; 52% female). Among these participants, 838 had complete data and were included in the current study. The racial/ethnic composition of the sample was: White (73.7%), African American (12.4%), Hispanic (11.1%), and other racial/ethnic backgrounds (2.7%). The Overall Test Battery Mean (OTBM) and the prevalence of low scores at various clinical cut-offs were calculated for the 13 primary scores from the D-KEFS Trail Making Test, Verbal Fluency Test, and Color-Word Interference Test. The OTBM and base rates were also calculated separately for those scores reflecting executive functioning (n = 7) and processing speed (n = 6). Healthy children and adolescents commonly obtained low scores on the D-KEFS tests considered here. Younger age, lower estimated full-scale intelligence quotient, and more test scores interpreted were associated with a greater frequency of low scores. Clinicians and researchers are encouraged to consider these multivariate base rates when assessing and attempting to identify executive functioning impairment among children and adolescents with the D-KEFS.

Brief iPad-Based Assessment of Cognitive Functioning with ImPACT® Pediatric

ImPACT® Pediatric is an examiner-administered iOS-based battery of neuropsychological tests designed to measure neurocognitive functioning in children ages 5-11. This study documented Multivariate Base Rates (prevalence of low scores when multiple test scores are considered simultaneously) in the ImPACT® Pediatric standardization sample (N = 892). In the total sample, it was common for children to obtain at least one low factor score using the 25th percentile [T 43; base rate (BR) = 54.2%], 16th percentile (T40, BR = 38.1%), and the 10th percentile (T37, BR = 31.1%). However, it was uncommon for children to obtain two (or more) low factor scores using any of the above-listed cutoff score.

Concussion Biomarkers Assessed in Collegiate Student-Athletes (BASICS) I: Normative study

OBJECTIVE

To describe variability in concussion biomarker concentrations collected from serum in a sample of healthy collegiate athletes, as well as report reliability metrics in a subsample of female athletes.

METHODS

In this observational cohort study, β-amyloid peptide 42 (Aβ₄₂), total tau, S100 calcium binding protein B (S100B), ubiquitin carboxy-terminal hydrolyzing enzyme L1 (UCH-L1), glial fibrillary acidic protein, microtubule associated protein 2, and 2',3'-cyclic-nucleotide 3'-phosphodiesterase (CNPase) serum concentrations were measured in 415 (61% male, 40% white, aged 19.0 ± 1.2 years) nonconcussed collegiate athletes without recent exposure to head impacts. Standardized normative distributions are reported for each biomarker. We evaluated main effects (analyses of variance) of sex and race, reporting demographic-specific normative metrics when appropriate. In a subset of 31 female participants, test-retest reliability (Pearson r) and reliable change indices (80%, 90%, and 95% confidence intervals) across a 6- to 12-month interval are reported for Aβ₄₂, total tau, S100B, and UCH-L1.

RESULTS

Males exhibited higher UCH-L1 (p < 0.001, Cohen d = 0.75) and S100B (p < 0.001, d = 0.56) than females, while females had higher CNPase (p < 0.001, d = 0.43). Regarding race, black participants had higher baseline levels of UCH-L1 (p < 0.001, d = 0.61) and S100B (p < 0.001, d = 1.1) than white participants. Conversely, white participants had higher baseline levels of Aβ₄₂ (p = 0.005, d = 0.28) and CNPase (p < 0.001, d = 0.46). Test-retest reliability was generally poor, ranging from -0.02 to 0.40, and Aβ₄₂ significantly increased from time 1 to time 2.

CONCLUSION

Healthy collegiate athletes express concussion-related serum biomarkers in variable concentrations. Accounting for demographic factors such as sex and race is essential. Evidence suggested poor reliability for serum biomarkers; however, understanding how other factors influence biomarker expression, as well as knowledge of reliable change metrics, may improve clinical interpretation and future study designs.

Longitudinal Estimation of the Clinically Significant Change in the Treatment of Major Depression Disorder

Background: Although major depressive disorder is usually treated with antidepressants, only 50-70% of the patients respond to this treatment. This study applied Jacobson and Truax's (1991) methodology (reliable change index, RCI) to a sample of depressive patients being treated with one of two antidepressants to evaluate their functioning and the effect of certain variables such as severity and age. Method: Seventy-three depressive patients medicated with Escitalopram (n = 37) or Duloxetine (n = 36) were assessed using the Hamilton depression rating scale over a 24-week period. Results: They indicate that the RCI stabilizes in an absolute way starting in week 16, and it is not until week 24 that all of the patients become part of the functional population. We found limited statistical significance with respect to the RCI and the external variables. Conclusion: Our study suggests the need to accompany the traditional statistical methodology with some other clinical estimation systems capable of going beyond a simple subtraction between pre and posttreatment values. Hence, it is concluded that RCI estimations could be stronger and more stable than the classical statistical techniques.