Measurement Properties of the Multiple Errands Test: A Systematic Review

Comparing the Oxford Digital Multiple Errands Test (OxMET) to a real-life version: Convergence, feasibility, and acceptability

We aimed to assess the convergence, feasibility, and acceptability of the Oxford Digital Multiple Errands Test (OxMET) and the in-person Multiple Errands Test-Home version (MET-Home). Participants completed OxMET, MET-Home, Montreal Cognitive Assessment (MoCA), and questionnaires on activities of daily living, depression, technology usage, mobility, and disability. Forty-eight stroke survivors (mean age 69.61, 41.67% female, and average 16.5 months post-stroke) and 50 controls (mean age 71.46, 56.00% female) took part. No performance differences were found for healthy and stroke participants for MET-Home, and only found below p = .05 for OxMET but not below the corrected p = .006. Convergent validity was found between MET-Home and OxMET metrics (most r ≥ .30, p < .006). MET-Home accuracy was related to age (B = -.04, p = .03), sex (B = -.98, p = .03), disability (B = -0.63, p = .04), and MoCA (B = .26, p < .001), whereas OxMET accuracy was predicted by MoCA score (B = .40, p < .001). Feedback indicated that the OxMET was easy and fun and more acceptable than the MET-Home. The MET-Home was more stressful and interesting. The MET tasks demonstrated good convergent validity, with the OxMET digital administration providing a more feasible, inclusive, and acceptable assessment, especially to people with mobility restrictions and more severe stroke.

Principles for Adapting Assessments of Executive Function across Cultural Contexts

Direct assessments of executive functions (EFs) are increasingly used in research and clinical settings, with a central assumption that they assess "universal" underlying skills. Their use is spreading globally, raising questions about the cultural appropriateness of assessments devised in Western industrialized countries. We selectively reviewed multidisciplinary evidence and theory to identify sets of cultural preferences that may be at odds with the implicit assumptions of EF assessments. These preferences relate to motivation and compliance; cultural expectations for interpersonal engagement; contextualized vs. academic thinking; cultural notions of speed and time; the willingness to be silly, be incorrect, or do the opposite; and subject-matter familiarity. In each case, we discuss how the cultural preference may be incompatible with the assumptions of assessments, and how future research and practice can address the issue. Many of the cultural preferences discussed differ between interdependent and independent cultures and between schooled and unschooled populations. Adapting testing protocols to these cultural preferences in different contexts will be important for expanding our scientific understanding of EF from the narrow slice of the human population that has participated in the research to date.

Efficacy of Intervention of Participation and Executive Functions (I-PEX) for Adults Following Traumatic Brain Injury: A Preliminary Pilot Randomized Controlled Trial

BACKGROUND

Participation restrictions following traumatic brain injury are associated with executive function (EF) deficits (EFDs). The subacute recovery phase's specific characteristics (enhanced brain plasticity and impaired self-awareness) and contextual factors (inpatient setting) warrant adjusting cognitive rehabilitation protocols. The Intervention of Participation and Executive Functions (I-PEX) was designed to improve EFDs during subacute inpatient rehabilitation.

OBJECTIVE

To investigate the I-PEX's preliminary efficacy to improve EFDs during the performance of complex daily activities and enhance self-awareness, cognitive self-efficacy, participation, and quality of life postdischarge.

METHODS

A pilot pre-, post-, and follow-up double-blind randomized controlled trial with 25 participants randomly allocated to the I-PEX (n = 13) or treatment-as-usual (n = 12) group. Cognitive assessments were administered pre- and postintervention, and quality of life and participation questionnaires 1-month postdischarge. Data analysis included repeated measures analysis of variance mixed design and independent t-tests, extracting effect sizes.

RESULTS

Significant group-by-time interaction effect with a medium effect size was found for the primary outcome measure; EFs manifested in complex daily activities, indicating a larger improvement for the experimental group. The group effect was not significant. The experimental group's mean delta score (pre-post improvement) was significantly higher (1.75 ± 2.89; t(23) = 2.52, P = .019), with a large effect size (d = 1.012, 95% confidence interval [0.166-1.840]). We found no significant group and interaction effects for EFs, self-awareness, and cognitive self-efficacy or no significant differences in participation or quality of life postdischarge.

CONCLUSIONS

Results provide initial evidence for the I-PEX efficacy in treating EFDs in the subacute phase and could help determine effect size for future studies.

CLINICAL TRIAL REGISTRY NUMBER

ClinicalTrial.gov NCT04292925.

Further investigations into performance variance on the Multiple Errands Test

INTRODUCTION

The Multiple Errands Test (MET) is a complex, performance-based assessment that is useful for characterising the impact of impairments of executive function on everyday activities. However, performance variance amongst those without neurological pathology, and the impact of non-cognitive factors on this, requires further investigation.

METHODS

This was a cross-sectional analytic study, conducted with a convenience sample of 40 neurologically intact community-dwelling Australian adults. Participants completed a hospital or shopping centre version of the MET, where their Performance Efficiency, Task Completions and Rule Breaks were recorded. Non-cognitive factors of interest were demographic (age, sex and education), psychological (measured with the Hospital Anxiety and Depression Scale and self-ratings of test anxiety) and assessment-related (assessment site, self-reported site familiarity and observed strategy use). MET performance was analysed using descriptive statistics. A series of standard multiple and binary logistic regression analyses examined the relationships between MET performance and non-cognitive factors.

RESULTS

Most participants (n = 35, 87.5%) completed at least 10 of the 12 prescribed tasks and broke an average of four rules (SD = 2.36). They achieved an average performance efficiency rating of 0.75/1 (SD = 0.15), suggesting variability in the extent to which participants made non-essential location stops and/or failed to complete tasks whilst at an essential location. The assessment site and participant site familiarity had a statistically significant (p < 0.05) impact on Performance Efficiency and Task Completion scores, and psychological factors had a statistically significant (p < 0.05) relationship with Rule Breaks.

CONCLUSION

Findings suggest that the impact of factors other than cognition should be considered when interpreting MET performance. The assessment site and participant site familiarity may contribute to significant variability in Performance Efficiency scores. Clinicians should also be aware of the potential impact of these assessment-related factors on Task Completions and psychological distress on Rule Breaks.

Profiles of executive functioning following traumatic brain injury and stroke using the assessment of participation and executive functions: combined cross-sectional and longitudinal designs

OBJECTIVES

The Assessment of Participation and Executive Functions (A-PEX) evaluates executive functioning through daily participation in complex daily activities. This study examines its ability to discriminate between executive functioning profiles post-traumatic brain injury and post-stroke and its sensitivity to changes.

DESIGN

Cross-sectional with a longitudinal component.

PATIENTS

Adults with post-traumatic brain injury (n = 28) and post-stroke (n = 26) in a rehabilitation facility.

METHODS

Patients were administered the A-PEX, Multiple Errands Test-Hospital version and Color Trail Test at 2 time-points 1 month apart. The Montreal Cognitive Assessment was administered at the first time-point, and Executive Functions Performance Test's Internet-based Bill Payment subtest at the second. The analysis used Mann-Whitney and Wilcoxon signed-rank tests.

RESULTS

The stroke group's A-PEX scores were higher than the traumatic brain injury group's at the first time-point (p < 0.05). No differences were found in the other assessments. Within-group differences in both groups were significant in the A-PEX (-3.7 < r < - 2.3, p < 0.05) and Multiple Errands Test-Hospital version (-3.4 < r < -3.3, p < 0.01).

CONCLUSION

The A-PEX may provide valuable information about the uniqueness of executive functioning profiles and patients' progress.

Development of a generic version of the multiple errands test for severe acquired brain injuries

BACKGROUND

Impairment of executive functions is a frequent and disabling consequence of brain injury. In 1991, Shallice and Burgess elaborated the Multiple Errand Test (MET) to assess everyday troubles associated with this impairment.

OBJECTIVE

Our first aim was to create a generic version of MET (MET-G) readily adaptable to different settings and for retesting. The second aim was to validate the revised version to detect executive impairment in individuals with severe ABI.

METHODS

An inter-professional team of experts in neurorehabilitation created a revised version of MET. Twenty-one patients with sABI (severe Acquired Brain Injury) and a control group of 20 neurologically healthy subjects, took part in the testing of MET-G.

RESULTS

MET-G clearly distinguished people with sABI from healthy controls based on Total error score, task completion, and rule breaking. The test showed good inter-rater reliability and internal consistency.

CONCLUSIONS

The new, generic version of MET was able to differentiate adults with sABI from controls and proved to be a good tool for evaluating executive functions in these patients in daily-life contexts. Indications on how to adapt the test to different contexts and different scoring modalities are provided.

Preliminary validation of the Virtual Kitchen Challenge as an objective and sensitive measure of everyday function associated with cerebrovascular disease

Preliminary validity of a computer-based test of everyday function (Virtual Kitchen Challenge [VKC]) was examined against brain-imaging markers of cerebrovascular disease and in contrast to conventional neuropsychological and self-report measures. Twenty community-dwelling older adults (n = 6 mild cognitive impairment) performed simulated breakfast and lunch tasks using a computer touchscreen (VKC). Automated measures (completion time, proportion time off screen, etc.) were computed during training and test conditions. White matter hyperintensity (WMH) volumes from brain magnetic resonance imaging and conventional measures of cognition and function also were obtained. VKC completion time and proportion time off screen improved significantly from training to test and were significantly associated with WMH volume (r > 0.573). VKC measures and WMH were not significantly correlated with conventional cognitive or self-report measures. The VKC holds promise as a valid measure of subtle functional difficulties in older adults that is sensitive to change and cerebrovascular pathology, highlighting its potential for clinical trials.

Highlights

Virtual Kitchen Challenge (VKC) scores showed significant improvement from training to test conditions.VKC scores (completion time and proportion of time off screen) were associated with a neuroimaging biomarker of brain health (white matter hyperintensities).

Using Multiverse Analysis to Highlight Differences in Convergent Correlation Outcomes Due to Data Analytical and Study Design Choices

In neuropsychological research, there are a near limitless number of different approaches researchers can choose when designing studies. Here we showcase the multiverse/specification curve technique to establish the robustness of analytical pathways choices within classic psychometric test validation in an example test of executive function. We examined the impact of choices regarding sample groups, sample sizes, test metrics, and covariate inclusions on convergent validation correlations between tests of executive function. Data were available for 87 neurologically healthy adults and 117 stroke survivors, and a total of 2,220 different analyses were run in a multiverse analysis. We found that the type of sample group, sample size, and test metric used for analyses affected validation outcomes. Covariate inclusion choices did not affect the observed coefficients in our analyses. The present analysis demonstrates the importance of carefully justifying every aspect of a psychometric test validation study a priori with theoretical and statistical factors in mind. It is essential to thoroughly consider the purpose and use of a new tool when designing validation studies.

The Multiple Errands Test: A Guide for Site-Specific Version Development

Background. The complex and real-world nature of the Multiple Errands Test (MET) makes it a valuable and increasingly popular assessment of cognitive function. However, these same qualities make its local implementation challenging. Purpose. To produce an evidence-based guide for site-specific adaptation of the MET. Method. The CAN-IMPLEMENT© knowledge translation framework informed a structured approach to the creation of a guide to site-specific version development, informed by twenty-two published approaches to MET adaptation. Applicability of the guide was supported by a two-phase revision process, in which a site-specific hospital and community version produced from its recommendations were administered with forty-two neurologically intact participants and stakeholder feedback obtained. Findings. We offer an outline of core components which maintain the integrity of the MET, and adaptable peripheries which may be modified when required by the local setting. Implications. The proposed guide provides a systematic yet flexible guide for site-specific MET development.

Predictive validity of the Oxford digital multiple errands test (OxMET) for functional outcomes after stroke

The Oxford Digital Multiple Errands Test (OxMET) is a brief computer-tablet based cognitive screen, intended as an ecologically valid assessment of executive dysfunction. We examined aspects of predictive validity in relation to functional outcomes. Participants (≤ 2 months post-stroke) were recruited from an English-speaking stroke rehabilitation in-patient setting. Participants completed OxMET. The Barthel Index, Therapy Outcome Measure (TOMS), and modified Rankin Scale (mRS) were collected from medical notes. Participants were followed up after 6-months and completed the Nottingham Extended Activities of Daily Living (NEADL) scale. 117 participants were recruited (M = 26.18 days post-stroke (SD = 25.16), mean 74.44yrs (SD  = 12.88), median NIHSS 8.32 (IQR = 5-11)). Sixty-six completed a follow-up (M = 73.94yrs (SD  = 12.68), median NIHSS 8 (IQR = 4-11)). Significant associations were found between TOMS and mRS. At 6-month follow up, we found a moderate predictive relationship between the OxMET accuracy and NEADL (R2 = .29, p < .001), and we did not find this prediction with MoCA taken at 6-months. The subacute OxMET associated with measures of functionality and disability in a rehabilitation context, and in activities of daily living. The OxMET is an assessment of executive function with good predictive validity on clinically relevant functional outcome measures that may be more predictive than other cognitive tests.

Beyond cognitive deficits: how social class shapes social cognition

Lower social class is thought to contribute to poorer executive functioning and working memory. Nevertheless, lower social class individuals consistently outperform their higher-class counterparts on social cognitive tasks that rely on similar underlying cognitive processes (e.g., working memory and executive functioning). Why would lower social class inhibit such processes in one domain, but promote them in another? We argue that features of lower-class communities (e.g., resource scarcity) promote social cognition via cultural processes. We then argue that social cognition involves partially unique task and neural demands that are separate from nonsocial cognition. We conclude that unique task and neural demands, together with the distinctive cognitive proclivities of lower- and higher-class cultures, can explain variable associations between social class and cognitive functioning.

Validation of the UK English Oxford cognitive screen-plus in sub-acute and chronic stroke survivors

INTRODUCTION

Stroke survivors are routinely screened for cognitive impairment with tools that often fail to detect subtle impairments. The Oxford Cognitive Screen-Plus (OCS-Plus) is a brief tablet-based screen designed to detect subtle post-stroke cognitive impairments. We examined its psychometric properties in two UK English-speaking stroke cohorts (subacute: <3 months post-stroke, chronic: >6 months post-stroke) cross-sectionally.

PATIENTS AND METHODS

This study included 347 stroke survivors (mean age = 73 years; mean education = 13 years; 43.06% female; 74.42% ischaemic stroke). The OCS-Plus was completed by 181 sub-acute stroke survivors and 166 chronic stroke survivors. All participants also completed the Oxford Cognitive Screen (OCS) and a subset completed the Montreal Cognitive Assessment (MoCA) and further neuropsychological tests.

RESULTS

First, convergent construct validity of OCS-Plus tasks to task-matched standardized neuropsychological tests was confirmed (r > 0.30). Second, we evaluated divergent construct validity of all OCS-Plus subtasks (r < 0.19). Third, we report the sensitivity and specificity of each OCS-Plus subtask compared to neuropsychological test performance. Fourth, we found that OCS-Plus detected cognitive impairments in a large proportion of those classed as unimpaired on MoCA (100%) and OCS (98.50%).

DISCUSSION AND CONCLUSION

The OCS-Plus provides a valid screening tool for sensitive detection of subtle cognitive impairment in stroke patients. Indeed, the OCS-Plus detected subtle cognitive impairment at a similar level to validated neuropsychological assessments and exceeded detection of cognitive impairment compared to standard clinical screening tools.

Self-reported strategy generation and implementation in the multiple errands test: A qualitative description

The Multiple Errands Test (MET) is a naturalistic assessment of executive function. Strategy use during the MET can provide useful information for the development of a cognitive profile and intervention plan in patients with brain injury. However, while observed external strategy use in the MET is well-documented, information about internal strategy use and reference data with healthy controls is limited. Contextual influences on strategy selection in this real-world assessment are also not well understood. This qualitative descriptive study explored the internal and external strategies used during MET performance by cognitively intact adults. Strategies were categorized as planning, checking, and problem solving. When planning, participants simplified and familiarized themselves with MET requirements before developing an action plan. They checked their performance by asking for help and using cues in the physical environment. When problems arose, these were solved through self-talk, comparing alternatives, applying context and modifying their plan. Results highlighted that individuals employ both visible and hidden strategies during the MET. This suggests that reflective discussions with patients following cognitive task engagement may be important, to uncover and understand strategy use, both to inform analysis of performance and guide strategy training.

Self-reported questionnaires for lymphoedema: a systematic review of measurement properties using COSMIN framework

INTRODUCTION

Lymphoedema may develop as a result of numerous genetic and traumatic causes; however, treatment for cancer is the most common cause of its development in more economically developed nations. This systematic review critically appraised, compared and summarised the measurement properties of lymphoedema-specific self-reported questionnaires (SRQs) measuring various patient-reported outcomes including quality of life (QOL), function, morbidity, and symptoms.

METHODS

Seven databases were searched to identify studies of the measurement properties of SRQs. Two review teams independently evaluated the quality of the individual studies using the risk of bias tool from the Consensus-based Standards for the Selection of Health Measurement Instruments (COSMIN). Measurement properties of the SRQs presented in the studies were then rated. Study level ratings were summarised for an SRQ if they were reported in multiple studies, and their overall quality of the evidence were then graded.

RESULTS

Forty articles, reporting on 19 SRQs were identified from 8615 records. The focus of the 19 SRQs included eight on QOL, four on symptoms, two on function, and two on impairment. The other three SRQs were on illness perception, self-efficacy, and patient-relevant treatment benefit, respectively. Eight and three SRQs were upper limb and lower limb-specific, respectively, whereas seven questionnaires were for both upper and lower limb lymphoedema. One SRQ was developed for head and neck lymphoedema. According to the COSMIN framework, none of the SRQs reviewed had sufficient evidence to support all nine measurement properties. In lower limb questionnaires, the LYMQOL-leg has sufficient content, structural, and construct validity as well as internal consistency and reliability. For upper limb lymphoedema questionnaires, the Lymph-ICF-UL had sufficient content and construct validity as well as reliability.

CONCLUSION

LYMQOL-leg SRQ is recommended with confidence for evaluation of QOL of people with lower limb lymphoedema while the Lymph-ICF-UL is recommended for evaluation of the QOL of the breast cancer-related lymphoedema with some confidence. In view of the high level of the indeterminate ratings of the measurement properties of the existing SRQs, further research is desirable.

The Oxford digital multiple errands test (OxMET): Validation of a simplified computer tablet based multiple errands test

Impairments in executive functioning are common following Acquired Brain Injury, though there are few screening tools which present a time efficient and ecologically valid approach to assessing the consequences of executive impairments. We present the Oxford Digital Multiple Errands Test (OxMET), a novel and simplified computer-tablet version of a Multiple Errands Test. We recruited 124 neurologically healthy controls and 105 stroke survivors to complete the OxMET task. Normative data and internal consistency were established from the healthy control data. Convergent and divergent validation was assessed in a mixed subset of 158 participants who completed the OxMET and OCS-Plus. Test-retest reliability was examined across a mixed subset of 39 participants. Finally, we investigated the known-group discriminability of the OxMET. The OxMET demonstrated very high internal consistency, and stable group level test-retest performance as well as good convergent and divergent validity. The OxMET demonstrated high sensitivity and good specificity in overall differentiation of stroke survivors from controls. The Oxford Digital Multiple Errands Test is a brief, easy to administer tool, designed to quickly screen for potential consequences of executive impairments in a virtual environment shopping task on a computer tablet. Initial normative data and validation within a chronic stroke cohort is presented.