Evidence for validity for the Cognitive Load Inventory for Handoffs

Cognitive Load Assessment Scales in Simulation: Validity Evidence for a Novel Measure of Cognitive Load Types

INTRODUCTION

Cognitive load (CL) theory provides a framework to inform simulation instructional design. Reliable measures of CL types (intrinsic [IL], extraneous [EL], and germane load [GL]) in simulation are lacking. We developed the novel Cognitive Load Assessment Scales in Simulation (CLAS-Sim) and report validity evidence using Kane's framework.

METHODS

This quasi-experimental study tested the effect of a segmented/pause-and-debrief or standard/end-of-case-debrief intervention on pediatric residents' performance and self-rated CL in 2 complex- and simple-case simulations. After each simulation, participants completed 22 items measuring CL types. Three validity inferences were examined: scoring (instrument development and principal component analysis); generalization (internal consistency reliability of CL-component items across cases); and extrapolation [CLAS-Sim correlations with the single-item Paas scale, which measures overall CL; differences in primary task performance (high vs low); and discriminant validity of IL under different instructional-design conditions].

RESULTS

Seventy-four residents completed both simulations and postcase CLAS-Sim measures. The principal component analysis yielded 3 components: 4-item IL, 4-item EL, and 3-item GL scales (Cronbach's α, 0.68-0.77). The Paas scores correlated with CLAS-Sim IL and total CL scores in both cases ( rs range, 0.39-0.70; P ≤ 0.001). High complex-case performers reported lower IL and total CL (analyses of variance, each P < 0.001). In multivariate analyses of variance, CLAS-Sim IL, GL, and total CL varied across both cases by arm (each P ≤ 0.018); the segmented-debrief arm reported lower IL than the standard-debrief arm in both cases (each P ≤ 0.01).

CONCLUSIONS

The CLAS-Sim demonstrates preliminary validity evidence for distinguishing 3 CL types but requires further study to evaluate the impact of simulation-design elements on CL and learning.

Cognitive tasks and combined statistical methods to evaluate, model, and predict mental workload

Mental workload (MWL) is a concept that is used as a reference for assessing the mental cost of activities. In recent times, challenges related to user experience are determining the expected MWL value for a given activity and real-time adaptation of task complexity level to achieve or maintain desired MWL. As a consequence, it is important to have at least one task that can reliably predict the MWL level associated with a given complexity level. In this study, we used several cognitive tasks to meet this need, including the N-Back task, the commonly used reference test in the MWL literature, and the Corsi test. Tasks were adapted to generate different MWL classes measured via NASA-TLX and Workload Profile questionnaires. Our first objective was to identify which tasks had the most distinct MWL classes based on combined statistical methods. Our results indicated that the Corsi test satisfied our first objective, obtaining three distinct MWL classes associated with three complexity levels offering therefore a reliable model (about 80% accuracy) to predicted MWL classes. Our second objective was to achieve or maintain the desired MWL, which entailed the use of an algorithm to adapt the MWL class based on an accurate prediction model. This model needed to be based on an objective and real-time indicator of MWL. For this purpose, we identified different performance criteria for each task. The classification models obtained indicated that only the Corsi test would be a good candidate for this aim (more than 50% accuracy compared to a chance level of 33%) but performances were not sufficient to consider identifying and adapting the MWL class online with sufficient accuracy during a task. Thus, performance indicators require to be complemented by other types of measures like physiological ones. Our study also highlights the limitations of the N-back task in favor of the Corsi test which turned out to be the best candidate to model and predict the MWL among several cognitive tasks.

Handover of Critical Patients in Urgent Care and Emergency Settings: A Systematic Review of Validated Assessment Tools

The emergency handover of critical patients is used to describe the moment when responsibility for the care of a patient is transferred from one critical patient care healthcare team to another, requiring the accurate delivery of information. However, the literature provides few validated assessment tools for the transfer of critical patients in urgent care and emergency settings. To identify the available evaluation tools that assess the handover of critical patients in urgent and emergency care settings in addition to evaluations of their psychometric properties, a systematic review was carried out using PubMed, Scopus, Cinahl, Web of Science (WoS), and PsycINFO, in accordance with PRISMA guidelines. The quality of the studies was assessed using the COSMIN checklist. Finally, eight articles were identified, of which only three included validated tools for evaluating the handover of critical patients in emergency care. Content validity, construct validity, and internal consistency were the most studied psychometric properties. Three studies evaluated error and reliability, criterion validity, hypothesis testing, and sensitivity. None of them considered cross-cultural adaptation or the translation process. This systematic psychometric review shows the existing ambiguities in the handover of critically ill patients and the scarcity of validated evaluation tools. For all of these reasons, we consider it necessary to further investigate urgent care and emergency handover settings through the design and validation of an assessment tool.

Exploring the relationship between emotion and cognitive load types during patient handovers

Cognitive Load Theory has emerged as an important approach to improving instruction in the health professions workplace, including patient handovers. At the same time, there is growing recognition that emotion influences learning through numerous cognitive processes including motivation, attention, working memory, and long-term memory. This study explores how emotion influences the cognitive load experienced by trainees performing patient handovers. From January to March 2019, 693 (38.7%) of 1807 residents and fellows from a 24-hospital health system in New York city completed a survey after performing a handover. Participants rated their emotional state and cognitive load. The survey included questions about features of the learner, task, and instructional environment. The authors used factor analysis to identify the core dimensions of emotion. Regression analyses explored the relationship between the emotion factors and cognitive load types. Two emotion dimensions were identified representing invigoration and tranquility. In regression analyses, higher levels of invigoration, tranquility, and their interaction were independently associated with lower intrinsic load and extraneous load. The interaction of invigoration and tranquility predicted lower germane load. The addition of the emotion variables to multivariate models including other predictors of cognitive load types significantly increased the amount of variance explained. The study provides a model for measuring emotions in workplace learning. Because emotion appears to have a significant influence on cognitive load types, instructional designers should consider strategies that help trainees regulate emotion in order to reduce cognitive load and improve learning and performance.

Information handoffs in critical care and their implications for information quality: A socio-technical network approach

OBJECTIVE

The design of health ICTs, as well as human factors, have been shown to influence patient information quality. The aim of this study was to understand how patterns of interaction between these factors influence information quality aspects in a critical care environment.

METHOD

We conducted an ethnographic study of socio-technical information handoffs in a critical care unit. Data collection methods included non-participant observations and semi-structured interviews. Methodological principles from network analysis (SNA, VNA) were used to develop visual network diagrams, as well as to analyze the composition of the information network and its influence on patient information quality.

RESULTS

The network patterns that emerged uncover that human actors have many information processing and dissemination roles at the critical care unit. However, ICTs play key network roles, acting as information intermediaries and gatekeepers. We further identify three types of information handoffs in the critical care environment - human-human, human-ICT and ICT-human. On the one hand, we find that human-human and ICT-human handoffs influence contextual and intrinsic aspects of patient information, such as information completeness and accuracy. On the other hand, human-ICT handoffs influence information accessibility and representational quality, such as consistency and interpretability.

DISCUSSION

The results suggest that standardizing change of shift handoff communication may not be sufficient to prevent information decay in complex care trajectories. In particular, we argue that ensuring information consistency and interpretability across disciplines and professions is as important as ensuring information completeness and accuracy during change of shift handoffs. ICT and workflow design opportunities are discussed as means to address overlapping or conflicting information needs across disciplines and professions, increase information consistency, and reduce information redundancy across the network.

Are need for affect and cognition culture dependent? Implications for global public health campaigns: a cross-sectional study

Background

Cultural differences in affective and cognitive intrinsic motivation could pose challenges for global public health campaigns, which use cognitive or affective goals to evoke desired attitudes and proactive health-promoting actions. This study aimed to identify cross-cultural differences in affective and cognitive intrinsic motivation and discuss the potential value of this information for public health promotion.

Methods

A cross-sectional survey using cross-culturally validated need for affect (NFA) and need for cognition (NFC) scales was carried out among 1166 Chinese participants, and the results were compared with published data from 980 American participants. Additionally, we assessed a highly prevalent symbolic geriatric health condition, hearing loss, in 500 Chinese community-dwelling seniors. The Chinese NFA scale was developed following the translation-back translation procedure, and the psychometric evaluation was performed by applying confirmatory factor analysis (CFA), exploratory structural equation modeling (ESEM), correlation analysis and multigroup invariance test. MANOVA and Hedge’s g statistic were employed to compare the NFA and NFC levels between individuals from different countries and between Chinese seniors with and without hearing loss. The relation of early hearing intervention intention to NFA and NFC was also explored in the Chinese sample.

Results

A basic two-factor model of NFA adequately fit the sample data from Chinese and American cultures. The questionnaire demonstrated reasonable invariance of the factor structure and factor loadings across the groups. Those in the primary Chinese sample had lower NFA and NFC than their American peers. This difference held in the senior sample. Moreover, Chinese seniors with hearing loss had even lower NFA and NFC than those without hearing loss. Their early hearing intervention intention was low but was associated with intrinsic motivation.

Conclusions

The Need for Affect (NFA) construct may be generalized beyond its Western origins. There was a general lack of affective and cognitive intrinsic motivation in Chinese individuals, particularly in seniors with hearing loss, compared with their American peers. These differences point to a potential challenge in framing effective messages for some cultures in the geriatric public health domain. Ideally, recognizing and understanding this challenge will inspire the consideration of novel persuasive strategies for these audiences.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12889-021-10689-w.