How context changes the neural basis of perception and language

The N400 component reflecting semantic and repetition priming of visual scenes is suppressed during the attentional blink

In the attentional blink paradigm, participants attempt to identify two targets appearing in a rapidly presented stream of distractors. Report accuracy is typically high for the first target (T1) while identification of the second target (T2) is impaired when it follows within about 200-400 ms of T1. An important question is whether T2 is processed to a semantic level even when participants are unaware of its identity. We examined this issue in three studies that used natural scenes as stimuli and the N400 component of the event-related potential (ERP) as a measure of semantic priming. In the first experiment, the prime (e.g., a doghouse in a yard) was presented at the beginning of the trial and a test picture that was related (e.g., a dog standing in the kitchen) or unrelated (e.g., a coffee mug on a table) appeared as T2. In the second experiment, the prime was presented as T2 and the test picture appeared at the end of the picture sequence. In both experiments, we found robust semantic priming when participants were aware of the identity of the blinked picture and an absence of priming when they were unaware. In Experiment 3, we used identity priming to assess whether earlier representations preceding semantics were preserved, and again found that priming critically depended on awareness of the prime's identity. These results suggest that semantic priming in scenes, as measured with the N400, is a higher-level process that critically depends on attention and awareness.

The cognitive science of language diversity: achievements and challenges

Linguistics needs to embrace all the way down a key feature of language: its diversity. In this paper, we build on recent experimental findings and theoretical discussions about the neuroscience and the cognitive science of linguistic variation, but also on proposals by theoretical biology, to advance some future directions for a more solid neurocognitive approach to language diversity. We argue that the cognitive foundations and the neuroscience of human language will be better understood if we pursue a unitary explanation of four key dimensions of linguistic variation: the different functions performed by language, the diversity of sociolinguistic phenomena, the typological differences between human languages, and the diverse developmental paths to language. Succeeding in the cognitive and neurobiological examination and explanation of these four dimensions will not only result in a more comprehensive understanding of how our brain processes language, but also of how language evolved and the core properties of human language(s).

Shared representations of human actions across vision and language

Humans can recognize and communicate about many actions performed by others. How are actions organized in the mind, and is this organization shared across vision and language? We collected similarity judgments of human actions depicted through naturalistic videos and sentences, and tested four models of action categorization, defining actions at different levels of abstraction ranging from specific (action verb) to broad (action target: whether an action is directed towards an object, another person, or the self). The similarity judgments reflected a shared organization of action representations across videos and sentences, determined mainly by the target of actions, even after accounting for other semantic features. Furthermore, language model embeddings predicted the behavioral similarity of action videos and sentences, and captured information about the target of actions alongside unique semantic information. Together, our results show that action concepts are similarly organized in the mind across vision and language, and that this organization reflects socially relevant goals.

Why we should view the decision of medical trainees to cheat as the product of a person-by-situation interaction

BACKGROUND

Cheating during medical training is a delicate subject matter with varying opinions on the prevalence, causes and gravity of cheating during training.

PROPOSED FRAMEWORK

In this article, the authors suggest that the decision to cheat is best viewed as the product of a person-by-situation interaction rather than indicating inherent dishonesty and/or extrinsic motivation in those who participate in cheating. This framework can explain why individuals who would typically default to honesty may participate in cheating if there is perceived justification for cheating and where situational variables, such as ease of cheating, rewards for cheating and perceived risk associated with cheating, make the decision to cheat appear rational.

DISCUSSION

They discuss why the impression that there is a culture of cheating can provide perceived justification for medical trainees to cheat if they have the opportunity. They then describe how aspects of medical training and assessment may enable or hinder cheating by trainees. Consistent with the person-by-situation interaction framework, they contend that our response to cheating should include interventions directed at both the person who cheated and situational variables that enabled cheating. Recognising that some forms of cheating may be widespread, difficult to detect and contentious (such as the creation and use of exam reconstructs), their proposal for dealing with suspected and pervasive cheating is to identify and target enabling variables such that the decision to cheat becomes less rational. Their hope is that in so doing, we can gradually nudge trainees and the culture of medical training towards honesty.

Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that (1) in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and (2) lesioning these neurons by setting their output to 0 or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.

Emergence of Emotion Selectivity in Deep Neural Networks Trained to Recognize Visual Objects

Recent neuroimaging studies have shown that the visual cortex plays an important role in representing the affective significance of visual input. The origin of these affect-specific visual representations is debated: they are intrinsic to the visual system versus they arise through reentry from frontal emotion processing structures such as the amygdala. We examined this problem by combining convolutional neural network (CNN) models of the human ventral visual cortex pre-trained on ImageNet with two datasets of affective images. Our results show that in all layers of the CNN models, there were artificial neurons that responded consistently and selectively to neutral, pleasant, or unpleasant images and lesioning these neurons by setting their output to zero or enhancing these neurons by increasing their gain led to decreased or increased emotion recognition performance respectively. These results support the idea that the visual system may have the intrinsic ability to represent the affective significance of visual input and suggest that CNNs offer a fruitful platform for testing neuroscientific theories.

Beyond the tried and true: How virtual reality, dialog setups, and a focus on multimodality can take bilingual language production research forward

Bilinguals possess the ability of expressing themselves in more than one language, and typically do so in contextually rich and dynamic settings. Theories and models have indeed long considered context factors to affect bilingual language production in many ways. However, most experimental studies in this domain have failed to fully incorporate linguistic, social, or physical context aspects, let alone combine them in the same study. Indeed, most experimental psycholinguistic research has taken place in isolated and constrained lab settings with carefully selected words or sentences, rather than under rich and naturalistic conditions. We argue that the most influential experimental paradigms in the psycholinguistic study of bilingual language production fall short of capturing the effects of context on language processing and control presupposed by prominent models. This paper therefore aims to enrich the methodological basis for investigating context aspects in current experimental paradigms and thereby move the field of bilingual language production research forward theoretically. After considering extensions of existing paradigms proposed to address context effects, we present three far-ranging innovative proposals, focusing on virtual reality, dialog situations, and multimodality in the context of bilingual language production.

Discourse coherence modulates use of predictive processing during sentence comprehension

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze probability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, predictable words were processed more quickly and anomalous words more slowly when they occurred in highly coherent discourse. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating semantic and lexical information less when their expectations are less likely to be reliable.

Predictive processing of scenes and objects

Real-world visual input consists of rich scenes that are meaningfully composed of multiple objects which interact in complex, but predictable, ways. Despite this complexity, we recognize scenes, and objects within these scenes, from a brief glance at an image. In this review, we synthesize recent behavioral and neural findings that elucidate the mechanisms underlying this impressive ability. First, we review evidence that visual object and scene processing is partly implemented in parallel, allowing for a rapid initial gist of both objects and scenes concurrently. Next, we discuss recent evidence for bidirectional interactions between object and scene processing, with scene information modulating the visual processing of objects, and object information modulating the visual processing of scenes. Finally, we review evidence that objects also combine with each other to form object constellations, modulating the processing of individual objects within the object pathway. Altogether, these findings can be understood by conceptualizing object and scene perception as the outcome of a joint probabilistic inference, in which "best guesses" about objects act as priors for scene perception and vice versa, in order to concurrently optimize visual inference of objects and scenes.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

A large-scale fMRI dataset for human action recognition

Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.

Different kinds of simulation during literary reading: Insights from a combined fMRI and eye-tracking study

Mental simulation is an important aspect of narrative reading. In a previous study, we found that gaze durations are differentially impacted by different kinds of mental simulation. Motor simulation, perceptual simulation, and mentalizing as elicited by literary short stories influenced eye movements in distinguishable ways (Mak & Willems, 2019). In the current study, we investigated the existence of a common neural locus for these different kinds of simulation. We additionally investigated whether individual differences during reading, as indexed by the eye movements, are reflected in domain-specific activations in the brain. We found a variety of brain areas activated by simulation-eliciting content, both modality-specific brain areas and a general simulation area. Individual variation in percent signal change in activated areas was related to measures of story appreciation as well as personal characteristics (i.e., transportability, perspective taking). Taken together, these findings suggest that mental simulation is supported by both domain-specific processes grounded in previous experiences, and by the neural mechanisms that underlie higher-order language processing (e.g., situation model building, event indexing, integration).

Switching between emotions in the twenty-first century attention economy

The use of naturalistic stimuli in cognitive neuroscience experiments inspires and requires theoretical foundations that bring together different cognitive domains, such as emotion, language, and morality. By zooming in on the digital environments in which we often perceive emotional messages today, and inspired by the Mixed and Ambiguous Emotions and Morality model, we here argue that successfully processing emotional information in the twenty-first century will often have to rely not only on simulation and/or mentalizing, but also on executive control and attention regulation.

MA-EM: A neurocognitive model for understanding mixed and ambiguous emotions and morality

Understanding emotions and moral intentions of other people is integral to being human. Humanities scholars have long recognized the complex and ambiguous nature of emotions and morality. People are rarely 'just' happy, or sad. Neither are they 'just' good or bad people. Despite this, most knowledge about the psychological and neural basis of emotions and moral understanding comes from experiments investigating unidimensional and non-ambiguous emotions and morality. The goal of this paper is twofold. First I want to point out why mixed and ambiguous emotions and morality are a promising research topic for cognitive neuroscientists. Observing or experiencing mixed or ambiguous emotions and morality tends to have a strong impact on humans. This impact is clearly visible in narratives and fiction, and I will argue that narratives make an excellent stimulus to study the effects of emotional and moral ambiguity. Second, I will sketch a model to help guide research in this promising corner of human cognition.

Enhancing learning and retention with distinctive virtual reality environments and mental context reinstatement

Memory is inherently context-dependent: internal and environmental cues become bound to learnt information, and the later absence of these cues can impair recall. Here, we developed an approach to leverage context-dependence to optimise learning of challenging, interference-prone material. While navigating through desktop virtual reality (VR) contexts, participants learnt 80 foreign words in two phonetically similar languages. Those participants who learnt each language in its own unique context showed reduced interference and improved one-week retention (92%), relative to those who learnt the languages in the same context (76%)-however, this advantage was only apparent if participants subjectively experienced VR-based contexts as "real" environments. A follow-up fMRI experiment confirmed that reinstatement of brain activity patterns associated with the original encoding context during word retrieval was associated with improved recall performance. These findings establish that context-dependence can be harnessed with VR to optimise learning and showcase the important role of mental context reinstatement.

Similar mechanisms of temporary bindings for identity and location of objects in healthy ageing: an eye-tracking study with naturalistic scenes

The ability to maintain visual working memory (VWM) associations about the identity and location of objects has at times been found to decrease with age. To date, however, this age-related difficulty was mostly observed in artificial visual contexts (e.g., object arrays), and so it is unclear whether it may manifest in naturalistic contexts, and in which ways. In this eye-tracking study, 26 younger and 24 healthy older adults were asked to detect changes in a critical object situated in a photographic scene (192 in total), about its identity (the object becomes a different object but maintains the same position), location (the object only changes position) or both (the object changes in location and identity). Aging was associated with a lower change detection performance. A change in identity was harder to detect than a location change, and performance was best when both features changed, especially in younger adults. Eye movements displayed minor differences between age groups (e.g., shorter saccades in older adults) but were similarly modulated by the type of change. Latencies to the first fixation were longer and the amplitude of incoming saccades was larger when the critical object changed in location. Once fixated, the target object was inspected for longer when it only changed in identity compared to location. Visually salient objects were fixated earlier, but saliency did not affect any other eye movement measures considered, nor did it interact with the type of change. Our findings suggest that even though aging results in lower performance, it does not selectively disrupt temporary bindings of object identity, location, or their association in VWM, and highlight the importance of using naturalistic contexts to discriminate the cognitive processes that undergo detriment from those that are instead spared by aging.

Social-affective features drive human representations of observed actions

Humans observe actions performed by others in many different visual and social settings. What features do we extract and attend when we view such complex scenes, and how are they processed in the brain? To answer these questions, we curated two large-scale sets of naturalistic videos of everyday actions and estimated their perceived similarity in two behavioral experiments. We normed and quantified a large range of visual, action-related, and social-affective features across the stimulus sets. Using a cross-validated variance partitioning analysis, we found that social-affective features predicted similarity judgments better than, and independently of, visual and action features in both behavioral experiments. Next, we conducted an electroencephalography experiment, which revealed a sustained correlation between neural responses to videos and their behavioral similarity. Visual, action, and social-affective features predicted neural patterns at early, intermediate, and late stages, respectively, during this behaviorally relevant time window. Together, these findings show that social-affective features are important for perceiving naturalistic actions and are extracted at the final stage of a temporal gradient in the brain.