Consciousness is more than meets the eye: a call for a multisensory study of subjective experience

The influence of feature-based attention and response requirements on ERP correlates of auditory awareness

In search for the neural correlates of consciousness (NCCs), it is important to isolate the true NCCs from their prerequisites, consequences, and co-occurring processes. To date, little is known about how attention affects the event-related potential (ERP) correlates of auditory awareness and there is contradictory evidence on whether one of them, the late positivity (LP), is affected by response requirements. By implementing a GO-NOGO design with target and nontarget stimuli, we controlled for feature-based attention and response requirements in the same experiment, while participants rated their awareness using a perceptual awareness scale. The results showed a prolonged auditory awareness negativity (AAN) for aware trials, which was influenced neither by attention nor by response requirement. The LP was affected by both attention and response requirements. Consistent with the levels of processing hypothesis, the LP was related to consciousness as a correlate of the processing of higher-level stimulus features, likely requiring access to a "global workspace." Our findings further suggest that AAN is a proper ERP correlate of auditory consciousness and thus a true NCC in the auditory modality.

Unconscious multisensory integration: behavioral and neural evidence from subliminal stimuli

Introduction

The prevailing theories of consciousness consider the integration of different sensory stimuli as a key component for this phenomenon to rise on the brain level. Despite many theories and models have been proposed for multisensory integration between supraliminal stimuli (e.g., the optimal integration model), we do not know if multisensory integration occurs also for subliminal stimuli and what psychophysical mechanisms it follows.

Methods

To investigate this, subjects were exposed to visual (Virtual Reality) and/or haptic stimuli (Electro-Cutaneous Stimulation) above or below their perceptual threshold. They had to discriminate, in a two-Alternative Forced Choice Task, the intensity of unimodal and/or bimodal stimuli. They were then asked to discriminate the sensory modality while recording their EEG responses.

Results

We found evidence of multisensory integration for supraliminal condition, following the classical optimal model. Importantly, even for subliminal trials participant's performances in the bimodal condition were significantly more accurate when discriminating the intensity of the stimulation. Moreover, significant differences emerged between unimodal and bimodal activity templates in parieto-temporal areas known for their integrative role.

Discussion

These converging evidences - even if preliminary and needing confirmation from the collection of further data - suggest that subliminal multimodal stimuli can be integrated, thus filling a meaningful gap in the debate about the relationship between consciousness and multisensory integration.

Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes

The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.

Metacognitive awareness in the sound-induced flash illusion

Hundreds (if not thousands) of multisensory studies provide evidence that the human brain can integrate temporally and spatially discrepant stimuli from distinct modalities into a singular event. This process of multisensory integration is usually portrayed in the scientific literature as contributing to our integrated, coherent perceptual reality. However, missing from this account is an answer to a simple question: how do confidence judgements compare between multisensory information that is integrated across multiple sources, and multisensory information that comes from a single, congruent source in the environment? In this paper, we use the sound-induced flash illusion to investigate if confidence judgements are similar across multisensory conditions when the numbers of auditory and visual events are the same, and the numbers of auditory and visual events are different. Results showed that congruent audiovisual stimuli produced higher confidence than incongruent audiovisual stimuli, even when the perceptual report was matched across the two conditions. Integrating these behavioural findings with recent neuroimaging and theoretical work, we discuss the role that prefrontal cortex may play in metacognition, multisensory causal inference and sensory source monitoring in general. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Multisensory decisions from self to world

Classic Bayesian models of perceptual inference describe how an ideal observer would integrate 'unisensory' measurements (multisensory integration) and attribute sensory signals to their origin(s) (causal inference). However, in the brain, sensory signals are always received in the context of a multisensory bodily state-namely, in combination with other senses. Moreover, sensory signals from both interoceptive sensing of one's own body and exteroceptive sensing of the world are highly interdependent and never occur in isolation. Thus, the observer must fundamentally determine whether each sensory observation is from an external (versus internal, self-generated) source to even be considered for integration. Critically, solving this primary causal inference problem requires knowledge of multisensory and sensorimotor dependencies. Thus, multisensory processing is needed to separate sensory signals. These multisensory processes enable us to simultaneously form a sense of self and form distinct perceptual decisions about the external world. In this opinion paper, we review and discuss the similarities and distinctions between multisensory decisions underlying the sense of self and those directed at acquiring information about the world. We call attention to the fact that heterogeneous multisensory processes take place all along the neural hierarchy (even in forming 'unisensory' observations) and argue that more integration of these aspects, in theory and experiment, is required to obtain a more comprehensive understanding of multisensory brain function. This article is part of the theme issue 'Decision and control processes in multisensory perception'.

Cross-modal metacognition: Visual and tactile confidence share a common scale

Humans can judge the quality of their perceptual decisions-an ability known as perceptual confidence. Previous work suggested that confidence can be evaluated on an abstract scale that can be sensory modality-independent or even domain-general. However, evidence is still scarce on whether confidence judgments can be directly made across visual and tactile decisions. Here, we investigated in a sample of 56 adults whether visual and tactile confidence share a common scale by measuring visual contrast and vibrotactile discrimination thresholds in a confidence-forced choice paradigm. Confidence judgments were made about the correctness of the perceptual decision between two trials involving either the same or different modalities. To estimate confidence efficiency, we compared discrimination thresholds obtained from all trials to those from trials judged to be relatively more confident. We found evidence for metaperception because higher confidence was associated with better perceptual performance in both modalities. Importantly, participants were able to judge their confidence across modalities without any costs in metaperceptual sensitivity and only minor changes in response times compared to unimodal confidence judgments. In addition, we were able to predict cross-modal confidence well from unimodal judgments. In conclusion, our findings show that perceptual confidence is computed on an abstract scale and that it can assess the quality of our decisions across sensory modalities.

Relating different Dimensions of Bodily Experiences: Review and proposition of an integrative model relying on phenomenology, predictive brain and neuroscience of the self

How we mentally experience our body has been studied in a variety research domains. Each of these domains focuses in its own ways on different aspects of the body, namely the neurophysiological, perceptual, affective or social components, and proposes different conceptual taxonomies. It is therefore difficult to find one's way through this vast literature and to grasp the relationships between the different dimensions of bodily experiences. In this narrative review, we summarize the existing research directions and present their limits. We propose an integrative framework, grounded in studies on phenomenal consciousness, self-consciousness and bodily self-consciousness, that can provide a common basis for evaluating findings on different dimensions of bodily experiences. We review the putative mechanisms, relying on predictive processes, and neural substrates that support this model. We discuss how this model enables a conceptual assessment of the interrelationships between multiple dimensions of bodily experiences and potentiate interdisciplinary approaches.

Development and preliminary validation of the Self-Awareness Situation-Based Observation Lists for children with Profound Intellectual and Multiple Disabilities

BACKGROUND

A primary, pre-reflective form of self-awareness develops during the first two years of life, through sensory perceptions and bodily and social experiences. Although this is a central dimension in the development of children with PIMD, no tools are available to guide caregivers' observations.

AIMS

To present the development procedure and the results of the first validation step of a battery aimed at direct observation of primary self-awareness behaviors in children with PIMD.

METHODS AND PROCEDURE

We built a criterion-referenced tool composed of five scales proposed in two complementary lists. The first list comprises inducing standardized tasks; the second natural observation situations. The battery was administered three times to 18 children with PIMD.

OUTCOMES AND RESULTS

The inducing tasks list was shown to be reliable, with good internal consistency, inter- and intra-rater reliability, high procedural reliability and high test-retest reliability. The natural observation situations list has a lower internal consistency, but high test-retest reliability. The social validity of both lists is deemed to be excellent.

CONCLUSION AND IMPLICATIONS

The tool looks promising. Even if these first results need to be confirmed by further research, it opens up perspectives for assessment and intervention on a key dimension of human functioning.

Modality-specific and modality-general electrophysiological correlates of visual and auditory awareness: Evidence from a bimodal ERP experiment

To date, most studies on the event-related potential (ERP) correlates of conscious perception have examined a single perceptual modality. We compared electrophysiological correlates of visual and auditory awareness in the same experiment to test whether there are modality-specific and modality-general correlates of conscious perception. We used near threshold stimulation and analyzed event-related potentials in response to aware and unaware trials in visual, auditory and bimodal conditions. The results showed modality-specific negative amplitude correlates of conscious perception between 200 and 300 ms after stimulus onset. A combination of these auditory and visual awareness negativities was observed in the bimodal condition. A later positive amplitude difference, whose early part was modality-specific, possibly reflecting access to global workspace, and later part shared modality-general features, possibly indicating higher level cognitive processing involving the decision making, was also observed.

V1 as an egocentric cognitive map

We typically distinguish between V1 as an egocentric perceptual map and the hippocampus as an allocentric cognitive map. In this article, we argue that V1 also functions as a post-perceptual egocentric cognitive map. We argue that three well-documented functions of V1, namely (i) the estimation of distance, (ii) the estimation of size, and (iii) multisensory integration, are better understood as post-perceptual cognitive inferences. This argument has two important implications. First, we argue that V1 must function as the neural correlates of the visual perception/cognition distinction and suggest how this can be accommodated by V1's laminar structure. Second, we use this insight to propose a low-level account of visual consciousness in contrast to mid-level accounts (recurrent processing theory; integrated information theory) and higher-level accounts (higher-order thought; global workspace theory). Detection thresholds have been traditionally used to rule out such an approach, but we explain why it is a mistake to equate visibility (and therefore the presence/absence of visual experience) with detection thresholds.

Does Vergence Affect Perceived Size?

Since Kepler (1604) and Descartes (1637), it has been suggested that 'vergence' (the angular rotation of the eyes) plays a key role in size constancy. However, this has never been tested divorced from confounding cues such as changes in the retinal image. In our experiment, participants viewed a target which grew or shrank in size over 5 s. At the same time, the fixation distance specified by vergence was reduced from 50 to 25 cm. The question was whether this change in vergence affected the participants' judgements of whether the target grew or shrank in size? We found no evidence of any effect, and therefore no evidence that eye movements affect perceived size. If this is correct, then our finding has three implications. First, perceived size is much more reliant on cognitive influences than previously thought. This is consistent with the argument that visual scale is purely cognitive in nature (Linton, 2017; 2018). Second, it leads us to question whether the vergence modulation of V1 contributes to size constancy. Third, given the interaction between vergence, proprioception, and the retinal image in the Taylor illusion, it leads us to ask whether this cognitive approach could also be applied to multisensory integration.

Whatever Next and Close to My Self-The Transparent Senses and the "Second Skin": Implications for the Case of Depersonalization

In his paper “Whatever next? Predictive brains, situated agents, and the future of cognitive science,” Andy Clark seminally proposed that the brain's job is to predict whatever information is coming “next” on the basis of prior inputs and experiences. Perception fundamentally subserves survival and self-preservation in biological agents, such as humans. Survival however crucially depends on rapid and accurate information processing of what is happening in the here and now. Hence, the term “next” in Clark's seminal formulation must include not only the temporal dimension (i.e., what is perceived now) but also the spatial dimension (i.e., what is perceived here or next-to-my-body). In this paper, we propose to focus on perceptual experiences that happen “next,” i.e., close-to-my-body. This is because perceptual processing of proximal sensory inputs has a key impact on the organism's survival. Specifically, we focus on tactile experiences mediated by the skin and what we will call the “extended skin” or “second skin,” that is, immediate objects/materials that envelop closely to our skin, namely, clothes. We propose that the skin and tactile experiences are not a mere border separating the self and world. Rather, they simultaneously and inherently distinguish and connect the bodily self to its environment. Hence, these proximal and pervasive tactile experiences can be viewed as a “transparent bridge” intrinsically relating and facilitating exchanges between the self and the physical and social world. We conclude with potential implications of this observation for the case of Depersonalization Disorder, a condition that makes people feel estranged and detached from their self, body, and the world.

The first prior: From co-embodiment to co-homeostasis in early life

The idea that our perceptions in the here and now are influenced by prior events and experiences has recently received substantial support and attention from the proponents of the Predictive Processing (PP) and Active Inference framework in philosophy and computational neuroscience. In this paper we look at how perceptual experiences get off the ground from the outset, in utero. One basic yet overlooked aspect of current PP approaches is that human organisms first develop within another human body. Crucially, while not all humans will have the experience of being pregnant or carrying a baby, the experience of being carried and growing within another person's body is universal. Specifically, we focus on the development of minimal selfhood in utero as a process co-embodiment and co-homeostasis, and highlight their close relationship. We conclude with some implications on several critical questions fuelling current debates on the nature of conscious experiences, minimal self and social cognition.

Confidence in subjective pain is predicted by reaction time during decision making

Self-report is the gold standard for measuring pain. However, decisions about pain can vary substantially within and between individuals. We measured whether self-reported pain is accompanied by metacognition and variations in confidence, similar to perceptual decision-making in other modalities. Eighty healthy volunteers underwent acute thermal pain and provided pain ratings followed by confidence judgments on continuous visual analogue scales. We investigated whether eye fixations and reaction time during pain rating might serve as implicit markers of confidence. Confidence varied across trials and increased confidence was associated with faster pain rating reaction times. The association between confidence and fixations varied across individuals as a function of the reliability of individuals’ association between temperature and pain. Taken together, this work indicates that individuals can provide metacognitive judgments of pain and extends research on confidence in perceptual decision-making to pain.

Detecting (Un)seen Change: The Neural Underpinnings of (Un)conscious Prediction Errors

Detecting changes in the environment is fundamental for our survival. According to predictive coding theory, detecting these irregularities relies both on incoming sensory information and our top-down prior expectations (or internal generative models) about the world. Prediction errors (PEs), detectable in event-related potentials (ERPs), occur when there is a mismatch between the sensory input and our internal model (i.e., a surprise event). Many changes occurring in our environment are irrelevant for survival and may remain unseen. Such changes, even if subtle, can nevertheless be detected by the brain without emerging into consciousness. What remains unclear is how these changes are processed in the brain at the network level. Here, we used a visual oddball paradigm in which participants engaged in a central letter task during electroencephalographic (EEG) recordings while presented with task-irrelevant high- or low-coherence background, random-dot motion. Critically, once in a while, the direction of the dots changed. After the EEG session, we confirmed that changes in motion direction at high- and low-coherence were visible and invisible, respectively, using psychophysical measurements. ERP analyses revealed that changes in motion direction elicited PE regardless of the visibility, but with distinct spatiotemporal patterns. To understand these responses, we applied dynamic causal modeling (DCM) to the EEG data. Bayesian Model Averaging showed visible PE relied on a release from adaptation (repetition suppression) within bilateral MT+, whereas invisible PE relied on adaptation at bilateral V1 (and left MT+). Furthermore, while feedforward upregulation was present for invisible PE, the visible change PE also included downregulation of feedback between right MT+ to V1. Our findings reveal a complex interplay of modulation in the generative network models underlying visible and invisible motion changes.

Transcranial Magnetic Stimulation-Induced Motor Cortex Activity Influences Visual Awareness Judgments

The influence of non-visual information on visual awareness judgments has recently gained substantial interest. Using single-pulse transcranial magnetic stimulation (TMS), we investigate the potential contribution of evidence from the motor system to judgment of visual awareness. We hypothesized that TMS-induced activity in the primary motor cortex (M1) would increase reported visual awareness as compared to the control condition. Additionally, we investigated whether TMS-induced motor-evoked potential (MEP) could measure accumulated evidence for stimulus perception. Following stimulus presentation and TMS, participants first rated their visual awareness verbally using the Perceptual Awareness Scale (PAS), after which they responded manually to a Gabor orientation identification task. Delivering TMS to M1 resulted in higher average awareness ratings as compared to the control condition, in both correct and incorrect identification task response trials, when the hand with which participants responded was contralateral to the stimulated hemisphere (TMS-response-congruent trials). This effect was accompanied by longer PAS response times (RTs), irrespective of the congruence between TMS and identification response. Moreover, longer identification RTs were observed in TMS-response-congruent trials in the M1 condition as compared to the control condition. Additionally, the amplitudes of MEPs were related to the awareness ratings when response congruence was taken into account. We argue that MEP can serve as an indirect measure of evidence accumulated for stimulus perception and that longer PAS RTs and higher amplitudes of MEPs in the M1 condition reflect integration of additional evidence with visual awareness judgment. In conclusion, we advocate that motor activity influences perceptual awareness judgments.

First-person view of one's body in immersive virtual reality: Influence on episodic memory

Episodic memories (EMs) are recollections of contextually rich and personally relevant past events. EM has been linked to the sense of self, allowing one to mentally travel back in subjective time and re-experience past events. However, the sense of self has recently been linked to online multisensory processing and bodily self-consciousness (BSC). It is currently unknown whether EM depends on BSC mechanisms. Here, we used a new immersive virtual reality (VR) system that maintained the perceptual richness of life episodes and fully controlled the experimental stimuli during encoding and retrieval, including the participant’s body. Our data reveal a classical EM finding, which shows that memory for complex real-life like scenes decays over time. However, here we also report a novel finding that delayed retrieval performance can be enhanced when participants view their body as part of the virtual scene during encoding. This body effect was not observed when no virtual body or a moving control object was shown, thereby linking the sense of self, and BSC in particular, to EMs. The present VR methodology and the present behavioral findings will enable to study key aspects of EM in healthy participants and may be especially beneficial for the restoration of self-relevant memories in future experiments.

Tactile motion biases visual motion perception in binocular rivalry

There is an ongoing debate whether or not multisensory interactions require awareness of the sensory signals. Static visual and tactile stimuli have been shown to influence each other even in the absence of visual awareness. However, it is unclear if this finding generalizes to dynamic contexts. In the present study, we presented visual and tactile motion stimuli and induced fluctuations of visual awareness by means of binocular rivalry: two gratings which drifted in opposite directions were displayed, one to each eye. One visual motion stimulus dominated and reached awareness while the other visual stimulus was suppressed from awareness. Tactile motion stimuli were presented at random time points during the visual stimulation. The motion direction of a tactile stimulus always matched the direction of one of the concurrently presented visual stimuli. The visual gratings were differently tinted, and participants reported the color of the currently seen stimulus. Tactile motion delayed perceptual switches that ended dominance periods of congruently moving visual stimuli compared to switches during visual-only stimulation. In addition, tactile motion fostered the return to dominance of suppressed, congruently moving visual stimuli, but only if the tactile motion started at a late stage of the ongoing visual suppression period. At later stages, perceptual suppression is typically decreasing. These results suggest that visual awareness facilitates but does not gate multisensory interactions between visual and tactile motion signals.

Attention periodically samples competing stimuli during binocular rivalry

The attentional sampling hypothesis suggests that attention rhythmically enhances sensory processing when attending to a single (~8 Hz), or multiple (~4 Hz) objects. Here, we investigated whether attention samples sensory representations that are not part of the conscious percept during binocular rivalry. When crossmodally cued toward a conscious image, subsequent changes in consciousness occurred at ~8 Hz, consistent with the rates of undivided attentional sampling. However, when attention was cued toward the suppressed image, changes in consciousness slowed to ~3.5 Hz, indicating the division of attention away from the conscious visual image. In the electroencephalogram, we found that at attentional sampling frequencies, the strength of inter-trial phase-coherence over fronto-temporal and parieto-occipital regions correlated with changes in perception. When cues were not task-relevant, these effects disappeared, confirming that perceptual changes were dependent upon the allocation of attention, and that attention can flexibly sample away from a conscious image in a task-dependent manner.

Invisible Flashes Alter Perceived Sound Location

Information integration across the senses is fundamental for effective interactions with our environment. The extent to which signals from different senses can interact in the absence of awareness is controversial. Combining the spatial ventriloquist illusion and dynamic continuous flash suppression (dCFS), we investigated in a series of two experiments whether visual signals that observers do not consciously perceive can influence spatial perception of sounds. Importantly, dCFS obliterated visual awareness only on a fraction of trials allowing us to compare spatial ventriloquism for physically identical flashes that were judged as visible or invisible. Our results show a stronger ventriloquist effect for visible than invisible flashes. Critically, a robust ventriloquist effect emerged also for invisible flashes even when participants were at chance when locating the flash. Collectively, our findings demonstrate that signals that we are not aware of in one sensory modality can alter spatial perception of signals in another sensory modality.