Functional imaging of human crossmodal identification and object recognition

An evaluation of recent advancements in biological sensory organ-inspired neuromorphically tuned biomimetic devices

In the field of neuroscience, significant progress has been made regarding how the brain processes information. Unlike computer processors, the brain comprises neurons and synapses instead of memory blocks and transistors. Despite advancements in artificial neural networks, a complete understanding concerning brain functions remains elusive. For example, to achieve more accurate neuron replication, we must better understand signal transmission during synaptic processes, neural network tunability, and the creation of nanodevices featuring neurons and synapses. This study discusses the latest algorithms utilized in neuromorphic systems, the production of synaptic devices, differences between single and multisensory gadgets, recent advances in multisensory devices, and the promising research opportunities available in this field. We also explored the ability of an artificial synaptic device to mimic biological neural systems across diverse applications. Despite existing challenges, neuroscience-based computing technology holds promise for attracting scientists seeking to enhance solutions and augment the capabilities of neuromorphic devices, thereby fostering future breakthroughs in algorithms and the widespread application of cutting-edge technologies.

HD-tDCS to the lateral occipital complex improves haptic object recognition

High-definition transcranial direct current stimulation (HD-tDCS) is a non-invasive brain stimulation technique that has been shown to be safe and effective in modulating neuronal activity. The present study investigates the effect of anodal HD-tDCS on haptic object perception and memory through stimulation of the lateral occipital complex (LOC), a structure that has been shown to be involved in both visual and haptic object recognition. In this single-blind, sham-controlled, between-subjects study, blindfolded healthy, sighted participants used their right (dominant) hand to perform haptic discrimination and recognition tasks with 3D-printed, novel objects called "Greebles" while receiving 20 min of 2 milliamp (mA) anodal stimulation (or sham) to the left or right LOC. Compared to sham, those who received left LOC stimulation (contralateral to the hand used) showed an improvement in haptic object recognition but not discrimination-a finding that was evident from the start of the behavioral tasks. A second experiment showed that this effect was not observed with right LOC stimulation (ipsilateral to the hand used). These results suggest that HD-tDCS to the left LOC can improve recognition of objects perceived via touch. Overall, this work sheds light on the LOC as a multimodal structure that plays a key role in object recognition in both the visual and haptic modalities.

Longstanding Auditory Sensory and Semantic Differences in Preterm Born Children

More than 10% of births are preterm, and the long-term consequences on sensory and semantic processing of non-linguistic information remain poorly understood. 17 very preterm-born children (born at < 33 weeks gestational age) and 15 full-term controls were tested at 10 years old with an auditory object recognition task, while 64-channel auditory evoked potentials (AEPs) were recorded. Sounds consisted of living (animal and human vocalizations) and manmade objects (e.g. household objects, instruments, and tools). Despite similar recognition behavior, AEPs strikingly differed between full-term and preterm children. Starting at 50ms post-stimulus onset, AEPs from preterm children differed topographically from their full-term counterparts. Over the 108-224ms post-stimulus period, full-term children showed stronger AEPs in response to living objects, whereas preterm born children showed the reverse pattern; i.e. stronger AEPs in response to manmade objects. Differential brain activity between semantic categories could reliably classify children according to their preterm status. Moreover, this opposing pattern of differential responses to semantic categories of sounds was also observed in source estimations within a network of occipital, temporal and frontal regions. This study highlights how early life experience in terms of preterm birth shapes sensory and object processing later on in life.

Superior Attentional Efficiency of Auditory Cue via the Ventral Auditory-thalamic Pathway

Auditory commands are often executed more efficiently than visual commands. However, empirical evidence on the underlying behavioral and neural mechanisms remains scarce. In two experiments, we manipulated the delivery modality of informative cues and the prediction violation effect and found consistently enhanced RT benefits for the matched auditory cues compared with the matched visual cues. At the neural level, when the bottom-up perceptual input matched the prior prediction induced by the auditory cue, the auditory-thalamic pathway was significantly activated. Moreover, the stronger the auditory-thalamic connectivity, the higher the behavioral benefits of the matched auditory cue. When the bottom-up input violated the prior prediction induced by the auditory cue, the ventral auditory pathway was specifically involved. Moreover, the stronger the ventral auditory-prefrontal connectivity, the larger the behavioral costs caused by the violation of the auditory cue. In addition, the dorsal frontoparietal network showed a supramodal function in reacting to the violation of informative cues irrespective of the delivery modality of the cue. Taken together, the results reveal novel behavioral and neural evidence that the superior efficiency of the auditory cue is twofold: The auditory-thalamic pathway is associated with improvements in task performance when the bottom-up input matches the auditory cue, whereas the ventral auditory-prefrontal pathway is involved when the auditory cue is violated.

Learning and navigating digitally rendered haptic spatial layouts

Learning spatial layouts and navigating through them rely not simply on sight but rather on multisensory processes, including touch. Digital haptics based on ultrasounds are effective for creating and manipulating mental images of individual objects in sighted and visually impaired participants. Here, we tested if this extends to scenes and navigation within them. Using only tactile stimuli conveyed via ultrasonic feedback on a digital touchscreen (i.e., a digital interactive map), 25 sighted, blindfolded participants first learned the basic layout of an apartment based on digital haptics only and then one of two trajectories through it. While still blindfolded, participants successfully reconstructed the haptically learned 2D spaces and navigated these spaces. Digital haptics were thus an effective means to learn and translate, on the one hand, 2D images into 3D reconstructions of layouts and, on the other hand, navigate actions within real spaces. Digital haptics based on ultrasounds represent an alternative learning tool for complex scenes as well as for successful navigation in previously unfamiliar layouts, which can likely be further applied in the rehabilitation of spatial functions and mitigation of visual impairments.

The effect of multisensory semantic congruency on unisensory object recognition in schizophrenia

Multisensory, as opposed to unisensory processing of stimuli, has been found to enhance the performance (e.g., reaction time, accuracy, and discrimination) of healthy individuals across various tasks. However, this enhancement is not as pronounced in patients with schizophrenia (SZ), indicating impaired multisensory integration (MSI) in these individuals. To the best of our knowledge, no study has yet investigated the impact of MSI deficits in the context of working memory, a domain highly reliant on multisensory processing and substantially impaired in schizophrenia. To address this research gap, we employed two adopted versions of the continuous object recognition task to investigate the effect of single-trail multisensory encoding on subsequent object recognition in 21 schizophrenia patients and 21 healthy controls (HC). Participants were tasked with discriminating between initial and repeated presentations. For the initial presentations, half of the stimuli were audiovisual pairings, while the other half were presented unimodal. The task-relevant stimuli were then presented a second time in a unisensory manner (either auditory stimuli in the auditory task or visual stimuli in the visual task). To explore the impact of semantic context on multisensory encoding, half of the audiovisual pairings were selected to be semantically congruent, while the remaining pairs were not semantically related to each other. Consistent with prior studies, our findings demonstrated that the impact of single-trial multisensory presentation during encoding remains discernible during subsequent object recognition. This influence could be distinguished based on the semantic congruity between the auditory and visual stimuli presented during the encoding. This effect was more robust in the auditory task. In the auditory task, when congruent multisensory pairings were encoded, both participant groups demonstrated a multisensory facilitation effect. This effect resulted in improved accuracy and RT performance. Regarding incongruent audiovisual encoding, as expected, HC did not demonstrate an evident multisensory facilitation effect on memory performance. In contrast, SZs exhibited an atypically accelerated reaction time during the subsequent auditory object recognition. Based on the predictive coding model we propose that this observed deviations indicate a reduced semantic modulatory effect and anomalous predictive errors signaling, particularly in the context of conflicting cross-modal sensory inputs in SZ.

The Physiology of Cognition in Autism Spectrum Disorder: Current and Future Challenges

Cognitive impairment is among the most challenging characteristics of autism spectrum disorder (ASD). Although ASD is one of the common neurodevelopmental disorders, we are still behind in diagnosing and treating cognitive impairment in ASD. Cognitive impairment in ASD varies, meaning it could be at the sensory perception level to cognitive processing, learning, and memory. There are no diagnostic criteria for cognitive impairment that are specific to ASD. The leading causes of cognitive impairment in ASD could be neurological, immune, and gastrointestinal dysfunction. Immune dysfunction might lead to neuroinflammation, affecting neural connectivity, glutamate/gamma-aminobutyric acid (GABA) balance, and plasticity. The gut-brain axes are essential in the developing brain. Special retinal changes have recently been detected in ASD, which need clinical investigation to find their possible role in early diagnosis. Early intervention is crucial for ASD cognitive dysfunction. Due to the heterogeneity of the disease, the clinical manifestation of ASD makes it difficult for clinicians to develop gold-standard diagnostic and therapeutic criteria. We suggest a triad for diagnosis, which includes clinical tests for immune and gastrointestinal dysfunction biomarkers, clinical examination for the retina, and an objective neurocognitive evaluation for ASD, and to develop a treatment strategy involving these three aspects. Developing clear treatment criteria for cognitive impairment for ASD would improve the quality of life of ASD people and their caregivers and would delay or prevent dementia-related disorders in ASD people.

Aberrant Intrinsic Brain Network Functional Connectivity During a Face-Matching Task in Women Diagnosed With Premenstrual Dysphoric Disorder

BACKGROUND

Premenstrual dysphoric disorder (PMDD) is characterized by affective, cognitive, and physical symptoms, suggesting alterations at the brain network level. Women with PMDD demonstrate aberrant discrimination of facial emotions during the luteal phase of the menstrual cycle and altered reactivity to emotional stimuli. However, previous studies assessing emotional task-related brain reactivity using region-of-interest or whole-brain analysis have reported conflicting findings. Therefore, we utilized both region-of-interest task-reactivity and seed-voxel functional connectivity (FC) approaches to test for differences in the default mode network, salience network, and central executive network between women with PMDD and control participants during an emotional-processing task that yields an optimal setup for investigating brain network changes in PMDD.

METHODS

Twenty-four women with PMDD and 27 control participants were classified according to the Daily Record of Severity of Problems. Participants underwent functional magnetic resonance imaging scans while completing the emotional face-matching task during the midfollicular and late-luteal phases of their menstrual cycle.

RESULTS

No significant between-group differences in brain reactivity were found using region-of-interest analysis. In the FC analysis, a main effect of diagnosis was found showing decreased default mode network connectivity, increased salience network connectivity, and decreased central executive network connectivity in women with PMDD compared with control participants. A significant interaction between menstrual cycle phase and diagnosis was found in the central executive network for right posterior parietal cortex and left inferior lateral occipital cortex connectivity. A post hoc analysis revealed stronger FC during the midfollicular than the late-luteal phase of PMDD.

CONCLUSIONS

Aberrant FC in the 3 brain networks involved in PMDD may indicate vulnerability to experience affective and cognitive symptoms of the disorder.

Elevated Insulin Levels Engage the Salience Network during Multisensory Perception

INTRODUCTION

Brain insulin reactivity has been reported in connection with systematic energy metabolism, enhancement in cognition, olfactory sensitivity, and neuroendocrine circuits. High receptor densities exist in regions important for sensory processing. The main aim of the study was to examine whether intranasal insulin would modulate the activity of areas in charge of olfactory-visual integration.

METHODS

As approach, a placebo-controlled double-blind within crossover design was chosen. The experiments were conducted in a research unit of a university hospital. On separate mornings, twenty-six healthy normal-weight males aged between 19 and 31 years received either 40 IU intranasal insulin or placebo vehicle. Subsequently, they underwent 65 min of functional magnetic resonance imaging whilst performing an odor identification task. Functional brain activations of olfactory, visual, and multisensory integration as well as insulin versus placebo were assessed. Regarding the odor identification task, reaction time, accuracy, pleasantness, and intensity measurements were taken to examine the role of integration and treatment. Blood samples were drawn to control for peripheral hormone concentrations.

RESULTS

Intranasal insulin administration during olfactory-visual stimulation revealed strong bilateral engagement of frontoinsular cortices, anterior cingulate, prefrontal cortex, mediodorsal thalamus, striatal, and hippocampal regions (p ≤ 0.001 familywise error [FWE] corrected). In addition, the integration contrast showed increased activity in left intraparietal sulcus, left inferior frontal gyrus, left superior frontal gyrus, and left middle frontal gyrus (p ≤ 0.013 FWE corrected).

CONCLUSIONS

Intranasal insulin application in lean men led to enhanced activation in multisensory olfactory-visual integration sites and salience hubs which indicates stimuli valuation modulation. This effect can serve as a basis for understanding the connection of intracerebral insulin and olfactory-visual processing.

Benefit of visual speech information for word comprehension in post-stroke aphasia

Aphasia is a language disorder that often involves speech comprehension impairments affecting communication. In face-to-face settings, speech is accompanied by mouth and facial movements, but little is known about the extent to which they benefit aphasic comprehension. This study investigated the benefit of visual information accompanying speech for word comprehension in people with aphasia (PWA) and the neuroanatomic substrates of any benefit. Thirty-six PWA and 13 neurotypical matched control participants performed a picture-word verification task in which they indicated whether a picture of an animate/inanimate object matched a subsequent word produced by an actress in a video. Stimuli were either audiovisual (with visible mouth and facial movements) or auditory-only (still picture of a silhouette) with audio being clear (unedited) or degraded (6-band noise-vocoding). We found that visual speech information was more beneficial for neurotypical participants than PWA, and more beneficial for both groups when speech was degraded. A multivariate lesion-symptom mapping analysis for the degraded speech condition showed that lesions to superior temporal gyrus, underlying insula, primary and secondary somatosensory cortices, and inferior frontal gyrus were associated with reduced benefit of audiovisual compared to auditory-only speech, suggesting that the integrity of these fronto-temporo-parietal regions may facilitate cross-modal mapping. These findings provide initial insights into our understanding of the impact of audiovisual information on comprehension in aphasia and the brain regions mediating any benefit.

Age-Related Changes to Multisensory Integration and Audiovisual Speech Perception

Multisensory integration is essential for the quick and accurate perception of our environment, particularly in everyday tasks like speech perception. Research has highlighted the importance of investigating bottom-up and top-down contributions to multisensory integration and how these change as a function of ageing. Specifically, perceptual factors like the temporal binding window and cognitive factors like attention and inhibition appear to be fundamental in the integration of visual and auditory information-integration that may become less efficient as we age. These factors have been linked to brain areas like the superior temporal sulcus, with neural oscillations in the alpha-band frequency also being implicated in multisensory processing. Age-related changes in multisensory integration may have significant consequences for the well-being of our increasingly ageing population, affecting their ability to communicate with others and safely move through their environment; it is crucial that the evidence surrounding this subject continues to be carefully investigated. This review will discuss research into age-related changes in the perceptual and cognitive mechanisms of multisensory integration and the impact that these changes have on speech perception and fall risk. The role of oscillatory alpha activity is of particular interest, as it may be key in the modulation of multisensory integration.

Similar object shape representation encoded in the inferolateral occipitotemporal cortex of sighted and early blind people

We can sense an object's shape by vision or touch. Previous studies suggested that the inferolateral occipitotemporal cortex (ILOTC) implements supramodal shape representations as it responds more to seeing or touching objects than shapeless textures. However, such activation in the anterior portion of the ventral visual pathway could be due to the conceptual representation of an object or visual imagery triggered by touching an object. We addressed these possibilities by directly comparing shape and conceptual representations of objects in early blind (who lack visual experience/imagery) and sighted participants. We found that bilateral ILOTC in both groups showed stronger activation during a shape verification task than during a conceptual verification task made on the names of the same manmade objects. Moreover, the distributed activity in the ILOTC encoded shape similarity but not conceptual association among objects. Besides the ILOTC, we also found shape representation in both groups' bilateral ventral premotor cortices and intraparietal sulcus (IPS), a frontoparietal circuit relating to object grasping and haptic processing. In contrast, the conceptual verification task activated both groups' left perisylvian brain network relating to language processing and, interestingly, the cuneus in early blind participants only. The ILOTC had stronger functional connectivity to the frontoparietal circuit than to the left perisylvian network, forming a modular structure specialized in shape representation. Our results conclusively support that the ILOTC selectively implements shape representation independently of visual experience, and this unique functionality likely comes from its privileged connection to the frontoparietal haptic circuit.

Auditory scene context facilitates visual recognition of objects in consistent visual scenes

Visual object recognition is facilitated by contextually consistent scenes in which the object is embedded. Scene gist representations extracted from the scenery backgrounds yield this scene consistency effect. Here we examined whether the scene consistency effect is specific to the visual domain or if it is crossmodal. Through four experiments, the accuracy of the naming of briefly presented visual objects was assessed. In each trial, a 4-s sound clip was presented and a visual scene containing the target object was briefly shown at the end of the sound clip. In a consistent sound condition, an environmental sound associated with the scene in which the target object typically appears was presented (e.g., forest noise for a bear target object). In an inconsistent sound condition, a sound clip contextually inconsistent with the target object was presented (e.g., city noise for a bear). In a control sound condition, a nonsensical sound (sawtooth wave) was presented. When target objects were embedded in contextually consistent visual scenes (Experiment 1: a bear in a forest background), consistent sounds increased object-naming accuracy. In contrast, sound conditions did not show a significant effect when target objects were embedded in contextually inconsistent visual scenes (Experiment 2: a bear in a pedestrian crossing background) or in a blank background (Experiments 3 and 4). These results suggested that auditory scene context has weak or no direct influence on visual object recognition. It seems likely that consistent auditory scenes indirectly facilitate visual object recognition by promoting visual scene processing.

Long-term memory representations for audio-visual scenes

In this study, we investigated the nature of long-term memory representations for naturalistic audio-visual scenes. Whereas previous research has shown that audio-visual scenes are recognized more accurately than their unimodal counterparts, it remains unclear whether this benefit stems from audio-visually integrated long-term memory representations or a summation of independent retrieval cues. We tested two predictions for audio-visually integrated memory representations. First, we used a modeling approach to test whether recognition performance for audio-visual scenes is more accurate than would be expected from independent retrieval cues. This analysis shows that audio-visual integration is not necessary to explain the benefit of audio-visual scenes relative to purely auditory or purely visual scenes. Second, we report a series of experiments investigating the occurrence of study-test congruency effects for unimodal and audio-visual scenes. Most importantly, visually encoded information was immune to additional auditory information presented during testing, whereas auditory encoded information was susceptible to additional visual information presented during testing. This renders a true integration of visual and auditory information in long-term memory representations unlikely. In sum, our results instead provide evidence for visual dominance in long-term memory. Whereas associative auditory information is capable of enhancing memory performance, the long-term memory representations appear to be primarily visual.

Do our hands see what our eyes see? Investigating spatial and haptic abilities

Spatial abilities (SAs) are cognitive resources used to mentally manipulate representations of objects to solve problems. Haptic abilities (HAs) represent tactile interactions with real-world objects transforming somatic information into mental representations. Both are proposed to be factors in anatomy education, yet relationships between SAs and HAs remain unknown. The objective of the current study was to explore SA-HA interactions. A haptic ability test (HAT) was developed based on the mental rotations test (MRT) with three-dimensional (3D) objects. The HAT was undertaken in three sensory conditions: (1) sighted, (2) sighted with haptics, and (3) haptics. Participants (n = 22; 13 females, 9 males) completed the MRT and were categorized into high spatial abilities (HSAs) (n = 12, mean± standard deviation: 13.7 ± 3.0) and low spatial abilities (LSAs) (n = 10, 5.6 ± 2.0) based on score distributions about the overall mean. Each SA group's HAT scores were compared across the three sensory conditions. Spearman's correlation coefficients between MRT and HAT scores indicated a statistically significant correlation in sighted condition (r = 0.553, p = 0.015) but were not significant in the sighted with haptics (r = 0.0.078, p = 0.212) and haptics (r = 0.043, p = 0.279) conditions. These data suggest HAs appear unrelated to SAs. With haptic exploration, LSA HAT scores were compensated; comparing HSA with LSA: sighted with haptics [median (lower and upper quartiles): 12 (12,13) vs. 12 (11,13), p = 0.254], and haptics [12 (11,13) vs. 12 (10,12), p = 0.381] conditions. Migrations to online anatomy teaching may unwittingly remove important sensory modalities from the learner. Understanding learner behaviors and performance when haptic inputs are removed from the learning environment represents valuable insight informing future anatomy curriculum and resource development.

The impact of COVID-19 on the everyday life of blind and sighted individuals

The COVID-19 pandemic caused unexpected and unavoidable changes in daily life worldwide. Governments and communities found ways to mitigate the impact of these changes, but many solutions were inaccessible to people with visual impairments. This work aimed to investigate how blind individuals subjectively experienced the restrictions and isolation caused by the COVID-19 pandemic. To this end, a group of twenty-seven blind and seventeen sighted people took part in a survey addressing how COVID-19 impacted life practically and psychologically, how it affected their daily habits, and how it changed their experiences of themselves and others. Results demonstrated that both sighted and blind individuals had a hard time adapting to the new situation. However, while sighted people struggled more with personal and social aspects, the frustration of the blind population derived mostly from more practical and logistical issues. Likely as consequences, results showed that blind people engaged more in their inner life and experienced fear and anger as main emotions. This study suggests that changes in life associated with COVID-19 have been subjectively experienced differently based on the presence or not of blindness, and that tailored future interventions should be considered to take care of the different needs of blind individuals.

Early blindness modulates haptic object recognition

Haptic object recognition is usually an efficient process although slower and less accurate than its visual counterpart. The early loss of vision imposes a greater reliance on haptic perception for recognition compared to the sighted. Therefore, we may expect that congenitally blind persons could recognize objects through touch more quickly and accurately than late blind or sighted people. However, the literature provided mixed results. Furthermore, most of the studies on haptic object recognition focused on performance, devoting little attention to the exploration procedures that conducted to that performance. In this study, we used iCube, an instrumented cube recording its orientation in space as well as the location of the points of contact on its faces. Three groups of congenitally blind, late blind and age and gender-matched blindfolded sighted participants were asked to explore the cube faces where little pins were positioned in varying number. Participants were required to explore the cube twice, reporting whether the cube was the same or it differed in pins disposition. Results showed that recognition accuracy was not modulated by the level of visual ability. However, congenitally blind touched more cells simultaneously while exploring the faces and changed more the pattern of touched cells from one recording sample to the next than late blind and sighted. Furthermore, the number of simultaneously touched cells negatively correlated with exploration duration. These findings indicate that early blindness shapes haptic exploration of objects that can be held in hands.

Making sense of sensory language: Acquisition of sensory knowledge by individuals with congenital sensory impairments

The present article provides a narrative review on how language communicates sensory information and how knowledge of sight and sound develops in individuals born deaf or blind. Studying knowledge of the perceptually inaccessible sensory domain for these populations offers a lens into how humans learn about that which they cannot perceive. We first review the linguistic strategies within language that communicate sensory information. Highlighting the power of language to shape knowledge, we next review the detailed knowledge of sensory information by individuals with congenital sensory impairments, limitations therein, and neural representations of imperceptible phenomena. We suggest that the acquisition of sensory knowledge is supported by language, experience with multiple perceptual domains, and cognitive and social abilities which mature over the first years of life, both in individuals with and without sensory impairment. We conclude by proposing a developmental trajectory for acquiring sensory knowledge in the absence of sensory perception.

The Spatiotemporal Neural Dynamics of Intersensory Attention Capture of Salient Stimuli: A Large-Scale Auditory-Visual Modeling Study

The spatiotemporal dynamics of the neural mechanisms underlying endogenous (top-down) and exogenous (bottom-up) attention, and how attention is controlled or allocated in intersensory perception are not fully understood. We investigated these issues using a biologically realistic large-scale neural network model of visual-auditory object processing of short-term memory. We modeled and incorporated into our visual-auditory object-processing model the temporally changing neuronal mechanisms for the control of endogenous and exogenous attention. The model successfully performed various bimodal working memory tasks, and produced simulated behavioral and neural results that are consistent with experimental findings. Simulated fMRI data were generated that constitute predictions that human experiments could test. Furthermore, in our visual-auditory bimodality simulations, we found that increased working memory load in one modality would reduce the distraction from the other modality, and a possible network mediating this effect is proposed based on our model.

Whether attentional loads influence audiovisual integration depends on semantic associations

Neuronal studies have shown that selectively attending to a common object in one sensory modality results in facilitated processing of that object's representations in the ignored sensory modality. Thus, the audiovisual (AV) integration of common objects can be observed under modality-specific selective attention. However, little is known about whether this AV integration can also occur under increased attentional load conditions. Additionally, whether semantic associations between multisensory features of common objects modulate the influence of increased attentional loads on this cross-modal integration remains unknown. In the present study, participants completed an AV integration task (ignored auditory stimuli) under various attentional load conditions: no load, low load, and high load. The semantic associations between AV stimuli were composed of animal pictures presented concurrently with semantically congruent, semantically incongruent, or semantically unrelated auditory stimuli. Our results demonstrated that attentional loads did not disrupt the integration of semantically congruent AV stimuli but suppressed the potential alertness effects induced by incongruent or unrelated auditory stimuli under the condition of modality-specific selective attention. These findings highlight the critical role of semantic association between AV stimuli in modulating the effect of attentional loads on the AV integration of modality-specific selective attention.

Abnormal cortical responses elicited by audiovisual movies in patients with autism spectrum disorder with atypical sensory behavior: A magnetoencephalographic study

BACKGROUND

Atypical sensory behavior disrupts behavioral adaptation in children with autism spectrum disorder (ASD); however, neural correlates of sensory dysfunction using magnetoencephalography (MEG) remain unclear.

METHOD

We used MEG to measure the cortical activation elicited by visual (uni)/audiovisual (multisensory) movies in 46 children (7-14 years) were included in final analysis: 13 boys with atypical audiovisual behavior in ASD (AAV+), 10 without this condition, and 23 age-matched typically developing boys.

RESULTS

The AAV+ group demonstrated an increase in the cortical activation in the bilateral insula in response to unisensory movies and in the left occipital, right superior temporal sulcus (rSTS), and temporal regions to multisensory movies. These increased responses were correlated with severity of the sensory impairment. Increased theta-low gamma oscillations were observed in the rSTS in AAV+.

CONCLUSION

The findings suggest that AAV is attributed to atypical neural networks centered on the rSTS.

How are visemes and graphemes integrated with speech sounds during spoken word recognition? ERP evidence for supra-additive responses during audiovisual compared to auditory speech processing

Both visual articulatory gestures and orthography provide information on the phonological content of speech. This EEG study investigated the integration between speech and these two visual inputs. A comparison of skilled readers' brain responses elicited by a spoken word presented alone versus synchronously with a static image of a viseme or a grapheme of the spoken word's onset showed that while neither visual input induced audiovisual integration on N1 acoustic component, both led to a supra-additive integration on P2, with a stronger integration between speech and graphemes on left-anterior electrodes. This pattern persisted in P350 time-window and generalized to all electrodes. The finding suggests a strong impact of spelling knowledge on phonetic processing and lexical access. It also indirectly indicates that the dynamic and predictive value present in natural lip movements but not in static visemes is particularly critical to the contribution of visual articulatory gestures to speech processing.

Evidence for an A-Modal Number Sense: Numerosity Adaptation Generalizes Across Visual, Auditory, and Tactile Stimuli

Humans and other species share a perceptual mechanism dedicated to the representation of approximate quantities that allows to rapidly and reliably estimate the numerosity of a set of objects: an Approximate Number System (ANS). Numerosity perception shows a characteristic shared by all primary visual features: it is susceptible to adaptation. As a consequence of prolonged exposure to a large/small quantity (“adaptor”), the apparent numerosity of a subsequent (“test”) stimulus is distorted yielding a robust under- or over-estimation, respectively. Even if numerosity adaptation has been reported across several sensory modalities (vision, audition, and touch), suggesting the idea of a central and a-modal numerosity processing system, evidence for cross-modal effects are limited to vision and audition, two modalities that are known to preferentially encode sensory stimuli in an external coordinate system. Here we test whether numerosity adaptation for visual and auditory stimuli also distorts the perceived numerosity of tactile stimuli (and vice-versa) despite touch being a modality primarily coded in an internal (body-centered) reference frame. We measured numerosity discrimination of stimuli presented sequentially after adaptation to series of either few (around 2 Hz; low adaptation) or numerous (around 8 Hz; high adaptation) impulses for all possible combinations of visual, auditory, or tactile adapting and test stimuli. In all cases, adapting to few impulses yielded a significant overestimation of the test numerosity with the opposite occurring as a consequence of adaptation to numerous stimuli. The overall magnitude of adaptation was robust (around 30%) and rather similar for all sensory modality combinations. Overall, these findings support the idea of a truly generalized and a-modal mechanism for numerosity representation aimed to process numerical information independently from the sensory modality of the incoming signals.

Reading-Related Brain Changes in Audiovisual Processing: Cross-Sectional and Longitudinal MEG Evidence

The ability to establish associations between visual objects and speech sounds is essential for human reading. Understanding the neural adjustments required for acquisition of these arbitrary audiovisual associations can shed light on fundamental reading mechanisms and help reveal how literacy builds on pre-existing brain circuits. To address these questions, the present longitudinal and cross-sectional MEG studies characterize the temporal and spatial neural correlates of audiovisual syllable congruency in children (age range, 4-9 years; 22 males and 20 females) learning to read. Both studies showed that during the first years of reading instruction children gradually set up audiovisual correspondences between letters and speech sounds, which can be detected within the first 400 ms of a bimodal presentation and recruit the superior portions of the left temporal cortex. These findings suggest that children progressively change the way they treat audiovisual syllables as a function of their reading experience. This reading-specific brain plasticity implies (partial) recruitment of pre-existing brain circuits for audiovisual analysis.SIGNIFICANCE STATEMENT Linking visual and auditory linguistic representations is the basis for the development of efficient reading, while dysfunctional audiovisual letter processing predicts future reading disorders. Our developmental MEG project included a longitudinal and a cross-sectional study; both studies showed that children's audiovisual brain circuits progressively change as a function of reading experience. They also revealed an exceptional degree of neuroplasticity in audiovisual neural networks, showing that as children develop literacy, the brain progressively adapts so as to better detect new correspondences between letters and speech sounds.

The role of the left and right inferior frontal gyrus in processing metaphoric and unrelated co-speech gestures

Gestures are an integral part of in-person conversations and complement the meaning of the speech they accompany. The neural processing of co-speech gestures is supported by a mostly left-lateralized network of fronto-temporal regions. However, in contrast to iconic gestures, metaphoric as well as unrelated gestures have been found to more strongly engage the left and right inferior frontal gyrus (IFG), respectively. With this study, we conducted the first systematic comparison of all three types of gestures and resulting potential laterality effects. During collection of functional imaging data, 74 subjects were presented with 5 s videos of abstract speech with related metaphoric gestures, concrete speech with related iconic gestures and concrete speech with unrelated gestures. They were asked to judge whether the content of the speech and gesture matched or not. Differential contrasts revealed that both abstract related and concrete unrelated compared to concrete related stimuli elicited stronger activation of the bilateral IFG. Analyses of lateralization indices for IFG activation further showed a left hemispheric dominance for metaphoric gestures and a right hemispheric dominance for unrelated gestures. Our results give support to the hypothesis that the bilateral IFG is activated specifically when processing load for speech-gesture combinations is high. In addition, laterality effects indicate a stronger involvement of the right IFG in mismatch detection and conflict processing, whereas the left IFG performs the actual integration of information from speech and gesture.

Trimodal processing of complex stimuli in inferior parietal cortex is modality-independent

In humans, multisensory mechanisms facilitate object processing through integration of sensory signals that match in their temporal and spatial occurrence as well as their meaning. The generalizability of such integration processes across different sensory modalities is, however, to date not well understood. As such, it remains unknown whether there are cerebral areas that process object-related signals independently of the specific senses from which they arise, and whether these areas show different response profiles depending on the number of sensory channels that carry information. To address these questions, we presented participants with dynamic stimuli that simultaneously emitted object-related sensory information via one, two, or three channels (sight, sound, smell) in the MR scanner. By comparing neural activation patterns between various integration processes differing in type and number of stimulated senses, we showed that the left inferior frontal gyrus and areas within the left inferior parietal cortex were engaged independently of the number and type of sensory input streams. Activation in these areas was enhanced during bimodal stimulation, compared to the sum of unimodal activations, and increased even further during trimodal stimulation. Taken together, our findings demonstrate that activation of the inferior parietal cortex during processing and integration of meaningful multisensory stimuli is both modality-independent and modulated by the number of available sensory modalities. This suggests that the processing demand placed on the parietal cortex increases with the number of sensory input streams carrying meaningful information, likely due to the increasing complexity of such stimuli.

Underconnectivity Between Visual and Salience Networks and Links With Sensory Abnormalities in Autism Spectrum Disorders

OBJECTIVE

The anterior insular cortex (AI), which is a part of the salience network, is critically involved in visual awareness, multisensory perception, and social and emotional processing, among other functions. In children and adolescents with autism spectrum disorders (ASDs), evidence has suggested aberrant functional connectivity (FC) of AI compared with typically developing peers. While recent studies have primarily focused on the functional connections between salience and social networks, much less is known about connectivity between AI and primary sensory regions, including visual areas, and how these patterns may be linked to autism symptomatology.

METHOD

The current investigation implemented functional magnetic resonance imaging to examine resting-state FC patterns of salience and visual networks in children and adolescents with ASDs compared with typically developing controls, and to relate them to behavioral measures.

RESULTS

Functional underconnectivity was found in the ASD group between left AI and bilateral visual cortices. Moreover, in an ASD subgroup with more atypical visual sensory profiles, FC was positively correlated with abnormal social motivational responsivity.

CONCLUSION

Findings of reduced FC between salience and visual networks in ASDs potentially indicate deficient selection of salient information. Moreover, in children and adolescents with ASDs who show strongly atypical visual sensory profiles, connectivity at seemingly more neurotypical levels may be paradoxically associated with greater impairment of social motivation.

Neuro-Clinical Signatures of Language Impairments: A Theoretical Framework for Function-to-structure Mapping in Clinics

In the past decades, neuroscientists and clinicians have collected a considerable amount of data and drastically increased our knowledge about the mapping of language in the brain. The emerging picture from the accumulated knowledge is that there are complex and combinatorial relationships between language functions and anatomical brain regions. Understanding the underlying principles of this complex mapping is of paramount importance for the identification of the brain signature of language and Neuro-Clinical signatures that explain language impairments and predict language recovery after stroke. We review recent attempts to addresses this question of language-brain mapping. We introduce the different concepts of mapping (from diffeomorphic one-to-one mapping to many-to-many mapping). We build those different forms of mapping to derive a theoretical framework where the current principles of brain architectures including redundancy, degeneracy, pluri-potentiality and bow-tie network are described.

Sensory and Behavioral Components of Neocortical Signal Flow in Discrimination Tasks with Short-Term Memory

In the neocortex, each sensory modality engages distinct sensory areas that route information to association areas. Where signal flow converges for maintaining information in short-term memory and how behavior may influence signal routing remain open questions. Using wide-field calcium imaging, we compared cortex-wide neuronal activity in layer 2/3 for mice trained in auditory and tactile tasks with delayed response. In both tasks, mice were either active or passive during stimulus presentation, moving their body or sitting quietly. Irrespective of behavioral strategy, auditory and tactile stimulation activated distinct subdivisions of the posterior parietal cortex, anterior area A and rostrolateral area RL, which held stimulus-related information necessary for the respective tasks. In the delay period, in contrast, behavioral strategy rather than sensory modality determined short-term memory location, with activity converging frontomedially in active trials and posterolaterally in passive trials. Our results suggest behavior-dependent routing of sensory-driven cortical signals flow from modality-specific posterior parietal cortex (PPC) subdivisions to higher association areas.

Semantic Congruency Modulates the Effect of Attentional Load on the Audiovisual Integration of Animate Images and Sounds

Attentional processes play a complex and multifaceted role in the integration of input from different sensory modalities. However, whether increased attentional load disrupts the audiovisual (AV) integration of common objects that involve semantic content remains unclear. Furthermore, knowledge regarding how semantic congruency interacts with attentional load to influence the AV integration of common objects is limited. We investigated these questions by examining AV integration under various attentional-load conditions. AV integration was assessed by adopting an animal identification task using unisensory (animal images and sounds) and AV stimuli (semantically congruent AV objects and semantically incongruent AV objects), while attentional load was manipulated by using a rapid serial visual presentation task. Our results indicate that attentional load did not attenuate the integration of semantically congruent AV objects. However, semantically incongruent animal sounds and images were not integrated (as there was no multisensory facilitation), and the interference effect produced by the semantically incongruent AV objects was reduced by increased attentional-load manipulations. These findings highlight the critical role of semantic congruency in modulating the effect of attentional load on the AV integration of common objects.

Supramodal neural networks support top-down processing of social signals

The perception of facial and vocal stimuli is driven by sensory input and cognitive top‐down influences. Important top‐down influences are attentional focus and supramodal social memory representations. The present study investigated the neural networks underlying these top‐down processes and their role in social stimulus classification. In a neuroimaging study with 45 healthy participants, we employed a social adaptation of the Implicit Association Test. Attentional focus was modified via the classification task, which compared two domains of social perception (emotion and gender), using the exactly same stimulus set. Supramodal memory representations were addressed via congruency of the target categories for the classification of auditory and visual social stimuli (voices and faces). Functional magnetic resonance imaging identified attention‐specific and supramodal networks. Emotion classification networks included bilateral anterior insula, pre‐supplementary motor area, and right inferior frontal gyrus. They were pure attention‐driven and independent from stimulus modality or congruency of the target concepts. No neural contribution of supramodal memory representations could be revealed for emotion classification. In contrast, gender classification relied on supramodal memory representations in rostral anterior cingulate and ventromedial prefrontal cortices. In summary, different domains of social perception involve different top‐down processes which take place in clearly distinguishable neural networks.

Momentary lapses of attention in multisensory environment

Momentary lapses in attention disrupt goal-directed behaviors, and have been associated with increased pre-stimulus activity in the default mode network (DMN). The human brain often encounters multisensory inputs. It remains unknown, however, whether the neural mechanisms underlying attentional lapses are supra-modal or modality-dependent. To answer this question in the present functional magnetic resonance imaging (fMRI) study, we asked participants to respond to either visual or auditory targets in a multisensory paradigm, and focused on the pre-stimulus neural signals underlying attentional lapses, which resulted in impaired task performance, in terms of both delayed RTs and behavioral errors, in different sensory modalities. Behaviorally, mean reaction times (RTs) were equivalent between the visual and auditory modality. At the neural level, increased pre-stimulus neural activity in the majority of the core DMN regions, including medial prefrontal cortex (mPFC), posterior cingulate cortex (PCC), and left angular gyrus (AG), predicted delayed RTs more effectively in the visual than auditory modality. Especially, increased pre-stimulus activity in the mPFC predicted not only delayed RTs but also errors, more effectively in the visual than auditory modality. On the other hand, increased pre-stimulus activity in the anterior precuneus predicted both prolonged RTs and errors more effectively in the auditory than visual modality. Moreover, a supra-modal mechanism was revealed in the left middle temporal gyrus (MTG), which belongs to the posterior DMN. Increased pre-stimulus neural activity in the left MTG predicted impaired task performance in both the visual and auditory modality. Taken together, the core DMN regions manifest vision-dependent mechanisms of attentional lapses while a novel region in the anterior precuneus shows audition-dependent mechanisms of attentional lapses. Moreover, left MTG in the posterior DMN manifests a supra-modal mechanism of attentional lapses, independent of the modality of sensory inputs.

Auditory Enhancement of Illusory Contour Perception

Illusory contours (ICs) are borders that are perceived in the absence of contrast gradients. Until recently, IC processes were considered exclusively visual in nature and presumed to be unaffected by information from other senses. Electrophysiological data in humans indicates that sounds can enhance IC processes. Despite cross-modal enhancement being observed at the neurophysiological level, to date there is no evidence of direct amplification of behavioural performance in IC processing by sounds. We addressed this knowledge gap. Healthy adults ( n = 15) discriminated instances when inducers were arranged to form an IC from instances when no IC was formed (NC). Inducers were low-constrast and masked, and there was continuous background acoustic noise throughout a block of trials. On half of the trials, i.e., independently of IC vs NC, a 1000-Hz tone was presented synchronously with the inducer stimuli. Sound presence improved the accuracy of indicating when an IC was presented, but had no impact on performance with NC stimuli (significant IC presence/absence × Sound presence/absence interaction). There was no evidence that this was due to general alerting or to a speed-accuracy trade-off (no main effect of sound presence on accuracy rates and no comparable significant interaction on reaction times). Moreover, sound presence increased sensitivity and reduced bias on the IC vs NC discrimination task. These results demonstrate that multisensory processes augment mid-level visual functions, exemplified by IC processes. Aside from the impact on neurobiological and computational models of vision, our findings may prove clinically beneficial for low-vision or sight-restored patients.

Task-Irrelevant Sound Corrects Leftward Spatial Bias in Blindfolded Haptic Placement Task

We often rely on our sense of vision for understanding the spatial location of objects around us. If vision cannot be used, one must rely on other senses, such as hearing and touch, in order to build spatial representations. Previous work has found evidence of a leftward spatial bias in visual and tactile tasks. In this study, we sought evidence of this leftward bias in a non-visual haptic object location memory task and assessed the influence of a task-irrelevant sound. In Experiment 1, blindfolded right-handed sighted participants used their non-dominant hand to haptically locate an object on the table, then used their dominant hand to place the object back in its original location. During placement, participants either heard nothing (no-sound condition) or a task-irrelevant repeating tone to the left, right, or front of the room. The results showed that participants exhibited a leftward placement bias on no-sound trials. On sound trials, this leftward bias was corrected; placements were faster and more accurate (regardless of the direction of the sound). One explanation for the leftward bias could be that participants were overcompensating their reach with the right hand during placement. Experiment 2 tested this explanation by switching the hands used for exploration and placement, but found similar results as Experiment 1. A third Experiment found evidence supporting the explanation that sound corrects the leftward bias by heightening attention. Together, these findings show that sound, even if task-irrelevant and semantically unrelated, can correct one's tendency to place objects too far to the left.

Active touch facilitates object size perception in children but not adults: A multisensory event related potential study

In order to increase perceptual precision the adult brain dynamically combines redundant information from different senses depending on their reliability. During object size estimation, for example, visual, auditory and haptic information can be integrated to increase the precision of the final size estimate. Young children, however, do not integrate sensory information optimally and instead rely on active touch. Whether this early haptic dominance is reflected in age-related differences in neural mechanisms and whether it is driven by changes in bottom-up perceptual or top-down attentional processes has not yet been investigated. Here, we recorded event-related-potentials from a group of adults and children aged 5-7 years during an object size perception task using auditory, visual and haptic information. Multisensory information was presented either congruently (conveying the same information) or incongruently (conflicting information). No behavioral responses were required from participants. When haptic size information was available via actively tapping the objects, response amplitudes in the mid-parietal area were significantly reduced by information congruency in children but not in adults between 190 ms-250 ms and 310 ms-370 ms. These findings indicate that during object size perception only children's brain activity is modulated by active touch supporting a neural maturational shift from sensory dominance in early childhood to optimal multisensory benefit in adulthood.

Functional and effective connectivity based features of EEG signals for object recognition

Classifying different object categories is one of the most important aims of brain-computer interface researches. Recently, interactions between brain regions were studied using different methods, such as functional and effective connectivity techniques. Functional and effective connectivity techniques are applied to estimate human brain areas connectivity. The main purpose of this study is to compare classification accuracy of the most advanced functional and effective methods in order to classify 12 basic object categories using Electroencephalography (EEG) signals. In this paper, 19 channels EEG signals were collected from 10 healthy subjects; when they were visiting color images and instructed to select the target images among others. Correlation, magnitude square coherence, wavelet coherence (WC), phase synchronization and mutual information were applied to estimate functional cortical connectivity. On the other hand, directed transfer function, partial directed coherence, generalized partial directed coherence (GPDC) were used to obtain effective cortical connectivity. After feature extraction, the scalar feature selection methods including T-test and one-sided-anova were applied to rank and select the most informative features. The selected features were classified by a one-against-one support vector machine classifier. The results indicated that the use of different techniques led to different classifying accuracy and brain lobes analysis. WC and GPDC are the most accurate methods with performances of 80.15% and 64.43%, respectively.

Hand-Feel Touch Cues and Their Influences on Consumer Perception and Behavior with Respect to Food Products: A Review

There has been a great deal of research investigating intrinsic/extrinsic cues and their influences on consumer perception and purchasing decisions at points of sale, product usage, and consumption. Consumers create expectations toward a food product through sensory information extracted from its surface (intrinsic cues) or packaging (extrinsic cues) at retail stores. Packaging is one of the important extrinsic cues that can modulate consumer perception, liking, and decision making of a product. For example, handling a product packaging during consumption, even just touching the packaging while opening or holding it during consumption, may result in a consumer expectation of the package content. Although hand-feel touch cues are an integral part of the food consumption experience, as can be observed in such an instance, little has been known about their influences on consumer perception, acceptability, and purchase behavior of food products. This review therefore provided a better understanding about hand-feel touch cues and their influences in the context of food and beverage experience with a focus on (1) an overview of touch as a sensory modality, (2) factors influencing hand-feel perception, (3) influences of hand-feel touch cues on the perception of other sensory modalities, and (4) the effects of hand-feel touch cues on emotional responses and purchase behavior.

Who's that Knocking at My Door? Neural Bases of Sound Source Identification

When hearing knocking on a door, a listener typically identifies both the action (forceful and repeated impacts) and the object (a thick wooden board) causing the sound. The current work studied the neural bases of sound source identification by switching listeners' attention toward these different aspects of a set of simple sounds during functional magnetic resonance imaging scanning: participants either discriminated the action or the material that caused the sounds, or they simply discriminated meaningless scrambled versions of them. Overall, discriminating action and material elicited neural activity in a left-lateralized frontoparietal network found in other studies of sound identification, wherein the inferior frontal sulcus and the ventral premotor cortex were under the control of selective attention and sensitive to task demand. More strikingly, discriminating materials elicited increased activity in cortical regions connecting auditory inputs to semantic, motor, and even visual representations, whereas discriminating actions did not increase activity in any regions. These results indicate that discriminating and identifying material requires deeper processing of the stimuli than discriminating actions. These results are consistent with previous studies suggesting that auditory perception is better suited to comprehend the actions than the objects producing sounds in the listeners' environment.

Reduced Dynamic Interactions Within Intrinsic Functional Brain Networks in Early Blind Patients

Neuroimaging studies in early blind (EB) patients have shown altered connections or brain networks. However, it remains unclear how the causal relationships are disrupted within intrinsic brain networks. In our study, we used spectral dynamic causal modeling (DCM) to estimate the causal interactions using resting-state data in a group of 20 EB patients and 20 healthy controls (HC). Coupling parameters in specific regions were estimated, including the medial prefrontal cortex (mPFC), posterior cingulate cortex (PCC), and inferior parietal lobule (IPC) in the default mode network (DMN); dorsal anterior cingulate cortex (dACC) and bilateral anterior insulae (AI) in the salience network (SN), and bilateral frontal eye fields (FEF) and superior parietal lobes (SPL) within the dorsal attention network (DAN). Statistical analyses found that all endogenous connections and the connections from the mPFC to bilateral IPCs in EB patients were significantly reduced within the DMN, and the effective connectivity from the PCC and lIPC to the mPFC, and from the mPFC to the PCC were enhanced. For the SN, all significant connections in EB patients were significantly decreased, except the intrinsic right AI connections. Within the DAN, more significant effective connections were observed to be reduced between the EB and HC groups, while only the connections from the right SPL to the left SPL and the intrinsic connection in the left SPL were significantly enhanced. Furthermore, discovery of more decreased effective connections in the EB subjects suggested that the disrupted causal interactions between specific regions are responsive to the compensatory brain plasticity in early deprivation.

A functional MRI investigation of crossmodal interference in an audiovisual Stroop task

The visual color-word Stroop task is widely used in clinical and research settings as a measure of cognitive control. Numerous neuroimaging studies have used color-word Stroop tasks to investigate the neural resources supporting cognitive control, but to our knowledge all have used unimodal (typically visual) Stroop paradigms. Thus, it is possible that this classic measure of cognitive control is not capturing the resources involved in multisensory cognitive control. The audiovisual integration and crossmodal correspondence literatures identify regions sensitive to congruency of auditory and visual stimuli, but it is unclear how these regions relate to the unimodal cognitive control literature. In this study we aimed to identify brain regions engaged by crossmodal cognitive control during an audiovisual color-word Stroop task, and how they relate to previous unimodal Stroop and audiovisual integration findings. First, we replicated previous behavioral audiovisual Stroop findings in an fMRI-adapted audiovisual Stroop paradigm: incongruent visual information increased reaction time towards an auditory stimulus and congruent visual information decreased reaction time. Second, we investigated the brain regions supporting cognitive control during an audiovisual color-word Stroop task using fMRI. Similar to unimodal cognitive control tasks, a left superior parietal region exhibited an interference effect of visual information on the auditory stimulus. This superior parietal region was also identified using a standard audiovisual integration localizing procedure, indicating that audiovisual integration resources are sensitive to cognitive control demands. Facilitation of the auditory stimulus by congruent visual information was found in posterior superior temporal cortex, including in the posterior STS which has been found to support audiovisual integration. The dorsal anterior cingulate cortex, often implicated in unimodal Stroop tasks, was not modulated by the audiovisual Stroop task. Overall the findings indicate that an audiovisual color-word Stroop task engages overlapping resources with audiovisual integration and overlapping but distinct resources compared to unimodal Stroop tasks.

Decoding natural scenes based on sounds of objects within scenes using multivariate pattern analysis

Scene recognition plays an important role in spatial navigation and scene classification. It remains unknown whether the occipitotemporal cortex could represent the semantic association between the scenes and sounds of objects within the scenes. In this study, we used the functional magnetic resonance imaging (fMRI) technique and multivariate pattern analysis to assess whether diff ; ;erent scenes could be discriminated based on the patterns evoked by sounds of objects within the scenes. We found that patterns evoked by scenes could be predicted with patterns evoked by sounds of objects within the scenes in the posterior fusiform area (pF), lateral occipital area (LO) and superior temporal sulcus (STS). The further functional connectivity analysis suggested significant correlations between pF, LO and parahippocampal place area (PPA) except that between STS and other three regions under the scene and sound conditions. A distinct network in processing scenes and sounds was discovered using a seed-to-voxel analysis with STS as the seed. This study may provide a cross-modal channel of scene decoding through the sounds of objects within the scenes in the occipitotemporal cortex, which could complement the single-modal channel of scene decoding based on the global scene properties or objects within the scenes.

A Novel Method for Training Mice in Visuo-Tactile 3-D Object Discrimination and Recognition

Perceiving, recognizing and remembering 3-dimensional (3-D) objects encountered in the environment has a very high survival value; unsurprisingly, this ability is shared among many animal species, including humans. The psychological, psychophysical and neural basis for object perception, discrimination, recognition and memory has been extensively studied in humans, monkeys, pigeons and rodents, but is still far from understood. Nearly all 3-D object recognition studies in the rodent used the "novel object recognition" paradigm, which relies on innate rather than learned behavior; however, this procedure has several important limitations. Recently, investigators have begun to recognize the power of behavioral tasks learned through reinforcement training (operant conditioning) to reveal the sensorimotor and cognitive abilities of mice and to elucidate their underlying neural mechanisms. Here, we describe a novel method for training and testing mice in visual and tactile object discrimination, recognition and memory, and use it to begin to examine the underlying sensory basis for these cognitive capacities. A custom-designed Y maze was used to train mice to associate one of two 3-D objects with a food reward. Out of nine mice trained in two cohorts, seven reached performance criterion in about 20-35 daily sessions of 20 trials each. The learned association was retained, or rapidly re-acquired, after a 6 weeks hiatus in training. When tested under low light conditions, individual animals differed in the degree to which they used tactile or visual cues to identify the objects. Switching to total darkness resulted only in a transient dip in performance, as did subsequent trimming of all large whiskers (macrovibrissae). Additional removal of the small whiskers (microvibrissae) did not degrade performance, but transiently increased the time spent inspecting the object. This novel method can be combined in future studies with the large arsenal of genetic tools available in the mouse, to elucidate the neural basis of object perception, recognition and memory.

Neural Mechanisms of Material Perception: Quest on Shitsukan

In recent years, a growing body of research has addressed the nature and mechanism of material perception. Material perception entails perceiving and recognizing a material, surface quality or internal state of an object based on sensory stimuli such as visual, tactile, and/or auditory sensations. This process is ongoing in every aspect of daily life. We can, for example, easily distinguish whether an object is made of wood or metal, or whether a surface is rough or smooth. Judging whether the ground is wet or dry or whether a fish is fresh also involves material perception. Information obtained through material perception can be used to govern actions toward objects and to make decisions about whether to approach an object or avoid it. Because the physical processes leading to sensory signals related to material perception is complicated, it has been difficult to manipulate experimental stimuli in a rigorous manner. However, that situation is now changing thanks to advances in technology and knowledge in related fields. In this article, we will review what is currently known about the neural mechanisms responsible for material perception. We will show that cortical areas in the ventral visual pathway are strongly involved in material perception. Our main focus is on vision, but every sensory modality is involved in material perception. Information obtained through different sensory modalities is closely linked in material perception. Such cross-modal processing is another important feature of material perception, and will also be covered in this review.

Effects of androstadienone on dominance perception in males with low and high social anxiety

Increasing evidence suggests that humans can communicate both trait-dominance and state-dominance via body odor. Androstadienone (androsta-4,16,-dien-3-one), a chemosignal found in human sweat, seems to be a likely candidate for signaling dominance in humans. The aim of the current study was to investigate the effects of androstadienone on the perception of social dominance. Moreover, we examined whether high levels of social anxiety, a psychopathology involving concerns that specifically pertain to social dominance, are associated with increased sensitivity to androstadienone as a chemical cue of dominance. In a double-blind, placebo-controlled, within-subject design, 64 heterosexual male participants (32 with high social anxiety and 32 with low social anxiety) viewed facial images of males depicting dominant, neutral and submissive postures, and were asked to recognize and rate the dominance expressed in those images. Participants completed the task twice, once under exposure to androstadienone and once under exposure to a control solution. The results indicate that androstadienone increased the perceived dominance of men's faces, specifically among participants with high social anxiety. These findings suggest a direct influence of androstadienone on dominance perception and further highlight the preferential processing of dominance and social threat signals evident in social anxiety.

An oscillatory neural network model that demonstrates the benefits of multisensory learning

Since the world consists of objects that stimulate multiple senses, it is advantageous for a vertebrate to integrate all the sensory information available. However, the precise mechanisms governing the temporal dynamics of multisensory processing are not well understood. We develop a computational modeling approach to investigate these mechanisms. We present an oscillatory neural network model for multisensory learning based on sparse spatio-temporal encoding. Recently published results in cognitive science show that multisensory integration produces greater and more efficient learning. We apply our computational model to qualitatively replicate these results. We vary learning protocols and system dynamics, and measure the rate at which our model learns to distinguish superposed presentations of multisensory objects. We show that the use of multiple channels accelerates learning and recall by up to 80%. When a sensory channel becomes disabled, the performance degradation is less than that experienced during the presentation of non-congruent stimuli. This research furthers our understanding of fundamental brain processes, paving the way for multiple advances including the building of machines with more human-like capabilities.

Endogenous modulation of human visual cortex activity improves perception at twilight

Perception, particularly in the visual domain, is drastically influenced by rhythmic changes in ambient lighting conditions. Anticipation of daylight changes by the circadian system is critical for survival. However, the neural bases of time-of-day-dependent modulation in human perception are not yet understood. We used fMRI to study brain dynamics during resting-state and close-to-threshold visual perception repeatedly at six times of the day. Here we report that resting-state signal variance drops endogenously at times coinciding with dawn and dusk, notably in sensory cortices only. In parallel, perception-related signal variance in visual cortices decreases and correlates negatively with detection performance, identifying an anticipatory mechanism that compensates for the deteriorated visual signal quality at dawn and dusk. Generally, our findings imply that decreases in spontaneous neural activity improve close-to-threshold perception.

Effects of modality and repetition in a continuous recognition memory task: Repetition has no effect on auditory recognition memory

Previous research has shown that auditory recognition memory is poorer compared to visual and cross-modal (visual and auditory) recognition memory. The effect of repetition on memory has been robust in showing improved performance. It is not clear, however, how auditory recognition memory compares to visual and cross-modal recognition memory following repetition. Participants performed a recognition memory task, making old/new discriminations to new stimuli, stimuli repeated for the first time after 4-7 intervening items (R1), or repeated for the second time after 36-39 intervening items (R2). Depending on the condition, participants were either exposed to visual stimuli (2D line drawings), auditory stimuli (spoken words), or cross-modal stimuli (pairs of images and associated spoken words). Results showed that unlike participants in the visual and cross-modal conditions, participants in the auditory recognition did not show improvements in performance on R2 trials compared to R1 trials. These findings have implications for pedagogical techniques in education, as well as for interventions and exercises aimed at boosting memory performance.