Multisensory Integration as per Technological Advances: A Review

Brain imaging studies of multisensory integration in emotion perception: A scoping review

Emotions are essential in everyday human functioning, yet accurately perceiving them is a complex task. Emotional cues originating from multiple sensory modalities need to be efficiently integrated to form holistic percepts. This process, known as emotional multisensory integration (eMSI), involves the recruitment of multiple brain regions and has been shown to enhance emotion perception. The present scoping review provides an updated and exhaustive overview of the current literature on brain regions underlying eMSI. In total, 12 relevant studies were identified for inclusion across four academic databases. Most included studies employed audio-visual paradigms, examined the integration of happy and angry emotional cues, and utilized functional magnetic resonance imaging to identify brain regions involved in eMSI. Findings support that brain regions underlying both multisensory (e.g., thalamus, posterior superior temporal sulcus/gyrus) and emotional integration (e.g., fusiform gyrus, medial temporal gyrus) interact to enhance eMSI, leading to improved emotion detection. A need for additional research was also identified due to a lack of diverse populations, sensory combinations, and emotions investigated, as well as methodological issues identified across included studies. Future research should aim to expand its focus to address these underexplored areas, thereby advancing our understanding of eMSI.

Interactions Between Vibroacoustic Discomfort and Visual Stimuli: Comparison of Real, 3D and 360 Environments

The building industry and the design of interior environments are increasingly focusing on the user experience, incorporating sensory analysis to reconsider how office environments can be optimized. New immersive technologies offer significant opportunities for sensory science, enhancing our understanding of human perception and enabling the collection of multi-sensory data under controlled laboratory conditions. While the potential of Virtual Reality (VR) for these types of studies is well recognized, certain limitations still need to be addressed, including the lack of standardized research practices and the challenge of ensuring the simulated environment closely mirrors the real world. In this study, we compare 360° and 3D formats, to real-life settings in order to determine which format offers greater ecological validity for visual perception and immersion. Additionally, we examine the effects of vibroacoustic stimuli with different levels of intensity on perception and cognition of 30 participants. Subjective, physiological and cognitive data was collected throughout the test to tackle the participant's experience. This preliminary study introduces an immersive methodology that leverages advanced techniques to gain deeper insights into multisensory user experience in VR, marking a significant step forward in the optimization of VR for building evaluation.

Modulation of thermal perception by VR-based visual stimulation to the embodied virtual body

Visual stimulation to the embodied virtual body could modulate human perception, however the associated neurophysiological mechanisms have not elucidated yet. The present study aimed to reveal the underlying neurophysiological mechanisms from a neurophysiological viewpoint. Fifteen healthy participants were subjected to three visual conditions (i.e., fire, water, and non-visual effect conditions) and psychological pain stimulation (thermal grill stimulation). Oscillatory neural activities during stimulation were measured with electroencephalogram. The association between accessory visual stimulation applied to the embodied virtual body, induced by virtual reality, and perception was examined through neuronal oscillatory analysis using electroencephalogram data. Regression analysis was performed to obtain data on brain regions contributing to sensory modulation with body illusion. The results of subjective measures under the fire and water conditions showed that thermal perception were modulated by a visual stimulus to the virtual hand. Furthermore, we found that the insula was commonly associated with thermal perception under the fire and water conditions. This result indicate that the insula may control sensory information as a gatekeeper as well as facilitate the access to human attention and cognition as a hub, suggesting the influence on perception and cognition.

A conceptual ethical framework to preserve natural human presence in the use of AI systems in education

In recent years, there has been a remarkable increase of interest in the ethical use of AI systems in education. On one hand, the potential for such systems is undeniable. Used responsibly, they can meaningfully support and enhance the interactive process of teaching and learning. On the other hand, there is a risk that natural human presence may be gradually replaced by arbitrarily created AI systems, particularly due to their rapidly increasing yet partially unguided capabilities. State-of-the-art ethical frameworks suggest high-level principles, requirements, and guidelines, but lack detailed low-level models of concrete processes and according properties of the involved actors in education. In response, this article introduces a detailed Unified Modeling Language (UML)-based ancillary framework that includes a novel set of low-level properties. Whilst not incorporated in related work, particularly the ethical behavior and visual representation of the actors are intended to improve transparency and reduce the potential for misinterpretation and misuse of AIS. The framework primarily focuses on school education, resulting in a more restrictive model, however, reflects on potentials and challenges in terms of improving flexibility toward different educational levels. The article concludes with a discussion of key findings and implications of the presented framework, its limitations, and potential future research directions to sustainably preserve natural human presence in the use of AI systems in education.

Clinical Efficacy of Virtual Reality Cave Automatic Virtual Environments (CAVE) for Chronic Musculoskeletal Pain in Older Adults: A Randomized Controlled Trial

OBJECTIVES

To assess whether tai chi assisted by the Cave Automatic Virtual Environment (CAVE), a novel virtual reality (VR) technology, was superior to tai chi alone in managing chronic pain in older adults. CAVE may offer a promising alternative to head-mounted displays in chronic pain treatment.

DESIGN

The study was a 12-week, 2-arm, parallel, randomized controlled trial implemented in VR and non-VR groups (N = 80, each arm = 40). The VR group underwent an 8-week tai chi program in a 3-wall VR-CAVE with projections of nature scenes and music, whereas the control group received tai chi only.

SETTING AND PARTICIPANTS

Community-dwelling older adults with chronic musculoskeletal pain were randomized (1:1) to VR and non-VR groups.

MEASURES

Participants were assessed at baseline, posttreatment at 8 weeks, and 12 weeks. The primary outcome was the Brief Pain Inventory severity score at 8 weeks, modeled within an intention-to-treat framework using generalized estimating equations.

RESULTS

Participants had a mean age of 65.1 ± 5.6 years, with 78.8% female and mean BPI-pain severity score of 4.4 ± 1.5. At 8 weeks, the VR group demonstrated a statistically significant improvement in BPI severity score compared with the non-VR group (β = -0.75, 95% CI -1.48 to -0.03, P = .043), with the effect sustained to 12 weeks (β = -1.18, 95% CI -1.90 to -0.46, P = .001). No major adverse events were reported.

CONCLUSIONS AND IMPLICATIONS

VR-CAVE tai chi was superior to non-VR tai chi for chronic musculoskeletal pain. Future trials that are longer-term, larger in scale, and include other forms of exercise will further inform VR-CAVE's role in post-acute and long-term rehabilitation.

Recent Advances in Artificial Sensory Neurons: Biological Fundamentals, Devices, Applications, and Challenges

Spike-based neural networks, which use spikes or action potentials to represent information, have gained a lot of attention because of their high energy efficiency and low power consumption. To fully leverage its advantages, converting the external analog signals to spikes is an essential prerequisite. Conventional approaches including analog-to-digital converters or ring oscillators, and sensors suffer from high power and area costs. Recent efforts are devoted to constructing artificial sensory neurons based on emerging devices inspired by the biological sensory system. They can simultaneously perform sensing and spike conversion, overcoming the deficiencies of traditional sensory systems. This review summarizes and benchmarks the recent progress of artificial sensory neurons. It starts with the presentation of various mechanisms of biological signal transduction, followed by the systematic introduction of the emerging devices employed for artificial sensory neurons. Furthermore, the implementations with different perceptual capabilities are briefly outlined and the key metrics and potential applications are also provided. Finally, we highlight the challenges and perspectives for the future development of artificial sensory neurons.

Synchrony perception across senses: A systematic review of temporal binding window changes from infancy to adolescence in typical and atypical development

Sensory integration is increasingly acknowledged as being crucial for the development of cognitive and social abilities. However, its developmental trajectory is still little understood. This systematic review delves into the topic by investigating the literature about the developmental changes from infancy through adolescence of the Temporal Binding Window (TBW) - the epoch of time within which sensory inputs are perceived as simultaneous and therefore integrated. Following comprehensive searches across PubMed, Elsevier, and PsycInfo databases, only experimental, behavioral, English-language, peer-reviewed studies on multisensory temporal processing in 0-17-year-olds have been included. Non-behavioral, non-multisensory, and non-human studies have been excluded as those that did not directly focus on the TBW. The selection process was independently performed by two Authors. The 39 selected studies involved 2859 participants in total. Findings indicate a predisposition towards cross-modal asynchrony sensitivity and a composite, still unclear, developmental trajectory, with atypical development associated to increased asynchrony tolerance. These results highlight the need for consistent and thorough research into TBW development to inform potential interventions.

A behavioral economics approach to enhancing HIV preexposure and postexposure prophylaxis implementation

PURPOSE OF REVIEW

The 'PrEP cliff' phenomenon poses a critical challenge in global HIV PrEP implementation, marked by significant dropouts across the entire PrEP care continuum. This article reviews new strategies to address 'PrEP cliff'.

RECENT FINDINGS

Canadian clinicians have developed a service delivery model that offers presumptive PEP to patients in need and transits eligible PEP users to PrEP. Early findings are promising. This service model not only establishes a safety net for those who were not protected by PrEP, but it also leverages the immediate salience and perceived benefits of PEP as a natural nudge towards PrEP use. Aligning with Behavioral Economics, specifically the Salience Theory, this strategy holds potential in tackling PrEP implementation challenges.

SUMMARY

A natural pathway between PEP and PrEP has been widely observed. The Canadian service model exemplifies an innovative strategy that leverages this organic pathway and enhances the utility of both PEP and PrEP services. We offer theoretical insights into the reasons behind these PEP-PrEP transitions and evolve the Canadian model into a cohesive framework for implementation.

Multimodal processing of noisy cues in bumblebees

Multimodal cues can improve behavioral responses by enhancing the detection and localization of sensory cues and reducing response times. Across species, studies have shown that multisensory integration of visual and olfactory cues can improve response accuracy. However, in real-world settings, sensory cues are often noisy; visual and olfactory cues can be deteriorated, masked, or mixed, making the target cue less clear to the receiver. In this study, we use an associative learning paradigm (Free Moving Proboscis Extension Reflex, FMPER) to show that having multimodal cues may improve the accuracy of bees' responses to noisy cues. Adding a noisy visual cue improves the accuracy of response to a noisy olfactory cue, despite neither the clear nor noisy visual cue being sufficient when paired with a novel olfactory cue. This may provide insight into the neural mechanisms underlying multimodal processing and the effects of environmental change on pollination services.

Alterations of Audiovisual Integration in Alzheimer's Disease

Audiovisual integration is a vital information process involved in cognition and is closely correlated with aging and Alzheimer's disease (AD). In this review, we evaluated the altered audiovisual integrative behavioral symptoms in AD. We further analyzed the relationships between AD pathologies and audiovisual integration alterations bidirectionally and suggested the possible mechanisms of audiovisual integration alterations underlying AD, including the imbalance between energy demand and supply, activity-dependent degeneration, disrupted brain networks, and cognitive resource overloading. Then, based on the clinical characteristics including electrophysiological and imaging data related to audiovisual integration, we emphasized the value of audiovisual integration alterations as potential biomarkers for the early diagnosis and progression of AD. We also highlighted that treatments targeted audiovisual integration contributed to widespread pathological improvements in AD animal models and cognitive improvements in AD patients. Moreover, investigation into audiovisual integration alterations in AD also provided new insights and comprehension about sensory information processes.

Olfactory-auditory sensory integration in the lateral entorhinal cortex

Multisensory integration plays an important role in animal cognition. Although many studies have focused on visual-auditory integration, studies on olfactory-auditory integration are rare. Here, we investigated neural activity patterns and odor decoding in the lateral entorhinal cortex (LEC) under uni-sensory and multisensory stimuli in awake, head-fixed mice. Using specific retrograde tracing, we verified that the LEC receives direct inputs from the primary auditory cortex (AC) and the medial geniculate body (MGB). Strikingly, we found that mitral/tufted cells (M/Ts) in the olfactory bulb (OB) and neurons in the LEC respond to both olfactory and auditory stimuli. Sound decreased the neural responses evoked by odors in both the OB and LEC, for both excitatory and inhibitory responses. Interestingly, significant changes in odor decoding performance and modulation of odor-evoked local field potentials (LFPs) were observed only in the LEC. These data indicate that the LEC is a critical center for olfactory-auditory multisensory integration, with direct projections from both olfactory and auditory centers.

Scent-delivery devices as a digital healthcare tool for olfactory training: A pilot focus group study in Parkinson's disease patients

Parkinson's disease (PD) patients display a combination of motor and non-motor symptoms. The most common non-motor symptom is scent (olfactory) impairment, occurring at least four years prior to motor symptom onset. Recent and growing interest in digital healthcare technology used in PD has resulted in more technologies developed for motor rather than non-motor symptoms. Human-computer interaction (HCI), which uses computer technology to explore human activity and work, could be combined with digital healthcare technologies to better understand and support olfaction via scent training - leading to the development of a scent-delivery device (SDD). In this pilot study, three PD patients were invited to an online focus group to explore the association between PD and olfaction, understand HCI and sensory technologies and were demonstrated a new multichannel SDD with an associated mobile app. Participants had a preconceived link, a result of personal experience, between olfactory impairment and PD. Participants felt that healthcare professionals did not take olfactory dysfunction concerns seriously prior to PD diagnosis. Two were not comfortable with sharing scent loss experiences with others. Participants expected the multichannel SDD to be small, portable and easy-to-use, with customisable cartridges to deliver chosen scents and the mobile app to create a sense of community. None of the participants regularly performed scent training but would consider doing so if some scent function could be regained. Standardised digital SDDs for regular healthcare check-ups may facilitate improvement in olfactory senses in PD patients and potential earlier PD diagnosis, allowing earlier therapeutic and symptomatic PD management.

Central vestibular dysfunction: don't forget vestibular rehabilitation

INTRODUCTION

Vestibular rehabilitation (VR) is now a subject of active studies and has been shown to be effective for multiple vestibular disorders, peripheral or central. VR is a physical therapy that helps train the central nervous system to compensate for vestibular dysfunction. There is moderate to strong evidence that VR is safe and effective for the management of peripheral vestibular dysfunction. Nonetheless, the studies on how VR works on central vestibular dysfunction remains scanty.

AREAS COVERED

This article addressed the rehabilitation strategies and possible mechanisms, including how central vestibular function might improve upon rehabilitation. In addition, it provides some examples concerning the effect of VR on central vestibular dysfunction.

EXPERT OPINION

VR works on the vestibular system through repetition of specific physical exercises that activate central neuroplastic mechanisms to achieve adaptive compensation of the impaired functions. VR has become a mainstay in the management of patients with dizziness and balance dysfunction. Individualized VR programs are a safe and effective treatment option for a large percentage of patients with central vestibular disease reporting imbalance and dizziness. Exploration of various treatment strategies and possible mechanisms will help develop the best and personalized VR treatment for patients with central vestibular dysfunction.

Consumer Consciousness in Multisensory Extended Reality

The reality-virtuality continuum encompasses a multitude of objects, events and environments ranging from real-world multisensory inputs to interactive multisensory virtual simulators, in which sensory integration can involve very different combinations of both physical and digital inputs. These different ways of stimulating the senses can affect the consumer's consciousness, potentially altering their judgements and behaviours. In this perspective paper, we explore how technologies such as Augmented Reality (AR) and Virtual Reality (VR) can, by generating and modifying the human sensorium, act on consumer consciousness. We discuss the potential impact of this altered consciousness for consumer behaviour while, at the same time, considering how it may pave the way for further research.

OpenSync: An opensource platform for synchronizing multiple measures in neuroscience experiments

BACKGROUND

The human mind is multimodal. Yet most behavioral studies rely on century-old measures such as task accuracy and latency. To create a better understanding of human behavior and brain functionality, we should introduce other measures and analyze behavior from various aspects. However, it is technically complex and costly to design and implement the experiments that record multiple measures. To address this issue, a platform that allows synchronizing multiple measures from human behavior is needed.

METHOD

This paper introduces an opensource platform named OpenSync, which can be used to synchronize multiple measures in neuroscience experiments. This platform helps to automatically integrate, synchronize and record physiological measures (e.g., electroencephalogram (EEG), galvanic skin response (GSR), eye-tracking, body motion, etc.), user input response (e.g., from mouse, keyboard, joystick, etc.), and task-related information (stimulus markers). In this paper, we explain the structure and details of OpenSync, provide two case studies in PsychoPy and Unity. Comparison with existing tools: Unlike proprietary systems (e.g., iMotions), OpenSync is free and it can be used inside any opensource experiment design software (e.g., PsychoPy, OpenSesame, Unity, etc., https://pypi.org/project/OpenSync/ and https://github.com/moeinrazavi/OpenSync_Unity).

RESULTS

Our experimental results show that the OpenSync platform is able to synchronize multiple measures with microsecond resolution.

A Reflection on the State of Multisensory Human–Food Interaction Research

We present a perspective article on the state of multisensory human–food interaction (MHFI) research and lay out some reflections for research and development in this area of inquiry, based on a revision of the different spaces that we have co-created with researchers in this space. We begin by conceptualizing and defining MHFI, before moving onto presenting some of its major themes, as well as possible ways in which such themes can guide future research in the area. This article provides key definitions and foundations for the area of MHFI, as well as a first point of contact for those interested in it.

Impossible (Food) Experiences in Extended Reality

We introduce a model to think about impossible experiences in mixed and virtual reality, while emphasizing the role of said experiences in the context of food. This reality-impossibility model includes two continua, namely, the reality-fantasy character of objects and environments, and the extent to which they follow the laws of physics-other laws. We present a series of examples in each of the quadrants of the model and discuss both the research possibilities and implications of impossible experiences.

Getting started with virtual reality for sensory and consumer science: Current practices and future perspectives

While virtual reality (VR) has become increasingly popular in food-related research, there has been a lack of clarity, precision, and guidelines regarding what exactly constitutes a virtual reality study, as well as the options available to the researcher for designing and implementing it. This review provides a practical guide for sensory and consumer scientists interested in exploring the emerging opportunities offered by VR. We take a deep dive into the components that make up a VR study, including hardware, software, and response measurement methods, all the while being grounded in immersion and presence theory. We then review how these building blocks are put together to create two major categories of research scenarios: product selection, which can be entirely created in VR, and food evaluation, which involve tasting products in real life. For each category, we review current literature with a focus on experimental design, then highlight future avenues and technical development opportunities within sensory and consumer research. Finally, we evaluate limitations and ethical issues in VR food research, and offer future perspectives which go above and beyond ensuring ecological validity in product testing.