Audiovisual Integration Varies With Target and Environment Richness in Immersive Virtual Reality

Spatiotemporal Congruency Modulates Weighting of Visuotactile Information in Displacement Judgments

Combining or integrating information from multiple senses often provides richer and more reliable estimates for the perception of objects and events. In daily life, sensory information from the same source often is in close spatiotemporal proximity. This can be an important determinant of whether and how multisensory signals are combined. The introduction of advanced technical display systems allows to present multisensory information in virtual environments. However, technical displays can lack the spatiotemporal fidelity of the real world due the rendering delays. Thus, any spatiotemporal incongruency could alter how information is combined. In the current study we tested this by investigating if and how spatially and temporally discrepant tactile displacement cues can supplement imprecise visual displacement cues. Participants performed a visual displacement task with visual and tactile displacement cues under spatial and temporal incongruency conditions. We modelled how participants combined visual and tactile information in visuotactile condition using their performance in visual only condition. We found that temporal incongruency lead to an increase in tactile weights although they were correlated with the congruency condition. In contrast, the spatial incongruency led to individual differences altering cue combination strategies. Our results illustrate the importance of spatiotemporal congruency for combining tactile and visual cues when making visual displacement judgments. Given the altered cue combination strategies and individual differences, we recommend developers to adopt individual spatiotemporal calibration procedures to improve the efficiency of the sensory augmentation.

The Redundant Signals Effect and the Full Body Illusion: not Multisensory, but Unisensory Tactile Stimuli Are Affected by the Illusion

During a full body illusion (FBI), participants experience a change in self-location towards a body that they see in front of them from a third-person perspective and experience touch to originate from this body. Multisensory integration is thought to underlie this illusion. In the present study we tested the redundant signals effect (RSE) as a new objective measure of the illusion that was designed to directly tap into the multisensory integration underlying the illusion. The illusion was induced by an experimenter who stroked and tapped the participant's shoulder and underarm, while participants perceived the touch on the virtual body in front of them via a head-mounted display. Participants performed a speeded detection task, responding to visual stimuli on the virtual body, to tactile stimuli on the real body and to combined (multisensory) visual and tactile stimuli. Analysis of the RSE with a race model inequality test indicated that multisensory integration took place in both the synchronous and the asynchronous condition. This surprising finding suggests that simultaneous bodily stimuli from different (visual and tactile) modalities will be transiently integrated into a multisensory representation even when no illusion is induced. Furthermore, this finding suggests that the RSE is not a suitable objective measure of body illusions. Interestingly however, responses to the unisensory tactile stimuli in the speeded detection task were found to be slower and had a larger variance in the asynchronous condition than in the synchronous condition. The implications of this finding for the literature on body representations are discussed.

The additive nature of the human multisensory evoked pupil response

Pupillometry has received increased interest for its usefulness in measuring various sensory processes as an alternative to behavioural assessments. This is also apparent for multisensory investigations. Studies of the multisensory pupil response, however, have produced conflicting results. Some studies observed super-additive multisensory pupil responses, indicative of multisensory integration (MSI). Others observed additive multisensory pupil responses even though reaction time (RT) measures were indicative of MSI. Therefore, in the present study, we investigated the nature of the multisensory pupil response by combining methodological approaches of previous studies while using supra-threshold stimuli only. In two experiments we presented auditory and visual stimuli to observers that evoked a(n) (onset) response (be it constriction or dilation) in a simple detection task and a change detection task. In both experiments, the RT data indicated MSI as shown by race model inequality violation. Still, the multisensory pupil response in both experiments could best be explained by linear summation of the unisensory pupil responses. We conclude that the multisensory pupil response for supra-threshold stimuli is additive in nature and cannot be used as a measure of MSI, as only a departure from additivity can unequivocally demonstrate an interaction between the senses.

Role of Kinematics Assessment and Multimodal Sensorimotor Training for Motion Deficits in Breast Cancer Chemotherapy-Induced Polyneuropathy: A Perspective on Virtual Reality Avatars

Chemotherapy-induced polyneuropathy (CIPN), one of the most severe and incapacitating side effects of chemotherapeutic drugs, is a serious concern in breast cancer therapy leading to dose diminution, delay, or cessation. The reversibility of CIPN is of increasing importance since active chemotherapies prolong survival. Clinical assessment tools show that patients experiencing sensorimotor CIPN symptoms not only do they have to cope with loss in autonomy and life quality, but CIPN has become a key restricting factor in treatment. CIPN incidence poses a clinical challenge and has lacked established and efficient therapeutic options up to now. Complementary, non-opioid therapies are sought for both prevention and management of CIPN. In this perspective, we explore the potential that digital interventions have for sensorimotor CIPN rehabilitation in breast cancer patients. Our primary goal is to emphasize the benefits and impact that Virtual Reality (VR) avatars and Machine Learning have in combination in a digital intervention aiming at (1) assessing the complete kinematics of deficits through learning underlying patient sensorimotor parameters, and (2) parameterize a multimodal VR simulation to drive personalized deficit compensation. We support our perspective by evaluating sensorimotor effects of chemotherapy, the metrics to assess sensorimotor deficits, and relevant clinical studies. We subsequently analyse the neurological substrate of VR sensorimotor rehabilitation, with multisensory integration acting as a key element. Finally, we propose a closed-loop patient-centered design recommendation for CIPN sensorimotor rehabilitation. Our aim is to provoke the scientific community toward the development and use of such digital interventions for more efficient and targeted rehabilitation.