Studying the Effects of Congruence of Auditory and Visual Stimuli on Virtual Reality Experiences

Interactions Between Vibroacoustic Discomfort and Visual Stimuli: Comparison of Real, 3D and 360 Environments

The building industry and the design of interior environments are increasingly focusing on the user experience, incorporating sensory analysis to reconsider how office environments can be optimized. New immersive technologies offer significant opportunities for sensory science, enhancing our understanding of human perception and enabling the collection of multi-sensory data under controlled laboratory conditions. While the potential of Virtual Reality (VR) for these types of studies is well recognized, certain limitations still need to be addressed, including the lack of standardized research practices and the challenge of ensuring the simulated environment closely mirrors the real world. In this study, we compare 360° and 3D formats, to real-life settings in order to determine which format offers greater ecological validity for visual perception and immersion. Additionally, we examine the effects of vibroacoustic stimuli with different levels of intensity on perception and cognition of 30 participants. Subjective, physiological and cognitive data was collected throughout the test to tackle the participant's experience. This preliminary study introduces an immersive methodology that leverages advanced techniques to gain deeper insights into multisensory user experience in VR, marking a significant step forward in the optimization of VR for building evaluation.

Affording Social Experience for Adolescents Using Immersive Virtual Reality: A Moderated Mediation Analysis

BACKGROUND

Immersive virtual reality (IVR) serves as a promising tool to provide adolescents with enriched social experience due to its high-fidelity simulations and multimodal interaction. This study aims to design and develop a multi-user IVR collaborative game utilizing simultaneous localization and mapping (SLAM)-based inside-out tracking technique to foster social experience among students. Also, this study explored the mechanism by which technology acceptance affected social experience in the IVR collaboration game, focusing on the mediating effects of presence, collective efficacy, and group effectiveness, as well as the moderating effect of social-emotional competence (SEC).

METHODS

A total of 104 seventh graders from a middle school in Central China participated in this study and completed the questionnaire. Finally, 87 valid questionnaire responses were retrieved.

RESULTS

The results revealed that technology acceptance both directly and indirectly influenced social experience. The mediation analysis revealed a key pathway influencing social experience: technology acceptance → presence → collective efficacy → group effectiveness → social experience. However, no moderating effect of SEC was found in the relationship between technology acceptance and social experience, group effectiveness, and social experience.

CONCLUSIONS

Based on these results, more appropriate IVR interventions could be developed for social-emotional learning among children and adolescents.

Spatiotemporal Congruency Modulates Weighting of Visuotactile Information in Displacement Judgments

Combining or integrating information from multiple senses often provides richer and more reliable estimates for the perception of objects and events. In daily life, sensory information from the same source often is in close spatiotemporal proximity. This can be an important determinant of whether and how multisensory signals are combined. The introduction of advanced technical display systems allows to present multisensory information in virtual environments. However, technical displays can lack the spatiotemporal fidelity of the real world due the rendering delays. Thus, any spatiotemporal incongruency could alter how information is combined. In the current study we tested this by investigating if and how spatially and temporally discrepant tactile displacement cues can supplement imprecise visual displacement cues. Participants performed a visual displacement task with visual and tactile displacement cues under spatial and temporal incongruency conditions. We modelled how participants combined visual and tactile information in visuotactile condition using their performance in visual only condition. We found that temporal incongruency lead to an increase in tactile weights although they were correlated with the congruency condition. In contrast, the spatial incongruency led to individual differences altering cue combination strategies. Our results illustrate the importance of spatiotemporal congruency for combining tactile and visual cues when making visual displacement judgments. Given the altered cue combination strategies and individual differences, we recommend developers to adopt individual spatiotemporal calibration procedures to improve the efficiency of the sensory augmentation.

Modeling the Impact of Head-Body Rotations on Audio-Visual Spatial Perception for Virtual Reality Applications

Humans perceive the world by integrating multimodal sensory feedback, including visual and auditory stimuli, which holds true in virtual reality (VR) environments. Proper synchronization of these stimuli is crucial for perceiving a coherent and immersive VR experience. In this work, we focus on the interplay between audio and vision during localization tasks involving natural head-body rotations. We explore the impact of audio-visual offsets and rotation velocities on users' directional localization acuity for various viewing modes. Using psychometric functions, we model perceptual disparities between visual and auditory cues and determine offset detection thresholds. Our findings reveal that target localization accuracy is affected by perceptual audio-visual disparities during head-body rotations, but remains consistent in the absence of stimuli-head relative motion. We then showcase the effectiveness of our approach in predicting and enhancing users' localization accuracy within realistic VR gaming applications. To provide additional support for our findings, we implement a natural VR game wherein we apply a compensatory audio-visual offset derived from our measured psychometric functions. As a result, we demonstrate a substantial improvement of up to 40% in participants' target localization accuracy. We additionally provide guidelines for content creation to ensure coherent and seamless VR experiences.

D-SAV360: A Dataset of Gaze Scanpaths on 360° Ambisonic Videos

Understanding human visual behavior within virtual reality environments is crucial to fully leverage their potential. While previous research has provided rich visual data from human observers, existing gaze datasets often suffer from the absence of multimodal stimuli. Moreover, no dataset has yet gathered eye gaze trajectories (i.e., scanpaths) for dynamic content with directional ambisonic sound, which is a critical aspect of sound perception by humans. To address this gap, we introduce D-SAV360, a dataset of 4,609 head and eye scanpaths for 360° videos with first-order ambisonics. This dataset enables a more comprehensive study of multimodal interaction on visual behavior in virtual reality environments. We analyze our collected scanpaths from a total of 87 participants viewing 85 different videos and show that various factors such as viewing mode, content type, and gender significantly impact eye movement statistics. We demonstrate the potential of D-SAV360 as a benchmarking resource for state-of-the-art attention prediction models and discuss its possible applications in further research. By providing a comprehensive dataset of eye movement data for dynamic, multimodal virtual environments, our work can facilitate future investigations of visual behavior and attention in virtual reality.

Effects of immersion in a simulated natural environment on stress reduction and emotional arousal: A systematic review and meta-analysis

Background

Although the mental health benefits of exposure to simulated natural environments are well established by researchers from environmental psychology, landscape architecture, and public health, it is unclear whether and to what extent technological immersion affects these benefits.

Methods

Systematical literature searches were conducted in May 2022 from six databases. The risk of bias was evaluated using the Cochrane's Risk of Bias tool 2.0 and the Risk of Bias in Non-randomized Studies of Interventions tool. We performed a random-effects meta-regression to investigate the heterogeneity. The immersion levels of included studies were classified by projection devices and motion capture, and then subgroup analysis was conducted.

Results

Twenty-six publications were included. Exposure to simulated nature was confirmed to be associated with increased positive affect 0.40 [95% confidence interval (CI): 0.22, 0.58], vigor 0.58 (95% CI: 0.30, 0.86), calmness 0.54 (95% CI: 0.17, 0.92) and decreased perceived stress -0.38 (95% CI: -0.71, -0.06), total mood disturbance -0.87 (95% CI: -1.17, -0.57), tension -0.70 (95% CI: -0.99, -0.41), fatigue -0.60 (95% CI: -0.91, -0.28), anxiety -0.72 (95% CI: -1.43, -0.02), depression -0.33 (95% CI: -0.52, -0.14), confusion -0.79 (95% CI: -1.19, -0.40), and anger -0.54 (95% CI: -0.76, -0.31). Gender, health status, study design, mean age, and single exposure duration were not significant when entered in a meta-regression. For positive affect, medium immersion was observed to produce a larger effect than low and high immersion. All included studies had a moderate to high risk of bias.

Conclusion

Audio-visual exposure to simulated nature contributes to stress relief and emotional arousal. The immersion level explains the heterogeneity of positive affect triggered by simulated nature. Focusing on the technical features will open up new possibilities for combining actual and simulated nature's mental health benefits.