3D Tune-In Toolkit: An open-source library for real-time binaural spatialisation

HEaring and LIstening eXperience (HELIX): Evaluation of a co-designed serious game for auditory-cognitive training

In the dementia field, a number of applications are being developed aimed at boosting functional abilities. There is an interesting gap as to how utilising serious games can further the knowledge on the potential relationship between hearing and cognitive health in mid-life. The aim of this study was to evaluate the auditory-cognitive training application HELIX, against outcome measures for speech-in-noise, cognitive tasks, communication confidence, quality of life, and usability. A randomised-controlled trial was completed for 43 participants with subjective hearing loss and/or cognitive impairment, over a play period of 4 weeks and a follow-up period of another 4 weeks. Outcome measures included a new online implementation of the Digit-Triplet-Test, a battery of online cognitive tests, and quality of life questionnaires. Paired semi-structured interviews and usability measures were completed to assess HELIX's impact on quality of life and usability. An improvement in the performance of the Digit-Triplet-Test, measured 4 and 8 weeks after the baseline, was found within the training group; however, this improvement was not significant between the training and control groups. No significant improvements were found in any other outcome measures. Thematic analysis suggested HELIX prompted the realisation of difficulties and actions required, improved listening, and positive behaviour change. Employing a participatory design approach has ensured HELIX is relevant and useful for participants who may be at risk of developing age-related hearing loss and cognitive decline. Although an improvement in the Digit-Triplet-Test was seen, it is not possible to conclude whether this was as a result of playing HELIX.

On the generalization of accommodation to head-related transfer functions

To date, there is strong evidence indicating that humans with normal hearing can adapt to non-individual head-related transfer functions (HRTFs). However, less attention has been given to studying the generalization of this adaptation to untrained conditions. This study investigated how adaptation to one set of HRTFs can generalize to another set of HRTFs. Participants were divided into two groups and trained to localize a speech stimulus reproduced binaurally using either individual or non-individual HRTFs. Training led to an improved localization performance with the trained HRTFs for both groups of participants. Results also showed that there was no difference in the localization performance improvement between the trained and untrained HRTFs for both groups, indicating a generalization of adaptation to HRTFs. The findings did not allow to precisely determine which type of learning (procedural or perceptual) primarily contributed to the generalization, thus highlighting the potential need to expose participants to longer training protocols.

Virtual reality games for spatial hearing training in children and young people with bilateral cochlear implants: the "Both Ears (BEARS)" approach

Spatial hearing relies on the encoding of perceptual sound location cues in space. It is critical for communicating in background noise, and understanding where sounds are coming from (sound localization). Although there are some monoaural spatial hearing cues (i.e., from one ear), most of our spatial hearing skills require binaural hearing (i.e., from two ears). Cochlear implants (CIs) are often the most appropriate rehabilitation for individuals with severe-to-profound hearing loss, with those aged 18 years of age and younger typically receiving bilateral implants (one in each ear). As experience with bilateral hearing increases, individuals tend to improve their spatial hearing skills. Extensive research demonstrates that training can enhance sound localization, speech understanding in noise, and music perception. The BEARS (Both Ears) approach utilizes Virtual Reality (VR) games specifically designed for young people with bilateral CIs to train and improve spatial hearing skills. This paper outlines the BEARS approach by: (i) emphasizing the need for more robust and engaging rehabilitation techniques, (ii) presenting the BEARS logic model that underpins the intervention, and (iii) detailing the assessment tools that will be employed in a clinical trial to evaluate the effectiveness of BEARS in alignment with the logic model.

The Suite for the Assessment of Low-Level cues on Orientation (SALLO): The psychophysics of spatial orientation in virtual reality

Spatial orientation is a complex ability that emerges from the interaction of several systems in a way that is still unclear. One of the reasons limiting the research on the topic is the lack of methodologies aimed at studying multimodal psychophysics in an ecological manner and with affordable settings. Virtual reality can provide a workaround to this impasse by using virtual stimuli rather than real ones. However, the available virtual reality development platforms are not meant for psychophysical testing; therefore, using them as such can be very difficult for newcomers, especially the ones new to coding. For this reason, we developed SALLO, the Suite for the Assessment of Low-Level cues on Orientation, which is a suite of utilities that simplifies assessing the psychophysics of multimodal spatial orientation in virtual reality. The tools in it cover all the fundamental steps to design a psychophysical experiment. Plus, dedicated tracks guide the users in extending the suite components to simplify developing new experiments. An experimental use-case used SALLO and virtual reality to show that the head posture affects both the egocentric and the allocentric mental representations of spatial orientation. Such a use-case demonstrated how SALLO and virtual reality can be used to accelerate hypothesis testing concerning the psychophysics of spatial orientation and, more broadly, how the community of researchers in the field may benefit from such a tool to carry out their investigations.

Spatial hearing training in virtual reality with simulated asymmetric hearing loss

Sound localization is essential to perceive the surrounding world and to interact with objects. This ability can be learned across time, and multisensory and motor cues play a crucial role in the learning process. A recent study demonstrated that when training localization skills, reaching to the sound source to determine its position reduced localization errors faster and to a greater extent as compared to just naming sources' positions, despite the fact that in both tasks, participants received the same feedback about the correct position of sound sources in case of wrong response. However, it remains to establish which features have made reaching to sound more effective as compared to naming. In the present study, we introduced a further condition in which the hand is the effector providing the response, but without it reaching toward the space occupied by the target source: the pointing condition. We tested three groups of participants (naming, pointing, and reaching groups) each while performing a sound localization task in normal and altered listening situations (i.e. mild-moderate unilateral hearing loss) simulated through auditory virtual reality technology. The experiment comprised four blocks: during the first and the last block, participants were tested in normal listening condition, while during the second and the third in altered listening condition. We measured their performance, their subjective judgments (e.g. effort), and their head-related behavior (through kinematic tracking). First, people's performance decreased when exposed to asymmetrical mild-moderate hearing impairment, more specifically on the ipsilateral side and for the pointing group. Second, we documented that all groups decreased their localization errors across altered listening blocks, but the extent of this reduction was higher for reaching and pointing as compared to the naming group. Crucially, the reaching group leads to a greater error reduction for the side where the listening alteration was applied. Furthermore, we documented that, across blocks, reaching and pointing groups increased the implementation of head motor behavior during the task (i.e., they increased approaching head movements toward the space of the sound) more than naming. Third, while performance in the unaltered blocks (first and last) was comparable, only the reaching group continued to exhibit a head behavior similar to those developed during the altered blocks (second and third), corroborating the previous observed relationship between the reaching to sounds task and head movements. In conclusion, this study further demonstrated the effectiveness of reaching to sounds as compared to pointing and naming in the learning processes. This effect could be related both to the process of implementing goal-directed motor actions and to the role of reaching actions in fostering the implementation of head-related motor strategies.

Spatial release from masking in the median plane with non-native speakers using individual and mannequin head related transfer functions

Spatial release from masking (SRM) in speech-on-speech tasks has been widely studied in the horizontal plane, where interaural cues play a fundamental role. Several studies have also observed SRM for sources located in the median plane, where (monaural) spectral cues are more important. However, a relatively unexplored research question concerns the impact of head-related transfer function (HRTF) personalisation on SRM, for example, whether using individually-measured HRTFs results in better performance if compared with the use of mannequin HRTFs. This study compares SRM in the median plane in a speech-on-speech virtual task rendered using both individual and mannequin HRTFs. SRM is obtained using English sentences with non-native English speakers. Our participants show lower SRM performances compared to those found by others using native English participants. Furthermore, SRM is significantly larger when the source is spatialised using the individual HRTF, and this effect is more marked for those with lower English proficiency. Further analyses using a spectral distortion metric and the estimation of the better-ear effect, show that the observed SRM can only partially be explained by HRTF-specific factors and that the effect of the familiarity with individual spatial cues is likely to be the most significant element driving these results.

Toward Sound Localization Testing in Virtual Reality to Aid in the Screening of Auditory Processing Disorders

Sound localization testing is key for comprehensive hearing evaluations, particularly in cases of suspected auditory processing disorders. However, sound localization is not commonly assessed in clinical practice, likely due to the complexity and size of conventional measurement systems, which require semicircular loudspeaker arrays in large and acoustically treated rooms. To address this issue, we investigated the feasibility of testing sound localization in virtual reality (VR). Previous research has shown that virtualization can lead to an increase in localization blur. To measure these effects, we conducted a study with a group of normal-hearing adults, comparing sound localization performance in different augmented reality and VR scenarios. We started with a conventional loudspeaker-based measurement setup and gradually moved to a virtual audiovisual environment, testing sound localization in each scenario using a within-participant design. The loudspeaker-based experiment yielded results comparable to those reported in the literature, and the results of the virtual localization test provided new insights into localization performance in state-of-the-art VR environments. By comparing localization performance between the loudspeaker-based and virtual conditions, we were able to estimate the increase in localization blur induced by virtualization relative to a conventional test setup. Notably, our study provides the first proxy normative cutoff values for sound localization testing in VR. As an outlook, we discuss the potential of a VR-based sound localization test as a suitable, accessible, and portable alternative to conventional setups and how it could serve as a time- and resource-saving prescreening tool to avoid unnecessarily extensive and complex laboratory testing.

Audio-Corsi: a novel system to evaluate audio-spatial memory skills

Spatial memory (SM) is a multimodal representation of the external world, which different sensory inputs can mediate. It is essential in accomplishing everyday activities and strongly correlates with sleep processes. However, despite valuable knowledge of the spatial mechanisms in the visual modality, the multi-sensory aspects of SM have yet to be thoroughly investigated due to a lack of proper technologies.This work presents a novel acoustic system built around 3D audio spatial technology. Our goal was to examine if an afternoon nap can improve memory performance, measured through the acoustic version of the Corsi Block Tapping Task (CBTT), named Audio-Corsi. We tested five adults over two days. During one of the two days (Wake), participants performed the Audio-Corsi before (Pre) and after (Post) a wake resting period; while the other day (Sleep), participants performed the Audio-Corsi before (Pre) and after (Post) a nap. Day orders were randomized. We calculated the memory span for the Pre and Post session in both the Wake and Sleep days. Preliminary results show a significant difference in the memory span between the Wake and Sleep days. Specifically, memory span decreased between the pre-and post-test during the wake day. The opposite trend was found for the sleep day. Results indicate that SM can be improved by sleeping also in the acoustic modality other than the visual one.Clinical Relevance- The technology and procedure we designed and developed could be suitable in clinical and experimental settings to study high-level cognitive skills in the auditory sensory modality and their relationship with sleep, especially when vision is absent or distorted (i.e. blindness).

How much I moved: Robust biases in self-rotation perception

Vestibular cues are crucial to sense the linear and angular acceleration of our head in three-dimensional space. Previous literature showed that vestibular information precociously combines with other sensory modalities, such as proprioceptive and visual, to facilitate spatial navigation. Recent studies suggest that auditory cues may improve self-motion perception as well. The present study investigated the ability to estimate passive rotational displacements with and without virtual acoustic landmarks to determine how vestibular and auditory information interact in processing self-motion information. We performed two experiments. In both, healthy participants sat on a Rotational-Translational Chair. They experienced yaw rotations along the earth-vertical axis and performed a self-motion discrimination task. Their goal was to estimate both clockwise and counterclockwise rotations’ amplitude, with no visual information available, reporting whether they felt to be rotated more or less than 45°. According to the condition, vestibular-only or audio-vestibular information was present. Between the two experiments, we manipulated the procedure of presentation of the auditory cues (passive vs. active production of sounds). We computed the point of subjective equality (PSE) as a measure of accuracy and the just noticeable difference (JND) as the precision of the estimations for each condition and direction of rotations. Results in both experiments show a strong overestimation bias of the rotations, regardless of the condition, the direction, and the sound generation conditions. Similar to previously found heading biases, this bias in rotation estimation may facilitate the perception of substantial deviations from the most relevant directions in daily navigation activities.

Evaluating Spatial Hearing Using a Dual-Task Approach in a Virtual-Acoustics Environment

Spatial hearing is critical for communication in everyday sound-rich environments. It is important to gain an understanding of how well users of bilateral hearing devices function in these conditions. The purpose of this work was to evaluate a Virtual Acoustics (VA) version of the Spatial Speech in Noise (SSiN) test, the SSiN-VA. This implementation uses relatively inexpensive equipment and can be performed outside the clinic, allowing for regular monitoring of spatial-hearing performance. The SSiN-VA simultaneously assesses speech discrimination and relative localization with changing source locations in the presence of noise. The use of simultaneous tasks increases the cognitive load to better represent the difficulties faced by listeners in noisy real-world environments. Current clinical assessments may require costly equipment which has a large footprint. Consequently, spatial-hearing assessments may not be conducted at all. Additionally, as patients take greater control of their healthcare outcomes and a greater number of clinical appointments are conducted remotely, outcome measures that allow patients to carry out assessments at home are becoming more relevant. The SSiN-VA was implemented using the 3D Tune-In Toolkit, simulating seven loudspeaker locations spaced at 30° intervals with azimuths between -90° and +90°, and rendered for headphone playback using the binaural spatialization technique. Twelve normal-hearing participants were assessed to evaluate if SSiN-VA produced patterns of responses for relative localization and speech discrimination as a function of azimuth similar to those previously obtained using loudspeaker arrays. Additionally, the effect of the signal-to-noise ratio (SNR), the direction of the shift from target to reference, and the target phonetic contrast on performance were investigated. SSiN-VA led to similar patterns of performance as a function of spatial location compared to loudspeaker setups for both relative localization and speech discrimination. Performance for relative localization was significantly better at the highest SNR than at the lowest SNR tested, and a target shift to the right was associated with an increased likelihood of a correct response. For word discrimination, there was an interaction between SNR and word group. Overall, these outcomes support the use of virtual audio for speech discrimination and relative localization testing in noise.

Involving Children and Teenagers With Bilateral Cochlear Implants in the Design of the BEARS (Both EARS) Virtual Reality Training Suite Improves Personalization

Older children and teenagers with bilateral cochlear implants often have poor spatial hearing because they cannot fuse sounds from the two ears. This deficit jeopardizes speech and language development, education, and social well-being. The lack of protocols for fitting bilateral cochlear implants and resources for spatial-hearing training contribute to these difficulties. Spatial hearing develops with bilateral experience. A large body of research demonstrates that sound localisation can improve with training, underpinned by plasticity-driven changes in the auditory pathways. Generalizing training to non-trained auditory skills is best achieved by using a multi-modal (audio-visual) implementation and multi-domain training tasks (localisation, speech-in-noise, and spatial music). The goal of this work was to develop a package of virtual-reality games (BEARS, Both EARS) to train spatial hearing in young people (8–16 years) with bilateral cochlear implants using an action-research protocol. The action research protocol used formalized cycles for participants to trial aspects of the BEARS suite, reflect on their experiences, and in turn inform changes in the game implementations. This participatory design used the stakeholder participants as co-creators. The cycles for each of the three domains (localisation, spatial speech-in-noise, and spatial music) were customized to focus on the elements that the stakeholder participants considered important. The participants agreed that the final games were appropriate and ready to be used by patients. The main areas of modification were: the variety of immersive scenarios to cover age range and interests, the number of levels of complexity to ensure small improvements were measurable, feedback, and reward schemes to ensure positive reinforcement, and an additional implementation on an iPad for those who had difficulties with the headsets due to age or balance issues. The effectiveness of the BEARS training suite will be evaluated in a large-scale clinical trial to determine if using the games lead to improvements in speech-in-noise, quality of life, perceived benefit, and cost utility. Such interventions allow patients to take control of their own management reducing the reliance on outpatient-based rehabilitation. For young people, a virtual-reality implementation is more engaging than traditional rehabilitation methods, and the participatory design used here has ensured that the BEARS games are relevant.

Speech signal enhancement in cocktail party scenarios by deep learning based virtual sensing of head-mounted microphones

The cocktail party effect refers to the human sense of hearing's ability to pay attention to a single conversation while filtering out all other background noise. To mimic this human hearing ability for people with hearing loss, scientists integrate beamforming algorithms into the signal processing path of hearing aids or implants' audio processors. Although these algorithms' performance strongly depends on the number and spatial arrangement of the microphones, most devices are equipped with a small number of microphones mounted close to each other on the audio processor housing. We measured and evaluated the impact of the number and spatial arrangement of hearing aid or head-mounted microphones on the performance of the established Minimum Variance Distortionless Response beamformer in cocktail party scenarios. The measurements revealed that the optimal microphone placement exploits monaural cues (pinna-effect), is close to the target signal, and creates a large distance spread due to its spatial arrangement. However, this microphone placement is impractical for hearing aid or implant users, as it includes microphone positions such as on the forehead. To overcome microphones' placement at impractical positions, we propose a deep virtual sensing estimation of the corresponding audio signals. The results of objective measures and a subjective listening test with 20 participants showed that the virtually sensed microphone signals significantly improved the speech quality, especially in cocktail party scenarios with low signal-to-noise ratios. Subjective speech quality was assessed using a 3-alternative forced choice procedure to determine which of the presented speech mixtures was most pleasant to understand. Hearing aid and cochlear implant (CI) users might benefit from the presented approach using virtually sensed microphone signals, especially in noisy environments.

Impact of non-individualised head related transfer functions on speech-in-noise performances within a synthesised virtual environment

When performing binaural spatialisation, it is widely accepted that the choice of the head related transfer functions (HRTFs), and in particular the use of individually measured ones, can have an impact on localisation accuracy, externalization, and overall realism. Yet the impact of HRTF choices on speech-in-noise performances in cocktail party-like scenarios has not been investigated in depth. This paper introduces a study where 22 participants were presented with a frontal speech target and two lateral maskers, spatialised using a set of non-individual HRTFs. Speech reception threshold (SRT) was measured for each HRTF. Furthermore, using the SRT predicted by an existing speech perception model, the measured values were compensated in the attempt to remove overall HRTF-specific benefits. Results show significant overall differences among the SRTs measured using different HRTFs, consistently with the results predicted by the model. Individual differences between participants related to their SRT performances using different HRTFs could also be found, but their significance was reduced after the compensation. The implications of these findings are relevant to several research areas related to spatial hearing and speech perception, suggesting that when testing speech-in-noise performances within binaurally rendered virtual environments, the choice of the HRTF for each individual should be carefully considered.

Design and Evaluation of a Web- and Mobile-Based Binaural Audio Platform for Cultural Heritage

PlugSonic is a suite of web- and mobile-based applications for the curation and experience of 3D interactive soundscapes and sonic narratives in the cultural heritage context. It was developed as part of the PLUGGY EU project (Pluggable Social Platform for Heritage Awareness and Participation) and consists of two main applications: PlugSonic Sample, to edit and apply audio effects, and PlugSonic Soundscape, to create and experience 3D soundscapes for headphones playback. The audio processing within PlugSonic is based on the Web Audio API and the 3D Tune-In Toolkit, while the mobile exploration of soundscapes in a physical space is obtained using Apple’s ARKit. The main goal of PlugSonic is technology democratisation; PlugSonic users—whether cultural institutions or citizens—are all given the instruments needed to create, process and experience 3D soundscapes and sonic narratives; without the need for specific devices, external tools (software and/or hardware), specialised knowledge or custom development. The aims of this paper are to present the design and development choices, the user involvement processes as well as a final evaluation conducted with inexperienced users on three tasks (creation, curation and experience), demonstrating how PlugSonic is indeed a simple, effective, yet powerful tool.

Perceptual implications of different Ambisonics-based methods for binaural reverberation

Reverberation is essential for the realistic auralisation of enclosed spaces. However, it can be computationally expensive to render with high fidelity and, in practice, simplified models are typically used to lower costs while preserving perceived quality. Ambisonics-based methods may be employed to this purpose as they allow us to render a reverberant sound field more efficiently by limiting its spatial resolution. The present study explores the perceptual impact of two simplifications of Ambisonics-based binaural reverberation that aim to improve efficiency. First, a "hybrid Ambisonics" approach is proposed in which the direct sound path is generated by convolution with a spatially dense head related impulse response set, separately from reverberation. Second, the reverberant virtual loudspeaker method (RVL) is presented as a computationally efficient approach to dynamically render binaural reverberation for multiple sources with the potential limitation of inaccurately simulating listener's head rotations. Numerical and perceptual evaluations suggest that the perceived quality of hybrid Ambisonics auralisations of two measured rooms ceased to improve beyond the third order, which is a lower threshold than what was found by previous studies in which the direct sound path was not processed separately. Additionally, RVL is shown to produce auralisations with comparable perceived quality to Ambisonics renderings.

Speech stream segregation to control an ERP-based auditory BCI

OBJECTIVE

The use of natural sounds in auditory Brain-Computer Interfaces (BCI) has been shown to improve classification results and usability. Some auditory BCIs are based on stream segregation, in which the subjects must attend one audio stream and ignore the other(s); these streams include some kind of stimuli to be detected. In this work we focus on Event-Related Potentials (ERP) and study whether providing intelligible content to each audio stream could help the users to better concentrate on the desired stream and so to better attend the target stimuli and to ignore the non-target ones.

APPROACH

In addition to a control condition, two experimental conditions, based on the selective attention and the cocktail party effect, were tested using two simultaneous and spatialized audio streams: i) the condition A2 consisted of an overlap of auditory stimuli (single syllables) on a background consisting of natural speech for each stream, ii) in condition A3, brief alterations of the natural flow of each speech were used as stimuli.

MAIN RESULTS

The two experimental proposals improved the results of the control condition (single words as stimuli without a speech background) both in a cross validation analysis of the calibration part and in the online test. The analysis of the ERP responses also presented better discriminability for the two proposals in comparison to the control condition. The results of subjective questionnaires support the better usability of the first experimental condition.

SIGNIFICANCE

The use of natural speech as background improves the stream segregation in an ERP-based auditory BCI (with significant results in the performance metrics, the ERP waveforms, and in the preference parameter in subjective questionnaires). Future work in the field of ERP-based stream segregation should study the use of natural speech in combination with easily perceived but not distracting stimuli.

Multichannel acoustic source and image dataset for the cocktail party effect in hearing aid and implant users

The Cocktail Party Effect refers to the ability of the human sense of hearing to extract a specific target sound source from a mixture of background noises in complex acoustic scenarios. The ease with which normal hearing people perform this challenging task is in stark contrast to the difficulties that hearing-impaired subjects face in these situations. To help patients with hearing aids and implants, scientists are trying to imitate this ability of human hearing, with modest success so far. To support the scientific community in its efforts, we provide the Bern Cocktail Party (BCP) dataset consisting of 55938 Cocktail Party scenarios recorded from 20 people and a head and torso simulator wearing cochlear implant audio processors. The data were collected in an acoustic chamber with 16 synchronized microphones placed at purposeful positions on the participants' heads. In addition to the multi-channel audio source and image recordings, the spatial coordinates of the microphone positions were digitized for each participant. Python scripts were provided to facilitate data processing.

Play&Tune: User Feedback in the Development of a Serious Game for Optimizing Hearing Aid Orientation

Many hearing aid (HA) users are dissatisfied with HA performance in social situations. One way to improve HA outcomes is training the users to understand how HAs work. Play&Tune was designed to provide this training and to foster autonomy in hearing rehabilitation. We carried out two prototype evaluations and a prerelease evaluation of Play&Tune with 71 HA users, using an interview or online survey. Users gave detailed feedback on their experiences with the app. Most participants enjoyed learning about HAs and expressed a desire for autonomy over their HA settings. Our case study reinforces the importance of user feedback during app development.