Superhuman Hearing - Virtual Prototyping of Artificial Hearing: a Case Study on Interactions and Acoustic Beamforming

Physics-based generative adversarial network for real-time acoustic holography

Acoustic holography (AH) encodes the acoustic fields in high dimensions into two-dimensional holograms without information loss. Phase-only holography (POH) modulates only the phase profiles of the encoded hologram, establishing its superiority over alternative modulation schedules due to its information volume and storage efficiency. Moreover, POH implemented by a phased array of transducers (PAT) facilitates active and dynamic manipulation by independently modulating the phase of each transducer. However, existing algorithms for POH calculation suffer from a deficiency in terms of high fidelity and good real-time performance. Thus, a deep learning algorithm reinforced by the physical model, i.e. Angular Spectrum Method (ASM), is proposed to learn the inverse physical mapping from the target field to the source POH. This method comprises a generative adversarial network (GAN) evaluated by soft label, which is referred to as soft-GAN. Furthermore, to avoid the intrinsic limitation of neural networks on high-frequency features, a Y-Net structure is developed with two decoder branches in frequency and spatial domain, respectively. The proposed method achieves the reconstruction performance with a state-of-the-art (SOTA) Peak Signal-to-Noise Ratio (PSNR) of 24.05 dB. Experiment results demonstrated that the POH calculated by the proposed method enables accurate and real-time hologram reconstruction, showing enormous potential for applications.

A bioinspired configurable cochlea based on memristors

Cochleas are the basis for biology to process and recognize speech information, emulating which with electronic devices helps us construct high-efficient intelligent voice systems. Memristor provides novel physics for performing neuromorphic engineering beyond complementary metal-oxide-semiconductor technology. This work presents an artificial cochlea based on the shallen-key filter model configured with memristors, in which one filter emulates one channel. We first fabricate a memristor with the TiN/HfOx/TaOx/TiN structure to implement such a cochlea and demonstrate the non-volatile multilevel states through electrical operations. Then, we build the shallen-key filter circuit and experimentally demonstrate the frequency-selection function of cochlea’s five channels, whose central frequency is determined by the memristor’s resistance. To further demonstrate the feasibility of the cochlea for system applications, we use it to extract the speech signal features and then combine it with a convolutional neural network to recognize the Free Spoken Digit Dataset. The recognition accuracy reaches 92% with 64 channels, compatible with the traditional 64 Fourier transform transformation points of mel-frequency cepstral coefficients method with 95% recognition accuracy. This work provides a novel strategy for building cochleas, which has a great potential to conduct configurable, high-parallel, and high-efficient auditory systems for neuromorphic robots.

Invisible Boundaries for VR: Auditory and Haptic Signals as Indicators for Real World Boundaries

Maintaining awareness of real world boundaries whilst being immersed in virtual reality (VR) with head mounted displays (HMDs), is a necessity for the physical integrity of the user. This paper explores whether individual human senses can be allocated to the real and the virtual world and what effect this has on workload, presence, performance and perceived safety. We present the results of a lab study ( N=33) where the auditory and haptic sense of participants was trained to be an indicator for real world boundaries, while their visual sense was bound to a VR experience with an HMD. Our results suggests that allocating senses increases workload. However, while performance is comparable to purely visual indications of boundaries, sense allocation seems to improve presence. Participants prefer the signals to be separate or combined subsequently, depending on the priority and proximity to the boundary. This exploratory study is valuable for developers and researchers who want to start including audio and haptic signals as indicators for real world boundaries.