Two-dimensional shape discrimination by sighted people using simulated virtual echoes

EchoSee: An Assistive Mobile Application for Real-Time 3D Environment Reconstruction and Sonification Supporting Enhanced Navigation for People with Vision Impairments

Improving the quality of life for people with vision impairments has been an important goal in the research and design of assistive devices for several decades. This paper seeks to further that goal by introducing a novel assistive technology platform that leverages real-time 3D spatial audio to promote safe and efficient navigation for people who are blind or visually impaired (PVI). The presented platform, EchoSee, uses modern 3D scanning technology on a mobile device to construct a live, digital 3D map of a user's environment as they move about their surroundings. Spatialized, virtual audio sources (i.e., virtual speakers) are dynamically placed within the digital 3D scan of the world, providing the navigator with a real-time 3D stereo audio "soundscape." The digital 3D map, and its resultant soundscape, are continuously updated as the user moves about their environment. The generated soundscape is played back through headphones connected to the navigator's device. This paper details (1) the underlying technical components and how they were integrated to produce the mobile application that generates a dynamic soundscape on a consumer mobile device, (2) a methodology for analyzing navigation performance with the application, (3) the design and execution of a user study investigating the effectiveness of the presented system, and (4) a discussion of the results of that study along with a proposed future study and possible improvements. Altogether, this paper presents a novel software platform aimed at assisting individuals with vision impairments to navigate and understand spaces safely, efficiently, and independently and the results of a feasibility study analyzing the viability of the approach.

Object recognition via echoes: quantifying the crossmodal transfer of three-dimensional shape information between echolocation, vision, and haptics

Active echolocation allows blind individuals to explore their surroundings via self-generated sounds, similarly to dolphins and other echolocating animals. Echolocators emit sounds, such as finger snaps or mouth clicks, and parse the returning echoes for information about their surroundings, including the location, size, and material composition of objects. Because a crucial function of perceiving objects is to enable effective interaction with them, it is important to understand the degree to which three-dimensional shape information extracted from object echoes is useful in the context of other modalities such as haptics or vision. Here, we investigated the resolution of crossmodal transfer of object-level information between acoustic echoes and other senses. First, in a delayed match-to-sample task, blind expert echolocators and sighted control participants inspected common (everyday) and novel target objects using echolocation, then distinguished the target object from a distractor using only haptic information. For blind participants, discrimination accuracy was overall above chance and similar for both common and novel objects, whereas as a group, sighted participants performed above chance for the common, but not novel objects, suggesting that some coarse object information (a) is available to both expert blind and novice sighted echolocators, (b) transfers from auditory to haptic modalities, and (c) may be facilitated by prior object familiarity and/or material differences, particularly for novice echolocators. Next, to estimate an equivalent resolution in visual terms, we briefly presented blurred images of the novel stimuli to sighted participants (N = 22), who then performed the same haptic discrimination task. We found that visuo-haptic discrimination performance approximately matched echo-haptic discrimination for a Gaussian blur kernel σ of ~2.5°. In this way, by matching visual and echo-based contributions to object discrimination, we can estimate the quality of echoacoustic information that transfers to other sensory modalities, predict theoretical bounds on perception, and inform the design of assistive techniques and technology available for blind individuals.

Auditory orientation and distance estimation of sighted humans using virtual echolocation with artificial and self-generated sounds

Active echolocation of sighted humans using predefined synthetic and self-emitted sounds, as habitually used by blind individuals, was investigated. Using virtual acoustics, distance estimation and directional localization of a wall in different rooms were assessed. A virtual source was attached to either the head or hand with realistic or increased source directivity. A control condition was tested with a virtual sound source located at the wall. Untrained echolocation performance comparable to performance in the control condition was achieved on an individual level. On average, the echolocation performance was considerably lower than in the control condition, however, it benefitted from increased directivity.

Analysis of echolocation behavior of bats in "echo space" using acoustic simulation

BACKGROUND

Echolocating bats use echo information to perceive space, control their behavior, and adjust flight navigation strategies in various environments. However, the echolocation behavior of bats, including echo information, has not been thoroughly investigated as it is technically difficult to measure all the echoes that reach the bats during flight, even with the conventional telemetry microphones currently in use. Therefore, we attempted to reproduce the echoes received at the location of bats during flight by combining acoustic simulation and behavioral experiments with acoustic measurements. By using acoustic simulation, echoes can be reproduced as temporal waveforms (including diffracted waves and multiple reflections), and detailed echo analysis is possible even in complex obstacle environments.

RESULTS

We visualized the spatiotemporal changes in the echo incidence points detected by bats during flight, which enabled us to investigate the "echo space" revealed through echolocation for the first time. We then hypothesized that by observing the differences in the "echo space" before and after spatial learning, the bats' attentional position would change. To test this hypothesis, we examined how the distribution of visualized echoes concentrated at the obstacle edges after the bats became more familiar with their environment. The echo incidence points appeared near the edge even when the pulse direction was not toward the edge. Furthermore, it was found that the echo direction correlated with the turn rate of the bat's flight path, revealing for the first time the relationship between the echo direction and the bat's flight path.

CONCLUSIONS

We were able to clarify for the first time how echoes space affects echolocation behavior in bats by combining acoustic simulations and behavioral experiments.