Evidence Integration in Natural Acoustic Textures during Active and Passive Listening

Sensory Evidence Accumulation Using Optic Flow in a Naturalistic Navigation Task

Sensory evidence accumulation is considered a hallmark of decision-making in noisy environments. Integration of sensory inputs has been traditionally studied using passive stimuli, segregating perception from action. Lessons learned from this approach, however, may not generalize to ethological behaviors like navigation, where there is an active interplay between perception and action. We designed a sensory-based sequential decision task in virtual reality in which humans and monkeys navigated to a memorized location by integrating optic flow generated by their own joystick movements. A major challenge in such closed-loop tasks is that subjects' actions will determine future sensory input, causing ambiguity about whether they rely on sensory input rather than expectations based solely on a learned model of the dynamics. To test whether subjects integrated optic flow over time, we used three independent experimental manipulations, unpredictable optic flow perturbations, which pushed subjects off their trajectory; gain manipulation of the joystick controller, which changed the consequences of actions; and manipulation of the optic flow density, which changed the information borne by sensory evidence. Our results suggest that both macaques (male) and humans (female/male) relied heavily on optic flow, thereby demonstrating a critical role for sensory evidence accumulation during naturalistic action-perception closed-loop tasks.SIGNIFICANCE STATEMENT The temporal integration of evidence is a fundamental component of mammalian intelligence. Yet, it has traditionally been studied using experimental paradigms that fail to capture the closed-loop interaction between actions and sensations inherent in real-world continuous behaviors. These conventional paradigms use binary decision tasks and passive stimuli with statistics that remain stationary over time. Instead, we developed a naturalistic visuomotor visual navigation paradigm that mimics the causal structure of real-world sensorimotor interactions and probed the extent to which participants integrate sensory evidence by adding task manipulations that reveal complementary aspects of the computation.

Representation Learning for Fine-Grained Change Detection

Fine-grained change detection in sensor data is very challenging for artificial intelligence though it is critically important in practice. It is the process of identifying differences in the state of an object or phenomenon where the differences are class-specific and are difficult to generalise. As a result, many recent technologies that leverage big data and deep learning struggle with this task. This review focuses on the state-of-the-art methods, applications, and challenges of representation learning for fine-grained change detection. Our research focuses on methods of harnessing the latent metric space of representation learning techniques as an interim output for hybrid human-machine intelligence. We review methods for transforming and projecting embedding space such that significant changes can be communicated more effectively and a more comprehensive interpretation of underlying relationships in sensor data is facilitated. We conduct this research in our work towards developing a method for aligning the axes of latent embedding space with meaningful real-world metrics so that the reasoning behind the detection of change in relation to past observations may be revealed and adjusted. This is an important topic in many fields concerned with producing more meaningful and explainable outputs from deep learning and also for providing means for knowledge injection and model calibration in order to maintain user confidence.

Assessing the state of consciousness for individual patients using complex, statistical stimuli

Patients with prolonged disorders of consciousness (PDOC) are often unable to communicate their state of consciousness. Determining the latter is essential for the patient's care and prospects of recovery. Auditory stimulation in combination with neural recordings is a promising technique towards an objective assessment of conscious awareness. Here, we investigated the potential of complex, acoustic stimuli to elicit EEG responses suitable for classifying multiple subject groups, from unconscious to responding. We presented naturalistic auditory textures with unexpectedly changing statistics to human listeners. Awake, active listeners were asked to indicate the change by button press, while all other groups (awake passive, asleep, minimally conscious state (MCS), and unresponsive wakefulness syndrome (UWS)) listened passively. We quantified the evoked potential at stimulus onset and change in stimulus statistics, as well as the complexity of neural response during the change of stimulus statistics. On the group level, onset and change potentials classified patients and healthy controls successfully but failed to differentiate between the UWS and MCS groups. Conversely, the Lempel-Ziv complexity of the scalp-level potential allowed reliable differentiation between UWS and MCS even for individual subjects, when compared with the clinical assessment aligned to the EEG measurements. The accuracy appears to improve further when taking the latest available clinical diagnosis into account. In summary, EEG signal complexity during onset and changes in complex acoustic stimuli provides an objective criterion for distinguishing states of consciousness in clinical patients. These results suggest EEG-recordings as a cost-effective tool to choose appropriate treatments for non-responsive PDOC patients.