Sensory augmentation for a rapid motor task in a multisensory environment

BACKGROUND

Sensory substitution and augmentation systems (SSASy) seek to either replace or enhance existing sensory skills by providing a new route to access information about the world. Tests of such systems have largely been limited to untimed, unisensory tasks.

OBJECTIVE

To test the use of a SSASy for rapid, ballistic motor actions in a multisensory environment.

METHODS

Participants played a stripped-down version of air hockey in virtual reality with motion controls (Oculus Touch). They were trained to use a simple SASSy (novel audio cue) for the puck's location. They were tested on ability to strike an oncoming puck with the SASSy, degraded vision, or both.

RESULTS

Participants coordinated vision and the SSASy to strike the target with their hand more consistently than with the best single cue alone, t(13) = 9.16, p <.001, Cohen's d = 2.448.

CONCLUSIONS

People can adapt flexibly to using a SSASy in tasks that require tightly timed, precise, and rapid body movements. SSASys can augment and coordinate with existing sensorimotor skills rather than being limited to replacement use cases - in particular, there is potential scope for treating moderate vision loss. These findings point to the potential for augmenting human abilities, not only for static perceptual judgments, but in rapid and demanding perceptual-motor tasks.

Multisensory perception and decision-making with a new sensory skill

It is clear that people can learn a new sensory skill-a new way of mapping sensory inputs onto world states. It remains unclear how flexibly a new sensory skill can become embedded in multisensory perception and decision-making. To address this, we trained typically sighted participants (N = 12) to use a new echo-like auditory cue to distance in a virtual world, together with a noisy visual cue. Using model-based analyses, we tested for key markers of efficient multisensory perception and decision-making with the new skill. We found that 12 of 14 participants learned to judge distance using the novel auditory cue. Their use of this new sensory skill showed three key features: (a) It enhanced the speed of timed decisions; (b) it largely resisted interference from a simultaneous digit span task; and (c) it integrated with vision in a Bayes-like manner to improve precision. We also show some limits following this relatively short training: Precision benefits were lower than the Bayes-optimal prediction, and there was no forced fusion of signals. We conclude that people already embed new sensory skills in flexible multisensory perception and decision-making after a short training period. A key application of these insights is to the development of sensory augmentation systems that can enhance human perceptual abilities in novel ways. The limitations we reveal (sub-optimality, lack of fusion) provide a foundation for further investigations of the limits of these abilities and their brain basis. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

An adaptive cue selection model of allocentric spatial reorientation

After becoming disoriented, an organism must use the local environment to reorient and recover vectors to important locations. A new theory, adaptive combination, suggests that the information from different spatial cues is combined with Bayesian efficiency during reorientation. To test this further, we modified the standard reorientation paradigm to be more amenable to Bayesian cue combination analyses while still requiring reorientation in an allocentric (i.e., world-based, not egocentric) frame. Twelve adults and 20 children at ages 5 to 7 years old were asked to recall locations in a virtual environment after a disorientation. Results were not consistent with adaptive combination. Instead, they are consistent with the use of the most useful (nearest) single landmark in isolation. We term this adaptive selection. Experiment 2 suggests that adults also use the adaptive selection method when they are not disoriented but are still required to use a local allocentric frame. This suggests that the process of recalling a location in the allocentric frame is typically guided by the single most useful landmark rather than a Bayesian combination of landmarks. These results illustrate that there can be important limits to Bayesian theories of the cognition, particularly for complex tasks such as allocentric recall.

Whether studying the development of children’s spatial cognition, creating artificial intelligence with human-like capacities, or designing civic spaces, we can benefit from a strong understanding of how humans process the space around them. Here we tested a prominent theory that brings together statistical theory and psychological theory (Bayesian models of perception and memory) but found that it could not satisfactorily explain our data. Our findings suggest that when tracking the spatial relations between objects from different viewpoints, rather than efficiently combining all the available landmarks, people often fall back to the much simpler method of tracking the spatial relation to the nearest landmark.

Coding Locations Relative to One or Many Landmarks in Childhood

Cognitive development studies how information processing in the brain changes over the course of development. A key part of this question is how information is represented and stored in memory. This study examined allocentric (world-based) spatial memory, an important cognitive tool for planning routes and interacting with the space around us. This is typically theorized to use multiple landmarks all at once whenever it operates. In contrast, here we show that allocentric spatial memory frequently operates over a limited spatial window, much less than the full proximal scene, for children between 3.5 and 8.5 years old. The use of multiple landmarks increases gradually with age. Participants were asked to point to a remembered target location after a change of view in immersive virtual reality. A k-fold cross-validation model-comparison selected a model where young children usually use the target location's vector to the single nearest landmark and rarely take advantage of the vectors to other nearby landmarks. The comparison models, which attempt to explain the errors as generic forms of noise rather than encoding to a single spatial cue, did not capture the distribution of responses as well. Parameter fits of this new single- versus multi-cue model are also easily interpretable and related to other variables of interest in development (age, executive function). Based on this, we theorize that spatial memory in humans develops through three advancing levels (but not strict stages): most likely to encode locations egocentrically (relative to the self), then allocentrically (relative to the world) but using only one landmark, and finally, most likely to encode locations relative to multiple parts of the scene.

Boundaries in spatial cognition: Looking like a boundary is more important than being a boundary

Large walls and other typical boundaries strongly influence neural activity related to navigation and the representations of spatial layouts. They are also major aids to reliable navigation behavior in young children and nonhuman animals. Is this because they are physical boundaries (barriers to movement), or because they present certain visual features, such as visually extended 3D surfaces? Here, these 2 factors were dissociated by using immersive virtual reality and real boundaries. Eighty adults recalled target locations in 1 of 4 environments: plywood, where a virtual wall coincided with a large piece of real plywood; pass through, where the virtual wall coincided with empty space and participants could pass through it; pass over, where the virtual wall was projected downward to be visible underneath a transparent floor; and cones, where the walls were replaced with traffic cones. One condition had features that were boundaries and looked like boundaries (plywood); 2 had features that were not boundaries but looked like boundaries (pass over/through); and 1 had features that were not boundaries and did not look like boundaries (cones). The precision and bias of responses changed only as a function of looking like a boundary. This suggests that variations in spatial coding are more closely linked to the visual properties of environmental layouts than to whether they contain physical boundaries (barriers to movement). (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Bayes-Like Integration of a New Sensory Skill with Vision

Humans are effective at dealing with noisy, probabilistic information in familiar settings. One hallmark of this is Bayesian Cue Combination: combining multiple noisy estimates to increase precision beyond the best single estimate, taking into account their reliabilities. Here we show that adults also combine a novel audio cue to distance, akin to human echolocation, with a visual cue. Following two hours of training, subjects were more precise given both cues together versus the best single cue. This persisted when we changed the novel cue's auditory frequency. Reliability changes also led to a re-weighting of cues without feedback, showing that they learned something more flexible than a rote decision rule for specific stimuli. The main findings replicated with a vibrotactile cue. These results show that the mature sensory apparatus can learn to flexibly integrate new sensory skills. The findings are unexpected considering previous empirical results and current models of multisensory learning.

Strong evidence that callous-unemotional traits are not related to risk-taking task performance

A hypothesized association between callous-unemotional (CU) traits and risk-taking may account for the link between CU traits and real-world risky behaviors, such as illegal behavior. Prior findings show that reward and punishment responsivity differs in relation to CU traits, but is not associated with general risk-taking. However this has only been examined previously with one task, only with a frequentist framework, and with limited interpretation. Here, we expand to another task and to Bayesian analyses. A total of 657 participants (52% female) completed the Inventory of Callous-Unemotional Traits, the Balloon Analogue Risk Task (essentially a gambling task), and the Stoplight driving task, which repeatedly presents participants with riskier or less risky choices to make while driving. We found strong evidence for the null model, in which there is no relation between the two risk-taking tasks and CU traits (R ² = 0.001; BF ₁₀ = 1/60.22). These results suggest that general risk-taking does not underlie the real-world risky behavior of people with CU traits. Alternative explanations include a different method of valuing certain outcomes.

Four-Year-Olds Use a Mixture of Spatial Reference Frames

Keeping track of unseen objects is an important spatial skill. In order to do this, people must situate the object in terms of different frames of reference, including body position (egocentric frame of reference), landmarks in the surrounding environment (extrinsic frame reference), or other attached features (intrinsic frame of reference). Nardini et al. hid a toy in one of 12 cups in front of children, turned the array when they were not looking, and then asked them to point to the cup with the toy. This forced children to use the intrinsic frame (information about the array of cups) to locate the hidden toy. Three-year-olds made systematic errors by using the wrong frame of reference, 4-year-olds were at chance, and only 5- and 6-year-olds were successful. Can we better understand the developmental change that takes place at four years? This paper uses a modelling approach to re-examine the data and distinguish three possible strategies that could lead to the previous results at four years: (1) Children were choosing cups randomly, (2) Children were pointing between the egocentric/extrinsic-cued location and the correct target, and (3) Children were pointing near the egocentric/extrinsic-cued location on some trials and near the target on the rest. Results heavily favor the last possibility: 4-year-olds were not just guessing or trying to combine the available frames of reference. They were using the intrinsic frame on some trials, but not doing so consistently. These insights suggest that accounts of improving spatial performance at 4 years need to explain why there is a mixture of responses. Further application of the selected model also suggests that children become both more reliant on the correct frame and more accurate with any chosen frame as they mature.

Is there really a link between exact-number knowledge and approximate number system acuity in young children?

Although everyone perceives approximate numerosities, some people make more accurate estimates than others. The accuracy of this estimation is called approximate number system (ANS) acuity. Recently, several studies have reported that individual differences in young children's ANS acuity are correlated with their knowledge of exact numbers such as the word 'six' (Mussolin et al., 2012, Trends Neurosci. Educ., 1, 21; Shusterman et al., 2011, Connecting early number word knowledge and approximate number system acuity; Wagner & Johnson, 2011, Cognition, 119, 10; see also Abreu-Mendoza et al., 2013, Front. Psychol., 4, 1). This study argues that this correlation should not be trusted. It seems to be an artefact of the procedure used to assess ANS acuity in children. The correlation arises because (1) some experimental designs inadvertently allow children to answer correctly based on the size (rather than the number) of dots in the display and/or (2) young children with little exact-number knowledge may not understand the phrase 'more dots' to mean numerically more. When the task is modified to make sure that children respond on the basis of numerosity, the correlation between ANS acuity and exact-number knowledge in normally developing children disappears.

Are bilingual children better at ignoring perceptually misleading information? A novel test

Does speaking more than one language help a child perform better on certain types of cognitive tasks? One possibility is that bilingualism confers either specific or general cognitive advantages on tasks that require selective attention to one dimension over another (e.g. Bialystok, ; Hilchey & Klein, ). Other studies have looked for such an advantage but found none (e.g. Morton & Harper, ; Paap & Greenberg, ). The present study compared monolingual and bilingual children's performance on a numerical discrimination task, which required children to ignore area and attend to number. Ninety-two children, ages 3 to 6 years, were asked which of two arrays of dots had 'more dots'. Half of the trials were congruent, where the numerically greater array was also larger in total area, and half were incongruent, where the numerically greater array was smaller in total area. All children performed better on congruent than on incongruent trials. Older children were more successful than younger children at ignoring area in favor of number. Bilingual children did not perform differently from monolingual children either in number discrimination itself (i.e. identifying which array had more dots) or at selectively attending to number. The present study thus finds no evidence of a bilingual advantage on this task for children of this age.

A number of options: rationalist, constructivist, and Bayesian insights into the development of exact-number concepts

The question of how human beings acquire exact-number concepts has interested cognitive developmentalists since the time of Piaget. The answer will owe something to both the rationalist and constructivist traditions. On the one hand, some aspects of numerical cognition (e.g. approximate number estimation and the ability to track small sets of one to four individuals) are innate or early-developing and are shared widely among species. On the other hand, only humans create representations of exact, large numbers such as 42, as distinct from both 41 and 43. These representations seem to be constructed slowly, over a period of months or years during early childhood. The task for researchers is to distinguish the innate representational resources from those that are constructed, and to characterize the construction process. Bayesian approaches can be useful to this project in at least three ways: (1) As a way to analyze data, which may have distinct advantages over more traditional methods (e.g. making it possible to find support for a nuli hypothesis); (2) as a way of modeling children's performance on specific tasks: Peculiarities of the task are captured as a prior; the child's knowledge is captured in the way the prior is updated; and behavior is captured as a posterior distribution; and (3) as a way of modeling learning itself, by providing a formal account of how learners might choose among alternative hypotheses.

Number-concept acquisition and general vocabulary development

How is number-concept acquisition related to overall language development? Experiments 1 and 2 measured number-word knowledge and general vocabulary in a total of 59 children, ages 30-60 months. A strong correlation was found between number-word knowledge and vocabulary, independent of the child's age, contrary to previous results (D. Ansari et al., 2003). This result calls into question arguments that (a) the number-concept creation process is scaffolded mainly by visuo-spatial development and (b) that language only becomes integrated after the concepts are created (D. Ansari et al., 2003). Instead, this may suggest that having a larger nominal vocabulary helps children learn number words. Experiment 3 shows that the differences with previous results are likely due to changes in how the data were analyzed.