Enhancing Robotic Perception through Synchronized Simulation and Physical Common-Sense Reasoning

We introduce both conceptual and empirical findings arising from the amalgamation of a robotics cognitive architecture with an embedded physics simulator, aligning with the principles outlined in the intuitive physics literature. The employed robotic cognitive architecture, named CORTEX, leverages a highly efficient distributed working memory known as deep state representation. This working memory inherently encompasses a fundamental ontology, state persistency, geometric and logical relationships among elements, and tools for reading, updating, and reasoning about its contents. Our primary objective is to investigate the hypothesis that the integration of a physics simulator into the architecture streamlines the implementation of various functionalities that would otherwise necessitate extensive coding and debugging efforts. Furthermore, we categorize these enhanced functionalities into broad types based on the nature of the problems they address. These include addressing challenges related to occlusion, model-based perception, self-calibration, scene structural stability, and human activity interpretation. To demonstrate the outcomes of our experiments, we employ CoppeliaSim as the embedded simulator and both a Kinova Gen3 robotic arm and the Open-Manipulator-P as the real-world scenarios. Synchronization is maintained between the simulator and the stream of real events. Depending on the ongoing task, numerous queries are computed, and the results are projected into the working memory. Participating agents can then leverage this information to enhance overall performance.

The (Un)ideal Physicist: How Humans Rely on Object Interaction for Friction Estimates

To estimate object properties such as mass or friction, our brain relies on visual information to efficiently compute approximations. The role of sensorimotor feedback, however, is not well understood. Here we tested healthy adults (N = 79) in an inclined-plane problem, that is, how much a plane can be tilted before an object starts to slide, and contrasted the interaction group with observation groups who accessed involved forces by watching objects being manipulated. We created objects of different masses and levels of friction and asked participants to estimate the critical tilt angle after pushing an object, lifting it, or both. Estimates correlated with applied forces and were biased toward object mass, with higher estimates for heavier objects. Our findings highlight that inferences about physical object properties are tightly linked to the human sensorimotor system and that humans integrate sensorimotor information even at the risk of nonveridical perceptual estimates.

Seeing and understanding epistemic actions

Many actions have instrumental aims, in which we move our bodies to achieve a physical outcome in the environment. However, we also perform actions with epistemic aims, in which we move our bodies to acquire information and learn about the world. A large literature on action recognition investigates how observers represent and understand the former class of actions; but what about the latter class? Can one person tell, just by observing another person's movements, what they are trying to learn? Here, five experiments explore epistemic action understanding. We filmed volunteers playing a "physics game" consisting of two rounds: Players shook an opaque box and attempted to determine i) the number of objects hidden inside, or ii) the shape of the objects inside. Then, independent subjects watched these videos and were asked to determine which videos came from which round: Who was shaking for number and who was shaking for shape? Across several variations, observers successfully determined what an actor was trying to learn, based only on their actions (i.e., how they shook the box)-even when the box's contents were identical across rounds. These results demonstrate that humans can infer epistemic intent from physical behaviors, adding a new dimension to research on action understanding.

Grounding Intuitive Physics in Perceptual Experience

This review article explores the foundation of laypeople's understanding of the physical world rooted in perceptual experience. Beginning with a concise historical overview of the study of intuitive physics, the article presents the hypothesis that laypeople possess accurate internalized representations of physical laws. A key aspect of this hypothesis is the contention that correct representations of physical laws emerge in ecological experimental conditions, where the scenario being examined resembles everyday life experiences. The article critically examines empirical evidence both supporting and challenging this claim, revealing that despite everyday-life-like conditions, fundamental misconceptions often persist. Many of these misconceptions can be attributed to a domain-general heuristic that arises from the overgeneralization of perceptual-motor experiences with physical objects. To conclude, the article delves into ongoing controversies and highlights promising future avenues in the field of intuitive physics, including action-judgment dissociations, insights from developmental psychology, and computational models integrating artificial intelligence.

Physical inference of falling objects involves simulation of occluded trajectories in early visual areas

Humans possess an intuitive understanding of the environment's physical properties and dynamics, which allows them to predict the outcomes of physical scenarios and successfully interact with the physical world. This predictive ability is thought to rely on mental simulations and has been shown to involve frontoparietal areas. Here, we investigate whether such mental simulations may be accompanied by visual imagery of the predicted physical scene. We designed an intuitive physical inference task requiring participants to infer the parabolic trajectory of an occluded ball falling in accordance with Newtonian physics. Participants underwent fMRI while (i) performing the physical inference task alternately with a visually matched control task, and (ii) passively observing falling balls depicting the trajectories that had to be inferred during the physical inference task. We found that performing the physical inference task activates early visual areas together with a frontoparietal network when compared with the control task. Using multivariate pattern analysis, we show that these regions contain information specific to the trajectory of the occluded ball (i.e., fall direction), despite the absence of visual inputs. Using a cross-classification approach, we further show that in early visual areas, trajectory-specific activity patterns evoked by the physical inference task resemble those evoked by the passive observation of falling balls. Together, our findings suggest that participants simulated the ball trajectory when solving the task, and that the outcome of these simulations may be represented in form of the perceivable sensory consequences in early visual areas.

Seeing Soft Materials Draped Over Objects: A Case Study of Intuitive Physics in Perception, Attention, and Memory

We typically think of intuitive physics in terms of high-level cognition, but might aspects of physics also be extracted during lower-level visual processing? Might we not only think about physics, but also see it? We explored this using multiple tasks in online adult samples with objects covered by soft materials-as when you see a chair with a blanket draped over it-where you must account for the physical interactions between cloth, gravity, and object. In multiple change-detection experiments (n = 200), observers from an online testing marketplace were better at detecting image changes involving underlying object structure versus those involving only the superficial folds of cloths-even when the latter were more extreme along several dimensions. And in probe-comparison experiments (n = 100), performance was worse when both probes (vs. only one) appeared on image regions reflective of underlying object structure (equating visual properties). This work collectively shows how vision uses intuitive physics to recover the deeper underlying structure of scenes.

What would have happened? Counterfactuals, hypotheticals and causal judgements

How do people make causal judgements? In this paper, I show that counterfactual simulations are necessary for explaining causal judgements about events, and that hypotheticals do not suffice. In two experiments, participants viewed video clips of dynamic interactions between billiard balls. In Experiment 1, participants either made hypothetical judgements about whether ball B would go through the gate if ball A were not present in the scene, or counterfactual judgements about whether ball B would have gone through the gate if ball A had not been present. Because the clips featured a block in front of the gate that sometimes moved and sometimes stayed put, hypothetical and counterfactual judgements came apart. A computational model that evaluates hypotheticals and counterfactuals by running noisy physical simulations accurately captured participants' judgements. In Experiment 2, participants judged whether ball A caused ball B to go through the gate. The results showed a tight fit between counterfactual and causal judgements, whereas hypotheticals did not predict causal judgements. I discuss the implications of this work for theories of causality, and for studying the development of counterfactual thinking in children. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Visual object detection biases escape trajectories following acoustic startle in larval zebrafish

In this study, we investigated whether the larval zebrafish is sensitive to the presence of obstacles in its environment. Zebrafish execute fast escape swims when in danger of predation. We posited that collisions with solid objects during escape would be maladaptive to the fish, and therefore, the direction of escape swims should be informed by the locations of barriers. To test this idea, we developed a closed-loop imaging rig outfitted with barriers of various qualities. We show that when larval zebrafish escape in response to a non-directional vibrational stimulus, they use visual scene information to avoid collisions with obstacles. Our study demonstrates that barrier avoidance rate corresponds to the absolute distance of obstacles, as distant barriers outside of collision range elicit less bias than nearby collidable barriers that occupy the same amount of visual field. The computation of barrier avoidance is covert: the fact that fish will avoid barriers during escape cannot be predicted by its routine swimming behavior in the barrier arena. Finally, two-photon laser ablation experiments suggest that excitatory bias is provided to the Mauthner cell ipsilateral to approached barriers, either via direct excitation or a multi-step modulation process. We ultimately propose that zebrafish detect collidable objects via an integrative visual computation that is more complex than retinal occupancy alone, laying a groundwork for understanding how cognitive physical models observed in humans are implemented in an archetypal vertebrate brain. VIDEO ABSTRACT.

Invariant representation of physical stability in the human brain

Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.

Melting Ice With Your Mind: Representational Momentum for Physical States

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Precise functional connections between the dorsal anterior cingulate cortex and areas recruited for physical inference

Recent work has identified brain areas that are engaged when people predict how the physical behavior of the world will unfold - an ability termed intuitive physics. Among the many unanswered questions about the neural mechanisms of intuitive physics is where the key inputs come from: which brain regions connect up with intuitive physics processes to regulate when and how they are engaged in service of our goals? In the present work, we targeted the dorsal anterior cingulate cortex (dACC) for study based on characteristics that make it well-positioned to regulate intuitive physics processes. The dACC is richly interconnected with frontoparietal regions and is implicated in mapping contexts to actions, a process that would benefit from physical predictions to indicate which action(s) would produce the desired physical outcomes. We collected resting state functional MRI data in seventeen participants and used independent task-related runs to find the pattern of activity during a physical inference task in each individual participant. We found that the strongest resting state functional connections of the dACC not only aligned well with physical inference-related activity at the group level, it also mirrored individual differences in the positioning of physics-related activity across participants. Our results suggest that the dACC might be a key structure for regulating the engagement of intuitive physics processes in the brain.

Naturalistic embodied interactions elicit intuitive physical behaviour in accordance with Newtonian physics

The success of visuomotor interactions in everyday activities such as grasping or sliding a cup is inescapably governed by the laws of physics. Research on intuitive physics has predominantly investigated reasoning about objects' behaviour involving binary forced choice responses. We investigated how the type of visuomotor response influences participants' beliefs about physical quantities and their lawful relationship implicit in their active behaviour. Participants propelled pucks towards targets positioned at different distances. Analysis with a probabilistic model of interactions showed that subjects adopted the non-linear control prescribed by Newtonian physics when sliding real pucks in a virtual environment even in the absence of visual feedback. However, they used a linear heuristic when viewing the scene on a monitor and interactions were implemented through key presses. These results support the notion of probabilistic internal physics models but additionally suggest that humans can take advantage of embodied, sensorimotor, multimodal representations in physical scenarios.

A role for visual areas in physics simulations

To engage with the world, we must regularly make predictions about the outcomes of physical scenes. How do we make these predictions? Recent computational evidence points to simulation-the idea that we can introspectively manipulate rich, mental models of the world-as one explanation for how such predictions are accomplished. However, questions about the potential neural mechanisms of simulation remain. We hypothesized that the process of simulating physical events would evoke imagery-like representations in visual areas of those same events. Using functional magnetic resonance imaging, we find that when participants are asked to predict the likely trajectory of a falling ball, motion-sensitive brain regions are activated. We demonstrate that this activity, which occurs even though no motion is being sensed, resembles activity patterns that arise while participants perceive the ball's motion. This finding thus suggests that mental simulations recreate sensory depictions of how a physical scene is likely to unfold.

Physically Implied Surfaces

In addition to seeing objects that are directly in view, we also represent objects that are merely implied (e.g., by occlusion, motion, and other cues). What can imply the presence of an object? Here, we explored (in three preregistered experiments; N = 360 adults) the role of physical interaction in creating impressions of objects that are not actually present. After seeing an actor collide with an invisible wall or step onto an invisible box, participants gave facilitated responses to actual, visible surfaces that appeared where the implied wall or box had been-a Stroop-like pattern of facilitation and interference that suggested automatic inferences about the relevant implied surfaces. Follow-up experiments ruled out confounding geometric cues and anticipatory responses. We suggest that physical interactions can trigger representations of the participating surfaces such that we automatically infer the presence of objects implied only by their physical consequences.

The Perception of Relations

The world contains not only objects and features (red apples, glass bowls, wooden tables), but also relations holding between them (apples contained in bowls, bowls supported by tables). Representations of these relations are often developmentally precocious and linguistically privileged; but how does the mind extract them in the first place? Although relations themselves cast no light onto our eyes, a growing body of work suggests that even very sophisticated relations display key signatures of automatic visual processing. Across physical, eventive, and social domains, relations such as support, fit, cause, chase, and even socially interact are extracted rapidly, are impossible to ignore, and influence other perceptual processes. Sophisticated and structured relations are not only judged and understood, but also seen - revealing surprisingly rich content in visual perception itself.

The trajectory of counterfactual simulation in development

Young children often struggle to answer the question "what would have happened?" particularly in cases where the adult-like "correct" answer has the same outcome as the event that actually occurred. Previous work has assumed that children fail because they cannot engage in accurate counterfactual simulations. Children have trouble considering what to change and what to keep fixed when comparing counterfactual alternatives to reality. However, most developmental studies on counterfactual reasoning have relied on binary yes/no responses to counterfactual questions about complex narratives and so have only been able to document when these failures occur but not why and how. Here, we investigate counterfactual reasoning in a domain in which specific counterfactual possibilities are very concrete: simple collision interactions. In Experiment 1, we show that 5- to 10-year-old children (recruited from schools and museums in Connecticut) succeed in making predictions but struggle to answer binary counterfactual questions. In Experiment 2, we use a multiple-choice method to allow children to select a specific counterfactual possibility. We find evidence that 4- to 6-year-old children (recruited online from across the United States) do conduct counterfactual simulations, but the counterfactual possibilities younger children consider differ from adult-like reasoning in systematic ways. Experiment 3 provides further evidence that young children engage in simulation rather than using a simpler visual matching strategy. Together, these experiments show that the developmental changes in counterfactual reasoning are not simply a matter of whether children engage in counterfactual simulation but also how they do so. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning

Many animals, and an increasing number of artificial agents, display sophisticated capabilities to perceive and manipulate objects. But human beings remain distinctive in their capacity for flexible, creative tool use-using objects in new ways to act on the world, achieve a goal, or solve a problem. To study this type of general physical problem solving, we introduce the Virtual Tools game. In this game, people solve a large range of challenging physical puzzles in just a handful of attempts. We propose that the flexibility of human physical problem solving rests on an ability to imagine the effects of hypothesized actions, while the efficiency of human search arises from rich action priors which are updated via observations of the world. We instantiate these components in the "sample, simulate, update" (SSUP) model and show that it captures human performance across 30 levels of the Virtual Tools game. More broadly, this model provides a mechanism for explaining how people condense general physical knowledge into actionable, task-specific plans to achieve flexible and efficient physical problem solving.

Broken Physics: A Conjunction-Fallacy Effect in Intuitive Physical Reasoning

One remarkable aspect of human cognition is our ability to reason about physical events. This article provides novel evidence that intuitive physics is subject to a peculiar error, the classic conjunction fallacy, in which people rate the probability of a conjunction of two events as more likely than one constituent (a logical impossibility). Participants viewed videos of physical scenarios and judged the probability that either a single event or a conjunction of two events would occur. In Experiment 1 (n = 60), participants consistently rated conjunction events as more likely than single events for the same scenes. Experiment 2 (n = 180) extended these results to rule out several alternative explanations. Experiment 3 (n = 100) generalized the finding to different scenes. This demonstration of conjunction errors contradicts claims that such errors should not appear in intuitive physics and presents a serious challenge to current theories of mental simulation in physical reasoning.

Phenomenal Causality and Sensory Realism

One of the most important tasks for humans is the attribution of causes and effects in all wakes of life. The first systematical study of visual perception of causality-often referred to as phenomenal causality-was done by Albert Michotte using his now well-known launching events paradigm. Launching events are the seeming collision and seeming transfer of movement between two objects-abstract, featureless stimuli ("objects") in Michotte's original experiments. Here, we study the relation between causal ratings for launching events in Michotte's setting and launching collisions in a photorealistically computer-rendered setting. We presented launching events with differing temporal gaps, the same launching processes with photorealistic billiard balls, as well as photorealistic billiard balls with realistic motion dynamics, that is, an initial rebound of the first ball after collision and a short sliding phase of the second ball due to momentum and friction. We found that providing the normal launching stimulus with realistic visuals led to lower causal ratings, but realistic visuals together with realistic motion dynamics evoked higher ratings. Two-dimensional versus three-dimensional presentation, on the other hand, did not affect phenomenal causality. We discuss our results in terms of intuitive physics as well as cue conflict.

Invariant representations of mass in the human brain

An intuitive understanding of physical objects and events is critical for successfully interacting with the world. Does the brain achieve this understanding by running simulations in a mental physics engine, which represents variables such as force and mass, or by analyzing patterns of motion without encoding underlying physical quantities? To investigate, we scanned participants with fMRI while they viewed videos of objects interacting in scenarios indicating their mass. Decoding analyses in brain regions previously implicated in intuitive physical inference revealed mass representations that generalized across variations in scenario, material, friction, and motion energy. These invariant representations were found during tasks without action planning, and tasks focusing on an orthogonal dimension (object color). Our results support an account of physical reasoning where abstract physical variables serve as inputs to a forward model of dynamics, akin to a physics engine, in parietal and frontal cortex.

Development of basic intuitions about physical support during early childhood: Evidence from a novel eye-tracking paradigm

Using a novel eye-tracking paradigm, we assessed the development of 2- to 6-year-old children's intuitions about the physical support of symmetrical and asymmetrical objects in two experiments (Experiment 1: N = 98; Experiment 2: N = 288). Children were presented with video sequences demonstrating how two identical blocks were lowered onto a platform before being released simultaneously. In the critical test trials, both blocks remained in place although only one of them was sufficiently supported. As expected, children tended to look longer at the block, which should have fallen. Taken together, the results indicate that even 2-year-old children are sensitive to the amount of contact between symmetrical blocks and a supporting platform and even anticipate which block is going to fall. Nonetheless, we found a considerable improvement with age in this respect. Two-year-olds did not consider an object's weight distribution reliably when assessing its stability and even older preschoolers performed much more poorly with asymmetrical than symmetrical blocks. We conclude that intuitions about support are still weak and limited in toddlers and that they improve considerably during early childhood.

Speed Biases With Real-Life Video Clips

We live almost literally immersed in an artificial visual world, especially motion pictures. In this exploratory study, we asked whether the best speed for reproducing a video is its original, shooting speed. By using adjustment and double staircase methods, we examined speed biases in viewing real-life video clips in three experiments, and assessed their robustness by manipulating visual and auditory factors. With the tested stimuli (short clips of human motion, mixed human-physical motion, physical motion and ego-motion), speed underestimation was the rule rather than the exception, although it depended largely on clip content, ranging on average from 2% (ego-motion) to 32% (physical motion). Manipulating display size or adding arbitrary soundtracks did not modify these speed biases. Estimated speed was not correlated with estimated duration of these same video clips. These results indicate that the sense of speed for real-life video clips can be systematically biased, independently of the impression of elapsed time. Measuring subjective visual tempo may integrate traditional methods that assess time perception: speed biases may be exploited to develop a simple, objective test of reality flow, to be used for example in clinical and developmental contexts. From the perspective of video media, measuring speed biases may help to optimize video reproduction speed and validate "natural" video compression techniques based on sub-threshold temporal squeezing.

Eye-Tracking Causality

How do people make causal judgments? What role, if any, does counterfactual simulation play? Counterfactual theories of causal judgments predict that people compare what actually happened with what would have happened if the candidate cause had been absent. Process theories predict that people focus only on what actually happened, to assess the mechanism linking candidate cause and outcome. We tracked participants' eye movements while they judged whether one billiard ball caused another one to go through a gate or prevented it from going through. Both participants' looking patterns and their judgments demonstrated that counterfactual simulation played a critical role. Participants simulated where the target ball would have gone if the candidate cause had been removed from the scene. The more certain participants were that the outcome would have been different, the stronger the causal judgments. These results provide the first direct evidence for spontaneous counterfactual simulation in an important domain of high-level cognition.

Testing Bayesian and heuristic predictions of mass judgments of colliding objects

Mass judgments of colliding objects have been used to explore people's understanding of the physical world because they are ecologically relevant, yet people display biases that are most easily explained by a small set of heuristics. Recent work has challenged the heuristic explanation, by producing the same biases from a model that copes with perceptual uncertainty by using Bayesian inference with a prior based on the correct combination rules from Newtonian mechanics (noisy Newton). Here I test the predictions of the leading heuristic model (Gilden and Proffitt, 1989) against the noisy Newton model using a novel manipulation of the standard mass judgment task: making one of the objects invisible post-collision. The noisy Newton model uses the remaining information to predict above-chance performance, while the leading heuristic model predicts chance performance when one or the other final velocity is occluded. An experiment using two different types of occlusion showed better-than-chance performance and response patterns that followed the predictions of the noisy Newton model. The results demonstrate that people can make sensible physical judgments even when information critical for the judgment is missing, and that a Bayesian model can serve as a guide in these situations. Possible algorithmic-level accounts of this task that more closely correspond to the noisy Newton model are explored.