Mind Games: Game Engines as an Architecture for Intuitive Physics

Dynamic tracking of objects in the macaque dorsomedial frontal cortex

A central tenet of cognitive neuroscience is that humans build an internal model of the external world and use mental simulation of the model to perform physical inferences. Decades of human experiments have shown that behaviors in many physical reasoning tasks are consistent with predictions from the mental simulation theory. However, evidence for the defining feature of mental simulation - that neural population dynamics reflect simulations of physical states in the environment - is limited. We test the mental simulation hypothesis by combining a naturalistic ball-interception task, large-scale electrophysiology in non-human primates, and recurrent neural network modeling. We find that neurons in the monkeys' dorsomedial frontal cortex (DMFC) represent task-relevant information about the ball position in a multiplexed fashion. At a population level, the activity pattern in DMFC comprises a low-dimensional neural embedding that tracks the ball both when it is visible and invisible, serving as a neural substrate for mental simulation. A systematic comparison of different classes of task-optimized RNN models with the DMFC data provides further evidence supporting the mental simulation hypothesis. Our findings provide evidence that neural dynamics in the frontal cortex are consistent with internal simulation of external states in the environment.

Monkeys engage in visual simulation to solve complex problems

Visual simulation-i.e., using internal reconstructions of the world to experience potential future versions of events that are not currently happening-is among the most sophisticated capacities of the human mind. But is this ability in fact uniquely human? To answer this question, we tested monkeys on a series of experiments involving the "Planko" game, which we have previously used to evoke visual simulation in human participants. We found that monkeys were able to successfully play the game using a simulation strategy, predicting the trajectory of a ball through a field of planks while demonstrating a level of accuracy and behavioral signatures comparable with those of humans. Computational analyses further revealed that the monkeys' strategy while playing Planko aligned with a recurrent neural network (RNN) that approached the task using a spontaneously learned simulation strategy. Finally, we carried out awake functional magnetic resonance imaging while monkeys played Planko. We found activity in motion-sensitive regions of the monkey brain during hypothesized simulation periods, even without any perceived visual motion cues. This neural result closely mirrors previous findings from human research, suggesting a shared mechanism of visual simulation across species. Taken together, these findings challenge traditional views of animal cognition, proposing that nonhuman primates possess a complex cognitive landscape, capable of invoking imaginative and predictive mental experiences to solve complex everyday problems.

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

Electrophysiology Reveals That Intuitive Physics Guides Visual Tracking and Working Memory

Starting in early infancy, our perception and predictions are rooted in strong expectations about the behavior of everyday objects. These intuitive physics expectations have been demonstrated in numerous behavioral experiments, showing that even pre-verbal infants are surprised when something impossible happens (e.g., when objects magically appear or disappear). However, it remains unclear whether and how physical expectations shape different aspects of moment-by-moment online visual scene processing, unrelated to explicit physical reasoning. In two EEG experiments, people watched short videos like those used in behavioral studies with adults and infants, and more recently in AI benchmarks. Objects moved on a stage, and were briefly hidden behind an occluder, with the scene either unfolding as expected, or violating object permanence (adding or removing an object). We measured the contralateral delay activity, an electrophysiological marker of online processing, to examine participants' working memory (WM) representations, as well as their ability to continuously track the objects in the scene. We found that both types of object permanence violations disrupted tracking, even though violations involved perceptually non-salient events (magical vanishing) or new objects that weren't previously tracked (magical creation). Physical violations caused WM to reset, i.e., to discard the original scene representation before it could recover and represent the updated number of items. Providing a physical explanation for the violations (a hole behind the occluder) restored object tracking, and we found evidence that WM continued to represent items that disappeared 'down the hole'. Our results show how intuitive physical expectations shape online representations, and form the basis of dynamic object tracking.

Synthesizing the temporal self: robotic models of episodic and autobiographical memory

Episodic memories are experienced as belonging to a self that persists in time. We review evidence concerning the nature of human episodic memory and of the sense of self and how these emerge during development, proposing that the younger child experiences a persistent self that supports a subjective experience of remembering. We then explore recent research in cognitive architectures for robotics that has investigated the possibility of forms of synthetic episodic and autobiographical memory. We show that recent advances in generative modeling can support an understanding of the emergence of self and of episodic memory, and that cognitive architectures which include a language capacity are showing progress towards the construction of a narrative self with autobiographical memory capabilities for robots. We conclude by considering the prospects for a more complete model of mental time travel in robotics and the implications of this modeling work for understanding human episodic memory and the self in time. This article is part of the theme issue 'Elements of episodic memory: lessons from 40 years of research'.

The role of action concepts in physical reasoning: insights from late childhood

A fundamental component of human cognition is the ability to intuitively reason about behaviours of objects and systems in the physical world without resorting to explicit scientific knowledge. This skill was traditionally considered a symbolic process. However, in the last decades, there has been a shift towards ideas of embodiment, suggesting that accessing physical knowledge and predicting physical outcomes is grounded in bodily interactions with the environment. Infants and children, who learn mainly through their embodied experiences, serve as a model to probe the link between reasoning and physical concepts. Here, we tested school-aged children (5- to 15-year-olds) in online reasoning games that involve different physical action concepts such as supporting, launching and clearing. We assessed changes in children's performance and strategies over development and their relationships with the different action concepts. Children reasoned more accurately in problems that involved supporting actions compared to launching or clearing actions. Moreover, when children failed, they were more strategic in subsequent attempts when problems involved support rather than launching or clearing. Children improved with age, but improvements differed across action concepts. Our findings suggest that accessing physical knowledge and predicting physical events are affected by action concepts, and those effects change over development. This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Counterfactual simulation in causal cognition

How do people make causal judgments and assign responsibility? In this review article, I argue that counterfactual simulations are key. To simulate counterfactuals, we need three ingredients: a generative mental model of the world, the ability to perform interventions on that model, and the capacity to simulate the consequences of these interventions. The counterfactual simulation model (CSM) uses these ingredients to capture people's intuitive understanding of the physical and social world. In the physical domain, the CSM predicts people's causal judgments about dynamic collision events, complex situations that involve multiple causes, omissions as causes, and causes that sustain physical stability. In the social domain, the CSM predicts responsibility judgments in helping and hindering scenarios.

Causal relational problem solving in toddlers

We investigate young children's capacity for "causal relational reasoning": the ability to use relational reasoning to design novel interventions and bring about novel outcomes. In two experiments, we show that 24-30-month-old toddlers and three-year-old preschoolers use relational reasoning in a causal problem-solving task. Even toddlers rapidly inferred relational causal rules and applied this knowledge to solve novel problems--thus demonstrating both surprisingly early competence in relational reasoning and sophisticated causal inference. In both experiments, children observed a handful of trials in which a mechanistically opaque machine made objects larger or smaller. When prompted to solve a new problem, they used the machine to change the relative size of a novel object - even though its appearance and absolute size differed from previous observations, and even though subjects had never seen the machine generate objects of the required size before. This suggests that children quickly inferred abstract causal relations and then generalized these relations to determine which intervention would bring about the novel outcome required to solve the problem. These findings suggest a close link between early relational reasoning and active causal learning and inference.

Tangled Physics: Knots Strain Intuitive Physical Reasoning

Whereas decades of research have cataloged striking errors in physical reasoning, a resurgence of interest in intuitive physics has revealed humans' remarkable ability to successfully predict the unfolding of physical scenes. A leading interpretation intended to resolve these opposing results is that physical reasoning recruits a general-purpose mechanism that reliably models physical scenarios (explaining recent successes), but overly contrived tasks or impoverished and ecologically invalid stimuli can produce poor performance (accounting for earlier failures). But might there be tasks that persistently strain physical understanding, even in naturalistic contexts? Here, we explore this question by introducing a new intuitive physics task: evaluating the strength of knots and tangles. Knots are ubiquitous across cultures and time-periods, and evaluating them correctly often spells the difference between safety and peril. Despite this, 5 experiments show that observers fail to discern even very large differences in strength between knots. In a series of two-alternative forced-choice tasks, observers viewed a variety of simple "bends" (knots joining two pieces of thread) and decided which would require more force to undo. Though the strength of these knots is well-documented, observers' judgments completely failed to reflect these distinctions, across naturalistic photographs (E1), idealized renderings (E2), dynamic videos (E3), and even when accompanied by schematic diagrams of the knots' structures (E4). Moreover, these failures persisted despite accurate identification of the topological differences between the knots (E5); in other words, even when observers correctly perceived the underlying structure of the knot, they failed to correctly judge its strength. These results expose a blindspot in physical reasoning, placing new constraints on general-purpose theories of scene understanding.

Infants' representations of michottean triggering events

The classic Michottean 'launching' event is consistent with a real-world Newtonian elastic collision. Previous research has shown that adult humans distinguish launching events that obey some of the physical constraints on Newtonian elastic collisions from events that do not do so early in visual processing, and that infants do so early in development (< 9 months of age). These include that in a launching event, the speed of the agent can be 3 times faster (or more) than that of the patient but the speed of the patient cannot be detectably greater than the speed of the agent. Experiment 1 shows that 7-8-month-old infants also distinguish canonical launching events from events in which the motion of the patient is rotated 90° from the trajectory of the motion of the agent (another outcome ruled out by the physics of elastic collisions). Violations of both the relative speed and the angle constraints create Michottean 'triggering' events, in which adults describe the motion of the patient as autonomous but not spontaneous, i.e., still initiated by contact with the causal agent. Experiments 2 and 3 begin to explore whether infants of this age construe Michottean triggering events as causal. We find that infants of this age are not sensitive to a reversal of the agent and patient in triggering events, thus failing to exhibit one of the signatures of representing an event as causal. We argue that there are likely several independent events schemas with causal content represented by young infants, and the literature on the origins of causal cognition in infancy would benefit from systematic investigations of event schemas other than launching events.

The Cognitive Architecture of Infant Attachment

Meta-analytic evidence indicates that the quality of the attachment relationship that infants establish with their primary caregiver has enduring significance for socioemotional and cognitive outcomes. However, the mechanisms by which early attachment experiences contribute to subsequent development remain underspecified. According to attachment theory, early attachment experiences become embodied in the form of cognitive-affective representations, referred to as internal working models (IWMs), that guide future behavior. Little is known, however, about the cognitive architecture of IWMs in infancy. In this article, we discuss significant advances made in the field of infant cognitive development and propose that leveraging insights from this research has the potential to fundamentally shape our understanding of the cognitive architecture of attachment representations in infancy. We also propose that the integration of attachment research into cognitive research can shed light on the role of early experiences, individual differences, and stability and change in infant cognition, as well as open new routes of investigation in cognitive studies, which will further our understanding of human knowledge. We provide recommendations for future research throughout the article and conclude by using our collaborative research as an example.

Response to commentaries on What Babies Know

Twenty-five commentaries raise questions concerning the origins of knowledge, the interplay of iconic and propositional representations in mental life, the architecture of numerical and social cognition, the sources of uniquely human cognitive capacities, and the borders among core knowledge, perception, and thought. They also propose new methods, drawn from the vibrant, interdisciplinary cognitive sciences, for addressing these questions and deepening understanding of infant minds.

Perceptual (roots of) core knowledge

Some core knowledge may be rooted in - or even identical to - well-characterized mechanisms of mid-level visual perception and attention. In the decades since it was first proposed, this possibility has inspired (and has been supported by) several discoveries in both infant cognition and adult perception, but it also faces several challenges. To what degree does What Babies Know reflect how babies see and attend?

A Phone in a Basket Looks Like a Knife in a Cup: Role-Filler Independence in Visual Processing

When a piece of fruit is in a bowl, and the bowl is on a table, we appreciate not only the individual objects and their features, but also the relations containment and support, which abstract away from the particular objects involved. Independent representation of roles (e.g., containers vs. supporters) and "fillers" of those roles (e.g., bowls vs. cups, tables vs. chairs) is a core principle of language and higher-level reasoning. But does such role-filler independence also arise in automatic visual processing? Here, we show that it does, by exploring a surprising error that such independence can produce. In four experiments, participants saw a stream of images containing different objects arranged in force-dynamic relations-e.g., a phone contained in a basket, a marker resting on a garbage can, or a knife sitting in a cup. Participants had to respond to a single target image (e.g., a phone in a basket) within a stream of distractors presented under time constraints. Surprisingly, even though participants completed this task quickly and accurately, they false-alarmed more often to images matching the target's relational category than to those that did not-even when those images involved completely different objects. In other words, participants searching for a phone in a basket were more likely to mistakenly respond to a knife in a cup than to a marker on a garbage can. Follow-up experiments ruled out strategic responses and also controlled for various confounding image features. We suggest that visual processing represents relations abstractly, in ways that separate roles from fillers.

Realizing the gravity of the simulation: adaptation to simulated hypogravity leads to altered predictive control

Accurate predictive abilities are important for a wide variety of animal behaviors. Inherent to many of these predictions is an understanding of the physics that underlie the behavior. Humans are specifically attuned to the physics on Earth but can learn to move in other environments (e.g., the surface of the Moon). However, the adjustments made to their physics-based predictions in the face of altered gravity are not fully understood. The current study aimed to characterize the locomotor adaptation to a novel paradigm for simulated reduced gravity. We hypothesized that exposure to simulated hypogravity would result in updated predictions of gravity-based movement. Twenty participants took part in a protocol that had them perform vertically targeted countermovement jumps before (PRE), during, and after (POST) a physical simulation of hypogravity. Jumping in simulated hypogravity had different neuromechanics from the PRE condition, with reduced ground impulses (p ≤ .009) and muscle activity prior to the time of landing (i.e., preactivation; p ≤ .016). In the 1 g POST condition, muscle preactivation remained reduced (p ≤ .033) and was delayed (p ≤ .008) by up to 33% for most muscles of the triceps surae, reflecting an expectation of hypogravity. The aftereffects in muscle preactivation, along with little-to-no change in muscle dynamics during ground contact, point to a neuromechanical adaptation that affects predictive, feed-forward systems over feedback systems. As such, we conclude that the neural representation, or internal model, of gravity is updated after exposure to simulated hypogravity.

A stochastic world model on gravity for stability inference

The fact that objects without proper support will fall to the ground is not only a natural phenomenon, but also common sense in mind. Previous studies suggest that humans may infer objects' stability through a world model that performs mental simulations with a priori knowledge of gravity acting upon the objects. Here we measured participants' sensitivity to gravity to investigate how the world model works. We found that the world model on gravity was not a faithful replica of the physical laws, but instead encoded gravity's vertical direction as a Gaussian distribution. The world model with this stochastic feature fit nicely with participants' subjective sense of objects' stability and explained the illusion that taller objects are perceived as more likely to fall. Furthermore, a computational model with reinforcement learning revealed that the stochastic characteristic likely originated from experience-dependent comparisons between predictions formed by internal simulations and the realities observed in the external world, which illustrated the ecological advantage of stochastic representation in balancing accuracy and speed for efficient stability inference. The stochastic world model on gravity provides an example of how a priori knowledge of the physical world is implemented in mind that helps humans operate flexibly in open-ended environments.

Infant neuroscience: how to measure brain activity in the youngest minds

The functional properties of the infant brain are poorly understood. Recent advances in cognitive neuroscience are opening new avenues for measuring brain activity in human infants. These include novel uses of existing technologies such as electroencephalography (EEG) and magnetoencephalography (MEG), the availability of newer technologies including functional near-infrared spectroscopy (fNIRS) and optically pumped magnetometry (OPM), and innovative applications of functional magnetic resonance imaging (fMRI) in awake infants during cognitive tasks. In this review article we catalog these available non-invasive methods, discuss the challenges and opportunities encountered when applying them to human infants, and highlight the potential they may ultimately hold for advancing our understanding of the youngest minds.

Enhancing Robotic Perception through Synchronized Simulation and Physical Common-Sense Reasoning

We introduce both conceptual and empirical findings arising from the amalgamation of a robotics cognitive architecture with an embedded physics simulator, aligning with the principles outlined in the intuitive physics literature. The employed robotic cognitive architecture, named CORTEX, leverages a highly efficient distributed working memory known as deep state representation. This working memory inherently encompasses a fundamental ontology, state persistency, geometric and logical relationships among elements, and tools for reading, updating, and reasoning about its contents. Our primary objective is to investigate the hypothesis that the integration of a physics simulator into the architecture streamlines the implementation of various functionalities that would otherwise necessitate extensive coding and debugging efforts. Furthermore, we categorize these enhanced functionalities into broad types based on the nature of the problems they address. These include addressing challenges related to occlusion, model-based perception, self-calibration, scene structural stability, and human activity interpretation. To demonstrate the outcomes of our experiments, we employ CoppeliaSim as the embedded simulator and both a Kinova Gen3 robotic arm and the Open-Manipulator-P as the real-world scenarios. Synchronization is maintained between the simulator and the stream of real events. Depending on the ongoing task, numerous queries are computed, and the results are projected into the working memory. Participating agents can then leverage this information to enhance overall performance.

Monkeys engage in visual simulation to solve complex problems

Visual simulation - i.e., using internal reconstructions of the world to experience potential future versions of events that are not currently happening - is among the most sophisticated capacities of the human mind. But is this ability in fact uniquely human? To answer this question, we tested monkeys on a series of experiments involving the 'Planko' game, which we have previously used to evoke visual simulation in human participants. We found that monkeys were able to successfully play the game using a simulation strategy, predicting the trajectory of a ball through a field of planks while demonstrating a level of accuracy and behavioral signatures comparable to humans. Computational analyses further revealed that the monkeys' strategy while playing Planko aligned with a recurrent neural network (RNN) that approached the task using a spontaneously learned simulation strategy. Finally, we carried out awake functional magnetic resonance imaging while monkeys played Planko. We found activity in motion-sensitive regions of the monkey brain during hypothesized simulation periods, even without any perceived visual motion cues. This neural result closely mirrors previous findings from human research, suggesting a shared mechanism of visual simulation across species. In all, these findings challenge traditional views of animal cognition, proposing that nonhuman primates possess a complex cognitive landscape, capable of invoking imaginative and predictive mental experiences to solve complex everyday problems.

The (Un)ideal Physicist: How Humans Rely on Object Interaction for Friction Estimates

To estimate object properties such as mass or friction, our brain relies on visual information to efficiently compute approximations. The role of sensorimotor feedback, however, is not well understood. Here we tested healthy adults (N = 79) in an inclined-plane problem, that is, how much a plane can be tilted before an object starts to slide, and contrasted the interaction group with observation groups who accessed involved forces by watching objects being manipulated. We created objects of different masses and levels of friction and asked participants to estimate the critical tilt angle after pushing an object, lifting it, or both. Estimates correlated with applied forces and were biased toward object mass, with higher estimates for heavier objects. Our findings highlight that inferences about physical object properties are tightly linked to the human sensorimotor system and that humans integrate sensorimotor information even at the risk of nonveridical perceptual estimates.

Better Together: 14-Month-Old Infants Expect Agents to Cooperate

Humans engage in cooperative activities from early on and the breadth of human cooperation is unparalleled. Human preference for cooperation might reflect cognitive and motivational mechanisms that drive engagement in cooperative activities. Here we investigate early indices of humans' cooperative abilities and test whether 14-month-old infants expect agents to prefer cooperative over individual goal achievement. Three groups of infants saw videos of agents facing a choice between two actions that led to identical rewards but differed in the individual costs. Our results show that, in line with prior research, infants expect agents to make instrumentally rational choices and prefer the less costly of two individual action alternatives. In contrast, when one of the action alternatives is cooperative, infants expect agents to choose cooperation over individual action, even though the cooperative action demands more effort from each agent to achieve the same outcome. Finally, we do not find evidence that infants expect agents to choose the less costly alternative when both options entail cooperative action. Combined, these results indicate an ontogenetically early expectation of cooperation, and raise interesting implications and questions regarding the nature of infants' representations of cooperative actions and their utility.

Object geometry serves humans' intuitive physics of stability

How do humans judge physical stability? A prevalent account emphasizes the mental simulation of physical events implemented by an intuitive physics engine in the mind. Here we test the extent to which the perceptual features of object geometry are sufficient for supporting judgments of falling direction. In all experiments, adults and children judged the falling direction of a tilted object and, across experiments, objects differed in the geometric features (i.e., geometric centroid, object height, base size and/or aspect ratio) relevant to the judgment. Participants' performance was compared to computational models trained on geometric features, as well as a deep convolutional neural network (ResNet-50), none of which incorporated mental simulation. Adult and child participants' performance was well fit by models of object geometry, particularly the geometric centroid. ResNet-50 also provided a good account of human performance. Altogether, our findings suggest that object geometry may be sufficient for judging the falling direction of tilted objects, independent of mental simulation.

A dedicated mental resource for intuitive physics

Countless decisions and actions in daily life draw on a mental model of the physical structure and dynamics of the world - from stepping carefully around a patch of slippery pavement to stacking delicate produce in a shopping basket. People can make fast and accurate inferences about how physical interactions will unfold, but it remains unclear whether we do so by applying a general set of physical principles across scenarios, or instead by reasoning about the physics of individual scenarios in an ad-hoc fashion. Here, we hypothesized that humans possess a dedicated and flexible mental resource for physical inference, and we tested for such a resource using a battery of fine-tuned tasks to capture individual differences in performance. Despite varying scene contents across tasks, we found that performance was highly correlated among them and well-explained by a unitary intuitive physics resource, distinct from other facets of cognition such as spatial reasoning and working memory.

Rotating objects cue spatial attention via the perception of frictive surface contact

We report a new attentional cueing effect, which shows how attention models the physical force of friction. Most objects we see are in frictive contact with a 'floor', such that clockwise rotation causes rightward movement and counterclockwise rotation leftward movement. Is this regularity encoded in spatial orienting responses? In Experiment 1, seeing a clockwise-rotating 'wheel' produced faster responses to subsequent targets appearing on the right vs. left (and vice versa for counterclockwise rotation). Thus, when viewing a lone rotating wheel, we orient attention toward where we predict it will move next, assuming frictive floor contact. But what happens if the rotating wheel is seen touching another visible surface? In Experiment 2, rotational cueing was stronger for wheels touching a visible floor, was abolished for wheels near but not touching another surface, and reversed for wheels touching a ceiling. We conclude that the visual system makes an assumption of frictive floor contact, and rapidly analyzes visual cues to frictive contact with other surfaces, in order to orient attention toward where objects are likely to move next.

Thought Experiments as an Error Detection and Correction Tool

The ability to recognize and correct errors in one's explanatory understanding is critically important for learning. However, little is known about the mechanisms that determine when and under what circumstances errors are detected and how they are corrected. The present study investigated thought experiments as a potential tool that can reveal errors and trigger belief revision in the service of error correction. Across two experiments, 1149 participants engaged in reasoning about force and motion (a domain with well-documented misconceptions) in a pre-training-training-post-training design. The two experiments manipulated the type of mental model manipulated in the thought experiments (i.e., whether participants reasoned about forces acting on their own bodies vs. on external objects), as well as the level of relational and argumentative reasoning about the outcomes of the thought experiments. The results showed that: (i) thought experiments can serve as a tool to elicit inconsistencies in one's representations; (ii) the level of relational and argumentative reasoning determines the level of belief revision in the service of error correction; and (iii) the type of mental model manipulated in a thought experiment determines its outcome and its potential to initiate belief revision. Thought experiments can serve as a valuable teaching and learning tool, and they can help us better understand the nature of error detection and correction systems.

Generative replay underlies compositional inference in the hippocampal-prefrontal circuit

Human reasoning depends on reusing pieces of information by putting them together in new ways. However, very little is known about how compositional computation is implemented in the brain. Here, we ask participants to solve a series of problems that each require constructing a whole from a set of elements. With fMRI, we find that representations of novel constructed objects in the frontal cortex and hippocampus are relational and compositional. With MEG, we find that replay assembles elements into compounds, with each replay sequence constituting a hypothesis about a possible configuration of elements. The content of sequences evolves as participants solve each puzzle, progressing from predictable to uncertain elements and gradually converging on the correct configuration. Together, these results suggest a computational bridge between apparently distinct functions of hippocampal-prefrontal circuitry and a role for generative replay in compositional inference and hypothesis testing.

The Pulfrich solidity illusion: a surprising demonstration of the visual system's tolerance of solidity violations

Physical objects behave following the principle of solidity: One solid object cannot pass through another. To what extent does the visual system integrate this physical regularity as a prior constraint? A new variant of the Pulfrich effect demonstrates a surprising degree of tolerance for violations of solidity when pitted against motion and depth cues. When adult participants view a pendulum swinging in the fronto-parallel plane with both eyes (one of which was covered by a light-attenuating filter), they falsely perceive the pendulum as swinging in an elliptical path (known as the "Pulfrich effect"). Here, we show that even when the pendulum's motion takes place entirely behind a solid horizontal bar, observers nevertheless see the pendulum pass through the bar while moving in an ellipse. This illusion suggests that the Pulfrich effect and the underlying stereoscopic depth cues can be robust to object solidity.

Combining forces for causal reasoning: Children's predictions about physical interactions

Reasoning about causal relations is essential for children's early cognitive development. The current study investigated 4-year-olds' (N = 58) reasoning about complex causal physical interactions in terms of predicting the endpoint of motion. In an online task, children were presented with four configurations that involved different interactions of forces and consequently different patterns of motion. These were Cause (one force moving an object), Enable (a secondary force promoting the motion), Prevent-180° (an opposing force hindering the motion), and Prevent-90° (two-dimensional; a perpendicular force altering the motion). Each prediction was made in terms of either the Distance or Direction of the motion, which was novel in this task compared with previous assessments. Results revealed differences between the configurations, with Cause being the easiest and Prevent-90° being the most difficult to predict. Furthermore, predictions were more accurate when options were about the motion's Direction, whereas Distance options may have aggravated reasoning. The current study extends previous findings on children's intuitive physics and causal cognition by showing that accuracy in reasoning not only is dependent on the number of forces and dimensions at work but also interacts with estimating the motion's Distance and Direction.

Neither neural networks nor the language-of-thought alone make a complete game

Cognitive science has evolved since early disputes between radical empiricism and radical nativism. The authors are reacting to the revival of radical empiricism spurred by recent successes in deep neural network (NN) models. We agree that language-like mental representations (language-of-thoughts [LoTs]) are part of the best game in town, but they cannot be understood independent of the other players.

Grounding Intuitive Physics in Perceptual Experience

This review article explores the foundation of laypeople's understanding of the physical world rooted in perceptual experience. Beginning with a concise historical overview of the study of intuitive physics, the article presents the hypothesis that laypeople possess accurate internalized representations of physical laws. A key aspect of this hypothesis is the contention that correct representations of physical laws emerge in ecological experimental conditions, where the scenario being examined resembles everyday life experiences. The article critically examines empirical evidence both supporting and challenging this claim, revealing that despite everyday-life-like conditions, fundamental misconceptions often persist. Many of these misconceptions can be attributed to a domain-general heuristic that arises from the overgeneralization of perceptual-motor experiences with physical objects. To conclude, the article delves into ongoing controversies and highlights promising future avenues in the field of intuitive physics, including action-judgment dissociations, insights from developmental psychology, and computational models integrating artificial intelligence.

Non-commitment in mental imagery

We examine non-commitment in the imagination. Across 5 studies (N > 1, 800), we find that most people are non-committal about basic aspects of their mental images, including features that would be readily apparent in real images. While previous work on the imagination has discussed the possibility of non-commitment, this paper is the first, to our knowledge, to examine this systematically and empirically. We find that people do not commit to basic properties of specified mental scenes (Studies 1 and 2), and that people report non-commitment rather than uncertainty or forgetfulness (Study 3). Such non-commitment is present even for people with generally vivid imaginations, and those who report imagining the specified scene very vividly (Studies 4a, 4b). People readily confabulate properties of their mental images when non-commitment is not offered as an explicit option (Study 5). Taken together, these results establish non-commitment as a pervasive component of mental imagery.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

The psychophysics of bouncing: Perceptual constraints, physical constraints, animacy, and phenomenal causality

In the present study we broadly explored the perception of physical and animated motion in bouncing-like scenarios through four experiments. In the first experiment, participants were asked to categorize bouncing-like displays as physical bounce, animated motion, or other. Several parameters of the animations were manipulated, that is, the simulated coefficient of restitution, the value of simulated gravitational acceleration, the motion pattern (uniform acceleration/deceleration or constant speed) and the number of bouncing cycles. In the second experiment, a variable delay at the moment of the collision between the bouncing object and the bouncing surface was introduced. Main results show that, although observers appear to have realistic representations of physical constraints like energy conservation and gravitational acceleration/deceleration, the amount of visual information available in the scene has a strong modulation effect on the extent to which they rely on these representations. A coefficient of restitution >1 was a crucial cue to animacy in displays showing three bouncing cycles, but not in displays showing one bouncing cycle. Additionally, bouncing impressions appear to be driven by perceptual constraints that are unrelated to the physical realism of the scene, like preference for simulated gravitational attraction smaller than g and perceived temporal contiguity between the different phases of bouncing. In the third experiment, the visible opaque bouncing surface was removed from the scene, and the results showed that this did not have any substantial effect on the resulting impressions of physical bounce or animated motion, suggesting that the visual system can fill-in the scene with the missing element. The fourth experiment explored visual impressions of causality in bouncing scenarios. At odds with claims of current causal perception theories, results indicate that a passive object can be perceived as the direct cause of the motion behavior of an active object.

Précis of What Babies Know

Where does human knowledge begin? Research on human infants, children, adults, and nonhuman animals, using diverse methods from the cognitive, brain, and computational sciences, provides evidence for six early emerging, domain-specific systems of core knowledge. These automatic, unconscious systems are situated between perceptual systems and systems of explicit concepts and beliefs. They emerge early in infancy, guide children's learning, and function throughout life.

F = ma. Is the macaque brain Newtonian?

Intuitive Physics, the ability to anticipate how the physical events involving mass objects unfold in time and space, is a central component of intelligent systems. Intuitive physics is a promising tool for gaining insight into mechanisms that generalize across species because both humans and non-human primates are subject to the same physical constraints when engaging with the environment. Physical reasoning abilities are widely present within the animal kingdom, but monkeys, with acute 3D vision and a high level of dexterity, appreciate and manipulate the physical world in much the same way humans do.

Large Language Models and the Reverse Turing Test

Large language models (LLMs) have been transformative. They are pretrained foundational models that are self-supervised and can be adapted with fine-tuning to a wide range of natural language tasks, each of which previously would have required a separate network model. This is one step closer to the extraordinary versatility of human language. GPT-3 and, more recently, LaMDA, both of them LLMs, can carry on dialogs with humans on many topics after minimal priming with a few examples. However, there has been a wide range of reactions and debate on whether these LLMs understand what they are saying or exhibit signs of intelligence. This high variance is exhibited in three interviews with LLMs reaching wildly different conclusions. A new possibility was uncovered that could explain this divergence. What appears to be intelligence in LLMs may in fact be a mirror that reflects the intelligence of the interviewer, a remarkable twist that could be considered a reverse Turing test. If so, then by studying interviews, we may be learning more about the intelligence and beliefs of the interviewer than the intelligence of the LLMs. As LLMs become more capable, they may transform the way we interact with machines and how they interact with each other. Increasingly, LLMs are being coupled with sensorimotor devices. LLMs can talk the talk, but can they walk the walk? A road map for achieving artificial general autonomy is outlined with seven major improvements inspired by brain systems and how LLMs could in turn be used to uncover new insights into brain function.

New Approaches to 3D Vision

New approaches to 3D vision are enabling new advances in artificial intelligence and autonomous vehicles, a better understanding of how animals navigate the 3D world, and new insights into human perception in virtual and augmented reality. Whilst traditional approaches to 3D vision in computer vision (SLAM: simultaneous localization and mapping), animal navigation (cognitive maps), and human vision (optimal cue integration) start from the assumption that the aim of 3D vision is to provide an accurate 3D model of the world, the new approaches to 3D vision explored in this issue challenge this assumption. Instead, they investigate the possibility that computer vision, animal navigation, and human vision can rely on partial or distorted models or no model at all. This issue also highlights the implications for artificial intelligence, autonomous vehicles, human perception in virtual and augmented reality, and the treatment of visual disorders, all of which are explored by individual articles. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Seeing Soft Materials Draped Over Objects: A Case Study of Intuitive Physics in Perception, Attention, and Memory

We typically think of intuitive physics in terms of high-level cognition, but might aspects of physics also be extracted during lower-level visual processing? Might we not only think about physics, but also see it? We explored this using multiple tasks in online adult samples with objects covered by soft materials-as when you see a chair with a blanket draped over it-where you must account for the physical interactions between cloth, gravity, and object. In multiple change-detection experiments (n = 200), observers from an online testing marketplace were better at detecting image changes involving underlying object structure versus those involving only the superficial folds of cloths-even when the latter were more extreme along several dimensions. And in probe-comparison experiments (n = 100), performance was worse when both probes (vs. only one) appeared on image regions reflective of underlying object structure (equating visual properties). This work collectively shows how vision uses intuitive physics to recover the deeper underlying structure of scenes.

What would have happened? Counterfactuals, hypotheticals and causal judgements

How do people make causal judgements? In this paper, I show that counterfactual simulations are necessary for explaining causal judgements about events, and that hypotheticals do not suffice. In two experiments, participants viewed video clips of dynamic interactions between billiard balls. In Experiment 1, participants either made hypothetical judgements about whether ball B would go through the gate if ball A were not present in the scene, or counterfactual judgements about whether ball B would have gone through the gate if ball A had not been present. Because the clips featured a block in front of the gate that sometimes moved and sometimes stayed put, hypothetical and counterfactual judgements came apart. A computational model that evaluates hypotheticals and counterfactuals by running noisy physical simulations accurately captured participants' judgements. In Experiment 2, participants judged whether ball A caused ball B to go through the gate. The results showed a tight fit between counterfactual and causal judgements, whereas hypotheticals did not predict causal judgements. I discuss the implications of this work for theories of causality, and for studying the development of counterfactual thinking in children. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Visual object detection biases escape trajectories following acoustic startle in larval zebrafish

In this study, we investigated whether the larval zebrafish is sensitive to the presence of obstacles in its environment. Zebrafish execute fast escape swims when in danger of predation. We posited that collisions with solid objects during escape would be maladaptive to the fish, and therefore, the direction of escape swims should be informed by the locations of barriers. To test this idea, we developed a closed-loop imaging rig outfitted with barriers of various qualities. We show that when larval zebrafish escape in response to a non-directional vibrational stimulus, they use visual scene information to avoid collisions with obstacles. Our study demonstrates that barrier avoidance rate corresponds to the absolute distance of obstacles, as distant barriers outside of collision range elicit less bias than nearby collidable barriers that occupy the same amount of visual field. The computation of barrier avoidance is covert: the fact that fish will avoid barriers during escape cannot be predicted by its routine swimming behavior in the barrier arena. Finally, two-photon laser ablation experiments suggest that excitatory bias is provided to the Mauthner cell ipsilateral to approached barriers, either via direct excitation or a multi-step modulation process. We ultimately propose that zebrafish detect collidable objects via an integrative visual computation that is more complex than retinal occupancy alone, laying a groundwork for understanding how cognitive physical models observed in humans are implemented in an archetypal vertebrate brain. VIDEO ABSTRACT.

Subjective selection and the evolution of complex culture

Why is culture the way it is? Here I argue that a major force shaping culture is subjective (cultural) selection, or the selective retention of cultural variants that people subjectively perceive as satisfying their goals. I show that people evaluate behaviors and beliefs according to how useful they are, especially for achieving goals. As they adopt and pass on those variants that seem best, they iteratively craft culture into increasingly effective-seeming forms. I argue that this process drives the development of many cumulatively complex cultural products, including effective technology, magic and ritual, aesthetic traditions, and institutions. I show that it can explain cultural dependencies, such as how certain beliefs create corresponding new practices, and I outline how it interacts with other cultural evolutionary processes. Cultural practices everywhere, from spears to shamanism, develop because people subjectively evaluate them to be effective means of satisfying regular goals.

Recurrent neural networks with explicit representation of dynamic latent variables can mimic behavioral patterns in a physical inference task

Primates can richly parse sensory inputs to infer latent information. This ability is hypothesized to rely on establishing mental models of the external world and running mental simulations of those models. However, evidence supporting this hypothesis is limited to behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of primates (humans and monkeys) in a ball interception task to that of a large set of recurrent neural network (RNN) models with or without the capacity to dynamically track the underlying latent variables. Humans and monkeys exhibit similar behavioral patterns. This primate behavioral pattern is best captured by RNNs endowed with dynamic inference, consistent with the hypothesis that the primate brain uses dynamic inferences to support flexible physical predictions. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.

The building blocks of intuitive physics in the mind and brain

There has been a recent wave of interest in understanding the mental processes underlying intuitive physics - our ability to apprehend the physical structure of the world and anticipate how objects will behave as a scene's dynamics unfold. While work to uncover the neural mechanisms of intuitive physics is just in its beginnings, vibrant lines of neuropsychological research are investigating the many facets of cognition intimately linked with the 'physics engine in the mind'. This special issue brings together a collection of papers that delve into the interactions between intuitive physics and related domains such as audiovisual scene analysis, action planning, and decision making, providing a view of the larger landscape of mental processes that allow us to predict how physical events will unfold in the next moments and plan our behaviors accordingly.

Do capuchin monkeys (Sapajus apella) use exploration to form intuitions about physical properties?

Humans' flexible innovation relies on our capacity to accurately predict objects' behaviour. These predictions may originate from a "physics-engine" in the brain which simulates our environment. To explore the evolutionary origins of intuitive physics, we investigate whether capuchin monkeys' object exploration supports learning. Two capuchin groups experienced exploration sessions involving multiple copies of two objects, one object was easily opened (functional), the other was not (non-functional). We used two within-subject conditions (enrichment-then-test, and test-only) with two object sets per group. Monkeys then underwent individual test sessions where the objects contained rewards, and they choose one to attempt to open. The monkeys spontaneously explored, performing actions which yielded functional information. At test, both groups chose functional objects above chance. While high performance of the test-only group precluded us from establishing learning during exploration, this study reveals the promise of harnessing primates' natural exploratory tendencies to understand how they see the world.

Invariant representation of physical stability in the human brain

Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.

I overthink-Therefore I am not: An active inference account of altered sense of self and agency in depersonalisation disorder

This paper considers the phenomenology of depersonalisation disorder, in relation to predictive processing and its associated pathophysiology. To do this, we first establish a few mechanistic tenets of predictive processing that are necessary to talk about phenomenal transparency, mental action, and self as subject. We briefly review the important role of 'predicting precision' and how this affords mental action and the loss of phenomenal transparency. We then turn to sensory attenuation and the phenomenal consequences of (pathophysiological) failures to attenuate or modulate sensory precision. We then consider this failure in the context of depersonalisation disorder. The key idea here is that depersonalisation disorder reflects the remarkable capacity to explain perceptual engagement with the world via the hypothesis that "I am an embodied perceiver, but I am not in control of my perception". We suggest that individuals with depersonalisation may believe that 'another agent' is controlling their thoughts, perceptions or actions, while maintaining full insight that the 'other agent' is 'me' (the self). Finally, we rehearse the predictions of this formal analysis, with a special focus on the psychophysical and physiological abnormalities that may underwrite the phenomenology of depersonalisation.

Melting Ice With Your Mind: Representational Momentum for Physical States

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Similarities in Procedures Used to Solve Mathematical Problems and Video Games

Video game use is widespread among all age groups, from young children to older adults. The wide variety of video game genres, which are adapted to all tastes and needs, is one of the factors that makes them so attractive. In many cases, video games function as an outlet for stress associated with everyday life by providing an escape from reality. We took advantage of this recreational aspect of video games when investigating whether there are similarities between the procedures used to pass a video game level and those used to solve a mathematical problem. Moreover, we also questioned whether the use of video games can reduce the negative emotions generated by mathematical problems and logical–mathematical knowledge in general. To verify this, we used the Portal 2 video game as a research method or tool. This video game features concepts from the spatial–geometric field that the students must identify and relate in order to carry out the procedures required to solve challenges in each level. The procedures were recorded in a questionnaire that was separated into two blocks of content in order to compare them with the procedures used to solve mathematical problems. The first block pertains to the procedures employed and the second block to the emotions that the students experienced when playing the video game and when solving a mathematical problem. The results reveal that the recreational aspect of video games is more important than the educational aspect. However, the students were not aware of using the problem-solving procedures they learned at school to solve different challenges in the video games. Furthermore, overcoming video game challenges stimulates positive emotions as opposed to the negative emotions generated when solving mathematical problems.

Causal Judgment in the Wild: Evidence from the 2020 U.S. Presidential Election

When explaining why an event occurred, people intuitively highlight some causes while ignoring others. How do people decide which causes to select? Models of causal judgment have been evaluated in simple and controlled laboratory experiments, but they have yet to be tested in a complex real-world setting. Here, we provide such a test, in the context of the 2020 U.S. presidential election. Across tens of thousands of simulations of possible election outcomes, we computed, for each state, an adjusted measure of the correlation between a Biden victory in that state and a Biden election victory. These effect size measures accurately predicted the extent to which U.S. participants (N = 207, preregistered) viewed victory in a given state as having caused Biden to win the presidency. Our findings support the theory that people intuitively select as causes of an outcome the factors with the largest standardized causal effect on that outcome across possible counterfactual worlds.