Functional neuroanatomy of intuitive physical inference

Dissociating language and thought in large language models

Large language models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction between formal linguistic competence (knowledge of linguistic rules and patterns) and functional linguistic competence (understanding and using language in the world). We ground this distinction in human neuroscience, which has shown that formal and functional competence rely on different neural mechanisms. Although LLMs are surprisingly good at formal competence, their performance on functional competence tasks remains spotty and often requires specialized fine-tuning and/or coupling with external modules. We posit that models that use language in human-like ways would need to master both of these competence types, which, in turn, could require the emergence of separate mechanisms specialized for formal versus functional linguistic competence.

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

A stochastic world model on gravity for stability inference

The fact that objects without proper support will fall to the ground is not only a natural phenomenon, but also common sense in mind. Previous studies suggest that humans may infer objects' stability through a world model that performs mental simulations with a priori knowledge of gravity acting upon the objects. Here we measured participants' sensitivity to gravity to investigate how the world model works. We found that the world model on gravity was not a faithful replica of the physical laws, but instead encoded gravity's vertical direction as a Gaussian distribution. The world model with this stochastic feature fit nicely with participants' subjective sense of objects' stability and explained the illusion that taller objects are perceived as more likely to fall. Furthermore, a computational model with reinforcement learning revealed that the stochastic characteristic likely originated from experience-dependent comparisons between predictions formed by internal simulations and the realities observed in the external world, which illustrated the ecological advantage of stochastic representation in balancing accuracy and speed for efficient stability inference. The stochastic world model on gravity provides an example of how a priori knowledge of the physical world is implemented in mind that helps humans operate flexibly in open-ended environments.

Perception of 3D shape integrates intuitive physics and analysis-by-synthesis

Many surface cues support three-dimensional shape perception, but humans can sometimes still see shape when these features are missing-such as when an object is covered with a draped cloth. Here we propose a framework for three-dimensional shape perception that explains perception in both typical and atypical cases as analysis-by-synthesis, or inference in a generative model of image formation. The model integrates intuitive physics to explain how shape can be inferred from the deformations it causes to other objects, as in cloth draping. Behavioural and computational studies comparing this account with several alternatives show that it best matches human observers (total n = 174) in both accuracy and response times, and is the only model that correlates significantly with human performance on difficult discriminations. We suggest that bottom-up deep neural network models are not fully adequate accounts of human shape perception, and point to how machine vision systems might achieve more human-like robustness.

A dedicated mental resource for intuitive physics

Countless decisions and actions in daily life draw on a mental model of the physical structure and dynamics of the world - from stepping carefully around a patch of slippery pavement to stacking delicate produce in a shopping basket. People can make fast and accurate inferences about how physical interactions will unfold, but it remains unclear whether we do so by applying a general set of physical principles across scenarios, or instead by reasoning about the physics of individual scenarios in an ad-hoc fashion. Here, we hypothesized that humans possess a dedicated and flexible mental resource for physical inference, and we tested for such a resource using a battery of fine-tuned tasks to capture individual differences in performance. Despite varying scene contents across tasks, we found that performance was highly correlated among them and well-explained by a unitary intuitive physics resource, distinct from other facets of cognition such as spatial reasoning and working memory.

The Role of Agentive and Physical Forces in the Neural Representation of Motion Events

How does the brain represent information about motion events in relation to agentive and physical forces? In this study, we investigated the neural activity patterns associated with observing animated actions of agents (e.g., an agent hitting a chair) in comparison to similar movements of inanimate objects that were either shaped solely by the physics of the scene (e.g., gravity causing an object to fall down a hill and hit a chair) or initiated by agents (e.g., a visible agent causing an object to hit a chair). Using an fMRI-based multivariate pattern analysis (MVPA), this design allowed testing where in the brain the neural activity patterns associated with motion events change as a function of, or are invariant to, agentive versus physical forces behind them. A total of 29 human participants (nine male) participated in the study. Cross-decoding revealed a shared neural representation of animate and inanimate motion events that is invariant to agentive or physical forces in regions spanning frontoparietal and posterior temporal cortices. In contrast, the right lateral occipitotemporal cortex showed a higher sensitivity to agentive events, while the left dorsal premotor cortex was more sensitive to information about inanimate object events that were solely shaped by the physics of the scene.

The "Little Circles Test" (LCT): a dusted-off tool for assessing fine visuomotor function

BACKGROUND

The fine visuomotor function is commonly impaired in several neurological conditions. However, there is a scarcity of reliable neuropsychological tools to assess such a critical domain.

AIMS

The aim of this study is to explore the psychometric properties and provide normative data for the Visual-Motor Speed and Precision Test (VMSPT).

RESULTS

Our normative sample included 220 participants (130 females) aged 18-86 years (mean education = 15.24 years, SD = 3.98). Results showed that raw VMSPT scores were affected by higher age and lower education. No effect of sex or handedness was shown. Age- and education-based norms were provided. VMSPT exhibited weak-to-strong correlations with well-known neuropsychological tests, encompassing a wide range of cognitive domains of clinical relevance. By gradually intensifying the cognitive demands, the test becomes an indirect, performance-oriented measure of executive functioning. Finally, VMSPT seems proficient in capturing the speed-accuracy trade-off typically observed in the aging population.

CONCLUSIONS

This study proposes the initial standardization of a versatile, time-efficient, and cost-effective neuropsychological tool for assessing fine visuomotor coordination. We propose renaming the VMSPT as the more approachable "Little Circles Test" (LCT).

Neural Foundations of Mental Simulation: Future Prediction of Latent Representations on Dynamic Scenes

Humans and animals have a rich and flexible understanding of the physical world, which enables them to infer the underlying dynamical trajectories of objects and events, plausible future states, and use that to plan and anticipate the consequences of actions. However, the neural mechanisms underlying these computations are unclear. We combine a goal-driven modeling approach with dense neurophysiological data and high-throughput human behavioral readouts that contain thousands of comparisons to directly impinge on this question. Specifically, we construct and evaluate several classes of sensory-cognitive networks to predict the future state of rich, ethologically-relevant environments, ranging from self-supervised end-to-end models with pixel-wise or object-slot objectives, to models that future predict in the latent space of purely static image-pretrained or dynamic video-pretrained foundation models. We find that "scale is not all you need", and that many state-of-the-art machine learning models fail to perform well on our neural and behavioral benchmarks for future prediction. In fact, only one class of models matches these data well overall. We find that neural responses are currently best predicted by models trained to predict the future state of their environment in the latent space of pretrained foundation models optimized for dynamic scenes in a self-supervised manner. These models also approach the neurons' ability to predict the environmental state variables that are visually hidden from view, despite not being explicitly trained to do so. Finally, we find that not all foundation model latents are equal. Notably, models that future predict in the latent space of video foundation models that are optimized to support a diverse range of egocentric sensorimotor tasks, reasonably match both human behavioral error patterns and neural dynamics across all environmental scenarios that we were able to test. Overall, these findings suggest that the neural mechanisms and behaviors of primate mental simulation have strong inductive biases associated with them, and are thus far most consistent with being optimized to future predict on reusable visual representations that are useful for Embodied AI more generally.

Autism-Related Differences in Cortical Activation When Observing, Producing, and Imitating Communicative Gestures: An fNIRS Study

Children with autism spectrum disorder (ASD) have difficulties in gestural communication during social interactions. However, the neural mechanisms involved in naturalistic gestural communication remain poorly understood. In this study, cortical activation patterns associated with gestural communication were examined in thirty-two children with and without ASD (mean age: 11.0 years, SE: 0.6 years). Functional near-infrared spectroscopy (fNIRS) was used to record cortical activation while children produced, observed, or imitated communicative gestures. Children with ASD demonstrated more spatial and temporal errors when performing and imitating communicative gestures. Although both typically developing (TD) children and children with ASD showed left-lateralized cortical activation during gesture production, children with ASD showed hyperactivation in the middle/inferior frontal gyrus (MIFG) during observation and imitation, and hypoactivation in the middle/superior temporal gyrus (MSTG) during gesture production compared to their TD peers. More importantly, children with ASD exhibited greater MSTG activation during imitation than during gesture production, suggesting that imitation could be an effective intervention strategy to engage cortical regions crucial for processing and producing gestures. Our study provides valuable insights into the neural mechanisms underlying gestural communication difficulties in ASD, while also identifying potential neurobiomarkers that could serve as objective measures for evaluating intervention effectiveness in children with ASD.

Object representation in a gravitational reference frame

When your head tilts laterally, as in sports, reaching, and resting, your eyes counterrotate less than 20%, and thus eye images rotate, over a total range of about 180°. Yet, the world appears stable and vision remains normal. We discovered a neural strategy for rotational stability in anterior inferotemporal cortex (IT), the final stage of object vision in primates. We measured object orientation tuning of IT neurons in macaque monkeys tilted +25 and -25° laterally, producing ~40° difference in retinal image orientation. Among IT neurons with consistent object orientation tuning, 63% remained stable with respect to gravity across tilts. Gravitational tuning depended on vestibular/somatosensory but also visual cues, consistent with previous evidence that IT processes scene cues for gravity's orientation. In addition to stability across image rotations, an internal gravitational reference frame is important for physical understanding of a world where object position, posture, structure, shape, movement, and behavior interact critically with gravity.

Socially intelligent machines that learn from humans and help humans learn

A hallmark of human intelligence is the ability to understand and influence other minds. Humans engage in inferential social learning (ISL) by using commonsense psychology to learn from others and help others learn. Recent advances in artificial intelligence (AI) are raising new questions about the feasibility of human-machine interactions that support such powerful modes of social learning. Here, we envision what it means to develop socially intelligent machines that can learn, teach, and communicate in ways that are characteristic of ISL. Rather than machines that simply predict human behaviours or recapitulate superficial aspects of human sociality (e.g. smiling, imitating), we should aim to build machines that can learn from human inputs and generate outputs for humans by proactively considering human values, intentions and beliefs. While such machines can inspire next-generation AI systems that learn more effectively from humans (as learners) and even help humans acquire new knowledge (as teachers), achieving these goals will also require scientific studies of its counterpart: how humans reason about machine minds and behaviours. We close by discussing the need for closer collaborations between the AI/ML and cognitive science communities to advance a science of both natural and artificial intelligence. This article is part of a discussion meeting issue 'Cognitive artificial intelligence'.

A shared neural code for the physics of actions and object events

Observing others' actions recruits frontoparietal and posterior temporal brain regions - also called the action observation network. It is typically assumed that these regions support recognizing actions of animate entities (e.g., person jumping over a box). However, objects can also participate in events with rich meaning and structure (e.g., ball bouncing over a box). So far, it has not been clarified which brain regions encode information specific to goal-directed actions or more general information that also defines object events. Here, we show a shared neural code for visually presented actions and object events throughout the action observation network. We argue that this neural representation captures the structure and physics of events regardless of animacy. We find that lateral occipitotemporal cortex encodes information about events that is also invariant to stimulus modality. Our results shed light onto the representational profiles of posterior temporal and frontoparietal cortices, and their roles in encoding event information.

fNIRS-Based Differences in Cortical Activation during Tool Use, Pantomimed Actions, and Meaningless Actions between Children with and without Autism Spectrum Disorder (ASD)

Children with autism spectrum disorder (ASD) have difficulties with tool use and pantomime actions. The current study utilized functional near-infrared spectroscopy (fNIRS) to examine the neural mechanisms underlying these gestural difficulties. Thirty-one children with and without ASD (age (mean ± SE) = 11.0 ± 0.6) completed a naturalistic peg-hammering task using an actual hammer (hammer condition), pantomiming hammering actions (pantomime condition), and performing meaningless actions with similar joint motions (meaningless condition). Children with ASD exhibited poor praxis performance (praxis error: TD = 17.9 ± 1.7; ASD = 27.0 ± 2.6, p < 0.01), which was significantly correlated with their cortical activation (R = 0.257 to 0.543). Both groups showed left-lateralized activation, but children with ASD demonstrated more bilateral activation during all gestural conditions. Compared to typically developing children, children with ASD showed hyperactivation of the inferior parietal lobe and hypoactivation of the middle/inferior frontal and middle/superior temporal regions. Our findings indicate intact technical reasoning (typical left-IPL activation) but atypical visuospatial and proprioceptive processing (hyperactivation of the right IPL) during tool use in children with ASD. These results have important implications for clinicians and researchers, who should focus on facilitating/reducing the burden of visuospatial and proprioceptive processing in children with ASD. Additionally, fNIRS-related biomarkers could be used for early identification through early object play/tool use and to examine neural effects following gesture-based interventions.

What is "Where": Physical Reasoning Informs Object Location

A central puzzle the visual system tries to solve is: "what is where?" While a great deal of research attempts to model object recognition ("what"), a comparatively smaller body of work seeks to model object location ("where"), especially in perceiving everyday objects. How do people locate an object, right now, in front of them? In three experiments collecting over 35,000 judgements on stimuli spanning different levels of realism (line drawings, real images, and crude forms), participants clicked "where" an object is, as if pointing to it. We modeled their responses with eight different methods, including both human response-based models (judgements of physical reasoning, spatial memory, free-response "click anywhere" judgements, and judgements of where people would grab the object), and image-based models (uniform distributions over the image, convex hull, saliency map, and medial axis). Physical reasoning was the best predictor of "where," performing significantly better than even spatial memory and free-response judgements. Our results offer insight into the perception of object locations while also raising interesting questions about the relationship between physical reasoning and visual perception.

F = ma. Is the macaque brain Newtonian?

Intuitive Physics, the ability to anticipate how the physical events involving mass objects unfold in time and space, is a central component of intelligent systems. Intuitive physics is a promising tool for gaining insight into mechanisms that generalize across species because both humans and non-human primates are subject to the same physical constraints when engaging with the environment. Physical reasoning abilities are widely present within the animal kingdom, but monkeys, with acute 3D vision and a high level of dexterity, appreciate and manipulate the physical world in much the same way humans do.

Bringing cumulative technological culture beyond copying versus reasoning

The dominant view of cumulative technological culture suggests that high-fidelity transmission rests upon a high-fidelity copying ability, which allows individuals to reproduce the tool-use actions performed by others without needing to understand them (i.e., without causal reasoning). The opposition between copying versus reasoning is well accepted but with little supporting evidence. In this article, we investigate this distinction by examining the cognitive science literature on tool use. Evidence indicates that the ability to reproduce others' tool-use actions requires causal understanding, which questions the copying versus reasoning distinction and the cognitive reality of the so-called copying ability. We conclude that new insights might be gained by considering causal understanding as a key driver of cumulative technological culture.

Recurrent neural networks with explicit representation of dynamic latent variables can mimic behavioral patterns in a physical inference task

Primates can richly parse sensory inputs to infer latent information. This ability is hypothesized to rely on establishing mental models of the external world and running mental simulations of those models. However, evidence supporting this hypothesis is limited to behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of primates (humans and monkeys) in a ball interception task to that of a large set of recurrent neural network (RNN) models with or without the capacity to dynamically track the underlying latent variables. Humans and monkeys exhibit similar behavioral patterns. This primate behavioral pattern is best captured by RNNs endowed with dynamic inference, consistent with the hypothesis that the primate brain uses dynamic inferences to support flexible physical predictions. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.

A study on brain neuronal activation based on the load in upper limb exercise (STROBE)

This study aimed to determine the level of brain activation in separate regions, including the lobes, cerebellum, and limbic system, depending on the weight of an object during elbow flexion and extension exercise using functional magnetic resonance imaging (fMRI). The study was conducted on ten male undergraduates (22.4 ± 1.2 years). The functional images of the brain were obtained using the 3T MRI. The participants performed upper limb flexion and extension exercise at a constant speed and as the weight of the object for lifting was varied (0 g and 1000 g). The experiment consisted of four blocks that constituted 8 minutes. Each block was designed to comprise a rest phase (1 minute) and a lifting phase (1 minute). The results showed that, in the parietal lobe, the activation was higher for the 0 g-motion condition than for the 1000 g-motion condition; however, in the occipital lobe, cerebellum, sub-lobar, and limbic system, the activation was higher for the 1000 g-motion condition than for the 0 g-motion condition. The brain region for the perception of object weight was identified as the ventral area (occipital, temporal, and frontal lobe), and the activation of the ventral pathway is suggested to have increased as the object came into vision and as its shape, size, and weight were perceived. For holding an object in hand, compared to not holding it, the exercise load was greater for controlling the motion to maintain the posture (arm angle at 90°), controlling the speed to repeat the motion at a constant speed, and producing an accurate posing. Therefore, to maintain such varied conditions, the activation level increased in the regions associated with control and regulation through the motion coordination from vision to arm movements (control of muscles). A characteristic reduced activation was observed in the regions associated with visuo-vestibular interaction and voluntary movement when the exercise involved lifting a 1000-g object compared to the exercise without object lifting.

The cortical thickness of the area PF of the left inferior parietal cortex mediates technical-reasoning skills

Most recent research highlights how a specific form of causal understanding, namely technical reasoning, may support the increasing complexity of tools and techniques developed by humans over generations, i.e., the cumulative technological culture (CTC). Thus, investigating the neurocognitive foundations of technical reasoning is essential to comprehend the emergence of CTC in our lineage. Whereas functional neuroimaging evidence started to highlight the critical role of the area PF of the left inferior parietal cortex (IPC) in technical reasoning, no studies explored the links between the structural characteristics of such a brain region and technical reasoning skills. Therefore, in this study, we assessed participants’ technical-reasoning performance by using two ad-hoc psycho-technical tests; then, we extracted from participants’ 3 T T1-weighted magnetic-resonance brain images the cortical thickness (i.e., a volume-related measure which is associated with cognitive performance as reflecting the size, density, and arrangement of cells in a brain region) of all the IPC regions for both hemispheres. We found that the cortical thickness of the left area PF predicts participants’ technical-reasoning performance. Crucially, we reported no correlations between technical reasoning and the other IPC regions, possibly suggesting the specificity of the left area PF in generating technical knowledge. We discuss these findings from an evolutionary perspective, by speculating about how the evolution of parietal lobes may have supported the emergence of technical reasoning in our lineage.

Neurocomputations of strategic behavior: From iterated to novel interactions

Strategic interactions, where an individual's payoff depends on the decisions of multiple intelligent agents, are ubiquitous among social animals. They span a variety of important social behaviors such as competition, cooperation, coordination, and communication, and often involve complex, intertwining cognitive operations ranging from basic reward processing to higher‐order mentalization. Here, we review the progress and challenges in probing the neural and cognitive mechanisms of strategic behavior of interacting individuals, drawing an analogy to recent developments in studies of reward‐seeking behavior, in particular, how research focuses in the field of strategic behavior have been expanded from adaptive behavior based on trial‐and‐error to flexible decisions based on limited prior experience. We highlight two important research questions in the field of strategic behavior: (i) How does the brain exploit past experience for learning to behave strategically? and (ii) How does the brain decide what to do in novel strategic situations in the absence of direct experience? For the former, we discuss the utility of learning models that have effectively connected various types of neural data with strategic learning behavior and helped elucidate the interplay among multiple learning processes. For the latter, we review the recent evidence and propose a neural generative mechanism by which the brain makes novel strategic choices through simulating others' goal‐directed actions according to rational or bounded‐rational principles obtained through indirect social knowledge.

This article is categorized under:

Economics > Interactive Decision‐Making

Psychology > Reasoning and Decision Making

Neuroscience > Cognition

Do capuchin monkeys (Sapajus apella) use exploration to form intuitions about physical properties?

Humans' flexible innovation relies on our capacity to accurately predict objects' behaviour. These predictions may originate from a "physics-engine" in the brain which simulates our environment. To explore the evolutionary origins of intuitive physics, we investigate whether capuchin monkeys' object exploration supports learning. Two capuchin groups experienced exploration sessions involving multiple copies of two objects, one object was easily opened (functional), the other was not (non-functional). We used two within-subject conditions (enrichment-then-test, and test-only) with two object sets per group. Monkeys then underwent individual test sessions where the objects contained rewards, and they choose one to attempt to open. The monkeys spontaneously explored, performing actions which yielded functional information. At test, both groups chose functional objects above chance. While high performance of the test-only group precluded us from establishing learning during exploration, this study reveals the promise of harnessing primates' natural exploratory tendencies to understand how they see the world.

The role of the anterior temporal cortex in action: evidence from fMRI multivariate searchlight analysis during real object grasping

Intelligent manipulation of handheld tools marks a major discontinuity between humans and our closest ancestors. Here we identified neural representations about how tools are typically manipulated within left anterior temporal cortex, by shifting a searchlight classifier through whole-brain real action fMRI data when participants grasped 3D-printed tools in ways considered typical for use (i.e., by their handle). These neural representations were automatically evocated as task performance did not require semantic processing. In fact, findings from a behavioural motion-capture experiment confirmed that actions with tools (relative to non-tool) incurred additional processing costs, as would be suspected if semantic areas are being automatically engaged. These results substantiate theories of semantic cognition that claim the anterior temporal cortex combines sensorimotor and semantic content for advanced behaviours like tool manipulation.

Invariant representation of physical stability in the human brain

Successful engagement with the world requires the ability to predict what will happen next. Here, we investigate how the brain makes a fundamental prediction about the physical world: whether the situation in front of us is stable, and hence likely to stay the same, or unstable, and hence likely to change in the immediate future. Specifically, we ask if judgments of stability can be supported by the kinds of representations that have proven to be highly effective at visual object recognition in both machines and brains, or instead if the ability to determine the physical stability of natural scenes may require generative algorithms that simulate the physics of the world. To find out, we measured responses in both convolutional neural networks (CNNs) and the brain (using fMRI) to natural images of physically stable versus unstable scenarios. We find no evidence for generalizable representations of physical stability in either standard CNNs trained on visual object and scene classification (ImageNet), or in the human ventral visual pathway, which has long been implicated in the same process. However, in frontoparietal regions previously implicated in intuitive physical reasoning we find both scenario-invariant representations of physical stability, and higher univariate responses to unstable than stable scenes. These results demonstrate abstract representations of physical stability in the dorsal but not ventral pathway, consistent with the hypothesis that the computations underlying stability entail not just pattern classification but forward physical simulation.

Physical understanding in neurodegenerative diseases

This quantitative review gives an overview of physical understanding (i.e., the ability to represent and use the laws of physics to interact with the physical world) impairments in Alzheimer's disease (AD), semantic dementia (SD), and corticobasal syndrome (CBS), as assessed mainly with mechanical problem-solving and tool use tests. This review shows that: (1) SD patients have apraxia of tool use because of semantic tool knowledge deficits, but normal performance in tests of physical understanding; (2) AD and CBS patients show impaired performance in mechanical problem-solving tests, probably not because of intrinsic deficits of physical understanding, but rather because of additional cognitive (AD) or motor impairments (CBS); (3) As a result, the performance in mechanical problem-solving tests is not a good predictor of familiar tool use in dementia; (4) Actual deficits of physical understanding are probably observed only in late stages of neurodegenerative diseases, and associated with functional loss.

Melting Ice With Your Mind: Representational Momentum for Physical States

When a log burns, it transforms from a block of wood into a pile of ash. Such state changes are among the most dramatic ways objects change, going beyond mere changes of position or orientation. How does the mind represent changes of state? A foundational result in visual cognition is that memory extrapolates the positions of moving objects-a distortion called representational momentum. Here, five experiments (N = 400 adults) exploited this phenomenon to investigate mental representations in state space. Participants who viewed objects undergoing state changes (e.g., ice melting, logs burning, or grapes shriveling) remembered them as more changed (e.g., more melted, burned, or shriveled) than they actually were. This pattern extended to several types of state changes, went beyond their low-level properties, and even adhered to their natural trajectories in state space. Thus, mental representations of objects actively incorporate how they change-not only in their relation to their environment, but also in their essential qualities.

Precise functional connections between the dorsal anterior cingulate cortex and areas recruited for physical inference

Recent work has identified brain areas that are engaged when people predict how the physical behavior of the world will unfold - an ability termed intuitive physics. Among the many unanswered questions about the neural mechanisms of intuitive physics is where the key inputs come from: which brain regions connect up with intuitive physics processes to regulate when and how they are engaged in service of our goals? In the present work, we targeted the dorsal anterior cingulate cortex (dACC) for study based on characteristics that make it well-positioned to regulate intuitive physics processes. The dACC is richly interconnected with frontoparietal regions and is implicated in mapping contexts to actions, a process that would benefit from physical predictions to indicate which action(s) would produce the desired physical outcomes. We collected resting state functional MRI data in seventeen participants and used independent task-related runs to find the pattern of activity during a physical inference task in each individual participant. We found that the strongest resting state functional connections of the dACC not only aligned well with physical inference-related activity at the group level, it also mirrored individual differences in the positioning of physics-related activity across participants. Our results suggest that the dACC might be a key structure for regulating the engagement of intuitive physics processes in the brain.

Bottom-up and top-down modulation of route selection in imitation

The cognitive system selects the most appropriate action imitative process: a semantic process - relying on long-term memory representations for known actions, and low-level visuomotor transformations for unknown actions. These two processes work in parallel; however, how context regularities and cognitive control modulate them is unclear. In this study, process selection was triggered contextually by presenting mixed known and new actions in predictable or unpredictable lists, while a cue on the forthcoming action triggered top-down control. Known were imitated faster than the new actions in the predictable lists only. Accuracy was higher and reaction times faster in the uncued conditions, and the predictable faster than the unpredictable list in the uncued condition only. In the latter condition, contextual factors modulate process selection, as participants use statistical regularities to perform the task at best. With the cue, the cognitive system tries to control response selection, resulting in more errors and longer reaction times.

On the psychological origins of tool use

The ubiquity of tool use in human life has generated multiple lines of scientific and philosophical investigation to understand the development and expression of humans' engagement with tools and its relation to other dimensions of human experience. However, existing literature on tool use faces several epistemological challenges in which the same set of questions generate many different answers. At least four critical questions can be identified, which are intimately intertwined-(1) What constitutes tool use? (2) What psychological processes underlie tool use in humans and nonhuman animals? (3) Which of these psychological processes are exclusive to tool use? (4) Which psychological processes involved in tool use are exclusive to Homo sapiens? To help advance a multidisciplinary scientific understanding of tool use, six author groups representing different academic disciplines (e.g., anthropology, psychology, neuroscience) and different theoretical perspectives respond to each of these questions, and then point to the direction of future work on tool use. We find that while there are marked differences among the responses of the respective author groups to each question, there is a surprising degree of agreement about many essential concepts and questions. We believe that this interdisciplinary and intertheoretical discussion will foster a more comprehensive understanding of tool use than any one of these perspectives (or any one of these author groups) would (or could) on their own.

Behavioral and neural representations en route to intuitive action understanding

When we observe another person's actions, we process many kinds of information - from how their body moves to the intention behind their movements. What kinds of information underlie our intuitive understanding about how similar actions are to each other? To address this question, we measured the intuitive similarities among a large set of everyday action videos using multi-arrangement experiments, then used a modeling approach to predict this intuitive similarity space along three hypothesized properties. We found that similarity in the actors' inferred goals predicted the intuitive similarity judgments the best, followed by similarity in the actors' movements, with little contribution from the videos' visual appearance. In opportunistic fMRI analyses assessing brain-behavior correlations, we found suggestive evidence for an action processing hierarchy, in which these three kinds of action similarities are reflected in the structure of brain responses along a posterior-to-anterior gradient on the lateral surface of the visual cortex. Altogether, this work joins existing literature suggesting that humans are naturally tuned to process others' intentions, and that the visuo-motor cortex computes the perceptual precursors of the higher-level representations over which intuitive action perception operates.

Naturalistic embodied interactions elicit intuitive physical behaviour in accordance with Newtonian physics

The success of visuomotor interactions in everyday activities such as grasping or sliding a cup is inescapably governed by the laws of physics. Research on intuitive physics has predominantly investigated reasoning about objects' behaviour involving binary forced choice responses. We investigated how the type of visuomotor response influences participants' beliefs about physical quantities and their lawful relationship implicit in their active behaviour. Participants propelled pucks towards targets positioned at different distances. Analysis with a probabilistic model of interactions showed that subjects adopted the non-linear control prescribed by Newtonian physics when sliding real pucks in a virtual environment even in the absence of visual feedback. However, they used a linear heuristic when viewing the scene on a monitor and interactions were implemented through key presses. These results support the notion of probabilistic internal physics models but additionally suggest that humans can take advantage of embodied, sensorimotor, multimodal representations in physical scenarios.

World model learning and inference

Understanding information processing in the brain-and creating general-purpose artificial intelligence-are long-standing aspirations of scientists and engineers worldwide. The distinctive features of human intelligence are high-level cognition and control in various interactions with the world including the self, which are not defined in advance and are vary over time. The challenge of building human-like intelligent machines, as well as progress in brain science and behavioural analyses, robotics, and their associated theoretical formalisations, speaks to the importance of the world-model learning and inference. In this article, after briefly surveying the history and challenges of internal model learning and probabilistic learning, we introduce the free energy principle, which provides a useful framework within which to consider neuronal computation and probabilistic world models. Next, we showcase examples of human behaviour and cognition explained under that principle. We then describe symbol emergence in the context of probabilistic modelling, as a topic at the frontiers of cognitive robotics. Lastly, we review recent progress in creating human-like intelligence by using novel probabilistic programming languages. The striking consensus that emerges from these studies is that probabilistic descriptions of learning and inference are powerful and effective ways to create human-like artificial intelligent machines and to understand intelligence in the context of how humans interact with their world.

The neuroscience of advanced scientific concepts

Cognitive neuroscience methods can identify the fMRI-measured neural representation of familiar individual concepts, such as apple, and decompose them into meaningful neural and semantic components. This approach was applied here to determine the neural representations and underlying dimensions of representation of far more abstract physics concepts related to matter and energy, such as fermion and dark matter, in the brains of 10 Carnegie Mellon physics faculty members who thought about the main properties of each of the concepts. One novel dimension coded the measurability vs. immeasurability of a concept. Another novel dimension of representation evoked particularly by post-classical concepts was associated with four types of cognitive processes, each linked to particular brain regions: (1) Reasoning about intangibles, taking into account their separation from direct experience and observability; (2) Assessing consilience with other, firmer knowledge; (3) Causal reasoning about relations that are not apparent or observable; and (4) Knowledge management of a large knowledge organization consisting of a multi-level structure of other concepts. Two other underlying dimensions, previously found in physics students, periodicity, and mathematical formulation, were also present in this faculty sample. The data were analyzed using factor analysis of stably responding voxels, a Gaussian-naïve Bayes machine-learning classification of the activation patterns associated with each concept, and a regression model that predicted activation patterns associated with each concept based on independent ratings of the dimensions of the concepts. The findings indicate that the human brain systematically organizes novel scientific concepts in terms of new dimensions of neural representation.

What tool representation, intuitive physics, and action have in common: The brain's first-person physics engine

An overlapping set of brain regions in parietal and frontal cortex are engaged by different types of tasks and stimuli: (i) making inferences about the physical structure and dynamics of the world, (ii) passively viewing, or actively interacting with, manipulable objects, and (iii) planning and execution of reaching and grasping actions. We suggest the observed neural overlap is because a common superordinate computation is engaged by each of those different tasks: A forward model of physical reasoning about how first-person actions will affect the world and be affected by unfolding physical events. This perspective offers an account of why some physical predictions are systematically incorrect - there can be a mismatch between how physical scenarios are experimentally framed and the native format of the inferences generated by the brain's first-person physics engine. This perspective generates new empirical expectations about the conditions under which physical reasoning may exhibit systematic biases.

A role for visual areas in physics simulations

To engage with the world, we must regularly make predictions about the outcomes of physical scenes. How do we make these predictions? Recent computational evidence points to simulation-the idea that we can introspectively manipulate rich, mental models of the world-as one explanation for how such predictions are accomplished. However, questions about the potential neural mechanisms of simulation remain. We hypothesized that the process of simulating physical events would evoke imagery-like representations in visual areas of those same events. Using functional magnetic resonance imaging, we find that when participants are asked to predict the likely trajectory of a falling ball, motion-sensitive brain regions are activated. We demonstrate that this activity, which occurs even though no motion is being sensed, resembles activity patterns that arise while participants perceive the ball's motion. This finding thus suggests that mental simulations recreate sensory depictions of how a physical scene is likely to unfold.

The Cognitive Science of Technology

Technology is central to human life but hard to define and study. This review synthesizes advances in fields from anthropology to evolutionary biology and neuroscience to propose an interdisciplinary cognitive science of technology. The foundation of this effort is an evolutionarily motivated definition of technology that highlights three key features: material production, social collaboration, and cultural reproduction. This broad scope respects the complexity of the subject but poses a challenge for theoretical unification. Addressing this challenge requires a comparative approach to reduce the diversity of real-world technological cognition to a smaller number of recurring processes and relationships. To this end, a synthetic perceptual-motor hypothesis (PMH) for the evolutionary-developmental-cultural construction of technological cognition is advanced as an initial target for investigation.

Mathematics Meets Science in the Brain

Mathematics and science are highly integrated disciplines, but the brain association between mathematics and science remains unclear. The current study used functional magnetic resonance imaging (fMRI) scans of 34 undergraduates (17 males, mean age = 20.3±1.64 years old) while they completed mathematical, physical and chemical principles, arithmetic computation, and sentence comprehension. We examined neural activation level, neural activation pattern, and neural connectivity to investigate the neural associations between mathematics and science (including physics and chemistry). The results showed that mathematical, physical, and chemical principles elicited similar neural activation level and neural activation pattern in the visuospatial network (mainly in the middle frontal gyrus and inferior parietal lobule), which were different from those elicited by sentence comprehension; those three principles also elicited similar neural activation level and neural activation pattern in the semantic network (mainly in the middle temporal gyrus, angular gyrus, inferior frontal gyrus, and dorsomedial prefrontal cortex), in contrast to that elicited by arithmetic computation. Effective connectivity analyses showed stronger connectivity between the middle temporal gyrus and inferior parietal lobule for mathematical, physical, and chemical principles than for sentence comprehension. The results suggest that visuospatial and semantic networks were critical for processing both mathematics and science.

Hand-Selective Visual Regions Represent How to Grasp 3D Tools: Brain Decoding during Real Actions

Most neuroimaging experiments that investigate how tools and their actions are represented in the brain use visual paradigms where tools or hands are displayed as 2D images and no real movements are performed. These studies discovered selective visual responses in occipitotemporal and parietal cortices for viewing pictures of hands or tools, which are assumed to reflect action processing, but this has rarely been directly investigated. Here, we examined the responses of independently visually defined category-selective brain areas when participants grasped 3D tools (N = 20; 9 females). Using real-action fMRI and multivoxel pattern analysis, we found that grasp typicality representations (i.e., whether a tool is grasped appropriately for use) were decodable from hand-selective areas in occipitotemporal and parietal cortices, but not from tool-, object-, or body-selective areas, even if partially overlapping. Importantly, these effects were exclusive for actions with tools, but not for biomechanically matched actions with control nontools. In addition, grasp typicality decoding was significantly higher in hand than tool-selective parietal regions. Notably, grasp typicality representations were automatically evoked even when there was no requirement for tool use and participants were naive to object category (tool vs nontools). Finding a specificity for typical tool grasping in hand-selective, rather than tool-selective, regions challenges the long-standing assumption that activation for viewing tool images reflects sensorimotor processing linked to tool manipulation. Instead, our results show that typicality representations for tool grasping are automatically evoked in visual regions specialized for representing the human hand, the primary tool of the brain for interacting with the world.

The Language Network Is Recruited but Not Required for Nonverbal Event Semantics

The ability to combine individual concepts of objects, properties, and actions into complex representations of the world is often associated with language. Yet combinatorial event-level representations can also be constructed from nonverbal input, such as visual scenes. Here, we test whether the language network in the human brain is involved in and necessary for semantic processing of events presented nonverbally. In Experiment 1, we scanned participants with fMRI while they performed a semantic plausibility judgment task versus a difficult perceptual control task on sentences and line drawings that describe/depict simple agent-patient interactions. We found that the language network responded robustly during the semantic task performed on both sentences and pictures (although its response to sentences was stronger). Thus, language regions in healthy adults are engaged during a semantic task performed on pictorial depictions of events. But is this engagement necessary? In Experiment 2, we tested two individuals with global aphasia, who have sustained massive damage to perisylvian language areas and display severe language difficulties, against a group of age-matched control participants. Individuals with aphasia were severely impaired on the task of matching sentences to pictures. However, they performed close to controls in assessing the plausibility of pictorial depictions of agent-patient interactions. Overall, our results indicate that the left frontotemporal language network is recruited but not necessary for semantic processing of nonverbally presented events.

See, feel, act: Hierarchical learning for complex manipulation skills with multisensory fusion

Humans are able to seamlessly integrate tactile and visual stimuli with their intuitions to explore and execute complex manipulation skills. They not only see but also feel their actions. Most current robotic learning methodologies exploit recent progress in computer vision and deep learning to acquire data-hungry pixel-to-action policies. These methodologies do not exploit intuitive latent structure in physics or tactile signatures. Tactile reasoning is omnipresent in the animal kingdom, yet it is underdeveloped in robotic manipulation. Tactile stimuli are only acquired through invasive interaction, and interpretation of the data stream together with visual stimuli is challenging. Here, we propose a methodology to emulate hierarchical reasoning and multisensory fusion in a robot that learns to play Jenga, a complex game that requires physical interaction to be played effectively. The game mechanics were formulated as a generative process using a temporal hierarchical Bayesian model, with representations for both behavioral archetypes and noisy block states. This model captured descriptive latent structures, and the robot learned probabilistic models of these relationships in force and visual domains through a short exploration phase. Once learned, the robot used this representation to infer block behavior patterns and states as it played the game. Using its inferred beliefs, the robot adjusted its behavior with respect to both its current actions and its game strategy, similar to the way humans play the game. We evaluated the performance of the approach against three standard baselines and show its fidelity on a real-world implementation of the game.

How do the object-file and physical-reasoning systems interact? Evidence from priming effects with object arrays or novel labels

How do infants reason about simple physical events such as containment, tube, and support events? According to the two-system model, two cognitive systems, the object-file (OF) and physical-reasoning (PR) systems, work together to guide infants' responses to these events. When an event begins, the OF system sends categorical information about the objects and their arrangements to the PR system. This system then categorizes the event, assigns event roles to the objects, and taps the OF system for information about features previously identified as causally relevant for the event category selected. All of the categorical and featural information included in the event's representation is interpreted by the PR system's domain knowledge, which includes core principles such as persistence and gravity. The present research tested a novel prediction of the model: If the OF system could be primed to also send, at the beginning of an event, information about an as-yet-unidentified feature, the PR system would then interpret this information using its core principles, allowing infants to detect core violations involving the feature earlier than they normally would. We examined this prediction using two types of priming manipulations directed at the OF system, object arrays and novel labels. In six experiments, infants aged 7-13 months (N = 304) were tested using different event categories and as-yet-unidentified features (color in containment events, height in tube events, and proportional distribution in support events) as well as different tasks (violation-of-expectation and action tasks). In each case, infants who were effectively primed reasoned successfully about the as-yet-unidentified feature, sometimes as early as six months before they would typically do so. These converging results provide strong support for the two-system model and for the claim that uncovering how the OF and PR systems represent and exchange information is essential for understanding how infants respond to physical events.

Understanding Human Cognitive Uniqueness

Humanity has regarded itself as intellectually superior to other species for millennia, yet human cognitive uniqueness remains poorly understood. Here, we evaluate candidate traits plausibly underlying our distinctive cognition (including mental time travel, tool use, problem solving, social cognition, and communication) as well as domain generality, and we consider how human cognitive uniqueness may have evolved. We conclude that there are no traits present in humans and absent in other animals that in isolation explain our species' superior cognitive performance; rather, there are many cognitive domains in which humans possess unusually potent capabilities compared to those found in other species. Humans are flexible cognitive all-rounders, whose proficiency arises through interactions and reinforcement between cognitive domains at multiple scales.

Comprehension of computer code relies primarily on domain-general executive brain regions

Computer programming is a novel cognitive tool that has transformed modern society. What cognitive and neural mechanisms support this skill? Here, we used functional magnetic resonance imaging to investigate two candidate brain systems: the multiple demand (MD) system, typically recruited during math, logic, problem solving, and executive tasks, and the language system, typically recruited during linguistic processing. We examined MD and language system responses to code written in Python, a text-based programming language (Experiment 1) and in ScratchJr, a graphical programming language (Experiment 2); for both, we contrasted responses to code problems with responses to content-matched sentence problems. We found that the MD system exhibited strong bilateral responses to code in both experiments, whereas the language system responded strongly to sentence problems, but weakly or not at all to code problems. Thus, the MD system supports the use of novel cognitive tools even when the input is structurally similar to natural language.

Large-scale dissociations between views of objects, scenes, and reachable-scale environments in visual cortex

Space-related processing recruits a network of brain regions separate from those recruited in object processing. This dissociation has largely been explored by contrasting views of navigable-scale spaces to views of close-up, isolated objects. However, in naturalistic visual experience, we encounter spaces intermediate to these extremes, like the tops of desks and kitchen counters, which are not navigable but typically contain multiple objects. How are such reachable-scale views represented in the brain? In three human functional neuroimaging experiments, we find evidence for a large-scale dissociation of reachable-scale views from both navigable scene views and close-up object views. Three brain regions were identified that showed a systematic response preference to reachable views, located in the posterior collateral sulcus, the inferior parietal sulcus, and superior parietal lobule. Subsequent analyses suggest that these three regions may be especially sensitive to the presence of multiple objects. Further, in all classic scene and object regions, reachable-scale views dissociated from both objects and scenes with an intermediate response magnitude. Taken together, these results establish that reachable-scale environments have a distinct representational signature from both scene and object views in visual cortex.

Social interaction networks in the primate brain

Primate brains have evolved to understand and engage with their social world. Much about the structure of this world can be gleaned from social interactions. Circuits for the analysis of and participation in social interactions have now been mapped. Increased knowledge about their functional specializations and relative spatial locations promises to greatly improve the understanding of the functional organization of the primate social brain. Detailed electrophysiology, as in the case of the face-processing network, of local operations and functional interactions between areas is necessary to uncover neural mechanisms and computation principles of social cognition. New naturalistic behavioral paradigms, behavioral tracking, and new analytical approaches for parallel non-stationary data will be important components toward a neuroscientific theory of primates' interactive minds.