1
|
Koul A, Novembre G. How accurately can we estimate spontaneous body kinematics from video recordings? Effect of movement amplitude on OpenPose accuracy. Behav Res Methods 2025; 57:38. [PMID: 39747756 PMCID: PMC11695451 DOI: 10.3758/s13428-024-02546-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/10/2024] [Indexed: 01/04/2025]
Abstract
Estimating how the human body moves in space and time-body kinematics-has important applications for industry, healthcare, and several research fields. Gold-standard methodologies capturing body kinematics are expensive and impractical for naturalistic recordings as they rely on infrared-reflective wearables and bulky instrumentation. To overcome these limitations, several algorithms have been developed to extract body kinematics from plain video recordings. This comes with a drop in accuracy, which however has not been clearly quantified. To fill this knowledge gap, we analysed a dataset comprising 46 human participants exhibiting spontaneous movements of varying amplitude. Body kinematics were estimated using OpenPose (video-based) and Vicon (infrared-based) motion capture systems simultaneously. OpenPose accuracy was assessed using Vicon estimates as ground truth. We report that OpenPose accuracy is overall moderate and varies substantially across participants and body parts. This is explained by variability in movement amplitude. OpenPose estimates are weak for low-amplitude movements. Conversely, large-amplitude movements (i.e., > ~ 10 cm) yield highly accurate estimates. The relationship between accuracy and movement amplitude is not linear (but mostly exponential or power) and relatively robust to camera-body distance. Together, these results dissect the limits of video-based motion capture and provide useful guidelines for future studies.
Collapse
Affiliation(s)
- Atesh Koul
- Neuroscience of Perception and Action Lab, Italian Institute of Technology (IIT), Viale Regina Elena 291, 00161, Rome, Italy.
| | - Giacomo Novembre
- Neuroscience of Perception and Action Lab, Italian Institute of Technology (IIT), Viale Regina Elena 291, 00161, Rome, Italy.
| |
Collapse
|
3
|
Eijk L, Rasenberg M, Arnese F, Blokpoel M, Dingemanse M, Doeller CF, Ernestus M, Holler J, Milivojevic B, Özyürek A, Pouw W, van Rooij I, Schriefers H, Toni I, Trujillo J, Bögels S. The CABB dataset: A multimodal corpus of communicative interactions for behavioural and neural analyses. Neuroimage 2022; 264:119734. [PMID: 36343884 DOI: 10.1016/j.neuroimage.2022.119734] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2022] [Revised: 10/07/2022] [Accepted: 11/03/2022] [Indexed: 11/06/2022] Open
Abstract
We present a dataset of behavioural and fMRI observations acquired in the context of humans involved in multimodal referential communication. The dataset contains audio/video and motion-tracking recordings of face-to-face, task-based communicative interactions in Dutch, as well as behavioural and neural correlates of participants' representations of dialogue referents. Seventy-one pairs of unacquainted participants performed two interleaved interactional tasks in which they described and located 16 novel geometrical objects (i.e., Fribbles) yielding spontaneous interactions of about one hour. We share high-quality video (from three cameras), audio (from head-mounted microphones), and motion-tracking (Kinect) data, as well as speech transcripts of the interactions. Before and after engaging in the face-to-face communicative interactions, participants' individual representations of the 16 Fribbles were estimated. Behaviourally, participants provided a written description (one to three words) for each Fribble and positioned them along 29 independent conceptual dimensions (e.g., rounded, human, audible). Neurally, fMRI signal evoked by each Fribble was measured during a one-back working-memory task. To enable functional hyperalignment across participants, the dataset also includes fMRI measurements obtained during visual presentation of eight animated movies (35 min total). We present analyses for the various types of data demonstrating their quality and consistency with earlier research. Besides high-resolution multimodal interactional data, this dataset includes different correlates of communicative referents, obtained before and after face-to-face dialogue, allowing for novel investigations into the relation between communicative behaviours and the representational space shared by communicators. This unique combination of data can be used for research in neuroscience, psychology, linguistics, and beyond.
Collapse
Affiliation(s)
- Lotte Eijk
- Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
| | - Marlou Rasenberg
- Centre for Language Studies, Radboud University, Nijmegen, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
| | - Flavia Arnese
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - Mark Blokpoel
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - Mark Dingemanse
- Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
| | - Christian F Doeller
- Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany; Kavli Institute for Systems Neuroscience, Centre for Neural Computation, The Egil and Pauline Braathen and Fred Kavli Centre for Cortical Microcircuits, Jebsen Centre for Alzheimer's Disease, Norwegian University of Science and Technology, Trondheim, Norway; Wilhelm Wundt Institute of Psychology, Leipzig University, Leipzig, Germany
| | - Mirjam Ernestus
- Centre for Language Studies, Radboud University, Nijmegen, the Netherlands
| | - Judith Holler
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
| | - Branka Milivojevic
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - Asli Özyürek
- Centre for Language Studies, Radboud University, Nijmegen, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands; Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - Wim Pouw
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
| | - Iris van Rooij
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Department of Linguistics, Cognitive Science, and Semiotics, and the Interacting Minds Centre at Aarhus University, Denmark
| | - Herbert Schriefers
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - Ivan Toni
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands
| | - James Trujillo
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
| | - Sara Bögels
- Donders Institute for Brain, Cognition, and Behaviour, Centre for Cognitive Neuroimaging, Radboud University, P.O.Box 9010, Nijmegen, Gelderland 6500, the Netherlands; Department of Cognition and Communication, Tilburg University, the Netherlands.
| |
Collapse
|
5
|
Trujillo JP, Levinson SC, Holler J. A multi-scale investigation of the human communication system's response to visual disruption. ROYAL SOCIETY OPEN SCIENCE 2022; 9:211489. [PMID: 35425638 PMCID: PMC9006025 DOI: 10.1098/rsos.211489] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/20/2021] [Accepted: 03/25/2022] [Indexed: 05/03/2023]
Abstract
In human communication, when the speech is disrupted, the visual channel (e.g. manual gestures) can compensate to ensure successful communication. Whether speech also compensates when the visual channel is disrupted is an open question, and one that significantly bears on the status of the gestural modality. We test whether gesture and speech are dynamically co-adapted to meet communicative needs. To this end, we parametrically reduce visibility during casual conversational interaction and measure the effects on speakers' communicative behaviour using motion tracking and manual annotation for kinematic and acoustic analyses. We found that visual signalling effort was flexibly adapted in response to a decrease in visual quality (especially motion energy, gesture rate, size, velocity and hold-time). Interestingly, speech was also affected: speech intensity increased in response to reduced visual quality (particularly in speech-gesture utterances, but independently of kinematics). Our findings highlight that multi-modal communicative behaviours are flexibly adapted at multiple scales of measurement and question the notion that gesture plays an inferior role to speech.
Collapse
Affiliation(s)
- James P. Trujillo
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, The Netherlands
- Max Planck Institute for Psycholinguistics, Wundtlaan 1, 6525XD Nijmegen, The Netherlands
| | - Stephen C. Levinson
- Max Planck Institute for Psycholinguistics, Wundtlaan 1, 6525XD Nijmegen, The Netherlands
| | - Judith Holler
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, The Netherlands
- Max Planck Institute for Psycholinguistics, Wundtlaan 1, 6525XD Nijmegen, The Netherlands
| |
Collapse
|