Differences in audiovisual temporal processing in autistic adults are specific to simultaneity judgments

Research has shown that children on the autism spectrum and adults with high levels of autistic traits are less sensitive to audiovisual asynchrony compared to their neurotypical peers. However, this evidence has been limited to simultaneity judgments (SJ) which require participants to consider the timing of two cues together. Given evidence of partly divergent perceptual and neural mechanisms involved in making temporal order judgments (TOJ) and SJ, and given that SJ require a more global type of processing which may be impaired in autistic individuals, here we ask whether the observed differences in audiovisual temporal processing are task and stimulus specific. We examined the ability to detect audiovisual asynchrony in a group of 26 autistic adult males and a group of age and IQ-matched neurotypical males. Participants were presented with beep-flash, point-light drumming, and face-voice displays with varying degrees of asynchrony and asked to make SJ and TOJ. The results indicated that autistic participants were less able to detect audiovisual asynchrony compared to the control group, but this effect was specific to SJ and more complex social stimuli (e.g., face-voice) with stronger semantic correspondence between the cues, requiring a more global type of processing. This indicates that audiovisual temporal processing is not generally different in autistic individuals and that a similar level of performance could be achieved by using a more local type of processing, thus informing multisensory integration theory as well as multisensory training aimed to aid perceptual abilities in this population.

Past and present experience shifts audiovisual temporal perception in rats

Our brains have a propensity to integrate closely-timed auditory and visual stimuli into a unified percept; a phenomenon that is highly malleable based on prior sensory experiences, and is known to be altered in clinical populations. While the neural correlates of audiovisual temporal perception have been investigated using neuroimaging and electroencephalography techniques in humans, animal research will be required to uncover the underlying cellular and molecular mechanisms. Prior to conducting such mechanistic studies, it is important to first confirm the translational potential of any prospective animal model. Thus, in the present study, we conducted a series of experiments to determine if rats show the hallmarks of audiovisual temporal perception observed in neurotypical humans, and whether the rat behavioral paradigms could reveal when they experienced perceptual disruptions akin to those observed in neurodevelopmental disorders. After training rats to perform a temporal order judgment (TOJ) or synchrony judgment (SJ) task, we found that the rats' perception was malleable based on their past and present sensory experiences. More specifically, passive exposure to asynchronous audiovisual stimulation in the minutes prior to behavioral testing caused the rats' perception to predictably shift in the direction of the leading stimulus; findings which represent the first time that this form of audiovisual perceptual malleability has been reported in non-human subjects. Furthermore, rats performing the TOJ task also showed evidence of rapid recalibration, in which their audiovisual temporal perception on the current trial was predictably influenced by the timing lag between the auditory and visual stimuli in the preceding trial. Finally, by manipulating either experimental testing parameters or altering the rats' neurochemistry with a systemic injection of MK-801, we showed that the TOJ and SJ tasks could identify when the rats had difficulty judging the timing of audiovisual stimuli. These findings confirm that the behavioral paradigms are indeed suitable for future testing of rats with perceptual disruptions in audiovisual processing. Overall, our collective results highlight that rats represent an excellent animal model to study the cellular and molecular mechanisms underlying the acuity and malleability of audiovisual temporal perception, as they showcase the perceptual hallmarks commonly observed in humans.

Spatial attention modulates auditory dominance in audiovisual order judgment

Auditory dominance in audiovisual temporal order judgment is shown here to be modulated by exogenous orienting of attention to a spatial cue independent of the cue modality. The visual stimulus has to lead the auditory one further in advance for cued relative to uncued locations in order for the two to be perceived simultaneously, possibly suggesting an inhibitory function of spatial attention on temporal processing.

The Influence of Personal Harmony Value on Temporal Order Perception

Empirical studies have demonstrated that self-relevant information affects temporal order perception. Therefore, the question arises of whether personal values-which are the core components of the self-influence temporal order perception. To explore this problem, we chose harmony, one of the most common values in Chinese culture, as the starting point. First, the harmony scale was used to measure the harmony values of the participants, and the participants were divided into high- and low-harmony groups. The validity of the grouping was then verified using an implicit-association test. Furthermore, two temporal order judgment (TOJ) tasks were used to explore the impact of harmony values on temporal order perception. The results revealed that in both TOJ tasks, participants in the high-harmony group tended to perceive harmonious stimuli before non-harmonious stimuli, while the effect was not found in the low-harmony group. We conclude that harmony values affect temporal order perception, and only if the values are important to the individual.

Temporal order judgment of multisensory stimuli in rat and human

We do not fully understand the resolution at which temporal information is processed by different species. Here we employed a temporal order judgment (TOJ) task in rats and humans to test the temporal precision with which these species can detect the order of presentation of simple stimuli across two modalities of vision and audition. Both species reported the order of audiovisual stimuli when they were presented from a central location at a range of stimulus onset asynchronies (SOA)s. While both species could reliably distinguish the temporal order of stimuli based on their sensory content (i.e., the modality label), rats outperformed humans at short SOAs (less than 100 ms) whereas humans outperformed rats at long SOAs (greater than 100 ms). Moreover, rats produced faster responses compared to humans. The reaction time data further revealed key differences in decision process across the two species: at longer SOAs, reaction times increased in rats but decreased in humans. Finally, drift-diffusion modeling allowed us to isolate the contribution of various parameters including evidence accumulation rates, lapse and bias to the sensory decision. Consistent with the psychophysical findings, the model revealed higher temporal sensitivity and a higher lapse rate in rats compared to humans. These findings suggest that these species applied different strategies for making perceptual decisions in the context of a multimodal TOJ task.

Reduced audiovisual temporal sensitivity in Chinese children with dyslexia

Background

Temporal processing deficits regarding audiovisual cross-modal stimuli could affect children's speed and accuracy of decoding.

Aim

To investigate the characteristics of audiovisual temporal sensitivity (ATS) in Chinese children, with and without developmental dyslexia and its impact on reading ability.

Methods

The audiovisual simultaneity judgment and temporal order judgment tasks were performed to investigate the ATS of 106 Chinese children (53 with dyslexia) aged 8 to 12 and 37 adults without a history of dyslexia. The predictive effect of children's audiovisual time binding window on their reading ability and the effects of extra cognitive processing in the temporal order judgment task on participants' ATS were also investigated.

Outcomes and results

With increasing inter-stimulus intervals, the percentage of synchronous responses in adults declined more rapidly than in children. Adults and typically developing children had significantly narrower time binding windows than children with dyslexia. The size of visual stimuli preceding auditory stimuli time binding window had a marginally significant predictive effect on children's reading fluency. Compared with the simultaneity judgment task, the extra cognitive processing of the temporal order judgment task affected children's ATS.

Conclusion and implications

The ATS of 8-12-year-old Chinese children is immature. Chinese children with dyslexia have lower ATS than their peers.

Early Capture of Attention by Self-Face: Investigation Using a Temporal Order Judgment Task

Earlier work on self-face processing has reported a bias in the processing of self-face result in faster response to self-face in comparison to other familiar and unfamiliar faces (termed as self-face advantage or SFA). Even though most studies agree that the SFA occurs due to an attentional bias, there is little agreement regarding the stage at which it occurs. While a large number of studies show self-face influencing processing later at disengagement stage, early event-related potential components show differential activity for the self-face suggesting that SFA occurs early. We address this contradiction using a cueless temporal order judgment task that allows us to investigate early perceptual processing, while bias due to top-down expectation is controlled. A greater shift in point of subjective simultaneity for self-face would indicate a greater processing advantage at early perceptual stage. With help of two experiments, we show an early perceptual advantage for self-face, compared to both a friend's face and an unfamiliar face (Experiment 1). This advantage is present even when the effect of criterion shift is minimized (Experiment 2). Interestingly, the magnitude of advantage is similar for self-friend and self-unfamiliar pair. The evidence from the two experiments suggests early capture of attention as a likely reason for the SFA, which is present for the self-face but not for other familiar faces.

No Evidence for an Effect of the Distance Between the Hands on Tactile Temporal Order Judgments

Localizing somatosensory stimuli is an important process, as it allows us to spatially guide our actions toward the object entering in contact with the body. Accordingly, the positions of tactile inputs are coded according to both somatotopic and spatiotopic representations, the latter one considering the position of the stimulated limbs in external space. The spatiotopic representation has often been evidenced by means of temporal order judgment (TOJ) tasks. Participants' judgments about the order of appearance of two successive somatosensory stimuli are less accurate when the hands are crossed over the body midline than uncrossed but also when participants' hands are placed close together when compared with farther away. Moreover, these postural effects might depend on the vision of the stimulated limbs. The aim of this study was to test the influence of seeing the hands, on the modulation of tactile TOJ by the spatial distance between the stimulated limbs. The results showed no influence of the distance between the stimulated hands on TOJ performance and prevent us from concluding whether vision of the hands affects TOJ performance, or whether these variables interact. The reliability of such distance effect to investigate the spatial representations of tactile inputs is questioned.

The Effect of Rhythmic Tactile Stimuli Under the Voluntary Movement on Audio-Tactile Temporal Order Judgement

The simultaneous perception of multimodal sensory information is important for effective reactions to the external environment. In relation to the effect on time perception, voluntary movement and rhythmic stimuli have already been identified in previous studies to be associated with improved accuracy of temporal order judgments (TOJs). Here, we examined whether the combination of voluntary movement and rhythmic stimuli improves the just noticeable difference (JND) in audio-tactile TOJ Tasks. Four different experimental conditions were studied, involving two types of movements (voluntary movement, involuntary movement) and two types of stimulus presentation (rhythmic, one-time only). In the voluntary movement condition (VM), after the auditory stimulus (cue sound) participants moved their right index finger voluntarily and naturally, while in the involuntary movement condition (IM), their right index finger was moved by the tactile device. The stimuli were provided in a rhythmic or one-time only manner by hitting inside the first joint of the participants' right index finger using a tactile device. Furthermore, in the rhythmical tactile (RT) conditions, tactile stimuli were presented rhythmically to the right index finger 5 times consecutively. On the other hand, in the one-time tactile (1T) conditions, tactile stimuli was presented one-time only to the right index finger. Participants made an order judgment for the fifth tactile stimuli and the first and only auditory stimuli. In our TOJ tasks, auditory-tactile stimulus pairs were presented to participants with varying stimulus-onset asynchronies (SOAs; intervals between the within-pair onsets of the auditory and tactile stimuli). For the two stimuli presented at a time that were shifted by the SOA, the participants were asked to judge which one was presented first, and they were given a two-choice answer. Using a non-parametric test, our results showed that voluntary movement and rhythmic tactile stimuli were both effective in improving the JNDs in TOJ Tasks. However, in the combination of voluntary movement and rhythmic tactile stimuli, we found that there was no significant difference in JNDs in our experiments.

Mindfulness Meditation Biases Visual Temporal Order Discrimination but Not Under Conditions of Temporal Ventriloquism

This study examined how cognitive plasticity acquired from a long (8 weeks) course of mindfulness training can modulate the perceptual processing of temporal order judgment (TOJ) on a sub-second scale. Observers carried out a TOJ on two visual disks, with or without concurrent paired beeps. A temporal ventriloquism paradigm was used in which the sound beeps either were synchronized with the two disks or bracketed the visual stimuli by leading the first disk by 50 ms and lagging the other by 50 ms. A left-to-right bias in TOJ was found under the visual-only condition after mindfulness training. This bias was positively correlated with “acting with awareness,” a factor in the Five Facet Mindfulness Questionnaire, showing that awareness of every moment and enhanced attention focus magnify the left-to-right bias. However, the effect of mindfulness training may be short-lived and was not present when attention was diverted by auditory events in the cross-modal temporal ventriloquism illusion.

Neural Basis of Extremely High Temporal Sensitivity: Insights From a Patient With Autism

The human brain is sensitive to incoming sensory information across multiple time scales. Temporal scales of information represented in the brain generally constrain behavior. Despite reports of the neural correlates of millisecond timing, how the human brain processes sensory stimuli in the sub-second range (≤100 ms) and its behavioral implications are areas of active scientific inquiry. An autism spectrum disorder (ASD) patient showed a tactile discrimination threshold of 6.49 ms on a temporal order judgment (TOJ) task which was approximately 10-fold superior than other ASD and healthy controls (59 and 69 ms, respectively). To investigate the brain regions of this extremely high temporal resolution in the patient, we used functional magnetic resonance imaging (fMRI) during TOJ. We observed greater activity notably in the left superior temporal gyrus (STG) and precentral gyrus (PrG) compared to that of controls. Generally, the left superior frontal gyrus (SFG) correlated positively, while the opercular part of right inferior frontal gyrus (IFG) correlated negatively, with the correct TOJ rate across all subjects (the patient + 22 healthy controls). We found that the performance was negatively correlated with the strength of neural responses in the right IFG overall in 30 participants (the patient + 22 healthy and 7 ASD controls). Our data reveal superior ability of this particular case of ASD in the millisecond scale for sensory inputs. We highlight several neural correlates of TOJ underlying the facilitation and/or inhibition of temporal resolution in humans.

Behavioral Plasticity of Audiovisual Perception: Rapid Recalibration of Temporal Sensitivity but Not Perceptual Binding Following Adult-Onset Hearing Loss

The ability to accurately integrate or bind stimuli from more than one sensory modality is highly dependent on the features of the stimuli, such as their intensity and relative timing. Previous studies have demonstrated that the ability to perceptually bind stimuli is impaired in various clinical conditions such as autism, dyslexia, schizophrenia, as well as aging. However, it remains unknown if adult-onset hearing loss, separate from aging, influences audiovisual temporal acuity. In the present study, rats were trained using appetitive operant conditioning to perform an audiovisual temporal order judgment (TOJ) task or synchrony judgment (SJ) task in order to investigate the nature and extent that audiovisual temporal acuity is affected by adult-onset hearing loss, with a specific focus on the time-course of perceptual changes following loud noise exposure. In our first series of experiments, we found that audiovisual temporal acuity in normal-hearing rats was influenced by sound intensity, such that when a quieter sound was presented, the rats were biased to perceive the audiovisual stimuli as asynchronous (SJ task), or as though the visual stimulus was presented first (TOJ task). Psychophysical testing demonstrated that noise-induced hearing loss did not alter the rats' temporal sensitivity 2-3 weeks post-noise exposure, despite rats showing an initial difficulty in differentiating the temporal order of audiovisual stimuli. Furthermore, consistent with normal-hearing rats, the timing at which the stimuli were perceived as simultaneous (i.e., the point of subjective simultaneity, PSS) remained sensitive to sound intensity following hearing loss. Contrary to the TOJ task, hearing loss resulted in persistent impairments in asynchrony detection during the SJ task, such that a greater proportion of trials were now perceived as synchronous. Moreover, psychophysical testing found that noise-exposed rats had altered audiovisual synchrony perception, consistent with impaired audiovisual perceptual binding (e.g., an increase in the temporal window of integration on the right side of simultaneity; right temporal binding window (TBW)). Ultimately, our collective results show for the first time that adult-onset hearing loss leads to behavioral plasticity of audiovisual perception, characterized by a rapid recalibration of temporal sensitivity but a persistent impairment in the perceptual binding of audiovisual stimuli.

Violation of the Unity Assumption Disrupts Temporal Ventriloquism Effect in Starlings

When stimuli from different sensory modalities are received, they may be combined by the brain to form a multisensory percept. One key mechanism for multisensory binding is the unity assumption under which multisensory stimuli that share certain physical properties like temporal and/or spatial correspondence are grouped together as deriving from one object. In humans, evidence for a role of the unity assumption has been found in spatial tasks and also in temporal tasks using stimuli that share physical properties (speech-related stimuli, musical and synesthetically congruent stimuli). In our study, we investigate the role of the unity assumption in an animal model in a temporal order judgment task. When subjects are asked to indicate which of two spatially separated visual stimuli appeared first in time, performance improves when the visual stimuli are paired (in time) with spatially non-informative acoustic cues, a phenomenon known as the temporal ventriloquism effect. Here, we show that European starlings perform better when one singleton acoustic cue is paired with the first visual stimulus as compared to pairing with the second visual stimulus. This shows, in combination with our previous study, that a non-informative singleton acoustic cue, when temporally paired with the first visual stimulus, triggers alerting while, when temporally pairing with the second visual stimulus, it prevents a temporal ventriloquism effect because the unity assumption is violated. Thus, the unity assumption influences sensory perception not only in humans but also in an animal model. The importance of the unity assumption in this task supports the idea that the temporal ventriloquism effect, similar to the spatial ventriloquism effect, is based on multisensory binding and integration but not on alerting effects.

Visual Experience Shapes the Neural Networks Remapping Touch into External Space

Localizing touch relies on the activation of skin-based and externally defined spatial frames of reference. Psychophysical studies have demonstrated that early visual deprivation prevents the automatic remapping of touch into external space. We used fMRI to characterize how visual experience impacts the brain circuits dedicated to the spatial processing of touch. Sighted and congenitally blind humans performed a tactile temporal order judgment (TOJ) task, either with the hands uncrossed or crossed over the body midline. Behavioral data confirmed that crossing the hands has a detrimental effect on TOJ judgments in sighted but not in early blind people. Crucially, the crossed hand posture elicited enhanced activity, when compared with the uncrossed posture, in a frontoparietal network in the sighted group only. Psychophysiological interaction analysis revealed, however, that the congenitally blind showed enhanced functional connectivity between parietal and frontal regions in the crossed versus uncrossed hand postures. Our results demonstrate that visual experience scaffolds the neural implementation of the location of touch in space.SIGNIFICANCE STATEMENT In daily life, we seamlessly localize touch in external space for action planning toward a stimulus making contact with the body. For efficient sensorimotor integration, the brain has therefore to compute the current position of our limbs in the external world. In the present study, we demonstrate that early visual deprivation alters the brain activity in a dorsal parietofrontal network typically supporting touch localization in the sighted. Our results therefore conclusively demonstrate the intrinsic role that developmental vision plays in scaffolding the neural implementation of touch perception.

Modulation of Illusory Reversal in Tactile Temporal Order by the Phase of Posterior α Rhythm

The subjective temporal order of tactile stimuli, delivered sequentially to each hand with an interval of 100-300 ms, is often inverted when the arms are crossed. Based on data from behavioral and neuroimaging studies, it has been proposed that the reversal is due to a conflict between anatomical and spatial representations of the tactile signal or to the production of an inverted apparent motion signal. Because the α rhythms, which consist of a few distinct components, reportedly modulate tactile perception and apparent motion and serve as a 10 Hz timer, we hypothesized that the illusory reversal would be regulated by some of the α rhythms. To test this hypothesis, we conducted magnetoencephalographic recordings in both male and female participants during the tactile temporal order judgment task. We decomposed the α rhythms into five independent components and discovered that the illusory reversal was modulated by the phase of one independent component with strong current sources near the parieto-occipital (PO) sulcus (peri-PO component). As expected, the estimated current sources distributed over the human MST implicated to represent tactile apparent motion, in addition to the intraparietal region implicated in mapping tactile signals in space. However, the strongest source was located in the precuneus that occupies a central hub region in the cortical networks and receives tactile inputs through a tecto-thalamic pathway. These results suggest that the peri-PO component plays an essential role in regulating tactile temporal perception by modulating the thalamic nuclei that interconnect the superior colliculus with the cortical networks.SIGNIFICANCE STATEMENT Despite a long-held hypothesis that the posterior α rhythm serves as a 10 Hz timer that regulates human temporal perception, the contribution of the α rhythms in temporal perception is still unclear. We examined how the α rhythms influence tactile temporal order judgment. Judgment reversal depended on the phase of one particular α rhythm with its source near the parieto-occipital sulcus. The peri-parieto-occipital α rhythm may play a crucial role in organizing tactile temporal perception.

Audiovisual Temporal Processing and Synchrony Perception in the Rat

Extensive research on humans has improved our understanding of how the brain integrates information from our different senses, and has begun to uncover the brain regions and large-scale neural activity that contributes to an observer’s ability to perceive the relative timing of auditory and visual stimuli. In the present study, we developed the first behavioral tasks to assess the perception of audiovisual temporal synchrony in rats. Modeled after the parameters used in human studies, separate groups of rats were trained to perform: (1) a simultaneity judgment task in which they reported whether audiovisual stimuli at various stimulus onset asynchronies (SOAs) were presented simultaneously or not; and (2) a temporal order judgment task in which they reported whether they perceived the auditory or visual stimulus to have been presented first. Furthermore, using in vivo electrophysiological recordings in the lateral extrastriate visual (V2L) cortex of anesthetized rats, we performed the first investigation of how neurons in the rat multisensory cortex integrate audiovisual stimuli presented at different SOAs. As predicted, rats (n = 7) trained to perform the simultaneity judgment task could accurately (~80%) identify synchronous vs. asynchronous (200 ms SOA) trials. Moreover, the rats judged trials at 10 ms SOA to be synchronous, whereas the majority (~70%) of trials at 100 ms SOA were perceived to be asynchronous. During the temporal order judgment task, rats (n = 7) perceived the synchronous audiovisual stimuli to be “visual first” for ~52% of the trials, and calculation of the smallest timing interval between the auditory and visual stimuli that could be detected in each rat (i.e., the just noticeable difference (JND)) ranged from 77 ms to 122 ms. Neurons in the rat V2L cortex were sensitive to the timing of audiovisual stimuli, such that spiking activity was greatest during trials when the visual stimulus preceded the auditory by 20–40 ms. Ultimately, given that our behavioral and electrophysiological results were consistent with studies conducted on human participants and previous recordings made in multisensory brain regions of different species, we suggest that the rat represents an effective model for studying audiovisual temporal synchrony at both the neuronal and perceptual level.

Early Visual Perception Potentiated by Object Affordances: Evidence From a Temporal Order Judgment Task

Perceived objects automatically potentiate afforded action. Object affordances also facilitate perception of such objects, and this occurrence is known as the affordance effect. This study examined whether object affordances facilitate the initial visual processing stage, or perceptual entry processes, using the temporal order judgment task. The onset of the graspable (right-handled) coffee cup was perceived earlier than that of the less graspable (left-handled) cup for right-handed participants. The affordance effect was eliminated when the coffee cups were inverted, which presumably conveyed less affordance information. These results suggest that objects preattentively potentiate the perceptual entry processes in response to their affordances.

Motor Planning Influences the Perceived Timing of Vibrotactile Stimuli in an Amplitude-Dependent Manner

The authors characterized how motor planning influences temporal order judgment (TOJ) tasks. They examined this by applying vibrotactile stimulation during the planning stages of a bimanual arm movement that would bring the arms into a crossed configuration. The authors have previously shown that planning to cross the arms induces a subjective reversal of spatially defined temporal order judgments that evolves over the course of the planning period. It was unclear, however, whether this effect is modulated by the extent to which the arms would be crossed after movement. The authors examined this issue by having participants plan to move to 4 different targets that would leave the arms in crossed configurations of varying extents. The results demonstrate that even though cutaneous stimuli were applied before the movements, if participants were planning to move into a more crossed configuration, performance on the TOJ task worsened depending on where they were in the planning process. This data suggest the brain uses planning signals to predict sensations from impending movements in a context-dependent manner.

Metaplasticity in human primary somatosensory cortex: effects on physiology and tactile perception

Theta-burst stimulation (TBS) over human primary motor cortex evokes plasticity and metaplasticity, the latter contributing to the homeostatic balance of excitation and inhibition. Our knowledge of TBS-induced effects on primary somatosensory cortex (SI) is limited, and it is unknown whether TBS induces metaplasticity within human SI. Sixteen right-handed participants (6 females, mean age 23 yr) received two TBS protocols [continuous TBS (cTBS) and intermittent TBS (iTBS)] delivered in six different combinations over SI in separate sessions. TBS protocols were delivered at 30 Hz and were as follows: a single cTBS protocol, a single iTBS protocol, cTBS followed by cTBS, iTBS followed by iTBS, cTBS followed by iTBS, and iTBS followed by cTBS. Measures included the amplitudes of the first and second somatosensory evoked potentials (SEPs) via median nerve stimulation, their paired-pulse ratio (PPR), and temporal order judgment (TOJ). Dependent measures were obtained before TBS and at 5, 25, 50, and 90 min following stimulation. Results indicate similar effects following cTBS and iTBS; increased amplitudes of the second SEP and PPR without amplitude changes to SEP 1, and impairments in TOJ. Metaplasticity was observed such that TOJ impairments following a single cTBS protocol were abolished following consecutive cTBS protocols. Additionally, consecutive iTBS protocols altered the time course of effects when compared with a single iTBS protocol. In conclusion, 30-Hz cTBS and iTBS protocols delivered in isolation induce effects consistent with a TBS-induced reduction in intracortical inhibition within SI. Furthermore, cTBS- and iTBS-induced metaplasticity appear to follow homeostatic and nonhomeostatic rules, respectively.

Awareness of Temporal Lag is Necessary for Motor-Visual Temporal Recalibration

Consistent exposure to a temporal lag between observers' voluntary action and its visual feedback induced recalibration of temporal order perception between a motor action and a visual stimulus. It remains unclear what kinds of processing underlie this motor–visual temporal recalibration. This study examined the necessity of awareness of a temporal lag between a motor action and its visual feedback for motor–visual temporal recalibration. In Experiment 1, we allocated observers to either the multiple-step or single-step lag conditions. In the multiple-step lag condition, we first inserted a small temporal lag and subsequently increased it with progress of the adaptation period, to make observers unaware of the temporal lag during the adaptation period. In the single-step lag condition, we instructed observers about the temporal lag before adaptation, and inserted a substantial temporal lag from the beginning of the adaptation period to ensure that they were aware of the temporal lag. We found significant recalibration only in the single-step lag condition. In Experiment 2, we exposed all observers to a substantial temporal lag from the beginning of adaptation period with no instruction about insertion of the temporal lag. We asked observers at the end of the experiment whether they were aware of the temporal lag. We found significant recalibration for only observers who were aware of the lag. These results suggest that awareness of the temporal lag plays a crucial role in motor–visual temporal recalibration.

Dissociation of perception and action in audiovisual multisensory integration

The ‘temporal rule’ of multisensory integration (MI) proposes that unisensory stimuli, and the neuronal responses they evoke, must fall within a window of integration. Ecological validity demands that MI should occur only for physically simultaneous events (which may give rise to non‐simultaneous neural activations), and spurious neural response simultaneities unrelated to environmental multisensory occurrences must somehow be rejected. Two experiments investigated the requirements of simultaneity for facilitative MI. Experiment 1 employed an reaction time (RT)/race model paradigm to measure audiovisual (AV) MI as a function of AV stimulus‐onset asynchrony (SOA) under fully dark adapted conditions for visual stimuli that were either rod‐ or cone‐isolating. Auditory stimulus intensity was constant. Despite a 155‐ms delay in mean RT to the scotopic vs. photopic stimulus, facilitative AV MI in both conditions occurred exclusively at an AV SOA of 0 ms. Thus, facilitative MI demands both physical and physiological simultaneity. Experiment 2 investigated the accuracy of simultaneity and temporal order judgements under the same stimulus conditions. Judgements of AV stimulus simultaneity or temporal order were significantly influenced by stimulus intensity, indicating different simultaneity requirements for these tasks. The possibility was considered that there are mechanisms by which the nervous system may take account of variations in response latency arising from changes in stimulus intensity in order to selectively integrate only those physiological simultaneities that arise from physical simultaneities. It was proposed that separate subsystems for AV MI exist that pertain to action and perception.

The simultaneous perception of auditory-tactile stimuli in voluntary movement

The simultaneous perception of multimodal information in the environment during voluntary movement is very important for effective reactions to the environment. Previous studies have found that voluntary movement affects the simultaneous perception of auditory and tactile stimuli. However, the results of these experiments are not completely consistent, and the differences may be attributable to methodological differences in the previous studies. In this study, we investigated the effect of voluntary movement on the simultaneous perception of auditory and tactile stimuli using a temporal order judgment task with voluntary movement, involuntary movement, and no movement. To eliminate the potential effect of stimulus predictability and the effect of spatial information associated with large-scale movement in the previous studies, we randomized the interval between the start of movement and the first stimulus, and used small-scale movement. As a result, the point of subjective simultaneity (PSS) during voluntary movement shifted from the tactile stimulus being first during involuntary movement or no movement to the auditory stimulus being first. The just noticeable difference (JND), an indicator of temporal resolution, did not differ across the three conditions. These results indicate that voluntary movement itself affects the PSS in auditory–tactile simultaneous perception, but it does not influence the JND. In the discussion of these results, we suggest that simultaneous perception may be affected by the efference copy.

Memory in time: electrophysiological comparison between reality filtering and temporal order judgment

Orbitofrontal reality filtering (ORF) denotes a little known but vital memory control mechanism, expressed at 200-300ms after stimulus presentation, that allows one to sense whether evoked memories (thoughts) refer to present reality and can be acted upon, or not. Its failure induces reality confusion evident in confabulations that patients act upon and disorientation. In what way ORF differs from temporal order judgment (TOJ), that is, the conscious knowledge about when something happened in the past, has never been explored. Here we used evoked potential analysis to compare ORF and TOJ within a combined experimental task and within a comparable time frame, close to the experienced present. Seventeen healthy human subjects performed an experiment using continuous recognition tasks that combined the challenges of ORF and TOJ. We found that the two mechanisms dissociated behaviorally: subjects were markedly slower and less accurate in TOJ than ORF. Both mechanisms evoked similar potentials at 240-280ms, when ORF normally occurs. However, they rapidly dissociated in terms of amplitude differences and electrical source from 310 to 360ms and again from 530 to 560ms. We conclude that the task of consciously ordering memories in the immediate past (TOJ) is effortful and slow in contrast to sensing memories' relation with the present (ORF). Both functions invoke similar early electrocortical processes which then rapidly dissociate and engage different brain areas. The results are consistent with the different consequences of the two mechanisms' dysfunction: while failure of ORF has a known clinical manifestation (reality confusion as evident in confabulation and disorientation), the failure of TOJ, as tested here, has no such known clinical correlate.

The effect of numerical magnitude on the perceptual processing speed of a digit

In this study, we investigated whether the numerical information of a digit would affect perceptual processing speed for that digit. In Experiment 1, participants performed a temporal order judgment (TOJ) task in which they judged the order of two digits presented briefly to the left or right of the fixation point with a short asynchrony. The point of subjective simultaneity (PSS) was significantly shifted such that large numbers had to be presented before small numbers in order to be perceived as simultaneous, implying that small numbers are perceptually processed faster than large numbers. Given the susceptibility of a TOJ task to response bias, this result might have simply reflected the conceptual association between magnitude (e.g., small) and response selection (e.g., first). To exclude the potential influence of response bias, we adopted a simultaneity judgment (SJ) task in Experiment 2. Most participants in Experiment 2 had participated in Experiment 1. The participants judged whether the two digits were presented simultaneously or successively. The maximal possibility of simultaneous response was obtained when a large digit preceded a small digit by 5 ms, suggesting that small numbers were indeed perceived earlier than large numbers. Our findings indicated that small numbers were processed faster than large ones and that perceptual mechanisms contribute to this temporal advantage. In addition, although the TOJ and SJ task produced a similar processing speed advantage for small numbers, the PSSs of the two tasks were not correlated, which implied that different cognitive mechanisms were involved in the two tasks.

Temporal order judgments are disrupted more by reflexive than by voluntary saccades

We do not always perceive the sequence of events as they actually unfold. For example, when two events occur before a rapid eye movement (saccade), the interval between them is often perceived as shorter than it really is and the order of those events can be sometimes reversed (Morrone MC, Ross J, Burr DC. Nat Neurosci 8: 950-954, 2005). In the present article we show that these misperceptions of the temporal order of events critically depend on whether the saccade is reflexive or voluntary. In the first experiment, participants judged the temporal order of two visual stimuli that were presented one after the other just before a reflexive or voluntary saccadic eye movement. In the reflexive saccade condition, participants moved their eyes to a target that suddenly appeared. In the voluntary saccade condition, participants moved their eyes to a target that was present already. Similarly to the above-cited study, we found that the temporal order of events was often misjudged just before a reflexive saccade to a suddenly appearing target. However, when people made a voluntary saccade to a target that was already present, there was a significant reduction in the probability of misjudging the temporal order of the same events. In the second experiment, the reduction was seen in a memory-delay task. It is likely that the nature of the motor command and its origin determine how time is perceived during the moments preceding the motor act.

Priming makes a stimulus more salient

The present study used visual prior entry to determine which of two stimuli received attention first. Observers were asked to judge whether two test stimuli across a range of stimulus onset asynchronies (SOAs) were synchronized or not (simultaneity judgment task; SJ), or to report the temporal order of the two test stimuli (temporal order judgment task; TOJ). Before the presentation of the two test stimuli, a single noninformative stimulus that matched the color of one of the test stimuli was presented in the center of the display. The results showed that, in both the TOJ and SJ tasks, the noninformative stimulus caused a shift in the psychometric function such that the test stimulus that had the same color as the preceding noninformative stimulus was seen earlier in time than the test stimulus that had a color that did not match. In other words, the mere processing of the color of a noninformative stimulus, rendered the stimulus having that same color more salient, an effect that we attributed to priming. Because priming made one of the stimuli more salient, it received attention first and accelerated its processing, causing prior entry into awareness. Importantly, when the noninformative stimulus was a color word, no such priming effect was observed. We conclude that a primed test stimulus has the ability to capture attention in an automatic way.

Remapping time across space

Multiple lines of evidence indicate that visual attention's temporal properties differ between the left and right visual fields (LVF and RVF). Notably, recent electroencephalograph recordings indicate that event-related potentials peak earlier for LVF than for RVF targets on bilateral-stream rapid serial visual presentation (RSVP) identification tasks. Might this hastened neural response render LVF targets perceptually available sooner than RVF targets? If so, how might the visual system reconcile these timing differences to estimate simultaneity across the LVF and RVF? We approached these questions by presenting bilateral-stream RSVP displays that contained opposite-hemifield targets and requiring participants to judge both the targets' temporal order and simultaneity. The temporal order judgments (TOJs) revealed that participants perceived LVF targets ∼134 ms sooner than RVF targets. This LVF hastening approximates a full cycle of visual attention's canonical ∼10 Hz (∼100 ms) temporal resolution. In contrast, performance on the simultaneity task did not exhibit the LVF hastening observed on the TOJ task, despite identical retinal stimulation across the two tasks. This finding rules out a stimulus-driven "bottom-up" explanation for the task-specific behavior. Moreover, error patterns across the two tasks revealed that, within the decision stage of simultaneity judgments, participants remapped LVF targets, but not RVF targets, to a later time in the RSVP sequence. Such hemifield-specific remapping would effectively compensate for the cross-hemifield asymmetries in neural response latencies that could otherwise impair simultaneity estimates.

Effect of visual stimuli on temporal order judgments of a sequence of pure tones

We investigated what effect visual spatial information had on auditory temporal order judgments (TOJs) and examined what effect visual stimuli had on the TOJs of sequences of pure tones in experiment 1. The auditory stimuli were sequences of four distinct pure tones. The visual stimuli consisted of two vertically aligned flashes: one flashed before the first tone and the other flashed after the last tone. Participants judged whether the temporal order of the second and third tones in auditory stimuli occurred with the higher tone being first or the lower tone being first. As a result, the proportion of responses for higher-tone-first increased when the flash of the upper LED preceded that of the lower LED, independent of the actual temporal order. Participants in experiment 2 were asked to make simultaneity judgments instead, which were also affected by visual stimuli. The auditory stimuli in experiment 3 were the same as those in experiment 1, whereas the visual stimuli consisted of two horizontally aligned flashes. Furthermore, the participants made TOJs, which were not affected by the horizontally aligned visual stimuli. We concluded that vertically aligned visual stimuli had an effect on auditory TOJs with some response bias.

Non-retinotopic motor-visual recalibration to temporal lag

Temporal order judgment (TOJ) between the voluntary motor action and its perceptual feedback is important in distinguishing between a sensory feedback which is caused by observer's own action and other stimulus, which are irrelevant to that action. Prolonged exposure to fixed temporal lag between motor action and visual feedback recalibrates motor-visual temporal relationship, and consequently shifts the point of subjective simultaneity (PSS). Previous studies on the audio-visual temporal recalibration without voluntary action revealed that the low level processing is involved. However, it is not clear how the low and high level processings affect the recalibration to constant temporal lag between voluntary action and visual feedback. This study examined retinotopic specificity of the motor-visual temporal recalibration. During the adaptation phase, observers repeatedly pressed a key, and visual stimulus was presented in left or right visual field with a fixed temporal lag (0 or 200 ms). In the test phase, observers performed a TOJ for observer's voluntary keypress and test stimulus, which was presented in the same as or opposite to the visual field in which the stimulus was presented in the adaptation phase. We found that the PSS was shifted toward the exposed lag in both visual fields. These results suggest that the low visual processing, which is retinotopically specific, has minor contribution to the motor-visual temporal lag adaptation, and that the adaptation to shift the PSS mainly depends upon the high level processing such as attention to specific properties of the stimulus.

A neural model for temporal order judgments and their active recalibration: a common mechanism for space and time?

When observers experience a constant delay between their motor actions and sensory feedback, their perception of the temporal order between actions and sensations adapt (Stetson et al., 2006). We present here a novel neural model that can explain temporal order judgments (TOJs) and their recalibration. Our model employs three ubiquitous features of neural systems: (1) information pooling, (2) opponent processing, and (3) synaptic scaling. Specifically, the model proposes that different populations of neurons encode different delays between motor-sensory events, the outputs of these populations feed into rivaling neural populations (encoding "before" and "after"), and the activity difference between these populations determines the perceptual judgment. As a consequence of synaptic scaling of input weights, motor acts which are consistently followed by delayed sensory feedback will cause the network to recalibrate its point of subjective simultaneity. The structure of our model raises the possibility that recalibration of TOJs is a temporal analog to the motion aftereffect (MAE). In other words, identical neural mechanisms may be used to make perceptual determinations about both space and time. Our model captures behavioral recalibration results for different numbers of adapting trials and different adapting delays. In line with predictions of the model, we additionally demonstrate that temporal recalibration can last through time, in analogy to storage of the MAE.

Acquisition of multiple prior distributions in tactile temporal order judgment

The Bayesian estimation theory proposes that the brain acquires the prior distribution of a task and integrates it with sensory signals to minimize the effect of sensory noise. Psychophysical studies have demonstrated that our brain actually implements Bayesian estimation in a variety of sensory-motor tasks. However, these studies only imposed one prior distribution on participants within a task period. In this study, we investigated the conditions that enable the acquisition of multiple prior distributions in temporal order judgment of two tactile stimuli across the hands. In Experiment 1, stimulation intervals were randomly selected from one of two prior distributions (biased to right hand earlier and biased to left hand earlier) in association with color cues (green and red, respectively). Although the acquisition of the two priors was not enabled by the color cues alone, it was significant when participants shifted their gaze (above or below) in response to the color cues. However, the acquisition of multiple priors was not significant when participants moved their mouths (opened or closed). In Experiment 2, the spatial cues (above and below) were used to identify which eye position or retinal cue position was crucial for the eye-movement-dependent acquisition of multiple priors in Experiment 1. The acquisition of the two priors was significant when participants moved their gaze to the cues (i.e., the cue positions on the retina were constant across the priors), as well as when participants did not shift their gazes (i.e., the cue positions on the retina changed according to the priors). Thus, both eye and retinal cue positions were effective in acquiring multiple priors. Based on previous neurophysiological reports, we discuss possible neural correlates that contribute to the acquisition of multiple priors.

Response errors explain the failure of independent-channels models of perception of temporal order

Independent-channels models of perception of temporal order (also referred to as threshold models or perceptual latency models) have been ruled out because two formal properties of these models (monotonicity and parallelism) are not borne out by data from ternary tasks in which observers must judge whether stimulus A was presented before, after, or simultaneously with stimulus B. These models generally assume that observed responses are authentic indicators of unobservable judgments, but blinks, lapses of attention, or errors in pressing the response keys (maybe, but not only, motivated by time pressure when reaction times are being recorded) may make observers misreport their judgments or simply guess a response. We present an extension of independent-channels models that considers response errors and we show that the model produces psychometric functions that do not satisfy monotonicity and parallelism. The model is illustrated by fitting it to data from a published study in which the ternary task was used. The fitted functions describe very accurately the absence of monotonicity and parallelism shown by the data. These characteristics of empirical data are thus consistent with independent-channels models when response errors are taken into consideration. The implications of these results for the analysis and interpretation of temporal order judgment data are discussed.

Size Isn't All that Matters: Noticing Differences in Size and Temporal Order

The ability to represent time and size is essential for thought and action. These domains have traditionally been investigated independently. However, the processing of events in time and space is postulated to have considerable anatomical and behavioral overlap. Here we formally tested for associations and dissociations of abilities in these domains. We examined patterns of impairments in temporal order and relative size judgments in 40 patients with unilateral brain lesions and 20 age-matched control participants. While brain damage can impair both size and temporal order judgments (TOJ), we did not find evidence for global hemispheric differences. When patients were analyzed individually compared to control subjects, we found double dissociations in performances on both kinds of judgments. Voxel lesion symptom mapping allowed us to investigate shared and unique contributions of brain damage to deficits in judgments noticing differences in temporal order and in spatial extent. We found that size and temporal order estimations have overlapping cortical vulnerabilities within the left inferior frontal gyrus, left superior temporal cortex, and bilateral inferior parietal lobule. However, vulnerability unique to TOJ occurred with damage predominantly in left lateralized regions involving inferior and middle frontal cortex and inferior parietal lobule. Conversely, vulnerability unique to size judgments occurred with damage predominantly in right lateralized regions in the supramarginal gyrus and superior temporal cortex. These data provide evidence for interactions between the processing of spatial extent and temporal order; however, they do not provide evidence for right lateralized systems.