Content-specific evidence accumulation in inferior temporal cortex during perceptual decision-making

Intracranial electroencephalography reveals effector-independent evidence accumulation dynamics in multiple human brain regions

Neural representations of perceptual decision formation that are abstracted from specific motor requirements have previously been identified in humans using non-invasive electrophysiology; however, it is currently unclear where these originate in the brain. Here we capitalized on the high spatiotemporal precision of intracranial EEG to localize such abstract decision signals. Participants undergoing invasive electrophysiological monitoring for epilepsy were asked to judge the direction of random-dot stimuli and respond either with a speeded button press (N = 24), or vocally, after a randomized delay (N = 12). We found a widely distributed motor-independent network of regions where high-frequency activity exhibited key characteristics consistent with evidence accumulation, including a gradual buildup that was modulated by the strength of the sensory evidence, and an amplitude that predicted participants' choice accuracy and response time. Our findings offer a new view on the brain networks governing human decision-making.

Functional imaging analyses reveal prototype and exemplar representations in a perceptual single-category task

Similarity-based categorization can be performed by memorizing category members as exemplars or by abstracting the central tendency of the category – the prototype. In similarity-based categorization of stimuli with clearly identifiable dimensions from two categories, prototype representations were previously located in the hippocampus and the ventromedial prefrontal cortex (vmPFC) and exemplar representations in areas supporting visual memory. However, the neural implementation of exemplar and prototype representations in perceptual similarity-based categorization of single categories is unclear. To investigate these representations, we applied model-based univariate and multivariate analyses of functional imaging data from a dot-pattern paradigm-based task. Univariate prototype and exemplar representations occurred bilaterally in visual areas. Multivariate analyses additionally identified prototype representations in parietal areas and exemplar representations in the hippocampus. Bayesian analyses supported the non-presence of prototype representations in the hippocampus and the vmPFC. We additionally demonstrate that some individuals form both representation types simultaneously, probably granting flexibility in categorization strategies.

Model-based univariate and multivariate analyses of fMRI data from 62 healthy participants in a dot-pattern paradigm-based task provide further insight into the neural basis of similarity-based categorization.

Neural Substrates of the Drift-Diffusion Model in Brain Disorders

Many studies on the drift-diffusion model (DDM) explain decision-making based on a unified analysis of both accuracy and response times. This review provides an in-depth account of the recent advances in DDM research which ground different DDM parameters on several brain areas, including the cortex and basal ganglia. Furthermore, we discuss the changes in DDM parameters due to structural and functional impairments in several clinical disorders, including Parkinson's disease, Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorders, Obsessive-Compulsive Disorder (OCD), and schizophrenia. This review thus uses DDM to provide a theoretical understanding of different brain disorders.

Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

When exposed to complementary features of information across sensory modalities, our brains formulate cross-modal associations between features of stimuli presented separately to multiple modalities. For example, auditory pitch-visual size associations map high-pitch tones with small-size visual objects, and low-pitch tones with large-size visual objects. Preferential, or congruent, cross-modal associations have been shown to affect behavioural performance, i.e. choice accuracy and reaction time (RT) across multisensory decision-making paradigms. However, the neural mechanisms underpinning such influences in perceptual decision formation remain unclear. Here, we sought to identify when perceptual improvements from associative congruency emerge in the brain during decision formation. In particular, we asked whether such improvements represent ‘early’ sensory processing benefits, or ‘late’ post-sensory changes in decision dynamics. Using a modified version of the Implicit Association Test (IAT), coupled with electroencephalography (EEG), we measured the neural activity underlying the effect of auditory stimulus-driven pitch-size associations on perceptual decision formation. Behavioural results showed that participants responded significantly faster during trials when auditory pitch was congruent, rather than incongruent, with its associative visual size counterpart. We used multivariate Linear Discriminant Analysis (LDA) to characterise the spatiotemporal dynamics of EEG activity underpinning IAT performance. We found an ‘Early’ component (∼100–110 ms post-stimulus onset) coinciding with the time of maximal discrimination of the auditory stimuli, and a ‘Late’ component (∼330–340 ms post-stimulus onset) underlying IAT performance. To characterise the functional role of these components in decision formation, we incorporated a neurally-informed Hierarchical Drift Diffusion Model (HDDM), revealing that the Late component decreases response caution, requiring less sensory evidence to be accumulated, whereas the Early component increased the duration of sensory-encoding processes for incongruent trials. Overall, our results provide a mechanistic insight into the contribution of ‘early’ sensory processing, as well as ‘late’ post-sensory neural representations of associative congruency to perceptual decision formation.

A temporal hierarchical feedforward model explains both the time and the accuracy of object recognition

Brain can recognize different objects as ones it has previously experienced. The recognition accuracy and its processing time depend on different stimulus properties such as the viewing conditions, the noise levels, etc. Recognition accuracy can be explained well by different models. However, most models paid no attention to the processing time, and the ones which do, are not biologically plausible. By modifying a hierarchical spiking neural network (spiking HMAX), the input stimulus is represented temporally within the spike trains. Then, by coupling the modified spiking HMAX model, with an accumulation-to-bound decision-making model, the generated spikes are accumulated over time. The input category is determined as soon as the firing rates of accumulators reaches a threshold (decision bound). The proposed object recognition model accounts for both recognition time and accuracy. Results show that not only does the model follow human accuracy in a psychophysical task better than the well-known non-temporal models, but also it predicts human response time in each choice. Results provide enough evidence that the temporal representation of features is informative, since it can improve the accuracy of a biologically plausible decision maker over time. In addition, the decision bound is able to adjust the speed-accuracy trade-off in different object recognition tasks.

Functional heterogeneity in the left lateral posterior parietal cortex during visual and haptic crossmodal dot-surface matching

BACKGROUND

Vision and touch are thought to contribute information to object perception in an independent but complementary manner. The left lateral posterior parietal cortex (LPPC) has long been associated with multisensory information processing, and it plays an important role in visual and haptic crossmodal information retrieval. However, it remains unclear how LPPC subregions are involved in visuo-haptic crossmodal retrieval processing.

METHODS

In the present study, we used an fMRI experiment with a crossmodal delayed match-to-sample paradigm to reveal the functional role of LPPC subregions related to unimodal and crossmodal dot-surface retrieval.

RESULTS

The visual-to-haptic condition enhanced the activity of the left inferior parietal lobule relative to the haptic unimodal condition, whereas the inverse condition enhanced the activity of the left superior parietal lobule. By contrast, activation of the left intraparietal sulcus did not differ significantly between the crossmodal and unimodal conditions. Seed-based resting connectivity analysis revealed that these three left LPPC subregions engaged distinct networks, confirming their different functions in crossmodal retrieval processing.

CONCLUSION

Taken together, the findings suggest that functional heterogeneity of the left LPPC during visuo-haptic crossmodal dot-surface retrieval processing reflects that the left LPPC does not simply contribute to retrieval of past information; rather, each subregion has a specific functional role in resolving different task requirements.

Different activation signatures in the primary sensorimotor and higher-level regions for haptic three-dimensional curved surface exploration

Haptic object perception begins with continuous exploratory contact, and the human brain needs to accumulate sensory information continuously over time. However, it is still unclear how the primary sensorimotor cortex (PSC) interacts with these higher-level regions during haptic exploration over time. This functional magnetic resonance imaging (fMRI) study investigates time-dependent haptic object processing by examining brain activity during haptic 3D curve and roughness estimations. For this experiment, we designed sixteen haptic stimuli (4 kinds of curves × 4 varieties of roughness) for the haptic curve and roughness estimation tasks. Twenty participants were asked to move their right index and middle fingers along the surface twice and to estimate one of the two features-roughness or curvature-depending on the task instruction. We found that the brain activity in several higher-level regions (e.g., the bilateral posterior parietal cortex) linearly increased as the number of curves increased during the haptic exploration phase. Surprisingly, we found that the contralateral PSC was parametrically modulated by the number of curves only during the late exploration phase but not during the early exploration phase. In contrast, we found no similar parametric modulation activity patterns during the haptic roughness estimation task in either the contralateral PSC or in higher-level regions. Thus, our findings suggest that haptic 3D object perception is processed across the cortical hierarchy, whereas the contralateral PSC interacts with other higher-level regions across time in a manner that is dependent upon the features of the object.

Neural Correlates of Evidence and Urgency During Human Perceptual Decision-Making in Dynamically Changing Conditions

Current models of decision-making assume that the brain gradually accumulates evidence and drifts toward a threshold that, once crossed, results in a choice selection. These models have been especially successful in primate research; however, transposing them to human fMRI paradigms has proved it to be challenging. Here, we exploit the face-selective visual system and test whether decoded emotional facial features from multivariate fMRI signals during a dynamic perceptual decision-making task are related to the parameters of computational models of decision-making. We show that trial-by-trial variations in the pattern of neural activity in the fusiform gyrus reflect facial emotional information and modulate drift rates during deliberation. We also observed an inverse-urgency signal based in the caudate nucleus that was independent of sensory information but appeared to slow decisions, particularly when information in the task was ambiguous. Taken together, our results characterize how decision parameters from a computational model (i.e., drift rate and urgency signal) are involved in perceptual decision-making and reflected in the activity of the human brain.

Cognitive Effort Modulates Connectivity between Dorsal Anterior Cingulate Cortex and Task-Relevant Cortical Areas

Investment of cognitive effort is required in everyday life and has received ample attention in recent neurocognitive frameworks. The neural mechanism of effort investment is thought to be structured hierarchically, with dorsal anterior cingulate cortex (dACC) at the highest level, recruiting task-specific upstream areas. In the current fMRI study, we tested whether dACC is generally active when effort demand is high across tasks with different stimuli, and whether connectivity between dACC and task-specific areas is increased depending on the task requirements and effort level at hand. For that purpose, a perceptual detection task was administered that required male and female human participants to detect either a face or a house in a noisy image. Effort demand was manipulated by adding little (low effort) or much (high effort) noise to the images. Results showed a network of dACC, anterior insula (AI), and intraparietal sulcus (IPS) to be more active when effort demand was high, independent of the performed task (face or house detection). Importantly, effort demand modulated functional connectivity between dACC and face-responsive or house-responsive perceptual areas, depending on the task at hand. This shows that dACC, AI, and IPS constitute a general effort-responsive network and suggests that the neural implementation of cognitive effort involves dACC-initiated sensitization of task-relevant areas.SIGNIFICANCE STATEMENT Although cognitive effort is generally perceived as aversive, its investment is inevitable when navigating an increasingly complex society. In this study, we demonstrate how the human brain tailors the implementation of effort to the requirements of the task at hand. We show increased effort-related activity in a network of brain areas consisting of dorsal anterior cingulate cortex (dACC), anterior insula, and intraparietal sulcus, independent of task specifics. Crucially, we also show that effort-induced functional connectivity between dACC and task-relevant areas tracks specific task demands. These results demonstrate how brain regions specialized to solve a task may be energized by dACC when effort demand is high.

Computational and neural signatures of pre and post-sensory expectation bias in inferior temporal cortex

As we gather noisy sensory information from the environment, prior knowledge about the likely cause(s) of sensory input can be leveraged to facilitate perceptual judgments. Here, we investigated the computational and neural manifestation of cued expectations in human subjects as they performed a probabilistic face/house discrimination task in which face and house stimuli were preceded by informative or neutral cues. Drift-diffusion modeling of behavioral data showed that cued expectations biased both the baseline (pre-sensory) and drift-rate (post-sensory) of evidence accumulation. By employing a catch-trial functional MRI design we were able to isolate neural signatures of expectation during pre- and post-sensory stages of decision processing in face- and house-selective areas of inferior temporal cortex (ITC). Cue-evoked timecourses were modulated by cues in a manner consistent with a pre-sensory prediction signal that scaled with probability. Sensory-evoked timecourses resembled a prediction-error signal, greater in magnitude for surprising than expected stimuli. Individual differences in baseline and drift-rate biases showed a clear mapping onto pre- and post-sensory fMRI activity in ITC. These findings highlight the specificity of perceptual expectations and provide new insight into the convergence of top-down and bottom-up signals in ITC and their distinct interactions prior to and during sensory processing.

Distinct Stages of Moment-to-Moment Processing in the Cinguloopercular and Frontoparietal Networks

Control of goal-directed tasks is putatively carried out via the cinguloopercular (CO) and frontoparietal (FP) systems. However, it remains unclear whether these systems show dissociable moment-to-moment processing during distinct stages of a trial. Here, we characterize dynamics in the CO and FP networks in a meta-analysis of 5 decision-making tasks using fMRI, with a specialized "slow reveal" paradigm which allows us to measure the temporal characteristics of trial responses. We find that activations in left FP, right FP, and CO systems form separate clusters, pointing to distinct roles in decision-making. Left FP shows early "accumulator-like" responses, suggesting a role in pre-decision processing. CO has a late onset and transient response linked to the decision event, suggesting a role in performance reporting. The majority of right FP regions show late onsets with prolonged responses, suggesting a role in post-recognition processing. These findings expand upon past models, arguing that the CO and FP systems relate to distinct stages of processing within a trial. Furthermore, the findings provide evidence for a heterogeneous profile in the FP network, with left and right FP taking on specialized roles. This evidence informs our understanding of how distinct control networks may coordinate moment-to-moment components of complex actions.

Perceptual and categorical decision making: goal-relevant representation of two domains at different levels of abstraction

To date it has been unclear whether perceptual decision making and rule-based categorization reflect activation of similar cognitive processes and brain regions. On one hand, both map potentially ambiguous stimuli to a smaller set of motor responses. On the other hand, decisions about perceptual salience typically concern concrete sensory representations derived from a noisy stimulus, while categorization is typically conceptualized as an abstract decision about membership in a potentially arbitrary set. Previous work has primarily examined these types of decisions in isolation. Here we independently varied salience in both the perceptual and categorical domains in a random dot-motion framework by manipulating dot-motion coherence and motion direction relative to a category boundary, respectively. Behavioral and modeling results suggest that categorical (more abstract) information, which is more relevant to subjects' decisions, is weighted more strongly than perceptual (more concrete) information, although they also have significant interactive effects on choice. Within the brain, BOLD activity within frontal regions strongly differentiated categorical salience and weakly differentiated perceptual salience; however, the interaction between these two factors activated similar frontoparietal brain networks. Notably, explicitly evaluating feature interactions revealed a frontal-parietal dissociation: parietal activity varied strongly with both features, but frontal activity varied with the combined strength of the information that defined the motor response. Together, these data demonstrate that frontal regions are driven by decision-relevant features and argue that perceptual decisions and rule-based categorization reflect similar cognitive processes and activate similar brain networks to the extent that they define decision-relevant stimulus-response mappings.NEW & NOTEWORTHY Here we study the behavioral and neural dynamics of perceptual categorization when decision information varies in multiple domains at different levels of abstraction. Behavioral and modeling results suggest that categorical (more abstract) information is weighted more strongly than perceptual (more concrete) information but that perceptual and categorical domains interact to influence decisions. Frontoparietal brain activity during categorization flexibly represents decision-relevant features and highlights significant dissociations in frontal and parietal activity during decision making.

Sounds facilitate visual motion discrimination via the enhancement of late occipital visual representations

Sensory discriminations, such as judgements about visual motion, often benefit from multisensory evidence. Despite many reports of enhanced brain activity during multisensory conditions, it remains unclear which dynamic processes implement the multisensory benefit for an upcoming decision in the human brain. Specifically, it remains difficult to attribute perceptual benefits to specific processes, such as early sensory encoding, the transformation of sensory representations into a motor response, or to more unspecific processes such as attention. We combined an audio-visual motion discrimination task with the single-trial mapping of dynamic sensory representations in EEG activity to localize when and where multisensory congruency facilitates perceptual accuracy. Our results show that a congruent sound facilitates the encoding of motion direction in occipital sensory - as opposed to parieto-frontal - cortices, and facilitates later - as opposed to early (i.e. below 100 ms) - sensory activations. This multisensory enhancement was visible as an earlier rise of motion-sensitive activity in middle-occipital regions about 350 ms from stimulus onset, which reflected the better discriminability of motion direction from brain activity and correlated with the perceptual benefit provided by congruent multisensory information. This supports a hierarchical model of multisensory integration in which the enhancement of relevant sensory cortical representations is transformed into a more accurate choice.

•

Feature specific multisensory integration occurs in sensory not amodal cortex.

•

Feature specific integration occurs late, i.e. around 350 ms post stimulus onset.

•

Acoustic and visual representations interact in occipital motion regions.

Neural signatures of experience-based improvements in deterministic decision-making

Feedback about our choices is a crucial part of how we gather information and learn from our environment. It provides key information about decision experiences that can be used to optimize future choices. However, our understanding of the processes through which feedback translates into improved decision-making is lacking. Using neuroimaging (fMRI) and cognitive models of decision-making and learning, we examined the influence of feedback on multiple aspects of decision processes across learning. Subjects learned correct choices to a set of 50 word pairs across eight repetitions of a concurrent discrimination task. Behavioral measures were then analyzed with both a drift-diffusion model and a reinforcement learning model. Parameter values from each were then used as fMRI regressors to identify regions whose activity fluctuates with specific cognitive processes described by the models. The patterns of intersecting neural effects across models support two main inferences about the influence of feedback on decision-making. First, frontal, anterior insular, fusiform, and caudate nucleus regions behave like performance monitors, reflecting errors in performance predictions that signal the need for changes in control over decision-making. Second, temporoparietal, supplementary motor, and putamen regions behave like mnemonic storage sites, reflecting differences in learned item values that inform optimal decision choices. As information about optimal choices is accrued, these neural systems dynamically adjust, likely shifting the burden of decision processing from controlled performance monitoring to bottom-up, stimulus-driven choice selection. Collectively, the results provide a detailed perspective on the fundamental ability to use past experiences to improve future decisions.

Global Visual Motion Sensitivity: Associations with Parietal Area and Children's Mathematical Cognition

Sensitivity to global visual motion has been proposed as a signature of brain development, related to the dorsal rather than ventral cortical stream. Thresholds for global motion have been found to be elevated more than for global static form in many developmental disorders, leading to the idea of "dorsal stream vulnerability." Here we explore the association of global motion thresholds with individual differences in children's brain development, in a group of typically developing 5- to 12-year-olds. Good performance was associated with a relative increase in parietal lobe surface area, most strongly around the intraparietal sulcus and decrease in occipital area. In line with the involvement of intraparietal sulcus, areas in visuospatial and numerical cognition, we also found that global motion performance was correlated with tests of visuomotor integration and numerical skills. Individual differences in global form detection showed none of these anatomical or cognitive correlations. This suggests that the correlations with motion sensitivity are unlikely to reflect general perceptual or attentional abilities required for both form and motion. We conclude that individual developmental variations in global motion processing are not linked to greater area in the extrastriate visual areas, which initially process such motion, but in the parietal systems that make decisions based on this information. The overlap with visuospatial and numerical abilities may indicate the anatomical substrate of the "dorsal stream vulnerability" proposed as characterizing neurodevelopmental disorders.

Evidence against perfect integration of sensory information during perceptual decision making

Perceptual decision making is often modeled as perfect integration of sequential sensory samples until the accumulated total reaches a fixed decision bound. In that view, the buildup of neural activity during perceptual decision making is attributed to temporal integration. However, an alternative explanation is that sensory estimates are computed quickly with a low-pass filter and combined with a growing signal reflecting the urgency to respond and it is the latter that is primarily responsible for neural activity buildup. These models are difficult to distinguish empirically because they make similar predictions for tasks in which sensory information is constant within a trial, as in most previous studies. Here we presented subjects with a variant of the classic constant-coherence motion discrimination (CMD) task in which we inserted brief motion pulses. We examined the effect of these pulses on reaction times (RTs) in two conditions: 1) when the CMD trials were blocked and subjects responded quickly and 2) when the same CMD trials were interleaved among trials of a variable-motion coherence task that motivated slower decisions. In the blocked condition, early pulses had a strong effect on RTs but late pulses did not, consistent with both models. However, when subjects slowed their decision policy in the interleaved condition, later pulses now became effective while early pulses lost their efficacy. This last result contradicts models based on perfect integration of sensory evidence and implies that motion signals are processed with a strong leak, equivalent to a low-pass filter with a short time constant.

Categorical evidence, confidence, and urgency during probabilistic categorization

We used a temporally extended categorization task to investigate the neural substrates underlying our ability to integrate information over time and across multiple stimulus features. Using model-based fMRI, we tracked the temporal evolution of two important variables as participants deliberated about impending choices: (1) categorical evidence, and (2) confidence (the total amount of evidence provided by the stimuli, irrespective of the particular category favored). Importantly, in each model, we also included a covariate that allowed us to differentiate signals related to information accumulation from other, evidence-independent functions that increased monotonically with time (such as urgency or cognitive load). We found that somatomotor regions tracked the temporal evolution of categorical evidence, while regions in both medial and lateral prefrontal cortex, inferior parietal cortex, and the striatum tracked decision confidence. As both theory and experimental work suggest that patterns of activity thought to be related to information-accumulation may reflect, in whole or in part, an interaction between sensory evidence and urgency, we additionally investigated whether urgency might modulate the slopes of the two evidence-dependent functions. We found that the slopes of both functions were likely modulated by urgency such that the difference between the high and low evidence states increased as the response deadline loomed.

Evidence Accumulation and Choice Maintenance Are Dissociated in Human Perceptual Decision Making

Perceptual decision making in monkeys relies on decision neurons, which accumulate evidence and maintain choices until a response is given. In humans, several brain regions have been proposed to accumulate evidence, but it is unknown if these regions also maintain choices. To test if accumulator regions in humans also maintain decisions we compared delayed and self-paced responses during a face/house discrimination decision making task. Computational modeling and fMRI results revealed dissociated processes of evidence accumulation and decision maintenance, with potential accumulator activations found in the dorsomedial prefrontal cortex, right inferior frontal gyrus and bilateral insula. Potential maintenance activation spanned the frontal pole, temporal gyri, precuneus and the lateral occipital and frontal orbital cortices. Results of a quantitative reverse inference meta-analysis performed to differentiate the functions associated with the identified regions did not narrow down potential accumulation regions, but suggested that response-maintenance might rely on a verbalization of the response.