Role of inferior temporal neurons in visual memory. I. Temporal encoding of information about visual images, recalled images, and behavioral context

Visual recognition in rhesus monkeys requires area TE but not TEO

The primate visual system is often described as a hierarchical feature-conjunction pathway, whereby each level represents an increasingly complex combination of image elements, culminating in the representation of whole coherent images in anterior inferior temporal cortex. Although many models of the ventral visual stream emphasize serial feedforward processing ((Poggio T, Mutch J, Leibo J, Rosasco L, Tacchetti A. The computationalmagic of the ventral stream: sketch of a theory (and why some deep architectures work). TechRep MIT-CSAIL-TR-2012-035. MIT CSAIL, Cambridge, MA. 2012); (Yamins DLK, DiCarlo JJ. Eight open questions in the computational modeling of higher sensory cortex. Curr Opin Neurobiol. 2016:37:114-120.)), anatomical studies show connections that bypass intermediate areas and that feedback to preceding areas ((Distler C, Boussaoud D, Desimone R, Ungerleider LG. Cortical connections of inferior temporal area TEO in macaque monkeys. J Comp Neurol. 1993:334(1):125-150.); (Kravitz DJ, Saleem KS, Baker CI, Mishkin M. A new neural framework for visuospatial processing. Nat Rev Neurosci. 2011:12(4):217-230.)). Prior studies on visual discrimination and object transforms also provide evidence against a strictly feed-forward serial transfer of information between adjacent areas ((Kikuchi R, Iwai E. The locus of the posterior subdivision of the inferotemporal visual learning area in the monkey. Brain Res. 1980:198(2):347-360.); (Weiskrantz L, Saunders RC. Impairments of visual object transforms in monkeys. Brain. 1984:107(4):1033-1072.); (Kar K, DiCarlo JJ. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust Core visual object recognition. Neuron. 2021:109(1):164-176.e5.)). Thus, we sought to investigate whether behaviorally relevant propagation of visual information is as strictly sequential as sometimes supposed. We compared the accuracy of visual recognition after selective removal of specific subregions of inferior temporal cortex-area TEO, area TE, or both areas combined. Removal of TEO alone had no detectable effect on recognition memory, whereas removal of TE alone produced a large and significant impairment. Combined removal of both areas created no additional deficit relative to removal of TE alone. Thus, area TE is critical for rapid visual object recognition, and detailed image-level visual information can reach area TE via a route other than through TEO.

Contribution of Individual Features to Repetition Suppression in Macaque Inferotemporal Cortex

When an image is presented twice in succession, neurons in area TE of macaque inferotemporal cortex exhibit repetition suppression, responding less strongly to the second presentation than to the first. Suppression is known to occur if the adapter and the test image are subtly different from each other. However, it is not known whether cross-suppression occurs between images that are radically different from each other but that share a subset of features. To explore this issue, we measured repetition suppression using colored shapes. On interleaved trials, the test image might be identical to the adapter, might share its shape or color alone or might differ from it totally. At the level of the neuronal population as a whole, suppression was especially deep when adapter and test were identical, intermediate when they shared only one attribute and minimal when they shared neither attribute. At the level of the individual neuron, the degree of suppression depended not only on the properties of the two images but also on the preferences of the neuron. Suppression was deeper when the repeated color or shape was preferred by the neuron than when it was not. This effect might arise from feature-specific adaptation or alternatively from adapter-induced fatigue. Both mechanisms conform to the principle that the degree of suppression is determined by the preferences of the neuron.

The integration of visual and target signals in V4 and IT during visual object search

Searching for a specific visual object requires our brain to compare the items in view with a remembered representation of the sought target to determine whether a target match is present. This comparison is thought to be implemented, in part, via the combination of top-down modulations reflecting target identity with feed-forward visual representations. However, it remains unclear whether top-down signals are integrated at a single locus within the ventral visual pathway (e.g., V4) or at multiple stages [e.g., both V4 and inferotemporal cortex (IT)]. To investigate, we recorded neural responses in V4 and IT as rhesus monkeys performed a task that required them to identify when a target object appeared across variation in position, size, and background context. We found nonvisual, task-specific signals in both V4 and IT. To evaluate whether V4 was the only locus for the integration of top-down signals, we evaluated several feed-forward accounts of processing from V4 to IT, including a model in which IT preferentially sampled from the best V4 units and a model that allowed for nonlinear IT computation. IT task-specific modulation was not accounted for by any of these feed-forward descriptions, suggesting that during object search, top-down signals are integrated directly within IT.NEW & NOTEWORTHY To find specific objects, the brain must integrate top-down, target-specific signals with visual information about objects in view. However, the exact route of this integration in the ventral visual pathway is unclear. In the first study to systematically compare V4 and inferotemporal cortex (IT) during an invariant object search task, we demonstrate that top-down signals found in IT cannot be described as being inherited from V4 but rather must be integrated directly within IT itself.

Recency memory effects in Macaques during sequential delayed match-to-sample task with visual noise

Visual object recognition requires both visual sensory information and memory, and its mechanisms are often studied using old-world monkeys. Wittig et al. (2014, 2016) reported that Rhesus monkeys and humans seem to adopt different strategies in a short-term visual memory task. The Rhesus monkeys seemed to rely on recency of stimulus repetition, whereas humans relied on specific memorization. We conducted experiments using a sequential delayed match-to-sample task with random dot visual noise using Rhesus and Japanese monkeys and found that recency effect was observed in both species. There were differences in the noise effect on behavioral performances across species.

Inferotemporal cortex multiplexes behaviorally-relevant target match signals and visual representations in a manner that minimizes their interference

Finding a sought visual target object requires combining visual information about a scene with a remembered representation of the target to create a "target match" signal that indicates when a target is in view. Target match signals have been reported to exist within high-level visual brain areas including inferotemporal cortex (IT), where they are mixed with representations of image and object identity. However, these signals are not well understood, particularly in the context of the real-world challenge that the objects we search for typically appear at different positions, sizes, and within different background contexts. To investigate these signals, we recorded neural responses in IT as two rhesus monkeys performed a delayed-match-to-sample object search task in which target objects could appear at a variety of identity-preserving transformations. Consistent with the existence of behaviorally-relevant target match signals in IT, we found that IT contained a linearly separable target match representation that reflected behavioral confusions on trials in which the monkeys made errors. Additionally, target match signals were highly distributed across the IT population, and while a small fraction of units reflected target match signals as target match suppression, most units reflected target match signals as target match enhancement. Finally, we found that the potentially detrimental impact of target match signals on visual representations was mitigated by target match modulation that was approximately (albeit imperfectly) multiplicative. Together, these results support the existence of a robust, behaviorally-relevant target match representation in IT that is configured to minimally interfere with IT visual representations.

Altered regional homogeneity in patients with late monocular blindness: a resting-state functional MRI study

Many previous studies have demonstrated that the blindness patients have has functional and anatomical abnormalities in the visual and other vision-related cortex. However, changes in the brain function in late monocular blindness (MB) at rest are largely unknown. In this study, we investigated the underlying regional homogeneity (ReHo) of brain-activity abnormalities in patients with late MB and their relationship with clinical features. A total of 32 patients with MB (25 male and seven female) and 32 healthy controls (HCs) (25 male and seven female) closely matched in age, sex, and education underwent resting-state functional MRI scans. The ReHo method was used to assess local features of spontaneous brain activities. Patients with MB were distinguishable from HCs using the receiver operating characteristic curve. The relationship between the mean ReHo in brain regions and the behavioral performance was calculated using correlation analysis. Compared with HCs, patients with MB showed significantly decreased ReHo values in the right rectal gyrus, right cuneus, right anterior cingulate, and right lateral occipital cortex and increased ReHo values in the right inferior temporal gyrus, right frontal middle orbital, left posterior cingulate/precuneus, and left middle frontal gyrus. However, there was no significant relationship between the different mean ReHo values in the brain regions and the clinical features. Late MB involves abnormalities of the visual cortex and other vision-related brain regions, which may reflect brain dysfunction in these regions.

Dynamic representation of partially occluded objects in primate prefrontal and visual cortex

Successful recognition of partially occluded objects is presumed to involve dynamic interactions between brain areas responsible for vision and cognition, but neurophysiological evidence for the involvement of feedback signals is lacking. Here, we demonstrate that neurons in the ventrolateral prefrontal cortex (vlPFC) of monkeys performing a shape discrimination task respond more strongly to occluded than unoccluded stimuli. In contrast, neurons in visual area V4 respond more strongly to unoccluded stimuli. Analyses of V4 response dynamics reveal that many neurons exhibit two transient response peaks, the second of which emerges after vlPFC response onset and displays stronger selectivity for occluded shapes. We replicate these findings using a model of V4/vlPFC interactions in which occlusion-sensitive vlPFC neurons feed back to shape-selective V4 neurons, thereby enhancing V4 responses and selectivity to occluded shapes. These results reveal how signals from frontal and visual cortex could interact to facilitate object recognition under occlusion.

Information Accumulation over Time in Monkey Inferior Temporal Cortex Neurons Explains Pattern Recognition Reaction Time under Visual Noise

We recognize objects even when they are partially degraded by visual noise. We studied the relation between the amount of visual noise (5, 10, 15, 20, or 25%) degrading 8 black-and-white stimuli and stimulus identification in 2 monkeys performing a sequential delayed match-to-sample task. We measured the accuracy and speed with which matching stimuli were identified. The performance decreased slightly (errors increased) as the amount of visual noise increased for both monkeys. The performance remained above 80% correct, even with 25% noise. However, the reaction times markedly increased as the noise increased, indicating that the monkeys took progressively longer to decide what the correct response would be as the amount of visual noise increased, showing that the monkeys trade time to maintain accuracy. Thus, as time unfolds the monkeys act as if they are accumulating the information and/or testing hypotheses about whether the test stimulus is likely to be a match for the sample being held in short-term memory. We recorded responses from 13 single neurons in area TE of the 2 monkeys. We found that stimulus-selective information in the neuronal responses began accumulating when the match stimulus appeared. We found that the greater the amount of noise obscuring the test stimulus, the more slowly stimulus-related information by the 13 neurons accumulated. The noise induced slowing was about the same for both behavior and information. These data are consistent with the hypothesis that area TE neuron population carries information about stimulus identity that accumulates over time in such a manner that it progressively overcomes the signal degradation imposed by adding visual noise.

A robot-based behavioural task to quantify impairments in rapid motor decisions and actions after stroke

Background

Stroke can affect our ability to perform daily activities, although it can be difficult to identify the underlying functional impairment(s). Recent theories highlight the importance of sensory feedback in selecting future motor actions. This selection process can involve multiple processes to achieve a behavioural goal, including selective attention, feature/object recognition, and movement inhibition. These functions are often impaired after stroke, but existing clinical measures tend to explore these processes in isolation and without time constraints. We sought to characterize patterns of post-stroke impairments in a dynamic situation where individuals must identify and select spatial targets rapidly in a motor task engaging both arms. Impairments in generating rapid motor decisions and actions could guide functional rehabilitation targets, and identify potential of individuals to perform daily activities such as driving.

Methods

Subjects were assessed in a robotic exoskeleton. Subjects used virtual paddles attached to their hands to hit away 200 virtual target objects falling towards them while avoiding 100 virtual distractors. The inclusion of distractor objects required subjects to rapidly assess objects located across the workspace and make motor decisions about which objects to hit.

Results

As many as 78 % of the 157 subjects with subacute stroke had impairments in individual global, spatial, temporal, or hand-specific task parameters relative to the 95 % performance bounds for 309 non-disabled control subjects. Subjects with stroke and neglect (Behavioural Inattention Test score <130; n = 28) were more often impaired in task parameters than other subjects with stroke. Approximately half of subjects with stroke hit proportionally more distractor objects than 95 % of controls, suggesting they had difficulty in attending to and selecting appropriate objects. This impairment was observed for affected and unaffected limbs including some whose motor performance was comparable to controls. The proportion of distractors hit also significantly correlated with the Montreal Cognitive Assessment scores for subjects with stroke (r_s < = − 0.48, P < 10⁻⁹).

Conclusions

A simple robot-based task identified that many subjects with stroke have impairments in the rapid selection and generation of motor responses to task specific spatial goals in the workspace.

A neural substrate for object permanence in monkey inferotemporal cortex

We take it for granted that objects continue to exist after being occluded. This knowledge – known as object permanence – is present even in childhood, but its neural basis is not fully understood. Here, we show that monkey inferior temporal (IT) neurons carry potential signals of object permanence even in animals that received no explicit behavioral training. We compared two conditions with identical visual stimulation: the same object emerged from behind an occluder as expected following its occlusion, or unexpectedly after occlusion of a different object. Some neurons produced a larger (surprise) signal when the object emerged unexpectedly, whereas other neurons produced a larger (match) signal when the object reappeared as expected. Neurons carrying match signals also reinstated selective delay period activity just before the object emerged. Thus, signals related to object permanence are present in IT neurons and may arise through an interplay of memory and match computations.

Alterations in regional shape on ipsilateral and contralateral cortex contrast in children with unilateral cerebral palsy and are predictive of multiple outcomes

Congenital brain lesions result in a wide range of cerebral tissue alterations observed in children with cerebral palsy (CP) that are associated with a range of functional impairments. The relationship between injury severity and functional outcomes, however, remains poorly understood. This research investigates the differences in cortical shape between children with congenital brain lesions and typically developing children (TDC) and investigates the correlations between cortical shape and functional outcome in a large cohort of patients diagnosed with unilateral CP. Using 139 structural magnetic resonance images, including 95 patients with clinically diagnosed CP and 44 TDC, cortical segmentations were obtained using a modified expectation maximization algorithm. Three shape characteristics (cortical thickness, curvature, and sulcal depth) were computed within a number of cortical regions. Significant differences in these shape measures compared to the TDC were observed on both the injured hemisphere of children with CP (P < 0.004), as well as on the apparently uninjured hemisphere, illustrating potential compensatory mechanisms in these children. Furthermore, these shape measures were significantly correlated with several functional outcomes, including motor, cognition, vision, and communication (P < 0.012), with three out of these four models performing well on test set validation. This study highlights that cortical neuroplastic effects may be quantified using MR imaging, allowing morphological changes to be studied longitudinally, including any influence of treatment. Ultimately, such approaches could be used for the long term prediction of outcomes and the tailoring of treatment to individuals. Hum Brain Mapp 37:3588-3603, 2016. © 2016 Wiley Periodicals, Inc.

Auditory short-term memory in the primate auditory cortex

Sounds are fleeting, and assembling the sequence of inputs at the ear into a coherent percept requires auditory memory across various time scales. Auditory short-term memory comprises at least two components: an active ׳working memory' bolstered by rehearsal, and a sensory trace that may be passively retained. Working memory relies on representations recalled from long-term memory, and their rehearsal may require phonological mechanisms unique to humans. The sensory component, passive short-term memory (pSTM), is tractable to study in nonhuman primates, whose brain architecture and behavioral repertoire are comparable to our own. This review discusses recent advances in the behavioral and neurophysiological study of auditory memory with a focus on single-unit recordings from macaque monkeys performing delayed-match-to-sample (DMS) tasks. Monkeys appear to employ pSTM to solve these tasks, as evidenced by the impact of interfering stimuli on memory performance. In several regards, pSTM in monkeys resembles pitch memory in humans, and may engage similar neural mechanisms. Neural correlates of DMS performance have been observed throughout the auditory and prefrontal cortex, defining a network of areas supporting auditory STM with parallels to that supporting visual STM. These correlates include persistent neural firing, or a suppression of firing, during the delay period of the memory task, as well as suppression or (less commonly) enhancement of sensory responses when a sound is repeated as a ׳match' stimulus. Auditory STM is supported by a distributed temporo-frontal network in which sensitivity to stimulus history is an intrinsic feature of auditory processing. This article is part of a Special Issue entitled SI: Auditory working memory.

Finding and Not Finding Rat Perirhinal Neuronal Responses to Novelty

There is much evidence that the perirhinal cortex of both rats and monkeys is important for judging the relative familiarity of visual stimuli. In monkeys many studies have found that a proportion of perirhinal neurons respond more to novel than familiar stimuli. There are fewer studies of perirhinal neuronal responses in rats, and those studies based on exploration of objects, have raised into question the encoding of stimulus familiarity by rat perirhinal neurons. For this reason, recordings of single neuronal activity were made from the perirhinal cortex of rats so as to compare responsiveness to novel and familiar stimuli in two different behavioral situations. The first situation was based upon that used in “paired viewing” experiments that have established rat perirhinal differences in immediate early gene expression for novel and familiar visual stimuli displayed on computer monitors. The second situation was similar to that used in the spontaneous object recognition test that has been widely used to establish the involvement of rat perirhinal cortex in familiarity discrimination. In the first condition 30 (25%) of 120 perirhinal neurons were visually responsive; of these responsive neurons 19 (63%) responded significantly differently to novel and familiar stimuli. In the second condition eight (53%) of 15 perirhinal neurons changed activity significantly in the vicinity of objects (had “object fields”); however, for none (0%) of these was there a significant activity change related to the familiarity of an object, an incidence significantly lower than for the first condition. Possible reasons for the difference are discussed. It is argued that the failure to find recognition‐related neuronal responses while exploring objects is related to its detectability by the measures used, rather than the absence of all such signals in perirhinal cortex. Indeed, as shown by the results, such signals are found when a different methodology is used. © 2016 The Authors Hippocampus Published by Wiley Periodicals, Inc.

Dynamic target match signals in perirhinal cortex can be explained by instantaneous computations that act on dynamic input from inferotemporal cortex

Finding sought objects requires the brain to combine visual and target signals to determine when a target is in view. To investigate how the brain implements these computations, we recorded neural responses in inferotemporal cortex (IT) and perirhinal cortex (PRH) as macaque monkeys performed a delayed-match-to-sample target search task. Our data suggest that visual and target signals were combined within or before IT in the ventral visual pathway and then passed onto PRH, where they were reformatted into a more explicit target match signal over ∼10-15 ms. Accounting for these dynamics in PRH did not require proposing dynamic computations within PRH itself but, rather, could be attributed to instantaneous PRH computations performed upon an input representation from IT that changed with time. We found that the dynamics of the IT representation arose from two commonly observed features: individual IT neurons whose response preferences were not simply rescaled with time and variable response latencies across the population. Our results demonstrate that these types of time-varying responses have important consequences for downstream computation and suggest that dynamic representations can arise within a feedforward framework as a consequence of instantaneous computations performed upon time-varying inputs.

Neural correlates of auditory short-term memory in rostral superior temporal cortex

BACKGROUND

Auditory short-term memory (STM) in the monkey is less robust than visual STM and may depend on a retained sensory trace, which is likely to reside in the higher-order cortical areas of the auditory ventral stream.

RESULTS

We recorded from the rostral superior temporal cortex as monkeys performed serial auditory delayed match-to-sample (DMS). A subset of neurons exhibited modulations of their firing rate during the delay between sounds, during the sensory response, or during both. This distributed subpopulation carried a predominantly sensory signal modulated by the mnemonic context of the stimulus. Excitatory and suppressive effects on match responses were dissociable in their timing and in their resistance to sounds intervening between the sample and match.

CONCLUSIONS

Like the monkeys' behavioral performance, these neuronal effects differ from those reported in the same species during visual DMS, suggesting different neural mechanisms for retaining dynamic sounds and static images in STM.

In search of a recognition memory engram

•

The role of the perirhinal cortex in familiarity discrimination is reviewed.

•

Behavioural, pharmacological and electrophysiological evidence is considered.

•

The cortex is found to be essential for memory acquisition, retrieval and storage.

•

The evidence indicates that perirhinal synaptic weakening is critically involved.

A large body of data from human and animal studies using psychological, recording, imaging, and lesion techniques indicates that recognition memory involves at least two separable processes: familiarity discrimination and recollection. Familiarity discrimination for individual visual stimuli seems to be effected by a system centred on the perirhinal cortex of the temporal lobe. The fundamental change that encodes prior occurrence within the perirhinal cortex is a reduction in the responses of neurones when a stimulus is repeated. Neuronal network modelling indicates that a system based on such a change in responsiveness is potentially highly efficient in information theoretic terms. A review is given of findings indicating that perirhinal cortex acts as a storage site for recognition memory of objects and that such storage depends upon processes producing synaptic weakening.

The role of visual area V4 in the discrimination of partially occluded shapes

The primate brain successfully recognizes objects, even when they are partially occluded. To begin to elucidate the neural substrates of this perceptual capacity, we measured the responses of shape-selective neurons in visual area V4 while monkeys discriminated pairs of shapes under varying degrees of occlusion. We found that neuronal shape selectivity always decreased with increasing occlusion level, with some neurons being notably more robust to occlusion than others. The responses of neurons that maintained their selectivity across a wider range of occlusion levels were often sufficiently sensitive to support behavioral performance. Many of these same neurons were distinctively selective for the curvature of local boundary features and their shape tuning was well fit by a model of boundary curvature (curvature-tuned neurons). A significant subset of V4 neurons also signaled the animal's upcoming behavioral choices; these decision signals had short onset latencies that emerged progressively later for higher occlusion levels. The time course of the decision signals in V4 paralleled that of shape selectivity in curvature-tuned neurons: shape selectivity in curvature-tuned neurons, but not others, emerged earlier than the decision signals. These findings provide evidence for the involvement of contour-based mechanisms in the segmentation and recognition of partially occluded objects, consistent with psychophysical theory. Furthermore, they suggest that area V4 participates in the representation of the relevant sensory signals and the generation of decision signals underlying discrimination.

On brain activity mapping: insights and lessons from Brain Decoding Project to map memory patterns in the hippocampus

The BRAIN project recently announced by the president Obama is the reflection of unrelenting human quest for cracking the brain code, the patterns of neuronal activity that define who we are and what we are. While the Brain Activity Mapping proposal has rightly emphasized on the need to develop new technologies for measuring every spike from every neuron, it might be helpful to consider both the theoretical and experimental aspects that would accelerate our search for the organizing principles of the brain code. Here we share several insights and lessons from the similar proposal, namely, Brain Decoding Project that we initiated since 2007. We provide a specific example in our initial mapping of real-time memory traces from one part of the memory circuit, namely, the CA1 region of the mouse hippocampus. We show how innovative behavioral tasks and appropriate mathematical analyses of large datasets can play equally, if not more, important roles in uncovering the specific-to-general feature-coding cell assembly mechanism by which episodic memory, semantic knowledge, and imagination are generated and organized. Our own experiences suggest that the bottleneck of the Brain Project is not only at merely developing additional new technologies, but also the lack of efficient avenues to disseminate cutting edge platforms and decoding expertise to neuroscience community. Therefore, we propose that in order to harness unique insights and extensive knowledge from various investigators working in diverse neuroscience subfields, ranging from perception and emotion to memory and social behaviors, the BRAIN project should create a set of International and National Brain Decoding Centers at which cutting-edge recording technologies and expertise on analyzing large datasets analyses can be made readily available to the entire community of neuroscientists who can apply and schedule to perform cutting-edge research.

On initial Brain Activity Mapping of episodic and semantic memory code in the hippocampus

It has been widely recognized that the understanding of the brain code would require large-scale recording and decoding of brain activity patterns. In 2007 with support from Georgia Research Alliance, we have launched the Brain Decoding Project Initiative with the basic idea which is now similarly advocated by BRAIN project or Brain Activity Map proposal. As the planning of the BRAIN project is currently underway, we share our insights and lessons from our efforts in mapping real-time episodic memory traces in the hippocampus of freely behaving mice. We show that appropriate large-scale statistical methods are essential to decipher and measure real-time memory traces and neural dynamics. We also provide an example of how the carefully designed, sometime thinking-outside-the-box, behavioral paradigms can be highly instrumental to the unraveling of memory-coding cell assembly organizing principle in the hippocampus. Our observations to date have led us to conclude that the specific-to-general categorical and combinatorial feature-coding cell assembly mechanism represents an emergent property for enabling the neural networks to generate and organize not only episodic memory, but also semantic knowledge and imagination.

Signals in inferotemporal and perirhinal cortex suggest an untangling of visual target information

Finding sought visual targets requires our brains to flexibly combine working memory information about what we are looking for with visual information about what we are looking at. To investigate the neural computations involved in finding visual targets, we recorded neural responses in inferotemporal (IT) and perirhinal (PRH) cortex as macaque monkeys performed a task that required them to find targets within sequences of distractors. We found similar amounts of total task-specific information in both areas, however, information about whether a target was in view was more accessible using a linear read-out (i.e. was more “untangled”) in PRH. Consistent with the flow of information from IT to PRH, we also found that task-relevant information arrived earlier in IT. PRH responses were well-described by a functional model in which “untangling” computations in PRH reformat input from IT by combining neurons with asymmetric tuning correlations for target matches and distractors.

Working memory and decision processes in visual area v4

Recognizing and responding to a remembered stimulus requires the coordination of perception, working memory, and decision-making. To investigate the role of visual cortex in these processes, we recorded responses of single V4 neurons during performance of a delayed match-to-sample task that incorporates rapid serial visual presentation of natural images. We found that neuronal activity during the delay period after the cue but before the images depends on the identity of the remembered image and that this change persists while distractors appear. This persistent response modulation has been identified as a diagnostic criterion for putative working memory signals; our data thus suggest that working memory may involve reactivation of sensory neurons. When the remembered image reappears in the neuron’s receptive field, visually evoked responses are enhanced; this match enhancement is a diagnostic criterion for decision. One model that predicts these data is the matched filter hypothesis, which holds that during search V4 neurons change their tuning so as to match the remembered cue, and thus become detectors for that image. More generally, these results suggest that V4 neurons participate in the perceptual, working memory, and decision processes that are needed to perform memory-guided decision-making.

Category-selective phase coding in the superior temporal sulcus

Object perception and categorization can occur so rapidly that behavioral responses precede or co-occur with the firing rate changes in the object-selective neocortex. Phase coding could, in principle, support rapid representation of object categories, whereby the first spikes evoked by a stimulus would appear at different phases of an oscillation, depending on the object category. To determine whether object-selective regions of the neo-cortex demonstrate phase coding, we presented images of faces and objects to two monkeys while recording local field potentials (LFP) and single unit activity from object-selective regions in the upper bank superior temporal sulcus. Single units showed preferred phases of firing that depended on stimulus category, emerging with the initiation of spiking responses after stimulus onset. Differences in phase of firing were seen below 20 Hz and in the gamma and high-gamma frequency ranges. For all but the <20-Hz cluster, phase differences remained category-specific even when controlling for stimulus-locked activity, revealing that phase-specific firing is not a simple consequence of category-specific differences in the evoked responses of the LFP. In addition, we tested for firing rate-to-phase conversion. Category-specific differences in firing rates accounted for 30-40% of the explained variance in phase occurring at lower frequencies (<20 Hz) during the initial response, but was limited (<20% of the explained variance) in the 30- to 60-Hz frequency range, suggesting that gamma phase-of-firing effects reflect more than evoked LFP and firing rate responses. The present results are consistent with theoretical models of rapid object processing and extend previous observations of phase coding to include object-selective neocortex.

Contrast induced changes in response latency depend on stimulus specificity

Neurones in visual cortex show increasing response latency with decreasing stimulus contrast. Neurophysiological recordings from neurones in inferior temporal cortex (IT) and the superior temporal sulcus (STS), show that the increment in response latency with decreasing stimulus contrast is considerably greater in higher visual areas than that seen in primary visual cortex. This suggests that the majority of the latency change is not retinal or V1 in origin, instead each cortical processing area adds latency at low contrast. I show that, as in earlier visual areas, response latency is more strongly dependent on stimulus contrast than stimulus identity. There is large variation in the extent to which response latency increases with decreasing stimulus contrast. I show that this between cell variability is, at least in part, related to the stimulus specificity of the neurones: the increase in response latency as stimulus contrast decreases is greater for neurones that respond to few stimuli compared to neurones that respond to many stimuli.

A color-based visualization technique for multielectrode spike trains

Multi electrode recordings of neuronal activity provide an overwhelming amount of data that is often difficult to analyze and interpret. Although various methods exist for treating multielectrode datasets quantitatively, there is a particularly prominent lack of techniques that enable a quick visual exploration of such datasets. Here, by using Kohonen self-organizing maps, we propose a simple technique that allows for the representation of multiple spike trains through a sequence of color-coded population activity vectors. When multiple color sequences are grouped according to a certain criterion, e.g., by stimulation condition or recording time, one can inspect an entire dataset visually and extract quickly information about the identity, stimulus-locking and temporal distribution of multi-neuron activity patterns. Color sequences can be computed on various time scales revealing different aspects of the temporal dynamics and can emphasize high-order correlation patterns that are not detectable with pairwise techniques. Furthermore, this technique is useful for determining the stability of neuronal responses during a recording session. Due to its simplicity and reliance on perceptual grouping, the method is useful for both quick on-line visualization of incoming data and for more detailed post hoc analyses.

Neural dynamics in inferior temporal cortex during a visual working memory task

Intelligent organisms are capable of tracking objects even when they temporarily disappear from sight, a cognitive capacity commonly referred to as visual working memory (VWM). The neural basis of VWM has been the subject of significant scientific debate, with recent work focusing on the relative roles of posterior visual areas, such as the inferior temporal cortex (ITC), and the prefrontal cortex. Here we reexamined the contribution of ITC to VWM by recording from highly selective individual ITC neurons as monkeys engaged in multiple versions of an occlusion-based memory task. As expected, we found strong evidence for a role of ITC in stimulus encoding. We also found that almost half of these selective cells showed stimulus-selective delay period modulation, with a small but significant fraction exhibiting differential responses even in the presence of simultaneously visible interfering information. When we combined the informational content of multiple neurons, we found that the accuracy with which we could decode memory content increased drastically. The memory epoch analyses suggest that behaviorally relevant visual memories were reinstated in ITC. Furthermore, we observed a population-wide enhancement of neuronal response to a match stimulus compared with the same stimulus presented as a nonmatch. The single-cell enhancement preceded any match effects identified in the local field potential, leading us to speculate that enhancement is the result of neural processing local to ITC. Moreover, match enhancement was only later followed by the more commonly observed match suppression. Altogether, the data support the hypothesis that, when a stimulus is held in memory, ITC neurons are actively biased in favor of task-relevant visual representations and that this bias can immediately impact subsequent recognition events.

Short-term memory trace in rapidly adapting synapses of inferior temporal cortex

Visual short-term memory tasks depend upon both the inferior temporal cortex (ITC) and the prefrontal cortex (PFC). Activity in some neurons persists after the first (sample) stimulus is shown. This delay-period activity has been proposed as an important mechanism for working memory. In ITC neurons, intervening (nonmatching) stimuli wipe out the delay-period activity; hence, the role of ITC in memory must depend upon a different mechanism. Here, we look for a possible mechanism by contrasting memory effects in two architectonically different parts of ITC: area TE and the perirhinal cortex. We found that a large proportion (80%) of stimulus-selective neurons in area TE of macaque ITCs exhibit a memory effect during the stimulus interval. During a sequential delayed matching-to-sample task (DMS), the noise in the neuronal response to the test image was correlated with the noise in the neuronal response to the sample image. Neurons in perirhinal cortex did not show this correlation. These results led us to hypothesize that area TE contributes to short-term memory by acting as a matched filter. When the sample image appears, each TE neuron captures a static copy of its inputs by rapidly adjusting its synaptic weights to match the strength of their individual inputs. Input signals from subsequent images are multiplied by those synaptic weights, thereby computing a measure of the correlation between the past and present inputs. The total activity in area TE is sufficient to quantify the similarity between the two images. This matched filter theory provides an explanation of what is remembered, where the trace is stored, and how comparison is done across time, all without requiring delay period activity. Simulations of a matched filter model match the experimental results, suggesting that area TE neurons store a synaptic memory trace during short-term visual memory.

To know whether one is looking at an object seen a few seconds ago or not depends on visual short-term memory. To study short-term memory, we recorded single neuronal activity from two brain areas of monkeys, the TE and the perirhinal cortex of the temporal lobe, known to be important in visual pattern recognition and memory. The monkeys performed a short-term visual memory task, a sequential match-to-sample. The monkeys had to signal when a sample stimulus reappeared in a short sequence of stimuli. In area TE only, small fluctuations occurring for a sample-elicited response were correlated with the responses when a match stimulus reappeared, as if a snapshot of the sample-induced response was stored and recalled. In our modeling, we propose that each TE neuron stores and compares the signals during short-term memory by storing the response to the sample in local and rapidly adapting synapses. Subsequent stimulus-elicited responses are then automatically multiplied by the locally stored signal. Here, we show that the match can be detected when the sum of the outputs of the population of TE neurons crosses a threshold. Correlated fluctuations will be a signature this type of local memory storage wherever it occurs in the brain.

Linking cortical spike pattern codes to auditory perception

Abstract Neurometric analysis has proven to be a powerful tool for studying links between neural activity and perception, especially in visual and somatosensory cortices, but conventional neurometrics are based on a simplistic rate-coding hypothesis that is clearly at odds with the rich and complex temporal spiking patterns evoked by many natural stimuli. In this study, we investigated the possible relationships between temporal spike pattern codes in the primary auditory cortex (A1) and the perceptual detection of subtle changes in the temporal structure of a natural sound. Using a two-alternative forced-choice oddity task, we measured the ability of human listeners to detect local time reversals in a marmoset twitter call. We also recorded responses of neurons in A1 of anesthetized and awake ferrets to these stimuli, and analyzed these responses using a novel neurometric approach that is sensitive to temporal discharge patterns. We found that although spike count-based neurometrics were inadequate to account for behavioral performance on this auditory task, neurometrics based on the temporal discharge patterns of populations of A1 units closely matched the psychometric performance curve, but only if the spiking patterns were resolved at temporal resolutions of 20 msec or better. These results demonstrate that neurometric discrimination curves can be calculated for temporal spiking patterns, and they suggest that such an extension of previous spike count-based approaches is likely to be essential for understanding the neural correlates of the perception of stimuli with a complex temporal structure.

Cortical mechanisms of action selection: the affordance competition hypothesis

At every moment, the natural world presents animals with two fundamental pragmatic problems: selection between actions that are currently possible and specification of the parameters or metrics of those actions. It is commonly suggested that the brain addresses these by first constructing representations of the world on which to build knowledge and make a decision, and then by computing and executing an action plan. However, neurophysiological data argue against this serial viewpoint. In contrast, it is proposed here that the brain processes sensory information to specify, in parallel, several potential actions that are currently available. These potential actions compete against each other for further processing, while information is collected to bias this competition until a single response is selected. The hypothesis suggests that the dorsal visual system specifies actions which compete against each other within the fronto-parietal cortex, while a variety of biasing influences are provided by prefrontal regions and the basal ganglia. A computational model is described, which illustrates how this competition may take place in the cerebral cortex. Simulations of the model capture qualitative features of neurophysiological data and reproduce various behavioural phenomena.

Effects of task demands on the responses of color-selective neurons in the inferior temporal cortex

Categorization and fine discrimination are two different functions in visual perception, and we can switch between these two functions depending on the situation or task demands. To explore how visual cortical neurons behave in such situations, we recorded the activities of color-selective neurons in the inferior temporal (IT) cortex of two monkeys trained to perform a color categorization task, a color discrimination task and a simple fixation task. Many IT neurons changed their activity depending upon the task, although color selectivity was well conserved. A majority of neurons showed stronger responses during the categorization task. Moreover, for the population of IT neurons as a whole, signals contributing to performing the categorization task were enhanced. These results imply that judgment of color category by color-selective IT neurons is facilitated during the categorization task and suppressed during the discrimination task as a consequence of task-dependent modulation of their activities.

Enhancement of object representations in primate perirhinal cortex during a visual working-memory task

We compared single-cell activities in perirhinal cortex (PRh) as well as adjacent visual cortex (area TE) across two tasks. One task required the monkey to identify any stimulus repetition within a sequence of object stimuli. In the other task, the same stimuli were presented, but the monkey didn't have to remember them. PRh responses during the object-memory task were elevated relative to those during the second task. In TE, on the other hand, there were no significant task-related differences in responses. We did not observe task-related differences related to repetition effects in either brain area. The onset of the enhanced signal in PRh during the object-memory task occurred with a latency of 80 ms after the onset of the stimulus response, suggesting that it was the result of top-down feedback.

Organizing principles of real-time memory encoding: neural clique assemblies and universal neural codes

Recent identification of network-level coding units, termed neural cliques, in the hippocampus has enabled real-time patterns of memory traces to be mathematically described, directly visualized, and dynamically deciphered. These memory coding units are functionally organized in a categorical and hierarchical manner, suggesting that internal representations of external events in the brain is achieved not by recording exact details of those events, but rather by recreating its own selective pictures based on cognitive importance. This neural-clique-based hierarchical-extraction and parallel-binding process enables the brain to acquire not only large storage capacity but also abstraction and generalization capability. In addition, activation patterns of the neural clique assemblies can be converted to strings of binary codes that would permit universal categorizations of internal brain representations across individuals and species.

Prefrontal activity during serial probe reproduction task: encoding, mnemonic, and retrieval processes

To study the prefrontal neuronal mechanism for the encoding and mnemonic processing of multiple objects, the order of object presentation, and the retrieval of an object among objects in the working memory, we recorded neuronal activity from the lateral prefrontal cortex while two monkeys performed the serial probe reproduction task. In the task, two objects (C1 and C2) were presented sequentially interleaved with a delay (D1) period, and after the second delay (D2) period, a color cue was presented. Monkeys were trained to select one target object on the basis of the color stimulus. During the C1 and C2 periods, we found responses that depended on the order of presentation (order-selective response). During the D1 and/or D2 periods, two-thirds of the neurons with object-selective delay-period activity showed order-selective activity coding either C1 or C2. Neurons with larger response magnitudes during the C2 period showed order-selective delay-period activity during the D2 period. These order-selective responses during the C2 period could also contribute to order-selective delay-period activity, and order-selective delay-period activity during the D1 and D2 periods could play an essential role in storing information on both the object and the temporal order of presentation. During the color cue period, two-thirds of the neurons with responses showed target object selectivity (CT and T responses), although the target object was not presented during this period. The CT and T responses could play a critical role in the retrieval of an item among various items in the working memory.

Neural correlates of executive control in the avian brain

Executive control, the ability to plan one's behaviour to achieve a goal, is a hallmark of frontal lobe function in humans and other primates. In the current study we report neural correlates of executive control in the avian nidopallium caudolaterale, a region analogous to the mammalian prefrontal cortex. Homing pigeons (Columba livia) performed a working memory task in which cues instructed them whether stimuli should be remembered or forgotten. When instructed to remember, many neurons showed sustained activation throughout the memory period. When instructed to forget, the sustained activation was abolished. Consistent with the neural data, the behavioural data showed that memory performance was high after instructions to remember, and dropped to chance after instructions to forget. Our findings indicate that neurons in the avian nidopallium caudolaterale participate in one of the core forms of executive control, the control of what should be remembered and what should be forgotten. This form of executive control is fundamental not only to working memory, but also to all cognition.

Effect of visual noise on pattern recognition

We recognize objects even when they are partially degraded by visual noise. Using monkeys performing a sequential delayed match-to-sample task, we studied the relation between the amount of visual noise (5, 10, 15, 20 or 25%) degrading the eight black and white stimuli used here, and the accuracy and speed with which matching stimuli were identified. The correct response rate decreased slightly as the amount of visual noise increased for both monkeys. Even at the 25% noise level, the correct response rate was more than 80%, indicating that the monkeys can recognize the pattern they are trying to match when the pattern is masked with visual noise. In contrast, the reaction time to the match stimulus increased substantially as the amount of visual noise increased. Thus, the monkeys appear to be trading time to maintain accuracy, suggesting that the monkeys are accumulating information and/or testing hypotheses about whether the test stimulus is likely to be a match for the sample being held in short-term memory.

Relationship between stimulus complexity and neuronal activity in the inferotemporal cortex of the macaque monkey

The single-unit activity of 217 cells was recorded from the inferotemporal cortex (IT) of two awake macaque monkeys while they performed a fixation task. The stimuli were coloured geometrical shapes or coloured representations of natural or artificial objects. To determine whether the stimuli could be separated into groups on the basis on neuronal population behaviour, the responses to the images were analysed by factor analysis and cluster analysis. It was a common result of each analysis that, on the basis of neuronal responses, the stimulus set could be separated into two groups, despite the lack of difference in mean response rate to them. Similar groups were formed when only the first half of the responses was analysed. The results suggest a differential coding of the images of simple geometrical shapes and of the images of complex, real (photographic) objects. We found significant differences between the two stimulus groups in physical features, other than size or luminance. Our results suggest that the same neurone population might respond differently to simple and complex images in the first 150 ms of their responses. The differences might be attributed to "non-obvious" physical features of the stimuli, such as the amount of internal lines in the images, colourfulness and the length of perimeter of the stimuli.

An information-processing analysis of the functional architecture of the primate neocortex

Working at the systems level of analysis, we will use the term functional architecture to concern what processing components exist, how they are interconnected, and what information-processing functions each is involved in. In this paper, experimental evidence for the primate neocortex is analysed for conclusions concerning the existence of neural areas, for corticocortical connectivity among neural areas, and for the involvement of each cortical neural area in the functioning of the brain. We characterize the information-processing function for each neural area in terms of the types of information it is associated with, and conceive of its activity as processing, storage and transmission of data of the corresponding types for that area. We also adapt concepts of goal, plan, sequence, event and context for the description of information processing in the neocortex. This analysis shows that the primate neocortex consists in the main of a perception hierarchy, an action hierarchy and connections between them. In other words, from an information-processing point of view, the primate neocortex has a hierarchical perception-action architecture.

Short-term memory and perceptual decision for three-dimensional visual features in the caudal intraparietal sulcus (Area CIP)

The purpose of the present study was to examine whether neurons in the caudolateral part of the intraparietal sulcus (area CIP), a part of the posterior parietal cortex, contribute to short-term memory and perceptual decision of three-dimensional (3D) surface orientation, in addition to its purely visual nature of responding selectively to 3D surface orientation. Activities of CIP neurons were recorded while monkeys performed a modified delayed matching-to-sample (DMTS) task using stereoscopic stimuli. Seventy-seven neurons were examined with a routine of the DMTS task, and 94% (72 of 77) of them showed selectivity to surface orientation. Furthermore, 82% (63 of 77) of the examined neurons showed sustained activity during delay, and 60% (38 of 63) of them showed selective delay activity depending on the sample stimulus, suggesting that they contribute to short-term memory of 3D visual features. On the other hand, 53% (41 of 77) of the examined neurons showed modulation of visual response depending on whether a stimulus appeared as a sample, match, or nonmatch stimulus (contextual modulation). The majority (73%, 30 of 41) of these neurons with contextual modulation showed activity change depending on whether the test stimuli did or did not match the sample stimuli (match-nonmatch modulation), suggesting their involvement in matching, or perceptual decision, concerning 3D visual features. These findings suggest that CIP neurons play important roles not only in the perception of 3D visual features but also in cognitive functions such as short-term memory and perceptual decision of 3D visual information.

The temporal resolution of neural codes: does response latency have a unique role?

This article reviews the nature of the neural code in non-human primate cortex and assesses the potential for neurons to carry two or more signals simultaneously. Neurophysiological recordings from visual and motor systems indicate that the evidence for a role for precisely timed spikes relative to other spike times (ca. 1-10 ms resolution) is inconclusive. This indicates that the visual system does not carry a signal that identifies whether the responses were elicited when the stimulus was attended or not. Simulations show that the absence of such a signal reduces, but does not eliminate, the increased discrimination between stimuli that are attended compared with when the stimuli are unattended. The increased accuracy asymptotes with increased gain control, indicating limited benefit from increasing attention. The absence of a signal identifying the attentional state under which stimuli were viewed can produce the greatest discrimination between attended and unattended stimuli. Furthermore, the greatest reduction in discrimination errors occurs for a limited range of gain control, again indicating that attention effects are limited. By contrast to precisely timed patterns of spikes where the timing is relative to other spikes, response latency provides a fine temporal resolution signal (ca. 10 ms resolution) that carries information that is unavailable from coarse temporal response measures. Changes in response latency and changes in response magnitude can give rise to different predictions for the patterns of reaction times. The predictions are verified, and it is shown that the standard method for distinguishing executive and slave processes is only valid if the representations of interest, as evidenced by the neural code, are known. Overall, the data indicate that the signalling evident in neural signals is restricted to the spike count and the precise times of spikes relative to stimulus onset (response latency). These coding issues have implications for our understanding of cognitive models of attention and the roles of executive and slave systems.

Evidence concerning how neurons of the perirhinal cortex may effect familiarity discrimination

Many studies indicate that recognition memory involves at least two separable processes, familiarity discrimination and recollection. Aspects of what is known of potential neuronal substrates of familiarity discrimination are reviewed. Lesion studies have established that familiarity discrimination for individual visual stimuli is effected by a system centred on the perirhinal cortex of the temporal lobe. The fundamental change that encodes prior occurrence of such stimuli appears to be a reduction in the response of neurons in anterior inferior temporal (including perirhinal) cortex when a stimulus is repeated. The neuronal responses rapidly signal the presence of a novel stimulus, and are evidence of long-lasting learning after a single exposure. Computational modelling indicates that a neuronal network based on such a change in responsiveness is potentially highly efficient in information theoretic terms. Processes that occur in long-term depression within the perirhinal cortex provide candidate synaptic plastic mechanisms for that underlying the change, but such linkage remains to be experimentally established.

Weakly modulated spike trains: significance, precision, and correction for sample size

Many single-unit electrophysiological studies of visual cortex have investigated strong evoked responses to simple stimuli such as oriented gratings. Experiments involving other types of stimuli, such as natural scenes, higher-order features, and surface brightness, produce single-unit responses that are more difficult to interpret. Experiments with brightness, in particular, evoke single-unit responses that are typically weakly modulated. When the brightness is generated by a visual illusion such as the Cornsweet illusion, statistical tests are often necessary to distinguish true responses from baseline fluctuations. Here, using data collected from cat Areas 17 and 18 in response to real and illusory brightness stimuli, we provide a method for detecting and quantifying weak but significant periodic responses. By randomizing spike trains (via bootstrap methods), we provide confidence levels for response significance, permitting the evaluation of both weak and strong responses. We show that because of a strong dependence on total spike number, response significance can only be appropriately determined with randomized spike trains of similar spike number. Such randomizations can be performed for both stimulus-elicited and spontaneously occurring spike trains. By developing a method for generating randomized modulated spike trains (phase-restricted randomization) from actual recordings, we calculate upper and lower confidence limits of modulated spike trains and describe how measurement precision varies as a function of total spike count. Finally, using this randomization method, we describe how a correction function can be determined to correct for measurement bias introduced at low spike counts. These methods may also be useful in the study of small but potentially significant responses in other systems.

Information loss in an optimal maximum likelihood decoding

The mutual information between a set of stimuli and the elicited neural responses is compared to the corresponding decoded information. The decoding procedure is presented as an artificial distortion of the joint probabilities between stimuli and responses. The information loss is quantified. Whenever the probabilities are only slightly distorted, the information loss is shown to be quadratic in the distortion.

Consistency of encoding in monkey visual cortex

Are different kinds of stimuli (for example, different classes of geometric images or naturalistic images) encoded differently by visual cortex, or are the principles of encoding the same for all stimuli? We examine two response properties: (1) the range of spike counts that can be elicited from a neuron in epochs representative of short periods of fixation (up to 400 msec), and (2) the relation between mean and variance of spike counts elicited by different stimuli, that together characterize the information processing capabilities of a neuron using the spike count code. In monkey primary visual cortex (V1) complex cells, we examine responses elicited by static stimuli of four kinds (photographic images, bars, gratings, and Walsh patterns); in area TE of inferior temporal cortex, we examine responses elicited by static stimuli in the sample, nonmatch, and match phases of a delayed match-to-sample task. In each area, the ranges of mean spike counts and the relation between mean and variance of spike counts elicited are sufficiently similar across experimental conditions that information transmission is unaffected by the differences across stimulus set or behavioral conditions [although in 10 of 27 (37%) of the V1 neurons there are statistically significant but small differences, the median difference in transmitted information for these neurons was 0.9%]. Encoding therefore appears to be consistent across experimental conditions for neurons in both V1 and TE, and downstream neurons could decode all incoming signals using a single set of rules.

Representational capacity of a set of independent neurons

The capacity with which a system of independent neuron-like units represents a given set of stimuli is studied by calculating the mutual information between the stimuli and the neural responses. Both discrete noiseless and continuous noisy neurons are analyzed. In both cases, the information grows monotonically with the number of neurons considered. Under the assumption that neurons are independent, the mutual information rises linearly from zero, and approaches exponentially its maximum value. We find the dependence of the initial slope on the number of stimuli and on the sparseness of the representation.

Clustering of perirhinal neurons with similar properties following visual experience in adult monkeys

The functional organization of early visual areas seems to be largely determined during development. However, the organization of areas important for learning and memory, such as perirhinal cortex, may be modifiable in adults. To test this hypothesis, we recorded from pairs of neurons in perirhinal cortex of macaques while they viewed multiple complex stimuli. For novel stimuli, neuronal response preferences for pairs of nearby neurons and far-apart neurons were uncorrelated. However, after one day of experience with the stimuli, response preferences of nearby neurons became more similar. We conclude that specific visual experience induces development of clusters of perirhinal neurons with similar stimulus preferences.