Investigating silent pauses in connected speech: integrating linguistic, neuropsychological, and neuroanatomical perspectives across narrative tasks in post-stroke aphasia

Introduction

Silent pauses are regarded as integral components of the temporal organization of speech. However, it has also been hypothesized that they serve as markers for internal cognitive processes, including word access, monitoring, planning, and memory functions. Although existing evidence across various pathological populations underscores the importance of investigating silent pauses' characteristics, particularly in terms of frequency and duration, there is a scarcity of data within the domain of post-stroke aphasia.

Methods

The primary objective of the present study is to scrutinize the frequency and duration of silent pauses in two distinct narrative tasks within a cohort of 32 patients with chronic post-stroke aphasia, in comparison with a control group of healthy speakers. Subsequently, we investigate potential correlation patterns between silent pause measures, i.e., frequency and duration, across the two narrative tasks within the patient group, their performance in neuropsychological assessments, and lesion data.

Results

Our findings showed that patients exhibited a higher frequency of longer-duration pauses in both narrative tasks compared to healthy speakers. Furthermore, within-group comparisons revealed that patients tended to pause more frequently and for longer durations in the picture description task, while healthy participants exhibited the opposite trend. With regard to our second research question, a marginally significant interaction emerged between performance in semantic verbal fluency and the narrative task, in relation to the location of silent pauses-whether between or within clauses-predicting the duration of silent pauses in the patient group. However, no significant results were observed for the frequency of silent pauses. Lastly, our study identified that the duration of silent pauses could be predicted by distinct Regions of Interest (ROIs) in spared tissue within the left hemisphere, as a function of the narrative task.

Discussion

Overall, this study follows an integrative approach of linguistic, neuropsychological and neuroanatomical data to define silent pauses in connected speech, and illustrates interrelations between cognitive components, temporal aspects of speech, and anatomical indices, while it further highlights the importance of studying connected speech indices using different narrative tasks.

Color discrimination in fixed saturation level of patients with acute traumatic injury

Introduction

Traumatic brain injury (TBI) is an important public health concern and that may lead to severe neural sequels, such as color vision deficits.

Methods

We evaluated the color vision of 10 TBI patients with normal cognitive function using a color discrimination test in a fixed saturation level. We also analyzed computerized tomography scans to identify the local of the brain damages.

Results

Four TBI patients that had lesions in brain areas of the ventral visual streams, five TBI patients had lesions inferred in brain areas of the dorsal visual stream, and one TBI patient had lesion in the occipital area. All the patients had cognitive and color vision screened and they had characterized the chromatic discrimination at high and low saturation. All participants had no significant cognitive impairment in the moment of the color vision test. Additionally, they had perfect performance for discrimination of chromatic stimulus at high saturation and similar to controls (n = 37 age-matched participants). Three of four TBI patients with lesions in the ventral brain and one patient with lesion in the occipital area had impairment of the chromatic discrimination at low saturation. All TBI patients with lesions in the dorsal brain had performance similar or slightly worse than the controls.

Conclusion

Chromatic discrimination at low saturation was associated to visual damage in the ventral region of the brain and is a potential tool for functional evaluation of brain damage in TBI patients.

Dissection of the Temporofrontal Extreme Capsule Fasciculus Using Diffusion MRI Tractography and Association with Lexical Retrieval

The well-known arcuate fasciculus that connects the posterior superior temporal region with the language production region in the ventrolateral frontal cortex constitutes the classic peri-Sylvian dorsal stream of language. A second temporofrontal white matter tract connects ventrally the anterior to intermediate lateral temporal cortex with frontal areas via the extreme capsule. This temporofrontal extreme capsule fasciculus (TFexcF) constitutes the ventral stream of language processing. The precise origin, course, and termination of this pathway has been examined in invasive tract tracing studies in macaque monkeys, but there have been no standard protocols for its reconstruction in the human brain using diffusion imaging tractography. Here we provide a protocol for the dissection of the TFexcF in vivo in the human brain using diffusion magnetic resonance imaging (MRI) tractography which provides a solid basis for exploring its functional role. A key finding of the current dissection protocol is the demonstration that the TFexcF is left hemisphere lateralized. Furthermore, using the present dissection protocol, we demonstrate that the TFexcF is related to lexical retrieval scores measured with the category fluency test, in contrast to the classical arcuate fasciculus (the dorsal language pathway) that was also dissected and was related to sentence repetition.

Differential functional reorganization of ventral and dorsal visual pathways following childhood hemispherectomy

Hemispherectomy is a surgical procedure in which an entire hemisphere of a patient's brain is resected or functionally disconnected to manage seizures in individuals with drug-resistant epilepsy. Despite the extensive loss of input from both ventral and dorsal visual pathways of one hemisphere, pediatric patients who have undergone hemispherectomy show a remarkably high degree of perceptual function across many domains. In the current study, we sought to understand the extent to which functions of the ventral and dorsal visual pathways reorganize to the contralateral hemisphere following childhood hemispherectomy. To this end, we collected fMRI data from an equal number of left and right hemispherectomy patients who completed tasks that typically elicit lateralized responses from the ventral or the dorsal pathway, namely, word (left ventral), face (right ventral), tool (left dorsal), and global form (right dorsal) perception. Overall, there was greater evidence of functional reorganization in the ventral pathway than in the dorsal pathway. Importantly, because ventral and dorsal reorganization was tested in the very same patients, these results cannot be explained by idiosyncratic factors such as disease etiology, age at the time of surgery, or age at testing. These findings suggest that because the dorsal pathway may mature earlier, it may have a shorter developmental window of plasticity than the ventral pathway and, hence, be less malleable.

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues

Convolutional Neural Networks (CNN) are a class of machine learning models predominately used in computer vision tasks and can achieve human-like performance through learning from experience. Their striking similarities to the structural and functional principles of the primate visual system allow for comparisons between these artificial networks and their biological counterparts, enabling exploration of how visual functions and neural representations may emerge in the real brain from a limited set of computational principles. After considering the basic features of CNNs, we discuss the opportunities and challenges of endorsing CNNs as in silico models of the primate visual system. Specifically, we highlight several emerging notions about the anatomical and physiological properties of the visual system that still need to be systematically integrated into current CNN models. These tenets include the implementation of parallel processing pathways from the early stages of retinal input and the reconsideration of several assumptions concerning the serial progression of information flow. We suggest design choices and architectural constraints that could facilitate a closer alignment with biology provide causal evidence of the predictive link between the artificial and biological visual systems. Adopting this principled perspective could potentially lead to new research questions and applications of CNNs beyond modeling object recognition.

Temporal continuity shapes visual responses of macaque face patch neurons

Macaque inferior temporal cortex neurons respond selectively to complex visual images, with recent work showing that they are also entrained reliably by the evolving content of natural movies. To what extent does temporal continuity itself shape the responses of high-level visual neurons? We addressed this question by measuring how cells in face-selective regions of the macaque visual cortex were affected by the manipulation of a movie's temporal structure. Sampling a 5-min movie at 1 s intervals, we measured neural responses to randomized, brief stimuli of different lengths, ranging from 800 ms dynamic movie snippets to 100 ms static frames. We found that the disruption of temporal continuity strongly altered neural response profiles, particularly in the early response period after stimulus onset. The results suggest that models of visual system function based on discrete and randomized visual presentations may not translate well to the brain's natural modes of operation.

White matter connections of high-level visual areas predict cytoarchitecture better than category-selectivity in childhood, but not adulthood

Ventral temporal cortex (VTC) consists of high-level visual regions that are arranged in consistent anatomical locations across individuals. This consistency has led to several hypotheses about the factors that constrain the functional organization of VTC. A prevailing theory is that white matter connections influence the organization of VTC, however, the nature of this constraint is unclear. Here, we test 2 hypotheses: (1) white matter tracts are specific for each category or (2) white matter tracts are specific to cytoarchitectonic areas of VTC. To test these hypotheses, we used diffusion magnetic resonance imaging to identify white matter tracts and functional magnetic resonance imaging to identify category-selective regions in VTC in children and adults. We find that in childhood, white matter connections are linked to cytoarchitecture rather than category-selectivity. In adulthood, however, white matter connections are linked to both cytoarchitecture and category-selectivity. These results suggest a rethinking of the view that category-selective regions in VTC have category-specific white matter connections early in development. Instead, these findings suggest that the neural hardware underlying the processing of categorical stimuli may be more domain-general than previously thought, particularly in childhood.

Temporal asymmetries and interactions between dorsal and ventral visual pathways during object recognition

Despite their anatomical and functional distinctions, there is growing evidence that the dorsal and ventral visual pathways interact to support object recognition. However, the exact nature of these interactions remains poorly understood. Is the presence of identity-relevant object information in the dorsal pathway simply a byproduct of ventral input? Or, might the dorsal pathway be a source of input to the ventral pathway for object recognition? In the current study, we used high-density EEG-a technique with high temporal precision and spatial resolution sufficient to distinguish parietal and temporal lobes-to characterise the dynamics of dorsal and ventral pathways during object viewing. Using multivariate analyses, we found that category decoding in the dorsal pathway preceded that in the ventral pathway. Importantly, the dorsal pathway predicted the multivariate responses of the ventral pathway in a time-dependent manner, rather than the other way around. Together, these findings suggest that the dorsal pathway is a critical source of input to the ventral pathway for object recognition.

Structural and functional connectivity of the inferior temporal numeral area

A growing body of evidence suggests that in adults, there is a spatially consistent "inferior temporal numeral area" (ITNA) in the occipitotemporal cortex that appears to preferentially process Arabic digits relative to non-numerical symbols and objects. However, very little is known about why the ITNA is spatially segregated from regions that process other orthographic stimuli such as letters, and why it is spatially consistent across individuals. In the present study, we used diffusion-weighted imaging and functional magnetic resonance imaging to contrast structural and functional connectivity between left and right hemisphere ITNAs and a left hemisphere letter-preferring region. We found that the left ITNA had stronger structural and functional connectivity than the letter region to inferior parietal regions involved in numerical magnitude representation and arithmetic. Between hemispheres, the left ITNA showed stronger structural connectivity with the left inferior frontal gyrus (Broca's area), while the right ITNA showed stronger structural connectivity to the ipsilateral inferior parietal cortex and stronger functional coupling with the bilateral IPS. Based on their relative connectivity, our results suggest that the left ITNA may be more readily involved in mapping digits to verbal number representations, while the right ITNA may support the mapping of digits to quantity representations.

Does the brain's ventral visual pathway compute object shape?

A rich behavioral literature has shown that human object recognition is supported by a representation of shape that is tolerant to variations in an object's appearance. Such 'global' shape representations are achieved by describing objects via the spatial arrangement of their local features, or structure, rather than by the appearance of the features themselves. However, accumulating evidence suggests that the ventral visual pathway - the primary substrate underlying object recognition - may not represent global shape. Instead, ventral representations may be better described as a basis set of local image features. We suggest that this evidence forces a reevaluation of the role of the ventral pathway in object perception and posits a broader network for shape perception that encompasses contributions from the dorsal pathway.

High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks

Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance. The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model’s performance in high-level visual areas. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas.

The Dorsal Visual Pathway Represents Object-Centered Spatial Relations for Object Recognition

Although there is mounting evidence that input from the dorsal visual pathway is crucial for object processes in the ventral pathway, the specific functional contributions of dorsal cortex to these processes remain poorly understood. Here, we hypothesized that dorsal cortex computes the spatial relations among an object's parts, a process crucial for forming global shape percepts, and transmits this information to the ventral pathway to support object categorization. Using fMRI with human participants (females and males), we discovered regions in the intraparietal sulcus (IPS) that were selectively involved in computing object-centered part relations. These regions exhibited task-dependent functional and effective connectivity with ventral cortex, and were distinct from other dorsal regions, such as those representing allocentric relations, 3D shape, and tools. In a subsequent experiment, we found that the multivariate response of posterior (p)IPS, defined on the basis of part-relations, could be used to decode object category at levels comparable to ventral object regions. Moreover, mediation and multivariate effective connectivity analyses further suggested that IPS may account for representations of part relations in the ventral pathway. Together, our results highlight specific contributions of the dorsal visual pathway to object recognition. We suggest that dorsal cortex is a crucial source of input to the ventral pathway and may support the ability to categorize objects on the basis of global shape.SIGNIFICANCE STATEMENT Humans categorize novel objects rapidly and effortlessly. Such categorization is achieved by representing an object's global shape structure, that is, the relations among object parts. Yet, despite their importance, it is unclear how part relations are represented neurally. Here, we hypothesized that object-centered part relations may be computed by the dorsal visual pathway, which is typically implicated in visuospatial processing. Using fMRI, we identified regions selective for the part relations in dorsal cortex. We found that these regions can support object categorization, and even mediate representations of part relations in the ventral pathway, the region typically thought to support object categorization. Together, these findings shed light on the broader network of brain regions that support object categorization.

Representation of Color, Form, and their Conjunction across the Human Ventral Visual Pathway

Despite decades of research, our understanding of the relationship between color and form processing in the primate ventral visual pathway remains incomplete. Using fMRI multivoxel pattern analysis, we examined coding of color and form, using a simple form feature (orientation) and a mid-level form feature (curvature), in human ventral visual processing regions. We found that both color and form could be decoded from activity in early visual areas V1 to V4, as well as in the posterior color-selective region and shape-selective regions in ventral and lateral occipitotemporal cortex defined based on their univariate selectivity to color or shape, respectively (the central color region only showed color but not form decoding). Meanwhile, decoding biases towards one feature or the other existed in the color- and shape-selective regions, consistent with their univariate feature selectivity reported in past studies. Additional extensive analyses show that while all these regions contain independent (linearly additive) coding for both features, several early visual regions also encode the conjunction of color and the simple, but not the complex, form feature in a nonlinear, interactive manner. Taken together, the results show that color and form are encoded in a biased distributed and largely independent manner across ventral visual regions in the human brain.

Dorsal and Ventral Stream Function in Children With Developmental Coordination Disorder

Dorsal stream cortical networks underpin a cluster of visuomotor, visuospatial, and visual attention functions. Sensitivity to global coherence of motion and static form is considered a signature of visual cortical processing in the dorsal stream (motion) relative to the ventral stream (form). Poorer sensitivity to global motion compared to global static form has been found across a diverse range of neurodevelopmental disorders, suggesting a “dorsal stream vulnerability.” However, previous studies of global coherence sensitivity in Developmental Coordination Disorder (DCD) have shown conflicting findings. We examined two groups totalling 102 children with DCD (age 5–12 years), using the “Ball in the Grass” psychophysical test to compare sensitivity to global motion and global static form. Motor impairment was measured using the Movement-ABC (M-ABC). Global coherence sensitivity was compared with a typically developing control group (N = 69) in the same age range. Children with DCD showed impaired sensitivity to global motion (p = 0.002), but not global form (p = 0.695), compared to controls. Within the DCD group, motor impairment showed a significant linear relationship with global form sensitivity (p < 0.001). There was also a significant quadratic relationship between motor impairment and global motion sensitivity (p = 0.046), where poorer global motion sensitivity was only apparent with greater motor impairment. We suggest that two distinct visually related components, associated with global form and global motion sensitivity, contribute to DCD differentially over the range of severity of the disorder. Possible neural circuitry underlying these relationships is discussed.

Multiple Adjoining Word- and Face-Selective Regions in Ventral Temporal Cortex Exhibit Distinct Dynamics

The map of category-selectivity in human ventral temporal cortex (VTC) provides organizational constraints to models of object recognition. One important principle is lateral-medial response biases to stimuli that are typically viewed in the center or periphery of the visual field. However, little is known about the relative temporal dynamics and location of regions that respond preferentially to stimulus classes that are centrally viewed, such as the face- and word-processing networks. Here, word- and face-selective regions within VTC were mapped using intracranial recordings from 36 patients. Partially overlapping, but also anatomically dissociable patches of face- and word-selectivity, were found in VTC. In addition to canonical word-selective regions along the left posterior occipitotemporal sulcus, selectivity was also located medial and anterior to face-selective regions on the fusiform gyrus at the group level and within individual male and female subjects. These regions were replicated using 7 Tesla fMRI in healthy subjects. Left hemisphere word-selective regions preceded right hemisphere responses by 125 ms, potentially reflecting the left hemisphere bias for language, with no hemispheric difference in face-selective response latency. Word-selective regions along the posterior fusiform responded first, then spread medially and laterally, then anteriorally. Face-selective responses were first seen in posterior fusiform regions bilaterally, then proceeded anteriorally from there. For both words and faces, the relative delay between regions was longer than would be predicted by purely feedforward models of visual processing. The distinct time courses of responses across these regions, and between hemispheres, suggest that a complex and dynamic functional circuit supports face and word perception.SIGNIFICANCE STATEMENT Representations of visual objects in the human brain have been shown to be organized by several principles, including whether those objects tend to be viewed centrally or peripherally in the visual field. However, it remains unclear how regions that process objects that are viewed centrally, such as words and faces, are organized relative to one another. Here, invasive and noninvasive neuroimaging suggests that there is a mosaic of regions in ventral temporal cortex that respond selectively to either words or faces. These regions display differences in the strength and timing of their responses, both within and between brain hemispheres, suggesting that they play different roles in perception. These results illuminate extended, bilateral, and dynamic brain pathways that support face perception and reading.

Inferior Occipital Gyrus Is Organized along Common Gradients of Spatial and Face-Part Selectivity

The ventral visual stream of the human brain is subdivided into patches with categorical stimulus preferences, like faces or scenes. However, the functional organization within these areas is less clear. Here, we used functional magnetic resonance imaging and vertex-wise tuning models to independently probe spatial and face-part preferences in the inferior occipital gyrus (IOG) of healthy adult males and females. The majority of responses were well explained by Gaussian population tuning curves for both retinotopic location and the preferred relative position within a face. Parameter maps revealed a common gradient of spatial and face-part selectivity, with the width of tuning curves drastically increasing from posterior to anterior IOG. Tuning peaks clustered more idiosyncratically but were also correlated across maps of visual and face space. Preferences for the upper visual field went along with significantly increased coverage of the upper half of the face, matching recently discovered biases in human perception. Our findings reveal a broad range of neural face-part selectivity in IOG, ranging from narrow to "holistic." IOG is functionally organized along this gradient, which in turn is correlated with retinotopy.SIGNIFICANCE STATEMENT Brain imaging has revealed a lot about the large-scale organization of the human brain and visual system. For example, occipital cortex contains map-like representations of the visual field, while neurons in ventral areas cluster into patches with categorical preferences, like faces or scenes. Much less is known about the functional organization within these areas. Here, we focused on a well established face-preferring area-the inferior occipital gyrus (IOG). A novel neuroimaging paradigm allowed us to map the retinotopic and face-part tuning of many recording sites in IOG independently. We found a steep posterior-anterior gradient of decreasing face-part selectivity, which correlated with retinotopy. This suggests the functional role of ventral areas is not uniform and may follow retinotopic "protomaps."

Behavioural Features of Cerebral Visual Impairment Are Common in Children With Down Syndrome

It is widely recognised that children with Down syndrome have a broad range and a high prevalence of visual deficits and it has been suggested that those with Down syndrome are more likely to exhibit visual perception deficits indicative of cerebral visual impairment. This exploratory study aims to determine the prevalence of behavioural features suggestive of cerebral visual impairment (CVI) occurring with Down syndrome and whether the visual problems can be ascribed to optometric factors. A cohort of 226 families of children with Down syndrome (trisomy 21), aged 4–17, were invited to participate in a validated question inventory, to recognise visual perception issues. The clinical records of the participants were then reviewed retrospectively. A five-question screening instrument was used to indicate suspected CVI. The majority of the 81 families who responded to the questionnaire reported some level of visual perceptual difficulty in their child. Among this cohort, the prevalence of suspected CVI as indicated by the screening questionnaire was 38%. Only ametropia was found to have a significant association with suspected CVI, although this increased the correct prediction of suspected CVI outcome by only a small amount. Results suggest that children with Down syndrome are more likely to experience problems consistent with cerebral visual impairment, and that these may originate from a similar brain dysfunction to that which contributes to high levels of ametropia and failure to emmetropise. It is important that behavioural features of CVI are recognised in children with Down syndrome, further investigations initiated and appropriate management applied.

Pulvinar Modulates Contrast Responses in the Visual Cortex as a Function of Cortical Hierarchy

The pulvinar is the largest extrageniculate visual nucleus in mammals. Given its extensive reciprocal connectivity with the visual cortex, it allows the cortico-thalamocortical transfer of visual information. Nonetheless, knowledge of the nature of the pulvinar inputs to the cortex remains elusive. We investigated the impact of silencing the pulvinar on the contrast response function of neurons in 2 distinct hierarchical cortical areas in the cat (areas 17 and 21a). Pulvinar inactivation altered the response gain in both areas, but with larger changes observed in area 21a. A theoretical model was proposed, simulating the pulvinar contribution to cortical contrast responses by modifying the excitation-inhibition balanced state of neurons across the cortical hierarchy. Our experimental and theoretical data showed that the pulvinar exerts a greater modulatory influence on neuronal activity in area 21a than in the primary visual cortex, indicating that the pulvinar impact on cortical visual neurons varies along the cortical hierarchy.

Rational and Flexible Adaptation of Sentence Production to Ongoing Language Experience

Whether sentences are formulated primarily using lexically based or non-lexically based information has been much debated. In this perspective article, I review evidence for rational flexibility in the sentence production architecture. Sentences can be constructed flexibly via lexically dependent or independent routes, and rationally depending on the statistical properties of the input and the validity of lexical vs. abstract cues for predicting sentence structure. Different neural pathways appear to be recruited for individuals with different executive function abilities and for verbs with different statistical properties, suggesting that alternative routes are available for producing the same structure. Together, extant evidence indicates that the human brain adapts to ongoing language experience during adulthood, and that the nature of the adjustment may depend rationally on the statistical contingencies of the current context.

Within- and between-hemifield generalization of repetition suppression in inferior temporal cortex

The decrease in response with stimulus repetition is a common property observed in many sensory brain areas. This repetition suppression (RS) is ubiquitous in neurons of macaque inferior temporal (IT) cortex, the end-stage of the ventral visual pathway. The neural mechanisms of RS in IT are still unclear, and one possibility is that it is inherited from areas upstream to IT that show also RS. Since neurons in IT have larger receptive fields compared with earlier visual areas, we examined the inheritance hypothesis by presenting adapter and test stimuli at widely different spatial locations along both vertical and horizontal meridians and across hemifields. RS was present for distances between adapter and test stimuli up to 22° and when the two stimuli were presented in different hemifields. Also, we examined the position tolerance of the stimulus selectivity of adaptation by comparing the responses to a test stimulus following the same (repetition trial) or a different (alternation trial) adapter at a position different from the test stimulus. Stimulus-selective adaptation was still present and consistently stronger in the later phase of the response for distances up to 18°. Finally, we observed stimulus-selective adaptation in repetition trials even without a measurable excitatory response to the adapter stimulus. To accommodate these and previous data, we propose that at least part of the stimulus-selective adaptation in IT is based on short-term plasticity mechanisms within IT and/or reflects top-down activity from areas downstream to IT.NEW & NOTEWORTHY Neurons in inferior temporal cortex reduce their response when stimuli are repeated. To assess whether this repetition suppression is inherited from upstream visual areas, we examined the extent of its spatial generalization. We observed stimulus-selective adaptation when adapter and test stimuli were presented at widely different spatial positions and in different hemifields. These data suggest that at least part of the repetition suppression originates within inferior temporal cortex and/or reflects feedback from downstream areas.

Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence

A potentially organizing goal of the brain and cognitive sciences is to accurately explain domains of human intelligence as executable, neurally mechanistic models. Years of research have led to models that capture experimental results in individual behavioral tasks and individual brain regions. We here advocate for taking the next step: integrating experimental results from many laboratories into suites of benchmarks that, when considered together, push mechanistic models toward explaining entire domains of intelligence, such as vision, language, and motor control. Given recent successes of neurally mechanistic models and the surging availability of neural, anatomical, and behavioral data, we believe that now is the time to create integrative benchmarking platforms that incentivize ambitious, unified models. This perspective discusses the advantages and the challenges of this approach and proposes specific steps to achieve this goal in the domain of visual intelligence with the case study of an integrative benchmarking platform called Brain-Score.

Grip Constancy but Not Perceptual Size Constancy Survives Lesions of Early Visual Cortex

Object constancies are central constructs in theories of visual phenomenology. A powerful example is "size constancy," in which the perceived size of an object remains stable despite changes in viewing distance [1-4]. Evidence from neuropsychology [5], neuroimaging [6-11], transcranial magnetic stimulation [12, 13], single-unit and lesion studies in monkey [14-20], and computational modeling [21] suggests that re-entrant processes involving reciprocal interactions between primary visual cortex (V1) and extrastriate visual areas [22-26] play an essential role in mediating size constancy. It is seldom appreciated, however, that object constancies must also operate for the visual guidance of goal-directed action. For example, when reaching out to pick up an object, the hand's in-flight aperture scales with size of the goal object [27-30] and is refractory to the decrease in retinal-image size with increased viewing distance [31-41] (Figure 1), a phenomenon we call "grip constancy." Does grip constancy, like perceptual constancy, depend on V1 or can it be mediated by pathways that bypass it altogether? We tested these possibilities in an individual, M.C., who has bilateral lesions encompassing V1 and much of the ventral visual stream. We show that her perceptual estimates of object size co-vary with retinal-image size rather than real-world size as viewing distance varies. In contrast, M.C. shows near-normal scaling of in-flight grasp aperture to object size despite changes in viewing distance. Thus, although early visual cortex is necessary for perceptual object constancy, it is unnecessary for grip constancy, which is mediated instead by separate visual inputs to dorsal-stream visuomotor areas [42-48].

Pulvinar Modulates Synchrony across Visual Cortical Areas

The cortical visual hierarchy communicates in different oscillatory ranges. While gamma waves influence the feedforward processing, alpha oscillations travel in the feedback direction. Little is known how this oscillatory cortical communication depends on an alternative route that involves the pulvinar nucleus of the thalamus. We investigated whether the oscillatory coupling between the primary visual cortex (area 17) and area 21a depends on the transthalamic pathway involving the pulvinar in cats. To that end, visual evoked responses were recorded in areas 17 and 21a before, during and after inactivation of the pulvinar. Local field potentials were analyzed with Wavelet and Granger causality tools to determine the oscillatory coupling between layers. The results indicate that cortical oscillatory activity was enhanced during pulvinar inactivation, in particular for area 21a. In area 17, alpha band responses were represented in layers II/III. In area 21a, gamma oscillations, except for layer I, were significantly increased, especially in layer IV. Granger causality showed that the pulvinar modulated the oscillatory information between areas 17 and 21a in gamma and alpha bands for the feedforward and feedback processing, respectively. Together, these findings indicate that the pulvinar is involved in the mechanisms underlying oscillatory communication along the visual cortex.

Relating Visual Production and Recognition of Objects in Human Visual Cortex

Drawing is a powerful tool that can be used to convey rich perceptual information about objects in the world. What are the neural mechanisms that enable us to produce a recognizable drawing of an object, and how does this visual production experience influence how this object is represented in the brain? Here we evaluate the hypothesis that producing and recognizing an object recruit a shared neural representation, such that repeatedly drawing the object can enhance its perceptual discriminability in the brain. We scanned human participants (N = 31; 11 male) using fMRI across three phases of a training study: during training, participants repeatedly drew two objects in an alternating sequence on an MR-compatible tablet; before and after training, they viewed these and two other control objects, allowing us to measure the neural representation of each object in visual cortex. We found that: (1) stimulus-evoked representations of objects in visual cortex are recruited during visually cued production of drawings of these objects, even throughout the period when the object cue is no longer present; (2) the object currently being drawn is prioritized in visual cortex during drawing production, while other repeatedly drawn objects are suppressed; and (3) patterns of connectivity between regions in occipital and parietal cortex supported enhanced decoding of the currently drawn object across the training phase, suggesting a potential neural substrate for learning how to transform perceptual representations into representational actions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.SIGNIFICANCE STATEMENT Humans can produce simple line drawings that capture rich information about their perceptual experiences. However, the mechanisms that support this behavior are not well understood. Here we investigate how regions in visual cortex participate in the recognition of an object and the production of a drawing of it. We find that these regions carry diagnostic information about an object in a similar format both during recognition and production, and that practice drawing an object enhances transmission of information about it to downstream regions. Together, our study provides novel insight into the functional relationship between visual production and recognition in the brain.

Longitudinal progression of posterior cortical atrophy over 11 years: Relationship between lesion topology and clinical deficits

Posterior cortical atrophy (PCA) is a rare form of dementia primarily characterized by slowly progressing deterioration of visual processing corresponding to atrophy in the posterior parietal and occipital cortices with less prominent memory loss than are usually seen in other forms of dementia such as Alzheimer's Disease (AD). In the present case report, we describe longitudinal data over a period of 11 years regarding clinical and neuropsychological impairments and their relation to the location and extent of cortical changes related to higher order visual processing in a patient with posterior cortical atrophy. In our patient, visual processing deficits concerning space, motion and object perception emerged at the age of 50 and continued to worsen. By the age of 58, while the perception of contrast, color and figure-ground separation appeared undisturbed the patient exhibited pronounced dorsal- and ventral-related visual deficits, which continued to worsen with age. The patient's MRI scans over the course of the disease revealed increasing circumscribed and bilateral atrophy of the parietal and occipital cortices, with a right-sided predominance. The specific localization of cortical atrophy, the slow progression characterized by visual processing deficits and relatively preserved memory were the main criteria for the diagnosis of posterior cortical atrophy. The case report also highlights the importance of an early extensive neurological and neuropsychological evaluation of visual deficits that occur without the presence of ophthalmological disease.

The effect of mild traumatic brain injury on the visual processing of global form and motion

Cortical visual processing involves the ventral stream (form perception) and the dorsal stream (motion perception). We assessed whether mild traumatic brain injury (TBI) differentially affects these two streams. Eleven adults with mild TBI (28 ± 9 yrs, 17 ± 5 months post injury) and 25 controls (25 ± 5 yrs) participated. Participants completed tests of global processing involving Glass patterns (form) and random dot kinematograms (motion), measurement of contrast thresholds for motion direction discrimination, a comprehensive vision screening and the Post-Concussion Symptom Inventory (PCSI). Our results showed that the mild TBI group had significantly higher (worse) global form (mean ± SD: TBI 25 ± 6%, control 21 ± 5%) and motion (TBI 14 ± 7%, control 11 ± 3%) coherence thresholds than controls. The magnitude of the mild TBI group deficit did not differ between the two tasks. Contrast thresholds for motion direction discrimination did not differ between the groups, but were positively correlated with PCSI score (r² = 0.51. p = 0.01) in the mild TBI group. The mild TBI group had worse outcomes than controls for all clinical measurements of vision except distance visual acuity. In conclusion, mild TBI affects processing in both the dorsal and ventral cortical processing streams equally. In addition, spatiotemporal contrast sensitivity may be related to the symptoms of mild TBI.

Domain-Specific Diaschisis: Lesions to Parietal Action Areas Modulate Neural Responses to Tools in the Ventral Stream

Neural responses to small manipulable objects ("tools") in high-level visual areas in ventral temporal cortex (VTC) provide an opportunity to test how anatomically remote regions modulate ventral stream processing in a domain-specific manner. Prior patient studies indicate that grasp-relevant information can be computed about objects by dorsal stream structures independently of processing in VTC. Prior functional neuroimaging studies indicate privileged functional connectivity between regions of VTC exhibiting tool preferences and regions of parietal cortex supporting object-directed action. Here we test whether lesions to parietal cortex modulate tool preferences within ventral and lateral temporal cortex. We found that lesions to the left anterior intraparietal sulcus, a region that supports hand-shaping during object grasping and manipulation, modulate tool preferences in left VTC and in the left posterior middle temporal gyrus. Control analyses demonstrated that neural responses to "place" stimuli in left VTC were unaffected by lesions to parietal cortex, indicating domain-specific consequences for ventral stream neural responses in the setting of parietal lesions. These findings provide causal evidence that neural specificity for "tools" in ventral and lateral temporal lobe areas may arise, in part, from online inputs to VTC from parietal areas that receive inputs via the dorsal visual pathway.

Gesturing tool use and tool transport actions modulates inferior parietal functional connectivity with the dorsal and ventral object processing pathways

Interacting with manipulable objects (tools) requires the integration of diverse computations supported by anatomically remote regions. Previous functional neuroimaging research has demonstrated the left supramarginal (SMG) exhibits functional connectivity to both ventral and dorsal pathways, supporting the integration of ventrally-mediated tool properties and conceptual knowledge with dorsally-computed volumetric and structural representations of tools. This architecture affords us the opportunity to test whether interactions between the left SMG, ventral visual pathway, and dorsal visual pathway are differentially modulated when participants plan and generate tool-directed gestures emphasizing functional manipulation (tool use gesturing) or structure-based grasping (tool transport gesturing). We found that functional connectivity between the left SMG, ventral temporal cortex (bilateral fusiform gyri), and dorsal visual pathway (left superior parietal lobule/posterior intraparietal sulcus) was maximal for tool transport planning and gesturing, whereas functional connectivity between the left SMG, left ventral anterior temporal lobe, and left frontal operculum was maximal for tool use planning and gesturing. These results demonstrate that functional connectivity to the left SMG is differentially modulated by tool use and tool transport gesturing, suggesting that distinct tool features computed by the two object processing pathways are integrated in the parietal lobe in the service of tool-directed action.

Nonlinear Processing of Shape Information in Rat Lateral Extrastriate Cortex

In rodents, the progression of extrastriate areas located laterally to primary visual cortex (V1) has been assigned to a putative object-processing pathway (homologous to the primate ventral stream), based on anatomical considerations. Recently, we found functional support for such attribution (Tafazoli et al., 2017), by showing that this cortical progression is specialized for coding object identity despite view changes, the hallmark property of a ventral-like pathway. Here, we sought to clarify what computations are at the base of such specialization. To this aim, we performed multielectrode recordings from V1 and laterolateral area LL (at the apex of the putative ventral-like hierarchy) of male adult rats, during the presentation of drifting gratings and noise movies. We found that the extent to which neuronal responses were entrained to the phase of the gratings sharply dropped from V1 to LL, along with the quality of the receptive fields inferred through reverse correlation. Concomitantly, the tendency of neurons to respond to different oriented gratings increased, whereas the sharpness of orientation tuning declined. Critically, these trends are consistent with the nonlinear summation of visual inputs that is expected to take place along the ventral stream, according to the predictions of hierarchical models of ventral computations and a meta-analysis of the monkey literature. This suggests an intriguing homology between the mechanisms responsible for building up shape selectivity and transformation tolerance in the visual cortex of primates and rodents, reasserting the potential of the latter as models to investigate ventral stream functions at the circuitry level.SIGNIFICANCE STATEMENT Despite the growing popularity of rodents as models of visual functions, it remains unclear whether their visual cortex contains specialized modules for processing shape information. To addresses this question, we compared how neuronal tuning evolves from rat primary visual cortex (V1) to a downstream visual cortical region (area LL) that previous work has implicated in shape processing. In our experiments, LL neurons displayed a stronger tendency to respond to drifting gratings with different orientations while maintaining a sustained response across the whole duration of the drift cycle. These trends match the increased complexity of pattern selectivity and the augmented tolerance to stimulus translation found in monkey visual temporal cortex, thus revealing a homology between shape processing in rodents and primates.

'Artiphysiology' reveals V4-like shape tuning in a deep network trained for image classification

Deep networks provide a potentially rich interconnection between neuroscientific and artificial approaches to understanding visual intelligence, but the relationship between artificial and neural representations of complex visual form has not been elucidated at the level of single-unit selectivity. Taking the approach of an electrophysiologist to characterizing single CNN units, we found many units exhibit translation-invariant boundary curvature selectivity approaching that of exemplar neurons in the primate mid-level visual area V4. For some V4-like units, particularly in middle layers, the natural images that drove them best were qualitatively consistent with selectivity for object boundaries. Our results identify a novel image-computable model for V4 boundary curvature selectivity and suggest that such a representation may begin to emerge within an artificial network trained for image categorization, even though boundary information was not provided during training. This raises the possibility that single-unit selectivity in CNNs will become a guide for understanding sensory cortex.

Neuronal correlates of motion-defined shape perception in primate dorsal and ventral streams

Human and non-human primates can readily perceive the shape of objects using visual motion. Classically, shape, and motion are considered to be separately processed via ventral and dorsal cortical pathways, respectively. However, many lines of anatomical and physiological evidence have indicated that these two pathways are likely to be interconnected at some stage. For motion-defined shape perception, these two pathways should interact with each other because the ventral pathway must utilize motion, which the dorsal pathway processes, to extract shape signal. However, it is unknown how interactions between cortical pathways are involved in neural mechanisms underlying motion-defined shape perception. We review evidence from psychophysical, lesion, neuroimaging and physiological research on motion-defined shape perception and then discuss the effects of behavioral demands on neural activity in ventral and dorsal cortical areas. Further, we discuss functions of two candidate sets of levels: early and higher-order cortical areas. The extrastriate area V4 and middle temporal (MT) area, which are reciprocally connected, at the early level are plausible areas for extracting the shape and/or constituent parts of shape from motion cues because neural dynamics are different from those during luminance-defined shape perception. On the other hand, among other higher-order visual areas, the anterior superior temporal sulcus likely contributes to the processing of cue-invariant shape recognition rather than cue-dependent shape processing. We suggest that sharing information about motion and shape between the early visual areas in the dorsal and ventral pathways is dependent on visual cues and behavioral requirements, indicating the interplay between the pathways.

The Contribution of Primary Auditory Cortex to Auditory Categorization in Behaving Monkeys

The specific contribution of core auditory cortex to auditory perception –such as categorization– remains controversial. To identify a contribution of the primary auditory cortex (A1) to perception, we recorded A1 activity while monkeys reported whether a temporal sequence of tone bursts was heard as having a “small” or “large” frequency difference. We found that A1 had frequency-tuned responses that habituated, independent of frequency content, as this auditory sequence unfolded over time. We also found that A1 firing rate was modulated by the monkeys’ reports of “small” and “large” frequency differences; this modulation correlated with their behavioral performance. These findings are consistent with the hypothesis that A1 contributes to the processes underlying auditory categorization.

The Large-Scale Organization of Object-Responsive Cortex Is Reflected in Resting-State Network Architecture

Neural responses to visually presented objects have a large-scale spatial organization across the cortex, related to the dimensions of animacy and object size. Most proposals about the origins of this organization point to the influence of differential connectivity with other cortical regions as the key organizing force that drives distinctions in object-responsive cortex. To explore this possibility, we used resting-state functional connectivity to examine the relationship between stimulus-evoked organization of objects, and distinctions in functional network architecture. Using a data-driven analysis, we found evidence for three distinct whole-brain resting-state networks that route through object-responsive cortex, and these naturally manifest the tripartite structure of the stimulus-evoked organization. However, object-responsive regions were also highly correlated with each other at rest. Together, these results point to a nested network architecture, with a local interconnected network across object-responsive cortex and distinctive subnetworks that specifically route these key object distinctions to distinct long-range regions. Broadly, these results point to the viability that long-range connections are a driving force of the large-scale organization of object-responsive cortex.

Visual and visuomotor processing of hands and tools as a case study of cross talk between the dorsal and ventral streams

A major principle of organization of the visual system is between a dorsal stream that processes visuomotor information and a ventral stream that supports object recognition. Most research has focused on dissociating processing across these two streams. Here we focus on how the two streams interact. We tested neurologically-intact and impaired participants in an object categorization task over two classes of objects that depend on processing within both streams-hands and tools. We measured how unconscious processing of images from one of these categories (e.g., tools) affects the recognition of images from the other category (i.e., hands). Our findings with neurologically-intact participants demonstrated that processing an image of a hand hampers the subsequent processing of an image of a tool, and vice versa. These results were not present in apraxic patients (N = 3). These findings suggest local and global inhibitory processes working in tandem to co-register information across the two streams.

Hierarchical stimulus processing in rodent primary and lateral visual cortex as assessed through neuronal selectivity and repetition suppression

Similar to primates, visual cortex in rodents appears to be organized in two distinct hierarchical streams. However, there is still little known about how visual information is processed along those streams in rodents. In this study, we examined how repetition suppression and position and clutter tolerance of the neuronal representations evolve along the putative ventral visual stream in rats. To address this question, we recorded multiunit spiking activity in primary visual cortex (V1) and the more downstream visual laterointermediate (LI) area of head-restrained Long-Evans rats. We employed a paradigm reminiscent of the continuous carry-over design used in human neuroimaging. In both areas, stimulus repetition attenuated the early phase of the neuronal response to the repeated stimulus, with this response suppression being greater in area LI. Furthermore, stimulus preferences were more similar across positions (position tolerance) in area LI than in V1, even though the absolute responses in both areas were very sensitive to changes in position. In contrast, the neuronal representations in both areas were equally good at tolerating the presence of limited visual clutter, as modeled by the presentation of a single flank stimulus. When probing tolerance of the neuronal representations with stimulus-specific adaptation, we detected no position tolerance in either examined brain area, whereas, on the contrary, we revealed clutter tolerance in both areas. Overall, our data demonstrate similarities and discrepancies in processing of visual information along the ventral visual stream of rodents and primates. Moreover, our results stress caution in using neuronal adaptation to probe tolerance of the neuronal representations. NEW & NOTEWORTHY Rodents are emerging as a popular animal model that complement primates for studying higher level visual functions. Similar to findings in primates, we demonstrate a greater repetition suppression and position tolerance of the neuronal representations in the downstream laterointermediate area of Long-Evans rats compared with primary visual cortex. However, we report no difference in the degree of clutter tolerance between the areas. These findings provide additional evidence for hierarchical processing of visual stimuli in rodents.

Task Context Overrules Object- and Category-Related Representational Content in the Human Parietal Cortex

The dorsal, parietal visual stream is activated when seeing objects, but the exact nature of parietal object representations is still under discussion. Here we test 2 specific hypotheses. First, parietal cortex is biased to host some representations more than others, with a different bias compared with ventral areas. A prime example would be object action representations. Second, parietal cortex forms a general multiple-demand network with frontal areas, showing similar task effects and representational content compared with frontal areas. To differentiate between these hypotheses, we implemented a human neuroimaging study with a stimulus set that dissociates associated object action from object category while manipulating task context to be either action- or category-related. Representations in parietal as well as prefrontal areas represented task-relevant object properties (action representations in the action task), with no sign of the irrelevant object property (category representations in the action task). In contrast, irrelevant object properties were represented in ventral areas. These findings emphasize that human parietal cortex does not preferentially represent particular object properties irrespective of task, but together with frontal areas is part of a multiple-demand and content-rich cortical network representing task-relevant object properties.

Language Learning Variability within the Dorsal and Ventral Streams as a Cue for Compensatory Mechanisms in Aphasia Recovery

Dorsal and ventral pathways connecting perisylvian language areas have been shown to be functionally and anatomically segregated. Whereas the dorsal pathway integrates the sensory-motor information required for verbal repetition, the ventral pathway has classically been associated with semantic processes. The great individual differences characterizing language learning through life partly correlate with brain structure and function within these dorsal and ventral language networks. Variability and plasticity within these networks also underlie inter-individual differences in the recovery of linguistic abilities in aphasia. Despite the division of labor of the dorsal and ventral streams, studies in healthy individuals have shown how the interaction of them and the redundancy in the areas they connect allow for compensatory strategies in functions that are usually segregated. In this mini-review we highlight the need to examine compensatory mechanisms between streams in healthy individuals as a helpful guide to choosing the most appropriate rehabilitation strategies, using spared functions and targeting preserved compensatory networks for brain plasticity.

Multiple Transmitter Receptors in Regions and Layers of the Human Cerebral Cortex

We measured the densities (fmol/mg protein) of 15 different receptors of various transmitter systems in the supragranular, granular and infragranular strata of 44 areas of visual, somatosensory, auditory and multimodal association systems of the human cerebral cortex. Receptor densities were obtained after labeling of the receptors using quantitative in vitro receptor autoradiography in human postmortem brains. The mean density of each receptor type over all cortical layers and of each of the three major strata varies between cortical regions. In a single cortical area, the multi-receptor fingerprints of its strata (i.e., polar plots, each visualizing the densities of multiple different receptor types in supragranular, granular or infragranular layers of the same cortical area) differ in shape and size indicating regional and laminar specific balances between the receptors. Furthermore, the three strata are clearly segregated into well definable clusters by their receptor fingerprints. Fingerprints of different cortical areas systematically vary between functional networks, and with the hierarchical levels within sensory systems. Primary sensory areas are clearly separated from all other cortical areas particularly by their very high muscarinic M₂ and nicotinic α₄β₂ receptor densities, and to a lesser degree also by noradrenergic α₂ and serotonergic 5-HT₂ receptors. Early visual areas of the dorsal and ventral streams are segregated by their multi-receptor fingerprints. The results are discussed on the background of functional segregation, cortical hierarchies, microstructural types, and the horizontal (layers) and vertical (columns) organization in the cerebral cortex. We conclude that a cortical column is composed of segments, which can be assigned to the cortical strata. The segments differ by their patterns of multi-receptor balances, indicating different layer-specific signal processing mechanisms. Additionally, the differences between the strata-and area-specific fingerprints of the 44 areas reflect the segregation of the cerebral cortex into functionally and topographically definable groups of cortical areas (visual, auditory, somatosensory, limbic, motor), and reveals their hierarchical position (primary and unimodal (early) sensory to higher sensory and finally to multimodal association areas).

Highlights

Densities of transmitter receptors vary between areas of human cerebral cortex.

Multi-receptor fingerprints segregate cortical layers.

The densities of all examined receptor types together reach highest values in the supragranular stratum of all areas.

The lowest values are found in the infragranular stratum.

Multi-receptor fingerprints of entire areas and their layers segregate functional systems

Cortical types (primary sensory, motor, multimodal association) differ in their receptor fingerprints.

The Functional Neuroanatomy of Human Face Perception

Face perception is critical for normal social functioning and is mediated by a network of regions in the ventral visual stream. In this review, we describe recent neuroimaging findings regarding the macro- and microscopic anatomical features of the ventral face network, the characteristics of white matter connections, and basic computations performed by population receptive fields within face-selective regions composing this network. We emphasize the importance of the neural tissue properties and white matter connections of each region, as these anatomical properties may be tightly linked to the functional characteristics of the ventral face network. We end by considering how empirical investigations of the neural architecture of the face network may inform the development of computational models and shed light on how computations in the face network enable efficient face perception.

Goal-Directed Visual Processing Differentially Impacts Human Ventral and Dorsal Visual Representations

Recent studies have challenged the ventral/"what" and dorsal/"where" two-visual-processing-pathway view by showing the existence of "what" and "where" information in both pathways. Is the two-pathway distinction still valid? Here, we examined how goal-directed visual information processing may differentially impact visual representations in these two pathways. Using fMRI and multivariate pattern analysis, in three experiments on human participants (57% females), by manipulating whether color or shape was task-relevant and how they were conjoined, we examined shape-based object category decoding in occipitotemporal and parietal regions. We found that object category representations in all the regions examined were influenced by whether or not object shape was task-relevant. This task effect, however, tended to decrease as task-relevant and irrelevant features were more integrated, reflecting the well-known object-based feature encoding. Interestingly, task relevance played a relatively minor role in driving the representational structures of early visual and ventral object regions. They were driven predominantly by variations in object shapes. In contrast, the effect of task was much greater in dorsal than ventral regions, with object category and task relevance both contributing significantly to the representational structures of the dorsal regions. These results showed that, whereas visual representations in the ventral pathway are more invariant and reflect "what an object is," those in the dorsal pathway are more adaptive and reflect "what we do with it." Thus, despite the existence of "what" and "where" information in both visual processing pathways, the two pathways may still differ fundamentally in their roles in visual information representation.SIGNIFICANCE STATEMENT Visual information is thought to be processed in two distinctive pathways: the ventral pathway that processes "what" an object is and the dorsal pathway that processes "where" it is located. This view has been challenged by recent studies revealing the existence of "what" and "where" information in both pathways. Here, we found that goal-directed visual information processing differentially modulates shape-based object category representations in the two pathways. Whereas ventral representations are more invariant to the demand of the task, reflecting what an object is, dorsal representations are more adaptive, reflecting what we do with the object. Thus, despite the existence of "what" and "where" information in both pathways, visual representations may still differ fundamentally in the two pathways.

Effective Connectivity between Ventral Occipito-Temporal and Ventral Inferior Frontal Cortex during Lexico-Semantic Processing. A Dynamic Causal Modeling Study

It has been suggested that dorsal and ventral pathways support distinct aspects of language processing. Yet, the full extent of their involvement and their inter-regional connectivity in visual word recognition is still unknown. Studies suggest that they might reflect the dual-route model of reading, with the dorsal pathway more involved in grapho-phonological conversion during phonological tasks, and the ventral pathway performing lexico-semantic access during semantic tasks. Furthermore, this subdivision is also suggested at the level of the inferior frontal cortex, involving ventral and dorsal parts for lexico-semantic and phonological processing, respectively. In the present study, we assessed inter-regional brain connectivity and task-induced modulations of brain activity during a phoneme detection and semantic categorization tasks, using fMRI in healthy subject. We used a dynamic causal modeling approach to assess inter-regional connectivity and task demand modulation within the dorsal and ventral pathways, including the following network components: the ventral occipito-temporal cortex (vOTC; dorsal and ventral), the superior temporal gyrus (STG; dorsal), the dorsal inferior frontal gyrus (dIFG; dorsal), and the ventral IFG (vIFG; ventral). We report three distinct inter-regional interactions supporting orthographic information transfer from vOTC to other language regions (vOTC -> STG, vOTC -> vIFG and vOTC -> dIFG) regardless of task demands. Moreover, we found that (a) during semantic processing (direct ventral pathway) the vOTC -> vIFG connection strength specifically increased and (b) a lack of modulation of the vOTC -> dIFG connection strength by the task that could suggest a more general involvement of the dorsal pathway during visual word recognition. Results are discussed in terms of anatomo-functional connectivity of visual word recognition network.

Functional selectivity for face processing in the temporal voice area of early deaf individuals

Brain systems supporting face and voice processing both contribute to the extraction of important information for social interaction (e.g., person identity). How does the brain reorganize when one of these channels is absent? Here, we explore this question by combining behavioral and multimodal neuroimaging measures (magneto-encephalography and functional imaging) in a group of early deaf humans. We show enhanced selective neural response for faces and for individual face coding in a specific region of the auditory cortex that is typically specialized for voice perception in hearing individuals. In this region, selectivity to face signals emerges early in the visual processing hierarchy, shortly after typical face-selective responses in the ventral visual pathway. Functional and effective connectivity analyses suggest reorganization in long-range connections from early visual areas to the face-selective temporal area in individuals with early and profound deafness. Altogether, these observations demonstrate that regions that typically specialize for voice processing in the hearing brain preferentially reorganize for face processing in born-deaf people. Our results support the idea that cross-modal plasticity in the case of early sensory deprivation relates to the original functional specialization of the reorganized brain regions.

Visual perception skills: a comparison between patients with Noonan syndrome and 22q11.2 deletion syndrome

Ventral and dorsal streams are visual pathways deputed to transmit information from the photoreceptors of the retina to the lateral geniculate nucleus and then to the primary visual cortex (V1). Several studies investigated whether one pathway is more vulnerable than the other during development, and whether these streams develop at different rates. The results are still discordant. The aim of the present study was to understand the functionality of the dorsal and the ventral streams in two populations affected by different genetic disorders, Noonan syndrome (NS) and 22q11.2 deletion syndrome (22q11.2DS), and explore the possible genotype-phenotype relationships. 'Form coherence' abilities for the ventral stream and 'motion coherence' abilities for the dorsal stream were evaluated in 19 participants with NS and 20 participants with 22q11.2DS. Collected data were compared with 55 age-matched controls. Participants with NS and 22q11.2DS did not differ in the form coherence task, and their performance was significantly lower than that of controls. However, in the motion coherence task, the group with NS and controls did not differ, and both obtained significantly higher scores than the group with 22q11.2DS. Our findings indicate that deficits in the dorsal stream are related to the specific genotype, and that in our syndromic groups the ventral stream is more vulnerable than the dorsal stream.

What the success of brain imaging implies about the neural code

The success of fMRI places constraints on the nature of the neural code. The fact that researchers can infer similarities between neural representations, despite fMRI’s limitations, implies that certain neural coding schemes are more likely than others. For fMRI to succeed given its low temporal and spatial resolution, the neural code must be smooth at the voxel and functional level such that similar stimuli engender similar internal representations. Through proof and simulation, we determine which coding schemes are plausible given both fMRI’s successes and its limitations in measuring neural activity. Deep neural network approaches, which have been forwarded as computational accounts of the ventral stream, are consistent with the success of fMRI, though functional smoothness breaks down in the later network layers. These results have implications for the nature of the neural code and ventral stream, as well as what can be successfully investigated with fMRI.

DOI:http://dx.doi.org/10.7554/eLife.21397.001

We can appreciate that a cat is more similar to a dog than to a truck. The combined activity of millions of neurons in the brain somehow captures these everyday similarities, and this activity can be measured using imaging techniques such as functional magnetic resonance imaging (fMRI). However, fMRI scanners are not particularly precise – they average together the responses of many thousands of neurons over several seconds, which provides a blurry snapshot of brain activity. Nevertheless, the pattern of activity measured when viewing a photograph of a cat is more similar to that seen when viewing a picture of a dog than a picture of a truck. This tells us a lot about how the brain codes information, as only certain coding methods would allow fMRI to capture these similarities given the technique’s limitations.

There are many different models that attempt to describe how the brain codes similarity relations. Some models use the principle of neural networks, in which neurons can be considered as arranged into interconnected layers. In such models, neurons transmit information from one layer to the next.

By investigating which models are consistent with fMRI’s ability to capture similarity relations, Guest and Love have found that certain neural network models are plausible accounts of how the brain represents and processes information. These models include the deep learning networks that contain many layers of neurons and are popularly used in artificial intelligence. Other modeling approaches do not account for the ability of fMRI to capture similarity relations.

As neural networks become deeper with more layers, they should be less readily understood using fMRI: as the number of layers increases, the representations of objects with similarities (for example, cats and dogs) become more unrelated. One question that requires further investigation is whether this finding explains why certain parts of the brain are more difficult to image.

DOI:http://dx.doi.org/10.7554/eLife.21397.002

View-Tolerant Face Recognition and Hebbian Learning Imply Mirror-Symmetric Neural Tuning to Head Orientation

The primate brain contains a hierarchy of visual areas, dubbed the ventral stream, which rapidly computes object representations that are both specific for object identity and robust against identity-preserving transformations, like depth rotations [1, 2]. Current computational models of object recognition, including recent deep-learning networks, generate these properties through a hierarchy of alternating selectivity-increasing filtering and tolerance-increasing pooling operations, similar to simple-complex cells operations [3-6]. Here, we prove that a class of hierarchical architectures and a broad set of biologically plausible learning rules generate approximate invariance to identity-preserving transformations at the top level of the processing hierarchy. However, all past models tested failed to reproduce the most salient property of an intermediate representation of a three-level face-processing hierarchy in the brain: mirror-symmetric tuning to head orientation [7]. Here, we demonstrate that one specific biologically plausible Hebb-type learning rule generates mirror-symmetric tuning to bilaterally symmetric stimuli, like faces, at intermediate levels of the architecture and show why it does so. Thus, the tuning properties of individual cells inside the visual stream appear to result from group properties of the stimuli they encode and to reflect the learning rules that sculpted the information-processing system within which they reside.

Two Visual Pathways in Primates Based on Sampling of Space: Exploitation and Exploration of Visual Information

Evidence is strong that the visual pathway is segregated into two distinct streams—ventral and dorsal. Two proposals theorize that the pathways are segregated in function: The ventral stream processes information about object identity, whereas the dorsal stream, according to one model, processes information about either object location, and according to another, is responsible in executing movements under visual control. The models are influential; however recent experimental evidence challenges them, e.g., the ventral stream is not solely responsible for object recognition; conversely, its function is not strictly limited to object vision; the dorsal stream is not responsible by itself for spatial vision or visuomotor control; conversely, its function extends beyond vision or visuomotor control. In their place, we suggest a robust dichotomy consisting of a ventral stream selectively sampling high-resolution/focal spaces, and a dorsal stream sampling nearly all of space with reduced foveal bias. The proposal hews closely to the theme of embodied cognition: Function arises as a consequence of an extant sensory underpinning. A continuous, not sharp, segregation based on function emerges, and carries with it an undercurrent of an exploitation-exploration dichotomy. Under this interpretation, cells of the ventral stream, which individually have more punctate receptive fields that generally include the fovea or parafovea, provide detailed information about object shapes and features and lead to the systematic exploitation of said information; cells of the dorsal stream, which individually have large receptive fields, contribute to visuospatial perception, provide information about the presence/absence of salient objects and their locations for novel exploration and subsequent exploitation by the ventral stream or, under certain conditions, the dorsal stream. We leverage the dichotomy to unify neuropsychological cases under a common umbrella, account for the increased prevalence of multisensory integration in the dorsal stream under a Bayesian framework, predict conditions under which object recognition utilizes the ventral or dorsal stream, and explain why cells of the dorsal stream drive sensorimotor control and motion processing and have poorer feature selectivity. Finally, the model speculates on a dynamic interaction between the two streams that underscores a unified, seamless perception. Existing theories are subsumed under our proposal.

Off to a Good Start: The Early Development of the Neural Substrates Underlying Visual Working Memory

Current neuroscientific models describe the functional neural architecture of visual working memory (VWM) as an interaction of the frontal-parietal control network and more posterior areas in the ventral visual stream (Jonides et al., 2008; D'Esposito and Postle, 2015; Eriksson et al., 2015). These models are primarily based on adult neuroimaging studies. However, VWM undergoes significant development in infancy and early childhood, and the goal of this mini-review is to examine how recent findings from neuroscientific studies of early VWM development can be reconciled with this model. We surveyed 29 recent empirical reports that present neuroimaging findings in infants, toddlers, and preschoolers (using EEG, fNIRS, rs-fMRI) and neonatal lesion studies in non-human primates. We conclude that (1) both the frontal-parietal control network and the posterior cortical storage areas are active from early infancy; (2) this system undergoes focalization and some reorganization during early development; (3) and the MTL plays a significant role in this process as well. Motivated by both theoretical and methodological considerations, we offer some recommendations for future directions for the field.

Representation of Maximally Regular Textures in Human Visual Cortex

Naturalistic textures with an intermediate degree of statistical regularity can capture key structural features of natural images (Freeman and Simoncelli, 2011). V2 and later visual areas are sensitive to these features, while primary visual cortex is not (Freeman et al., 2013). Here we expand on this work by investigating a class of textures that have maximal formal regularity, the 17 crystallographic wallpaper groups (Fedorov, 1891). We used texture stimuli from four of the groups that differ in the maximum order of rotation symmetry they contain, and measured neural responses in human participants using functional MRI and high-density EEG. We found that cortical area V3 has a parametric representation of the rotation symmetries in the textures that is not present in either V1 or V2, the first discovery of a stimulus property that differentiates processing in V3 from that of lower-level areas. Parametric responses were also seen in higher-order ventral stream areas V4, VO1, and lateral occipital complex (LOC), but not in dorsal stream areas. The parametric response pattern was replicated in the EEG data, and source localization indicated that responses in V3 and V4 lead responses in LOC, which is consistent with a feedforward mechanism. Finally, we presented our stimuli to four well developed feedforward models and found that none of them were able to account for our results. Our results highlight structural regularity as an important stimulus dimension for distinguishing the early stages of visual processing, and suggest a previously unrecognized role for V3 in the visual form-processing hierarchy. Significance statement: Hierarchical processing is a fundamental organizing principle in visual neuroscience, with each successive processing stage being sensitive to increasingly complex stimulus properties. Here, we probe the encoding hierarchy in human visual cortex using a class of visual textures--wallpaper patterns--that are maximally regular. Through a combination of fMRI and EEG source imaging, we find specific responses to texture regularity that depend parametrically on the maximum order of rotation symmetry in the textures. These parametric responses are seen in several areas of the ventral visual processing stream, as well as in area V3, but not in V1 or V2. This is the first demonstration of a stimulus property that differentiates processing in V3 from that of lower-level visual areas.

Neural Representations of Natural and Scrambled Movies Progressively Change from Rat Striate to Temporal Cortex

In recent years, the rodent has come forward as a candidate model for investigating higher level visual abilities such as object vision. This view has been backed up substantially by evidence from behavioral studies that show rats can be trained to express visual object recognition and categorization capabilities. However, almost no studies have investigated the functional properties of rodent extrastriate visual cortex using stimuli that target object vision, leaving a gap compared with the primate literature. Therefore, we recorded single-neuron responses along a proposed ventral pathway in rat visual cortex to investigate hallmarks of primate neural object representations such as preference for intact versus scrambled stimuli and category-selectivity. We presented natural movies containing a rat or no rat as well as their phase-scrambled versions. Population analyses showed increased dissociation in representations of natural versus scrambled stimuli along the targeted stream, but without a clear preference for natural stimuli. Along the measured cortical hierarchy the neural response seemed to be driven increasingly by features that are not V1-like and destroyed by phase-scrambling. However, there was no evidence for category selectivity for the rat versus nonrat distinction. Together, these findings provide insights about differences and commonalities between rodent and primate visual cortex.