Reading with Sounds: Sensory Substitution Selectively Activates the Visual Word Form Area in the Blind

Is there a future for sensory substitution outside academic laboratories?

Sensory substitution devices (SSDs) have been developed with the ultimate purpose of supporting sensory deprived individuals in their daily activities. However, more than forty years after their first appearance in the scientific literature, SSDs still remain more common in research laboratories than in the daily life of people with sensory deprivation. Here, we seek to identify the reasons behind the limited diffusion of SSDs among the blind community by discussing the ergonomic, neurocognitive and psychosocial issues potentially associated with the use of these systems. We stress that these issues should be considered together when developing future devices or improving existing ones. We provide some examples of how to achieve this by adopting a multidisciplinary and participatory approach. These efforts would contribute not solely to address fundamental theoretical research questions, but also to better understand the everyday needs of blind people and eventually promote the use of SSDs outside laboratories.

Neural correlates of taste perception in congenital blindness

Sight is undoubtedly important for the perception and the assessment of the palatability of tastants. Although many studies have addressed the consequences of visual impairment on food selection, feeding behavior, eating habits and taste perception, nothing is known about the neural correlates of gustation in blindness. In the current study we examined brain responses during gustation using functional magnetic resonance imaging (fMRI). We scanned nine congenitally blind and 14 age- and sex-matched blindfolded sighted control subjects, matched in age, gender and body mass index (BMI), while they made judgments of either the intensity or the (un)pleasantness of different tastes (sweet, bitter) or artificial saliva that were delivered intra-orally. The fMRI data indicated that during gustation, congenitally blind individuals activate less strongly the primary taste cortex (right posterior insula and overlying Rolandic operculum) and the hypothalamus. In sharp contrast with results of multiple other sensory processing studies in congenitally blind subjects, including touch, audition and smell, the occipital cortex was not recruited during taste processing, suggesting the absence of taste-related compensatory crossmodal responses in the occipital cortex. These results underscore our earlier behavioral demonstration that congenitally blind subjects have a lower gustatory sensitivity compared to normal sighted individuals. We hypothesize that due to an underexposure to a variety of tastants, training-induced crossmodal sensory plasticity to gustatory stimulation does not occur in blind subjects.

Anatomical connections of the visual word form area

The visual word form area (VWFA), a region systematically involved in the identification of written words, occupies a reproducible location in the left occipitotemporal sulcus in expert readers of all cultures. Such a reproducible localization is paradoxical, given that reading is a recent invention that could not have influenced the genetic evolution of the cortex. Here, we test the hypothesis that the VWFA recycles a region of the ventral visual cortex that shows a high degree of anatomical connectivity to perisylvian language areas, thus providing an efficient circuit for both grapheme-phoneme conversion and lexical access. In two distinct experiments, using high-resolution diffusion-weighted data from 75 human subjects, we show that (1) the VWFA, compared with the fusiform face area, shows higher connectivity to left-hemispheric perisylvian superior temporal, anterior temporal and inferior frontal areas; (2) on a posterior-to-anterior axis, its localization within the left occipitotemporal sulcus maps onto a peak of connectivity with language areas, with slightly distinct subregions showing preferential projections to areas respectively involved in grapheme-phoneme conversion and lexical access. In agreement with functional data on the VWFA in blind subjects, the results suggest that connectivity to language areas, over and above visual factors, may be the primary determinant of VWFA localization.

Navigating from a Depth Image Converted into Sound

Background. Common manufactured depth sensors generate depth images that humans normally obtain from their eyes and hands. Various designs converting spatial data into sound have been recently proposed, speculating on their applicability as sensory substitution devices (SSDs). Objective. We tested such a design as a travel aid in a navigation task. Methods. Our portable device (MeloSee) converted 2D array of a depth image into melody in real-time. Distance from the sensor was translated into sound intensity, stereo-modulated laterally, and the pitch represented verticality. Twenty-one blindfolded young adults navigated along four different paths during two sessions separated by one-week interval. In some instances, a dual task required them to recognize a temporal pattern applied through a tactile vibrator while they navigated. Results. Participants learnt how to use the system on both new paths and on those they had already navigated from. Based on travel time and errors, performance improved from one week to the next. The dual task was achieved successfully, slightly affecting but not preventing effective navigation. Conclusions. The use of Kinect-type sensors to implement SSDs is promising, but it is restricted to indoor use and it is inefficient on too short range.

A number-form area in the blind

Distinct preference for visual number symbols was recently discovered in the human right inferior temporal gyrus (rITG). It remains unclear how this preference emerges, what is the contribution of shape biases to its formation and whether visual processing underlies it. Here we use congenital blindness as a model for brain development without visual experience. During fMRI, we present blind subjects with shapes encoded using a novel visual-to-music sensory-substitution device (The EyeMusic). Greater activation is observed in the rITG when subjects process symbols as numbers compared with control tasks on the same symbols. Using resting-state fMRI in the blind and sighted, we further show that the areas with preference for numerals and letters exhibit distinct patterns of functional connectivity with quantity and language-processing areas, respectively. Our findings suggest that specificity in the ventral ‘visual’ stream can emerge independently of sensory modality and visual experience, under the influence of distinct connectivity patterns.

The human visual cortex includes areas with preference for various object categories. Here, Abboud et al. demonstrate using visual-to-music substitution, that the congenitally blind show a similar preference for numerals in the right inferior temporal cortex as sighted individuals, despite having no visual experience.

Congenital visual pathway abnormalities: a window onto cortical stability and plasticity

Sensory systems project information in a highly organized manner to the brain, where it is preserved in maps of the sensory structures. These sensory projections are altered in congenital abnormalities, such as anophthalmia, albinism, achiasma, and hemihydranencephaly. Consequently, these abnormalities, profoundly affect the organization of the visual system. Surprisingly, visual perception remains largely intact, except for anophthalmia. Recent brain imaging advances shed light on the mechanisms that underlie this phenomenon. In contrast to animal models, in humans the plasticity of thalamocortical connections appears limited, thus demonstrating the importance of cortical adaptations. We suggest that congenital visual pathway abnormalities provide a valuable model to investigate the principles of plasticity that make visual representations available for perception and behavior in humans.

Color synesthesia. Insight into perception, emotion, and consciousness

Purpose of review

Synesthesia is an extraordinary perceptual phenomenon, in which individuals experience unusual percepts elicited by the activation of an unrelated sensory modality or by a cognitive process. Emotional reactions are commonly associated. The condition prompted philosophical debates on the nature of perception and impacted the course of art history. It recently generated a considerable interest among neuroscientists, but its clinical significance apparently remains underevaluated. This review focuses on the recent studies regarding variants of color synesthesia, the commonest form of the condition.

Recent findings

Synesthesia is commonly classified as developmental and acquired. Developmental forms predispose to changes in primary sensory processing and cognitive functions, usually with better performances in certain aspects and worse in others, and to heightened creativity. Acquired forms of synesthesia commonly arise from drug ingestion or neurological disorders, including thalamic lesions and sensory deprivation (e.g., blindness). Cerebral exploration using structural and functional imaging has demonstrated distinct patterns in cortical activation and brain connectivity for controls and synesthetes. Artworks of affected painters are most illustrative of the nature of synesthetic experiences.

Summary

Results of the recent investigations on synesthesia offered a remarkable insight into the mechanisms of perception, emotion and consciousness, and deserve attention both from neuroscientists and from clinicians.

Anatomical and functional plasticity in early blind individuals and the mixture of experts architecture

As described elsewhere in this special issue, recent advances in neuroimaging over the last decade have led to a rapid expansion in our knowledge of anatomical and functional correlations within the normal and abnormal human brain. Here, we review how early blindness has been used as a model system for examining the role of visual experience in the development of anatomical connections and functional responses. We discuss how lack of power in group comparisons may provide a potential explanation for why extensive anatomical changes in cortico-cortical connectivity are not observed. Finally we suggest a framework-cortical specialization via hierarchical mixtures of experts-which offers some promise in reconciling a wide range of functional and anatomical data.

Accessing orthographic representations from speech: the role of left ventral occipitotemporal cortex in spelling

The present fMRI study used a spelling task to investigate the hypothesis that the left ventral occipitotemporal cortex (vOT) hosts neuronal representations of whole written words. Such an orthographic word lexicon is posited by cognitive dual‐route theories of reading and spelling. In the scanner, participants performed a spelling task in which they had to indicate if a visually presented letter is present in the written form of an auditorily presented word. The main experimental manipulation distinguished between an orthographic word spelling condition in which correct spelling decisions had to be based on orthographic whole‐word representations, a word spelling condition in which reliance on orthographic whole‐word representations was optional and a phonological pseudoword spelling condition in which no reliance on such representations was possible. To evaluate spelling‐specific activations the spelling conditions were contrasted with control conditions that also presented auditory words and pseudowords, but participants had to indicate if a visually presented letter corresponded to the gender of the speaker. We identified a left vOT cluster activated for the critical orthographic word spelling condition relative to both the control condition and the phonological pseudoword spelling condition. Our results suggest that activation of left vOT during spelling can be attributed to the retrieval of orthographic whole‐word representations and, thus, support the position that the left vOT potentially represents the neuronal equivalent of the cognitive orthographic word lexicon. Hum Brain Mapp, 36:1393–1406, 2015. © 2014 The Authors Human Brain Mapping Published by Wiley Periodicals, Inc.

Timing the impact of literacy on visual processing

Learning to read requires the acquisition of an efficient visual procedure for quickly recognizing fine print. Thus, reading practice could induce a perceptual learning effect in early vision. Using functional magnetic resonance imaging (fMRI) in literate and illiterate adults, we previously demonstrated an impact of reading acquisition on both high- and low-level occipitotemporal visual areas, but could not resolve the time course of these effects. To clarify whether literacy affects early vs. late stages of visual processing, we measured event-related potentials to various categories of visual stimuli in healthy adults with variable levels of literacy, including completely illiterate subjects, early-schooled literate subjects, and subjects who learned to read in adulthood (ex-illiterates). The stimuli included written letter strings forming pseudowords, on which literacy is expected to have a major impact, as well as faces, houses, tools, checkerboards, and false fonts. To evaluate the precision with which these stimuli were encoded, we studied repetition effects by presenting the stimuli in pairs composed of repeated, mirrored, or unrelated pictures from the same category. The results indicate that reading ability is correlated with a broad enhancement of early visual processing, including increased repetition suppression, suggesting better exemplar discrimination, and increased mirror discrimination, as early as ∼ 100-150 ms in the left occipitotemporal region. These effects were found with letter strings and false fonts, but also were partially generalized to other visual categories. Thus, learning to read affects the magnitude, precision, and invariance of early visual processing.

Color improves "visual" acuity via sound

Visual-to-auditory sensory substitution devices (SSDs) convey visual information via sound, with the primary goal of making visual information accessible to blind and visually impaired individuals. We developed the EyeMusic SSD, which transforms shape, location, and color information into musical notes. We tested the “visual” acuity of 23 individuals (13 blind and 10 blindfolded sighted) on the Snellen tumbling-E test, with the EyeMusic. Participants were asked to determine the orientation of the letter “E.” The test was repeated twice: in one test, the letter “E” was drawn with a single color (white), and in the other test, with two colors (red and white). In the latter case, the vertical line in the letter, when upright, was drawn in red, with the three horizontal lines drawn in white. We found no significant differences in performance between the blind and the sighted groups. We found a significant effect of the added color on the “visual” acuity. The highest acuity participants reached in the monochromatic test was 20/800, whereas with the added color, acuity doubled to 20/400. We conclude that color improves “visual” acuity via sound.

Clinical characteristics and current therapies for inherited retinal degenerations

Inherited retinal degenerations (IRDs) encompass a large group of clinically and genetically heterogeneous diseases that affect approximately 1 in 3000 people (>2 million people worldwide) (Bessant DA, Ali RR, Bhattacharya SS. 2001. Molecular genetics and prospects for therapy of the inherited retinal dystrophies. Curr Opin Genet Dev 11: 307-316.). IRDs may be inherited as Mendelian traits or through mitochondrial DNA, and may affect the entire retina (e.g., rod-cone dystrophy, also known as retinitis pigmentosa, cone dystrophy, cone-rod dystrophy, choroideremia, Usher syndrome, and Bardet-Bidel syndrome) or be restricted to the macula (e.g., Stargardt disease, Best disease, and Sorsby fundus dystrophy), ultimately leading to blindness. IRDs are a major cause of severe vision loss, with profound impact on patients and society. Although IRDs remain untreatable today, significant progress toward therapeutic strategies for IRDs has marked the past two decades. This progress has been based on better understanding of the pathophysiological pathways of these diseases and on technological advances.

Time-interval for integration of stabilizing haptic and visual information in subjects balancing under static and dynamic conditions

Maintaining equilibrium is basically a sensorimotor integration task. The central nervous system (CNS) continually and selectively weights and rapidly integrates sensory inputs from multiple sources, and coordinates multiple outputs. The weighting process is based on the availability and accuracy of afferent signals at a given instant, on the time-period required to process each input, and possibly on the plasticity of the relevant pathways. The likelihood that sensory inflow changes while balancing under static or dynamic conditions is high, because subjects can pass from a dark to a well-lit environment or from a tactile-guided stabilization to loss of haptic inflow. This review article presents recent data on the temporal events accompanying sensory transition, on which basic information is fragmentary. The processing time from sensory shift to reaching a new steady state includes the time to (a) subtract or integrate sensory inputs; (b) move from allocentric to egocentric reference or vice versa; and (c) adjust the calibration of motor activity in time and amplitude to the new sensory set. We present examples of processes of integration of posture-stabilizing information, and of the respective sensorimotor time-intervals while allowing or occluding vision or adding or subtracting tactile information. These intervals are short, in the order of 1–2 s for different postural conditions, modalities and deliberate or passive shift. They are just longer for haptic than visual shift, just shorter on withdrawal than on addition of stabilizing input, and on deliberate than unexpected mode. The delays are the shortest (for haptic shift) in blind subjects. Since automatic balance stabilization may be vulnerable to sensory-integration delays and to interference from concurrent cognitive tasks in patients with sensorimotor problems, insight into the processing time for balance control represents a critical step in the design of new balance- and locomotion training devices.

Revisiting the adaptive and maladaptive effects of crossmodal plasticity

One of the most striking demonstrations of experience-dependent plasticity comes from studies of sensory-deprived individuals (e.g., blind or deaf), showing that brain regions deprived of their natural inputs change their sensory tuning to support the processing of inputs coming from the spared senses. These mechanisms of crossmodal plasticity have been traditionally conceptualized as having a double-edged sword effect on behavior. On one side, crossmodal plasticity is conceived as adaptive for the development of enhanced behavioral skills in the remaining senses of early-deaf or blind individuals. On the other side, crossmodal plasticity raises crucial challenges for sensory restoration and is typically conceived as maladaptive since its presence may prevent optimal recovery in sensory-re-afferented individuals. In the present review we stress that this dichotomic vision is oversimplified and we emphasize that the notions of the unavoidable adaptive/maladaptive effects of crossmodal reorganization for sensory compensation/restoration may actually be misleading. For this purpose we critically review the findings from the blind and deaf literatures, highlighting the complementary nature of these two fields of research. The integrated framework we propose here has the potential to impact on the way rehabilitation programs for sensory recovery are carried out, with the promising prospect of eventually improving their final outcomes.

An ALE meta-analysis on the audiovisual integration of speech signals

The brain improves speech processing through the integration of audiovisual (AV) signals. Situations involving AV speech integration may be crudely dichotomized into those where auditory and visual inputs contain (1) equivalent, complementary signals (validating AV speech) or (2) inconsistent, different signals (conflicting AV speech). This simple framework may allow the systematic examination of broad commonalities and differences between AV neural processes engaged by various experimental paradigms frequently used to study AV speech integration. We conducted an activation likelihood estimation metaanalysis of 22 functional imaging studies comprising 33 experiments, 311 subjects, and 347 foci examining "conflicting" versus "validating" AV speech. Experimental paradigms included content congruency, timing synchrony, and perceptual measures, such as the McGurk effect or synchrony judgments, across AV speech stimulus types (sublexical to sentence). Colocalization of conflicting AV speech experiments revealed consistency across at least two contrast types (e.g., synchrony and congruency) in a network of dorsal stream regions in the frontal, parietal, and temporal lobes. There was consistency across all contrast types (synchrony, congruency, and percept) in the bilateral posterior superior/middle temporal cortex. Although fewer studies were available, validating AV speech experiments were localized to other regions, such as ventral stream visual areas in the occipital and inferior temporal cortex. These results suggest that while equivalent, complementary AV speech signals may evoke activity in regions related to the corroboration of sensory input, conflicting AV speech signals recruit widespread dorsal stream areas likely involved in the resolution of conflicting sensory signals.

Compensatory plasticity: time matters

Plasticity in the human and animal brain is the rule, the base for development, and the way to deal effectively with the environment for making the most efficient use of all the senses. When the brain is deprived of one sensory modality, plasticity becomes compensatory: the exception that invalidates the general loss hypothesis giving the opportunity of effective change. Sensory deprivation comes with massive alterations in brain structure and function, behavioral outcomes, and neural interactions. Blind individuals do as good as the sighted and even more, show superior abilities in auditory, tactile and olfactory processing. This behavioral enhancement is accompanied with changes in occipital cortex function, where visual areas at different levels become responsive to non-visual information. The intact senses are in general used more efficiently in the blind but are also used more exclusively. New findings are disentangling these two aspects of compensatory plasticity. What is due to visual deprivation and what is dependent on the extended use of spared modalities? The latter seems to contribute highly to compensatory changes in the congenitally blind. Short-term deprivation through the use of blindfolds shows that cortical excitability of the visual cortex is likely to show rapid modulatory changes after few minutes of light deprivation and therefore changes are possible in adulthood. However, reorganization remains more pronounced in the congenitally blind. Cortico-cortical pathways between visual areas and the areas of preserved sensory modalities are inhibited in the presence of vision, but are unmasked after loss of vision or blindfolding as a mechanism likely to drive cross-modal information to the deafferented visual cortex. The development of specialized higher order visual pathways independently from early sensory experience is likely to preserve their function and switch to the intact modalities. Plasticity in the blind is also accompanied with neurochemical and morphological changes; both intrinsic connectivity and functional coupling at rest are altered but are likewise dependent on different sensory experience and training.

Multisensory integration: flexible use of general operations

Research into the anatomical substrates and "principles" for integrating inputs from separate sensory surfaces has yielded divergent findings. This suggests that multisensory integration is flexible and context dependent and underlines the need for dynamically adaptive neuronal integration mechanisms. We propose that flexible multisensory integration can be explained by a combination of canonical, population-level integrative operations, such as oscillatory phase resetting and divisive normalization. These canonical operations subsume multisensory integration into a fundamental set of principles as to how the brain integrates all sorts of information, and they are being used proactively and adaptively. We illustrate this proposition by unifying recent findings from different research themes such as timing, behavioral goal, and experience-related differences in integration.

Failure to learn a new spatial format in children with developmental dyslexia

A general problem in studying children with developmental dyslexia is how to separate inefficiency in learning on the one hand from exposure to written texts on the other. To evaluate dyslexic children's learning abilities with graphemic materials, we tested their improvement in a condition that minimized previous experience with words (i.e., “novel words”) and with the standard, horizontal spatial letter array (i.e., a non-canonical “zigzag” format). We selected five pairs of children with dyslexia and (younger) typically developing readers matched for reading speed and accuracy in these specific conditions. Reading performance on novel words in the zigzag format was measured in 23 sessions; learning curves were fitted by power functions. Similar to typically developing readers, children with dyslexia improved their reading of novel words presented in the new format; however, their rate of learning was slower than that of typically developing readers. Furthermore, their learning to read in the new format did not generalize to novel untrained items, whereas significant generalization was present in typically developing readers. As the failure to generalize learning of the spatial format could not be attributed to reduced experience, it indicates a genuine disability and points to impaired perceptual learning as a factor in developmental dyslexia.

Visual cortex extrastriate body-selective area activation in congenitally blind people "seeing" by using sounds

Vision is by far the most prevalent sense for experiencing others' body shapes, postures, actions, and intentions, and its congenital absence may dramatically hamper body-shape representation in the brain. We investigated whether the absence of visual experience and limited exposure to others' body shapes could still lead to body-shape selectivity. We taught congenitally fully-blind adults to perceive full-body shapes conveyed through a sensory-substitution algorithm topographically translating images into soundscapes [1]. Despite the limited experience of the congenitally blind with external body shapes (via touch of close-by bodies and for ~10 hr via soundscapes), once the blind could retrieve body shapes via soundscapes, they robustly activated the visual cortex, specifically the extrastriate body area (EBA; [2]). Furthermore, body selectivity versus textures, objects, and faces in both the blind and sighted control groups was not found in the temporal (auditory) or parietal (somatosensory) cortex but only in the visual EBA. Finally, resting-state data showed that the blind EBA is functionally connected to the temporal cortex temporal-parietal junction/superior temporal sulcus Theory-of-Mind areas [3]. Thus, the EBA preference is present without visual experience and with little exposure to external body-shape information, supporting the view that the brain has a sensory-independent, task-selective supramodal organization rather than a sensory-specific organization.

Retinal prostheses: clinical results and future challenges

Retinal prostheses aim at restoring visual perception in blind patients affected by retinal diseases leading to the loss of photoreceptors, such as age-related macular degeneration or retinitis pigmentosa. Recent clinical trials have demonstrated the feasibility of this approach for restoring useful vision. Despite a limited number of electrodes (60), and therefore of pixels, some patients were able to read words and to recognize high-contrast objects. Face recognition and independent locomotion in unknown urban environments imply technological breakthroughs to increase the number and density of electrodes. This review presents recent clinical results and discusses future solutions to answer the major technological challenges.

Relationship Between Cortical Thickness and Functional Activation in the Early Blind

Early blindness results in both structural and functional changes of the brain. However, these changes have rarely been studied in relation to each other. We measured alterations in cortical thickness (CT) caused by early visual deprivation and their relationship with cortical activity. Structural and functional magnetic resonance imaging was performed in 12 early blind (EB) humans and 12 sighted controls (SC). Experimental conditions included one-back tasks for auditory localization and pitch identification, and a simple sound-detection task. Structural and functional data were analyzed in a whole-brain approach and within anatomically defined regions of interest in sensory areas of the spared (auditory) and deprived (visual) modalities. Functional activation during sound-localization or pitch-identification tasks correlated negatively with CT in occipital areas of EB (calcarine sulcus, lingual gyrus, superior and middle occipital gyri, and cuneus) and in nonprimary auditory areas of SC. These results suggest a link between CT and activation and demonstrate that the relationship between cortical structure and function may depend on early sensory experience, probably via selective pruning of exuberant connections. Activity-dependent effects of early sensory deprivation and long-term practice are superimposed on normal maturation and aging. Together these processes shape the relationship between brain structure and function over the lifespan.

Flexibility and Stability in Sensory Processing Revealed Using Visual-to-Auditory Sensory Substitution

The classical view of sensory processing involves independent processing in sensory cortices and multisensory integration in associative areas. This hierarchical structure has been challenged by evidence of multisensory responses in sensory areas, and dynamic weighting of sensory inputs in associative areas, thus far reported independently. Here, we used a visual-to-auditory sensory substitution algorithm (SSA) to manipulate the information conveyed by sensory inputs while keeping the stimuli intact. During scan sessions before and after SSA learning, subjects were presented with visual images and auditory soundscapes. The findings reveal 2 dynamic processes. First, crossmodal attenuation of sensory cortices changed direction after SSA learning from visual attenuations of the auditory cortex to auditory attenuations of the visual cortex. Secondly, associative areas changed their sensory response profile from strongest response for visual to that for auditory. The interaction between these phenomena may play an important role in multisensory processing. Consistent features were also found in the sensory dominance in sensory areas and audiovisual convergence in associative area Middle Temporal Gyrus. These 2 factors allow for both stability and a fast, dynamic tuning of the system when required.

Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation

Sensory substitution devices (SSDs) have come a long way since first developed for visual rehabilitation. They have produced exciting experimental results, and have furthered our understanding of the human brain. Unfortunately, they are still not used for practical visual rehabilitation, and are currently considered as reserved primarily for experiments in controlled settings. Over the past decade, our understanding of the neural mechanisms behind visual restoration has changed as a result of converging evidence, much of which was gathered with SSDs. This evidence suggests that the brain is more than a pure sensory-machine but rather is a highly flexible task-machine, i.e., brain regions can maintain or regain their function in vision even with input from other senses. This complements a recent set of more promising behavioral achievements using SSDs and new promising technologies and tools. All these changes strongly suggest that the time has come to revive the focus on practical visual rehabilitation with SSDs and we chart several key steps in this direction such as training protocols and self-train tools.

[Restoring vision in blind patients following photoreceptor degeneration: clinical results and future challenges]

Retinal prostheses aim at restoring vision in patients blind from photoreceptor degeneration by electrically stimulating the residual retinal tissue. Currently, the most efficient implants are either inserted in the subretinal space or on the vitreal side of the retina (epi-retinal). Although the residual tissue can partly degenerate, it was shown that acute stimulation of residual neurones can induce visual percepts. Recently, a clinical trial with the epiretinal Argus2 device (60 electrodes) from the company 2nd Sight enabled most patients to orient and find light targets, some even reading words. This device has received a CE mark. Surprisingly, when the subretinal implant from the company Retina Implant AG displaying many more electrodes (1500 electrodes) was evaluated in clinical trials, the patient visual performances were fairly similar. The restored visual performances of the patients demonstrate that blind patients can recover some visual function when their residual retina is properly stimulated. However, the resolution is not yet sufficient to perform complex tasks such as autonomous locomotion, face identification or text reading. Several challenges remain to generate an increase in pixel density corresponding to the increase in electrode number and density. These challenges include the stimulation modality, the tissue/implant interface design, the electrode materials, and the visual information encoder. This review will discuss these great challenges after introducing the major clinical results.

Unimodal and multimodal regions for logographic language processing in left ventral occipitotemporal cortex

The human neocortex appears to contain a dedicated visual word form area (VWFA) and an adjacent multimodal (visual/auditory) area. However, these conclusions are based on functional magnetic resonance imaging (fMRI) of alphabetic language processing, languages that have clear grapheme-to-phoneme correspondence (GPC) rules that make it difficult to disassociate visual-specific processing from form-to-sound mapping. In contrast, the Chinese language has no clear GPC rules. Therefore, the current study examined whether native Chinese readers also have the same VWFA and multimodal area. Two cross-modal tasks, phonological retrieval of visual words and orthographic retrieval of auditory words, were adopted. Different task requirements were also applied to explore how different levels of cognitive processing modulate activation of putative VWFA-like and multimodal-like regions. Results showed that the left occipitotemporal sulcus (LOTS) responded exclusively to visual inputs and an adjacent region, the left inferior temporal gyrus (LITG), showed comparable activation for both visual and auditory inputs. Surprisingly, processing levels did not significantly alter activation of these two regions. These findings indicated that there are both unimodal and multimodal word areas for non-alphabetic language reading, and that activity in these two word-specific regions are independent of task demands at the linguistic level.

Right occipital cortex activation correlates with superior odor processing performance in the early blind

Using functional magnetic resonance imaging (fMRI) in ten early blind humans, we found robust occipital activation during two odor-processing tasks (discrimination or categorization of fruit and flower odors), as well as during control auditory-verbal conditions (discrimination or categorization of fruit and flower names). We also found evidence for reorganization and specialization of the ventral part of the occipital cortex, with dissociation according to stimulus modality: the right fusiform gyrus was most activated during olfactory conditions while part of the left ventral lateral occipital complex showed a preference for auditory-verbal processing. Only little occipital activation was found in sighted subjects, but the same right-olfactory/left-auditory-verbal hemispheric lateralization was found overall in their brain. This difference between the groups was mirrored by superior performance of the blind in various odor-processing tasks. Moreover, the level of right fusiform gyrus activation during the olfactory conditions was highly correlated with individual scores in a variety of odor recognition tests, indicating that the additional occipital activation may play a functional role in odor processing.

The emergence of semantic meaning in the ventral temporal pathway

In the ventral visual pathway, early visual areas encode light patterns on the retina in terms of image properties, for example, edges and color, whereas higher areas encode visual information in terms of objects and categories. At what point does semantic knowledge, as instantiated in human language, emerge? We examined this question by studying whether semantic similarity in language relates to the brain's organization of object representations in inferior temporal cortex (ITC), an area of the brain at the crux of several proposals describing how the brain might represent conceptual knowledge. Semantic relationships among words can be viewed as a geometrical structure with some pairs of words close in their meaning (e.g., man and boy) and other pairs more distant (e.g., man and tomato). ITC's representation of objects similarly can be viewed as a complex structure with some pairs of stimuli evoking similar patterns of activation (e.g., man and boy) and other pairs evoking very different patterns (e.g., man and tomato). In this study, we examined whether the geometry of visual object representations in ITC bears a correspondence to the geometry of semantic relationships between word labels used to describe the objects. We compared ITC's representation to semantic structure, evaluated by explicit ratings of semantic similarity and by five computational measures of semantic similarity. We show that the representational geometry of ITC-but not of earlier visual areas (V1)-is reflected both in explicit behavioral ratings of semantic similarity and also in measures of semantic similarity derived from word usage patterns in natural language. Our findings show that patterns of brain activity in ITC not only reflect the organization of visual information into objects but also represent objects in a format compatible with conceptual thought and language.

A depth-based head-mounted visual display to aid navigation in partially sighted individuals

Independent navigation for blind individuals can be extremely difficult due to the inability to recognise and avoid obstacles. Assistive techniques such as white canes, guide dogs, and sensory substitution provide a degree of situational awareness by relying on touch or hearing but as yet there are no techniques that attempt to make use of any residual vision that the individual is likely to retain. Residual vision can restricted to the awareness of the orientation of a light source, and hence any information presented on a wearable display would have to limited and unambiguous. For improved situational awareness, i.e. for the detection of obstacles, displaying the size and position of nearby objects, rather than including finer surface details may be sufficient. To test whether a depth-based display could be used to navigate a small obstacle course, we built a real-time head-mounted display with a depth camera and software to detect the distance to nearby objects. Distance was represented as brightness on a low-resolution display positioned close to the eyes without the benefit focussing optics. A set of sighted participants were monitored as they learned to use this display to navigate the course. All were able to do so, and time and velocity rapidly improved with practise with no increase in the number of collisions. In a second experiment a cohort of severely sight-impaired individuals of varying aetiologies performed a search task using a similar low-resolution head-mounted display. The majority of participants were able to use the display to respond to objects in their central and peripheral fields at a similar rate to sighted controls. We conclude that the skill to use a depth-based display for obstacle avoidance can be rapidly acquired and the simplified nature of the display may appropriate for the development of an aid for sight-impaired individuals.

"To see or not to see: that is the question." The "Protection-Against-Schizophrenia" (PaSZ) model: evidence from congenital blindness and visuo-cognitive aberrations

The causes of schizophrenia are still unknown. For the last 100 years, though, both “absent” and “perfect” vision have been associated with a lower risk for schizophrenia. Hence, vision itself and aberrations in visual functioning may be fundamental to the development and etiological explanations of the disorder. In this paper, we present the “Protection-Against-Schizophrenia” (PaSZ) model, which grades the risk for developing schizophrenia as a function of an individual's visual capacity. We review two vision perspectives: (1) “Absent” vision or how congenital blindness contributes to PaSZ and (2) “perfect” vision or how aberrations in visual functioning are associated with psychosis. First, we illustrate that, although congenitally blind and sighted individuals acquire similar world representations, blind individuals compensate for behavioral shortcomings through neurofunctional and multisensory reorganization. These reorganizations may indicate etiological explanations for their PaSZ. Second, we demonstrate that visuo-cognitive impairments are fundamental for the development of schizophrenia. Deteriorated visual information acquisition and processing contribute to higher-order cognitive dysfunctions and subsequently to schizophrenic symptoms. Finally, we provide different specific therapeutic recommendations for individuals who suffer from visual impairments (who never developed “normal” vision) and individuals who suffer from visual deterioration (who previously had “normal” visual skills). Rather than categorizing individuals as “normal” and “mentally disordered,” the PaSZ model uses a continuous scale to represent psychiatrically relevant human behavior. This not only provides a scientific basis for more fine-grained diagnostic assessments, earlier detection, and more appropriate therapeutic assignments, but it also outlines a trajectory for unraveling the causes of abnormal psychotic human self- and world-perception.

Distributed circuits, not circumscribed centers, mediate visual recognition

Increasingly, the neural mechanisms that support visual cognition are being conceptualized as a distributed but integrated system, as opposed to a set of individual, specialized regions that each subserve a particular visual behavior. Consequently, there is an emerging emphasis on characterizing the functional, structural,and computational properties of these broad networks [corrected]. We present a novel theoretical perspective, which elucidates the developmental emergence, computational properties, and vulnerabilities of integrated circuits using face and word recognition as model domains. Additionally, we suggest that, rather than being disparate and independent, these neural circuits are overlapping and subject to the same computational constraints. Specifically, we argue that both word and face recognition rely on fine-grained visual representations but, by virtue of pressure to couple visual and language areas and to keep connection length short, the left hemisphere becomes more finely tuned for word recognition and, consequently, the right hemisphere becomes more finely tuned for face recognition. Thus, both hemispheres ultimately participate in both forms of visual recognition, but their respective contributions are asymmetrically weighted.

Cortical plasticity and preserved function in early blindness

The "neural Darwinism" theory predicts that when one sensory modality is lacking, as in congenital blindness, the target structures are taken over by the afferent inputs from other senses that will promote and control their functional maturation (Edelman, 1993). This view receives support from both cross-modal plasticity experiments in animal models and functional imaging studies in man, which are presented here.