Cortical visual prostheses: from microstimulation to functional percept

Cortical Reorganization after Limb Loss: Bridging the Gap between Basic Science and Clinical Recovery

Despite the increasing incidence and prevalence of amputation across the globe, individuals with acquired limb loss continue to struggle with functional recovery and chronic pain. A more complete understanding of the motor and sensory remodeling of the peripheral and central nervous system that occurs postamputation may help advance clinical interventions to improve the quality of life for individuals with acquired limb loss. The purpose of this article is to first provide background clinical context on individuals with acquired limb loss and then to provide a comprehensive review of the known motor and sensory neural adaptations from both animal models and human clinical trials. Finally, the article bridges the gap between basic science researchers and clinicians that treat individuals with limb loss by explaining how current clinical treatments may restore function and modulate phantom limb pain using the underlying neural adaptations described above. This review should encourage the further development of novel treatments with known neurological targets to improve the recovery of individuals postamputation.Significance Statement In the United States, 1.6 million people live with limb loss; this number is expected to more than double by 2050. Improved surgical procedures enhance recovery, and new prosthetics and neural interfaces can replace missing limbs with those that communicate bidirectionally with the brain. These advances have been fairly successful, but still most patients experience persistent problems like phantom limb pain, and others discontinue prostheses instead of learning to use them daily. These problematic patient outcomes may be due in part to the lack of consensus among basic and clinical researchers regarding the plasticity mechanisms that occur in the brain after amputation injuries. Here we review results from clinical and animal model studies to bridge this clinical-basic science gap.

The Influence of Phosphene Synchrony in Driving Object Binding in a Simulation of Artificial Vision

Purpose

Electrical microstimulation techniques used in visual prostheses are designed to restore visual function following acquired blindness. Patterns of induced focal percepts, known as phosphenes, are achieved by applying localized electrical pulses to the visual pathway to bypass the impaired site in order to convey images from the external world. Here, we use a simulation of artificial vision to manipulate relationships between individual phosphenes to observe the effects on object binding and perception. We hypothesize that synchronous phosphene presentation will facilitate object perception as compared to asynchronous presentation.

Methods

A model system that tracks gaze position of normal, sighted participants to present patterns of phosphenes on a computer screen was used to simulate prosthetic vision. Participants performed a reading task at varying font sizes (1.1-1.4 logMAR) and under varying levels of phosphene temporal noise while reading accuracy and speed were measured.

Results

Reading performance was significantly affected by temporal noise in phosphene presentation, with increasing desynchronization leading to lower reading scores. A drop in performance was also observed when the total latency between the gaze position and phosphene update was increased without adding temporal noise.

Conclusions

Object perception (here, text perception) is enhanced with synchronously presented phosphenes as compared to asynchronously presented ones. These results are fundamental for developing an efficient temporal pattern of stimulation and for the creation of high-fidelity prosthetic vision.

Attitudes of potential recipients toward emerging visual prosthesis technologies

With the advent of multiple visual prosthesis devices to treat blindness, the question of how potential patients view such interventions becomes important in order to understand the levels of expectation and acceptance, and the perceived risk-reward balance across the different device approaches. Building on previous work on single device approaches done with blind individuals in Chicago and Detroit, USA, Melbourne, Australia, and Bejing, China, we investigated attitudes in blind individuals in Athens, Greece with coverage expanded to three of the contemporary approaches, Retinal, Thalamic, and Cortical. We presented an informational lecture on the approaches, had potential participants fill out a preliminary Questionnaire 1, then organized selected subjects into focus groups for guided discussion on visual prostheses, and finally had these subjects fill out a more detailed Questionnaire 2. We report here the first quantitative data that compares multiple prosthesis approaches. Our primary findings are that for these potential patients, perceived risk continues to outweigh perceived benefits, with the Retinal approach having the least negative overall impression and the Cortical approach the most negative. Concerns about the quality of restored vision were primary. Factors that drove the choice of hypothetical participation in a clinical trial were age and years of blindness. Secondary factors focused on positive clinical outcomes. The focus groups served to swing the impressions of each approach from neutrality toward the extremes of a Likert scale, and shifted the overall willingness to participate in a clinical trial from neutral to negative. These results, coupled with informal assessment of audience questions after the informational lecture, suggest that a substantial improvement in performance over currently available devices will be necessary before visual prostheses gain wide acceptance.

Effects of central nervous system electrical stimulation on non-neuronal cells

Over the past few decades, much progress has been made in the clinical use of electrical stimulation of the central nervous system (CNS) to treat an ever-growing number of conditions from Parkinson's disease (PD) to epilepsy as well as for sensory restoration and many other applications. However, little is known about the effects of microstimulation at the cellular level. Most of the existing research focuses on the effects of electrical stimulation on neurons. Other cells of the CNS such as microglia, astrocytes, oligodendrocytes, and vascular endothelial cells have been understudied in terms of their response to stimulation. The varied and critical functions of these cell types are now beginning to be better understood, and their vital roles in brain function in both health and disease are becoming better appreciated. To shed light on the importance of the way electrical stimulation as distinct from device implantation impacts non-neuronal cell types, this review will first summarize common stimulation modalities from the perspective of device design and stimulation parameters and how these different parameters have an impact on the physiological response. Following this, what is known about the responses of different cell types to different stimulation modalities will be summarized, drawing on findings from both clinical studies as well as clinically relevant animal models and in vitro systems.

Clinical Progress and Optimization of Information Processing in Artificial Visual Prostheses

Visual prostheses, used to assist in restoring functional vision to the visually impaired, convert captured external images into corresponding electrical stimulation patterns that are stimulated by implanted microelectrodes to induce phosphenes and eventually visual perception. Detecting and providing useful visual information to the prosthesis wearer under limited artificial vision has been an important concern in the field of visual prosthesis. Along with the development of prosthetic device design and stimulus encoding methods, researchers have explored the possibility of the application of computer vision by simulating visual perception under prosthetic vision. Effective image processing in computer vision is performed to optimize artificial visual information and improve the ability to restore various important visual functions in implant recipients, allowing them to better achieve their daily demands. This paper first reviews the recent clinical implantation of different types of visual prostheses, summarizes the artificial visual perception of implant recipients, and especially focuses on its irregularities, such as dropout and distorted phosphenes. Then, the important aspects of computer vision in the optimization of visual information processing are reviewed, and the possibilities and shortcomings of these solutions are discussed. Ultimately, the development direction and emphasis issues for improving the performance of visual prosthesis devices are summarized.

End-to-end optimization of prosthetic vision

Neural prosthetics may provide a promising solution to restore visual perception in some forms of blindness. The restored prosthetic percept is rudimentary compared to normal vision and can be optimized with a variety of image preprocessing techniques to maximize relevant information transfer. Extracting the most useful features from a visual scene is a nontrivial task and optimal preprocessing choices strongly depend on the context. Despite rapid advancements in deep learning, research currently faces a difficult challenge in finding a general and automated preprocessing strategy that can be tailored to specific tasks or user requirements. In this paper, we present a novel deep learning approach that explicitly addresses this issue by optimizing the entire process of phosphene generation in an end-to-end fashion. The proposed model is based on a deep auto-encoder architecture and includes a highly adjustable simulation module of prosthetic vision. In computational validation experiments, we show that such an approach is able to automatically find a task-specific stimulation protocol. The results of these proof-of-principle experiments illustrate the potential of end-to-end optimization for prosthetic vision. The presented approach is highly modular and our approach could be extended to automated dynamic optimization of prosthetic vision for everyday tasks, given any specific constraints, accommodating individual requirements of the end-user.

Real-world indoor mobility with simulated prosthetic vision: The benefits and feasibility of contour-based scene simplification at different phosphene resolutions

Neuroprosthetic implants are a promising technology for restoring some form of vision in people with visual impairments via electrical neurostimulation in the visual pathway. Although an artificially generated prosthetic percept is relatively limited compared with normal vision, it may provide some elementary perception of the surroundings, re-enabling daily living functionality. For mobility in particular, various studies have investigated the benefits of visual neuroprosthetics in a simulated prosthetic vision paradigm with varying outcomes. The previous literature suggests that scene simplification via image processing, and particularly contour extraction, may potentially improve the mobility performance in a virtual environment. In the current simulation study with sighted participants, we explore both the theoretically attainable benefits of strict scene simplification in an indoor environment by controlling the environmental complexity, as well as the practically achieved improvement with a deep learning-based surface boundary detection implementation compared with traditional edge detection. A simulated electrode resolution of 26 × 26 was found to provide sufficient information for mobility in a simple environment. Our results suggest that, for a lower number of implanted electrodes, the removal of background textures and within-surface gradients may be beneficial in theory. However, the deep learning-based implementation for surface boundary detection did not improve mobility performance in the current study. Furthermore, our findings indicate that, for a greater number of electrodes, the removal of within-surface gradients and background textures may deteriorate, rather than improve, mobility. Therefore, finding a balanced amount of scene simplification requires a careful tradeoff between informativity and interpretability that may depend on the number of implanted electrodes.

The impact of synchronous versus asynchronous electrical stimulation in artificial vision

Visual prosthesis devices designed to restore sight to the blind have been under development in the laboratory for several decades. Clinical translation continues to be challenging, due in part to gaps in our understanding of critical parameters such as how phosphenes, the electrically-generated pixels of artificial vision, can be combined to form images. In this review we explore the effects that synchronous and asynchronous electrical stimulation across multiple electrodes have in evoking phosphenes. Understanding how electrical patterns influence phosphene generation to control object binding and perception of visual form is fundamental to creation of a clinically successful prosthesis.

Visual cortical prosthesis: an electrical perspective

The electrical stimulation of the visual cortices has the potential to restore vision to blind individuals. Until now, the results of visual cortical prosthetics have been limited as no prosthesis has restored a full working vision but the field has shown a renewed interest these last years, thanks to wireless and technological advances. However, several scientific and technical challenges are still open to achieve the therapeutic benefit expected by these new devices. One of the main challenges is the electrical stimulation of the brain itself. In this review, we analyse the results in electrode-based visual cortical prosthetics from the electrical point of view. We first describe what is known about the electrode-tissue interface and safety of electrical stimulation. Then we focus on the psychophysics of prosthetic vision and the state-of-the-art on the interplay between the electrical stimulation of the visual cortex and the phosphene perception. Lastly, we discuss the challenges and perspectives of visual cortex electrical stimulation and electrode array design to develop the new generation implantable cortical visual prostheses.

Advances in Neuroscience, Not Devices, Will Determine the Effectiveness of Visual Prostheses

Background: Innovations in engineering and neuroscience have enabled the development of sophisticated visual prosthetic devices. In clinical trials, these devices have provided visual acuities as high as 20/460, enabled coarse navigation, and even allowed for reading of short words. However, long-term commercial viability arguably rests on attaining even better vision and more definitive improvements in tasks of daily living and quality of life. Purpose: Here we review technological and biological obstacles in the implementation of visual prosthetics. Conclusions: Research in the visual prosthetic field has tackled significant technical challenges, including biocompatibility, signal spread through neural tissue, and inadvertent activation of passing axons; however, significant gaps in knowledge remain in the realm of neuroscience, including the neural code of vision and visual plasticity. We assert that further optimization of prosthetic devices alone will not provide markedly improved visual outcomes without significant advances in our understanding of neuroscience.

Dynamic Stimulation of Visual Cortex Produces Form Vision in Sighted and Blind Humans

A visual cortical prosthesis (VCP) has long been proposed as a strategy for restoring useful vision to the blind, under the assumption that visual percepts of small spots of light produced with electrical stimulation of visual cortex (phosphenes) will combine into coherent percepts of visual forms, like pixels on a video screen. We tested an alternative strategy in which shapes were traced on the surface of visual cortex by stimulating electrodes in dynamic sequence. In both sighted and blind participants, dynamic stimulation enabled accurate recognition of letter shapes predicted by the brain's spatial map of the visual world. Forms were presented and recognized rapidly by blind participants, up to 86 forms per minute. These findings demonstrate that a brain prosthetic can produce coherent percepts of visual forms.

Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex

Blindness affects 40 million people across the world. A neuroprosthesis could one day restore functional vision in the blind. We implanted a 1024-channel prosthesis in areas V1 and V4 of the visual cortex of monkeys and used electrical stimulation to elicit percepts of dots of light (called phosphenes) on hundreds of electrodes, the locations of which matched the receptive fields of the stimulated neurons. Activity in area V4 predicted phosphene percepts that were elicited in V1. We simultaneously stimulated multiple electrodes to impose visible patterns composed of a number of phosphenes. The monkeys immediately recognized them as simple shapes, motions, or letters. These results demonstrate the potential of electrical stimulation to restore functional, life-enhancing vision in the blind.

Neurolight: A Deep Learning Neural Interface for Cortical Visual Prostheses

Visual neuroprosthesis, that provide electrical stimulation along several sites of the human visual system, constitute a potential tool for vision restoration for the blind. Scientific and technological progress in the fields of neural engineering and artificial vision comes with new theories and tools that, along with the dawn of modern artificial intelligence, constitute a promising framework for the further development of neurotechnology. In the framework of the development of a Cortical Visual Neuroprosthesis for the blind (CORTIVIS), we are now facing the challenge of developing not only computationally powerful tools and flexible approaches that will allow us to provide some degree of functional vision to individuals who are profoundly blind. In this work, we propose a general neuroprosthesis framework composed of several task-oriented and visual encoding modules. We address the development and implementation of computational models of the firing rates of retinal ganglion cells and design a tool — Neurolight — that allows these models to be interfaced with intracortical microelectrodes in order to create electrical stimulation patterns that can evoke useful perceptions. In addition, the developed framework allows the deployment of a diverse array of state-of-the-art deep-learning techniques for task-oriented and general image pre-processing, such as semantic segmentation and object detection in our system’s pipeline. To the best of our knowledge, this constitutes the first deep-learning-based system designed to directly interface with the visual brain through an intracortical microelectrode array. We implement the complete pipeline, from obtaining a video stream to developing and deploying task-oriented deep-learning models and predictive models of retinal ganglion cells’ encoding of visual inputs under the control of a neurostimulation device able to send electrical train pulses to a microelectrode array implanted at the visual cortex.

Injecting Information into the Mammalian Cortex: Progress, Challenges, and Promise

For 150 years artificial stimulation has been used to study the function of the nervous system. Such stimulation-whether electrical or optogenetic-eventually may be used in neuroprosthetic devices to replace lost sensory inputs and to otherwise introduce information into the nervous system. Efforts toward this goal can be classified broadly as either biomimetic or arbitrary. Biomimetic stimulation aims to mimic patterns of natural neural activity, so that the subject immediately experiences the artificial stimulation as if it were natural sensation. Arbitrary stimulation, in contrast, makes no attempt to mimic natural patterns of neural activity. Instead, different stimuli-at different locations and/or in different patterns-are assigned different meanings randomly. The subject's time and effort then are required to learn to interpret different stimuli, a process that engages the brain's inherent plasticity. Here we will examine progress in using artificial stimulation to inject information into the cerebral cortex and discuss the challenges for and the promise of future development.

A preliminary implementation of an active intraocular prosthesis as a new image acquisition device for a cortical visual prosthesis

An active intraocular prosthesis is herein proposed as a new image acquisition device for a cortical visual prosthesis. A conventional intraocular prosthesis is a passive device that helps blind patients underwent eye enucleation to maintain the shape of an eyeball. In contrast, an active intraocular prosthesis, which works as an implantable wireless camera, can capture real-time images and transmit them to a cortical visual prosthesis to restore partial vision of the patients. This active device has distinct advantages in that it can garner a variety of image information while focusing on objects in accordance with natural eye movements, compared with a glasses-mounted camera and implanted micro-photodiodes in typical artificial vision systems. Coated with an epoxy and sealed by an elastomer for biocompatibility as well as durability, the active intraocular prosthesis was fabricated in a spherical form miniaturized enough to be inserted into the eye. Its operation was evaluated by wireless image acquisition displaying a processed gray-scale image. Furthermore, signal-to-noise ratio measurements were conducted to find a reliable communication range of the fabricated prosthesis, while it was covered by an 8-mm-thick biological medium that mimicked in vivo environments. In conclusion, the feasibility of the active intraocular prosthesis to cooperate with a cortical visual prosthesis is discussed.

Sound- and current-driven laminar profiles and their application method mimicking acoustic responses in the mouse auditory cortex in vivo

The local application of electrical currents to the cortex is one of the most commonly used techniques to activate neurons, and this intracortical stimulation (ICS) could potentially lead to new types of neuroprosthetic devices that can be directly applied to the cortex. To identify whether ICS-activated circuits are physiological vs. profoundly artificial, it is necessary to record in vivo the responses of the same neuronal population to both natural sensory stimuli and artificial electric stimuli. However, few studies have extensively reported simultaneous electrophysiological recordings combined with ICS. Here, we evaluated the similarity between sound- and ICS-driven cortical response patterns in different cortical layers. In the mouse auditory cortex, we performed laminar recordings using 16-channel silicon electrodes and ICS using sharp glass-pipette electrodes containing biocytin for layer identification. In different cortical depths, short current pulses were delivered in vivo to mice under urethane anesthesia. For the recorded data, we mainly analyzed properties of local field potentials and current source densities (CSDs). We demonstrated that electrical stimulation evoked different excitation patterns according to the stimulated cortical layer; responses to electric stimuli in layer 4 were most likely to mimic acoustic responses. Next, we proposed a CSD-based stimulation method to artificially synthesize sound-driven responses, using an approximation method associated with a linear combination of CSD patterns electrically stimulated in the different cortical layers. The result indicates that synthesized responses were consistent with the canonical model of sound processing. Using these approaches, we provide a new technique in which natural sound-driven responses can be mimicked by well-designed computational stimulation pattern sequences in a layer-dependent manner. These findings may aid in the future development of an electrical stimulation methodology for a cortical prosthesis.

Residual vision activation and the brain-eye-vascular triad: Dysregulation, plasticity and restoration in low vision and blindness - a review

Vision loss due to ocular diseases such as glaucoma, optic neuropathy, macular degeneration, or diabetic retinopathy, are generally considered an exclusive affair of the retina and/or optic nerve. However, the brain, through multiple indirect influences, has also a major impact on functional visual impairment. Such indirect influences include intracerebral pressure, eye movements, top-down modulation (attention, cognition), and emotionally triggered stress hormone release affecting blood vessel dysregulation. Therefore, vision loss should be viewed as the result of multiple interactions within a “brain-eye-vascular triad”, and several eye diseases may also be considered as brain diseases in disguise. While the brain is part of the problem, it can also be part of the solution. Neuronal networks of the brain can “amplify” residual vision through neuroplasticity changes of local and global functional connectivity by activating, modulating and strengthening residual visual signals. The activation of residual vision can be achieved by different means such as vision restoration training, non-invasive brain stimulation, or blood flow enhancing medications. Modulating brain functional networks and improving vascular regulation may offer new opportunities to recover or restore low vision by increasing visual field size, visual acuity and overall functional vision. Hence, neuroscience offers new insights to better understand vision loss, and modulating brain and vascular function is a promising source for new opportunities to activate residual vision to achieve restoration and recovery to improve quality of live in patients suffering from low vision.

Development of visual Neuroprostheses: trends and challenges

Visual prostheses are implantable medical devices that are able to provide some degree of vision to individuals who are blind. This research field is a challenging subject in both ophthalmology and basic science that has progressed to a point where there are already several commercially available devices. However, at present, these devices are only able to restore a very limited vision, with relatively low spatial resolution. Furthermore, there are still many other open scientific and technical challenges that need to be solved to achieve the therapeutic benefits envisioned by these new technologies. This paper provides a brief overview of significant developments in this field and introduces some of the technical and biological challenges that still need to be overcome to optimize their therapeutic success, including long-term viability and biocompatibility of stimulating electrodes, the selection of appropriate patients for each artificial vision approach, a better understanding of brain plasticity and the development of rehabilitative strategies specifically tailored for each patient.