Broadband sensory networks with locally stored responsivities for neuromorphic machine vision

Binary Neural Network Based on a Programmable Graphene/Si Schottky Diode for In-Sensor Processing Image Sensors

Recent advancements in in-sensor computing technology have demonstrated significant advantages in time latency and energy efficiency in visual information processing through device-level integration of photosensing and neuromorphic computing. However, current implementations face challenges due to their single-layer architecture, creating an urgent demand for the development of devices that integrate front-end in-sensor processing with back-end computing layers. Here, we report a programmable graphene/Si Schottky diode (PGSSD) featuring gate-voltage-programmed photoresponsivity and rectification direction. The programmability of the photoresponsivity enables the application of reconfigurable convolution kernels to implement in-sensor convolution of optical images. Simultaneously, the programmable rectification direction permits analog-domain execution of quasi-binary multiply-accumulate (MAC) operations. Based on these capabilities, we constructed a complete binary neural network (BNN) using the PGSSDs and demonstrated its application for image recognition. The BNN combines front-end convolution processing and back-end computing layers, achieving an inference accuracy of 98.35% on the MNIST database.

Bidirectional Phosphorescent Neuroplasticity for All-Optical Neurovision

All-optical neuromorphics that can capture, process, and output photonic signals are in prospect to advance optical computing and imaging. Bidirectional neuroplasticity is essential for executing training and inference in optical neural networks, but most of the all-optical hardware only exhibits unidirectional weight modulation. Here, we explore bidirectional neuroplasticity in carbon dot phosphorescence (CDP) with potentiation and depression synaptic behaviors capable of neuroregulation for photonic intensity. This function enables the CDP as a neuroconverter to convert pulse light into excitatory and inhibitory light output for neuromorphic vision owing to the delayed release and superimposition dynamics of excitons in persistent phosphorescence, which allows for image digitization or direct observation. By integration with an optical neural network, the real-time motion tracking of light spots, including trajectory, direction, and speed, can be recorded and recognized, with a high accuracy of 96%. Such phosphor-based neuromorphics can be extended to other phosphorescent architectures for all-optical imaging and computing.

Heterointerface-Modulated Synthetic Synapses Exhibiting Complex Multiscale Plasticity

An asymmetric dual-gate heterointerface-regulated artificial synapse (HRAS) is developed, utilizing a main gate with distinct ion concentrations and a lateral gate to receive synaptic pulses, and through dielectric coupling and ionic effects, formed indium tin zinc oxide (ITZO) dual-interface channels that allow precise control over channel charge, thereby simulating multi-level coordinated actions of dual-neurotransmitters. The lateral modulation of the lateral gate significantly regulates ionic effects, achieving the intricate interplay among lateral inhibition/enhancement and short-/long-term plasticity at a multi-level scale for the first time. This interplay enables the HRAS device to simulate frequency-dependent image filtering and spike number-dependent dynamic visual persistence. By combining temporal synaptic inputs with lateral modulation, HRAS harnesses spatiotemporal properties for bio-inspired cryptographic applications, offering a versatile device-level platform for secure information processing. Furthermore, a novel dual-gate input neural network architecture based on HRAS has been proposed, which aids in weight update and demonstrates enhanced recognition capabilities in neural network tasks, highlighting its role in bio-inspired computing.

Bio-Inspired Retina by Regulating Ion-Confined Transport in Hydrogels

The effective and precise processing of visual information by the human eye primarily relies on the diverse contrasting functions achieved through synaptic regulation of ion transport in the retina. Developing a bio-inspired retina that uses ions as information carriers can more accurately replicate retina's natural signal processing capabilities, enabling high-performance machine vision. Herein, an ion-confined transport strategy is proposed to construct a bio-inspired retina by developing artificial synapses with inhibitory and excitatory contrasting functions. By fine-tuning the ionic hydrogel structures to control ion transport across the heterogeneous interfaces, inhibitory and excitatory synapses are realized to negatively or positively modulate the optical signal. The integration of these synapses facilitates advanced tasks such as image recognition and motion analysis. Moreover, as a proof of concept, guiding robot vehicles to perform path planning is demonstrated. This work offers a new idea for constructing the bio-inspired retina by precisely regulating ion transport, allowing it to reach a level closer to the biological retina.

Near Infrared Light-Based Non-Contact Sensing System for Robotics Applications

With the development of artificial intelligence and the Internet of Things, non-contact sensors are expected to realize complex human-computer interaction. However, current non-contact sensors are mainly limited by accuracy and stability. Herein, an intelligent infrared photothermal non-contact sensing system is developed that provides long-distance and high-accuracy non-contact sensing. A black phosphorus (BP)-based composite organogel is designed, which exhibits excellent photothermal properties and environmental stability, as the active material. This material can detect patterns created by near-infrared (NIR) light through various patterned masks monitored by an infrared thermal imager. The constructed non-contact sensing system is capable of accurately recognizing 26 letters with an impressive accuracy rate of 99.4%. Furthermore, even small size non-contact sensors can maintain high sensitivity and stability across a wide temperature range, at long working distances, and under different current intensities and dark conditions, demonstrating exceptional robustness. Combined with machine learning method, it is demonstrated that the non-contact sensing system excels in pattern recognition and human-computer interaction. These features highlight its potential applications in intelligent robotics and remote control systems.

Ionic Device: From Neuromorphic Computing to Interfacing with the Brain

In living organisms, the modulation of ion conductivity in ion channels of neuron cells enables intelligent behaviors, such as generating, transmitting, and storing neural signals. Drawing inspiration from these natural processes, researchers have fabricated ionic devices that replicate the functions of the nervous system. However, this field remains in its infancy, necessitating extensive foundational research in ionic device preparation, algorithm development, and biological interaction. This review summarizes recently developed neuromorphic ionic devices into three categories based on the materials states: liquid, semi-solid, and solid. The neural network algorithms embedded in these devices for neuromorphic computing are introduced, and future directions for the development of bidirectional human-computer interaction and hybrid human-computer intelligence are discussed.

Coexistence of WORM and RRAM Resistive Switching in Coumarin Derivatives: A Comprehensive Performance Analysis

Over the past few years, organic small molecules (OSM) having a π-conjugated heteroatomic aromatic backbone along with terminal donor-acceptor (D-A) groups have emerged as one of the most promising materials for organic resistive switching (ORS) devices. In this research, the resistive switching (RS) properties of two rationally synthesized coumarin derivatives, 7-(2-(benzylamino)ethoxy)-4-methyl-2H-chromen-2-one (CAMN1) and 7-(2-(4-methoxyphenylamino)ethoxy)-4-methyl-2H-chromen-2-one (CAMN2), have been exhaustively studied. The CAMN1-based ORS device exhibited WORM RS behavior with an excellent device yield of 97.22%, while the CAMN2-based device showed both WORM as well as RRAM RS behavior depending on the compliance current (CC) with a perfect device yield of 100%. Both devices exhibited superior read endurance on the order of 104 as well as a retention time of at least 3 × 104 s with a very good memory window of the order of 104 or more. Moreover, both devices exhibited superior long-term physical and thermal stability. The cyclability of the CAMN2-based device in the RRAM mode of operation was found to be 116 cycles. DFT-based calculations as well as absorption spectroscopic studies reveal the role of the intra/intermolecular charge transfer (CT) in the RS behavior of both the devices. Moreover, the presence of the methoxy (-OCH3) group in the CAMN2 molecule has been identified as the key reason behind the observed difference in the RS behaviors of the two molecules.

Adaptive In-Sensor Computing for Enhanced Feature Perception and Broadband Image Restoration

Traditional imaging systems struggle in weak or complex lighting environments due to their fixed spectral responses, resulting in spectral mismatches and degraded image quality. To address these challenges, a bioinspired adaptive broadband image sensor is developed. This innovative sensor leverages a meticulously designed type-I heterojunction alignment of 0D perovskite quantum dots (PQDs) and 2D black phosphorus (BP). This configuration enables efficient carrier injection control and advanced computing capabilities within an integrated phototransistor array. The sensor's unique responses to both visible and infrared (IR) light facilitate selective enhancement and precise feature extraction under varying lighting conditions. Furthermore, it supports real-time convolution and image restoration within a convolutional autoencoder (CAE) network, effectively countering image degradation by capturing spectral features. Remarkably, the hardware responsivity weights perform comparably to software-trained weights, achieving an image restoration accuracy of over 85%. This approach offers a robust and versatile solution for machine vision applications that demand precise and adaptive imaging in dynamic lighting environments.

Ion Intercalation-Enabled Reconfigurable Photodetector with Low Programming Voltage and Broadband Response Based on Van Der Waals Tin Disulfide

Artificial neural networks with integrated sensing and computing capabilities, leveraging reconfigurable optoelectronics, can effectively emulate biological neural networks, thereby enabling rapid and efficient information processing. However, realizing reconfigurable photoresponsivity is often blocked by the requirement for high programming voltages and the limits of the detection spectrum range. This greatly restricts the progress of energy-efficient and precise neuromorphic vision sensing. Herein, a reconfigurable photodetector with low programming voltage and broadband response is presented via in situ intercalation of Cu+ ions into the van der Waals (vdW) gaps of thermoelectric 2D material SnS2. Interestingly, the vdW gaps provide an ionic transport channel with lower energy barriers compared to oxide-based memristors, resulting in a low programming voltage (0.5 V). Furthermore, reversible conversion of photo-detection is achieved from photovoltaic to photo-thermoelectric (PTE) mode via voltage-controlled ion distribution, which modulates the phonon scattering rate in the neighboring SnS2 layer. As a result, the response spectrum switches from visible (532 nm) to long-wave infrared (10 µm) with an on/off ratio as high as 104. Thus, dual-mode conversion and broadband detection functionality in reconfigurable imaging are realized, suggesting a potential pathway for the development of highly energy-efficient reconfigurable optoelectronics with a spectrum far beyond human vision.

In situ training of an in-sensor artificial neural network based on ferroelectric photosensors

In-sensor computing has emerged as an ultrafast and low-power technique for next-generation machine vision. However, in situ training of in-sensor computing systems remains challenging due to the demands for both high-performance devices and efficient programming schemes. Here, we experimentally demonstrate the in situ training of an in-sensor artificial neural network (ANN) based on ferroelectric photosensors (FE-PSs). Our FE-PS exhibits self-powered, fast (<30 μs), and multilevel (>4 bits) photoresponses, as well as long retention (50 days), high endurance (109), high write speed (100 ns), and small cycle-to-cycle and device-to-device variations (~0.66% and ~2.72%, respectively), all of which are desirable for the in situ training. Additionally, a bi-directional closed-loop programming scheme is developed, achieving a precise and efficient weight update for the FE-PS. Using this programming scheme, an in-sensor ANN based on the FE-PSs is trained in situ to recognize traffic signs for commanding a prototype autonomous vehicle. Moreover, this in-sensor ANN operates 50 times faster than a von Neumann machine vision system. This study paves the way for the development of in-sensor computing systems with in situ training capability, which may find applications in new data-streaming machine vision tasks.

Perovskite retinomorphic image sensor for embodied intelligent vision

Retinomorphic systems that can see, recognize, and respond to real-time environmental information will extend the complexity and range of tasks that an exoskeleton robot can perform to better assist physically disabled people. However, the lack of ultrasensitive, reconfigurable, and large-scale integratable retinomorphic devices and advanced edge-processing algorithms makes it difficult to realize retinomorphic hardware. Here, we report the retinomorphic hardware prototype with a 4096-pixel perovskite image sensor array as core module to endow embodied intelligent vision functionalities. The retinomorphic sensor array, using a one photodetector-one transistor geometry to resemble retinal circuit with broadband, ultrahigh, and reconfigurable photoresponsivities, executes both adaptive imaging with a contrast enhancement of ~620% under a dim-lit intensity of 10 microwatts per square centimeter and an instantaneous one-dimensional feature extraction algorithm to decompose the origin visual scenarios into parsimoniously encoded spatiotemporal information. This retinomorphic system endows embodied intelligence with adaptive imaging, in situ processing, and decision-making capabilities and promises enormous potential for autonomous robot applications.

Interface-engineered non-volatile visible-blind photodetector for in-sensor computing

Ultraviolet (UV) detection is extensively used in a variety of applications. However, the storage and processing of information after detection require multiple components, resulting in increased energy consumption and data transmission latency. In this paper, a reconfigurable UV photodetector based on CeO2/SrTiO3 heterostructures is demonstrated with in-sensor computing capabilities achieved through interface engineering. We show that the non-volatile storage capability of the device could be significantly improved by the introduction of an oxygen reservoir. A photodetector array operated as a single-layer neural network was constructed, in which edge detection and pattern recognition were realized without the need for external memory and computing units. The location and classification of corona discharges in real-world environments were also simulated and achieved an accuracy of 100%. The approach proposed here offers promising avenues and material options for creating non-volatile smart photodetectors.

A Reconfigurable Polarimetric Photodetector Based on the MoS2/PdSe2 Heterostructure with a Charge-Trap Gate Stack

Besides the intensity and wavelength, the ability to analyze the optical polarization of detected light can provide a new degree of freedom for numerous applications, such as object recognition, biomedical applications, environmental monitoring, and remote sensing imaging. However, conventional filter-integrated polarimetric sensing systems require complex optical components and a complicated fabrication process, severely limiting their on-chip miniaturization and functionalities. Herein, the reconfigurable polarimetric photodetection with photovoltaic mode is developed based on a few-layer MoS2/PdSe2 heterostructure channel and a charge-trap structure composed of Al2O3/HfO2/Al2O3 (AHA)-stacked dielectrics. Because of the remarkable charge-trapping ability of carriers in the AHA stack, the MoS2/PdSe2 channel exhibits a high program/erase current ratio of 105 and a memory window exceeding 20 V. Moreover, the photovoltaic mode of the MoS2/PdSe2 Schottky diode can be operated and manipulable, resulting in high and distinct responsivities in the visible broadband. Interestingly, the linear polarization of the device can be modulated under program/erase states, enabling the reconfigurable capability of linearly polarized photodetection. This study demonstrates a new prototype heterostructure-based photodetector with the capability of both tunable responsivity and linear polarization, demonstrating great potential application toward reconfigurable photosensing and polarization-resolved imaging applications.

Charge-Selective 2D Heterointerface-Driven Multifunctional Floating Gate Memory for In Situ Sensing-Memory-Computing

Flash memory, dominating data storage due to its substantial storage density and cost efficiency, faces limitations such as slow response, high operating voltages, absence of optoelectronic response, etc., hindering the development of sensing-memory-computing capability. Here, we present an ultrathin platinum disulfide (PtS2)/hexagonal boron nitride (hBN)/multilayer graphene (MLG) van der Waals heterojunction with atomically sharp interfaces, achieving selective charge tunneling behavior and demonstrating ultrafast operations, a high on/off ratio (108), extremely low operating voltage, robust endurance (105 cycles), and retention exceeding 10 years. Additionally, we achieve highly linear synaptic potentiation and depression, and observe the reversibly gate-tunable transitions between positive and negative photoconductivity. Furthermore, we employed the VGG11 neural network for in situ trained in-sensor-memory-computing to classify the CIFAR-10 data set, pushing accuracy levels comparable to pure digital systems. This work could pave the way for seamlessly integrated sensing, memory, and computing capabilities for diverse edge computing.

Advancements in Nanowire-Based Devices for Neuromorphic Computing: A Review

Neuromorphic computing, inspired by the highly interconnected and energy-efficient way the human brain processes information, has emerged as a promising technology for post-Moore's law era. This emerging technology can emulate the structures and the functions of the human brain and is expected to overcome the fundamental limitation of the current von Neumann computing architecture. Neuromorphic devices stand out as the key components of future electronic systems, exhibiting potential in shaping the landscape of neuromorphic computing. Especially, nanowire (NW)-based neuromorphic devices, with their advantages of high integration, high-speed computing, and low power consumption, have recently emerged as candidates for neuromorphic computing technology. Here, a critical overview of the current development and relevant research in the field of NW-based neuromorphic devices are provided. Neuromorphic devices based on different NW materials are comprehensively discussed, including Ag NW-based, organic NW-based, metal oxide NW-based, and semiconductor NW-based devices. Finally, as a foresight perspective, the potentials and the challenges of these NW-based neuromorphic devices for use as future brain-like electronics are discussed.

An Aquatic Autonomic Nervous System

Reproducing human nervous systems with endogenous mechanisms has attracted increasing attention, driven by its great potential in streamlining the neuro-electronic interfaces with bilateral signaling. Here, an artificial aquatic autonomic nervous system (ANS) with switchable excitatory/inhibitory characteristics and acetylcholine (ACh)-mediated plasticity is reported based on the newly emerged organic photoelectrochemical transistor (OPECT). Under the modulation of spatial light and ACh, the system exhibits an immediate switch between excitation and inhibition, and many pulse patterns as well as advanced ANS functions are mimicked. To demonstrate its potential usage, the artificial ANS is then utilized to control artificial pupils and muscles to emulate real biological responses during an emergency. In contrast to previous solid-state attempts, this ANS is aqueous compatible just like biological nervous systems, which are capable of real neurotransmitter mediation.

Object Motion Detection Enabled by Reconfigurable Neuromorphic Vision Sensor under Ferroelectric Modulation

Increasing the demand for object motion detection (OMD) requires shifts of reducing redundancy, heightened power efficiency, and precise programming capabilities to ensure consistency and accuracy. Drawing inspiration from object motion-sensitive ganglion cells, we propose an OMD vision sensor with a simple device structure of a WSe2 homojunction modulated by a ferroelectric copolymer. Under optical mode and intermediate ferroelectric modulation, the vision sensor can generate progressive and bidirectional photocurrents with discrete multistates under zero power consumption. This design enables reconfigurable devices to emulate long-term potentiation and depression for synaptic weights updating, which exhibit 82 states (more than 6 bits) with a uniform step of 6 pA. Such OMD devices also demonstrate nonvolatility, reversibility, symmetry, and ultrahigh linearity, achieving a fitted R2 of 0.999 and nonlinearity values of 0.01/-0.01. Thus, a vision sensor could implement motion detection by sensing only dynamic information based on the brightness difference between frames, while eliminating redundant data from static scenes. Additionally, the neural network utilizing a linear result can recognize the essential moving information with a high recognition accuracy of 96.8%. We also present the scalable potential via a uniform 3 × 3 neuromorphic vision sensor array. Our work offers a platform to achieve motion detection based on controllable and energy-efficient ferroelectric programmability.

Multifunctional human visual pathway-replicated hardware based on 2D materials

Artificial visual system empowered by 2D materials-based hardware simulates the functionalities of the human visual system, leading the forefront of artificial intelligence vision. However, retina-mimicked hardware that has not yet fully emulated the neural circuits of visual pathways is restricted from realizing more complex and special functions. In this work, we proposed a human visual pathway-replicated hardware that consists of crossbar arrays with split floating gate 2D tungsten diselenide (WSe2) unit devices that simulate the retina and visual cortex, and related connective peripheral circuits that replicate connectomics between the retina and visual cortex. This hardware experimentally displays advanced multi-functions of red-green color-blindness processing, low-power shape recognition, and self-driven motion tracking, promoting the development of machine vision, driverless technology, brain-computer interfaces, and intelligent robotics.

Photoresponsive Two-Dimensional Magnetic Junctions for Reconfigurable In-Memory Sensing

Magnetic tunneling junctions (MTJs) lie in the core of magnetic random access memory, holding promise in integrating memory and computing to reduce hardware complexity, transition latency, and power consumption. However, traditional MTJs are insensitive to light, limiting their functionality in in-memory sensing─a crucial component for machine vision systems in artificial intelligence applications. Herein, the convergence of magnetic memory with optical sensing capabilities is achieved in the all-two-dimensional (2D) magnetic junction Fe3GaTe2/WSe2/Fe3GaTe2, which combines 2D magnetism and optoelectronic properties. The clean intrinsic band gap and prominent photoresponse of interlayer WSe2 endow the tunneling barrier with optical tunability. The on-off states of junctions and the magnetoresistance can be flexibly controlled by the intensity of the optical signal at room temperature. Based on the optical-tunable magnetoresistance in all-2D magnetic junctions, a machine vision system with the architecture of in-memory sensing and computing is constructed, which possesses high performance in image recognition. Our work exhibits the advantages of 2D magneto-electronic devices and extends the application scenarios of magnetic memory devices in artificial intelligence.

A Reconfigurable All-Optical-Controlled Synaptic Device for Neuromorphic Computing Applications

Retina-inspired visual sensors play a crucial role in the realization of neuromorphic visual systems. Nevertheless, significant obstacles persist in the pursuit of achieving bidirectional synaptic behavior and attaining high performance in the context of photostimulation. In this study, we propose a reconfigurable all-optical controlled synaptic device based on the IGZO/SnO/SnS heterostructure, which integrates sensing, storage and processing functions. Relying on the simple heterojunction stack structure and the role of energy band engineering, synaptic excitatory and inhibitory behaviors can be observed under the light stimulation of ultraviolet (266 nm) and visible light (405, 520 and 658 nm) without additional voltage modulation. In particular, junction field-effect transistors based on the IGZO/SnO/SnS heterostructure were fabricated to elucidate the underlying bidirectional photoresponse mechanism. In addition to optical signal processing, an artificial neural network simulator based on the optoelectrical synapse was trained and recognized handwritten numerals with a recognition rate of 91%. Furthermore, we prepared an 8 × 8 optoelectrical synaptic array and successfully demonstrated the process of perception and memory for image recognition in the human brain, as well as simulated the situation of damage to the retina by ultraviolet light. This work provides an effective strategy for the development of high-performance all-optical controlled optoelectronic synapses and a practical approach to the design of multifunctional artificial neural vision systems.

All-Photolithography Fabrication of Ion-Gated Flexible Organic Transistor Array for Multimode Neuromorphic Computing

Organic ion-gated transistors (OIGTs) demonstrate commendable performance for versatile neuromorphic systems. However, due to the fragility of organic materials to organic solvents, efficient and reliable all-photolithography methods for scalable manufacturing of high-density OIGT arrays with multimode neuromorphic functions are still missing, especially when all active layers are patterned in high-density. Here, a flexible high-density (9662 devices per cm2) OIGT array with high yield and minimal device-to-device variation is fabricated by a modified all-photolithography method. The unencapsulated flexible array can withstand 1000 times' bending at a radius of 1 mm, and 3 months' storage test in air, without obvious performance degradation. More interesting, the OIGTs can be configured between volatile and nonvolatile modes, suitable for constructing reservoir computing systems to achieve high accuracy in classifying handwritten digits with low training costs. This work proposes a promising design of organic and flexible electronics for affordable neuromorphic systems, encompassing both array and algorithm aspects.