In-sensor image memorization and encoding via optical neurons for bio-stimulus domain reduction toward visual cognitive processing

Binary Neural Network Based on a Programmable Graphene/Si Schottky Diode for In-Sensor Processing Image Sensors

Recent advancements in in-sensor computing technology have demonstrated significant advantages in time latency and energy efficiency in visual information processing through device-level integration of photosensing and neuromorphic computing. However, current implementations face challenges due to their single-layer architecture, creating an urgent demand for the development of devices that integrate front-end in-sensor processing with back-end computing layers. Here, we report a programmable graphene/Si Schottky diode (PGSSD) featuring gate-voltage-programmed photoresponsivity and rectification direction. The programmability of the photoresponsivity enables the application of reconfigurable convolution kernels to implement in-sensor convolution of optical images. Simultaneously, the programmable rectification direction permits analog-domain execution of quasi-binary multiply-accumulate (MAC) operations. Based on these capabilities, we constructed a complete binary neural network (BNN) using the PGSSDs and demonstrated its application for image recognition. The BNN combines front-end convolution processing and back-end computing layers, achieving an inference accuracy of 98.35% on the MNIST database.

Near-Sensor Edge Computing System Enabled by a CMOS Compatible Photonic Integrated Circuit Platform Using Bilayer AlN/Si Waveguides

The rise of large-scale artificial intelligence (AI) models, such as ChatGPT, DeepSeek, and autonomous vehicle systems, has significantly advanced the boundaries of AI, enabling highly complex tasks in natural language processing, image recognition, and real-time decision-making. However, these models demand immense computational power and are often centralized, relying on cloud-based architectures with inherent limitations in latency, privacy, and energy efficiency. To address these challenges and bring AI closer to real-world applications, such as wearable health monitoring, robotics, and immersive virtual environments, innovative hardware solutions are urgently needed. This work introduces a near-sensor edge computing (NSEC) system, built on a bilayer AlN/Si waveguide platform, to provide real-time, energy-efficient AI capabilities at the edge. Leveraging the electro-optic properties of AlN microring resonators for photonic feature extraction, coupled with Si-based thermo-optic Mach-Zehnder interferometers for neural network computations, the system represents a transformative approach to AI hardware design. Demonstrated through multimodal gesture and gait analysis, the NSEC system achieves high classification accuracies of 96.77% for gestures and 98.31% for gaits, ultra-low latency (< 10 ns), and minimal energy consumption (< 0.34 pJ). This groundbreaking system bridges the gap between AI models and real-world applications, enabling efficient, privacy-preserving AI solutions for healthcare, robotics, and next-generation human-machine interfaces, marking a pivotal advancement in edge computing and AI deployment.

All-Optically Modulated In-Sensor Computing Device Based on Ionic-Conducting CuInP2Se6

Inspired by the human visual system, in-sensor computing has emerged as a promising approach to address growing demands for real-time image processing while overcoming constraints in computational resources. However, existing in-sensor computing optoelectronic devices still face challenges such as complex heterostructures or limited optical modulation for operational efficiency, restricting their practical use. Here, a simple two-terminal optoelectronic device has been fabricated using the 2D material CuInP2Se6, achieving neuromorphic functionalities through all-optical modulation. The device exhibits a tunable photoresponse across the visible spectrum (400 to 700 nm) and enables bidirectional conductance modulation in response to light stimuli, driven by the interaction between Cu⁺ ions and photogenerated electrons. It shows high linearity with 300 discrete conductance states under red, green, and blue light, enabling color-specific image feature extraction, processing, and recognition across three channels. This approach significantly enhances color image recognition accuracy by 4.6% when integrated with a three-channel convolutional neural network. Additionally, the bidirectional photoresponse allows for efficient noise suppression during color image preprocessing, leading to a 490% improvement in signal-to-noise ratio. These findings highlight the potential of CuInP2Se6-based architecture for robust performance, paving the way for in-sensor neuromorphic vision systems in artificial intelligence and biomimetic computing.

In-sensor multilevel image adjustment for high-clarity contour extraction using adjustable synaptic phototransistors

Robotic vision has traditionally relied on high-performance yet resource-intensive computing solutions, which necessitate high-throughput data transmission from vision sensors to remote computing servers, sacrificing energy efficiency and processing speed. A promising solution is data compaction through contour extraction, visualizing only the outlines of objects while eliminating superfluous backgrounds. Here, we introduce an in-sensor multilevel image adjustment method using adjustable synaptic phototransistors, enabling the capture of well-defined images with optimal brightness and contrast suitable for achieving high-clarity contour extraction. This is enabled by emulating dopamine-mediated neuronal excitability regulation mechanisms. Electrostatic gating effect either facilitates or inhibits time-dependent photocurrent accumulation, adjusting photo-responses to varying lighting conditions. Through excitatory and inhibitory modes, the adjustable synaptic phototransistor enhances visibility of dim and bright regions, respectively, facilitating distinct contour extraction and high-accuracy semantic segmentation. Evaluations using road images demonstrate improvement of both object detection accuracy and intersection over union, and compression of data volume.

Low-Power Memristor for Neuromorphic Computing: From Materials to Applications

As an emerging memory device, memristor shows great potential in neuromorphic computing applications due to its advantage of low power consumption. This review paper focuses on the application of low-power-based memristors in various aspects. The concept and structure of memristor devices are introduced. The selection of functional materials for low-power memristors is discussed, including ion transport materials, phase change materials, magnetoresistive materials, and ferroelectric materials. Two common types of memristor arrays, 1T1R and 1S1R crossbar arrays are introduced, and physical diagrams of edge computing memristor chips are discussed in detail. Potential applications of low-power memristors in advanced multi-value storage, digital logic gates, and analogue neuromorphic computing are summarized. Furthermore, the future challenges and outlook of neuromorphic computing based on memristor are deeply discussed.

Enhanced In-Sensor Computing with Spike Number-Dependent Plasticity Characteristics in an InGaSnO Optical Neuromorphic Device for Accelerating Machine Vision

In-sensor computing systems based on optical neuromorphic devices have attracted increasing attention to improve the efficiency and accuracy of machine vision systems. However, most materials used in optical neuromorphic devices exhibit spike timing-dependent plasticity (STDP) behavior in response to input light signals, leading to complex in-sensor computing and reduced machine vision accuracy. To address this issue, we introduce an indium gallium tin oxide (IGTO) semiconductor designed to enhance spike number-dependent plasticity (SNDP) in response to input light signals while eliminating the STDP behavior. Here, an IGTO-based optical neuromorphic device shows enhanced SNDP characteristics, which are attributed to the strong Sn-O bonding, as verified by photoemission spectroscopy (PES) analysis. The IGTO-based device consistently reaches the same conduction state after 8 light pulses regardless of the pulse timing and also achieves a conduction state based on the number of input light pulses even when 15 different pulse sets are applied. These results exhibit the SNDP characteristics of the IGTO-based device. Notably, in-sensor computing with the SNDP-enhanced device reduces multilayer perceptron (MLP) training time by 87.7% while achieving high classification accuracy. This study demonstrates that in-sensor computing systems with SNDP characteristics in optical neuromorphic devices have significant potential to accelerate machine learning for highly efficient machine vision systems.

In-Sensor Computing with Visual-Tactile Perception Enabled by Mechano-Optical Artificial Synapse

In-sensor computing paradigm holds the promise of realizing rapid and low-power signal processing. Constructing crossmodal in-sensor computing systems to emulate human sensory and recognition capabilities has been a persistent pursuit for developing humanoid robotics. Here, an artificial mechano-optical synapse is reported to implement in-sensor dynamic computing with visual-tactile perception. By employing mechanoluminescence (ML) material, direct conversion of the mechanical signals into light emission is achieved and the light is transported to an adjacent photostimulated luminescence (PSL) layer without pre- and post-irradiation. The PSL layer acts as a photon reservoir as well as a processing unit for achieving in-memory computing. The approach based on ML coupled with PSL material is different from traditional circuit-constrained methods, enabling remote operation and easy accessibility. Individual and synergistic plasticity are elaborately investigated under force and light pulses, including paired-pulse facilitation, learning behavior, and short-term and long-term memory. A multisensory neural network is built for processing the obtained handwritten patterns with a tablet consisting of the device, achieving a recognition accuracy of up to 92.5%. Moreover, material identification has been explored based on visual-tactile sensing, with an accuracy rate of 98.6%. This work provides a promising strategy to construct in-sensor computing systems with crossmodal integration and recognition.

DC field-biased multibit/analog artificial synapse featuring an additional degree of freedom for performance tuning

Multibit/analog artificial synapses are in demand for neuromorphic computing systems. A problem hindering the utilization of memristive artificial synapses in commercial neuromorphic systems is the rigidity of their functional parameters, plasticity in particular. Here, we report fabricating polycrystalline rutile-based memristive memory segments with Ti/poly-TiO2/Ti structures featuring multibit/analog storage and the first use of a tunable DC-biasing for synaptic plasticity adjustment from short- to long-term. The unbiased device is of short-term plasticity, positive biasing increases the remanence of the recorded events and the device gains long-term plasticity at a specific biasing level determined from the device geometry. The adjustability of the biasing field provides an additional degree of freedom allowing performance tuning; the paired-pulse facilitation index of the device is tuned by the biasing level adjustment providing further functional versatility. An appropriately biased segment provides more than 10 synaptic weight levels linearly depending on the number and duration of the stimulating spikes. The relationship with spike magnitude is exponential. The experimentally determined nonlinearity coefficient of the biased device for 50 potentiating spikes is comparable to the best published data. The spike-timing-dependent plasticity determined experimentally for the biased device in its long-term plasticity mode fits the mathematical relationship developed for biological synapses. Fabricated on a titanium metal foil, the produced memristors are sturdy and flexible making them suitable for wearable and implantable intelligent electronics. Our findings are anticipated to raise the potential of forming artificial synapses out of polycrystalline metal oxide thin films.

All-Inorganic Perovskite Quantum-Dot Optical Neuromorphic Synapses for Near-Sensor Colored Image Recognition

As the demand for the neuromorphic vision system in image recognition experiences rapid growth, it is imperative to develop advanced architectures capable of processing perceived data proximal to sensory terminals. This approach aims to reduce data movement between sensory and computing units, minimizing the need for data transfer and conversion at the sensor-processor interface. Here, an optical neuromorphic synaptic (ONS) device is demonstrated by homogeneously integrating optical-sensing and synaptic functionalities into a unified material platform, constructed exclusively by all-inorganic perovskite CsPbBr3 quantum dots (QDs). The dual functionality of each unit within the ONS device, which can be operated as either an optical sensor or a synaptic device depending on applied electrical polarity, provides significant advantages over previous heterogeneous integration methods, particularly regarding material selection, structural compatibility, and device fabrication complexity. The ONS device exhibits distinct wavelength responses essential for emulating colored image recognition capability inherent in the human visual system. Additionally, the seamless integration of electronics and photonics within a unified material system establishes a novel paradigm for optical retrieval, enabling real-time perception of the encoded status of the ONS device. These findings represent substantial advancements in near-sensor computing platforms and open a new horizon for all-inorganic perovskite optoelectronic technologies.

Emerging 2D materials hardware for in-sensor computing

The advent of the novel in-sensor/near-sensor computing paradigm significantly eliminates the need for frequent data transfer between sensory terminals and processing units by integrating sensing and computing functions into a single device. This approach surpasses the traditional configuration of separate sensing and processing units, thereby greatly simplifying system complexity. Two-dimensional materials (2DMs) show immense promise for implementing in-sensor computing systems owing to their exceptional material properties and the flexibility they offer in designing innovative device architectures with heterostructures. This review highlights recent progress and advancements in 2DM-based in-sensor computing research, summarizing the unique physical mechanisms that can be leveraged in 2DM-based devices to achieve sensory responses and the essential biomimetic synaptic characteristics for computing functions. Additionally, the potential applications of 2DM-based in-sensor computing systems are discussed and categorized. This review concludes with a perspective on future development directions for 2DM-based in-sensor computing.

Full van der Waals Ambipolar Ferroelectric Configurable Optical Hetero-Synapses for In-Sensor Computing

The rapid development of visual neuromorphic hardware can be attributed to their ability to capture, store and process optical signals from the environment. The main limitation of existing visual neuromorphic hardware is that the realization of complex functions is premised on the increase of manufacturing cost, hardware volume and energy consumption. In this study, we demonstrated an optical synaptic device based on a three-terminal van der Waals (vdW) heterojunction that can realize the sensing functions of light wavelength and intensity as well as short-term and long-term synaptic plasticity. In the image recognition task, we constructed an optical reservoir neural network (ORNN) and a visible light communication system (VLC) composed of this optical synaptic device. The ORNN has a recognition rate of up to 84.9% for 50 000 color images in 10 categories in the CIFAR-10 color image dataset, and the VLC system can achieve high-speed transmission with an ultra-low power consumption of only 0.4 nW. This work shows that through reasonable design, vdW heterojunction structures have great application potential in low-power multifunctional fusion application tasks such as visual bionics.

Bio-Plausible Multimodal Learning with Emerging Neuromorphic Devices

Multimodal machine learning, as a prospective advancement in artificial intelligence, endeavors to emulate the brain's multimodal learning abilities with the objective to enhance interactions with humans. However, this approach requires simultaneous processing of diverse types of data, leading to increased model complexity, longer training times, and higher energy consumption. Multimodal neuromorphic devices have the capability to preprocess spatio-temporal information from various physical signals into unified electrical signals with high information density, thereby enabling more biologically plausible multimodal learning with low complexity and high energy-efficiency. Here, this work conducts a comparison between the expression of multimodal machine learning and multimodal neuromorphic computing, followed by an overview of the key characteristics associated with multimodal neuromorphic devices. The bio-plausible operational principles and the multimodal learning abilities of emerging devices are examined, which are classified into heterogeneous and homogeneous multimodal neuromorphic devices. Subsequently, this work provides a detailed description of the multimodal learning capabilities demonstrated by neuromorphic circuits and their respective applications. Finally, this work highlights the limitations and challenges of multimodal neuromorphic computing in order to hopefully provide insight into potential future research directions.

Inspiration from Visual Ecology for Advancing Multifunctional Robotic Vision Systems: Bio-inspired Electronic Eyes and Neuromorphic Image Sensors

In robotics, particularly for autonomous navigation and human-robot collaboration, the significance of unconventional imaging techniques and efficient data processing capabilities is paramount. The unstructured environments encountered by robots, coupled with complex missions assigned to them, present numerous challenges necessitating diverse visual functionalities, and consequently, the development of multifunctional robotic vision systems has become indispensable. Meanwhile, rich diversity inherent in animal vision systems, honed over evolutionary epochs to meet their survival demands across varied habitats, serves as a profound source of inspirations. Here, recent advancements in multifunctional robotic vision systems drawing inspiration from natural ocular structures and their visual perception mechanisms are delineated. First, unique imaging functionalities of natural eyes across terrestrial, aerial, and aquatic habitats and visual signal processing mechanism of humans are explored. Then, designs and functionalities of bio-inspired electronic eyes are explored, engineered to mimic key components and underlying optical principles of natural eyes. Furthermore, neuromorphic image sensors are discussed, emulating functional properties of synapses, neurons, and retinas and thereby enhancing accuracy and efficiency of robotic vision tasks. Next, integration examples of electronic eyes with mobile robotic/biological systems are introduced. Finally, a forward-looking outlook on the development of bio-inspired electronic eyes and neuromorphic image sensors is provided.

Network of artificial olfactory receptors for spatiotemporal monitoring of toxic gas

Excessive human exposure to toxic gases can lead to chronic lung and cardiovascular diseases. Thus, precise in situ monitoring of toxic gases in the atmosphere is crucial. Here, we present an artificial olfactory system for spatiotemporal recognition of NO2 gas flow by integrating a network of chemical receptors with a near-sensor computing. The artificial olfactory receptor features nano-islands of metal-based catalysts that cover the graphene surface on the heterostructure of an AlGaN/GaN two-dimensional electron gas (2DEG) channel. Catalytically dissociated NO2 molecules bind to graphene, thereby modulating the conductivity of the 2DEG channel. For the energy/resource-efficient gas flow monitoring, trust-region Bayesian optimization algorithm allocates many sensors optimally in a complex space. Integrated artificial neural networks on a compact microprocessor with a network of sensors provide in situ gas flow predictions. This system enhances protective measures against toxic environments through spatiotemporal monitoring of toxic gases.

Development of Photonic In-Sensor Computing Based on a Mid-Infrared Silicon Waveguide Platform

Neuromorphic in-sensor computing has provided an energy-efficient solution to smart sensor design and on-chip data processing. In recent years, various free-space-configured optoelectronic chips have been demonstrated for on-chip neuromorphic vision processing. However, on-chip waveguide-based in-sensor computing with different data modalities is still lacking. Here, by integrating a responsivity-tunable graphene photodetector onto the silicon waveguide, an on-chip waveguide-based in-sensor processing unit is realized in the mid-infrared wavelength range. The weighting operation is achieved by dynamically tuning the bias of the photodetector, which could reach 4 bit weighting precision. Three different neural network tasks are performed to demonstrate the capabilities of our device. First, image preprocessing is performed for handwritten digits and fashion product classification as a general task. Next, resistive-type glove sensor signals are reversed and applied to the photodetector as an input for gesture recognition. Finally, spectroscopic data processing for binary gas mixture classification is demonstrated by utilizing the broadband performance of the device from 3.65 to 3.8 μm. By extending the wavelength from near-infrared to mid-infrared, our work shows the capability of a waveguide-integrated tunable graphene photodetector as a viable weighting solution for photonic in-sensor computing. Furthermore, such a solution could be used for large-scale neuromorphic in-sensor computing in photonic integrated circuits at the edge.

A Plasmonic Optoelectronic Resistive Random-Access Memory for In-Sensor Color Image Cryptography

The optoelectronic resistive random-access memory (RRAM) with the integrated function of perception, storage and intrinsic randomness displays promising applications in the hardware level in-sensor image cryptography. In this work, 2D hexagonal boron nitride based optoelectronic RRAM is fabricated with semitransparent noble metal (Ag or Au) as top electrodes, which can simultaneous capture color image and generate physically unclonable function (PUF) key for in-sensor color image cryptography. Surface plasmons of noble metals enable the strong light absorption to realize an efficient modulation of filament growth at nanoscale. Resistive switching curves show that the optical stimuli can impede the filament aggregation and promote the filament annihilation, which originates from photothermal effects and photogenerated hot electrons in localized surface plasmon resonance of noble metals. By selecting noble metals, the optoelectronic RRAM array can respond to distinct wavelengths and mimic the biological dichromatic cone cells to perform the color perception. Due to the intrinsic and high-quality randomness, the optoelectronic RRAM can produce a PUF key in every exposure cycle, which can be applied in the reconfigurable cryptography. The findings demonstrate an effective strategy to build optoelectronic RRAM for in-sensor color image cryptography applications.

Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array

The bioinspired camera, comprising a single lens and a curved image sensor-a photodiode array on a curved surface-, was born of flexible electronics. Its economical build lends itself well to space-constrained machine vision applications. The curved sensor, much akin to the retina, helps image focusing, but the curvature also creates a problem of image distortion, which can undermine machine vision tasks such as object recognition. Here we report an anti-distortion single-lens camera, where 4096 silicon photodiodes arrayed on a curved surface in a nonuniform pattern assimilated to the distorting optics are the key to anti-distortion engineering. That is, the photo-pixel distribution pattern itself is warped in the same manner as images are warped, which correctively reverses distortion. Acquired images feature no appreciable distortion across a 120° horizontal view, as confirmed by their neural-network recognition accuracies. This distortion correction via photo-pixel array reconfiguration is a form of in-sensor computing.

IGZO/WO3-x-Heterostructured Artificial Optoelectronic Synaptic Devices Mimicking Image Segmentation and Motion Capture

Currently, artificial neural networks (ANNs) based on memristors are limited to recognizing static images of objects when simulating human visual system, preventing them from performing high-dimensional information perception, and achieving more complex biomimetic functions is subject to certain limitations. In this work, indium gallium zinc oxide (IGZO)/tungsten oxide (WO3-x)-heterostructured artificial optoelectronic synaptic devices mimicking image segmentation and motion capture exhibiting high-performance optoelectronic synaptic responses are proposed and demonstrated. Upon electrical and optical stimulations, the device shows a variety of fundamental and advanced electrical and optical synaptic plasticity. Most importantly, outstanding and repeatable linear synaptic weight changes are attained by the developed memristor. By taking advantage of the notable linear synaptic weight changes, ANNs have been constructed and successfully utilized to demonstrate two applications in the field of computer vision, including image segmentation and object tracking. The accuracy attained by the memristor-based ANNs is similar to that of the computer algorithms, while its power has been significantly reduced by 105 orders of magnitude. With successful emulations of the human brain reactions when observing objects, the demonstrated memristor and related ANNs can be effectively utilized in constructing artificial optoelectronic synaptic devices and show promising potential in emulating human visual perception.

A Reconfigurable Bipolar Image Sensor for High-Efficiency Dynamic Vision Recognition

Dynamic vision perception and processing (DVPP) is in high demand by booming edge artificial intelligence. However, existing imaging systems suffer from low efficiency or low compatibility with advanced machine vision techniques. Here, we propose a reconfigurable bipolar image sensor (RBIS) for in-sensor DVPP based on a two-dimensional WSe2/GeSe heterostructure device. Owing to the gate-tunable and reversible built-in electric field, its photoresponse shows bipolarity as being positive or negative. High-efficiency DVPP incorporating front-end RBIS and back-end CNN is then demonstrated. It shows a high recognition accuracy of over 94.9% on the derived DVS128 data set and requires much fewer neural network parameters than that without RBIS. Moreover, we demonstrate an optimized device with a vertically stacked structure and a stable nonvolatile bipolarity, which enables more efficient DVPP hardware. Our work demonstrates the potential of fabricating DVPP devices with a simple structure, high efficiency, and outputs compatible with advanced algorithms.

The Roadmap of 2D Materials and Devices Toward Chips

Due to the constraints imposed by physical effects and performance degradation, silicon-based chip technology is facing certain limitations in sustaining the advancement of Moore's law. Two-dimensional (2D) materials have emerged as highly promising candidates for the post-Moore era, offering significant potential in domains such as integrated circuits and next-generation computing. Here, in this review, the progress of 2D semiconductors in process engineering and various electronic applications are summarized. A careful introduction of material synthesis, transistor engineering focused on device configuration, dielectric engineering, contact engineering, and material integration are given first. Then 2D transistors for certain electronic applications including digital and analog circuits, heterogeneous integration chips, and sensing circuits are discussed. Moreover, several promising applications (artificial intelligence chips and quantum chips) based on specific mechanism devices are introduced. Finally, the challenges for 2D materials encountered in achieving circuit-level or system-level applications are analyzed, and potential development pathways or roadmaps are further speculated and outlooked.

Bionic visual-audio photodetectors with in-sensor perception and preprocessing

Serving as the "eyes" and "ears" of the Internet of Things, optical and acoustic sensors are the fundamental components in hardware systems. Nowadays, mainstream hardware systems, often comprising numerous discrete sensors, conversion modules, and processing units, tend to result in complex architectures that are less efficient compared to human sensory pathways. Here, a visual-audio photodetector inspired by the human perception system is proposed to enable all-in-one visual and acoustic signal detection with computing capability. This device not only captures light but also optically records sound waves, thus achieving "watching" and "listening" within a single unit. The gate-tunable positive, negative, and zero photoresponses lead to highly programmable responsivities. This programmability enables the execution of diverse functions, including visual feature extraction, object classification, and sound wave manipulation. These results showcase the potential of expanding perception approaches in neuromorphic devices, opening up new possibilities to craft intelligent and compact hardware systems.

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

High-performance robotic vision empowers mobile and humanoid robots to detect and identify their surrounding objects efficiently, which enables them to cooperate with humans and assist human activities. For error-free execution of these robots' tasks, efficient imaging and data processing capabilities are essential, even under diverse and complex environments. However, conventional technologies fall short of meeting the high-standard requirements of robotic vision under such circumstances. Here, we discuss recent progress in artificial vision systems with high-performance imaging and data processing capabilities enabled by distinctive electrical, optical, and mechanical characteristics of nanomaterials surpassing the limitations of traditional silicon technologies. In particular, we focus on nanomaterial-based electronic eyes and in-sensor processing devices inspired by biological eyes and animal visual recognition systems, respectively. We provide perspectives on key nanomaterials, device components, and their functionalities, as well as explain the remaining challenges and future prospects of the artificial vision systems.

Full hardware implementation of neuromorphic visual system based on multimodal optoelectronic resistive memory arrays for versatile image processing

In-sensor and near-sensor computing are becoming the next-generation computing paradigm for high-density and low-power sensory processing. To fulfil a high-density and efficient neuromorphic visual system with fully hierarchical emulation of the retina and visual cortex, emerging multimodal neuromorphic devices for multi-stage processing and a fully hardware-implemented system with versatile image processing functions are still lacking and highly desirable. Here we demonstrate an emerging multimodal-multifunctional resistive random-access memory (RRAM) device array based on modified silk fibroin protein (MSFP), exhibiting both optoelectronic RRAM (ORRAM) mode featured by unique negative and positive photoconductance memory and electrical RRAM (ERRAM) mode featured by analogue resistive switching. A full hardware implementation of the artificial visual system with versatile image processing functions is realised for the first time, including ORRAM mode array for the in-sensor image pre-processing (contrast enhancement, background denoising, feature extraction) and ERRAM mode array for near-sensor high-level image recognition, which hugely improves the integration density, and simply the circuit design and the fabrication and integration complexity.

Filament-free memristors for computing

Memristors have attracted increasing attention due to their tremendous potential to accelerate data-centric computing systems. The dynamic reconfiguration of memristive devices in response to external electrical stimuli can provide highly desirable novel functionalities for computing applications when compared with conventional complementary-metal-oxide-semiconductor (CMOS)-based devices. Those most intensively studied and extensively reviewed memristors in the literature so far have been filamentary type memristors, which typically exhibit a relatively large variability from device to device and from switching cycle to cycle. On the other hand, filament-free switching memristors have shown a better uniformity and attractive dynamical properties, which can enable a variety of new computing paradigms but have rarely been reviewed. In this article, a wide range of filament-free switching memristors and their corresponding computing applications are reviewed. Various junction structures, switching properties, and switching principles of filament-free memristors are surveyed and discussed. Furthermore, we introduce recent advances in different computing schemes and their demonstrations based on non-filamentary memristors. This Review aims to present valuable insights and guidelines regarding the key computational primitives and implementations enabled by these filament-free switching memristors.

On-chip phonon-magnon reservoir for neuromorphic computing

Reservoir computing is a concept involving mapping signals onto a high-dimensional phase space of a dynamical system called "reservoir" for subsequent recognition by an artificial neural network. We implement this concept in a nanodevice consisting of a sandwich of a semiconductor phonon waveguide and a patterned ferromagnetic layer. A pulsed write-laser encodes input signals into propagating phonon wavepackets, interacting with ferromagnetic magnons. The second laser reads the output signal reflecting a phase-sensitive mix of phonon and magnon modes, whose content is highly sensitive to the write- and read-laser positions. The reservoir efficiently separates the visual shapes drawn by the write-laser beam on the nanodevice surface in an area with a size comparable to a single pixel of a modern digital camera. Our finding suggests the phonon-magnon interaction as a promising hardware basis for realizing on-chip reservoir computing in future neuromorphic architectures.

Neuro-inspired optical sensor array for high-accuracy static image recognition and dynamic trace extraction

Neuro-inspired vision systems hold great promise to address the growing demands of mass data processing for edge computing, a distributed framework that brings computation and data storage closer to the sources of data. In addition to the capability of static image sensing and processing, the hardware implementation of a neuro-inspired vision system also requires the fulfilment of detecting and recognizing moving targets. Here, we demonstrated a neuro-inspired optical sensor based on two-dimensional NbS2/MoS2 hybrid films, which featured remarkable photo-induced conductance plasticity and low electrical energy consumption. A neuro-inspired optical sensor array with 10 × 10 NbS2/MoS2 phototransistors enabled highly integrated functions of sensing, memory, and contrast enhancement capabilities for static images, which benefits convolutional neural network (CNN) with a high image recognition accuracy. More importantly, in-sensor trajectory registration of moving light spots was experimentally implemented such that the post-processing could yield a high restoration accuracy. Our neuro-inspired optical sensor array could provide a fascinating platform for the implementation of high-performance artificial vision systems.

Four levels of in-sensor computing in bionic olfaction: from discrete components to multi-modal integrations

Sensing and computing are two important ways in which humans attempt to perceive and understand the analog world through digital devices. Analog-to-digital converters (ADCs) discretize analog signals while the data bus transmits digital data between the components of a computer. With the increase in sensor nodes and the application of deep neural networks, the energy and time consumption limit the increment of data throughput. In-sensor computing is a computing paradigm that integrates sensing, storage, and processing in one device without ADCs and data transfer. According to the integration degree, herein, we summarize four levels of in-sensor computing in the field of artificial olfactory. In the first level, we show that different functions are conducted by using discrete components. Next, the data conversion and transfer are exempt within the in-memory computing architecture with necessary data encoding. Subsequently, in-sensor computing is integrated into a single device. Finally, multi-modal in-sensor computing is proposed to improve the quality and reliability of the classification results. At the end of this minireview, we provide an outlook on the use of metal nanoparticle devices to achieve such in-sensor computing for bionic olfaction.

An artificial neuromuscular junction for enhanced reflexes and oculomotor dynamics based on a ferroelectric CuInP2S6/GaN HEMT

A neuromuscular junction (NMJ) is a particularized synapse that activates muscle fibers for macro-motions, requiring more energy than computation. Emulating the NMJ is thus challenging owing to the need for both synaptic plasticity and high driving power to trigger motions. Here, we present an artificial NMJ using CuInP2S6 (CIPS) as a gate dielectric integrated with an AlGaN/GaN-based high-electron mobility transistor (HEMT). The ferroelectricity of the CIPS is coupled with the two-dimensional electron gas channel in the HEMT, providing a wide programmable current range of 6 picoampere per millimeter to 5 milliampere per millimeter. The large output current window of the CIPS/GaN ferroelectric HEMT (FeHEMT) allows for amplifier-less actuation, emulating the biological NMJ functions of actuation and synaptic plasticity. We also demonstrate the emulation of biological oculomotor dynamics, including in situ object tracking and enhanced stimulus responses, using the fabricated artificial NMJ. We believe that the CIPS/GaN FeHEMT offers a promising pathway for bioinspired robotics and neuromorphic vision.

Broadband sensory networks with locally stored responsivities for neuromorphic machine vision

As the most promising candidates for the implementation of in-sensor computing, retinomorphic vision sensors can constitute built-in neural networks and directly implement multiply-and-accumulation operations using responsivities as the weights. However, existing retinomorphic vision sensors mainly use a sustained gate bias to maintain the responsivity due to its volatile nature. Here, we propose an ion-induced localized-field strategy to develop retinomorphic vision sensors with nonvolatile tunable responsivity in both positive and negative regimes and construct a broadband and reconfigurable sensory network with locally stored weights to implement in-sensor convolutional processing in spectral range of 400 to 1800 nanometers. In addition to in-sensor computing, this retinomorphic device can implement in-memory computing benefiting from the nonvolatile tunable conductance, and a complete neuromorphic visual system involving front-end in-sensor computing and back-end in-memory computing architectures has been constructed, executing supervised and unsupervised learning tasks as demonstrations. This work paves the way for the development of high-speed and low-power neuromorphic machine vision for time-critical and data-intensive applications.

Emerging Iontronic Neural Devices for Neuromorphic Sensory Computing

Living organisms have a very mysterious and powerful sensory computing system based on ion activity. Interestingly, studies on iontronic devices in the past few years have proposed a promising platform for simulating the sensing and computing functions of living organisms, because: 1) iontronic devices can generate, store, and transmit a variety of signals by adjusting the concentration and spatiotemporal distribution of ions, which analogs to how the brain performs intelligent functions by alternating ion flux and polarization; 2) through ionic-electronic coupling, iontronic devices can bridge the biosystem with electronics and offer profound implications for soft electronics; 3) with the diversity of ions, iontronic devices can be designed to recognize specific ions or molecules by customizing the charge selectivity, and the ionic conductivity and capacitance can be adjusted to respond to external stimuli for a variety of sensing schemes, which can be more difficult for electron-based devices. This review provides a comprehensive overview of emerging neuromorphic sensory computing by iontronic devices, highlighting representative concepts of both low-level and high-level sensory computing and introducing important material and device breakthroughs. Moreover, iontronic devices as a means of neuromorphic sensing and computing are discussed regarding the pending challenges and future directions.

Nanomaterial-Based Synaptic Optoelectronic Devices for In-Sensor Preprocessing of Image Data

With the advance in information technologies involving machine vision applications, the demand for energy- and time-efficient acquisition, transfer, and processing of a large amount of image data has rapidly increased. However, current architectures of the machine vision system have inherent limitations in terms of power consumption and data latency owing to the physical isolation of image sensors and processors. Meanwhile, synaptic optoelectronic devices that exhibit photoresponse similar to the behaviors of the human synapse enable in-sensor preprocessing, which makes the front-end part of the image recognition process more efficient. Herein, we review recent progress in the development of synaptic optoelectronic devices using functional nanomaterials and their unique interfacial characteristics. First, we provide an overview of representative functional nanomaterials and device configurations for the synaptic optoelectronic devices. Then, we discuss the underlying physics of each nanomaterial in the synaptic optoelectronic device and explain related device characteristics that allow for the in-sensor preprocessing. We also discuss advantages achieved by the application of the synaptic optoelectronic devices to image preprocessing, such as contrast enhancement and image filtering. Finally, we conclude this review and present a short prospect.