A multichannel optical computing architecture for advanced machine vision

Ultra-compact multi-task processor based on in-memory optical computing

To enhance the computational density and energy efficiency of on-chip neuromorphic hardware, this study introduces a novel network architecture for multi-task processing with in-memory optical computing. On-chip optical neural networks are celebrated for their capability to transduce a substantial volume of parameters into optical form while conducting passive computing, yet they encounter challenges in scalability and multitasking. Leveraging the principles of transfer learning, this approach involves embedding the majority of parameters into fixed optical components and a minority into adjustable electrical components. Furthermore, with deep regression algorithm in modeling physical propagation process, a compact optical neural network achieve to handle diverse tasks. In this work, two ultra-compact in-memory diffraction-based chips with integration of more than 60,000 parameters/mm2 were fabricated, employing deep neural network model and the hard parameter sharing algorithm, to perform multifaceted classification and regression tasks, respectively. The experimental results demonstrate that these chips achieve accuracies comparable to those of electrical networks while significantly reducing the power-intensive digital computation by 90%. Our work heralds strong potential for advancing in-memory optical computing frameworks and next generation of artificial intelligence platforms.

Flippable multitask diffractive neural networks based on double-sided metasurfaces

Diffractive neural networks (DNNs) have garnered significant attention in recent years as a physical computing framework, combining high computational speed, parallelism, and low-power consumption. However, the non-reconfigurability of cascaded diffraction layers limits the ability of DNNs to perform multitasking, and methods such as replacing diffraction layers or light sources, while theoretically feasible, are difficult to implement in practice. This Letter introduces a flippable diffractive neural network (F-DNN) in which the diffraction layer is an integrated structure processed on both sides of the substrate. This design allows rapid task switching by flipping diffraction layers and overcomes alignment challenges that arise when replacing layers. Classification-based simulation results demonstrate that F-DNN addresses the limitations of traditional multitask DNN architectures, offering both superior performance and scalability, which provides a new approach for realizing high-speed, low-power, and multitask artificial intelligence systems.

Adaptive In-Sensor Computing for Enhanced Feature Perception and Broadband Image Restoration

Traditional imaging systems struggle in weak or complex lighting environments due to their fixed spectral responses, resulting in spectral mismatches and degraded image quality. To address these challenges, a bioinspired adaptive broadband image sensor is developed. This innovative sensor leverages a meticulously designed type-I heterojunction alignment of 0D perovskite quantum dots (PQDs) and 2D black phosphorus (BP). This configuration enables efficient carrier injection control and advanced computing capabilities within an integrated phototransistor array. The sensor's unique responses to both visible and infrared (IR) light facilitate selective enhancement and precise feature extraction under varying lighting conditions. Furthermore, it supports real-time convolution and image restoration within a convolutional autoencoder (CAE) network, effectively countering image degradation by capturing spectral features. Remarkably, the hardware responsivity weights perform comparably to software-trained weights, achieving an image restoration accuracy of over 85%. This approach offers a robust and versatile solution for machine vision applications that demand precise and adaptive imaging in dynamic lighting environments.

Non-Interleaved Shared-Aperture Full-Stokes Metalens via Prior-Knowledge-Driven Inverse Design

Characterization of electromagnetic wave polarization states is critical in various applications of materials, biomedical, and imaging. The emergence of metasurfaces opens up the possibility of implementing highly integrated full-Stokes imagers. Despite rapid development, prevailing schemes on metasurface-based full-Stokes imagers require interleaved or cascaded designs, inevitably resulting in performance deterioration, bulky size, and complicating the imaging procedure due to misalignment. To overcome these challenges, a non-interleaved shared-aperture full-Stokes metalens enabled by the prior-knowledge-driven inverse design methodology is proposed. The metalens can be directly deployed into imagers, performing high-accuracy diffraction-limited full-Stokes imaging without other ancillary components, breaking intrinsic constraints of efficiency and resolution in interleaved design. To demonstrate this, the metalens is integrated into the W-band passive imaging system, as an alternative solution, to perform polarimetric imaging, which reduces the reliance on the high cost and high complexity of the ortho-mode transducer and broadband correlator in traditional polarimetric radiometers. Furthermore, with the assistance of a 3D reconstruction method, the feasibility of multi-polarization information for contactless surface slope measurements is explored. This work may open new paradigms for the full-Stokes imager designing and broaden the applications of metasurface polarimetric imaging.

Metasurface-Based Image Classification Using Diffractive Deep Neural Network

The computer-assisted inverse design of photonic computing, especially by leveraging artificial intelligence algorithms, offers great convenience to accelerate the speed of development and improve calculation accuracy. However, traditional thickness-based modulation methods are hindered by large volume and difficult fabrication process, making it hard to meet the data-driven requirements of flexible light modulation. Here, we propose a diffractive deep neural network (D2NN) framework based on a three-layer all-dielectric phased transmitarray as hidden layers, which can perform the classification of handwritten digits. By tailoring the radius of a silicon nanodisk of a meta-atom, the metasurface can realize the phase profile calculated by D2NN and maintain a relative high transmittance of 0.9 at a wavelength of 600 nm. The designed image classifier consists of three layers of phase-only metasurfaces, each of which contains 1024 units, mimicking a fully connected neural network through the diffraction of light fields. The classification task of handwriting digits from the '0' to '5' dataset is verified, with an accuracy of over 90% on the blind test dataset, as well as demonstrated by the full-wave simulation. Furthermore, the performance of the more complex animal image classification task is also validated by increasing the number of neurons to enhance the connectivity of the neural network. This study may provide a possible solution for practical applications such as biomedical detection, image processing, and machine vision based on all-optical computing.

Photoresponsive Two-Dimensional Magnetic Junctions for Reconfigurable In-Memory Sensing

Magnetic tunneling junctions (MTJs) lie in the core of magnetic random access memory, holding promise in integrating memory and computing to reduce hardware complexity, transition latency, and power consumption. However, traditional MTJs are insensitive to light, limiting their functionality in in-memory sensing─a crucial component for machine vision systems in artificial intelligence applications. Herein, the convergence of magnetic memory with optical sensing capabilities is achieved in the all-two-dimensional (2D) magnetic junction Fe3GaTe2/WSe2/Fe3GaTe2, which combines 2D magnetism and optoelectronic properties. The clean intrinsic band gap and prominent photoresponse of interlayer WSe2 endow the tunneling barrier with optical tunability. The on-off states of junctions and the magnetoresistance can be flexibly controlled by the intensity of the optical signal at room temperature. Based on the optical-tunable magnetoresistance in all-2D magnetic junctions, a machine vision system with the architecture of in-memory sensing and computing is constructed, which possesses high performance in image recognition. Our work exhibits the advantages of 2D magneto-electronic devices and extends the application scenarios of magnetic memory devices in artificial intelligence.

Optoelectronic nonlinear Softmax operator based on diffractive neural networks

Softmax, a pervasive nonlinear operation, plays a pivotal role in numerous statistics and deep learning (DL) models such as ChatGPT. To compute it is expensive especially for at-scale models. Several software and hardware speed-up strategies are proposed but still suffer from low efficiency, poor scalability. Here we propose a photonic-computing solution including massive programmable neurons that is capable to execute such operation in an accurate, computation-efficient, robust and scalable manner. Experimental results show our diffraction-based computing system exhibits salient generalization ability in diverse artificial and real-world tasks (mean square error <10-5). We further analyze its performances against several realistic restricted factors. Such flexible system not only contributes to optimizing Softmax operation mechanism but may provide an inspiration of manufacturing a plug-and-play module for general optoelectronic accelerators.

Large-scale photonic chiplet Taichi empowers 160-TOPS/W artificial general intelligence

The pursuit of artificial general intelligence (AGI) continuously demands higher computing performance. Despite the superior processing speed and efficiency of integrated photonic circuits, their capacity and scalability are restricted by unavoidable errors, such that only simple tasks and shallow models are realized. To support modern AGIs, we designed Taichi-large-scale photonic chiplets based on an integrated diffractive-interference hybrid design and a general distributed computing architecture that has millions-of-neurons capability with 160-tera-operations per second per watt (TOPS/W) energy efficiency. Taichi experimentally achieved on-chip 1000-category-level classification (testing at 91.89% accuracy in the 1623-category Omniglot dataset) and high-fidelity artificial intelligence-generated content with up to two orders of magnitude of improvement in efficiency. Taichi paves the way for large-scale photonic computing and advanced tasks, further exploiting the flexibility and potential of photonics for modern AGI.

Photonic neuromorphic architecture for tens-of-task lifelong learning

Scalable, high-capacity, and low-power computing architecture is the primary assurance for increasingly manifold and large-scale machine learning tasks. Traditional electronic artificial agents by conventional power-hungry processors have faced the issues of energy and scaling walls, hindering them from the sustainable performance improvement and iterative multi-task learning. Referring to another modality of light, photonic computing has been progressively applied in high-efficient neuromorphic systems. Here, we innovate a reconfigurable lifelong-learning optical neural network (L2ONN), for highly-integrated tens-of-task machine intelligence with elaborated algorithm-hardware co-design. Benefiting from the inherent sparsity and parallelism in massive photonic connections, L2ONN learns each single task by adaptively activating sparse photonic neuron connections in the coherent light field, while incrementally acquiring expertise on various tasks by gradually enlarging the activation. The multi-task optical features are parallelly processed by multi-spectrum representations allocated with different wavelengths. Extensive evaluations on free-space and on-chip architectures confirm that for the first time, L2ONN avoided the catastrophic forgetting issue of photonic computing, owning versatile skills on challenging tens-of-tasks (vision classification, voice recognition, medical diagnosis, etc.) with a single model. Particularly, L2ONN achieves more than an order of magnitude higher efficiency than the representative electronic artificial neural networks, and 14× larger capacity than existing optical neural networks while maintaining competitive performance on each individual task. The proposed photonic neuromorphic architecture points out a new form of lifelong learning scheme, permitting terminal/edge AI systems with light-speed efficiency and unprecedented scalability.

Parallel edge extraction operators on chip speed up photonic convolutional neural networks

We experimentally establish a 3 × 3 cross-shaped micro-ring resonator (MRR) array-based photonic multiplexing architecture relying on silicon photonics to achieve parallel edge extraction operations in images for photonic convolution neural networks. The main mathematical operations involved are convolution. Precisely, a faster convolutional calculation speed of up to four times is achieved by extracting four feature maps simultaneously with the same photonic hardware's structure and power consumption, where a maximum computility of 0.742 TOPS at an energy cost of 48.6 mW and a convolution accuracy of 95.1% is achieved in an MRR array chip. In particular, our experimental results reveal that this system using parallel edge extraction operators instead of universal operators can improve the imaging recognition accuracy for CIFAR-10 dataset by 6.2% within the same computing time, reaching a maximum of 78.7%. This work presents high scalability and efficiency of parallel edge extraction chips, furnishing a novel, to the best of our knowledge, approach to boost photonic computing speed.

Optical computing metasurfaces: applications and advances

Integrated photonic devices and artificial intelligence have presented a significant opportunity for the advancement of optical computing in practical applications. Optical computing technology is a unique computing system based on optical devices and computing functions, which significantly differs from the traditional electronic computing technology. On the other hand, optical computing technology offers the advantages such as fast speed, low energy consumption, and high parallelism. Yet there are still challenges such as device integration and portability. In the burgeoning development of micro-nano optics technology, especially the deeply ingrained concept of metasurface technique, it provides an advanced platform for optical computing applications, including edge detection, image or motion recognition, logic computation, and on-chip optical computing. With the aim of providing a comprehensive introduction and perspective for optical computing metasurface applications, we review the recent research advances of optical computing, from nanostructure and computing methods to practical applications. In this work, we review the challenges and analysis of optical computing metasurfaces in engineering field and look forward to the future development trends of optical computing.

Vector vortex beams sorting of 120 modes in visible spectrum

Polarization (P), angular index (l), and radius index (p) are three independent degrees of freedom (DoFs) of vector vortex beams, which have found extensive applications in various domains. While efficient sorting of a single DoF has been achieved successfully, simultaneous sorting of all these DoFs in a compact and efficient manner remains a challenge. In this study, we propose a beam sorter that simultaneously handles all the three DoFs using a diffractive deep neural network (D2NN), and demonstrate the robust sorting of 120 Laguerre-Gaussian (LG) modes experimentally in the visible spectrum. Our proposed beam sorter underscores the considerable potential of D2NN in optical field manipulation and promises to enhance the diverse applications of vector vortex beams.

Multi-wavelength diffractive neural network with the weighting method

Recently, the diffractive deep neural network (D2NN) has demonstrated the advantages to achieve large-scale computational tasks in terms of high speed, low power consumption, parallelism, and scalability. A typical D2NN with cascaded diffractive elements is designed for monochromatic illumination. Here, we propose a framework to achieve the multi-wavelength D2NN (MW-D2NN) based on the method of weight coefficients. In training, each wavelength is assigned a specific weighting and their output planes construct the wavelength weighting loss function. The trained MW-D2NN can implement the classification of images of handwritten digits at multi-wavelength incident beams. The designed 3-layers MW-D2NN achieves a simulation classification accuracy of 83.3%. We designed a 1-layer MW-D2NN. The simulation and experiment classification accuracy are 71.4% and 67.5% at RGB wavelengths. Furthermore, the proposed MW-D2NN can be extended to intelligent machine vision systems for multi-wavelength and incoherent illumination.

C-DONN: compact diffractive optical neural network with deep learning regression

A new method to improve the integration level of an on-chip diffractive optical neural network (DONN) is proposed based on a standard silicon-on-insulator (SOI) platform. The metaline, which represents a hidden layer in the integrated on-chip DONN, is composed of subwavelength silica slots, providing a large computation capacity. However, the physical propagation process of light in the subwavelength metalinses generally requires an approximate characterization using slot groups and extra length between adjacent layers, which limits further improvements of the integration of on-chip DONN. In this work, a deep mapping regression model (DMRM) is proposed to characterize the process of light propagation in the metalines. This method improves the integration level of on-chip DONN to over 60,000 and elimnates the need for approximate conditions. Based on this theory, a compact-DONN (C-DONN) is exploited and benchmarked on the Iris plants dataset to verify the performance, yielding a testing accuracy of 93.3%. This method provides a potential solution for future large-scale on-chip integration.

Planar Fourier optics for slab waveguides, surface plasmon polaritons, and 2D materials

Recent experimental work has demonstrated the potential of combining the merits of diffractive and on-chip photonic information processing devices in a single chip by making use of planar (or slab) waveguides. Here, arguments are developed to show that diffraction formulas familiar from 3D Fourier optics can be adapted to 2D under certain mild conditions on the operating speeds of the devices in question. In addition to serving those working in on-chip photonics, this Letter provides analytical tools for the study of surface plasmon polaritons, surface waves, and the optical, acoustic, and crystallographic properties of 2D materials.

Photonic multiplexing techniques for neuromorphic computing

The simultaneous advances in artificial neural networks and photonic integration technologies have spurred extensive research in optical computing and optical neural networks (ONNs). The potential to simultaneously exploit multiple physical dimensions of time, wavelength and space give ONNs the ability to achieve computing operations with high parallelism and large-data throughput. Different photonic multiplexing techniques based on these multiple degrees of freedom have enabled ONNs with large-scale interconnectivity and linear computing functions. Here, we review the recent advances of ONNs based on different approaches to photonic multiplexing, and present our outlook on key technologies needed to further advance these photonic multiplexing/hybrid-multiplexing techniques of ONNs.

Translation-invariant optical neural network for image classification

The classification performance of all-optical Convolutional Neural Networks (CNNs) is greatly influenced by components' misalignment and translation of input images in the practical applications. In this paper, we propose a free-space all-optical CNN (named Trans-ONN) which accurately classifies translated images in the horizontal, vertical, or diagonal directions. Trans-ONN takes advantages of an optical motion pooling layer which provides the translation invariance property by implementing different optical masks in the Fourier plane for classifying translated test images. Moreover, to enhance the translation invariance property, global average pooling (GAP) is utilized in the Trans-ONN structure, rather than fully connected layers. The comparative studies confirm that taking advantage of vertical and horizontal masks along GAP operation provide the best translation invariance property, compared to the alternative network models, for classifying horizontally and vertically shifted test images up to 50 pixel shifts of Kaggle Cats and Dogs, CIFAR-10, and MNIST datasets, respectively. Also, adopting the diagonal mask along GAP operation achieves the best classification accuracy for classifying translated test images in the diagonal direction for large number of pixel shifts (i.e. more than 30 pixel shifts). It is worth mentioning that the proposed translation invariant networks are capable of classifying the translated test images not included in the training procedure.