Experimentally realized in situ backpropagation for deep learning in photonic neural networks

Characterizing soil COPs eco-risk in China

Although soil combustible organic pollutants (COPs) pose a serious threat to human well-being, their spatial distribution patterns, responses to environmental constraints, and areas of risk throughout China are still unclear. This knowledge gap hinders the control of soil COPs, causing us to overlook their impact on climate change and the environment. In this study, a total of 420 soil samples, distributed in typical regions of China, were tested for COPs content, including black carbon (BC) and polycyclic aromatic hydrocarbons (PAHs). Interest points (POI) such as parking lots, gas stations, and car services have become the main factors that influence soil COPs enrichment, and can be considered new indicators in other organic pollution studies. By comparing various machine learning simulations and predictions, this study accurately predicted the content of soil COPs in China and pointed out that, as the "third pole of the world", the Qinghai Tibet Plateau will face an unprecedented crisis. We established a method for assessing the comprehensive risk of soil COPs and identified at-risk areas, which accounted for 38.9 % of China's total soil area. Our research findings emphasize the main driving factors for soil COPs and identify areas in China that require prioritized soil COPs control.

Probabilistic photonic computing for AI

Probabilistic computing excels in approximating combinatorial problems and modeling uncertainty. However, using conventional deterministic hardware for probabilistic models is challenging: (pseudo) random number generation introduces computational overhead and additional data shuffling. Therefore, there is a pressing need for different probabilistic computing architectures that achieve low latencies with reasonable energy consumption. Physical computing offers a promising solution, as these systems do not rely on an abstract deterministic representation of data but directly encode the information in physical quantities, enabling inherent probabilistic architectures utilizing entropy sources. Photonic computing is a prominent variant of physical computing due to the large available bandwidth, several orthogonal degrees of freedom for data encoding and optimal properties for in-memory computing and parallel data transfer. Here, we highlight key developments in physical photonic computing and photonic random number generation. We further provide insights into the realization of probabilistic photonic processors and their impact on artificial intelligence systems and future challenges.

Bidirectional Phosphorescent Neuroplasticity for All-Optical Neurovision

All-optical neuromorphics that can capture, process, and output photonic signals are in prospect to advance optical computing and imaging. Bidirectional neuroplasticity is essential for executing training and inference in optical neural networks, but most of the all-optical hardware only exhibits unidirectional weight modulation. Here, we explore bidirectional neuroplasticity in carbon dot phosphorescence (CDP) with potentiation and depression synaptic behaviors capable of neuroregulation for photonic intensity. This function enables the CDP as a neuroconverter to convert pulse light into excitatory and inhibitory light output for neuromorphic vision owing to the delayed release and superimposition dynamics of excitons in persistent phosphorescence, which allows for image digitization or direct observation. By integration with an optical neural network, the real-time motion tracking of light spots, including trajectory, direction, and speed, can be recorded and recognized, with a high accuracy of 96%. Such phosphor-based neuromorphics can be extended to other phosphorescent architectures for all-optical imaging and computing.

An integrated large-scale photonic accelerator with ultralow latency

Integrated photonics, particularly silicon photonics, have emerged as cutting-edge technology driven by promising applications such as short-reach communications, autonomous driving, biosensing and photonic computing1-4. As advances in AI lead to growing computing demands, photonic computing has gained considerable attention as an appealing candidate. Nonetheless, there are substantial technical challenges in the scaling up of integrated photonics systems to realize these advantages, such as ensuring consistent performance gains in upscaled integrated device clusters, establishing standard designs and verification processes for complex circuits, as well as packaging large-scale systems. These obstacles arise primarily because of the relative immaturity of integrated photonics manufacturing and the scarcity of advanced packaging solutions involving photonics. Here we report a large-scale integrated photonic accelerator comprising more than 16,000 photonic components. The accelerator is designed to deliver standard linear matrix multiply-accumulate (MAC) functions, enabling computing with high speed up to 1 GHz frequency and low latency as small as 3 ns per cycle. Logic, memory and control functions that support photonic matrix MAC operations were designed into a cointegrated electronics chip. To seamlessly integrate the electronics and photonics chips at the commercial scale, we have made use of an innovative 2.5D hybrid advanced packaging approach. Through the development of this accelerator system, we demonstrate an ultralow computation latency for heuristic solvers of computationally hard Ising problems whose performance greatly relies on the computing latency.

Asymmetrical estimator for training encapsulated deep photonic neural networks

Photonic neural networks (PNNs) are fast in-propagation and high bandwidth paradigms that aim to popularize reproducible NN acceleration with higher efficiency and lower cost. However, the training of PNN is known to be challenging, where the device-to-device and system-to-system variations create imperfect knowledge of the PNN. Despite backpropagation (BP)-based training algorithms being the industry standard for their robustness, generality, and fast gradient convergence for digital training, existing PNN-BP methods rely heavily on accurate intermediate state extraction or extensive computational resources for deep PNNs (DPNNs). The truncated photonic signal propagation and the computation overhead bottleneck DPNN's operation efficiency and increase system construction cost. Here, we introduce the asymmetrical training (AsyT) method, tailored for encapsulated DPNNs, where the signal is preserved in the analogue photonic domain for the entire structure. AsyT offers a lightweight solution for DPNNs with minimum readouts, fast and energy-efficient operation, and minimum system footprint. AsyT's ease of operation, error tolerance, and generality aim to promote PNN acceleration in a widened operational scenario despite the fabrication variations and imperfect controls. We demonstrated AsyT for encapsulated DPNN with integrated photonic chips, repeatably enhancing the performance from in-silico BP for different network structures and datasets.

Fast and efficient Sb-based type-II phototransistors integrated on silicon

Increasing the energy efficiency and reducing the footprint of on-chip photodetectors enable dense optical interconnects for emerging computational and sensing applications. While heterojunction phototransistors (HPTs) exhibit high energy efficiency and negligible excess noise factor, their gain-bandwidth product (GBP) has been inferior to that of avalanche photodiodes at low optical powers. Here, we demonstrate that utilizing type-II energy band alignment in an Sb-based HPT results in six times smaller junction capacitance per unit area and a significantly higher GBP at low optical powers. These type-II HPTs were scaled down to 2 μm in diameter and fully integrated with photonic waveguides on silicon. Thanks to their extremely low dark current and high internal gain, these devices exhibit a GBP similar to the best avalanche devices (∼270 GHz) but with one order of magnitude better energy efficiency. Their energy consumption is about 5 fJ/bit at 3.2 Gbps, with an error rate below 10-9 at -25 dBm optical power at 1550 nm. These features suggest new opportunities for creating highly efficient and compact optical receivers based on phototransistors with type-II band alignment.

Empowering nanophotonic applications via artificial intelligence: pathways, progress, and prospects

Empowering nanophotonic devices via artificial intelligence (AI) has revolutionized both scientific research methodologies and engineering practices, addressing critical challenges in the design and optimization of complex systems. Traditional methods for developing nanophotonic devices are often constrained by the high dimensionality of design spaces and computational inefficiencies. This review highlights how AI-driven techniques provide transformative solutions by enabling the efficient exploration of vast design spaces, optimizing intricate parameter systems, and predicting the performance of advanced nanophotonic materials and devices with high accuracy. By bridging the gap between computational complexity and practical implementation, AI accelerates the discovery of novel nanophotonic functionalities. Furthermore, we delve into emerging domains, such as diffractive neural networks and quantum machine learning, emphasizing their potential to exploit photonic properties for innovative strategies. The review also examines AI's applications in advanced engineering areas, e.g., optical image recognition, showcasing its role in addressing complex challenges in device integration. By facilitating the development of highly efficient, compact optical devices, these AI-powered methodologies are paving the way for next-generation nanophotonic systems with enhanced functionalities and broader applications.

Optimizing optical neural network design for enhanced compatibility with analog computation

This paper breaks away from traditional approaches that merely emulate digital neural networks. Using Mach-Zehnder interferometer (MZI) networks as a case study, we explore the impact of the inherent properties of analog computation on performance and identify the characteristics that optical neural networks (ONNs) components should possess to better adapt to these specific properties. Specifically, we examine the influence of analog computation on bias power and activation functions, as well as the impact of optical pruning on ONN's performance. The results show that a suitably larger bias power relative to normalized data and concave activation functions are more compatible with the characteristics of ONNs. These factors can significantly improve classification accuracy across different datasets and ξ values, with improvements reaching up to 35%. Additionally, optical pruning reduces the number of MZIs by two-thirds while maintaining performance. Moreover, these measures significantly enhance the robustness of ONNs against MZI losses and phase errors. Although this research primarily focuses on feedforward MZI-based networks, the proposed design principles are widely applicable to other types of ONNs.

Photonics for Neuromorphic Computing: Fundamentals, Devices, and Opportunities

In the dynamic landscape of Artificial Intelligence (AI), two notable phenomena are becoming predominant: the exponential growth of large AI model sizes and the explosion of massive amount of data. Meanwhile, scientific research such as quantum computing and protein synthesis increasingly demand higher computing capacities. As the Moore's Law approaches its terminus, there is an urgent need for alternative computing paradigms that satisfy this growing computing demand and break through the barrier of the von Neumann model. Neuromorphic computing, inspired by the mechanism and functionality of human brains, uses physical artificial neurons to do computations and is drawing widespread attention. This review studies the expansion of optoelectronic devices on photonic integration platforms that has led to significant growth in photonic computing, where photonic integrated circuits (PICs) have enabled ultrafast artificial neural networks (ANN) with sub-nanosecond latencies, low heat dissipation, and high parallelism. In particular, various technologies and devices employed in neuromorphic photonic AI accelerators, spanning from traditional optics to PCSEL lasers are examined. Lastly, it is recognized that existing neuromorphic technologies encounter obstacles in meeting the peta-level computing speed and energy efficiency threshold, and potential approaches in new devices, fabrication, materials, and integration to drive innovation are also explored. As the current challenges and barriers in cost, scalability, footprint, and computing capacity are resolved one-by-one, photonic neuromorphic systems are bound to co-exist with, if not replace, conventional electronic computers and transform the landscape of AI and scientific computing in the foreseeable future.

Ultrafast silicon photonic reservoir computing engine delivering over 200 TOPS

Reservoir computing (RC) is a powerful machine learning algorithm for information processing. Despite numerous optical implementations, its speed and scalability remain limited by the need to establish recurrent connections and achieve efficient optical nonlinearities. This work proposes a streamlined photonic RC design based on a new paradigm, called next-generation RC, which overcomes these limitations. Our design leads to a compact silicon photonic computing engine with an experimentally demonstrated processing speed of over 60 GHz. Experimental results demonstrate state-of-the-art performance in prediction, emulation, and classification tasks across various machine learning applications. Compared to traditional RC systems, our silicon photonic RC engine offers several key advantages, including no speed limitations, a compact footprint, and a high tolerance to fabrication errors. This work lays the foundation for ultrafast on-chip photonic RC, representing significant progress toward developing next-generation high-speed photonic computing and signal processing.

Training all-mechanical neural networks for task learning through in situ backpropagation

Recent advances unveiled physical neural networks as promising machine learning platforms, offering faster and more energy-efficient information processing. Compared with extensively-studied optical neural networks, the development of mechanical neural networks remains nascent and faces significant challenges, including heavy computational demands and learning with approximate gradients. Here, we introduce the mechanical analogue of in situ backpropagation to enable highly efficient training of mechanical neural networks. We theoretically prove that the exact gradient can be obtained locally, enabling learning through the immediate vicinity, and we experimentally demonstrate this backpropagation to obtain gradient with high precision. With the gradient information, we showcase the successful training of networks in simulations for behavior learning and machine learning tasks, achieving high accuracy in experiments of regression and classification. Furthermore, we present the retrainability of networks involving task-switching and damage, demonstrating the resilience. Our findings, which integrate the theory for training mechanical neural networks and experimental and numerical validations, pave the way for mechanical machine learning hardware and autonomous self-learning material systems.

Spatially varying nanophotonic neural networks

The explosive growth in computation and energy cost of artificial intelligence has spurred interest in alternative computing modalities to conventional electronic processors. Photonic processors, which use photons instead of electrons, promise optical neural networks with ultralow latency and power consumption. However, existing optical neural networks, limited by their designs, have not achieved the recognition accuracy of modern electronic neural networks. In this work, we bridge this gap by embedding parallelized optical computation into flat camera optics that perform neural network computations during capture, before recording on the sensor. We leverage large kernels and propose a spatially varying convolutional network learned through a low-dimensional reparameterization. We instantiate this network inside the camera lens with a nanophotonic array with angle-dependent responses. Combined with a lightweight electronic back-end of about 2K parameters, our reconfigurable nanophotonic neural network achieves 72.76% accuracy on CIFAR-10, surpassing AlexNet (72.64%), and advancing optical neural networks into the deep learning era.

120 GOPS Photonic tensor core in thin-film lithium niobate for inference and in situ training

Photonics offers a transformative approach to artificial intelligence (AI) and neuromorphic computing by enabling low-latency, high-speed, and energy-efficient computations. However, conventional photonic tensor cores face significant challenges in constructing large-scale photonic neuromorphic networks. Here, we propose a fully integrated photonic tensor core, consisting of only two thin-film lithium niobate (TFLN) modulators, a III-V laser, and a charge-integration photoreceiver. Despite its simple architecture, it is capable of implementing an entire layer of a neural network with a computational speed of 120 GOPS, while also allowing flexible adjustment of the number of inputs (fan-in) and outputs (fan-out). Our tensor core supports rapid in-situ training with a weight update speed of 60 GHz. Furthermore, it successfully classifies (supervised learning) and clusters (unsupervised learning) 112  × 112-pixel images through in-situ training. To enable in-situ training for clustering AI tasks, we offer a solution for performing multiplications between two negative numbers.

Deep learning with photonic neural cellular automata

Rapid advancements in deep learning over the past decade have fueled an insatiable demand for efficient and scalable hardware. Photonics offers a promising solution by leveraging the unique properties of light. However, conventional neural network architectures, which typically require dense programmable connections, pose several practical challenges for photonic realizations. To overcome these limitations, we propose and experimentally demonstrate Photonic Neural Cellular Automata (PNCA) for photonic deep learning with sparse connectivity. PNCA harnesses the speed and interconnectivity of photonics, as well as the self-organizing nature of cellular automata through local interactions to achieve robust, reliable, and efficient processing. We utilize linear light interference and parametric nonlinear optics for all-optical computations in a time-multiplexed photonic network to experimentally perform self-organized image classification. We demonstrate binary (two-class) classification of images using as few as 3 programmable photonic parameters, achieving high experimental accuracy with the ability to also recognize out-of-distribution data. The proposed PNCA approach can be adapted to a wide range of existing photonic hardware and provides a compelling alternative to conventional photonic neural networks by maximizing the advantages of light-based computing whilst mitigating their practical challenges. Our results showcase the potential of PNCA in advancing photonic deep learning and highlights a path for next-generation photonic computers.

Measuring, processing, and generating partially coherent light with self-configuring optics

Optical phenomena always display some degree of partial coherence between their respective degrees of freedom. Partial coherence is of particular interest in multimodal systems, where classical and quantum correlations between spatial, polarization, and spectral degrees of freedom can lead to fascinating phenomena (e.g., entanglement) and be leveraged for advanced imaging and sensing modalities (e.g., in hyperspectral, polarization, and ghost imaging). Here, we present a universal method to analyze, process, and generate spatially partially coherent light in multimode systems by using self-configuring optical networks. Our method relies on cascaded self-configuring layers whose average power outputs are sequentially optimized. Once optimized, the network separates the input light into its mutually incoherent components, which is formally equivalent to a diagonalization of the input density matrix. We illustrate our method with numerical simulations of Mach-Zehnder interferometer arrays and show how this method can be used to perform partially coherent environmental light sensing, generation of multimode partially coherent light with arbitrary coherency matrices, and unscrambling of quantum optical mixtures. We provide guidelines for the experimental realization of this method, including the influence of losses, paving the way for self-configuring photonic devices that can automatically learn optimal modal representations of partially coherent light fields.

Optical neural networks: progress and challenges

Artificial intelligence has prevailed in all trades and professions due to the assistance of big data resources, advanced algorithms, and high-performance electronic hardware. However, conventional computing hardware is inefficient at implementing complex tasks, in large part because the memory and processor in its computing architecture are separated, performing insufficiently in computing speed and energy consumption. In recent years, optical neural networks (ONNs) have made a range of research progress in optical computing due to advantages such as sub-nanosecond latency, low heat dissipation, and high parallelism. ONNs are in prospect to provide support regarding computing speed and energy consumption for the further development of artificial intelligence with a novel computing paradigm. Herein, we first introduce the design method and principle of ONNs based on various optical elements. Then, we successively review the non-integrated ONNs consisting of volume optical components and the integrated ONNs composed of on-chip components. Finally, we summarize and discuss the computational density, nonlinearity, scalability, and practical applications of ONNs, and comment on the challenges and perspectives of the ONNs in the future development trends.

Terahertz spoof plasmonic neural network for diffractive information recognition and processing

All-optical diffractive neural networks, as analog artificial intelligence accelerators, leverage parallelism and analog computation for complex data processing. However, their low space transmission efficiency or large spatial dimensions hinder miniaturization and broader application. Here, we propose a terahertz spoof plasmonic neural network on a planar diffractive platform for direct multi-target recognition. Our approach employs a spoof surface plasmon polariton coupler array to construct a diffractive network layer, resulting in a compact, efficient, and easily integrable architecture. We designed three schemes: basis vector classification, multi-user recognition, and MNIST handwritten digit classification. Experimental results reveal that the terahertz spoof plasmonic neural network successfully classifies basis vectors, recognizes multi-user orientation information, and directly processes handwritten digits using a designed input framework comprising a metal grating array, transmitters, and receivers. This work broadens the application of terahertz plasmonic metamaterials, paving the way for terahertz on-chip integration, intelligent communication, and advanced computing systems.

Fully forward mode training for optical neural networks

Optical computing promises to improve the speed and energy efficiency of machine learning applications1-6. However, current approaches to efficiently train these models are limited by in silico emulation on digital computers. Here we develop a method called fully forward mode (FFM) learning, which implements the compute-intensive training process on the physical system. The majority of the machine learning operations are thus efficiently conducted in parallel on site, alleviating numerical modelling constraints. In free-space and integrated photonics, we experimentally demonstrate optical systems with state-of-the-art performances for a given network size. FFM learning shows training the deepest optical neural networks with millions of parameters achieves accuracy equivalent to the ideal model. It supports all-optical focusing through scattering media with a resolution of the diffraction limit; it can also image in parallel the objects hidden outside the direct line of sight at over a kilohertz frame rate and can conduct all-optical processing with light intensity as weak as subphoton per pixel (5.40 × 1018- operations-per-second-per-watt energy efficiency) at room temperature. Furthermore, we prove that FFM learning can automatically search non-Hermitian exceptional points without an analytical model. FFM learning not only facilitates orders-of-magnitude-faster learning processes, but can also advance applied and theoretical fields such as deep neural networks, ultrasensitive perception and topological photonics.

Partial coherence enhances parallelized photonic computing

Advancements in optical coherence control1-5 have unlocked many cutting-edge applications, including long-haul communication, light detection and ranging (LiDAR) and optical coherence tomography6-8. Prevailing wisdom suggests that using more coherent light sources leads to enhanced system performance and device functionalities9-11. Our study introduces a photonic convolutional processing system that takes advantage of partially coherent light to boost computing parallelism without substantially sacrificing accuracy, potentially enabling larger-size photonic tensor cores. The reduction of the degree of coherence optimizes bandwidth use in the photonic convolutional processing system. This breakthrough challenges the traditional belief that coherence is essential or even advantageous in integrated photonic accelerators, thereby enabling the use of light sources with less rigorous feedback control and thermal-management requirements for high-throughput photonic computing. Here we demonstrate such a system in two photonic platforms for computing applications: a photonic tensor core using phase-change-material photonic memories that delivers parallel convolution operations to classify the gaits of ten patients with Parkinson's disease with 92.2% accuracy (92.7% theoretically) and a silicon photonic tensor core with embedded electro-absorption modulators (EAMs) to facilitate 0.108 tera operations per second (TOPS) convolutional processing for classifying the Modified National Institute of Standards and Technology (MNIST) handwritten digits dataset with 92.4% accuracy (95.0% theoretically).

Direct electromagnetic information processing with planar diffractive neural network

Diffractive neural network in electromagnetic wave-driven system has attracted great attention due to its ultrahigh parallel computing capability and energy efficiency. However, recent neural networks based on the diffractive framework still face the bottlenecks of misalignment and relatively large size limiting their further applications. Here, we propose a planar diffractive neural network (pla-NN) with a highly integrated and conformal architecture to achieve direct signal processing in the microwave frequency. On the basis of printed circuit fabrication process, the misalignment could be effectively circumvented while enabling flexible extension for multiple conformal and stacking designs. We first conduct validation on the fashion-MNIST dataset and experimentally build up a system using the proposed network architecture for direct recognition of different geometry structures in the electromagnetic space. We envision that the presented architecture, once combined with the advanced dynamic maneuvering techniques and flexible topology, would exhibit unlimited potentials in the areas of high-performance computing, wireless sensing, and flexible wearable electronics.

Super-resolution diffractive neural network for all-optical direction of arrival estimation beyond diffraction limits

Wireless sensing of the wave propagation direction from radio sources lays the foundation for communication, radar, navigation, etc. However, the existing signal processing paradigm for the direction of arrival estimation requires the radio frequency electronic circuit to demodulate and sample the multichannel baseband signals followed by a complicated computing process, which places the fundamental limit on its sensing speed and energy efficiency. Here, we propose the super-resolution diffractive neural networks (S-DNN) to process electromagnetic (EM) waves directly for the DOA estimation at the speed of light. The multilayer meta-structures of S-DNN generate super-oscillatory angular responses in local angular regions that can perform the all-optical DOA estimation with angular resolutions beyond the diffraction limit. The spatial-temporal multiplexing of passive and reconfigurable S-DNNs is utilized to achieve high-resolution DOA estimation over a wide field of view. The S-DNN is validated for the DOA estimation of multiple radio sources over 5 GHz frequency bandwidth with estimation latency over two to four orders of magnitude lower than the state-of-the-art commercial devices in principle. The results achieve the angular resolution over an order of magnitude, experimentally demonstrated with four times, higher than diffraction-limited resolution. We also apply S-DNN's edge computing capability, assisted by reconfigurable intelligent surfaces, for extremely low-latency integrated sensing and communication with low power consumption. Our work is a significant step towards utilizing photonic computing processors to facilitate various wireless sensing and communication tasks with advantages in both computing paradigms and performance over electronic computing.

A programmable topological photonic chip

Controlling topological phases of light allows the observation of abundant topological phenomena and the development of robust photonic devices. The prospect of more sophisticated control with topological photonic devices for practical implementations requires high-level programmability. Here we demonstrate a fully programmable topological photonic chip with large-scale integration of silicon photonic nanocircuits and microresonators. Photonic artificial atoms and their interactions in our compound system can be individually addressed and controlled, allowing the arbitrary adjustment of structural parameters and geometrical configurations for the observation of dynamic topological phase transitions and diverse photonic topological insulators. Individual programming of artificial atoms on the generic chip enables the comprehensive statistical characterization of topological robustness against relatively weak disorders, and counterintuitive topological Anderson phase transitions induced by strong disorders. This generic topological photonic chip can be rapidly reprogrammed to implement multifunctionalities, providing a flexible and versatile platform for applications across fundamental science and topological technologies.

Role of spatial coherence in diffractive optical neural networks

Diffractive optical neural networks (DONNs) have emerged as a promising optical hardware platform for ultra-fast and energy-efficient signal processing for machine learning tasks, particularly in computer vision. Previous experimental demonstrations of DONNs have only been performed using coherent light. However, many real-world DONN applications require consideration of the spatial coherence properties of the optical signals. Here, we study the role of spatial coherence in DONN operation and performance. We propose a numerical approach to efficiently simulate DONNs under incoherent and partially coherent input illumination and discuss the corresponding computational complexity. As a demonstration, we train and evaluate simulated DONNs on the MNIST dataset of handwritten digits to process light with varying spatial coherence.

A blueprint for precise and fault-tolerant analog neural networks

Analog computing has reemerged as a promising avenue for accelerating deep neural networks (DNNs) to overcome the scalability challenges posed by traditional digital architectures. However, achieving high precision using analog technologies is challenging, as high-precision data converters are costly and impractical. In this work, we address this challenge by using the residue number system (RNS) and composing high-precision operations from multiple low-precision operations, thereby eliminating the need for high-precision data converters and information loss. Our study demonstrates that the RNS-based approach can achieve ≥99% FP32 accuracy with 6-bit integer arithmetic for DNN inference and 7-bit for DNN training. The reduced precision requirements imply that using RNS can achieve several orders of magnitude higher energy efficiency while maintaining the same throughput compared to conventional analog hardware with the same precision. We also present a fault-tolerant dataflow using redundant RNS to protect the computation against noise and errors inherent within analog hardware.

A Reconfigurable Bipolar Image Sensor for High-Efficiency Dynamic Vision Recognition

Dynamic vision perception and processing (DVPP) is in high demand by booming edge artificial intelligence. However, existing imaging systems suffer from low efficiency or low compatibility with advanced machine vision techniques. Here, we propose a reconfigurable bipolar image sensor (RBIS) for in-sensor DVPP based on a two-dimensional WSe2/GeSe heterostructure device. Owing to the gate-tunable and reversible built-in electric field, its photoresponse shows bipolarity as being positive or negative. High-efficiency DVPP incorporating front-end RBIS and back-end CNN is then demonstrated. It shows a high recognition accuracy of over 94.9% on the derived DVS128 data set and requires much fewer neural network parameters than that without RBIS. Moreover, we demonstrate an optimized device with a vertically stacked structure and a stable nonvolatile bipolarity, which enables more efficient DVPP hardware. Our work demonstrates the potential of fabricating DVPP devices with a simple structure, high efficiency, and outputs compatible with advanced algorithms.

Inverse design of compact nonvolatile reconfigurable silicon photonic devices with phase-change materials

In the development of silicon photonics, the continued downsizing of photonic integrated circuits will further increase the integration density, which augments the functionality of photonic chips. Compared with the traditional design method, inverse design presents a novel approach for achieving compact photonic devices. However, achieving compact, reconfigurable photonic devices with the inverse design that employs the traditional modulation method exemplified by the thermo-optic effect poses a significant challenge due to the weak modulation capability. Low-loss phase change materials (PCMs) exemplified by Sb2Se3 are a promising candidate for solving this problem benefiting from their high refractive index contrast. In this work, we first developed a robust inverse design method to realize reconfigurable silicon and phase-change materials hybrid photonic devices including mode converter and optical switch. The mode converter exhibits a broadband operation of >100 nm. The optical switch shows an extinction ratio of >25 dB and a multilevel switching of 41 (>5 bits) by simply changing the crystallinity of Sb2Se3. Here, we experimentally demonstrated a Sb2Se3/Si hybrid integrated optical switch for the first time, wherein routing can be switched by the phase transition of the whole Sb2Se3. Our work provides an effective solution for the design of photonic devices that is insensitive to fabrication errors, thereby paving the way for high integration density in future photonic chips.

Towards large-scale programmable silicon photonic chip for signal processing

Optical signal processing has been playing a crucial part as powerful engine for various information systems in the practical applications. In particular, achieving large-scale programmable chips for signal processing are highly desirable for high flexibility, low cost and powerful processing. Silicon photonics, which has been developed successfully in the past decade, provides a promising option due to its unique advantages. Here, recent progress of large-scale programmable silicon photonic chip for signal processing in microwave photonics, optical communications, optical computing, quantum photonics as well as dispersion controlling are reviewed. Particularly, we give a discussion about the realization of high-performance building-blocks, including ultra-low-loss silicon photonic waveguides, 2 × 2 Mach-Zehnder switches and microring resonator switches. The methods for configuring large-scale programmable silicon photonic chips are also discussed. The representative examples are summarized for the applications of beam steering, optical switching, optical computing, quantum photonic processing as well as optical dispersion controlling. Finally, we give an outlook for the challenges of further developing large-scale programmable silicon photonic chips.

Monolithic back-end-of-line integration of phase change materials into foundry-manufactured silicon photonics

Monolithic integration of novel materials without modifying the existing photonic component library is crucial to advancing heterogeneous silicon photonic integrated circuits. Here we show the introduction of a silicon nitride etch stop layer at select areas, coupled with low-loss oxide trench, enabling incorporation of functional materials without compromising foundry-verified device reliability. As an illustration, two distinct chalcogenide phase change materials (PCMs) with remarkable nonvolatile modulation capabilities, namely Sb2Se3 and Ge2Sb2Se4Te1, were monolithic back-end-of-line integrated, offering compact phase and intensity tuning units with zero-static power consumption. By employing these building blocks, the phase error of a push-pull Mach-Zehnder interferometer optical switch could be reduced with a 48% peak power consumption reduction. Mirco-ring filters with >5-bit wavelength selective intensity modulation and waveguide-based >7-bit intensity-modulation broadband attenuators could also be achieved. This foundry-compatible platform could open up the possibility of integrating other excellent optoelectronic materials into future silicon photonic process design kits.

Programmable Photonic Time Circuits for Highly Scalable Universal Unitaries

Programmable photonic circuits (PPCs) have garnered substantial interest for their potential in facilitating deep learning accelerations and universal quantum computations. Although photonic computation using PPCs offers ultrafast operation, energy-efficient matrix calculations, and room-temperature quantum states, its poor scalability hinders integration. This challenge arises from the temporally one-shot operation of propagating light in conventional PPCs, resulting in a light-speed increase in device footprints. Here we propose the concept of programmable photonic time circuits, utilizing time-cycle-based computations analogous to gate cycling in the von Neumann architecture and quantum computation. Our building block is a reconfigurable SU(2) time gate, consisting of two resonators with tunable resonances, and coupled via time-coded dual-channel gauge fields. We demonstrate universal U(N) operations with high fidelity using an assembly of the SU(2) time gates, substantially improving scalability from O(N^{2}) to O(N) in terms of both the footprint and the number of gates. This result paves the way for PPC implementation in very large-scale integration.

Demixing microwave signals using system-on-chip photonic processor

The integrated photonic processor, co-packaged with electronic peripherals, is proposed for blind source separation of microwave signals, which separates signal-of-interest from dynamic interference with real-time adaptability.

Ultrabroadband high-resolution silicon RF-photonic beamformer

Microwave photonics aims to overcome the limitations of radiofrequency devices and systems by leveraging the unique properties of optics in terms of low loss and power consumption, broadband operation, immunity to interference and tunability. This enables versatile functions like beam steering, crucial in emerging applications such as the Internet of Things (IoT) and 5/6G networks. The main problem with current photonic beamforming architectures is that there is a tradeoff between resolution and bandwidth, which has not yet been solved. Here we propose and experimentally demonstrate a novel switched optical delay line beamformer architecture that is capable of achieving the desired maximum resolution (i.e., 2M pointing angles for M-bit coding) and provides broadband operation simultaneously. The concept is demonstrated by means of a compact (8 × 3 mm2) 8 (5-bit) delay line Silicon Photonic chip implementation capable of addressing 32 pointing angles and offering 20 GHz bandwidth operation.

Correlated optical convolutional neural network with "quantum speedup"

Compared with electrical neural networks, optical neural networks (ONNs) have the potentials to break the limit of the bandwidth and reduce the consumption of energy, and therefore draw much attention in recent years. By far, several types of ONNs have been implemented. However, the current ONNs cannot realize the acceleration as powerful as that indicated by the models like quantum neural networks. How to construct and realize an ONN with the quantum speedup is a huge challenge. Here, we propose theoretically and demonstrate experimentally a new type of optical convolutional neural network by introducing the optical correlation. It is called the correlated optical convolutional neural network (COCNN). We show that the COCNN can exhibit "quantum speedup" in the training process. The character is verified from the two aspects. One is the direct illustration of the faster convergence by comparing the loss function curves of the COCNN with that of the traditional convolutional neural network (CNN). Such a result is compatible with the training performance of the recently proposed quantum convolutional neural network (QCNN). The other is the demonstration of the COCNN's capability to perform the QCNN phase recognition circuit, validating the connection between the COCNN and the QCNN. Furthermore, we take the COCNN analog to the 3-qubit QCNN phase recognition circuit as an example and perform an experiment to show the soundness and the feasibility of it. The results perfectly match the theoretical calculations. Our proposal opens up a new avenue for realizing the ONNs with the quantum speedup, which will benefit the information processing in the era of big data.

Freeform direct-write and rewritable photonic integrated circuits in phase-change thin films

Photonic integrated circuits (PICs) with rapid prototyping and reprogramming capabilities promise revolutionary impacts on a plethora of photonic technologies. We report direct-write and rewritable photonic circuits on a low-loss phase-change material (PCM) thin film. Complete end-to-end PICs are directly laser-written in one step without additional fabrication processes, and any part of the circuit can be erased and rewritten, facilitating rapid design modification. We demonstrate the versatility of this technique for diverse applications, including an optical interconnect fabric for reconfigurable networking, a photonic crossbar array for optical computing, and a tunable optical filter for optical signal processing. By combining the programmability of the direct laser writing technique with PCM, our technique unlocks opportunities for programmable photonic networking, computing, and signal processing. Moreover, the rewritable photonic circuits enable rapid prototyping and testing in a convenient and cost-efficient manner, eliminate the need for nanofabrication facilities, and thus promote the proliferation of photonics research and education to a broader community.

Redundancy-free integrated optical convolver for optical neural networks based on arrayed waveguide grating

Optical neural networks (ONNs) have gained significant attention due to their potential for high-speed and energy-efficient computation in artificial intelligence. The implementation of optical convolutions plays a vital role in ONNs, as they are fundamental operations within neural network architectures. However, state-of-the-art convolution architectures often suffer from redundant inputs, leading to substantial resource waste. Here, we demonstrate an integrated optical convolution architecture that leverages the inherent routing principles of arrayed waveguide grating (AWG) to execute the sliding of convolution kernel and summation of results. M × N multiply-accumulate (MAC) operations are facilitated by M + N units within a single clock cycle, thus eliminating the redundancy. In the experiment, we achieved 5 bit precision and 91.9 % accuracy in the handwritten digit recognition task confirming the reliability of our approach. Its redundancy-free architecture, low power consumption, high compute density (8.53 teraOP mm-1 s-1) and scalability make it a valuable contribution to the field of optical neural networks, thereby paving the way for future advancements in high-performance computing and artificial intelligence applications.

Backpropagation-free training of deep physical neural networks

Recent successes in deep learning for vision and natural language processing are attributed to larger models but come with energy consumption and scalability issues. Current training of digital deep learning models primarily relies on backpropagation that is unsuitable for physical implementation. Here, we proposed a simple deep neural network architecture augmented by a physical local learning (PhyLL) algorithm, enabling supervised and unsupervised training of deep physical neural networks, without detailed knowledge of the nonlinear physical layer's properties. We trained diverse wave-based physical neural networks in vowel and image classification experiments, showcasing our approach's universality. Our method shows advantages over other hardware-aware training schemes by improving training speed, enhancing robustness, and reducing power consumption by eliminating the need for system modelling and thus decreasing digital computation.

Reproduction of mode-locked pulses by spectrotemporal domain-informed deep learning

The accurate reproduction of unique pulse states in a mode-locked fiber laser is an important scientific issue and has wide applications in the laser industry. We present what we believe to be a novel method for automatically and precisely reproducing targeted soliton states in a mode-locked fiber laser by spectrotemporal domain-informed deep learning. Targeted solitons are experimentally reproduced via a superior matching process with a spectrotemporal mean square error (MSE) of 3.99 × 10-5. The outstanding feature of our reproduction algorithm is that the pulse information in both the spectral and temporal domains is jointly adopted for reconstructing targeted soliton states from white noise, rather than establishing arbitrary mode-locked pulse states, as described in previous studies. Additionally, a single-layer perceptron model is proposed to retrieve the phase distribution of a mode-locked pulse, validating the physical completeness of our reproduction approach. Our approach advances ultrafast laser technology, enabling the precise control of pulse dynamics in applications such as optical communication and nonlinear optics.

Transferable learning on analog hardware

While analog neural network (NN) accelerators promise massive energy and time savings, an important challenge is to make them robust to static fabrication error. Present-day training methods for programmable photonic interferometer circuits, a leading analog NN platform, do not produce networks that perform well in the presence of static hardware errors. Moreover, existing hardware error correction techniques either require individual retraining of every analog NN (which is impractical in an edge setting with millions of devices), place stringent demands on component quality, or introduce hardware overhead. We solve all three problems by introducing one-time error-aware training techniques that produce robust NNs that match the performance of ideal hardware and can be exactly transferred to arbitrary highly faulty photonic NNs with hardware errors up to five times larger than present-day fabrication tolerances.

Biasing the quantum vacuum to control macroscopic probability distributions

Quantum field theory suggests that electromagnetic fields naturally fluctuate, and these fluctuations can be harnessed as a source of perfect randomness. Many potential applications of randomness rely on controllable probability distributions. We show that vacuum-level bias fields injected into multistable optical systems enable a controllable source of quantum randomness, and we demonstrated this concept in an optical parametric oscillator (OPO). By injecting bias pulses with less than one photon on average, we controlled the probabilities of the two possible OPO output states. The potential of our approach for sensing sub-photon-level fields was demonstrated by reconstructing the temporal shape of fields below the single-photon level. Our results provide a platform to study quantum dynamics in nonlinear driven-dissipative systems and point toward applications in probabilistic computing and weak field sensing.