Cuttlefish eye-inspired artificial vision for high-quality imaging under uneven illumination conditions

Seeing through arthropod eyes: An AI-assisted, biomimetic approach for high-resolution, multi-task imaging

Arthropods have intricate compound eyes and optic neuropils, exhibiting exceptional visual capabilities. Combining the strengths of digital imaging with the features of natural arthropod visual systems offers a promising approach to harness wide-angle vision and depth perception while addressing limitations like low resolving power. Here, we present an artificial intelligence-assisted biomimetic system modeled after arthropod vision. We developed a biomimetic compound eye camera with an effective pixel number of 4.3 megapixels capable of producing full-color panoramic images with a viewing angle of 165° and resolving power of 40 micrometers. Using rich visual information, our system achieves high-fidelity image reconstruction, precise 3D position prediction, high-accuracy classification, and pattern recognition through a multistage neural network. Moreover, our compact biomimetic visual system can simultaneously track the 3D motion of multiple miniature targets independently. The proof-of-concept biomimetic arthropod visual system offers a computational panoramic imaging solution, advancing applications in industry, medicine, and robotics.

In-sensor multilevel image adjustment for high-clarity contour extraction using adjustable synaptic phototransistors

Robotic vision has traditionally relied on high-performance yet resource-intensive computing solutions, which necessitate high-throughput data transmission from vision sensors to remote computing servers, sacrificing energy efficiency and processing speed. A promising solution is data compaction through contour extraction, visualizing only the outlines of objects while eliminating superfluous backgrounds. Here, we introduce an in-sensor multilevel image adjustment method using adjustable synaptic phototransistors, enabling the capture of well-defined images with optimal brightness and contrast suitable for achieving high-clarity contour extraction. This is enabled by emulating dopamine-mediated neuronal excitability regulation mechanisms. Electrostatic gating effect either facilitates or inhibits time-dependent photocurrent accumulation, adjusting photo-responses to varying lighting conditions. Through excitatory and inhibitory modes, the adjustable synaptic phototransistor enhances visibility of dim and bright regions, respectively, facilitating distinct contour extraction and high-accuracy semantic segmentation. Evaluations using road images demonstrate improvement of both object detection accuracy and intersection over union, and compression of data volume.

Ultrabroadband Detection and Self-Powered Functionality in Quasi-One-Dimensional Nb3Se12I Nanowire Photodetectors for Bionic Vision Applications

The burgeoning fields of the Internet of things (IoT) and artificial intelligence (AI) have escalated the demands for image sensing technologies, necessitating advancements in sensor efficiency and functionality. Traditional image sensors, structured on von Neumann architectures with discrete processing units, face challenges, such as high power consumption, latency, and escalated hardware costs. In this work, we introduced a unique approach through the development of a quasi-one-dimensional nanowire Nb3Se12I-based double-ended photosensor. The advanced sensor not only replicated the adaptive behavior of biological vision systems but also effectively managed the decreased sensitivity triggered by intense light stimuli. The integration of the photothermoelectric and bolometric effects allows the device to operate in a self-powered mode, offering broadband detectivity ranging from visible (405 nm) to midwave infrared (4060 nm). Additionally, the quasi-one-dimensional structure enables an angle-dependent response to polarized light with a polarization ratio of 1.83. Our findings suggest that the biomimetic vision adaptive sensor based on Nb3Se12I could effectively enhance the capabilities of smart optical sensors and machine vision systems.

Soft Materials and Devices Enabling Sensorimotor Functions in Soft Robots

Sensorimotor functions, the seamless integration of sensing, decision-making, and actuation, are fundamental for robots to interact with their environments. Inspired by biological systems, the incorporation of soft materials and devices into robotics holds significant promise for enhancing these functions. However, current robotics systems often lack the autonomy and intelligence observed in nature due to limited sensorimotor integration, particularly in flexible sensing and actuation. As the field progresses toward soft, flexible, and stretchable materials, developing such materials and devices becomes increasingly critical for advanced robotics. Despite rapid advancements individually in soft materials and flexible devices, their combined applications to enable sensorimotor capabilities in robots are emerging. This review addresses this emerging field by providing a comprehensive overview of soft materials and devices that enable sensorimotor functions in robots. We delve into the latest development in soft sensing technologies, actuation mechanism, structural designs, and fabrication techniques. Additionally, we explore strategies for sensorimotor control, the integration of artificial intelligence (AI), and practical application across various domains such as healthcare, augmented and virtual reality, and exploration. By drawing parallels with biological systems, this review aims to guide future research and development in soft robots, ultimately enhancing the autonomy and adaptability of robots in unstructured environments.

A Biomimetic Pose Estimation and Target Perception Strategy for Transmission Line Maintenance UAVs

High-voltage overhead power lines serve as the carrier of power transmission and are crucial to the stable operation of the power system. Therefore, it is particularly important to detect and remove foreign objects attached to transmission lines, as soon as possible. In this context, the widespread promotion and application of smart robots in the power industry can help address the increasingly complex challenges faced by the industry and ensure the efficient, economical, and safe operation of the power grid system. This article proposes a bionic-based UAV pose estimation and target perception strategy, which aims to address the lack of pattern recognition and automatic tracking capabilities of traditional power line inspection UAVs, as well as the poor robustness of visual odometry. Compared with the existing UAV environmental perception solutions, the bionic target perception algorithm proposed in this article can efficiently extract point and line features from infrared images and realize the target detection and automatic tracking function of small multi-rotor drones in the power line scenario, with low power consumption.

Inspiration from Visual Ecology for Advancing Multifunctional Robotic Vision Systems: Bio-inspired Electronic Eyes and Neuromorphic Image Sensors

In robotics, particularly for autonomous navigation and human-robot collaboration, the significance of unconventional imaging techniques and efficient data processing capabilities is paramount. The unstructured environments encountered by robots, coupled with complex missions assigned to them, present numerous challenges necessitating diverse visual functionalities, and consequently, the development of multifunctional robotic vision systems has become indispensable. Meanwhile, rich diversity inherent in animal vision systems, honed over evolutionary epochs to meet their survival demands across varied habitats, serves as a profound source of inspirations. Here, recent advancements in multifunctional robotic vision systems drawing inspiration from natural ocular structures and their visual perception mechanisms are delineated. First, unique imaging functionalities of natural eyes across terrestrial, aerial, and aquatic habitats and visual signal processing mechanism of humans are explored. Then, designs and functionalities of bio-inspired electronic eyes are explored, engineered to mimic key components and underlying optical principles of natural eyes. Furthermore, neuromorphic image sensors are discussed, emulating functional properties of synapses, neurons, and retinas and thereby enhancing accuracy and efficiency of robotic vision tasks. Next, integration examples of electronic eyes with mobile robotic/biological systems are introduced. Finally, a forward-looking outlook on the development of bio-inspired electronic eyes and neuromorphic image sensors is provided.

Dual-Coated Antireflective Film for Flexible and Robust Multi-Environmental Optoelectronic Applications

Artificial antireflective nanostructured surfaces, inspired by moth eyes, effectively reduce optical losses at interfaces, offering significant advantages in enhancing optical performance in various optoelectronic applications, including solar cells, light-emitting diodes, and cameras. However, their limited flexibility and low surface hardness constrain their broader use. In this study, we introduce a universal antireflective film by integrating nanostructures on both sides of a thin polycarbonate film. One side was thinly coated with Al2O3 for its high hardness, enhancing surface durability while maintaining flexibility. The opposite side was coated with SiO2 to optimize antireflective properties, making the film suitable for diverse environments (i.e., air, water, and adhesives). This dual-coating strategy resulted in a mechanically robust and flexible antireflective film with superior optical properties in various conditions. We demonstrated the universal capabilities of our antireflective film via optical simulations and experiments with the fabricated film in different environments.

Feline eye-inspired artificial vision for enhanced camouflage breaking under diverse light conditions

Biologically inspired artificial vision research has led to innovative robotic vision systems with low optical aberration, wide field of view, and compact form factor. However, challenges persist in object detection and recognition against complex backgrounds and varied lighting. Inspired by the feline eye, which features a vertically elongated pupil and tapetum lucidum, this study introduces an artificial vision system designed for superior object detection and recognition in a monocular framework. Using a slit-like elliptical aperture and a patterned metal reflector beneath a hemispherical silicon photodiode array, the system reduces excessive light and enhances photosensitivity. This design achieves clear focus under bright light and enhanced sensitivity in dim conditions. Theoretical and experimental analyses demonstrate the system's ability to filter redundant information and detect camouflaged objects in diverse lighting, representing a substantial advancement in monocular camera technology and the potential of biomimicry in optical innovations.

Cephalopod-Inspired Nanomaterials for Optical and Thermal Regulation: Mechanisms, Applications and Perspectives

The manipulation of interactions between light and matter plays a crucial role in the evolution of organisms and a better life for humans. As a result of natural selection, precise light-regulatory systems of biology have been engineered that provide many powerful and promising bioinspired strategies. As the "king of disguise", cephalopods, which can perfectly control the propagation of light and thus achieve excellent surrounding-matching via their delicate skin structure, have made themselves an exciting source of inspiration for developing optical and thermal regulation nanomaterials. This review presents cutting-edge advancements in cephalopod-inspired optical and thermal regulation nanomaterials, highlighting the key milestones and breakthroughs achieved thus far. We begin with the underlying mechanisms of the adaptive color-changing ability of cephalopods, as well as their special hierarchical skin structure. Then, different types of bioinspired nanomaterials and devices are comprehensively summarized. Furthermore, some advanced and emerging applications of these nanomaterials and devices, including camouflage, thermal management, pixelation, medical health, sensing and wireless communication, are addressed. Finally, some remaining but significant challenges and potential directions for future work are discussed. We anticipate that this comprehensive review will promote the further development of cephalopod-inspired nanomaterials for optical and thermal regulation and trigger ideas for bioinspired design of nanomaterials in multidisciplinary applications.

Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array

The bioinspired camera, comprising a single lens and a curved image sensor-a photodiode array on a curved surface-, was born of flexible electronics. Its economical build lends itself well to space-constrained machine vision applications. The curved sensor, much akin to the retina, helps image focusing, but the curvature also creates a problem of image distortion, which can undermine machine vision tasks such as object recognition. Here we report an anti-distortion single-lens camera, where 4096 silicon photodiodes arrayed on a curved surface in a nonuniform pattern assimilated to the distorting optics are the key to anti-distortion engineering. That is, the photo-pixel distribution pattern itself is warped in the same manner as images are warped, which correctively reverses distortion. Acquired images feature no appreciable distortion across a 120° horizontal view, as confirmed by their neural-network recognition accuracies. This distortion correction via photo-pixel array reconfiguration is a form of in-sensor computing.

Multidimensional vision sensors for information processing

The visual scene in the physical world integrates multidimensional information (spatial, temporal, polarization, spectrum and so on) and typically shows unstructured characteristics. Conventional image sensors cannot process this multidimensional vision data, creating a need for vision sensors that can efficiently extract features from substantial multidimensional vision data. Vision sensors are able to transform the unstructured visual scene into featured information without relying on sophisticated algorithms and complex hardware. The response characteristics of sensors can be abstracted into operators with specific functionalities, allowing for the efficient processing of perceptual information. In this Review, we delve into the hardware implementation of multidimensional vision sensors, exploring their working mechanisms and design principles. We exemplify multidimensional vision sensors built on emerging devices and silicon-based system integration. We further provide benchmarking metrics for multidimensional vision sensors and conclude with the principle of device-system co-design and co-optimization.

Avian eye-inspired perovskite artificial vision system for foveated and multispectral imaging

Avian eyes have deep central foveae as a result of extensive evolution. Deep foveae efficiently refract incident light, creating a magnified image of the target object and making it easier to track object motion. These features are essential for detecting and tracking remote objects in dynamic environments. Furthermore, avian eyes respond to a wide spectrum of light, including visible and ultraviolet light, allowing them to efficiently distinguish the target object from complex backgrounds. Despite notable advances in artificial vision systems that mimic animal vision, the exceptional object detection and targeting capabilities of avian eyes via foveated and multispectral imaging remain underexplored. Here, we present an artificial vision system that capitalizes on these aspects of avian vision. We introduce an artificial fovea and vertically stacked perovskite photodetector arrays whose designs were optimized by theoretical simulations for the demonstration of foveated and multispectral imaging. The artificial vision system successfully identifies colored and mixed-color objects and detects remote objects through foveated imaging. The potential for use in uncrewed aerial vehicles that need to detect, track, and recognize distant targets in dynamic environments is also discussed. Our avian eye-inspired perovskite artificial vision system marks a notable advance in bioinspired artificial visions.

Advanced visual components inspired by animal eyes

Artificial vision systems pervade our daily lives as a foremost sensing apparatus in various digital technologies, from smartphones to autonomous cars and robotics. The broad range of applications for conventional vision systems requires facile adaptation under extreme and dynamic visual environments. However, these current needs have complicated individual visual components for high-quality image acquisition and processing, which indeed leads to a decline in efficiency in the overall system. Here, we review recent advancements in visual components for high-performance visual processing based on strategies of biological eyes that execute diverse imaging functionalities and sophisticated visual processes with simple and concise ocular structures. This review first covers the structures and functions of biological eyes (i.e., single-lens eyes and compound eyes), which contain micro-optic components and nanophotonic structures. After that, we focus on their inspirations in imaging optics/photonics, light-trapping and filtering components, and retinomorphic devices. We discuss the remaining challenges and notable biological structures waiting to be implemented.

Bioinspired 3D flexible devices and functional systems

Flexible devices and functional systems with elaborated three-dimensional (3D) architectures can endow better mechanical/electrical performances, more design freedom, and unique functionalities, when compared to their two-dimensional (2D) counterparts. Such 3D flexible devices/systems are rapidly evolving in three primary directions, including the miniaturization, the increasingly merged physical/artificial intelligence and the enhanced adaptability and capabilities of heterogeneous integration. Intractable challenges exist in this emerging research area, such as relatively poor controllability in the locomotion of soft robotic systems, mismatch of bioelectronic interfaces, and signal coupling in multi-parameter sensing. By virtue of long-time-optimized materials, structures and processes, natural organisms provide rich sources of inspiration to address these challenges, enabling the design and manufacture of many bioinspired 3D flexible devices/systems. In this Review, we focus on bioinspired 3D flexible devices and functional systems, and summarize their representative design concepts, manufacturing methods, principles of structure-function relationship and broad-ranging applications. Discussions on existing challenges, potential solutions and future opportunities are also provided to usher in further research efforts toward realizing bioinspired 3D flexible devices/systems with precisely programmed shapes, enhanced mechanical/electrical performances, and high-level physical/artificial intelligence.

Flexible and Stretchable Light-Emitting Diodes and Photodetectors for Human-Centric Optoelectronics

Optoelectronic devices with unconventional form factors, such as flexible and stretchable light-emitting or photoresponsive devices, are core elements for the next-generation human-centric optoelectronics. For instance, these deformable devices can be utilized as closely fitted wearable sensors to acquire precise biosignals that are subsequently uploaded to the cloud for immediate examination and diagnosis, and also can be used for vision systems for human-interactive robotics. Their inception was propelled by breakthroughs in novel optoelectronic material technologies and device blueprinting methodologies, endowing flexibility and mechanical resilience to conventional rigid optoelectronic devices. This paper reviews the advancements in such soft optoelectronic device technologies, honing in on various materials, manufacturing techniques, and device design strategies. We will first highlight the general approaches for flexible and stretchable device fabrication, including the appropriate material selection for the substrate, electrodes, and insulation layers. We will then focus on the materials for flexible and stretchable light-emitting diodes, their device integration strategies, and representative application examples. Next, we will move on to the materials for flexible and stretchable photodetectors, highlighting the state-of-the-art materials and device fabrication methods, followed by their representative application examples. At the end, a brief summary will be given, and the potential challenges for further development of functional devices will be discussed as a conclusion.

Design and Fabrication of Stretchable Microwave Transmission Lines Based on a Quasi-Microstrip Structure

Radio frequency (RF) electronics are vital components of stretchable electronics that require wireless capabilities, ranging from skin-interfaced wearable systems to implantable devices to soft robotics. One of the key challenges in stretchable electronics is achieving near-lossless transmission line technology that can carry high-frequency electrical signals between various RF components. Almost all existing stretchable interconnection strategies only demonstrate direct current or low-frequency electrical properties, limiting their use in high frequencies, especially in the MHz to GHz range. Here, we describe the design and fabrication of a simple stretchable RF transmission line strategy that integrates a quasi-microstrip structure into a stretchable serpentine microscale interconnection. We show the effects of quasi-microstrip structural dimensions on the RF performance based on detailed quantitative analysis and experimentally demonstrate the optimized device capable of carrying RF signals with frequencies of up to 40 GHz with near-lossless characteristics. To show the potential application of our transmission line in stretchable microwave electronics, we designed a single-stage power amplifier system with a gain of 9.8 dB at 9 GHz that fully utilizes our quasi-microstrip transmission line technology.

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

High-performance robotic vision empowers mobile and humanoid robots to detect and identify their surrounding objects efficiently, which enables them to cooperate with humans and assist human activities. For error-free execution of these robots' tasks, efficient imaging and data processing capabilities are essential, even under diverse and complex environments. However, conventional technologies fall short of meeting the high-standard requirements of robotic vision under such circumstances. Here, we discuss recent progress in artificial vision systems with high-performance imaging and data processing capabilities enabled by distinctive electrical, optical, and mechanical characteristics of nanomaterials surpassing the limitations of traditional silicon technologies. In particular, we focus on nanomaterial-based electronic eyes and in-sensor processing devices inspired by biological eyes and animal visual recognition systems, respectively. We provide perspectives on key nanomaterials, device components, and their functionalities, as well as explain the remaining challenges and future prospects of the artificial vision systems.

Mechanically-Guided 3D Assembly for Architected Flexible Electronics

Architected flexible electronic devices with rationally designed 3D geometries have found essential applications in biology, medicine, therapeutics, sensing/imaging, energy, robotics, and daily healthcare. Mechanically-guided 3D assembly methods, exploiting mechanics principles of materials and structures to transform planar electronic devices fabricated using mature semiconductor techniques into 3D architected ones, are promising routes to such architected flexible electronic devices. Here, we comprehensively review mechanically-guided 3D assembly methods for architected flexible electronics. Mainstream methods of mechanically-guided 3D assembly are classified and discussed on the basis of their fundamental deformation modes (i.e., rolling, folding, curving, and buckling). Diverse 3D interconnects and device forms are then summarized, which correspond to the two key components of an architected flexible electronic device. Afterward, structure-induced functionalities are highlighted to provide guidelines for function-driven structural designs of flexible electronics, followed by a collective summary of their resulting applications. Finally, conclusions and outlooks are given, covering routes to achieve extreme deformations and dimensions, inverse design methods, and encapsulation strategies of architected 3D flexible electronics, as well as perspectives on future applications.

Perceiving like a Bat: Hierarchical 3D Geometric-Semantic Scene Understanding Inspired by a Biomimetic Mechanism

Geometric-semantic scene understanding is a spatial intelligence capability that is essential for robots to perceive and navigate the world. However, understanding a natural scene remains challenging for robots because of restricted sensors and time-varying situations. In contrast, humans and animals are able to form a complex neuromorphic concept of the scene they move in. This neuromorphic concept captures geometric and semantic aspects of the scenario and reconstructs the scene at multiple levels of abstraction. This article seeks to reduce the gap between robot and animal perception by proposing an ingenious scene-understanding approach that seamlessly captures geometric and semantic aspects in an unexplored environment. We proposed two types of biologically inspired environment perception methods, i.e., a set of elaborate biomimetic sensors and a brain-inspired parsing algorithm related to scene understanding, that enable robots to perceive their surroundings like bats. Our evaluations show that the proposed scene-understanding system achieves competitive performance in image semantic segmentation and volumetric-semantic scene reconstruction. Moreover, to verify the practicability of our proposed scene-understanding method, we also conducted real-world geometric-semantic scene reconstruction in an indoor environment with our self-developed drone.

Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.