Miniature curved artificial compound eyes

Avian eye-inspired perovskite artificial vision system for foveated and multispectral imaging

Avian eyes have deep central foveae as a result of extensive evolution. Deep foveae efficiently refract incident light, creating a magnified image of the target object and making it easier to track object motion. These features are essential for detecting and tracking remote objects in dynamic environments. Furthermore, avian eyes respond to a wide spectrum of light, including visible and ultraviolet light, allowing them to efficiently distinguish the target object from complex backgrounds. Despite notable advances in artificial vision systems that mimic animal vision, the exceptional object detection and targeting capabilities of avian eyes via foveated and multispectral imaging remain underexplored. Here, we present an artificial vision system that capitalizes on these aspects of avian vision. We introduce an artificial fovea and vertically stacked perovskite photodetector arrays whose designs were optimized by theoretical simulations for the demonstration of foveated and multispectral imaging. The artificial vision system successfully identifies colored and mixed-color objects and detects remote objects through foveated imaging. The potential for use in uncrewed aerial vehicles that need to detect, track, and recognize distant targets in dynamic environments is also discussed. Our avian eye-inspired perovskite artificial vision system marks a notable advance in bioinspired artificial visions.

Broadband Photodetectors and Imagers in Stretchable Electronics Packaging

The integration of flexible electronics with optics can help realize a powerful tool that facilitates the creation of a smart society wherein internal evaluations can be easily performed nondestructively from the surface of various objects that is used or encountered in daily lives. Here, organic-material-based stretchable optical sensors and imagers that possess both bending capability and rubber-like elasticity are reviewed. The latest trends in nondestructive evaluation equipment that enable simple on-site evaluations of health conditions and abnormalities are discussed without subjecting the targeted living bodies and various objects to mechanical stress. Real-time performance under real-life conditions is becoming increasingly important for creating smart societies interwoven with optical technologies. In particular, the terahertz (THz)-wave region offers a substance- and state-specific fingerprint spectrum that enables instantaneous analyses. However, to make THz sensors accessible, the following issues must be addressed: broadband and high-sensitivity at room temperature, stretchability to follow the surface movements of targets, and digital transformation compatibility. The materials, electronics packaging, and remote imaging systems used to overcome these issues are discussed in detail. Ultimately, stretchable optical sensors and imagers with highly sensitive and broadband THz sensors can facilitate the multifaceted on-site evaluation of solids, liquids, and gases.

Biomimetic nanocluster photoreceptors for adaptative circular polarization vision

Nanoclusters with atomically precise structures and discrete energy levels are considered as nanoscale semiconductors for artificial intelligence. However, nanocluster electronic engineering and optoelectronic behavior have remained obscure and unexplored. Hence, we create nanocluster photoreceptors inspired by mantis shrimp visual systems to satisfy the needs of compact but multi-task vision hardware and explore the photo-induced electronic transport. Wafer-scale arrayed photoreceptors are constructed by a nanocluster-conjugated molecule heterostructure. Nanoclusters perform as an in-sensor charge reservoir to tune the conductance levels of artificial photoreceptors by a light valve mechanism. A ligand-assisted charge transfer process takes place at nanocluster interface and it features an integration of spectral-dependent visual adaptation and circular polarization recognition. This approach is further employed for developing concisely structured, multi-task, and compact artificial visual systems and provides valuable guidelines for nanocluster neuromorphic devices.

Bioinspired 3D flexible devices and functional systems

Flexible devices and functional systems with elaborated three-dimensional (3D) architectures can endow better mechanical/electrical performances, more design freedom, and unique functionalities, when compared to their two-dimensional (2D) counterparts. Such 3D flexible devices/systems are rapidly evolving in three primary directions, including the miniaturization, the increasingly merged physical/artificial intelligence and the enhanced adaptability and capabilities of heterogeneous integration. Intractable challenges exist in this emerging research area, such as relatively poor controllability in the locomotion of soft robotic systems, mismatch of bioelectronic interfaces, and signal coupling in multi-parameter sensing. By virtue of long-time-optimized materials, structures and processes, natural organisms provide rich sources of inspiration to address these challenges, enabling the design and manufacture of many bioinspired 3D flexible devices/systems. In this Review, we focus on bioinspired 3D flexible devices and functional systems, and summarize their representative design concepts, manufacturing methods, principles of structure-function relationship and broad-ranging applications. Discussions on existing challenges, potential solutions and future opportunities are also provided to usher in further research efforts toward realizing bioinspired 3D flexible devices/systems with precisely programmed shapes, enhanced mechanical/electrical performances, and high-level physical/artificial intelligence.

A Bio-Inspired Probabilistic Neural Network Model for Noise-Resistant Collision Perception

Bio-inspired models based on the lobula giant movement detector (LGMD) in the locust's visual brain have received extensive attention and application for collision perception in various scenarios. These models offer advantages such as low power consumption and high computational efficiency in visual processing. However, current LGMD-based computational models, typically organized as four-layered neural networks, often encounter challenges related to noisy signals, particularly in complex dynamic environments. Biological studies have unveiled the intrinsic stochastic nature of synaptic transmission, which can aid neural computation in mitigating noise. In alignment with these biological findings, this paper introduces a probabilistic LGMD (Prob-LGMD) model that incorporates a probability into the synaptic connections between multiple layers, thereby capturing the uncertainty in signal transmission, interaction, and integration among neurons. Comparative testing of the proposed Prob-LGMD model and two conventional LGMD models was conducted using a range of visual stimuli, including indoor structured scenes and complex outdoor scenes, all subject to artificial noise. Additionally, the model's performance was compared to standard engineering noise-filtering methods. The results clearly demonstrate that the proposed model outperforms all comparative methods, exhibiting a significant improvement in noise tolerance. This study showcases a straightforward yet effective approach to enhance collision perception in noisy environments.

Miniature bioinspired artificial compound eyes: microfabrication technologies, photodetection and applications

As an outstanding visual system for insects and crustaceans to cope with the challenges of survival, compound eye has many unique advantages, such as wide field of view, rapid response, infinite depth of field, low aberration and fast motion capture. However, the complex composition of their optical systems also presents significant challenges for manufacturing. With the continuous development of advanced materials, complex 3D manufacturing technologies and flexible electronic detectors, various ingenious and sophisticated compound eye imaging systems have been developed. This paper provides a comprehensive review on the microfabrication technologies, photoelectric detection and functional applications of miniature artificial compound eyes. Firstly, a brief introduction to the types and structural composition of compound eyes in the natural world is provided. Secondly, the 3D forming manufacturing techniques for miniature compound eyes are discussed. Subsequently, some photodetection technologies for miniature curved compound eye imaging are introduced. Lastly, with reference to the existing prototypes of functional applications for miniature compound eyes, the future development of compound eyes is prospected.

Nanomaterial-Based Artificial Vision Systems: From Bioinspired Electronic Eyes to In-Sensor Processing Devices

High-performance robotic vision empowers mobile and humanoid robots to detect and identify their surrounding objects efficiently, which enables them to cooperate with humans and assist human activities. For error-free execution of these robots' tasks, efficient imaging and data processing capabilities are essential, even under diverse and complex environments. However, conventional technologies fall short of meeting the high-standard requirements of robotic vision under such circumstances. Here, we discuss recent progress in artificial vision systems with high-performance imaging and data processing capabilities enabled by distinctive electrical, optical, and mechanical characteristics of nanomaterials surpassing the limitations of traditional silicon technologies. In particular, we focus on nanomaterial-based electronic eyes and in-sensor processing devices inspired by biological eyes and animal visual recognition systems, respectively. We provide perspectives on key nanomaterials, device components, and their functionalities, as well as explain the remaining challenges and future prospects of the artificial vision systems.

Development of a bio-inspired optical system that mimics accommodation and lighting regulation like the human eye

Bio-inspired optical systems have recently been developed using polarizers and liquid or rigid lenses. In this work, we propose a bio-inspired opto-mechatronic system that imitates the accommodation and regulation of light intensity as the human eye does. The system uses a polymeric lens as a cornea, an adjustable diaphragm as an iris, a tunable solid elastic lens as a crystalline lens, and a commercial sensor as a retina. We also present the development of the electronic control system to accommodate and regulate the amount of light that enters the system, for which two stepper motors, an Arduino control system, and light and movement sensors are used. The characterization of the system is presented together with the results obtained, where it can be seen that the system works in an acceptable range as the human eye does.

Advanced Biomimetic Multispectral Curved Compound Eye Camera for Aerial Multispectral Imaging in a Large Field of View

In this work, we demonstrated a new type of biomimetic multispectral curved compound eye camera (BM3C) inspired by insect compound eyes for aerial multispectral imaging in a large field of view. The proposed system exhibits a maximum field of view (FOV) of 120 degrees and seven-waveband multispectral images ranging from visible to near-infrared wavelengths. Pinhole imaging theory and the image registration method from feature detection are used to reconstruct the multispectral 3D data cube. An airborne imaging experiment is performed by assembling the BM3C on an unmanned aerial vehicle (UAV). As a result, radiation intensity curves of several objects are successfully obtained, and a land type classification is performed using the K-means method based on the aerial image as well. The developed BM3C is proven to have the capability for large FOV aerial multispectral imaging and shows great potential applications for distant detecting based on aerial imaging.

Curved fiber compound eye camera inspired by the Strepsiptera vision

The Strepsiptera vision possesses intriguing features of a large field of view (FOV) and relatively high resolution compared to normal compound eyes. However, it presents a significant challenge of the mismatch between the curved compound eyelet lens array and the planar image sensor to image in a large FOV for artificial compound eyes (ACE). We propose what we believe to be a novel curved fiber compound eye camera (CFCEC) here, which employs coherent fiber bundles as the optical relay system to transmit sub-images curvilinearly. A total of 106 eyelets are arranged based on a scheme similar to the Goldberg polyhedron, with the advantages of uniform interval and minor edge blindness. Then, a prototype of the CFCEC is fabricated and assembled. A series of experiments are conducted to assess the FOV, contrast, resolution, and overlap rate of FOV of the prototype. The results prove that the CFCEC has a total FOV of up to 160°×160° and a total overlap rate of FOV of approximately 65%, demonstrating the promising potential of the CFCEC in various applications, such as panoramic surveillance, 3D detection, and motion tracking.

Motion perception based on ON/OFF channels: A survey

Motion perception is an essential ability for animals and artificially intelligent systems interacting effectively, safely with surrounding objects and environments. Biological visual systems, that have naturally evolved over hundreds-million years, are quite efficient and robust for motion perception, whereas artificial vision systems are far from such capability. This paper argues that the gap can be significantly reduced by formulation of ON/OFF channels in motion perception models encoding luminance increment (ON) and decrement (OFF) responses within receptive field, separately. Such signal-bifurcating structure has been found in neural systems of many animal species articulating early motion is split and processed in segregated pathways. However, the corresponding biological substrates, and the necessity for artificial vision systems have never been elucidated together, leaving concerns on uniqueness and advantages of ON/OFF channels upon building dynamic vision systems to address real world challenges. This paper highlights the importance of ON/OFF channels in motion perception through surveying current progress covering both neuroscience and computationally modelling works with applications. Compared to related literature, this paper for the first time provides insights into implementation of different selectivity to directional motion of looming, translating, and small-sized target movement based on ON/OFF channels in keeping with soundness and robustness of biological principles. Existing challenges and future trends of such bio-plausible computational structure for visual perception in connection with hotspots of machine learning, advanced vision sensors like event-driven camera finally are discussed.

Double-Glued Multi-Focal Bionic Compound Eye Camera

Compound eye cameras are a vital component of bionics. Compound eye lenses are currently used in light field cameras, monitoring imaging, medical endoscopes, and other fields. However, the resolution of the compound eye lens is still low at the moment, which has an impact on the application scene. Photolithography and negative pressure molding were used to create a double-glued multi-focal bionic compound eye camera in this study. The compound eye camera has 83 microlenses, with ommatidium diameters ranging from 400 μm to 660 μm, and a 92.3 degree field-of-view angle. The double-gluing structure significantly improves the optical performance of the compound eye lens, and the spatial resolution of the ommatidium is 57.00 lp mm-1. Additionally, the measurement of speed is investigated. This double-glue compound eye camera has numerous potential applications in the military, machine vision, and other fields.

Toward next-generation endoscopes integrating biomimetic video systems, nonlinear optical microscopy, and deep learning

According to the World Health Organization, the proportion of the world's population over 60 years will approximately double by 2050. This progressive increase in the elderly population will lead to a dramatic growth of age-related diseases, resulting in tremendous pressure on the sustainability of healthcare systems globally. In this context, finding more efficient ways to address cancers, a set of diseases whose incidence is correlated with age, is of utmost importance. Prevention of cancers to decrease morbidity relies on the identification of precursor lesions before the onset of the disease, or at least diagnosis at an early stage. In this article, after briefly discussing some of the most prominent endoscopic approaches for gastric cancer diagnostics, we review relevant progress in three emerging technologies that have significant potential to play pivotal roles in next-generation endoscopy systems: biomimetic vision (with special focus on compound eye cameras), non-linear optical microscopies, and Deep Learning. Such systems are urgently needed to enhance the three major steps required for the successful diagnostics of gastrointestinal cancers: detection, characterization, and confirmation of suspicious lesions. In the final part, we discuss challenges that lie en route to translating these technologies to next-generation endoscopes that could enhance gastrointestinal imaging, and depict a possible configuration of a system capable of (i) biomimetic endoscopic vision enabling easier detection of lesions, (ii) label-free in vivo tissue characterization, and (iii) intelligently automated gastrointestinal cancer diagnostic.

A monocular wide-field speed sensor inspired by the crabs' visual system for traffic analysis

The development of visual sensors for traffic analysis can benefit from mimicking two fundamental aspects of the visual system of crabs: their panoramic vision and their visual processing strategy adapted to a flat world. First, the use of omnidirectional cameras in urban environments allows for analyzing the simultaneous movement of many objects of interest over broad areas. This would reduce the costs and complications associated with infrastructure: installation, synchronization, maintenance, and operation of traditional vision systems that use multiple cameras with a limited field of view. Second, in urban traffic analysis, the objects of interest (e.g. vehicles and pedestrians) move on the ground surface. This constraint allows the calculation of the 3D trajectory of the vehicles using a single camera without the need to use binocular vision techniques.The main contribution of this work is to show that the strategy used by crabs to visually analyze their habitat (monocular omnidirectional vision with the assumption of a flat world ) is useful for developing a simple and effective method to estimate the speed of vehicles on long trajectories in urban environments. It is shown that the proposed method estimates the speed with a root mean squared error of 2.7 km h^-1.

A Novel Fast Servo Tool Device with Double Piezoelectric Driving

The fast tool servo (FTS) technology has unique advantages in the machining of complex surfaces such as special-shaped targets and free-form surfaces. In view of the shortcomings in the performance of the existing FTS device, this paper puts forward a novel FTS which uses two piezoelectric ceramics instead of flexure hinges to provide restoring force. Firstly, the feasibility of the double-drive principle is verified theoretically, and the corresponding mechanism is optimized accordingly. Then, the system control hysteresis model is established and identified, and the appropriate control strategy is designed. Finally, the performances of the proposed FTS device are tested, and a typical microstructure is machined based on the device and ultra-precision lathe. The results indicate that the proposed device effectively improves the performance of the FTS system, which is useful for the processing of microstructures and free-form surfaces.

Diffractive lensless imaging with optimized Voronoi-Fresnel phase

Lensless cameras are a class of imaging devices that shrink the physical dimensions to the very close vicinity of the image sensor by replacing conventional compound lenses with integrated flat optics and computational algorithms. Here we report a diffractive lensless camera with spatially-coded Voronoi-Fresnel phase to achieve superior image quality. We propose a design principle of maximizing the acquired information in optics to facilitate the computational reconstruction. By introducing an easy-to-optimize Fourier domain metric, Modulation Transfer Function volume (MTFv), which is related to the Strehl ratio, we devise an optimization framework to guide the optimization of the diffractive optical element. The resulting Voronoi-Fresnel phase features an irregular array of quasi-Centroidal Voronoi cells containing a base first-order Fresnel phase function. We demonstrate and verify the imaging performance for photography applications with a prototype Voronoi-Fresnel lensless camera on a 1.6-megapixel image sensor in various illumination conditions. Results show that the proposed design outperforms existing lensless cameras, and could benefit the development of compact imaging systems that work in extreme physical conditions.

Heterogeneous compound eye camera for dual-scale imaging in a large field of view

Multi-scale imaging with large field of view is pivotal for fast motion detection and target identification. However, existing single camera systems are difficult to achieve snapshot multi-scale imaging with large field of view. To solve this problem, we propose a design method for heterogeneous compound eye, and fabricate a prototype of heterogeneous compound eye camera (HeCECam). This prototype which consists of a heterogeneous compound eye array, an optical relay system and a CMOS sensor, is capable of dual-scale imaging in large field of view (360°×141°). The heterogeneous compound eye array is composed of 31 wide-angle (WA) subeyes and 226 high-definition (HD) subeyes. An optical relay system is introduced to re-image the curved focal surface formed by the heterogeneous compound eye array on a CMOS sensor, resulting in a heterogeneous compound eye image containing dual-scale subimages. To verify the imaging characteristics of this prototype, a series of experiments, such as large field of view imaging, imaging performance, and real-world scene imaging, were conducted. The experiment results show that this prototype can achieve dual-scale imaging in large field of view and has excellent imaging performance. This makes the HeCECam has great potential for UAV navigation, wide-area surveillance, and location tracking, and paves the way for the practical use of bio-inspired compound eye cameras.

Perovskite Wide-Angle Field-Of-View Camera

Researchers have attempted to create wide-angle field-of-view (FOV) cameras inspired by the structure of the eyes of animals, including fisheye and compound eye cameras. However, realizing wide-angle FOV cameras simultaneously exhibiting low distortion and high spatial resolution remains a significant challenge. In this study, a novel wide-angle FOV camera is developed by combining a single large-area flexible perovskite photodetector (FP-PD) using computational technology. With this camera, the proposed single-photodetector imaging technique can obtain high-spatial-resolution images using only a single detector, and the large-area FP-PD can be bent further to collect light from a wide-angle FOV. The proposed camera demonstrates remarkable features of an extraordinarily tunable wide FOV (greater than 150°), high spatial resolution of 256 × 256 pixels, and low distortion. It is believed that the proposed compatible and extensible camera prototype will promote the development of high-performance versatile FOV cameras.

Accommodating unobservability to control flight attitude with optic flow

Attitude control is an essential flight capability. Whereas flying robots commonly rely on accelerometers¹ for estimating attitude, flying insects lack an unambiguous sense of gravity^2,3. Despite the established role of several sense organs in attitude stabilization^3-5, the dependence of flying insects on an internal gravity direction estimate remains unclear. Here we show how attitude can be extracted from optic flow when combined with a motion model that relates attitude to acceleration direction. Although there are conditions such as hover in which the attitude is unobservable, we prove that the ensuing control system is still stable, continuously moving into and out of these conditions. Flying robot experiments confirm that accommodating unobservability in this manner leads to stable, but slightly oscillatory, attitude control. Moreover, experiments with a bio-inspired flapping-wing robot show that residual, high-frequency attitude oscillations from flapping motion improve observability. The presented approach holds a promise for robotics, with accelerometer-less autopilots paving the road for insect-scale autonomous flying robots⁶. Finally, it forms a hypothesis on insect attitude estimation and control, with the potential to provide further insight into known biological phenomena^5,7,8 and to generate new predictions such as reduced head and body attitude variance at higher flight speeds⁹.

Long-working-distance 3D measurement with a bionic curved compound-eye camera

The bionic curved compound-eye camera is a bionic-inspired multi-aperture camera, which can be designed to have an overlap on the field of view (FOV) in between adjacent ommatidia so that 3D measurement is possible. In this work, we demonstrate the 3D measurement with a working distance of up to 3.2 m by a curved compound-eye camera. In that there are hundreds of ommatidia in the compound-eye camera, traditional calibration boards with a fixed-pitch pattern arrays are not applicable. A batch calibration method based on the CALTag calibration board for the compound-eye camera was designed. Next, the 3D measurement principle was described and a 3D measurement algorithm for the compound-eye camera was developed. Finally, the 3D measurement experiment on objects placed at different distances and directions from the compound-eye camera was performed. The experimental results show that the working range for 3D measurement can cover the whole FOV of 98° and the working distance can be as long as 3.2 m. Moreover, a complete depth map was reconstructed from a raw image captured by the compound-eye camera and demonstrated as well. The 3D measurement capability of the compound-eye camera at long working distance in a large FOV demonstrated in this work has great potential applications in areas such as unmanned aerial vehicle (UAV) obstacle avoidance and robot navigation.

Miniature optoelectronic compound eye camera

Inspired by insect compound eyes (CEs) that feature unique optical schemes for imaging, there has recently been growing interest in developing optoelectronic CE cameras with comparable size and functions. However, considering the mismatch between the complex 3D configuration of CEs and the planar nature of available imaging sensors, it is currently challenging to reach this end. Here, we report a paradigm in miniature optoelectronic integrated CE camera by manufacturing polymer CEs with 19~160 logarithmic profile ommatidia via femtosecond laser two-photon polymerization. In contrast to μ-CEs with spherical ommatidia that suffer from defocusing problems, the as-obtained μ-CEs with logarithmic ommatidia permit direct integration with a commercial CMOS detector, because the depth-of-field and focus range of all the logarithmic ommatidia are significantly increased. The optoelectronic integrated μ-CE camera enables large field-of-view imaging (90°), spatial position identification and sensitive trajectory monitoring of moving targets. Moreover, the miniature μ-CE camera can be integrated with a microfluidic chip and serves as an on-chip camera for real-time microorganisms monitoring. The insect-scale optoelectronic μ-CE camera provides a practical route for integrating well-developed planar imaging sensors with complex micro-optics elements, holding great promise for cutting-edge applications in endoscopy and robot vision.

Design, fabrication, and performance evaluation of a concave lens array on an aspheric curved surface

A microlens array (MLA) is a fundamental optical element, which has been widely applied in the fields of imaging sensing, 3D display, and lighting source. However, it is still a challenge to design the MLAs simultaneously satisfying small size, wide field of view, and high image quality. Herein, a novel type of concave lens array on an aspheric convex substrate (CLAACs) is presented, which is composed of an aspheric substrate and a spherical concave subeye array. The facilely designed method of the CLAACs is described and its geometric model is also established by a numerical example. Furthermore, a fabrication method, which is directly machining the CLAACs on PMMA material, is proposed. To realize the ultra-precision machining of the lens, tool path planning is carried out before fabricating. The profile, surface quality, and imaging performance of the fabricated lens are then characterized to reveal its optical capabilities. The results show that the proposed method can realize the rapid design and fabrication of lenses flexibly and efficiently. The fabricated CLAACs exhibit excellent morphology uniformity, high imaging quality, and focusing performance. The study provides a feasible solution for the design and fabrication of such lens arrays with complex discontinuous surfaces.

A wide-field and high-resolution lensless compound eye microsystem for real-time target motion perception

Optical measurement systems suffer from a fundamental tradeoff between the field of view (FOV), the resolution and the update rate. A compound eye has the advantages of a wide FOV, high update rate and high sensitivity to motion, providing inspiration for breaking through the constraint and realizing high-performance optical systems. However, most existing studies on artificial compound eyes are limited by complex structure and low resolution, and they focus on imaging instead of precise measurement. Here, a high-performance lensless compound eye microsystem is developed to realize target motion perception through precise and fast orientation measurement. The microsystem splices multiple sub-FOVs formed by long-focal subeyes, images targets distributed in a panoramic range into a single multiplexing image sensor, and codes the subeye aperture array for distinguishing the targets from different sub-FOVs. A wide-field and high resolution are simultaneously realized in a simple and easy-to-manufacture microelectromechanical system (MEMS) aperture array. Moreover, based on the electronic rolling shutter technique of the image sensor, a hyperframe update rate is achieved by the precise measurement of multiple time-shifted spots of one target. The microsystem achieves an orientation measurement accuracy of 0.0023° (3σ) in the x direction and 0.0028° (3σ) in the y direction in a cone FOV of 120° with an update rate ~20 times higher than the frame rate. This study provides a promising approach for achieving optical measurements with comprehensive high performance and may have great significance in various applications, such as vision-controlled directional navigation and high-dynamic target tracking, formation and obstacle avoidance of unmanned aerial vehicles.

Insect-inspired AI for autonomous robots

Autonomous robots are expected to perform a wide range of sophisticated tasks in complex, unknown environments. However, available onboard computing capabilities and algorithms represent a considerable obstacle to reaching higher levels of autonomy, especially as robots get smaller and the end of Moore's law approaches. Here, we argue that inspiration from insect intelligence is a promising alternative to classic methods in robotics for the artificial intelligence (AI) needed for the autonomy of small, mobile robots. The advantage of insect intelligence stems from its resource efficiency (or parsimony) especially in terms of power and mass. First, we discuss the main aspects of insect intelligence underlying this parsimony: embodiment, sensory-motor coordination, and swarming. Then, we take stock of where insect-inspired AI stands as an alternative to other approaches to important robotic tasks such as navigation and identify open challenges on the road to its more widespread adoption. Last, we reflect on the types of processors that are suitable for implementing insect-inspired AI, from more traditional ones such as microcontrollers and field-programmable gate arrays to unconventional neuromorphic processors. We argue that even for neuromorphic processors, one should not simply apply existing AI algorithms but exploit insights from natural insect intelligence to get maximally efficient AI for robot autonomy.

Solar Concentrator Bio-Inspired by the Superposition Compound Eye for High-Concentration Photovoltaic System up to Thousands Fold Factor

We have proposed a fruitful design principle targeting a concentration ratio (CR) >1000× for a typical high concentrating photovoltaics (HCPV) system, on account of a two-concentrator system + homogenizer. The principle of a primary dual-lens concentrator unit, completely analogous basic optics seen in the superposition compound eyes, is a trend not hitherto reported for solar concentrators to our knowledge. Such a concentrator unit, consisting of two aspherical lenses, can be applied to minify the sunlight and reveal useful effects. We underline that, at this stage, the CR can be attained by two orders of magnitude simply by varying the radius ratio of such two lenses known from the optics side. The output beam is spatially minimized and nearly parallel, exactly as occurs in the superposition compound eye. In our scheme, thanks to such an array of dual-lens design, a sequence of equidistant focal points is formed. The secondary concentrator consists of a multi-reflective channel, which can collect all concentrated beams from the primary concentrator to a small area where a solar cell is placed. The secondary concentrator is located right underneath the primary concentrator. The optical characteristics are substantiated by optical simulations that confirm the applicability of thousands-fold gain in CR value, ~1100×. This, however, also reduced the uniformity of the illumination area. To regain the uniformity, we devise a fully new homogenizer, hinging on the scattering principle. A calculated optical efficiency for the entire system is ~75%. Experimentally, a prototype of such a dual-lens concentrator is implemented to evaluate the converging features. As a final note, we mention that the approach may be extended to implement an even higher CR, be it simply by taking an extra concentrator unit. With simple design of the concentrator part, which may allow the fabrication process by modeling method and large acceptant angle (0.6°), we assess its large potential as part of a general strategy to implement a highly efficient CPV system, with minimal critical elaboration steps and large flexibility.

Contact-Resistance-Free Stretchable Strain Sensors with High Repeatability and Linearity

Most of the existing stretchable strain sensors are based on the contact-resistance mechanism, where the stretchability and resistance variation depend on the change of the contact relationship of the conductive microstructures. These sensors usually exhibit large sensing ranges and gauge factors but unsatisfactory repeatability and linearity of the electrical responses because the contact is unstable. Here, we report a completely different design for stretchable strain sensors based on a contact-resistance-free structure, i.e., the off-axis serpentine sandwich structure (OASSS), with the mechanism of the stretch-bending-stretch transformation (SBST). Neither unstable contact resistance nor nonlinear constitutive and geometric behaviors occur for the OASSS while the sensor undergoes a large applied strain (50%), which guarantees high repeatability (repeatability error = 1.58%) and linearity (goodness-of-fit >0.999). Owing to such performances, the present sensors are not only applied to monitoring human activities and medical surgery but also to the ground tests of Tianwen-1, China's first Mars exploration mission.

Building 2D Model of Compound Eye Vision for Machine Learning

This paper presents a two-dimensional mathematical model of compound eye vision. Such a model is useful for solving navigation issues for autonomous mobile robots on the ground plane. The model is inspired by the insect compound eye that consists of ommatidia, which are tiny independent photoreception units, each of which combines a cornea, lens, and rhabdom. The model describes the planar binocular compound eye vision, focusing on measuring distance and azimuth to a circular feature with an arbitrary size. The model provides a necessary and sufficient condition for the visibility of a circular feature by each ommatidium. On this basis, an algorithm is built for generating a training data set to create two deep neural networks (DNN): the first detects the distance, and the second detects the azimuth to a circular feature. The hyperparameter tuning and the configurations of both networks are described. Experimental results showed that the proposed method could effectively and accurately detect the distance and azimuth to objects.

Digital optofluidic compound eyes with natural structures and zooming capability for large-area fluorescence sensing

Compound eyes are ubiquitous natural biosensors that possess high temporal resolution and large fields of view (FOVs). While for solid materials based artificial imaging systems, flexible zooming ability while keeping the constant FOV is still challenging, as well as the low-cost fabrication. Herein, liquid compound eyes with natural structures are presented that synthesize optofluidics and bionics in a non-trivial manner, which enables the deformation-free zooming and flexible cell fluorescence sensing. Experimental results indicate that the innovatively manufactured bionic template possesses low roughness and uniform lens configuration with more than two thousands units, which endows the eyes with high-quality and low aberration imaging ability. Besides, digital controlled miscible liquids switching enables the focus of ommatidia simultaneously be adjusted from 150 μm to 5 mm with 100° view angle, and without bending the microlens curvature, to avoid FOV changing and image aberration. Due to large FOV and tunable ability, large-area cell fluorescence signal arrays and dynamic cell motion are imaged using this liquid compound eyes. This work presents novel strategy for compound lens manufacture at low-cost, and proposes deformation-free and continuous focus-tuning strategy, offering potentials for numerous applications, including biomedical sensing and adaptive imaging with large FOV.

Ultralow Power Optical Synapses Based on MoS2 Layers by Indium-Induced Surface Charge Doping for Biomimetic Eyes

Biomimetic eyes, with their excellent imaging functions such as large fields of view and low aberrations, have shown great potentials in the fields of visual prostheses and robotics. However, high power consumption and difficulties in device integration severely restrict their rapid development. In this study, an artificial synaptic device consisting of a molybdenum disulfide (MoS₂ ) film coated with an electron injection enhanced indium (In) layer is proposed to increase the channel conductivity and reduce the power consumption. This artificial synaptic device achieves an ultralow power consumption of 68.9 aJ per spike, which is several hundred times lower than those of the optical artificial synapses reported in literature. Furthermore, the multilayer and polycrystalline MoS₂ film shows persistent photoconductivity performance, effectively resulting in short-term plasticity, long-term plasticity, and their transitions between each other. A 5 × 5 In/MoS₂ synaptic device array is constructed into a hemispherical electronic retina, demonstrating its impressive image sensing and learning functions. This research provides a new methodology for effective control of artificial synaptic devices, which have great opportunities used in bionic retinas, robots, and visual prostheses.

Biomimetic apposition compound eye fabricated using microfluidic-assisted 3D printing

After half a billion years of evolution, arthropods have developed sophisticated compound eyes with extraordinary visual capabilities that have inspired the development of artificial compound eyes. However, the limited 2D nature of most traditional fabrication techniques makes it challenging to directly replicate these natural systems. Here, we present a biomimetic apposition compound eye fabricated using a microfluidic-assisted 3D-printing technique. Each microlens is connected to the bottom planar surface of the eye via intracorporal, zero-crosstalk refractive-index-matched waveguides to mimic the rhabdoms of a natural eye. Full-colour wide-angle panoramic views and position tracking of a point source are realized by placing the fabricated eye directly on top of a commercial imaging sensor. As a biomimetic analogue to naturally occurring compound eyes, the eye's full-colour 3D to 2D mapping capability has the potential to enable a wide variety of applications from improving endoscopic imaging to enhancing machine vision for facilitating human-robot interactions.

Low-cost, thermoplastic micro-lens array with a carbon black light screen for bio-mimetic vision

Micro-lens array is a great example of bio-mimetic technology which was inspired by compound eyes found in insects and is used in lasers, optical communication, and 3D imaging. In this study, a micro-lens array was fabricated from cyclic olefin copolymer using a cost-effective method: compression molding and thermal reflow. Also, a light screen was installed between lenses to reduce the optical interference for clearer individual images. Cyclic olefin copolymer-based micro-lens array showed good optical results under a standard optical microscope. By placing the fabricated micro-lens array directly on an image sensor, it was observed that the light screen shows significant improvement in image quality. Also, the point spread function was analyzed to confirm the optical performance and the effectiveness of the micro-lens array with the light screen installed.

Biomimetic multispectral curved compound eye camera for real-time multispectral imaging in an ultra-large field of view

In this work, we demonstrate a prototype of a biomimetic multispectral curved compound eye camera (BMCCEC). In comparison with traditional multispectral imaging systems, the BMCCEC developed in this work has the distinct features of multi-spectral imaging on multiple targets in real time in an ultra-large field of view (FOV), which can be attributed to its biomimetic curved compound eye structure as well as the multispectral cluster network. Specifically, the BMCCEC has a total of 104 multispectral ommatidia and a FOV of 98°×98°, which is able to realize 7-band multispectral imaging with center wavelengths of 500 nm, 560 nm, 600 nm, 650 nm, 700 nm, 750 nm and 800 nm and a spectral resolution of 10 nm. A prototype of BMCCEC was then manufactured and multispectral imaging experiments were performed based on it. As a result, the red edge feature of the spectrum of green plants has been successfully obtained and retrieved with a good accuracy.

Multidimensional Tactile Sensor with a Thin Compound Eye-Inspired Imaging System

Artificial tactile sensing for robots is a counterpart to the human sense of touch, serving as a feedback interface for sensing and interacting with the environment. A vision-based tactile sensor has emerged as a novel and advantageous branch of artificial tactile sensors. Compared with conventional tactile sensors, vision-based tactile sensors possess stronger potential thanks to acquiring multimodal contact information in much higher spatial resolution, although they typically suffer from bulky size and fabrication challenges. In this article, we report a thin vision-based tactile sensor that draws inspiration from natural compound eye structures and demonstrate its capability of sensing three-dimensional (3D) force. The sensor is composed of an array of vision units, an elastic touching interface, and a supporting structure with illumination. Experiments validated the sensor's advantages, including competitive spatial resolution of deformation as high as 1016 dpi on a 5 × 8 mm² sensing area, superior accuracy of 3D force measurement at levels of 0.018 N for tangential force and 0.213 N (0.108 N at the center region) for normal force, and real-time processing at 30 Hz, while achieving a thin size of 5 mm. We further demonstrate the sensor capability in sensing 3D force and slip occurrence in real grasping experiments. This device paves the way for robotic applications that require rich tactile information with miniaturized sensor structure.

Artificial Compound Eye Systems and Their Application: A Review

The natural compound eye system has many outstanding properties, such as a more compact size, wider-angle view, better capacity to detect moving objects, and higher sensitivity to light intensity, compared to that of a single-aperture vision system. Thanks to the development of micro- and nano-fabrication techniques, many artificial compound eye imaging systems have been studied and fabricated to inherit fascinating optical features of the natural compound eye. This paper provides a review of artificial compound eye imaging systems. This review begins by introducing the principle of the natural compound eye, and then, the analysis of two types of artificial compound eye systems. We equally present the applications of the artificial compound eye imaging systems. Finally, we suggest our outlooks about the artificial compound eye imaging system.

Physical reservoir computing with origami and its application to robotic crawling

A new paradigm called physical reservoir computing has recently emerged, where the nonlinear dynamics of high-dimensional and fixed physical systems are harnessed as a computational resource to achieve complex tasks. Via extensive simulations based on a dynamic truss-frame model, this study shows that an origami structure can perform as a dynamic reservoir with sufficient computing power to emulate high-order nonlinear systems, generate stable limit cycles, and modulate outputs according to dynamic inputs. This study also uncovers the linkages between the origami reservoir's physical designs and its computing power, offering a guideline to optimize the computing performance. Comprehensive parametric studies show that selecting optimal feedback crease distribution and fine-tuning the underlying origami folding designs are the most effective approach to improve computing performance. Furthermore, this study shows how origami's physical reservoir computing power can apply to soft robotic control problems by a case study of earthworm-like peristaltic crawling without traditional controllers. These results can pave the way for origami-based robots with embodied mechanical intelligence.

Microlenses arrays: Fabrication, materials, and applications

Microlenses have become an indispensable optical element in many optical systems. The advancement of technology has led to a wider variety of microlenses fabrication methods, but these methods suffer from, more or less, some limitations. In this article, we review the manufacturing technology of microlenses from the direct and indirect perspectives. First, we present several fabrication methods and their advantages and disadvantages are discussed. Then, we discuss the commonly used materials for fabricating microlenses and the applications of microlenses in various fields. Finally, we point out the prospects for the future development of microlenses and their fabrication methods.

Biomimetic models of the human eye, and their applications

Replicating the functionality of the human eye has been a challenge for more than a century, creating a great wealth of biomimetic and bioinspired devices, and providing ever improving models of the eye for myriad research purposes. As improvements in microelectronics have proceeded, individual components of the eye have been replicated, and models of the optical behaviour of the eye have improved. This review explores both work developed for improving medical components, with an ultimate aim of a fully functioning prosthetic eye, and work looking at improving existing devices through biomimetic means. It is hoped that this holistic approach to the subject will aid in the cross pollination of ideas between the two research foci. The review starts by summarising the reported measurements of optical parameters of various components of the eye. It then charts the development of individual bionic components. Particular focus is put on the development of bionic and biomimetic forms of the two main adaptive components of the eye, namely the lens and the iris, and the challenges faced in modelling the light sensitive retina. Work on each of these components is thoroughly reviewed, including an overview of the principles behind the many different approaches used to mimic the functionality, and discussion of the pros and cons of each approach. This is concluded by an overview of several reported models of the complete or semi-complete eye, including details of the components used and a summary of the models' functionality. Finally, some consideration is given to the direction of travel of this field of research, and which existing approaches are likely to bring us closer to the long term goal of a fully functional analogue of the eye.

Locomotion of Miniature Soft Robots

Miniature soft robots are mobile devices, which are made of smart materials that can be actuated by external stimuli to realize their desired functionalities. Here, the key advancements and challenges of the locomotion producible by miniature soft robots in micro- to centimeter length scales are highlighted. It is highly desirable to endow these small machines with dexterous locomotive gaits as it enables them to easily access highly confined and enclosed spaces via a noninvasive manner. If miniature soft robots are able to capitalize this unique ability, they will have the potential to transform a vast range of applications, including but not limited to, minimally invasive medical treatments, lab-on-chip applications, and search-and-rescue missions. The gaits of miniature soft robots are categorized into terrestrial, aquatic, and aerial locomotion. Except for the centimeter-scale robots that can perform aerial locomotion, the discussions in this report are centered around soft robots that are in the micro- to millimeter length scales. Under each category of locomotion, prospective methods and strategies that can improve their gait performances are also discussed. This report provides critical analyses and discussions that can inspire future strategies to make miniature soft robots significantly more agile.

Design and Integration of the Single-Lens Curved Multi-Focusing Compound Eye Camera

Compared with a traditional optical system, the single-lens curved compound eye imaging system has superior optical performance, such as a large field of view (FOV), small size, and high portability. However, defocus and low resolution hinder the further development of single-lens curved compound eye imaging systems. In this study, the design of a nonuniform curved compound eye with multiple focal lengths was used to solve the defocus problem. A two-step gas-assisted process, which was combined with photolithography, soft photolithography, and ultraviolet curing, was proposed for fabricating the ommatidia with a large numerical aperture precisely. Ommatidia with high resolution were fabricated and arranged in five rings. Based on the imaging experimental results, it was demonstrated that the high-resolution and small-volume single-lens curved compound eye imaging system has significant advantages in large-field imaging and rapid recognition.

Bio-inspired multimodal 3D endoscope for image-guided and robotic surgery

Image-guided and robotic surgery based on endoscopic imaging technologies can enhance cancer treatment by ideally removing all cancerous tissue and avoiding iatrogenic damage to healthy tissue. Surgeons evaluate the tumor margins at the cost of impeding surgical workflow or working with dimmed surgical illumination, since current endoscopic imaging systems cannot simultaneous and real-time color and near-infrared (NIR) fluorescence imaging under normal surgical illumination. To overcome this problem, a bio-inspired multimodal 3D endoscope combining the excellent characteristics of human eyes and compound eyes of mantis shrimp is proposed. This 3D endoscope, which achieves simultaneous and real-time imaging of three-dimensional stereoscopic, color, and NIR fluorescence, consists of three parts: a broad-band binocular optical system like as human eye, an optical relay system, and a multiband sensor inspired by the mantis shrimp's compound eye. By introducing an optical relay system, the two sub-images after the broad-band binocular optical system can be projected onto one and the same multiband sensor. A series of experiments demonstrate that this bio-inspired multimodal 3D endoscope not only provides surgeons with real-time feedback on the location of tumor tissue and lymph nodes but also creates an immersive experience for surgeons without impeding surgical workflow. Its excellent characteristics and good scalability can promote the further development and application of image-guided and robotic surgery.

Study of asymmetric or decentered multi-view designs for uncooled infrared imaging applications

Multi-view architectures using lens arrays can bring interesting features like 3D or multispectral imagery over single aperture cameras. Combined with super-resolution algorithms, multi-view designs are a way to miniaturize cameras while maintaining their resolution. These optical designs can be adapted for thermal infrared imagery and can thus answer the size, weight and power (SWAP) challenge with advanced imagery functions. However, in this spectral range, the choice of an uncooled microbolometer detector imposes a high numerical aperture for the system which increases the size of the optics and makes difficult a multi-channel arrangement combined with a single focal plane array (FPA). In this paper, we theoretically investigate several asymmetric or decentered multi-view designs that allow both a high aperture for the optical channels and the use of a single FPA for the sub-images. Ray-traced designs will illustrate this study and their image quality will be checked with modulation transfer functions (MTF) for different field points.

Fractal Web Design of a Hemispherical Photodetector Array with Organic-Dye-Sensitized Graphene Hybrid Composites

The vision system of arthropods consists of a dense array of individual photodetecting elements across a curvilinear surface. This compound-eye architecture could be a useful model for optoelectronic sensing devices that require a large field of view and high sensitivity to motion. Strategies that aim to mimic the compound-eye architecture involve integrating photodetector pixels with a curved microlens, but their fabrication on a curvilinear surface is challenged by the use of standard microfabrication processes that are traditionally designed for planar, rigid substrates (e.g., Si wafers). Here, a fractal web design of a hemispherical photodetector array that contains an organic-dye-sensitized graphene hybrid composite is reported to serve as an effective photoactive component with enhanced light-absorbing capabilities. The device is first fabricated on a planar Si wafer at the microscale and then transferred to transparent hemispherical domes with different curvatures in a deterministic manner. The unique structural property of the fractal web design provides protection of the device from damage by effectively tolerating various external loads. Comprehensive experimental and computational studies reveal the essential design features and optoelectronic properties of the device, followed by the evaluation of its utility in the measurement of both the direction and intensity of incident light.

Fabrication and Characterization of Curved Compound Eyes Based on Multifocal Microlenses

Curved compound eyes have generated great interest owing to the wide field of view but the application of devices is hindered for the lack of proper detectors. One-lens curved compound eyes with multi-focal microlenses provide a solution for wide field imaging integrated in a commercial photo-detector. However, it is still a challenge for manufacturing this kind of compound eye. In this paper, a rapid and accurate method is proposed by a combination of photolithography, hot embossing, soft photolithography, and gas-assisted deformation techniques. Microlens arrays with different focal lengths were firstly obtained on a polymer, and then the planar structure was converted to the curved surface. A total of 581 compound eyes with diameters ranging from 152.8 µm to 240.9 µm were successfully obtained on one curved surface within a few hours, and the field of view of the compound eyes exceeded 108°. To verify the characteristics of the fabricated compound eyes, morphology deviation was measured by a probe profile and a scanning electron microscope. The optical performance and imaging capability were also tested and analyzed. As a result, the ommatidia made up of microlenses showed not only high accuracy in morphology, but also imaging uniformity on a focal plane. This flexible massive fabrication of compound eyes indicates great potential for miniaturized imaging systems.

Target orientation detection based on a neural network with a bionic bee-like compound eye

The compound eye of insects has many excellent characteristics. Directional navigation is one of the important features of compound eye, which is able to quickly and accurately determine the orientation of an objects. Therefore, bionic curved compound eye have great potential in detecting the orientation of the target. However, there is a serious non-linear relationship between the orientation of the target and the image obtained by the curved compound eye in wide field of view (FOV), and an effective model has not been established to detect the orientation of target. In this paper, a method for detecting the orientation of the target is proposed, which combines a virtual cylinder target with a neural network. To verify the feasibility of the method, a fiber-optic compound eye that is inspired by the structure of the bee's compound eye and that fully utilizes the transmission characteristics and flexibility of optical fibers is developed. A verification experiment shows that the proposed method is able to realize quantitative detection of orientations using a prototype of the fiber-optic compound eye. The average errors between the ground truth and the predicted values of the horizontal and elevation angles of a target are 0.5951 ^° and 0.6748^°, respectively. This approach has great potential for target tracking, obstacle avoidance by unmanned aerial vehicles, and directional navigation control.

Plasmonic ommatidia for lensless compound-eye vision

The vision system of arthropods such as insects and crustaceans is based on the compound-eye architecture, consisting of a dense array of individual imaging elements (ommatidia) pointing along different directions. This arrangement is particularly attractive for imaging applications requiring extreme size miniaturization, wide-angle fields of view, and high sensitivity to motion. However, the implementation of cameras directly mimicking the eyes of common arthropods is complicated by their curved geometry. Here, we describe a lensless planar architecture, where each pixel of a standard image-sensor array is coated with an ensemble of metallic plasmonic nanostructures that only transmits light incident along a small geometrically-tunable distribution of angles. A set of near-infrared devices providing directional photodetection peaked at different angles is designed, fabricated, and tested. Computational imaging techniques are then employed to demonstrate the ability of these devices to reconstruct high-quality images of relatively complex objects.

Biologically inspired artificial eyes and photonics

Natural visual systems have inspired scientists and engineers to mimic their intriguing features for the development of advanced photonic devices that can provide better solutions than conventional ones. Among various kinds of natural eyes, researchers have had intensive interest in mammal eyes and compound eyes due to their advantages in optical properties such as focal length tunability, high-resolution imaging, light intensity modulation, wide field of view, high light sensitivity, and efficient light management. A variety of different approaches in the broad field of science and technology have been tried and succeeded to duplicate the functions of natural eyes and develop bioinspired photonic devices for various applications. In this review, we present a comprehensive overview of bioinspired artificial eyes and photonic devices that mimic functions of natural eyes. After we briefly introduce visual systems in nature, we discuss optical components inspired by the mammal eyes, including tunable lenses actuated with different mechanisms, curved image sensors with low aberration, and light intensity modulators. Next, compound eye inspired photonic devices are presented, such as microlenses and micromirror arrays, imaging sensor arrays on curved surfaces, self-written waveguides with microlens arrays, and antireflective nanostructures (ARS). Subsequently, compound eyes with focal length tunability, photosensitivity enhancers, and polarization imaging sensors are described.

Algebraic approach towards the exploitation of “softness”: the input–output equation for morphological computation

Recently, soft robots that consist of soft and deformable materials have received much attention for their adaptability to uncertain environments. Although these robots are difficult to control with a conventional control theory owing to their complex body dynamics, research from different perspectives attempts to actively exploit these body dynamics as an asset rather than a drawback. This approach is called morphological computation, in which the soft materials are used for computation that includes a new kind of control strategy. In this article, we propose a novel approach to analyze the computational properties of soft materials based on an algebraic method, called the input–output equation used in systems analysis, particularly in systems biology. We mainly focus on the two scenarios relevant to soft robotics, that is, analysis of the computational capabilities of soft materials and design of the input force to soft devices to generate the target behaviors. The input–output equation directly describes the relationship between inputs and outputs of a system, and hence by using this equation, important properties, such as the echo state property that guarantees reproducible responses against the same input stream, can be investigated for soft structures. Several application scenarios of our proposed method are demonstrated using typical soft robotic settings in detail, including linear/nonlinear models and hydrogels driven by chemical reactions.