Active Vision and Surface Reconstruction for 3D Plant Shoot Modelling

Multi-View Fusion-Based Automated Full-Posture Cattle Body Size Measurement

Cattle farming is an important part of the global livestock industry, and cattle body size is the key indicator of livestock growth. However, traditional manual methods for measuring body sizes are not only time-consuming and labor-intensive but also incur significant costs. Meanwhile, automatic measurement techniques are prone to being affected by environmental conditions and the standing postures of livestock. To overcome these challenges, this study proposes a multi-view fusion-driven automatic measurement system for full-attitude cattle body measurements. Outdoors in natural light, three Zed2 cameras were installed covering different views of the channel. Multiple images, including RGB images, depth images, and point clouds, were automatically acquired from multiple views using the YOLOv8n algorithm. The point clouds from different views undergo multiple denoising to become local point clouds of the cattle body. The local point clouds are coarsely and finely aligned to become a complete point cloud of the cattle body. After detecting the 2D key points on the RGB image created by the YOLOv8x-pose algorithm, the 2D key points are mapped onto the 3D cattle body by combining the internal parameters of the camera and the depth values of the corresponding pixels of the depth map. Based on the mapped 3D key points, the body sizes of cows in different poses are automatically measured, including height, length, abdominal circumference, and chest circumference. In addition, support vector machines and Bézier curves are employed to rectify the missing and deformed circumference body sizes caused by environmental effects. The automatic body measurement system measured the height, length, abdominal circumference, and chest circumference of 47 Huaxi Beef Cattle, a breed native to China, and compared the results with manual measurements. The average relative errors were 2.32%, 2.27%, 3.67%, and 5.22%, respectively, when compared with manual measurements, demonstrating the feasibility and accuracy of the system.

An unmanned ground vehicle phenotyping-based method to generate three-dimensional multispectral point clouds for deciphering spatial heterogeneity in plant traits

Fusing three-dimensional (3D) and multispectral (MS) imaging data holds promise for high-throughput and comprehensive plant phenotyping to decipher genome-to-phenome knowledge. Acquiring high-quality 3D MS point clouds (3DMPCs) of plants remains challenging because of poor 3D data quality and limited radiometric calibration methods for plants with a complex canopy structure. Here, we present a novel 3D spatial-spectral data fusion approach to collect high-quality 3DMPCs of plants by integrating the next-best-view planning for adaptive data acquisition and neural reference field (NeREF) for radiometric calibration. This approach was used to acquire 3DMPCs of perilla, tomato, and rapeseed plants with diverse plant architecture and leaf morphological features evaluated by the accuracy of chlorophyll content and equivalent water thickness (EWT) estimation. The results showed that the completeness of plant point clouds collected by this approach was improved by an average of 23.6% compared with the fixed viewpoints alone. The NeREF-based radiometric calibration with the hemispherical reference outperformed the conventional calibration method by reducing the root mean square error (RMSE) of 58.93% for extracted reflectance spectra. The RMSE for chlorophyll content and EWT predictions decreased by 21.25% and 14.13% using partial least squares regression with the generated 3DMPCs. Collectively, our study provides an effective and efficient way to collect high-quality 3DMPCs of plants under natural light conditions, which improves the accuracy and comprehensiveness of phenotyping plant morphological and physiological traits, and thus will facilitate plant biology and genetic studies as well as crop breeding.

A New Camera Calibration Technique for Serious Distortion

A new camera calibration technique based on serious distortion is proposed, which only requires the camera to observe the plane pattern in an arbitrary azimuth. It uses the geometrical imaging principle and radial distortion model to acquire radial lens distortion coefficient and the image coordinate (u0, v0), and then solves the linear equation aiming at the other parameters of the camera. This method has the following characteristics: Firstly, the position of the camera and the plane is arbitrary, and the technique needs only a single observation for plane pattern. Secondly, it is suitable for camera calibration with serious distortion. Thirdly, it does not need expensive ancillary equipment, accurate movement, or lots of photos observed from different orientations. Having been authenticated by computer emulation and actual experiment, the results of the proposed technique have proved to be satisfactory. The research has also paved a new way in camera calibration for further studies.

Segmentation of structural parts of rosebush plants with 3D point-based deep learning methods

BACKGROUND

Segmentation of structural parts of 3D models of plants is an important step for plant phenotyping, especially for monitoring architectural and morphological traits. Current state-of-the art approaches rely on hand-crafted 3D local features for modeling geometric variations in plant structures. While recent advancements in deep learning on point clouds have the potential of extracting relevant local and global characteristics, the scarcity of labeled 3D plant data impedes the exploration of this potential.

RESULTS

We adapted six recent point-based deep learning architectures (PointNet, PointNet++, DGCNN, PointCNN, ShellNet, RIConv) for segmentation of structural parts of rosebush models. We generated 3D synthetic rosebush models to provide adequate amount of labeled data for modification and pre-training of these architectures. To evaluate their performance on real rosebush plants, we used the ROSE-X data set of fully annotated point cloud models. We provided experiments with and without the incorporation of synthetic data to demonstrate the potential of point-based deep learning techniques even with limited labeled data of real plants.

CONCLUSION

The experimental results show that PointNet++ produces the highest segmentation accuracy among the six point-based deep learning methods. The advantage of PointNet++ is that it provides a flexibility in the scales of the hierarchical organization of the point cloud data. Pre-training with synthetic 3D models boosted the performance of all architectures, except for PointNet.

3D Reconstruction of Non-Rigid Plants and Sensor Data Fusion for Agriculture Phenotyping

Technology has been promoting a great transformation in farming. The introduction of robotics; the use of sensors in the field; and the advances in computer vision; allow new systems to be developed to assist processes, such as phenotyping, of crop’s life cycle monitoring. This work presents, which we believe to be the first time, a system capable of generating 3D models of non-rigid corn plants, which can be used as a tool in the phenotyping process. The system is composed by two modules: an terrestrial acquisition module and a processing module. The terrestrial acquisition module is composed by a robot, equipped with an RGB-D camera and three sets of temperature, humidity, and luminosity sensors, that collects data in the field. The processing module conducts the non-rigid 3D plants reconstruction and merges the sensor data into these models. The work presented here also shows a novel technique for background removal in depth images, as well as efficient techniques for processing these images and the sensor data. Experiments have shown that from the models generated and the data collected, plant structural measurements can be performed accurately and the plant’s environment can be mapped, allowing the plant’s health to be evaluated and providing greater crop efficiency.

Surface reconstruction based on CAD model driven priori templates

In this paper, a method of reconstructing 3D (three dimensional) models from the original scanned point cloud using priori templates is proposed. Different from previous reconstruction methods that triangulate and fit the original scanning point cloud directly, we construct a priori template based on the CAD (computer aided design) model and guide the reconstruction of the original scanning point cloud with the priori template. Given a CAD model, the basic geometric elements are used as the basic units to extract the set elements of 3D shapes. Then the geometric elements are meshed, and the normal vectors at the mesh nodes are extracted. The corresponding point cloud data of each basic element are extracted from the original point cloud. The point cloud data near the normal of the guide point are searched, and the Gaussian weighted average value of the searched point represents the actual geometric parameters of the part at the guide point. Finally, the geometric elements of the basic unit are reconstructed locally by Non-Uniform Rational B-Splines surface fitting, and the complete reconstruction model is obtained by integrating the local reconstruction. Experiments show that our method can solve the problems of high quality reconstruction, sharp feature preservation, and detail recovery in surface reconstruction.