An Accurate Skeleton Extraction Approach From 3D Point Clouds of Maize Plants

Modern phenomics to empower holistic crop science, agronomy, and breeding research

Crop phenomics enables collection of diverse plant traits for a large number of samples along different time scales, representing a greater data collection throughput compared to the traditional measurements. Most of modern crop phenomics use different sensors to collect reflective, emitted and fluorescence signals etc., from plant organs at different spatial and temporal resolutions. Such multi-modal, high dimensional data not only accelerates basic research on crop physiology, genetics, and whole plant systems modeling, but also supports the optimization of field agronomic practices, internal environments of plant factories, and ultimately crop breeding. Major challenges and opportunities facing the current crop phenomics research community include developing community consensus or standards for data collection, management, sharing, and processing, developing capabilities to measure physiological parameters, and enabling farmers and breeders to effectively use phenomics in the field to directly support agricultural production.

The Importance of Using Realistic 3D Canopy Models to Calculate Light Interception in the Field

Quantifying canopy light interception provides insight into the effects of plant spacing, canopy structure, and leaf orientation on radiation distribution. This is essential for increasing crop yield and improving product quality. Canopy light interception can be quantified using 3-dimensional (3D) plant models and optical simulations. However, virtual 3D canopy models (VCMs) have often been used to quantify canopy light interception because realistic 3D canopy models (RCMs) are difficult to obtain in the field. This study aims to compare the differences in light interception between VCMs and RCM. A realistic 3D maize canopy model (RCM) was reconstructed over a large area of the field using an advanced unmanned aerial vehicle cross-circling oblique (CCO) route and the structure from motion-multi-view stereo method. Three types of VCMs (VCM-1, VCM-4, and VCM-8) were then created by replicating 1, 4, and 8 individual realistic plants constructed by CCO in the center of the corresponding RCM. The daily light interception per unit area (DLI), as computed for the 3 VCMs, exhibited marked deviation from the RCM, as evinced by the relative root mean square error (rRMSE) values of 20.22%, 17.38%, and 15.48%, respectively. Although this difference decreased as the number of plants used to replicate the virtual canopy increased, rRMSE of DLI for VCM-8 and RCM still reached 15.48%. It was also found that the difference in light interception between RCMs and VCMs was substantially smaller in the early stage (48 days after sowing [DAS]) than in the late stage (70 DAS). This study highlights the importance of using RCM when calculating light interception in the field, especially in the later growth stages of plants.

A fast phenotype approach of 3D point clouds of Pinus massoniana seedlings

The phenotyping of Pinus massoniana seedlings is essential for breeding, vegetation protection, resource investigation, and so on. Few reports regarding estimating phenotypic parameters accurately in the seeding stage of Pinus massoniana plants using 3D point clouds exist. In this study, seedlings with heights of approximately 15-30 cm were taken as the research object, and an improved approach was proposed to automatically calculate five key parameters. The key procedure of our proposed method includes point cloud preprocessing, stem and leaf segmentation, and morphological trait extraction steps. In the skeletonization step, the cloud points were sliced in vertical and horizontal directions, gray value clustering was performed, the centroid of the slice was regarded as the skeleton point, and the alternative skeleton point of the main stem was determined by the DAG single source shortest path algorithm. Then, the skeleton points of the canopy in the alternative skeleton point were removed, and the skeleton point of the main stem was obtained. Last, the main stem skeleton point after linear interpolation was restored, while stem and leaf segmentation was achieved. Because of the leaf morphological characteristics of Pinus massoniana, its leaves are large and dense. Even using a high-precision industrial digital readout, it is impossible to obtain a 3D model of Pinus massoniana leaves. In this study, an improved algorithm based on density and projection is proposed to estimate the relevant parameters of Pinus massoniana leaves. Finally, five important phenotypic parameters, namely plant height, stem diameter, main stem length, regional leaf length, and total leaf number, are obtained from the skeleton and the point cloud after separation and reconstruction. The experimental results showed that there was a high correlation between the actual value from manual measurement and the predicted value from the algorithm output. The accuracies of the main stem diameter, main stem length, and leaf length were 93.5%, 95.7%, and 83.8%, respectively, which meet the requirements of real applications.

How to make sense of 3D representations for plant phenotyping: a compendium of processing and analysis techniques

Computer vision technology is moving more and more towards a three-dimensional approach, and plant phenotyping is following this trend. However, despite its potential, the complexity of the analysis of 3D representations has been the main bottleneck hindering the wider deployment of 3D plant phenotyping. In this review we provide an overview of typical steps for the processing and analysis of 3D representations of plants, to offer potential users of 3D phenotyping a first gateway into its application, and to stimulate its further development. We focus on plant phenotyping applications where the goal is to measure characteristics of single plants or crop canopies on a small scale in research settings, as opposed to large scale crop monitoring in the field.

Phenotyping of Silique Morphology in Oilseed Rape Using Skeletonization with Hierarchical Segmentation

Silique morphology is an important trait that determines the yield output of oilseed rape (Brassica napus L.). Segmenting siliques and quantifying traits are challenging because of the complicated structure of an oilseed rape plant at the reproductive stage. This study aims to develop an accurate method in which a skeletonization algorithm was combined with the hierarchical segmentation (SHS) algorithm to separate siliques from the whole plant using 3-dimensional (3D) point clouds. We combined the L1-median skeleton with the random sample consensus for iteratively extracting skeleton points and optimized the skeleton based on information such as distance, angle, and direction from neighborhood points. Density-based spatial clustering of applications with noise and weighted unidirectional graph were used to achieve hierarchical segmentation of siliques. Using the SHS, we quantified the silique number (SN), silique length (SL), and silique volume (SV) automatically based on the geometric rules. The proposed method was tested with the oilseed rape plants at the mature stage grown in a greenhouse and field. We found that our method showed good performance in silique segmentation and phenotypic extraction with R ² values of 0.922 and 0.934 for SN and total SL, respectively. Additionally, SN, total SL, and total SV had the statistical significance of correlations with the yield of a plant, with R values of 0.935, 0.916, and 0.897, respectively. Overall, the SHS algorithm is accurate, efficient, and robust for the segmentation of siliques and extraction of silique morphological parameters, which is promising for high-throughput silique phenotyping in oilseed rape breeding.

DFSP: A fast and automatic distance field-based stem-leaf segmentation pipeline for point cloud of maize shoot

The 3D point cloud data are used to analyze plant morphological structure. Organ segmentation of a single plant can be directly used to determine the accuracy and reliability of organ-level phenotypic estimation in a point-cloud study. However, it is difficult to achieve a high-precision, automatic, and fast plant point cloud segmentation. Besides, a few methods can easily integrate the global structural features and local morphological features of point clouds relatively at a reduced cost. In this paper, a distance field-based segmentation pipeline (DFSP) which could code the global spatial structure and local connection of a plant was developed to realize rapid organ location and segmentation. The terminal point clouds of different plant organs were first extracted via DFSP during the stem-leaf segmentation, followed by the identification of the low-end point cloud of maize stem based on the local geometric features. The regional growth was then combined to obtain a stem point cloud. Finally, the instance segmentation of the leaf point cloud was realized using DFSP. The segmentation method was tested on 420 maize and compared with the manually obtained ground truth. Notably, DFSP had an average processing time of 1.52 s for about 15,000 points of maize plant data. The mean precision, recall, and micro F1 score of the DFSP segmentation algorithm were 0.905, 0.899, and 0.902, respectively. These findings suggest that DFSP can accurately, rapidly, and automatically achieve maize stem-leaf segmentation tasks and could be effective in maize phenotype research. The source code can be found at https://github.com/syau-miao/DFSP.git.

Geometric Wheat Modeling and Quantitative Plant Architecture Analysis Using Three-Dimensional Phytomers

The characterization, analysis, and evaluation of morphology and structure are crucial in wheat research. Quantitative and fine characterization of wheat morphology and structure from a three-dimensional (3D) perspective has great theoretical significance and application value in plant architecture identification, high light efficiency breeding, and cultivation. This study proposes a geometric modeling method of wheat plants based on the 3D phytomer concept. Specifically, 3D plant architecture parameters at the organ, phytomer, single stem, and individual plant scales were extracted based on the geometric models. Furthermore, plant architecture vector (PA) was proposed to comprehensively evaluate wheat plant architecture, including convergence index (C), leaf structure index (L), phytomer structure index (PHY), and stem structure index (S). The proposed method could quickly and efficiently achieve 3D wheat plant modeling by assembling 3D phytomers. In addition, the extracted PA quantifies the plant architecture differences in multi-scales among different cultivars, thus, realizing a shift from the traditional qualitative to quantitative analysis of plant architecture. Overall, this study promotes the application of the 3D phytomer concept to multi-tiller crops, thereby providing a theoretical and technical basis for 3D plant modeling and plant architecture quantification in wheat.

Characterization and genetic dissection of maize ear leaf midrib acquired by 3D digital technology

The spatial morphological structure of plant leaves is an important index to evaluate crop ideotype. In this study, we characterized the three-dimensional (3D) data of the ear leaf midrib of maize at the grain-filling stage using the 3D digitization technology and obtained the phenotypic values of 15 traits covering four different dimensions of the ear leaf midrib, of which 13 phenotypic traits were firstly proposed for featuring plant leaf spatial structure. Cluster analysis results showed that the 13 traits could be divided into four groups, Group I, -II, -III and -IV. Group I contains HorizontalLength, OutwardGrowthMeasure, LeafAngle and DeviationTip; Group II contains DeviationAngle, MaxCurvature and CurvaturePos; Group III contains LeafLength and ProjectionArea; Group IV contains TipTop, VerticalHeight, UpwardGrowthMeasure, and CurvatureRatio. To investigate the genetic basis of the ear leaf midrib curve, 13 traits with high repeatability were subjected to genome-wide association study (GWAS) analysis. A total of 828 significantly related SNPs were identified and 1365 candidate genes were annotated. Among these, 29 candidate genes with the highest significant and multi-method validation were regarded as the key findings. In addition, pathway enrichment analysis was performed on the candidate genes of traits to explore the potential genetic mechanism of leaf midrib curve phenotype formation. These results not only contribute to further understanding of maize leaf spatial structure traits but also provide new genetic loci for maize leaf spatial structure to improve the plant type of maize varieties.

Three-dimensional reconstruction and phenotype measurement of maize seedlings based on multi-view image sequences

As an important method for crop phenotype quantification, three-dimensional (3D) reconstruction is of critical importance for exploring the phenotypic characteristics of crops. In this study, maize seedlings were subjected to 3D reconstruction based on the imaging technology, and their phenotypic characters were analyzed. In the first stage, a multi-view image sequence was acquired via an RGB camera and video frame extraction method, followed by 3D reconstruction of maize based on structure from motion algorithm. Next, the original point cloud data of maize were preprocessed through Euclidean clustering algorithm, color filtering algorithm and point cloud voxel filtering algorithm to obtain a point cloud model of maize. In the second stage, the phenotypic parameters in the development process of maize seedlings were analyzed, and the maize plant height, leaf length, relative leaf area and leaf width measured through point cloud were compared with the corresponding manually measured values, and the two were highly correlated, with the coefficient of determination (R ²) of 0.991, 0.989, 0.926 and 0.963, respectively. In addition, the errors generated between the two were also analyzed, and results reflected that the proposed method was capable of rapid, accurate and nondestructive extraction. In the third stage, maize stem leaves were segmented and identified through the region growing segmentation algorithm, and the expected segmentation effect was achieved. In general, the proposed method could accurately construct the 3D morphology of maize plants, segment maize leaves, and nondestructively and accurately extract the phenotypic parameters of maize plants, thus providing a data support for the research on maize phenotypes.

Surface Defect Detection of Cabbage Based on Curvature Features of 3D Point Cloud

The dents and cracks of cabbage caused by mechanical damage during transportation have a direct impact on both commercial value and storage time. In this study, a method for surface defect detection of cabbage is proposed based on the curvature feature of the 3D point cloud. First, the red-green-blue (RGB) images and depth images are collected using a RealSense-D455 depth camera for 3D point cloud reconstruction. Then, the region of interest (ROI) is extracted by statistical filtering and Euclidean clustering segmentation algorithm, and the 3D point cloud of cabbage is segmented from background noise. Then, the curvature features of the 3D point cloud are calculated using the estimated normal vector based on the least square plane fitting method. Finally, the curvature threshold is determined according to the curvature characteristic parameters, and the surface defect type and area can be detected. The flat-headed cabbage and round-headed cabbage are selected to test the surface damage of dents and cracks. The test results show that the average detection accuracy of this proposed method is 96.25%, in which, the average detection accuracy of dents is 93.3% and the average detection accuracy of cracks is 96.67%, suggesting high detection accuracy and good adaptability for various cabbages. This study provides important technical support for automatic and non-destructive detection of cabbage surface defects.

A Self-Adaptive Optimization Individual Tree Modeling Method for Terrestrial LiDAR Point Clouds

Individual tree modeling for terrestrial LiDAR point clouds always involves heavy computation burden and low accuracy toward a complex tree structure. To solve these problems, this paper proposed a self-adaptive optimization individual tree modeling method. In this paper, we first proposed a joint neighboring growing method to segment wood points into object primitives. Subsequently, local object primitives were optimized to alleviate the computation burden. To build the topology relation among branches, branches were separated based on spatial connectivity analysis. And then the nodes corresponding to each object primitive were adopted to construct the graph structure of the tree. Furthermore, each object primitive was fitted as a cylinder. To revise the local abnormal cylinder, a self-adaptive optimization method based on the constructed graph structure was proposed. Finally, the constructed tree model was further optimized globally based on prior knowledge. Twenty-nine field datasets obtained from three forest sites were adopted to evaluate the performance of the proposed method. The experimental results show that the proposed method can achieve satisfying individual tree modeling accuracy. The mean volume deviation of the proposed method is 1.427 m3. In the comparison with two other famous tree modeling methods, the proposed method can achieve the best individual tree modeling result no matter which accuracy indicator is selected.

Fast Tree Skeleton Extraction Using Voxel Thinning Based on Tree Point Cloud

Tree skeletons play an important role in tree structure analysis and 3D model reconstruction. However, it is a challenge to extract a skeleton from a tree point cloud with complex branches. In this paper, an automatic and fast tree skeleton extraction method (FTSEM) based on voxel thinning is proposed. In this method, a wood–leaf classification algorithm was introduced to filter leaf points for the reduction of the leaf interference on tree skeleton generation, tree voxel thinning was adopted to extract a raw tree skeleton quickly, and a breakpoint connection algorithm was used to improve the skeleton connectivity and completeness. Experiments were carried out in Haidian Park, Beijing, in which 24 trees were scanned and processed to obtain tree skeletons. The graph search algorithm (GSA) was used to extract tree skeletons based on the same datasets. Compared with the GSA method, the FTSEM method obtained more complete tree skeletons. The time cost of the FTSEM method was evaluated using the runtime and time per million points (TPMP). The runtime of FTSEM was from 1.0 s to 13.0 s, and the runtime of GSA was from 6.4 s to 309.3 s. The average value of TPMP was 1.8 s for FTSEM and 22.3 s for GSA, respectively. The experimental results demonstrate that the proposed method is feasible, robust, and fast with good potential for tree skeleton extraction.

U-RISC: An Annotated Ultra-High-Resolution Electron Microscopy Dataset Challenging the Existing Deep Learning Algorithms

Connectomics is a developing field aiming at reconstructing the connection of the neural system at the nanometer scale. Computer vision technology, especially deep learning methods used in image processing, has promoted connectomic data analysis to a new era. However, the performance of the state-of-the-art (SOTA) methods still falls behind the demand of scientific research. Inspired by the success of ImageNet, we present an annotated ultra-high resolution image segmentation dataset for cell membrane (U-RISC), which is the largest cell membrane-annotated electron microscopy (EM) dataset with a resolution of 2.18 nm/pixel. Multiple iterative annotations ensured the quality of the dataset. Through an open competition, we reveal that the performance of current deep learning methods still has a considerable gap from the human level, different from ISBI 2012, on which the performance of deep learning is closer to the human level. To explore the causes of this discrepancy, we analyze the neural networks with a visualization method, which is an attribution analysis. We find that the U-RISC requires a larger area around a pixel to predict whether the pixel belongs to the cell membrane or not. Finally, we integrate the currently available methods to provide a new benchmark (0.67, 10% higher than the leader of the competition, 0.61) for cell membrane segmentation on the U-RISC and propose some suggestions in developing deep learning algorithms. The U-RISC dataset and the deep learning codes used in this study are publicly available.

3D modeling and reconstruction of plants and trees: A cross-cutting review across computer graphics, vision, and plant phenotyping

This paper reviews the past and current trends of three-dimensional (3D) modeling and reconstruction of plants and trees. These topics have been studied in multiple research fields, including computer vision, graphics, plant phenotyping, and forestry. This paper, therefore, provides a cross-cutting review. Representations of plant shape and structure are first summarized, where every method for plant modeling and reconstruction is based on a shape/structure representation. The methods were then categorized into 1) creating non-existent plants (modeling) and 2) creating models from real-world plants (reconstruction). This paper also discusses the limitations of current methods and possible future directions.

Advances in optical phenotyping of cereal crops

Optical sensors and sensing-based phenotyping techniques have become mainstream approaches in high-throughput phenotyping for improving trait selection and genetic gains in crops. We review recent progress and contemporary applications of optical sensing-based phenotyping (OSP) techniques in cereal crops and highlight optical sensing principles for spectral response and sensor specifications. Further, we group phenotypic traits determined by OSP into four categories - morphological, biochemical, physiological, and performance traits - and illustrate appropriate sensors for each extraction. In addition to the current status, we discuss the challenges of OSP and provide possible solutions. We propose that optical sensing-based traits need to be explored further, and that standardization of the language of phenotyping and worldwide collaboration between phenotyping researchers and other fields need to be established.

A miniaturized phenotyping platform for individual plants using multi-view stereo 3D reconstruction

Plant phenotyping is essential in plant breeding and management. High-throughput data acquisition and automatic phenotypes extraction are common concerns in plant phenotyping. Despite the development of phenotyping platforms and the realization of high-throughput three-dimensional (3D) data acquisition in tall plants, such as maize, handling small-size plants with complex structural features remains a challenge. This study developed a miniaturized shoot phenotyping platform MVS-Pheno V2 focusing on low plant shoots. The platform is an improvement of MVS-Pheno V1 and was developed based on multi-view stereo 3D reconstruction. It has the following four components: Hardware, wireless communication and control, data acquisition system, and data processing system. The hardware sets the rotation on top of the platform, separating plants to be static while rotating. A novel local network was established to realize wireless communication and control; thus, preventing cable twining. The data processing system was developed to calibrate point clouds and extract phenotypes, including plant height, leaf area, projected area, shoot volume, and compactness. This study used three cultivars of wheat shoots at four growth stages to test the performance of the platform. The mean absolute percentage error of point cloud calibration was 0.585%. The squared correlation coefficient R ² was 0.9991, 0.9949, and 0.9693 for plant height, leaf length, and leaf width, respectively. The root mean squared error (RMSE) was 0.6996, 0.4531, and 0.1174 cm for plant height, leaf length, and leaf width. The MVS-Pheno V2 platform provides an alternative solution for high-throughput phenotyping of low individual plants and is especially suitable for shoot architecture-related plant breeding and management studies.

Understanding the potential of root microbiome influencing salt-tolerance in plants and mechanisms involved at the transcriptional and translational level

Soil salinity severely affects plant growth and development and imparts inevitable losses to crop productivity. Increasing the concentration of salts in the vicinity of plant roots has severe consequences at the morphological, biochemical, and molecular levels. These include loss of chlorophyll, decrease in photosynthetic rate, reduction in cell division, ROS generation, inactivation of antioxidative enzymes, alterations in phytohormone biosynthesis and signaling, and so forth. The association of microorganisms, viz. plant growth-promoting rhizobacteria, endophytes, and mycorrhiza, with plant roots constituting the root microbiome can confer a greater degree of salinity tolerance in addition to their inherent ability to promote growth and induce defense mechanisms. The mechanisms involved in induced stress tolerance bestowed by these microorganisms involve the modulation of phytohormone biosynthesis and signaling pathways (including indole acetic acid, gibberellic acid, brassinosteroids, abscisic acid, and jasmonic acid), accumulation of osmoprotectants (proline, glycine betaine, and sugar alcohols), and regulation of ion transporters (SOS1, NHX, HKT1). Apart from this, salt-tolerant microorganisms are known to induce the expression of salt-responsive genes via the action of several transcription factors, as well as by posttranscriptional and posttranslational modifications. Moreover, the potential of these salt-tolerant microflora can be employed for sustainably improving crop performance in saline environments. Therefore, this review will briefly focus on the key responses of plants under salinity stress and elucidate the mechanisms employed by the salt-tolerant microorganisms in improving plant tolerance under saline environments.

3D phytomer-based geometric modelling method for plants-the case of maize

Geometric plant modelling is crucial in in silico plants. Existing geometric modelling methods have focused on the topological structure and basic organ profiles, simplifying the morphological features. However, the models cannot effectively differentiate cultivars, limiting FSPM application in crop breeding and management. This study proposes a 3D phytomer-based geometric modelling method with maize (Zea Mays) as the representative plant. Specifically, conversion methods between skeleton and mesh models of 3D phytomer are specified. This study describes the geometric modelling of maize shoots and populations by assembling 3D phytomers. Results show that the method can quickly and efficiently construct 3D models of maize plants and populations, with the ability to show morphological, structural and functional differences among four representative cultivars. The method takes into account both the geometric modelling efficiency and 3D detail features to achieve automatic operation of geometric modelling through the standardized description of 3D phytomers. Therefore, this study provides a theoretical and technical basis for the research and application of in silico plants.

Canopy occupation volume as an indicator of canopy photosynthetic capacity

Leaf angle and leaf area index together influence canopy light interception and canopy photosynthesis. However, so far, there is no effective method to identify the optimal combination of these two parameters for canopy photosynthesis. In this study, first a robust high-throughput method for accurate segmentation of maize organs based on 3D point clouds data was developed, then the segmented plant organs were used to generate new 3D point clouds for the canopy of altered architectures. With this, we simulated the synergistic effect of leaf area and leaf angle on canopy photosynthesis. The results show that, compared to the traditional parameters describing the canopy photosynthesis including leaf area index, facet angle and canopy coverage, a new parameter - the canopy occupation volume (COV) - can better explain the variations of canopy photosynthetic capacity. Specifically, COV can explain > 79% variations of canopy photosynthesis generated by changing leaf angle and > 84% variations of canopy photosynthesis generated by changing leaf area. As COV can be calculated in a high-throughput manner based on the canopy point clouds, it can be used to evaluate canopy architecture in breeding and agronomic research.

EasyIDP: A Python Package for Intermediate Data Processing in UAV-Based Plant Phenotyping

Unmanned aerial vehicle (UAV) and structure from motion (SfM) photogrammetry techniques are widely used for field-based, high-throughput plant phenotyping nowadays, but some of the intermediate processes throughout the workflow remain manual. For example, geographic information system (GIS) software is used to manually assess the 2D/3D field reconstruction quality and cropping region of interests (ROIs) from the whole field. In addition, extracting phenotypic traits from raw UAV images is more competitive than directly from the digital orthomosaic (DOM). Currently, no easy-to-use tools are available to implement previous tasks for commonly used commercial SfM software, such as Pix4D and Agisoft Metashape. Hence, an open source software package called easy intermediate data processor (EasyIDP; MIT license) was developed to decrease the workload in intermediate data processing mentioned above. The functions of the proposed package include (1) an ROI cropping module, assisting in reconstruction quality assessment and cropping ROIs from the whole field, and (2) an ROI reversing module, projecting ROIs to relative raw images. The result showed that both cropping and reversing modules work as expected. Moreover, the effects of ROI height selection and reversed ROI position on raw images to reverse calculation were discussed. This tool shows great potential for decreasing workload in data annotation for machine learning applications.

UAS-Based Plant Phenotyping for Research and Breeding Applications

Unmanned aircraft system (UAS) is a particularly powerful tool for plant phenotyping, due to reasonable cost of procurement and deployment, ease and flexibility for control and operation, ability to reconfigure sensor payloads to diversify sensing, and the ability to seamlessly fit into a larger connected phenotyping network. These advantages have expanded the use of UAS-based plant phenotyping approach in research and breeding applications. This paper reviews the state of the art in the deployment, collection, curation, storage, and analysis of data from UAS-based phenotyping platforms. We discuss pressing technical challenges, identify future trends in UAS-based phenotyping that the plant research community should be aware of, and pinpoint key plant science and agronomic questions that can be resolved with the next generation of UAS-based imaging modalities and associated data analysis pipelines. This review provides a broad account of the state of the art in UAS-based phenotyping to reduce the barrier to entry to plant science practitioners interested in deploying this imaging modality for phenotyping in plant breeding and research areas.

Novel 3D Imaging Systems for High-Throughput Phenotyping of Plants

The use of 3D plant models for high-throughput phenotyping is increasingly becoming a preferred method for many plant science researchers. Numerous camera-based imaging systems and reconstruction algorithms have been developed for the 3D reconstruction of plants. However, it is still challenging to build an imaging system with high-quality results at a low cost. Useful comparative information for existing imaging systems and their improvements is also limited, making it challenging for researchers to make data-based selections. The objective of this study is to explore the possible solutions to address these issues. We introduce two novel systems for plants of various sizes, as well as a pipeline to generate high-quality 3D point clouds and meshes. The higher accuracy and efficiency of the proposed systems make it a potentially valuable tool for enhancing high-throughput phenotyping by integrating 3D traits for increased resolution and measuring traits that are not amenable to 2D imaging approaches. The study shows that the phenotype traits derived from the 3D models are highly correlated with manually measured phenotypic traits (R2 > 0.91). Moreover, we present a systematic analysis of different settings of the imaging systems and a comparison with the traditional system, which provide recommendations for plant scientists to improve the accuracy of 3D construction. In summary, our proposed imaging systems are suggested for 3D reconstruction of plants. Moreover, the analysis results of the different settings in this paper can be used for designing new customized imaging systems and improving their accuracy.

Label3DMaize: toolkit for 3D point cloud data annotation of maize shoots

Background

The 3D point cloud is the most direct and effective data form for studying plant structure and morphology. In point cloud studies, the point cloud segmentation of individual plants to organs directly determines the accuracy of organ-level phenotype estimation and the reliability of the 3D plant reconstruction. However, highly accurate, automatic, and robust point cloud segmentation approaches for plants are unavailable. Thus, the high-throughput segmentation of many shoots is challenging. Although deep learning can feasibly solve this issue, software tools for 3D point cloud annotation to construct the training dataset are lacking.

Results

We propose a top-to-down point cloud segmentation algorithm using optimal transportation distance for maize shoots. We apply our point cloud annotation toolkit for maize shoots, Label3DMaize, to achieve semi-automatic point cloud segmentation and annotation of maize shoots at different growth stages, through a series of operations, including stem segmentation, coarse segmentation, fine segmentation, and sample-based segmentation. The toolkit takes ∼4–10 minutes to segment a maize shoot and consumes 10–20% of the total time if only coarse segmentation is required. Fine segmentation is more detailed than coarse segmentation, especially at the organ connection regions. The accuracy of coarse segmentation can reach 97.2% that of fine segmentation.

Conclusion

Label3DMaize integrates point cloud segmentation algorithms and manual interactive operations, realizing semi-automatic point cloud segmentation of maize shoots at different growth stages. The toolkit provides a practical data annotation tool for further online segmentation research based on deep learning and is expected to promote automatic point cloud processing of various plants.

Registration of spatio-temporal point clouds of plants for phenotyping

Plant phenotyping is a central task in crop science and plant breeding. It involves measuring plant traits to describe the anatomy and physiology of plants and is used for deriving traits and evaluating plant performance. Traditional methods for phenotyping are often time-consuming operations involving substantial manual labor. The availability of 3D sensor data of plants obtained from laser scanners or modern depth cameras offers the potential to automate several of these phenotyping tasks. This automation can scale up the phenotyping measurements and evaluations that have to be performed to a larger number of plant samples and at a finer spatial and temporal resolution. In this paper, we investigate the problem of registering 3D point clouds of the plants over time and space. This means that we determine correspondences between point clouds of plants taken at different points in time and register them using a new, non-rigid registration approach. This approach has the potential to form the backbone for phenotyping applications aimed at tracking the traits of plants over time. The registration task involves finding data associations between measurements taken at different times while the plants grow and change their appearance, allowing 3D models taken at different points in time to be compared with each other. Registering plants over time is challenging due to its anisotropic growth, changing topology, and non-rigid motion in between the time of the measurements. Thus, we propose a novel approach that first extracts a compact representation of the plant in the form of a skeleton that encodes both topology and semantic information, and then use this skeletal structure to determine correspondences over time and drive the registration process. Through this approach, we can tackle the data association problem for the time-series point cloud data of plants effectively. We tested our approach on different datasets acquired over time and successfully registered the 3D plant point clouds recorded with a laser scanner. We demonstrate that our method allows for developing systems for automated temporal plant-trait analysis by tracking plant traits at an organ level.

Robust Surface Reconstruction of Plant Leaves from 3D Point Clouds

The automation of plant phenotyping using 3D imaging techniques is indispensable. However, conventional methods for reconstructing the leaf surface from 3D point clouds have a trade-off between the accuracy of leaf surface reconstruction and the method's robustness against noise and missing points. To mitigate this trade-off, we developed a leaf surface reconstruction method that reduces the effects of noise and missing points while maintaining surface reconstruction accuracy by capturing two components of the leaf (the shape and distortion of that shape) separately using leaf-specific properties. This separation simplifies leaf surface reconstruction compared with conventional methods while increasing the robustness against noise and missing points. To evaluate the proposed method, we reconstructed the leaf surfaces from 3D point clouds of leaves acquired from two crop species (soybean and sugar beet) and compared the results with those of conventional methods. The result showed that the proposed method robustly reconstructed the leaf surfaces, despite the noise and missing points for two different leaf shapes. To evaluate the stability of the leaf surface reconstructions, we also calculated the leaf surface areas for 14 consecutive days of the target leaves. The result derived from the proposed method showed less variation of values and fewer outliers compared with the conventional methods.

Leveraging Image Analysis to Compute 3D Plant Phenotypes Based on Voxel-Grid Plant Reconstruction

High throughput image-based plant phenotyping facilitates the extraction of morphological and biophysical traits of a large number of plants non-invasively in a relatively short time. It facilitates the computation of advanced phenotypes by considering the plant as a single object (holistic phenotypes) or its components, i.e., leaves and the stem (component phenotypes). The architectural complexity of plants increases over time due to variations in self-occlusions and phyllotaxy, i.e., arrangements of leaves around the stem. One of the central challenges to computing phenotypes from 2-dimensional (2D) single view images of plants, especially at the advanced vegetative stage in presence of self-occluding leaves, is that the information captured in 2D images is incomplete, and hence, the computed phenotypes are inaccurate. We introduce a novel algorithm to compute 3-dimensional (3D) plant phenotypes from multiview images using voxel-grid reconstruction of the plant (3DPhenoMV). The paper also presents a novel method to reliably detect and separate the individual leaves and the stem from the 3D voxel-grid of the plant using voxel overlapping consistency check and point cloud clustering techniques. To evaluate the performance of the proposed algorithm, we introduce the University of Nebraska-Lincoln 3D Plant Phenotyping Dataset (UNL-3DPPD). A generic taxonomy of 3D image-based plant phenotypes are also presented to promote 3D plant phenotyping research. A subset of these phenotypes are computed using computer vision algorithms with discussion of their significance in the context of plant science. The central contributions of the paper are (a) an algorithm for 3D voxel-grid reconstruction of maize plants at the advanced vegetative stages using images from multiple 2D views; (b) a generic taxonomy of 3D image-based plant phenotypes and a public benchmark dataset, i.e., UNL-3DPPD, to promote the development of 3D image-based plant phenotyping research; and (c) novel voxel overlapping consistency check and point cloud clustering techniques to detect and isolate individual leaves and stem of the maize plants to compute the component phenotypes. Detailed experimental analyses demonstrate the efficacy of the proposed method, and also show the potential of 3D phenotypes to explain the morphological characteristics of plants regulated by genetic and environmental interactions.

Apple Tree Branch Information Extraction from Terrestrial Laser Scanning and Backpack-LiDAR

The branches of fruit trees provide support for the growth of leaves, buds, flowers, fruits, and other organs. The number and length of branches guarantee the normal growth, flowering, and fruiting of fruit trees and are thus important indicators of tree growth and yield. However, due to their low height and the high number of branches, the precise management of fruit trees lacks a theoretical basis and data support. In this paper, we introduce a method for extracting topological and structural information on fruit tree branches based on LiDAR (Light Detection and Ranging) point clouds and proved its feasibility for the study of fruit tree branches. The results show that based on Terrestrial Laser Scanning (TLS), the relative errors of branch length and number are 7.43% and 12% for first-order branches, and 16.75% and 9.67% for second-order branches. The accuracy of total branch information can reach 15.34% and 2.89%. We also evaluated the potential of backpack-LiDAR by comparing field measurements and quantitative structural models (QSMs) evaluations of 10 sample trees. This comparison shows that in addition to the first-order branch information, the information about other orders of branches is underestimated to varying degrees. The root means square error (RMSE) of the length and number of the first-order branches were 3.91 and 1.30 m, and the relative root means square error (NRMSE) was 14.62% and 11.96%, respectively. Our work represents the first automated classification of fruit tree branches, which can be used in support of precise fruit tree pruning, quantitative forecast of yield, evaluation of fruit tree growth, and the modern management of orchards.

A Low-Cost 3D Phenotype Measurement Method of Leafy Vegetables Using Video Recordings from Smartphones

Leafy vegetables are an essential source of the various nutrients that people need in their daily lives. The quantification of vegetable phenotypes and yield estimation are prerequisites for the selection of genetic varieties and for the improvement of planting methods. The traditional method is manual measurement, which is time-consuming and cumbersome. Therefore, there is a need for efficient and convenient in situ vegetable phenotype identification methods to provide data support for breeding research and for crop yield monitoring, thereby increasing vegetable yield. In this paper, a novel approach was developed for the in-situ determination of the three-dimensional (3D) phenotype of vegetables by recording video clips using smartphones. First, a smartphone was used to record the vegetable from different angles, and then the key frame containing the crop area in the video was obtained using an algorithm based on the vegetation index and scale-invariant feature transform algorithm (SIFT) matching. After obtaining the key frame, a dense point cloud of the vegetables was reconstructed using the Structure from Motion (SfM) method, and then the segmented point cloud and a point cloud skeleton were obtained using the clustering algorithm. Finally, the plant height, leaf number, leaf length, leaf angle, and other phenotypic parameters were obtained through the point cloud and point cloud skeleton. Comparing the obtained phenotypic parameters to the manual measurement results, the root-mean-square error (RMSE) of the plant height, leaf number, leaf length, and leaf angle were 1.82, 1.57, 2.43, and 4.7, respectively. The measurement accuracy of each indicators is greater than 80%. The results show that the proposed method provides a convenient, fast, and low-cost 3D phenotype measurement pipeline. Compared to other methods based on photogrammetry, this method does not need a labor-intensive image-capturing process and can reconstruct a high-quality point cloud model by directly recording videos of crops.

MVS-Pheno: A Portable and Low-Cost Phenotyping Platform for Maize Shoots Using Multiview Stereo 3D Reconstruction

Plant phenotyping technologies play important roles in plant research and agriculture. Detailed phenotypes of individual plants can guide the optimization of shoot architecture for plant breeding and are useful to analyze the morphological differences in response to environments for crop cultivation. Accordingly, high-throughput phenotyping technologies for individual plants grown in field conditions are urgently needed, and MVS-Pheno, a portable and low-cost phenotyping platform for individual plants, was developed. The platform is composed of four major components: a semiautomatic multiview stereo (MVS) image acquisition device, a data acquisition console, data processing and phenotype extraction software for maize shoots, and a data management system. The platform's device is detachable and adjustable according to the size of the target shoot. Image sequences for each maize shoot can be captured within 60-120 seconds, yielding 3D point clouds of shoots are reconstructed using MVS-based commercial software, and the phenotypic traits at the organ and individual plant levels are then extracted by the software. The correlation coefficient (R 2) between the extracted and manually measured plant height, leaf width, and leaf area values are 0.99, 0.87, and 0.93, respectively. A data management system has also been developed to store and manage the acquired raw data, reconstructed point clouds, agronomic information, and resulting phenotypic traits. The platform offers an optional solution for high-throughput phenotyping of field-grown plants, which is especially useful for large populations or experiments across many different ecological regions.

Skeletonization of Plant Point Cloud Data in Stochastic Optimization Framework.. [DOI: 10.1101/2020.02.15.950519] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Skeleton extraction from 3D plant point cloud data is an essential prior for myriads of phenotyping studies. Although skeleton extraction from 3D shapes have been studied extensively in the computer vision and graphics literature, handling the case of plants is still an open problem. Drawbacks of the existing approaches include the zigzag structure of the skeleton, nonuniform density of skeleton points, lack of points in the areas having complex geometry structure, and most importantly the lack of biological relevance. With the aim to improve existing skeleton structures of state-of-the-art, we propose a stochastic framework which is supported by the biological structure of the original plant (we consider plants without any leaves). Initially we estimate the branching structure of the plant by the notion of β-splines to form a curve tree defined as a finite set of curves joined in a tree topology with certain level of smoothness. In the next phase, we force the discrete points in the curve tree to move towards the original point cloud by treating each point in the curve tree as a center of Gaussian, and points in the input cloud data as observations from the Gaussians. The task is to find the correct locations of the Gaussian centroids by maximizing a likelihood. The optimization technique is iterative and is based on the Expectation Maximization (EM) algorithm. The E-step estimates which Gaussian the observed point cloud was sampled from, and the M-step maximizes the negative log-likelihood that the observed points were sampled from the Gaussian Mixture Model (GMM) with respect to the model parameters. We experiment with several real world and synthetic datasets and demonstrate the robustness of the approach over the state-of-the-art.

Skeletonization of Plant Point Cloud Data Using Stochastic Optimization Framework

Skeleton extraction from 3D plant point cloud data is an essential prior for myriads of phenotyping studies. Although skeleton extraction from 3D shapes have been studied extensively in the computer vision and graphics literature, handling the case of plants is still an open problem. Drawbacks of the existing approaches include the zigzag structure of the skeleton, nonuniform density of skeleton points, lack of points in the areas having complex geometry structure, and most importantly the lack of biological relevance. With the aim to improve existing skeleton structures of state-of-the-art, we propose a stochastic framework which is supported by the biological structure of the original plant (we consider plants without any leaves). Initially we estimate the branching structure of the plant by the notion of β-splines to form a curve tree defined as a finite set of curves joined in a tree topology with certain level of smoothness. In the next phase, we force the discrete points in the curve tree to move toward the original point cloud by treating each point in the curve tree as a center of Gaussian, and points in the input cloud data as observations from the Gaussians. The task is to find the correct locations of the Gaussian centroids by maximizing a likelihood. The optimization technique is iterative and is based on the Expectation Maximization (EM) algorithm. The E-step estimates which Gaussian the observed point cloud was sampled from, and the M-step maximizes the negative log-likelihood that the observed points were sampled from the Gaussian Mixture Model (GMM) with respect to the model parameters. We experiment with several real world and synthetic datasets and demonstrate the robustness of the approach over the state-of-the-art.

Measuring crops in 3D: using geometry for plant phenotyping

Using 3D sensing for plant phenotyping has risen within the last years. This review provides an overview on 3D traits for the demands of plant phenotyping considering different measuring techniques, derived traits and use-cases of biological applications. A comparison between a high resolution 3D measuring device and an established measuring tool, the leaf meter, is shown to categorize the possible measurement accuracy. Furthermore, different measuring techniques such as laser triangulation, structure from motion, time-of-flight, terrestrial laser scanning or structured light approaches enable the assessment of plant traits such as leaf width and length, plant size, volume and development on plant and organ level. The introduced traits were shown with respect to the measured plant types, the used measuring technique and the link to their biological use case. These were trait and growth analysis for measurements over time as well as more complex investigation on water budget, drought responses and QTL (quantitative trait loci) analysis. The used processing pipelines were generalized in a 3D point cloud processing workflow showing the single processing steps to derive plant parameters on plant level, on organ level using machine learning or over time using time series measurements. Finally the next step in plant sensing, the fusion of different sensor types namely 3D and spectral measurements is introduced by an example on sugar beet. This multi-dimensional plant model is the key to model the influence of geometry on radiometric measurements and to correct it. This publication depicts the state of the art for 3D measuring of plant traits as they were used in plant phenotyping regarding how the data is acquired, how this data is processed and what kind of traits is measured at the single plant, the miniplot, the experimental field and the open field scale. Future research will focus on highly resolved point clouds on the experimental and field scale as well as on the automated trait extraction of organ traits to track organ development at these scales.

A Clustering Framework for Monitoring Circadian Rhythm in Structural Dynamics in Plants From Terrestrial Laser Scanning Time Series

Terrestrial Laser Scanning (TLS) can be used to monitor plant dynamics with a frequency of several times per hour and with sub-centimeter accuracy, regardless of external lighting conditions. TLS point cloud time series measured at short intervals produce large quantities of data requiring fast processing techniques. These must be robust to the noise inherent in point clouds. This study presents a general framework for monitoring circadian rhythm in plant movements from TLS time series. Framework performance was evaluated using TLS time series collected from two Norway maples (Acer platanoides) and a control target, a lamppost. The results showed that the processing framework presented can capture a plant's circadian rhythm in crown and branches down to a spatial resolution of 1 cm. The largest movements in both Norway maples were observed before sunrise and at their crowns' outer edges. The individual cluster movements were up to 0.17 m (99th percentile) for the taller Norway maple and up to 0.11 m (99th percentile) for the smaller tree from their initial positions before sunset.