Image-based 3D canopy reconstruction to determine potential productivity in complex multi-species crop systems

How to make sense of 3D representations for plant phenotyping: a compendium of processing and analysis techniques

Computer vision technology is moving more and more towards a three-dimensional approach, and plant phenotyping is following this trend. However, despite its potential, the complexity of the analysis of 3D representations has been the main bottleneck hindering the wider deployment of 3D plant phenotyping. In this review we provide an overview of typical steps for the processing and analysis of 3D representations of plants, to offer potential users of 3D phenotyping a first gateway into its application, and to stimulate its further development. We focus on plant phenotyping applications where the goal is to measure characteristics of single plants or crop canopies on a small scale in research settings, as opposed to large scale crop monitoring in the field.

Photoacclimation and entrainment of photosynthesis by fluctuating light varies according to genotype in Arabidopsis thaliana

Acclimation of photosynthesis to light intensity (photoacclimation) takes days to achieve and so naturally fluctuating light presents a potential challenge where leaves may be exposed to light conditions that are beyond their window of acclimation. Experiments generally have focused on unchanging light with a relatively fixed combination of photosynthetic attributes to confer higher efficiency in those conditions. Here a controlled LED experiment and mathematical modelling was used to assess the acclimation potential of contrasting Arabidopsis thaliana genotypes following transfer to a controlled fluctuating light environment, designed to present frequencies and amplitudes more relevant to natural conditions. We hypothesize that acclimation of light harvesting, photosynthetic capacity and dark respiration are controlled independently. Two different ecotypes were selected, Wassilewskija-4 (Ws), Landsberg erecta (Ler) and a GPT2 knock out mutant on the Ws background (gpt2-), based on their differing abilities to undergo dynamic acclimation i.e. at the sub-cellular or chloroplastic scale. Results from gas exchange and chlorophyll content indicate that plants can independently regulate different components that could optimize photosynthesis in both high and low light; targeting light harvesting in low light and photosynthetic capacity in high light. Empirical modelling indicates that the pattern of 'entrainment' of photosynthetic capacity by past light history is genotype-specific. These data show flexibility of photoacclimation and variation useful for plant improvement.

Nutrient Accumulation Pattern in Mixtures of Wheat and Faba Bean Is Strongly Influenced by Cultivar Choice and Co-Existing Weeds

Cereal-legume mixtures are often associated with higher yields than the components grown as sole crops, but the underlying mechanisms are unclear. The study aims to evaluate how different cultivars in a two-species wheat-faba bean mixture influence above- and below-ground nitrogen (N) accumulation in the plant biomass, whether crop mixing affected the accumulation of other nutrients relative to the accumulation of N and phosphorus (P), and how the nutrient accumulation pattern in sole crops and mixtures is influenced by weed competition. Using a growth container experiment, we investigate nutrient accumulation patterns on specific wheat and faba bean cultivars grown as sole crops and mixtures, and with and without weed competition. We found that cereals in the mixture accumulated more N than in the sole crops, and the cultivar used influenced biomass accumulation in the legumes. Competition from weeds reduced the amount of plant N pools accumulated in the crop plant biomass. Based on stoichiometric scaling exponents, the plant neighbor affected the accumulation of other nutrients relative to the accumulation of N and P. These results are relevant for species and cultivar selection, all of which are important prerequisites for maximizing mixture performance.

Prediction of Photosynthetic, Biophysical, and Biochemical Traits in Wheat Canopies to Reduce the Phenotyping Bottleneck

To achieve food security, it is necessary to increase crop radiation use efficiency (RUE) and yield through the enhancement of canopy photosynthesis to increase the availability of assimilates for the grain, but its study in the field is constrained by low throughput and the lack of integrative measurements at canopy level. In this study, partial least squares regression (PLSR) was used with high-throughput phenotyping (HTP) data in spring wheat to build predictive models of photosynthetic, biophysical, and biochemical traits for the top, middle, and bottom layers of wheat canopies. The combined layer model predictions performed better than individual layer predictions with a significance as follows for photosynthesis R ² = 0.48, RMSE = 5.24 μmol m^-2 s^-1 and stomatal conductance: R ² = 0.36, RMSE = 0.14 mol m^-2 s^-1. The predictions of these traits from PLSR models upscaled to canopy level compared to field observations were statistically significant at initiation of booting (R ² = 0.3, p < 0.05; R ² = 0.29, p < 0.05) and at 7 days after anthesis (R ² = 0.15, p < 0.05; R ² = 0.65, p < 0.001). Using HTP allowed us to increase phenotyping capacity 30-fold compared to conventional phenotyping methods. This approach can be adapted to screen breeding progeny and genetic resources for RUE and to improve our understanding of wheat physiology by adding different layers of the canopy to physiological modeling.

Photoprotective energy dissipation is greater in the lower, not the upper, regions of a rice canopy: a 3D analysis

High light intensities raise photosynthetic and plant growth rates but can cause damage to the photosynthetic machinery. The likelihood and severity of deleterious effects are minimised by a set of photoprotective mechanisms, one key process being the controlled dissipation of energy from chlorophyll within PSII known as non-photochemical quenching (NPQ). Although ubiquitous, the role of NPQ in plant productivity is important because it momentarily reduces the quantum efficiency of photosynthesis. Rice plants overexpressing and deficient in the gene encoding a central regulator of NPQ, the protein PsbS, were used to assess the effect of protective effectiveness of NPQ (pNPQ) at the canopy scale. Using a combination of three-dimensional reconstruction, modelling, chlorophyll fluorescence, and gas exchange, the influence of altered NPQ capacity on the distribution of pNPQ was explored. A higher phototolerance in the lower layers of a canopy was found, regardless of genotype, suggesting a mechanism for increased protection for leaves that experience relatively low light intensities interspersed with brief periods of high light. Relative to wild-type plants, psbS overexpressors have a reduced risk of photoinactivation and early growth advantage, demonstrating that manipulating photoprotective mechanisms can impact both subcellular mechanisms and whole-canopy function.

Modelling of Soybean (Glycine max (L.) Merr.) Response to Blue Light Intensity in Controlled Environments

Low photosynthetic photon flux density (PPFD) under shade is associated with low blue photon flux density (BPFD), which independent from PPFD can induce shade responses, e.g., elongation growth. In this study, the response of soybean to six levels of BPFD under constant PPFD from LED lighting was investigated with regard to morphology, biomass and photosynthesis to increase the knowledge for optimizing the intensity of BPFD for a speed breeding system. The results showed that low BPFD increased plant height, leaf area and biomass and decreased leaf mass ratio. Photosynthetic rate and internode diameter were not influenced. A functional structural plant model of soybean was calibrated with the experimental data. A response function for internode length to the perceived BPFD by the internodes was derived from simulations and integrated into the model. With the aim to optimize lighting for a speed breeding system, simulations with alternative lighting scenarios indicated that decreasing BPFD during the growth period and using different chamber material with a higher reflectance could reduce energy consumption by 7% compared to the experimental setup, while inducing short soybean plants.

Active Vision and Surface Reconstruction for 3D Plant Shoot Modelling

Plant phenotyping is the quantitative description of a plant's physiological, biochemical, and anatomical status which can be used in trait selection and helps to provide mechanisms to link underlying genetics with yield. Here, an active vision- based pipeline is presented which aims to contribute to reducing the bottleneck associated with phenotyping of architectural traits. The pipeline provides a fully automated response to photometric data acquisition and the recovery of three-dimensional (3D) models of plants without the dependency of botanical expertise, whilst ensuring a non-intrusive and non-destructive approach. Access to complete and accurate 3D models of plants supports computation of a wide variety of structural measurements. An Active Vision Cell (AVC) consisting of a camera-mounted robot arm plus combined software interface and a novel surface reconstruction algorithm is proposed. This pipeline provides a robust, flexible, and accurate method for automating the 3D reconstruction of plants. The reconstruction algorithm can reduce noise and provides a promising and extendable framework for high throughput phenotyping, improving current state-of-the-art methods. Furthermore, the pipeline can be applied to any plant species or form due to the application of an active vision framework combined with the automatic selection of key parameters for surface reconstruction.

Quantification of light interception within image-based 3-D reconstruction of sole and intercropped canopies over the entire growth season

BACKGROUND AND AIMS

Light interception is closely related to canopy architecture. Few studies based on multi-view photography have been conducted in a field environment, particularly studies that link 3-D plant architecture with a radiation model to quantify the dynamic canopy light interception. In this study, we combined realistic 3-D plant architecture with a radiation model to quantify and evaluate the effect of differences in planting patterns and row orientations on canopy light interception.

METHODS

The 3-D architectures of maize and soybean plants were reconstructed for sole crops and intercrops based on multi-view images obtained at five growth dates in the field. We evaluated the accuracy of the calculated leaf length, maximum leaf width, plant height and leaf area according to the measured data. The light distribution within the 3-D plant canopy was calculated with a 3-D radiation model. Finally, we evaluated canopy light interception in different row orientations.

KEY RESULTS

There was good agreement between the measured and calculated phenotypic traits, with an R2 >0.97. The light distribution was more uniform for intercropped maize and more concentrated for sole maize. At the maize silking stage, 85 % of radiation was intercepted by approx. 55 % of the upper canopy region for maize and by approx. 33 % of the upper canopy region for soybean. There was no significant difference in daily light interception between the different row orientations for the entire intercropping and sole systems. However, for intercropped maize, near east-west orientations showed approx. 19 % higher daily light interception than near south-north orientations. For intercropped soybean, daily light interception showed the opposite trend. It was approx. 49 % higher for near south-north orientations than for near east-west orientations.

CONCLUSIONS

The accurate reconstruction of 3-D plants grown in the field based on multi-view images provides the possibility for high-throughput 3-D phenotyping in the field and allows a better understanding of the relationship between canopy architecture and the light environment.

Analysing the phenotype development of soybean plants using low-cost 3D reconstruction

With the development of digital agriculture, 3D reconstruction technology has been widely used to analyse crop phenotypes. To date, most research on 3D reconstruction of field crops has been limited to analysis of population characteristics. Therefore, in this study, we propose a method based on low-cost 3D reconstruction technology to analyse the phenotype development during the whole growth period. Based on the phenotypic parameters extracted from the 3D reconstruction model, we identified the "phenotypic fingerprint" of the relevant phenotypes throughout the whole growth period of soybean plants and completed analysis of the plant growth patterns using a logistic growth model. The phenotypic fingerprint showed that, before the R3 period, the growth of the five varieties was similar. After the R5 period, the differences among the five cultivars gradually increased. This result indicates that the phenotypic fingerprint can accurately reveal the patterns of phenotypic changes. The logistic growth model of soybean plants revealed the time points of maximum growth rate of the five soybean varieties, and this information can provide a basis for developing guidelines for water and fertiliser application to crops. These findings will provide effective guidance for breeding and field management of soybean and other crops.

All roads lead to growth: imaging-based and biochemical methods to measure plant growth

Plant growth is a highly complex biological process that involves innumerable interconnected biochemical and signalling pathways. Many different techniques have been developed to measure growth, unravel the various processes that contribute to plant growth, and understand how a complex interaction between genotype and environment determines the growth phenotype. Despite this complexity, the term 'growth' is often simplified by researchers; depending on the method used for quantification, growth is viewed as an increase in plant or organ size, a change in cell architecture, or an increase in structural biomass. In this review, we summarise the cellular and molecular mechanisms underlying plant growth, highlight state-of-the-art imaging and non-imaging-based techniques to quantitatively measure growth, including a discussion of their advantages and drawbacks, and suggest a terminology for growth rates depending on the type of technique used.

A photometric stereo-based 3D imaging system using computer vision and deep learning for tracking plant growth

BACKGROUND

Tracking and predicting the growth performance of plants in different environments is critical for predicting the impact of global climate change. Automated approaches for image capture and analysis have allowed for substantial increases in the throughput of quantitative growth trait measurements compared with manual assessments. Recent work has focused on adopting computer vision and machine learning approaches to improve the accuracy of automated plant phenotyping. Here we present PS-Plant, a low-cost and portable 3D plant phenotyping platform based on an imaging technique novel to plant phenotyping called photometric stereo (PS).

RESULTS

We calibrated PS-Plant to track the model plant Arabidopsis thaliana throughout the day-night (diel) cycle and investigated growth architecture under a variety of conditions to illustrate the dramatic effect of the environment on plant phenotype. We developed bespoke computer vision algorithms and assessed available deep neural network architectures to automate the segmentation of rosettes and individual leaves, and extract basic and more advanced traits from PS-derived data, including the tracking of 3D plant growth and diel leaf hyponastic movement. Furthermore, we have produced the first PS training data set, which includes 221 manually annotated Arabidopsis rosettes that were used for training and data analysis (1,768 images in total). A full protocol is provided, including all software components and an additional test data set.

CONCLUSIONS

PS-Plant is a powerful new phenotyping tool for plant research that provides robust data at high temporal and spatial resolutions. The system is well-suited for small- and large-scale research and will help to accelerate bridging of the phenotype-to-genotype gap.

Maize Plant Phenotyping: Comparing 3D Laser Scanning, Multi-View Stereo Reconstruction, and 3D Digitizing Estimates

High-throughput phenotyping technologies have become an increasingly important topic of crop science in recent years. Various sensors and data acquisition approaches have been applied to acquire the phenotyping traits. It is quite confusing for crop phenotyping researchers to determine an appropriate way for their application. In this study, three representative three-dimensional (3D) data acquisition approaches, including 3D laser scanning, multi-view stereo (MVS) reconstruction, and 3D digitizing, were evaluated for maize plant phenotyping in multi growth stages. Phenotyping traits accuracy, post-processing difficulty, device cost, data acquisition efficiency, and automation were considered during the evaluation process. 3D scanning provided satisfactory point clouds for medium and high maize plants with acceptable efficiency, while the results were not satisfactory for small maize plants. The equipment used in 3D scanning is expensive, but is highly automatic. MVS reconstruction provided satisfactory point clouds for small and medium plants, and point deviations were observed in upper parts of higher plants. MVS data acquisition, using low-cost cameras, exhibited the highest efficiency among the three evaluated approaches. The one-by-one pipeline data acquisition pattern allows the use of MVS high-throughput in further phenotyping platforms. Undoubtedly, enhancement of point cloud processing technologies is required to improve the extracted phenotyping traits accuracy for both 3D scanning and MVS reconstruction. Finally, 3D digitizing was time-consuming and labor intensive. However, it does not depend on any post-processing algorithms to extract phenotyping parameters and reliable phenotyping traits could be derived. The promising accuracy of 3D digitizing is a better verification choice for other 3D phenotyping approaches. Our study provides clear reference about phenotyping data acquisition of maize plants, especially for the affordable and portable field phenotyping platforms to be developed.

Three-dimensional plant architecture and sunlit-shaded patterns: a stochastic model of light dynamics in canopies

Background and Aims

Diurnal changes in solar position and intensity combined with the structural complexity of plant architecture result in highly variable and dynamic light patterns within the plant canopy. This affects productivity through the complex ways that photosynthesis responds to changes in light intensity. Current methods to characterize light dynamics, such as ray-tracing, are able to produce data with excellent spatio-temporal resolution but are computationally intensive and the resulting data are complex and high-dimensional. This necessitates development of more economical models for summarizing the data and for simulating realistic light patterns over the course of a day.

Methods

High-resolution reconstructions of field-grown plants are assembled in various configurations to form canopies, and a forward ray-tracing algorithm is applied to the canopies to compute light dynamics at high (1 min) temporal resolution. From the ray-tracer output, the sunlit or shaded state for each patch on the plants is determined, and these data are used to develop a novel stochastic model for the sunlit-shaded patterns. The model is designed to be straightforward to fit to data using maximum likelihood estimation, and fast to simulate from.

Key Results

For a wide range of contrasting 3-D canopies, the stochastic model is able to summarize, and replicate in simulations, key features of the light dynamics. When light patterns simulated from the stochastic model are used as input to a model of photoinhibition, the predicted reduction in carbon gain is similar to that from calculations based on the (extremely costly) ray-tracer data.

Conclusions

The model provides a way to summarize highly complex data in a small number of parameters, and a cost-effective way to simulate realistic light patterns. Simulations from the model will be particularly useful for feeding into larger-scale photosynthesis models for calculating how light dynamics affects the photosynthetic productivity of canopies.

Image-based dynamic quantification and high-accuracy 3D evaluation of canopy structure of plant populations

BACKGROUND AND AIMS

Global agriculture is facing the challenge of a phenotyping bottleneck due to large-scale screening/breeding experiments with improved breeds. Phenotypic analysis with high-throughput, high-accuracy and low-cost technologies has therefore become urgent. Recent advances in image-based 3D reconstruction offer the opportunity of high-throughput phenotyping. The main aim of this study was to quantify and evaluate the canopy structure of plant populations in two and three dimensions based on the multi-view stereo (MVS) approach, and to monitor plant growth and development from seedling stage to fruiting stage.

METHODS

Multi-view images of flat-leaf cucumber, small-leaf pepper and curly-leaf eggplant were obtained by moving a camera around the plant canopy. Three-dimensional point clouds were reconstructed from images based on the MVS approach and were then converted into surfaces with triangular facets. Phenotypic parameters, including leaf length, leaf width, leaf area, plant height and maximum canopy width, were calculated from reconstructed surfaces. Accurate evaluation in 2D and 3D for individual leaves was performed by comparing reconstructed phenotypic parameters with referenced values and by calculating the Hausdorff distance, i.e. the mean distance between two surfaces.

KEY RESULTS

Our analysis demonstrates that there were good agreements in leaf parameters between referenced and estimated values. A high level of overlap was also found between surfaces of image-based reconstructions and laser scanning. Accuracy of 3D reconstruction of curly-leaf plants was relatively lower than that of flat-leaf plants. Plant height of three plants and maximum canopy width of cucumber and pepper showed an increasing trend during the 70 d after transplanting. Maximum canopy width of eggplants reached its peak at the 40th day after transplanting. The larger leaf phenotypic parameters of cucumber were mostly found at the middle-upper leaf position.

CONCLUSIONS

High-accuracy 3D evaluation of reconstruction quality indicated that dynamic capture of the 3D canopy based on the MVS approach can be potentially used in 3D phenotyping for applications in breeding and field management.

Intercropping with wheat lowers nutrient uptake and biomass accumulation of maize, but increases photosynthetic rate of the ear leaf

Intercropping is an ancient agricultural practice that provides a possible pathway for sustainable increases in crop yields. Here, we determine how competition with wheat affects nutrient uptake (nitrogen and phosphorus) and leaf traits, such as photosynthetic rate, in maize. In a field experiment, maize was planted as a sole crop, in three different intercrop configurations with wheat (a replacement intercrop and two add-row intercrops), and as a skip-row system with one out of each three maize rows omitted. Nitrogen and phosphorus uptake were determined at flowering and maturity. Specific leaf area, leaf nitrogen concentration, chlorophyll content and photosynthetic rate of the ear leaf were determined at flowering. Nitrogen and phosphorus concentrations were significantly lower in intercropped maize than in sole maize and skip-row maize at flowering, but these differences were smaller at maturity. At flowering, specific leaf area was significantly greater in intercrops than in skip-row maize. Leaf nitrogen concentration was significantly lower in add-row intercrops than in sole maize, skip-row maize or maize in the replacement intercrop. Leaf chlorophyll content was highest in sole and skip-row maize, intermediate in maize in the replacement intercrop and lowest in maize grown in add-row intercrops. On the contrary, photosynthetic rate was significantly higher in the replacement intercrop than in sole maize, skip-row maize and the intercrop with an additional maize row. The findings indicate that competition with intercropped wheat severely constrained nutrient uptake in maize, while photosynthetic rate of the ear leaf was not negatively affected. Possible mechanisms for higher photosynthesis rate at lower leaf nitrogen content in intercropped maize are discussed.

A robot-assisted imaging pipeline for tracking the growths of maize ear and silks in a high-throughput phenotyping platform

BACKGROUND

In maize, silks are hundreds of filaments that simultaneously emerge from the ear for collecting pollen over a period of 1-7 days, which largely determines grain number especially under water deficit. Silk growth is a major trait for drought tolerance in maize, but its phenotyping is difficult at throughputs needed for genetic analyses.

RESULTS

We have developed a reproducible pipeline that follows ear and silk growths every day for hundreds of plants, based on an ear detection algorithm that drives a robotized camera for obtaining detailed images of ears and silks. We first select, among 12 whole-plant side views, those best suited for detecting ear position. Images are segmented, the stem pixels are labelled and the ear position is identified based on changes in width along the stem. A mobile camera is then automatically positioned in real time at 30 cm from the ear, for a detailed picture in which silks are identified based on texture and colour. This allows analysis of the time course of ear and silk growths of thousands of plants. The pipeline was tested on a panel of 60 maize hybrids in the PHENOARCH phenotyping platform. Over 360 plants, ear position was correctly estimated in 86% of cases, before it could be visually assessed. Silk growth rate, estimated on all plants, decreased with time consistent with literature. The pipeline allowed clear identification of the effects of genotypes and water deficit on the rate and duration of silk growth.

CONCLUSIONS

The pipeline presented here, which combines computer vision, machine learning and robotics, provides a powerful tool for large-scale genetic analyses of the control of reproductive growth to changes in environmental conditions in a non-invasive and automatized way. It is available as Open Source software in the OpenAlea platform.