OSC-CO2: coattention and cosegmentation framework for plant state change with multiple features

Cosegmentation and coattention are extensions of traditional segmentation methods aimed at detecting a common object (or objects) in a group of images. Current cosegmentation and coattention methods are ineffective for objects, such as plants, that change their morphological state while being captured in different modalities and views. The Object State Change using Coattention-Cosegmentation (OSC-CO2) is an end-to-end unsupervised deep-learning framework that enhances traditional segmentation techniques, processing, analyzing, selecting, and combining suitable segmentation results that may contain most of our target object's pixels, and then displaying a final segmented image. The framework leverages coattention-based convolutional neural networks (CNNs) and cosegmentation-based dense Conditional Random Fields (CRFs) to address segmentation accuracy in high-dimensional plant imagery with evolving plant objects. The efficacy of OSC-CO2 is demonstrated using plant growth sequences imaged with infrared, visible, and fluorescence cameras in multiple views using a remote sensing, high-throughput phenotyping platform, and is evaluated using Jaccard index and precision measures. We also introduce CosegPP+, a dataset that is structured and can provide quantitative information on the efficacy of our framework. Results show that OSC-CO2 out performed state-of-the art segmentation and cosegmentation methods by improving segementation accuracy by 3% to 45%.

Pitfalls and potential of high-throughput plant phenotyping platforms

Automated high-throughput plant phenotyping (HTPP) enables non-invasive, fast and standardized evaluations of a large number of plants for size, development, and certain physiological variables. Many research groups recognize the potential of HTPP and have made significant investments in HTPP infrastructure, or are considering doing so. To make optimal use of limited resources, it is important to plan and use these facilities prudently and to interpret the results carefully. Here we present a number of points that users should consider before purchasing, building or utilizing such equipment. They relate to (1) the financial and time investment for acquisition, operation, and maintenance, (2) the constraints associated with such machines in terms of flexibility and growth conditions, (3) the pros and cons of frequent non-destructive measurements, (4) the level of information provided by proxy traits, and (5) the utilization of calibration curves. Using data from an Arabidopsis experiment, we demonstrate how diurnal changes in leaf angle can impact plant size estimates from top-view cameras, causing deviations of more than 20% over the day. Growth analysis data from another rosette species showed that there was a curvilinear relationship between total and projected leaf area. Neglecting this curvilinearity resulted in linear calibration curves that, although having a high r2 (> 0.92), also exhibited large relative errors. Another important consideration we discussed is the frequency at which calibration curves need to be generated and whether different treatments, seasons, or genotypes require distinct calibration curves. In conclusion, HTPP systems have become a valuable addition to the toolbox of plant biologists, provided that these systems are tailored to the research questions of interest, and users are aware of both the possible pitfalls and potential involved.

Exploring plant responses to abiotic stress by contrasting spectral signature changes

In this study, daily changes over a short period and diurnal progression of spectral reflectance at the leaf level were used to identify spring wheat genotypes (Triticum aestivum L.) susceptible to adverse conditions. Four genotypes were grown in pots experiments under semi-controlled conditions in Chile and Spain. Three treatments were applied: i) control (C), ii) water stress (WS), and iii) combined water and heat shock (WS+T). Spectral reflectance, gas exchange and chlorophyll fluorescence measurements were performed on flag leaves for three consecutive days at anthesis. High canopy temperature ( H _CT ) genotypes showed less variability in their mean spectral reflectance signature and chlorophyll fluorescence, which was related to weaker responses to environmental fluctuations. While low canopy temperature ( L _CT ) genotypes showed greater variability. The genotypes spectral signature changes, in accordance with environmental fluctuation, were associated with variations in their stomatal conductance under both stress conditions (WS and WS+T); L _CT genotypes showed an anisohydric response compared that of H _CT , which was isohydric. This approach could be used in breeding programs for screening a large number of genotypes through proximal or remote sensing tools and be a novel but simple way to identify groups of genotypes with contrasting performances.

High-throughput phenotyping of physiological traits for wheat resilience to high temperature and drought stress

Interannual and local fluctuations in wheat crop yield are mostly explained by abiotic constraints. Heatwaves and drought, which are among the top stressors, commonly co-occur, and their frequency is increasing with global climate change. High-throughput methods were optimized to phenotype wheat plants under controlled water deficit and high temperature, with the aim to identify phenotypic traits conferring adaptative stress responses. Wheat plants of 10 genotypes were grown in a fully automated plant facility under 25/18 °C day/night for 30 d, and then the temperature was increased for 7 d (38/31 °C day/night) while maintaining half of the plants well irrigated and half at 30% field capacity. Thermal and multispectral images and pot weights were registered twice daily. At the end of the experiment, key metabolites and enzyme activities from carbohydrate and antioxidant metabolism were quantified. Regression machine learning models were successfully established to predict plant biomass using image-extracted parameters. Evapotranspiration traits expressed significant genotype-environment interactions (G×E) when acclimatization to stress was continuously monitored. Consequently, transpiration efficiency was essential to maintain the balance between water-saving strategies and biomass production in wheat under water deficit and high temperature. Stress tolerance included changes in carbohydrate metabolism, particularly in the sucrolytic and glycolytic pathways, and in antioxidant metabolism. The observed genetic differences in sensitivity to high temperature and water deficit can be exploited in breeding programmes to improve wheat resilience to climate change.

Automated hyperspectral vegetation index derivation using a hyperparameter optimisation framework for high-throughput plant phenotyping

Hyperspectral vegetation indices (VIs) are widely deployed in agriculture remote sensing and plant phenotyping to estimate plant biophysical and biochemical traits. However, existing VIs consist mainly of simple two-band indices that limit the net performance and often do not generalise well for traits other than those for which they were originally designed. We present an automated hyperspectral vegetation index (AutoVI) system for the rapid generation of novel two- to six-band trait-specific indices in a streamlined process covering model selection, optimisation and evaluation, driven by the tree parzen estimator algorithm. Its performance was tested in generating novel indices to estimate chlorophyll and sugar contents in wheat. Results showed that AutoVI can rapidly generate complex novel VIs (at least a four-band index) that correlated strongly (R²  > 0.8) with measured chlorophyll and sugar contents in wheat. Automated hyperspectral vegetation index-derived indices were used as features in simple and stepwise multiple linear regressions for chlorophyll and sugar content estimation, and outperformed the results achieved with the existing 47 VIs and those provided using partial least squares regression. The AutoVI system can deliver novel trait-specific VIs readily adoptable to high-throughput plant phenotyping platforms and should appeal to plant scientists and breeders. A graphical user interface for the AutoVI is provided here.

CBM: An IoT Enabled LiDAR Sensor for In-Field Crop Height and Biomass Measurements

The phenotypic characterization of crop genotypes is an essential, yet challenging, aspect of crop management and agriculture research. Digital sensing technologies are rapidly advancing plant phenotyping and speeding-up crop breeding outcomes. However, off-the-shelf sensors might not be fully applicable and suitable for agricultural research due to the diversity in crop species and specific needs during plant breeding selections. Customized sensing systems with specialized sensor hardware and software architecture provide a powerful and low-cost solution. This study designed and developed a fully integrated Raspberry Pi-based LiDAR sensor named CropBioMass (CBM), enabled by internet of things to provide a complete end-to-end pipeline. The CBM is a low-cost sensor, provides high-throughput seamless data collection in field, small data footprint, injection of data onto the remote server, and automated data processing. The phenotypic traits of crop fresh biomass, dry biomass, and plant height that were estimated by CBM data had high correlation with ground truth manual measurements in a wheat field trial. The CBM is readily applicable for high-throughput plant phenotyping, crop monitoring, and management for precision agricultural applications.

Robotic Technologies for High-Throughput Plant Phenotyping: Contemporary Reviews and Future Perspectives

Phenotyping plants is an essential component of any effort to develop new crop varieties. As plant breeders seek to increase crop productivity and produce more food for the future, the amount of phenotype information they require will also increase. Traditional plant phenotyping relying on manual measurement is laborious, time-consuming, error-prone, and costly. Plant phenotyping robots have emerged as a high-throughput technology to measure morphological, chemical and physiological properties of large number of plants. Several robotic systems have been developed to fulfill different phenotyping missions. In particular, robotic phenotyping has the potential to enable efficient monitoring of changes in plant traits over time in both controlled environments and in the field. The operation of these robots can be challenging as a result of the dynamic nature of plants and the agricultural environments. Here we discuss developments in phenotyping robots, and the challenges which have been overcome and others which remain outstanding. In addition, some perspective applications of the phenotyping robots are also presented. We optimistically anticipate that autonomous and robotic systems will make great leaps forward in the next 10 years to advance the plant phenotyping research into a new era.

UAV-Based Thermal, RGB Imaging and Gene Expression Analysis Allowed Detection of Fusarium Head Blight and Gave New Insights Into the Physiological Responses to the Disease in Durum Wheat

Wheat is one of the world's most economically important cereal crop, grown on 220 million hectares. Fusarium head blight (FHB) disease is considered a major threat to durum (Triticum turgidum subsp. durum (Desfontaines) Husnache) and bread wheat (T. aestivum L.) cultivars and is mainly managed by the application of fungicides at anthesis. However, fungicides are applied when FHB symptoms are clearly visible and the spikes are almost entirely bleached (% of diseased spikelets > 80%), by when it is too late to control FHB disease. For this reason, farmers often react by performing repeated fungicide treatments that, however, due to the advanced state of the infection, cause a waste of money and pose significant risks to the environment and non-target organisms. In the present study, we used unmanned aerial vehicle (UAV)-based thermal infrared (TIR) and red-green-blue (RGB) imaging for FHB detection in T. turgidum (cv. Marco Aurelio) under natural field conditions. TIR and RGB data coupled with ground-based measurements such as spike's temperature, photosynthetic efficiency and molecular identification of FHB pathogens, detected FHB at anthesis half-way (Zadoks stage 65, ZS 65), when the percentage (%) of diseased spikelets ranged between 20% and 60%. Moreover, in greenhouse experiments the transcripts of the key genes involved in stomatal closure were mostly up-regulated in F. graminearum-inoculated plants, demonstrating that the physiological mechanism behind the spike's temperature increase and photosynthetic efficiency decrease could be attributed to the closure of the guard cells in response to F. graminearum. In addition, preliminary analysis revealed that there is differential regulation of genes between drought-stressed and F. graminearum-inoculated plants, suggesting that there might be a possibility to discriminate between water stress and FHB infection. This study shows the potential of UAV-based TIR and RGB imaging for field phenotyping of wheat and other cereal crop species in response to environmental stresses. This is anticipated to have enormous promise for the detection of FHB disease and tremendous implications for optimizing the application of fungicides, since global food crop demand is to be met with minimal environmental impacts.

Nondestructive Determination of Nitrogen, Phosphorus and Potassium Contents in Greenhouse Tomato Plants Based on Multispectral Three-Dimensional Imaging

Measurement of plant nitrogen (N), phosphorus (P), and potassium (K) levels are important for determining precise fertilization management approaches for crops cultivated in greenhouses. To accurately, rapidly, stably, and nondestructively measure the NPK levels in tomato plants, a nondestructive determination method based on multispectral three-dimensional (3D) imaging was proposed. Multiview RGB-D images and multispectral images were synchronously collected, and the plant multispectral reflectance was registered to the depth coordinates according to Fourier transform principles. Based on the Kinect sensor pose estimation and self-calibration, the unified transformation of the multiview point cloud coordinate system was realized. Finally, the iterative closest point (ICP) algorithm was used for the precise registration of multiview point clouds and the reconstruction of plant multispectral 3D point cloud models. Using the normalized grayscale similarity coefficient, the degree of spectral overlap, and the Hausdorff distance set, the accuracy of the reconstructed multispectral 3D point clouds was quantitatively evaluated, the average value was 0.9116, 0.9343 and 0.41 cm, respectively. The results indicated that the multispectral reflectance could be registered to the Kinect depth coordinates accurately based on the Fourier transform principles, the reconstruction accuracy of the multispectral 3D point cloud model met the model reconstruction needs of tomato plants. Using back-propagation artificial neural network (BPANN), support vector machine regression (SVMR), and gaussian process regression (GPR) methods, determination models for the NPK contents in tomato plants based on the reflectance characteristics of plant multispectral 3D point cloud models were separately constructed. The relative error (RE) of the N content by BPANN, SVMR and GPR prediction models were 2.27%, 7.46% and 4.03%, respectively. The RE of the P content by BPANN, SVMR and GPR prediction models were 3.32%, 8.92% and 8.41%, respectively. The RE of the K content by BPANN, SVMR and GPR prediction models were 3.27%, 5.73% and 3.32%, respectively. These models provided highly efficient and accurate measurements of the NPK contents in tomato plants. The NPK contents determination performance of these models were more stable than those of single-view models.

Leveraging Image Analysis for High-Throughput Plant Phenotyping

The complex interaction between a genotype and its environment controls the biophysical properties of a plant, manifested in observable traits, i.e., plant's phenome, which influences resources acquisition, performance, and yield. High-throughput automated image-based plant phenotyping refers to the sensing and quantifying plant traits non-destructively by analyzing images captured at regular intervals and with precision. While phenomic research has drawn significant attention in the last decade, extracting meaningful and reliable numerical phenotypes from plant images especially by considering its individual components, e.g., leaves, stem, fruit, and flower, remains a critical bottleneck to the translation of advances of phenotyping technology into genetic insights due to various challenges including lighting variations, plant rotations, and self-occlusions. The paper provides (1) a framework for plant phenotyping in a multimodal, multi-view, time-lapsed, high-throughput imaging system; (2) a taxonomy of phenotypes that may be derived by image analysis for better understanding of morphological structure and functional processes in plants; (3) a brief discussion on publicly available datasets to encourage algorithm development and uniform comparison with the state-of-the-art methods; (4) an overview of the state-of-the-art image-based high-throughput plant phenotyping methods; and (5) open problems for the advancement of this research field.

Automated Alignment of Multi-Modal Plant Images Using Integrative Phase Correlation Approach

Modern facilities for high-throughput phenotyping provide plant scientists with a large amount of multi-modal image data. Combination of different image modalities is advantageous for image segmentation, quantitative trait derivation, and assessment of a more accurate and extended plant phenotype. However, visible light (VIS), fluorescence (FLU), and near-infrared (NIR) images taken with different cameras from different view points in different spatial resolutions exhibit not only relative geometrical transformations but also considerable structural differences that hamper a straightforward alignment and combined analysis of multi-modal image data. Conventional techniques of image registration are predominantly tailored to detection of relative geometrical transformations between two otherwise identical images, and become less accurate when applied to partially similar optical scenes. Here, we focus on a relatively new technical problem of FLU/VIS plant image registration. We present a framework for automated alignment of FLU/VIS plant images which is based on extension of the phase correlation (PC) approach - a frequency domain technique for image alignment, which relies on detection of a phase shift between two Fourier-space transforms. Primarily tailored to detection of affine image transformations between two structurally identical images, PC is known to be sensitive to structural image distortions. We investigate effects of image preprocessing and scaling on accuracy of image registration and suggest an integrative algorithmic scheme which allows to overcome shortcomings of conventional single-step PC by application to non-identical multi-modal images. Our experimental tests with FLU/VIS images of different plant species taken on different phenotyping facilities at different developmental stages, including difficult cases such as small plant shoots of non-specific shape and non-uniformly moving leaves, demonstrate improved performance of our extended PC approach within the scope of high-throughput plant phenotyping.

A Novel LiDAR-Based Instrument for High-Throughput, 3D Measurement of Morphological Traits in Maize and Sorghum

Recently, imaged-based approaches have developed rapidly for high-throughput plant phenotyping (HTPP). Imaging reduces a 3D plant into 2D images, which makes the retrieval of plant morphological traits challenging. We developed a novel LiDAR-based phenotyping instrument to generate 3D point clouds of single plants. The instrument combined a LiDAR scanner with a precision rotation stage on which an individual plant was placed. A LabVIEW program was developed to control the scanning and rotation motion, synchronize the measurements from both devices, and capture a 360° view point cloud. A data processing pipeline was developed for noise removal, voxelization, triangulation, and plant leaf surface reconstruction. Once the leaf digital surfaces were reconstructed, plant morphological traits, including individual and total leaf area, leaf inclination angle, and leaf angular distribution, were derived. The system was tested with maize and sorghum plants. The results showed that leaf area measurements by the instrument were highly correlated with the reference methods (R² > 0.91 for individual leaf area; R² > 0.95 for total leaf area of each plant). Leaf angular distributions of the two species were also derived. This instrument could fill a critical technological gap for indoor HTPP of plant morphological traits in 3D.