LeTra: a leaf tracking workflow based on convolutional neural networks and intersection over union

BACKGROUND

The study of plant photosynthesis is essential for productivity and yield. Thanks to the development of high-throughput phenotyping (HTP) facilities, based on chlorophyll fluorescence imaging, photosynthetic traits can be measured in a reliable, reproducible and efficient manner. In most state-of-the-art HTP platforms, these traits are automatedly analyzed at individual plant level, but information at leaf level is often restricted by the use of manual annotation. Automated leaf tracking over time is therefore highly desired. Methods for tracking individual leaves are still uncommon, convoluted, or require large datasets. Hence, applications and libraries with different techniques are required. New phenotyping platforms are initiated now more frequently than ever; however, the application of advanced computer vision techniques, such as convolutional neural networks, is still growing at a slow pace. Here, we provide a method for leaf segmentation and tracking through the fine-tuning of Mask R-CNN and intersection over union as a solution for leaf tracking on top-down images of plants. We also provide datasets and code for training and testing on both detection and tracking of individual leaves, aiming to stimulate the community to expand the current methodologies on this topic.

RESULTS

We tested the results for detection and segmentation on 523 Arabidopsis thaliana leaves at three different stages of development from which we obtained a mean F-score of 0.956 on detection and 0.844 on segmentation overlap through the intersection over union (IoU). On the tracking side, we tested nine different plants with 191 leaves. A total of 161 leaves were tracked without issues, accounting to a total of 84.29% correct tracking, and a Higher Order Tracking Accuracy (HOTA) of 0.846. In our case study, leaf age and leaf order influenced photosynthetic capacity and photosynthetic response to light treatments. Leaf-dependent photosynthesis varies according to the genetic background.

CONCLUSION

The method provided is robust for leaf tracking on top-down images. Although one of the strong components of the method is the low requirement in training data to achieve a good base result (based on fine-tuning), most of the tracking issues found could be solved by expanding the training dataset for the Mask R-CNN model.

Effect of varying UAV height on the precise estimation of potato crop growth

A phenotyping pipeline utilising DeepLab was developed for precisely estimating the height, volume, coverage and vegetation indices of European and Japanese varieties. Using this pipeline, the effect of varying UAV height on the precise estimation of potato crop growth properties was evaluated. A UAV fitted with a multispectral camera was flown at a height of 15 m and 30 m in an experimental field where various varieties of potatoes were grown. The properties of plant height, volume and NDVI were evaluated and compared with the manually obtained parameters. Strong linear correlations with R2 of 0.803 and 0.745 were obtained between the UAV obtained plant heights and manually estimated plant height when the UAV was flown at 15 m and 30 m respectively. Furthermore, high linear correlations with an R2 of 0.839 and 0.754 were obtained between the UAV-estimated volume and manually estimated volume when the UAV was flown at 15 m and 30 m respectively. For the vegetation indices, there were no observable differences in the NDVI values obtained from the UAV flown at the two heights. Furthermore, high linear correlations with R2 of 0.930 and 0.931 were obtained between UAV-estimated and manually measured NDVI at 15 m and 30 m respectively. It was found that UAV flown at the lower height had a higher ground sampling distance thus increased resolution leading to more precise estimation of both the height and volume of crops. For vegetation indices, flying the UAV at a higher height had no effect on the precision of NDVI estimates.

Insect exuviae as soil amendment affect flower reflectance and increase flower production and plant volatile emission

Soil composition and herbivory are two environmental factors that can affect plant traits including flower traits, thus potentially affecting plant-pollinator interactions. Importantly, soil composition and herbivory may interact in these effects, with consequences for plant fitness. We assessed the main effects of aboveground insect herbivory and soil amendment with exuviae of three different insect species on visual and olfactory traits of Brassica nigra plants, including interactive effects. We combined various methodological approaches including gas chromatography/mass spectrometry, spectroscopy and machine learning to evaluate changes in flower morphology, colour and the emission of volatile organic compounds (VOCs). Soil amended with insect exuviae increased the total number of flowers per plant and VOC emission, whereas herbivory reduced petal area and VOC emission. Soil amendment and herbivory interacted in their effect on the floral reflectance spectrum of the base part of petals and the emission of 10 VOCs. These findings demonstrate the effects of insect exuviae as soil amendment on plant traits involved in reproduction, with a potential for enhanced reproductive success by increasing the strength of signals attracting pollinators and by mitigating the negative effects of herbivory.

All-in-one: A spectral imaging laboratory system for standardised automated image acquisition and real-time spectral model deployment

Spectral imaging (SI) in analytical chemistry is widely used for the assessment of spatially distributed physicochemical properties of samples. Although massive development in instrument and chemometrics modelling has taken place in the recent years, the main challenge with SI is that available sensors require extensive system integration and calibration modelling before their use for routine analysis. Further, the models developed during one experiment are rarely useful once the system is reintegrated for a new experiment. To avoid system reintegration and reuse calibrated models, this study presents an intelligent All-In-One SI (ASI) laboratory system allowing standardised automated data acquisition and real-time spectral model deployment. The ASI system supplies a controlled standardised illumination environment, an in-built computing system, embedded software for automated image acquisition, and model deployment to predict the spatial distribution of sample properties in real-time. To show the capability of the ASI framework, exemplary cases of fruit property prediction in different fruits are presented. Furthermore, ASI is also benchmarked in performance against the current commercially available portable as well as high-end laboratory spectrometers.

Tomato Fruit Detection and Counting in Greenhouses Using Deep Learning

Accurately detecting and counting fruits during plant growth using imaging and computer vision is of importance not only from the point of view of reducing labor intensive manual measurements of phenotypic information, but also because it is a critical step toward automating processes such as harvesting. Deep learning based methods have emerged as the state-of-the-art techniques in many problems in image segmentation and classification, and have a lot of promise in challenging domains such as agriculture, where they can deal with the large variability in data better than classical computer vision methods. This paper reports results on the detection of tomatoes in images taken in a greenhouse, using the MaskRCNN algorithm, which detects objects and also the pixels corresponding to each object. Our experimental results on the detection of tomatoes from images taken in greenhouses using a RealSense camera are comparable to or better than the metrics reported by earlier work, even though those were obtained in laboratory conditions or using higher resolution images. Our results also show that MaskRCNN can implicitly learn object depth, which is necessary for background elimination.

Close Range Spectral Imaging for Disease Detection in Plants Using Autonomous Platforms: a Review on Recent Studies

Purpose of Review

A short introduction to the spectral imaging (SI) of plants along with a comprehensive overview of the recent research works related to disease detection in plants using autonomous phenotyping platforms is provided. Key benefits and challenges of SI for plant disease detection on robotic platforms are highlighted.

Recent Findings

SI is becoming a potential tool for autonomous platforms for non-destructive plant assessment. This is because it can provide information on the plant pigments such as chlorophylls, anthocyanins and carotenoids and supports quantification of biochemical parameters such as sugars, proteins, different nutrients, water and fat content. A plant suffering from diseases will exhibit different physicochemical parameters compared with a healthy plant, allowing the SI to capture those differences as a function of reflected or absorbed light.

Summary

Potential of SI to non-destructively capture physicochemical parameters in plants makes it a key technique to support disease detection on autonomous platforms. SI can be broadly used for crop disease detection by quantification of physicochemical changes in the plants.

Floral plasticity: Herbivore-species-specific-induced changes in flower traits with contrasting effects on pollinator visitation

Plant phenotypic plasticity in response to antagonists can affect other community members such as mutualists, conferring potential ecological costs associated with inducible plant defence. For flowering plants, induction of defences to deal with herbivores can lead to disruption of plant-pollinator interactions. Current knowledge on the full extent of herbivore-induced changes in flower traits is limited, and we know little about specificity of induction of flower traits and specificity of effect on flower visitors. We exposed flowering Brassica nigra plants to six insect herbivore species and recorded changes in flower traits (flower abundance, morphology, colour, volatile emission, nectar quantity, and pollen quantity and size) and the behaviour of two pollinating insects. Our results show that herbivory can affect multiple flower traits and pollinator behaviour. Most plastic floral traits were flower morphology, colour, the composition of the volatile blend, and nectar production. Herbivore-induced changes in flower traits resulted in positive, negative, or neutral effects on pollinator behaviour. Effects on flower traits and pollinator behaviour were herbivore species-specific. Flowers show extensive plasticity in response to antagonist herbivores, with contrasting effects on mutualist pollinators. Antagonists can potentially act as agents of selection on flower traits and plant reproduction via plant-mediated interactions with mutualists.

Potato Virus Y Detection in Seed Potatoes Using Deep Learning on Hyperspectral Images

Virus diseases are of high concern in the cultivation of seed potatoes. Once found in the field, virus diseased plants lead to declassification or even rejection of the seed lots resulting in a financial loss. Farmers put in a lot of effort to detect diseased plants and remove virus-diseased plants from the field. Nevertheless, dependent on the cultivar, virus diseased plants can be missed during visual observations in particular in an early stage of cultivation. Therefore, there is a need for fast and objective disease detection. Early detection of diseased plants with modern vision techniques can significantly reduce costs. Laboratory experiments in previous years showed that hyperspectral imaging clearly could distinguish healthy from virus infected potato plants. This paper reports on our first real field experiment. A new imaging setup was designed, consisting of a hyperspectral line-scan camera. Hyperspectral images were taken in the field with a line interval of 5 mm. A fully convolutional neural network was adapted for hyperspectral images and trained on two experimental rows in the field. The trained network was validated on two other rows, with different potato cultivars. For three of the four row/date combinations the precision and recall compared to conventional disease assessment exceeded 0.78 and 0.88, respectively. This proves the suitability of this method for real world disease detection.

Hyperspectral Sensors and Imaging Technologies in Phytopathology: State of the Art

Plant disease detection represents a tremendous challenge for research and practical applications. Visual assessment by human raters is time-consuming, expensive, and error prone. Disease rating and plant protection need new and innovative techniques to address forthcoming challenges and trends in agricultural production that require more precision than ever before. Within this context, hyperspectral sensors and imaging techniques-intrinsically tied to efficient data analysis approaches-have shown an enormous potential to provide new insights into plant-pathogen interactions and for the detection of plant diseases. This article provides an overview of hyperspectral sensors and imaging technologies for assessing compatible and incompatible plant-pathogen interactions. Within the progress of digital technologies, the vision, which is increasingly discussed in the society and industry, includes smart and intuitive solutions for assessing plant features in plant phenotyping or for making decisions on plant protection measures in the context of precision agriculture.

Metabolomics meets functional assays: coupling LC-MS and microfluidic cell-based receptor-ligand analyses

INTRODUCTION

Metabolomics has become a valuable tool in many research areas. However, generating metabolomics-based biochemical profiles without any related bioactivity is only of indirect value in understanding a biological process. Therefore, metabolomics research could greatly benefit from tools that directly determine the bioactivity of the detected compounds.

OBJECTIVE

We aimed to combine LC-MS metabolomics with a cell based receptor assay. This combination could increase the understanding of biological processes and may provide novel opportunities for functional metabolomics.

METHODS

We developed a flow through biosensor with human cells expressing both the TRPV1, a calcium ion channel which responds to capsaicin, and the fluorescent intracellular calcium ion reporter, YC3.6. We have analysed three contrasting Capsicum varieties. Two were selected with contrasting degrees of spiciness for characterization by HPLC coupled to high mass resolution MS. Subsequently, the biosensor was then used to link individual pepper compounds with TRPV1 activity.

RESULTS

Among the compounds in the crude pepper fruit extracts, we confirmed capsaicin and also identified both nordihydrocapsaicin and dihydrocapsaicin as true agonists of the TRPV1 receptor. Furthermore, the biosensor was able to detect receptor activity in extracts of both Capsicum fruits as well as a commercial product. Sensitivity of the biosensor to this commercial product was similar to the sensory threshold of a human sensory panel.

CONCLUSION

Our results demonstrate that the TRPV1 biosensor is suitable for detecting bioactive metabolites. Novel opportunities may lie in the development of a continuous functional assay, where the biosensor is directly coupled to the LC-MS.

Automated estimation of leaf area development in sweet pepper plants from image analysis

High-throughput automated plant phenotyping has recently received a lot of attention. Leaf area is an important characteristic in understanding plant performance, but time-consuming and destructive to measure accurately. In this research, we describe a method to use a histogram of image intensities to automatically measure plant leaf area of tall pepper (Capsicum annuum L.) plants in the greenhouse. With a device equipped with several cameras, images of plants were recorded at 5-cm intervals over a height of 3m, at a recording distance of less than 60cm. The images were reduced to a small set of principal components that defined the design matrix in a regression model for predicting manually measured leaf area as obtained from destructive harvesting. These regression calibrations were performed for six different developmental times. In addition, development of leaf area was investigated by fitting linear relations between predicted leaf area and time, with special attention given to the genotype by time interaction and its genetic basis in the form of quantitative trait loci (QTLs). The experiment comprised parents, F1 progeny and eight genotypes of a recombinant inbred population of pepper. Although the current trial contained a limited number of genotypes, an earlier identified QTL related to leaf area growth could be confirmed. Therefore, image analysis, as presented in this paper, provides a powerful and efficient way to study and identify the genetic basis of growth and developmental processes in plants.

SPICY: towards automated phenotyping of large pepper plants in the greenhouse

Most high-throughput systems for automated plant phenotyping involve a fixed recording cabinet to which plants are transported. However, important greenhouse plants like pepper are too tall to be transported. In this research we developed a system to automatically measure plant characteristics of tall pepper plants in the greenhouse. With a device equipped with multiple cameras, images of plants are recorded at a 5cm interval over a height of 3m. Two types of features are extracted: (1) features from a 3D reconstruction of the plant canopy; and (2) statistical features derived directly from RGB images. The experiment comprised 151 genotypes of a recombinant inbred population of pepper, to examine the heritability and quantitative trait loci (QTL) of the features. Features extracted from the 3D reconstruction of the canopy were leaf size and leaf angle, with heritabilities of 0.70 and 0.56 respectively. Three QTL were found for leaf size, and one for leaf angle. From the statistical features, plant height showed a good correlation (0.93) with manual measurements, and QTL were in accordance with QTL of manual measurements. For total leaf area, the heritability was 0.55, and two of the three QTL found by manual measurement were found by image analysis.