Assessment of Strawberry Ripeness Using Hyperspectral Imaging

SLFCNet: an ultra-lightweight and efficient strawberry feature classification network

Background

As modern agricultural technology advances, the automated detection, classification, and harvesting of strawberries have become an inevitable trend. Among these tasks, the classification of strawberries stands as a pivotal juncture. Nevertheless, existing object detection methods struggle with substantial computational demands, high resource utilization, and reduced detection efficiency. These challenges make deployment on edge devices difficult and lead to suboptimal user experiences.

Methods

In this study, we have developed a lightweight model capable of real-time detection and classification of strawberry fruit, named the Strawberry Lightweight Feature Classify Network (SLFCNet). This innovative system incorporates a lightweight encoder and a self-designed feature extraction module called the Combined Convolutional Concatenation and Sequential Convolutional (C3SC). While maintaining model compactness, this architecture significantly enhances its feature decoding capabilities. To evaluate the model's generalization potential, we utilized a high-resolution strawberry dataset collected directly from the fields. By employing image augmentation techniques, we conducted experimental comparisons between manually counted data and the model's inference-based detection and classification results.

Results

The SLFCNet model achieves an average precision of 98.9% in the mAP@0.5 metric, with a precision rate of 94.7% and a recall rate of 93.2%. Notably, SLFCNet features a streamlined design, resulting in a compact model size of only 3.57 MB. On an economical GTX 1080 Ti GPU, the processing time per image is a mere 4.1 ms. This indicates that the model can smoothly run on edge devices, ensuring real-time performance. Thus, it emerges as a novel solution for the automation and management of strawberry harvesting, providing real-time performance and presenting a new solution for the automatic management of strawberry picking.

Blackberry Fruit Classification in Underexposed Images Combining Deep Learning and Image Fusion Methods

Berry production is increasing worldwide each year; however, high production leads to labor shortages and an increase in wasted fruit during harvest seasons. This problem opened new research opportunities in computer vision as one main challenge to address is the uncontrolled light conditions in greenhouses and open fields. The high light variations between zones can lead to underexposure of the regions of interest, making it difficult to classify between vegetation, ripe, and unripe blackberries due to their black color. Therefore, the aim of this work is to automate the process of classifying the ripeness stages of blackberries in normal and low-light conditions by exploring the use of image fusion methods to improve the quality of the input image before the inference process. The proposed algorithm adds information from three sources: visible, an improved version of the visible, and a sensor that captures images in the near-infrared spectra, obtaining a mean F1 score of 0.909±0.074 and 0.962±0.028 in underexposed images, without and with model fine-tuning, respectively, which in some cases is an increase of up to 12% in the classification rates. Furthermore, the analysis of the fusion metrics showed that the method could be used in outdoor images to enhance their quality; the weighted fusion helps to improve only underexposed vegetation, improving the contrast of objects in the image without significant changes in saturation and colorfulness.

A data fusion approach for nondestructive tracking of the ripening process and quality attributes of green Hayward kiwifruit using artificial olfaction and proximal hyperspectral imaging techniques

A data fusion strategy based on hyperspectral imaging (HSI) and electronic nose (e-nose) systems was developed in this study to inspect the postharvest ripening process of Hayward kiwifruit. The extracted features from the e-nose and HSI techniques, in single or combined mode, were used to develop machine learning algorithms. Performance evaluations proved that the fusion of olfactory and reflectance data improves the performance of discriminative and predictive algorithms. Accordingly, with high classification accuracies of 100% and 94.44% in the calibration and test stages, the data fusion-based support vector machine (SVM) outperformed the partial least square discriminant analysis (PLSDA) for discriminating the kiwifruit samples into eight classes based on storage time. Moreover, the data fusion-based support vector regression (SVR) was a better predictor than partial least squares regression (PLSR) for kiwifruit firmness, soluble solids content (SSC), and titratable acidity (TA) measures. The prediction R 2 and RMSE criteria of the SVR algorithm on the test data were 0.962 and 0.408 for firmness, 0.964 and 0.337 for SSC, and 0.955 and 0.039 for TA, respectively. It was concluded that the hybrid of e-nose and HSI systems coupled with the SVM algorithm delivers an effective tool for accurate and nondestructive monitoring of kiwifruit quality during storage.

Research on Defect Detection in Kubo Peach Based on Hyperspectral Imaging Technology Combined with CARS-MIV-GA-SVM Method

Due to the dark red surface of ripe fresh peaches, their internal injury defects cannot be detected using the naked eye and conventional images. The rapid and accurate detection of fresh peach defects can improve the efficiency of fresh peach classification. The goal of this paper was to develop a nondestructive approach to simultaneously detecting internal injury defects and external injuries in fresh peaches. First, we collected spectral data from 347 Kubo peach samples using hyperspectral imaging technology (900-1700 nm) and carried out pretreatment. Four methods (the competitive adaptive reweighting algorithm (CARS), the combination of CARS and the average influence value algorithm (CARS-MIV), the combination of CARS and the successive projections algorithm (CARS-SPA), and the combination of CARS and uninformative variable elimination (CARS-UVE)) were used to extract the characteristic wavelength. Based on the characteristic wavelength extracted using the above methods, a genetic algorithm optimization support vector machine (GA-SVM) model and a least-squares support vector machine (LS-SVM) model were used to establish classification models. The results show that the combination of CARS and other feature wavelength extraction methods can effectively improve the prediction accuracy of the model when the number of wavelengths is small. Among them, the discriminant accuracy of the CARS-MIV-GA-SVM model reaches 93.15%. In summary, hyperspectral imaging technology can accomplish the accurate detection of Kubo peaches defects, and provides feasible ideas for the automatic classification of Kubo peaches.

Real-Time Detection of Strawberry Ripeness Using Augmented Reality and Deep Learning

Currently, strawberry harvesting relies heavily on human labour and subjective assessments of ripeness, resulting in inconsistent post-harvest quality. Therefore, the aim of this work is to automate this process and provide a more accurate and efficient way of assessing ripeness. We explored a unique combination of YOLOv7 object detection and augmented reality technology to detect and visualise the ripeness of strawberries. Our results showed that the proposed YOLOv7 object detection model, which employed transfer learning, fine-tuning and multi-scale training, accurately identified the level of ripeness of each strawberry with an mAP of 0.89 and an F1 score of 0.92. The tiny models have an average detection time of 18 ms per frame at a resolution of 1280 × 720 using a high-performance computer, thereby enabling real-time detection in the field. Our findings distinctly establish the superior performance of YOLOv7 when compared to other cutting-edge methodologies. We also suggest using Microsoft HoloLens 2 to overlay predicted ripeness labels onto each strawberry in the real world, providing a visual representation of the ripeness level. Despite some challenges, this work highlights the potential of augmented reality to assist farmers in harvesting support, which could have significant implications for current agricultural practices.

A fine recognition method of strawberry ripeness combining Mask R-CNN and region segmentation

As a fruit with high economic value, strawberry has a short ripeness period, and harvesting at an incorrect time will seriously affect the quality of strawberries, thereby reducing economic benefits. Therefore, the timing of its harvesting is very demanding. A fine ripeness recognition can provide more accurate crop information, and guide strawberry harvest management more timely and effectively. This study proposes a fine recognition method for field strawberry ripeness that combines deep learning and image processing. The method is divided into three stages: In the first stage, self-calibrated convolutions are added to the Mask R-CNN backbone network to improve the model performance, and then the model is used to extract the strawberry target in the image. In the second stage, the strawberry target is divided into four sub-regions by region segmentation method, and the color feature values of B, G, L, a and S channels are extracted for each sub-region. In the third stage, the strawberry ripeness is classified according to the color feature values and the results are visualized. Experimental results show that with the incorporation of self-calibrated convolutions into the Mask R-CNN, the model's performance has been substantially enhanced, leading to increased robustness against diverse occlusion interferences. As a result, the final average precision (AP) has improved to 0.937, representing a significant increase of 0.039 compared to the previous version. The strawberry ripeness classification effect is the best on the SVM classifier, and the accuracy under the combined channel BGLaS reaches 0.866. The classification results are better than common manual feature extraction methods and AlexNet, ResNet18 models. In order to clarify the role of the region segmentation method, the contribution of different sub-regions to each ripeness is also explored. The comprehensive results demonstrate that the proposed method enables the evaluation of six distinct ripeness levels of strawberries in the complex field environment. This method can provide accurate decision support for strawberry refined planting management.

A Self-Supervised Anomaly Detector of Fruits Based on Hyperspectral Imaging

Hyperspectral imaging combined with chemometric approaches is proven to be a powerful tool for the quality evaluation and control of fruits. In fruit defect-detection scenarios, developing an unsupervised anomaly detection framework is vital, as defect sample preparation is labor-intensive and time-consuming, especially for exploring potential defects. In this paper, a spectral-spatial, information-based, self-supervised anomaly detection (SSAD) approach is proposed. During training, an auxiliary classifier is proposed to identify the projection axes of principal component (PC) images that were transformed from the hyperspectral data cubes. In test time, the fully connected layer of the learned classifier was used as a 'spectral-spatial' feature extractor, and the feature similarity metric was adopted as the score function for the downstream anomaly evaluation task. The proposed network was evaluated with two fruit data sets: a strawberry data set with bruised, infected, chilling-injured, and contaminated test samples and a blueberry data set with bruised, infected, chilling-injured, and wrinkled samples as anomalies. The results show that the SSAD yielded the best anomaly detection performance (AUC = 0.923 on average) over the baseline methods, and the visualization results further confirmed its advantage in extracting effective 'spectral-spatial' latent representation. Moreover, the robustness of SSAD is verified with the data pollution experiment; it performed significantly better than the baselines when a portion of anomalous samples was involved in the training process.

Image-Based High-Throughput Phenotyping in Horticultural Crops

Plant phenotyping is the primary task of any plant breeding program, and accurate measurement of plant traits is essential to select genotypes with better quality, high yield, and climate resilience. The majority of currently used phenotyping techniques are destructive and time-consuming. Recently, the development of various sensors and imaging platforms for rapid and efficient quantitative measurement of plant traits has become the mainstream approach in plant phenotyping studies. Here, we reviewed the trends of image-based high-throughput phenotyping methods applied to horticultural crops. High-throughput phenotyping is carried out using various types of imaging platforms developed for indoor or field conditions. We highlighted the applications of different imaging platforms in the horticulture sector with their advantages and limitations. Furthermore, the principles and applications of commonly used imaging techniques, visible light (RGB) imaging, thermal imaging, chlorophyll fluorescence, hyperspectral imaging, and tomographic imaging for high-throughput plant phenotyping, are discussed. High-throughput phenotyping has been widely used for phenotyping various horticultural traits, which can be morphological, physiological, biochemical, yield, biotic, and abiotic stress responses. Moreover, the ability of high-throughput phenotyping with the help of various optical sensors will lead to the discovery of new phenotypic traits which need to be explored in the future. We summarized the applications of image analysis for the quantitative evaluation of various traits with several examples of horticultural crops in the literature. Finally, we summarized the current trend of high-throughput phenotyping in horticultural crops and highlighted future perspectives.

"How sweet are your strawberries?": Predicting sugariness using non-destructive and affordable hardware

Global soft fruit supply chains rely on trustworthy descriptions of product quality. However, crucial criteria such as sweetness and firmness cannot be accurately established without destroying the fruit. Since traditional alternatives are subjective assessments by human experts, it is desirable to obtain quality estimations in a consistent and non-destructive manner. The majority of research on fruit quality measurements analyzed fruits in the lab with uniform data collection. However, it is laborious and expensive to scale up to the level of the whole yield. The "harvest-first, analysis-second" method also comes too late to decide to adjust harvesting schedules. In this research, we validated our hypothesis of using in-field data acquirable via commodity hardware to obtain acceptable accuracies. The primary instance that the research concerns is the sugariness of strawberries, described by the juice's total soluble solid (TSS) content (unit: °Brix or Brix). We benchmarked the accuracy of strawberry Brix prediction using convolutional neural networks (CNN), variational autoencoders (VAE), principal component analysis (PCA), kernelized ridge regression (KRR), support vector regression (SVR), and multilayer perceptron (MLP), based on fusions of image data, environmental records, and plant load information, etc. Our results suggest that: (i) models trained by environment and plant load data can perform reliable prediction of aggregated Brix values, with the lowest RMSE at 0.59; (ii) using image data can further supplement the Brix predictions of individual fruits from (i), from 1.27 to as low up to 1.10, but they by themselves are not sufficiently reliable.

Assessing the Nutritional-Value-Based Therapeutic Potentials and Non-Destructive Approaches for Mulberry Fruit Assessment: An Overview

Among different fruits, mulberry is the most highlighted natural gift in its superior nutritional and bioactive composition, indispensable for continuing a healthy life. It also acts as a hepatoprotective immunostimulator and improves vision, anti-microbial, anti-cancer agent, anti-stress activity, atherosclerosis, neuroprotective functions, and anti-obesity action. The mulberry fruits also help reduce neurological disorders and mental illness. The main reason for that is the therapeutic potentials present in the nutritional components of the mulberry fruit. The available methods for assessing mulberry fruits are mainly chromatographic based, which are destructive and possess many limitations. However, recently some non-invasive techniques, including chlorophyll fluorescence, image processing, and hyperspectral imaging, were employed to detect various mulberry fruit attributes. The present review attempts to collect and explore available information regarding the nutritional and medicinal importance of mulberry fruit. Besides, non-destructive methods established for the fruit are also elaborated. This work helps encourage many more research works to dug out more hidden information about the essential nutrition of mulberry that can be helpful to resolve many mental-illness-related issues.

Application of Hyperspectral Imaging for Maturity and Soluble Solids Content Determination of Strawberry With Deep Learning Approaches

Maturity degree and quality evaluation are important for strawberry harvest, trade, and consumption. Deep learning has been an efficient artificial intelligence tool for food and agro-products. Hyperspectral imaging coupled with deep learning was applied to determine the maturity degree and soluble solids content (SSC) of strawberries with four maturity degrees. Hyperspectral image of each strawberry was obtained and preprocessed, and the spectra were extracted from the images. One-dimension residual neural network (1D ResNet) and three-dimension (3D) ResNet were built using 1D spectra and 3D hyperspectral image as inputs for maturity degree evaluation. Good performances were obtained for maturity identification, with the classification accuracy over 84% for both 1D ResNet and 3D ResNet. The corresponding saliency maps showed that the pigments related wavelengths and image regions contributed more to the maturity identification. For SSC determination, 1D ResNet model was also built, with the determination of coefficient (R ²) over 0.55 of the training, validation, and testing sets. The saliency maps of 1D ResNet for the SSC determination were also explored. The overall results showed that deep learning could be used to identify strawberry maturity degree and determine SSC. More efforts were needed to explore the use of 3D deep learning methods for the SSC determination. The close results of 1D ResNet and 3D ResNet for classification indicated that more samples might be used to improve the performances of 3D ResNet. The results in this study would help to develop 1D and 3D deep learning models for fruit quality inspection and other researches using hyperspectral imaging, providing efficient analysis approaches of fruit quality inspection using hyperspectral imaging.