Comparing visible and near infrared 'point' spectroscopy and hyperspectral imaging techniques to visualize the variability of apple firmness

In this work, visible and near-infrared 'point' (Vis-NIR) spectroscopy and hyperspectral imaging (Vis-NIR-HSI) techniques were applied on three different apple cultivars to compare their firmness prediction performances based on a large intra-variability of individual fruit, and develop rapid and simple models to visualize the variability of apple firmness on three apple cultivars. Apples with high degree of intra-variability can strongly affect the prediction model performances. The apple firmness prediction accuracy can be improved based on the large intra-variability samples with the coefficient variation (CV) values over 10%. The least squares-support vector machine (LS-SVM) models based on Vis-NIR-HSI spectra had better performances for firmness prediction than that of Vis-NIR spectroscopy, with the with the Rc2 over 0.84. Finally, The Vis-NIR-HSI technique combined with least squares-support vector machine (LS-SVM) models were successfully applied to visualize the spatial the variability of apple firmness.

Spectral separation degree method for Vis-NIR spectroscopic discriminant analysis of milk powder adulteration

Adulteration detection of adding ordinary milk powder to high-end dedicated milk powder is challenging due to the high similarity. Using visible and near-infrared (Vis-NIR) spectroscopy combined with k-nearest neighbor (kNN), the discriminant analysis models of pure brand milk powder and its adulterated milk powder (including unary and binary adulteration) were established. Standard normal variate transformation and Norris derivative filter (D = 2, S = 11, G = 5) were jointly used for spectral preprocessing. The separation degree and separation degree spectrum between two spectral populations were proposed and used to describe the differences between the two spectral populations, based on which, a novel wavelength selection method, named separation degree priority combination-kNN (SDPC-kNN), was proposed for wavelength optimization. SDPC-wavelength step-by-step phase-out-kNN (SDPC-WSP-kNN) models were established to further eliminate interference wavelengths and improve the model effect. The nineteen wavelengths in long-NIR region (1100-2498 nm) with a separation degree greater than 0 were used to establish single-wavelength kNN models, the total recognition-accuracy rates in prediction (RAR_P) all reached 100%, and the total recognition-accuracy rate in validation (RAR_V) of the optimal model (1174 nm) reached 97.4%. In the visible (400-780 nm) and short-NIR (780-1100 nm) regions with the separation degree all less than 0, the SDPC-WSP-kNN models were established. The two optimal models (N = 7, 22) were determined, the RAR_P values reached 100% and 97.4% respectively, and the RAR_V values reached 96.1% and 94.3% respectively. The results indicated that Vis-NIR spectroscopy combined with few-wavelength kNN has feasibility of high-precision milk powder adulteration discriminant. The few-wavelength schemes provided a valuable reference for designing dedicated miniaturized spectrometer of different spectral regions. The separation degree spectrum and SDPC can be used to improve the performance of spectral discriminant analysis. The SDPC method based on the separation degree priority proposed is a novel and effective wavelength selection method. It only needs to calculate the distance between two types of spectral sets at each wavelength with low computational complexity and good performance. In addition to combining with kNN, SDPC can also be combined with other classifier algorithms (e.g. PLS-DA, PCA-LDA) to expand the application scope of the method.

Exploring the potential of UAV hyperspectral image for estimating soil salinity: Effects of optimal band combination algorithm and random forest

Hyperspectral remote sensing by unmanned aerial vehicle (UAV) is an important technical tool for rapid, accurate, and real-time monitoring of soil salinity in arid zone agroecosystems. However, the key to effective soil salinity (electrical conductivity, EC) prediction by UAV visible and near-infrared (Vis-NIR) spectroscopy depends on the selection of effective features selection techniques and robust prediction characteristics algorithms. Therefore, in this study, two advanced feature selection methods and two commonly used modeling methods were applied to predict and characterize the spatial patterns of soil salinity (EC). The aim of this study was to explore the predictive performance of different feature band selection methods and to identify a robust soil salinity mapping strategy. The results demonstrated that standard normal variate (SNV) pre-processing broadened the absorption characteristics of the spectrum. Compared with competitive adaptive reweighted sampling (CARS), the optimal band combination algorithm (OBCA) strengthened the correlation with soil salinity and had a higher variable importance in the modeling. Random forest (RF) was more stable in mapping the spatial pattern of surface soil salinity compared to the partial least squares regression model (PLSR). Our results confirm the effectiveness of OBCA and RF in the developing UAV remote sensing models for surface soil salinity estimation and mapping.

Sedimentary structure discrimination with hyperspectral imaging in sediment cores

Hyperspectral imaging (HSI) is a non-destructive, high-resolution imaging technique that is currently under significant development for analyzing geological areas with remote devices or natural samples in a laboratory. In both cases, the hyperspectral image provides several sedimentary structures that must be separated to temporally and spatially describe the sample. Sediment sequences are composed of successive deposits (strata, homogenite, flood) that are visible depending on sample properties. The classical methods to identify them are time-consuming, have a low spatial resolution (millimeters) and are generally based on naked-eye counting. In this study, we compare several supervised classification algorithms to discriminate sedimentological structures in lake sediments. Instantaneous events in lake sediments are generally linked to extreme geodynamical events (e.g., floods, earthquakes), so their identification and counting are essential to understand long-term fluctuations and improve hazard assessments. Identification and counting are done by reconstructing a chronicle of event layer occurrence, including estimation of deposit thicknesses. Here, we applied two hyperspectral imaging sensors (Visible Near-Infrared, VNIR, 60 μm, 400-1000 nm; Short Wave Infrared, SWIR, 200 μm, 1000-2500 nm) on three sediment cores from different lake systems. We highlight that the SWIR sensor is the optimal one for creating robust classification models with discriminant analyses (prediction accuracies of 0.87-0.98). Indeed, the VNIR sensor is impacted by the surface reliefs and structures that are not in the learning set, which causes mis-classification. These observations are also valid for the combined sensor (VNIR-SWIR) and the RGB images. Several spatial and spectral pre-processing were also compared and enabled one to highlight discriminant information specific to a sample and a sensor. These works show that the combined use of hyperspectral imaging and machine learning improves the characterization of sedimentary structures compared to conventional methods.

Estimation of heavy metals using deep neural network with visible and infrared spectroscopy of soil

Heavy metal contamination in soil disturbs the chemical, biological, and physical soil conditions and adversely affects the health of living organisms. Visible and near-infrared spectroscopy (VNIRS) shows a potential feasibility for estimating heavy metal elements in soil. Moreover, deep learning models have been shown to successfully deal with complex multi-dimensional and multivariate nonlinear data. Thus, this study implemented a deep learning method on reflectance spectra of soil samples to estimate heavy metal concentrations. A convolutional neural network (CNN) was adopted to estimate arsenic (As), copper (Cu), and lead (Pb) concentrations using measured soil reflectance. In addition, a convolutional autoencoder was utilized as a joint method with the CNN for dimensionality reduction of the reflectance spectra. Furthermore, artificial neural network (ANN) and random forest regression (RFR) models were built for heavy metal estimation. Principal component analysis was utilized for dimensionality reduction of the ANN and RFR models. Among these models, the CNN model with convolutional autoencoder showed the highest accuracies for As, Cu, and Pb estimates, having R² values of 0.86, 0.74, and 0.82, respectively. The convolutional autoencoder disentangled the relevant features of multiple heavy metal elements and delivered robust features to the CNN model, thereby generating relatively accurate estimates.

Prediction of intramuscular fat in lamb by visible and near-infrared spectroscopy in an abattoir environment

The study used visible and near-infrared spectroscopy (Vis-NIR) in a large commercial processing plant, to test a system for meat quality (intramuscular fat; IMF) data collection within a supply chain for UK lamb meat. Crossbred Texel x Scotch Mule lambs (n = 220), finished on grass on 4 farms and slaughtered across 2 months, were processed through the abattoir and cutting plant and recorded using electronic identification. Vis-NIR scanning of the cut surface of the M. longissimus lumborum produced spectral data that predicted laboratory-measured IMF% with moderate accuracy (R² 0.38-0.48). Validation of the Vis-NIR prediction equations on an independent sample of 30 lambs slaughtered later in the season, provided similar accuracy of IMF prediction (R² 0.54). Values of IMF from four different laboratory tests were highly correlated with each other (r 0.82-0.95) and with Vis-NIR predicted IMF (r 0.66-0.75). Results suggest scope to collect lamb loin IMF data from a commercial UK abattoir, to sort cuts for different customers or to feed back to breeding programmes to improve meat quality.

Rapid and simultaneous analysis of direct and indirect bilirubin indicators in serum through reagent-free visible-near-infrared spectroscopy combined with chemometrics

Indirect (IBil), direct (DBil) and total (TBil) bilirubin are important clinical indicators of hepatobiliary diseases, which require rapid detection in diagnosis and treatment. IBil and DBil have a structural relationship with several macromolecules in hepatobiliary metabolism. Here, the rapid analysis models for bilirubin indicators using serum visible-near-infrared (Vis-NIR) spectroscopy were established. Norris derivative filter with optimisation was used for spectral pretreatment; the optimal parameters (derivative order, number of smoothing points, number of differential gaps) were (2, 15, 9) for IBil; (2, 13, 9) for DBil, respectively. Equidistant combination-partial least squares (EC-PLS) was used for large-scale wavelength screening. Wavelength step-by-step phase-out PLS (WSP-PLS) was used for secondary wavelength optimisation. The wavelength models of the optimal EC-WSP-PLS for IBil and DBil included 11 and 18 wavelengths, respectively. In independent validation, the root-mean-square errors and correlation coefficient for prediction (SEP, R_P), and ratio of performance-to-deviation (RPD) were 0.90 μmol L^-1, 0.975, and 4.4 for IBil; 0.71 μmol L^-1, 0.955, and 3.3 for DBil, respectively. TBil was subjected to spectral analysis, and the summation of the prediction values of IBil and DBil was compared. The latter was obviously better, and SEP, R_P, RPD were 0.82 μmol L^-1, 0.990, 7.1, respectively. The results for IBil, DBil and TBil indicated high correlation, low error and good overall prediction ability and confirmed the feasibility of the simultaneous analysis of bilirubin indicators through reagent-free serum Vis-NIR spectroscopy. The proposed method is crucial for the rapid screening of large populations and the treatment of hepatobiliary diseases.

A transferable spectroscopic diagnosis model for predicting arsenic contamination in soil

Visible and near-infrared reflectance (VNIR) spectroscopy is considered to be a potential and efficient means for monitoring soil arsenic (As) contamination. While current studies mainly focus on the evaluation of models' performance when training and verification samples are collected from the same region, whether the model developed at a specific region can be transferred to other regions is still unclear. To answer this question, this study collected a total of 247 samples for training and verification from regions with different geographical conditions, which are Yuanping and Baoding in northern China, Chenzhou and Hengyang in southern China. Afterward, we proposed a transfer component analysis (TCA) based spectroscopic diagnosis model, which aims at adapting a model learned from one region to other regions. This model was compared with the traditional modeling method in terms of the prediction accuracy by four experiments. The results show that: (1) The traditional modeling method trained by specific regional samples has no transfer capability to different regions, since the coefficient of determination (R²) and the ratio of prediction to deviation (RPD) were 0.02 and 0.65 for the first pair of study areas, 0.01 and 1.01 for the second pair of study areas; (2) A transfer model with favorable predictability can be constructed with the aid of TCA spectral transformation and a small amount off-site samples (R² and RPD were improved to 0.68 and 1.54 for the first pair of study areas, 0.64 and 1.66 for the second pair of study areas). Results suggest that it is promising to develop potential implementations of transferable spectroscopic diagnosis models for estimating soil As concentrations in large area with lower cost.

Predicting cadmium concentration in soils using laboratory and field reflectance spectroscopy

Visible and near-infrared spectroscopy (VNIRS, 350-2500 nm) is a promising alternative to rapidly investigate soil contamination by heavy metals. To explore the possibility of predicting heavy metal concentration in soils using laboratory and field reflectance spectroscopy and examine transferability of the prediction method, 46 soil samples from a mining area, 42 soil samples from an agricultural land, and the corresponding two sets of field soil spectra were collected. Cadmium (Cd) was taken as an example in this study. The collected soil samples were air-dried, ground, sieved, and then used for laboratory spectral measurement and chemical analysis. Soil reflectance spectroscopy associated with organic matter was extracted from the VNIRS and used to predict Cd concentration based on strong sorption and retention of Cd on soil organic matter. Genetic algorithm (GA) was adopted for band selection, and the selected bands were used to calibrate the prediction model with partial least squares regression (PLSR). Compared with the prediction using entire VNIR region, the ratio of prediction to deviation (RPD) and the coefficient of determination (R²) were improved from 1.473 and 0.508 to 2.997 and 0.881 for laboratory spectra and 1.437 and 0.484 to 1.992 and 0.731 for field spectra by using spectral bands associated with organic matter in the mining area. The RPD and R² values were improved from 1.919 and 0.707 to 3.727 and 0.923 for laboratory spectra and 1.057 and 0.036 to 1.747 and 0.646 for field spectra by the prediction method in the agricultural land. The improvement was further revealed by prediction of Cd concentration with a selected subset of soil samples from the mining area. The results suggest that predicting Cd concentration in soils with GA-PLSR using reflectance spectroscopy associated with organic matter is feasible and the prediction method could have the potential to be applied to field conditions.

Simultaneous determination of six main types of lipid-soluble pigments in green tea by visible and near-infrared spectroscopy

Lipid-soluble pigments make great contributions to the color of green tea. This study aimed to rapidly and simultaneously measure six main types of lipid-soluble pigments in green tea by using the visible and near-infrared (Vis-NIR) spectroscopy. A total of 135 tea samples with five kinds and three grades were collected for spectral scanning and color measurement, and their lipid-soluble pigments contents were measured by high performance liquid chromatography. It can be found that tea color was closely related to the six pigments. And there were significant differences in lipid-soluble pigments contents among these kinds and grades. Finally, quantitative determination models of the six pigments obtained excellent results with R_p² of 0.975, 0.973, 0.993, 0.919, 0.962 and 0.965 respectively based on multiple linear regression with the characteristic wavelengths. These results demonstrated that the Vis-NIR spectroscopy combined with chemometrics is a powerful tool for rapid determination of lipid-soluble pigments in green tea.

Visible and near-infrared spectroscopic comparison of five phyllosilicate mineral samples

Portable visible and near-infrared (vis/NIR) spectroscopy was used to characterize and differentiate the five phyllosilicate minerals and relate the bands to the mineral structure. The feature band at 2160-2170nm (4600-4630cm^-1) has been assigned to the high presence of Al-OH and is described as typical of dioctahedral phyllosilicate with OH groups coordinated around Al, and the feature occurred near 2322nm is considered to be due to a combination of the OH stretch with the MgOH deformation mode, which is a typical of trioctahedral phyllosilicates. The presence of the bands 1400 and 1900nm in vis/NIR spectrum indicated that some water is present in this sample. The absence of a 1900nm band but the presence of a 1400nm band indicates that only OH is present. Moreover, the significant differences between these five minerals were observed by the portable vis/NIR spectroscopy. The results show a potential for the application of vis/NIR spectroscopy in the identification and quantification of these minerals in the field. Further, such analysis can also provide important constraints on the nature of putative global and local-scale mineralogical transitions on Mars.

Computational optimization of the configuration of a spatially resolved spectroscopy sensor for milk analysis

A global optimizer has been developed, capable of computing the optimal configuration in a probe for spatially resolved reflectance spectroscopy (SRS). The main objective is to minimize the number of detection fibers, while maintaining an accurate estimation of both absorption and scattering profiles. Multiple fibers are necessary to robustify the estimation of optical properties against noise, which is typically present in the measured signals and influences the accuracy of the inverse estimation. The optimizer is based on a robust metamodel-based inverse estimation of the absorption coefficient and a reduced scattering coefficient from the acquired SRS signals. A genetic algorithm is used to evaluate the effect of the fiber placement on the performance of the inverse estimator to find the bulk optical properties of raw milk. The algorithm to find the optimal fiber placement was repeatedly executed for cases with a different number of detection fibers, ranging from 3 to 30. Afterwards, the optimal designs for each considered number of fibers were compared based on their performance in separating the absorption and scattering properties, and the significance of the differences was tested. A sensor configuration with 13 detection fibers was found to be the combination with the lowest number of fibers which provided an estimation performance which was not significantly worse than the one obtained with the best design (30 detection fibers). This design resulted in the root mean squared error of prediction (RMSEP) of 1.411 cm(-1) (R(2) = 0.965) for the estimation of the bulk absorption coefficient values, and 0.382 cm(-1) (R(2) = 0.996) for the reduced scattering coefficient values.

Rapid identification of soil cadmium pollution risk at regional scale based on visible and near-infrared spectroscopy

Soil heavy metal pollution due to long-term sewage irrigation is a serious environmental problem in many irrigation areas in northern China. Quickly identifying its pollution status is an important basis for remediation. Visible-near-infrared reflectance spectroscopy (VNIRS) provides a useful tool. In a case study, 76 soil samples were collected and their reflectance spectra were used to estimate cadmium (Cd) concentration by partial least squares regression (PLSR) and back propagation neural network (BPNN). To reduce noise, six pre-treatments were compared, in which orthogonal signal correction (OSC) was first used in soil Cd estimation. Spectral analysis and geostatistics were combined to identify Cd pollution hotspots. Results showed that Cd was accumulated in topsoil at the study area. OSC can effectively remove irrelevant information to improve prediction accuracy. More accurate estimation was achieved by applying a BPNN. Soil Cd pollution hotspots could be identified by interpolating the predicted values obtained from spectral estimates.