A Background and noise elimination method for quantitative calibration of near infrared spectra

A water probe for direct pH measurement of individual particles via micro-Raman spectroscopy

The acidity of atmospheric aerosols influences fundamental physicochemical processes that affect climate and human health. We recently developed a novel and facile water-probe-based method for directly measuring of the pH for micrometer-size droplets, providing a promising technique to better understand aerosol acidity in the atmosphere. The complex chemical composition of fine particles in the ambient air, however, poses certain challenges to using a water-probe for pH measurement, including interference from interactions between compositions and the influence of similar compositions on water structure. To explore the universality of our method, it was employed to measure the pH of ammonium, nitrate, carbonate, sulfate, and chloride particles. The pH of particles covering a broad range (0-14) were accurately determined, thereby demonstrating that our method can be generally applied, even to alkaline particles. Furthermore, a standard spectral library was developed by integrating the standard spectra of common hydrated ions extracted through the water-probe. The library can be employed to identify particle composition and overcome the spectral overlap problem resulting from similar effects. Using the spectral library, all ions were identified and their concentrations were determined, in turn allowing successful pH measurement of multicomponent (ammonium-sulfate-nitrate-chloride) particles. Insights into the synergistic effect of Cl-, NO3-, and NH4+ depletion obtained with our approach revealed the interplay between pH and volatile partitioning. Given the ubiquity of component partitioning and pH variation in particles, the water probe may provide a new perspective on the underlying mechanisms of aerosol aging and aerosol-cloud interaction.

An integrated CBLA-Net with fractional discrete wavelet transform and frequency-based CARS to predict heavy metal elements by XRF

BACKGROUND

X-ray fluorescence (XRF) emerges as a promising technique for estimating heavy metal elements. However, XRF spectra typically contain a significant amount of environmental information and signal noise, and the relationship between spectral intensity and element concentration is difficult to quantify using a single model, thereby reducing the predictive performance for low concentration elements.

RESULTS

This paper proposed a comprehensive framework for predicting elemental concentrations, encompassing preprocessing, variable selection, decision-making, to enable fast, non-destructive, and accurate estimation of element concentrations in soil. Firstly, an optimal denoising method based on fractional discrete wavelet transform (FDWT) was introduced to enhance signal quality. Furthermore, the frequency-based competitive adaptive reweighted sampling (FCARS) algorithm was employed for feature selection of XRF spectral variables, allowing extraction of the most informative features from the complex spectral data. Finally, a novel deep learning network, called ConvBiLSTM-Attention (CBLA-Net), was designed to achieve precise estimation of heavy metal elements concentration. Compared with other advanced algorithms, The CBLA-Net demonstrated the highest accuracy for V, Cr, Mn, Zn, Cd, and Pb, achieving the coefficient of determination (R2) of 0.9730, 0.9874, 0.9952, 0.9921, 0.9518, and 0.9741, respectively. The CBLA-Net not only effectively extract local features and capture global information, but also combines attention mechanism to focus on key information.

SIGNIFICANCE

The proposed novel deep learning quantitative framework, including preprocessing, feature selection, and CBLA-Net decision-making, significantly enhances the accuracy of elemental content prediction. It provides a new approach for accurately assessing the concentration of heavy metal elements in soil.

Study on robust model construction method of multi-batch fruit online sorting by near-infrared spectroscopy

In the online detection of fruit samples by near-infrared spectroscopy (NIRS), the natural change of sample states or the variations of instruments will often cause a large error in predicting different batches of samples. In this study, a total of 440 tomato samples were collected in four batches with each batch of 110 samples. The Spectral and soluble solids content (SSC) of single batch were collected every other day in batch order. The multivariate statistical process control (MSPC) method was adopted to establish a stability monitor model. The robustness regression (Rob-Reg) and partial least squares regression (PLSR) were used for mixed modeling of multiple batches of samples to eliminate the variability influence of sample and instrument states. The results show that MSPC can effectively monitor the consistency of the same batch samples measured at different times or different batches. The variation of sample attributes with spectral acquisition time has dramatically damaged the adaptation of PLSR models. The Rob-Reg method can predict the SSC of the different batches of samples at different collection times. Compared with the PLSR method, the correlation coefficient of prediction (R_p) was improved from 0.61 to 0.66, and the root mean square error of prediction (RMSEP) was decreased from 0.55 to 0.44 for Rob-Reg method. The RPD of 3.85 indicated that the model is excellent. The Robust modeling method can be well applied to fruit near-infrared online detection system.

Comprehensive evaluation of Dendrobium officinale from different geographical origins using near-infrared spectroscopy and chemometrics

Dendrobium officinale, often used as a kind of tea for daily drinks, has drawn increasing attention for its beneficial effects. Quality evaluation of D. officinale is of great significance to ensure its health care value and safeguard consumers' interest. Given that traditional analytical methods for assessing D. officinale quality are generally time-consuming and laborious, this study developed a comprehensive strategy, with the advantages of being rapid and efficient, enabling the quality evaluation of D. officinale from different geographical origins using near-infrared (NIR) spectroscopy and chemometrics. As the quality indicators, polysaccharides, polyphenols, total flavonoids, and total alkaloids were quantified. Three types of wavelength selection methods were used for model optimization and these were synergy interval (SI), genetic algorithm (GA), and competitive adaptive reweighted sampling (CARS). From the qualitative perspective, the geographical origins of D. officinale were differentiated by NIR spectroscopy combined with partial least squares-discriminant analysis (PLS-DA) and support vector classification (SVC). The PLS models constructed based on the wavelengths selected by CARS yielded the best performance for prediction of the contents of quality indicators in D. officinale. The root mean square error (RMSEP) and coefficient of determination (R_p²) in the independent test sets were 12.7768 g kg^-1 and 0.9586, 1.1346 g kg^-1 and 0.9670, 0.3938 g kg^-1 and 0.8803, 0.0825 and 0.7031 and for polysaccharides, polyphenols, total flavonoids, and total alkaloids, respectively. As for the origin identification, the nonlinear SVC was superior to the linear PLS-DA, with the correct recognition rates in calibration and prediction sets up to 100% and 100%, respectively. The overall results demonstrated the potential of NIR spectroscopy and chemometrics in the rapid determination of quality parameters and geographical origin. This study could provide a valuable reference for quality evaluation of D. officinale in a more rapid and comprehensive manner.

Spectral denoising based on Hilbert–Huang transform combined with F-test

Due to the influence of uncontrollable factors such as the environment and instruments, noise is unavoidable in a spectral signal, which may affect the spectral resolution and analysis result. In the present work, a novel spectral denoising method is developed based on the Hilbert–Huang transform (HHT) and F-test. In this approach, the original spectral signal is first decomposed by empirical mode decomposition (EMD). A series of intrinsic mode functions (IMFs) and a residual (r) are obtained. Then, the Hilbert transform (HT) is performed on each IMF and r to calculate their instantaneous frequencies. The mean and standard deviation of instantaneous frequencies are calculated to further illustrate the IMF frequency information. Third, the F-test is used to determine the cut-off point between noise frequency components and non-noise ones. Finally, the denoising signal is reconstructed by adding the IMF components after the cut-off point. Artificially chemical noised signal, X-ray diffraction (XRD) spectrum, and X-ray photoelectron spectrum (XPS) are used to validate the performance of the method in terms of the signal-to-noise ratio (SNR). The results show that the method provides superior denoising capabilities compared with Savitzky–Golay (SG) smoothing.

Diagnosis of Lung Cancer by FTIR Spectroscopy Combined With Raman Spectroscopy Based on Data Fusion and Wavelet Transform

Lung cancer is a fatal tumor threatening human health. It is of great significance to explore a diagnostic method with wide application range, high specificity, and high sensitivity for the detection of lung cancer. In this study, data fusion and wavelet transform were used in combination with Fourier transform infrared (FTIR) spectroscopy and Raman spectroscopy to study the serum samples of patients with lung cancer and healthy people. The Raman spectra of serum samples can provide more biological information than the FTIR spectra of serum samples. After selecting the optimal wavelet parameters for wavelet threshold denoising (WTD) of spectral data, the partial least squares–discriminant analysis (PLS-DA) model showed 93.41% accuracy, 96.08% specificity, and 90% sensitivity for the fusion data processed by WTD in the prediction set. The results showed that the combination of FTIR spectroscopy and Raman spectroscopy based on data fusion and wavelet transform can effectively diagnose patients with lung cancer, and it is expected to be applied to clinical screening and diagnosis in the future.

Water as a probe for pH measurement in individual particles using micro-Raman spectroscopy

Atmospheric aerosol acidity impacts numerous physicochemical processes, but the determination of particle pH remains a significant challenge due to the nonconservative nature of the H⁺ concentration ([H⁺]). Traditional measurements have difficulty in describing the practical state of an aerosol because they comprise chemical components or hypotheses that change the nature of the particles. In this work, we present a direct pH measurement that uses water as a general probe to detect [H⁺] in individual particles by micro-Raman spectroscopy. Containing the vibrational bands of ions and water influenced by ions, the spectra of hydrated ion were decomposed from the solution spectra as standard spectra by multivariate curve resolution analysis. Meanwhile, ratios of hydrated ions were calculated between the Raman spectra and standard spectra to evaluate concentration profiles of each ion. It demonstrated that good quantitative models between the ratio and concentration for all ions including H⁺ can be built with correlation coefficients (R²) higher than 0.95 for the solutions. The method was further applied to individual particle pH measurement. The pH value of sulfate aerosol particles was calculated, and the standard error was 0.09 using pH values calculated from the [HSO₄^-]/[SO₄^2-] as a reference. Furthermore, the applicability of the method was proven by detecting the pH value of chloride particles. Therefore, utilizing water, the most common substance, as the spectroscopic probe to measure [H⁺] without restriction of the ion system, this method has potential to measure the pH value of atmospheric particles with various compounds, although more work needs to be done to improve the sensitivity of the method.

Fraud Detection of Herbal Medicines Based on Modern Analytical Technologies Combine with Chemometrics Approach: A Review

Fraud in herbal medicines (HMs), commonplace throughout human history, is significantly related to medicinal effects with sometimes lethal consequences. Major HMs fraud events seem to occur with a certain regularity, such as substitution by counterfeits, adulteration by addition of inferior production-own materials, adulteration by chemical compounds, and adulteration by addition of foreign matter. The assessment of HMs fraud is in urgent demand to guarantee consumer protection against the four fraudulent activities. In this review, three analysis platforms (targeted, non-targeted, and the combination of non-targeted and targeted analysis) were introduced and summarized. Furthermore, the integration of analysis technology and chemometrics method (e.g., class-modeling, discrimination, and regression method) have also been discussed. Each integration shows different applicability depending on their advantages, drawbacks, and some factors, such as the explicit objective analysis or the nature of four types of HMs fraud. In an attempt to better solve four typical HMs fraud, appropriate analytical strategies are advised and illustrated with several typical studies. The article provides a general workflow of analysis methods that have been used for detection of HMs fraud. All analysis technologies and chemometrics methods applied can conduce to excellent reference value for further exploration of analysis methods in HMs fraud.

Accuracy improvement of quantitative LIBS analysis of coal properties using a hybrid model based on a wavelet threshold de-noising and feature selection method

A hybrid model based on a wavelet threshold de-noising (WTD) and recursive feature elimination with cross-validation (RFECV) method was proposed to improve the measurements in quantitative analysis of coal properties using laser-induced breakdown spectroscopy (LIBS). First, a modified threshold of WTD was proposed based on wavelet coefficient theory. Interference of noise in the LIBS spectrum was reduced by using this modified method. Then, the RFECV method was applied to extract effective features from the de-noised LIBS spectrum. Finally, support vector regression (SVR) models of coal properties were established by the selected features. A validation set was used to verify the effectiveness and robustness of the hybrid model. The improvement of the hybrid model on the quantitative analysis of each index of coal properties (heat value, ash, volatile content) was studied and discussed. By using the proposed model, the determination coefficient (R²), root mean square error of prediction, average relative error, and relative standard deviation were all significantly improved over the original spectra model. The results demonstrated that the proposed model could effectively improve the accuracy and precision of LIBS quantitative analysis for coal properties.

Data Fusion of Fourier Transform Mid-Infrared (MIR) and Near-Infrared (NIR) Spectroscopies to Identify Geographical Origin of Wild Paris polyphylla var. yunnanensis

Origin traceability is important for controlling the effect of Chinese medicinal materials and Chinese patent medicines. Paris polyphylla var. yunnanensis is widely distributed and well-known all over the world. In our study, two spectroscopic techniques (Fourier transform mid-infrared (FT-MIR) and near-infrared (NIR)) were applied for the geographical origin traceability of 196 wild P. yunnanensis samples combined with low-, mid-, and high-level data fusion strategies. Partial least squares discriminant analysis (PLS-DA) and random forest (RF) were used to establish classification models. Feature variables extraction (principal component analysis—PCA) and important variables selection models (recursive feature elimination and Boruta) were applied for geographical origin traceability, while the classification ability of models with the former model is better than with the latter. FT-MIR spectra are considered to contribute more than NIR spectra. Besides, the result of high-level data fusion based on principal components (PCs) feature variables extraction is satisfactory with an accuracy of 100%. Hence, data fusion of FT-MIR and NIR signals can effectively identify the geographical origin of wild P. yunnanensis.

A hybrid model combining wavelet transform and recursive feature elimination for running state evaluation of heat-resistant steel using laser-induced breakdown spectroscopy

Heat-resistant steel is widely used in various industries, and the running state is of great importance for equipment function and safety. In this work, laser-induced breakdown spectroscopy (LIBS) is applied to evaluate the running state of steel using indicators of micro and macro properties. The hybrid model based on wavelet threshold denoising (WTD) and K-fold-support vector machine-recursive feature elimination (K-SVM-RFE) is proposed to estimate the different indictors of various service conditions of steel. Fourteen T91 specimens, including 4 industrial specimens obtained from different service conditions in the power plant boiler, were used as the analytes. Firstly, the noise signal of the LIBS spectra of each specimen was analyzed and removed with WTD. Secondly, an improved approach K-SVM-RFE was applied to select the optimal feature subset and build the classification models of aging grade and hardness grade. The influence of denoising pretreatment on model performance was compared and discussed. Finally, the assessment matrix, established using the indicators from the aging grade and hardness grade, was used to evaluate the running state of steel. The results show that the test assessment matrix obtained with the hybrid model based on WTD and K-SVM-RFE is consistent with the reference matrix on the running state of steel.

Analytical-performance improvement of laser-induced breakdown spectroscopy for the processing degree of wheat flour using a continuous wavelet transform

Quality and safety of food are two of the most important matters in our lives. Wheat is one of the most important products in the modern agricultural processing industry. Issues of mislabeling and adulteration are of increasingly serious concern in the grain market. They threaten the credibility of producers and traders and the rights of the consumers. Therefore, it is very significant to guarantee the processing degree of wheat flour. In this work, two different spectral peak recognition methods, i.e., artificial spectral peak recognition and automatic spectral peak recognition, are carried out to study the adulteration problem in the food industry. Three grades of the processing degree of wheat flour from northern China are classified by laser-induced breakdown spectroscopy (LIBS). To search for an automatic classification model, continuous wavelet transform is used for the automatic recognition of the LIBS spectrum peak. Principal component analysis is used to reduce the collinearity of LIBS spectra data. First, 20 principal components were selected to represent the spectral data for the following discrimination analysis by a support vector machine. The results showed that the classification accuracies of automatic spectral peak recognition are better than those of artificial spectral peak recognition. The classification accuracies of artificial spectral peak recognition and automatic spectral peak recognition are 95.33% and 98.67%; the fivefold cross-validation classification accuracies are 94.67% and 96.67%; and the operation times were 240 min and 2 min, respectively. It can be concluded that LIBS can provide simpler and faster classification without the use of any chemical reagent, which represents a decisive advantage for applications dedicated to rapidly detecting the processing degree of wheat flour and other cereals.

Descriptor selection based on variable stability for predicting inhibitor activity

Quantitative structure-activity relationship (QSAR) has been a technique to study the relationship between chemical structures and properties, and variable selection is an important problem for finding the informative variables and building reliable models. A variable selection method based on variable stability is proposed and used for selecting the informative descriptors in the QSAR model of inhibitors. In the method, a series of models are built by leave-one-out cross validation (LOOCV), and variable stability is defined as the ratio of the absolute mean value and standard deviation of the regression coefficients in the models for a descriptor. Therefore, the descriptors with larger stabilities are more informative to the model. To further enhance the difference among the descriptors, an exponential parameter is used to modify the standard deviation. The results show that 13 descriptors are selected as informative ones from 1217 descriptors for the QSAR model of inhibitors. An effective prediction model can be constructed by them.

Investigating the Structural Change in Protein Aqueous Solution Using Temperature-Dependent Near-Infrared Spectroscopy and Continuous Wavelet Transform

The circulatory protein, human serum albumin (HSA), is widely used as a model protein for the study of protein structure. In this work, the structures of human serum albumin in aqueous solutions are studied using temperature-dependent near-infrared (NIR) spectroscopy with the aid of continuous wavelet transform (CWT). Near-infrared spectra of human serum albumin solutions with different concentrations were measured over a temperature range of 30-85 ℃. Then, continuous wavelet transform was performed on the spectra to enhance the resolution. As a result of the resolution enhancement, spectral bands around 4361, 4521, 4600 and 4260 cm^-1 were extracted from the overlapping low-resolution signals. The four bands can be assigned to the protein structures of α-helix, β-sheet, an intermediate state and side chains, respectively. The variations in intensity of the bands around 4361 and 4521 cm^-1 with temperature show that the increase of temperature leads to the loss of α-helical structure but the formation of β-sheet, and the denaturation temperature of human serum albumin is about 55 ℃. The variation of the band around 4600 cm^-1 indicates that the temperature-induced unfolding process of human serum albumin occurs through a stable intermediate state, and a significant change in the microenvironment of the side chains about 63 ℃ is observed from the variation of the band around 4260 cm^-1. On the other hand, the transformed spectra in the region of 8000-5600 cm^-1 provide an explicit evidence for the structural changes of water during the process of protein denaturation, and the unfolding process of HSA can be reflected by these changes.

UV spectrophotometric simultaneous determination of paracetamol and ibuprofen in combined tablets by derivative and wavelet transforms

The application of first-order derivative and wavelet transforms to UV spectra and ratio spectra was proposed for the simultaneous determination of ibuprofen and paracetamol in their combined tablets. A new hybrid approach on the combined use of first-order derivative and wavelet transforms to spectra was also discussed. In this application, DWT (sym6 and haar), CWT (mexh), and FWT were optimized to give the highest spectral recoveries. Calibration graphs in the linear concentration ranges of ibuprofen (12–32 mg/L) and paracetamol (20–40 mg/L) were obtained by measuring the amplitudes of the transformed signals. Our proposed spectrophotometric methods were statistically compared to HPLC in terms of precision and accuracy.

Accuracy improvement of quantitative analysis in laser-induced breakdown spectroscopy using modified wavelet transform

A modified algorithm of background removal based on wavelet transform was developed for spectrum correction in laser-induced breakdown spectroscopy (LIBS). The optimal type of wavelet function, decomposition level and scaling factor γ were determined by the root-mean-square error of calibration (RMSEC) of the univariate regression model of the analysis element, which is considered as the optimization criteria. After background removal by this modified algorithm with RMSEC, the root-mean-square error of cross-validation (RMSECV) and the average relative error (ARE) criteria, the accuracy of quantitative analysis on chromium (Cr), vanadium (V), cuprum (Cu), and manganese (Mn) in the low alloy steel was all improved significantly. The results demonstrated that the algorithm developed is an effective pretreatment method in LIBS to significantly improve the accuracy in the quantitative analysis.

Application of a hybrid variable selection method for determination of carbohydrate content in soy milk powder using visible and near infrared spectroscopy

Visible and near-infrared (Vis-NIR) spectroscopy was investigated to fast determine the carbohydrate content in soy milk powder. A hybrid variable selection method was proposed. In this method, a simulate annealing (SA) algorithm was first operated to search the optimal band (OB) in the wavelet packet transform (WPT) tree. The OB with 47 variables was further selected by SA (WTP-OB-SA). Finally, the number of variables was reduced from 47 to 20. The best partial least-squares prediction with a high residual predictive deviation (RPD) value of 12.2242 was obtained using these 20 variables with the correlation coefficient (r) and root-mean-square error of prediction (RMSEP) being 0.9967 and 0.1669, respectively. The results indicated that Vis-NIR spectroscopy could efficiently determine the carbohydrate content in soy milk powder. The WPT-OB-SA selection method eliminated redundant variables and improved the prediction ability.

Baseline noise and measurement uncertainty in liquid chromatography

The stochastic properties of baseline noise in HPLC systems with a UV photo-diode array, photo-multiplier and gamma-ray detector were examined by dividing the noise into auto-correlated random process (Markov process) and an independent process (white noise). The present work focused on the effect of the stochastic noise properties on a theoretical estimation of the standard deviation (SD) of area measurements in instrumental analyses. An estimation theory, called FUMI theory (Function of Mutual Information), was taken as an example. A computer simulation of noise was also used. It was shown that the reliability (confidence intervals) of theoretical SD estimates mainly depends on the following factors: the ratio of the white noise and Markov process occurring in the baselines; the number of data points used for the estimation; the width of a target peak for which the SD is estimated.

An adaptive strategy for selecting representative calibration samples in the continuous wavelet domain for near-infrared spectral analysis

Sample selection is often used to improve the cost-effectiveness of near-infrared (NIR) spectral analysis. When raw NIR spectra are used, however, it is not easy to select appropriate samples, because of background interference and noise. In this paper, a novel adaptive strategy based on selection of representative NIR spectra in the continuous wavelet transform (CWT) domain is described. After pretreatment with the CWT, an extension of the Kennard-Stone (EKS) algorithm was used to adaptively select the most representative NIR spectra, which were then submitted to expensive chemical measurement and multivariate calibration. With the samples selected, a PLS model was finally built for prediction. It is of great interest to find that selection of representative samples in the CWT domain, rather than raw spectra, not only effectively eliminates background interference and noise but also further reduces the number of samples required for a good calibration, resulting in a high-quality regression model that is similar to the model obtained by use of all the samples. The results indicate that the proposed method can effectively enhance the cost-effectiveness of NIR spectral analysis. The strategy proposed here can also be applied to different analytical data for multivariate calibration.

Simultaneous Wavelength Selection and Outlier Detection in Multivariate Regression of Near-Infrared Spectra

Near-infrared (NIR) spectrometry will present a more promising tool for quantitative measurement if the robustness and predictive ability of the partial least square (PLS) model are improved. In order to achieve the purpose, we present a new algorithm for simultaneous wavelength selection and outlier detection; at the same time, the problems of background and noise in multivariate calibration are also solved. The strategy is a combination of continuous wavelet transform (CWT) and modified iterative predictors and objects weighting PLS (mIPOW-PLS). CWT is performed as a pretreatment tool for eliminating background and noise synchronously; then, mIPOW-PLS is proposed to remove both the useless wavelengths and the multiple outliers in CWT domain. After pretreatment with CWT-mIPOW-PLS, a PLS model is built finally for prediction. The results indicate that the combination of CWT and mIPOW-PLS produces robust and parsimonious regression models with very few wavelengths.