Hyperspectral Imagery Classification Using Sparse Representations of Convolutional Neural Network Features

A Dual-Branch Fusion of a Graph Convolutional Network and a Convolutional Neural Network for Hyperspectral Image Classification

Semi-supervised graph convolutional networks (SSGCNs) have been proven to be effective in hyperspectral image classification (HSIC). However, limited training data and spectral uncertainty restrict the classification performance, and the computational demands of a graph convolution network (GCN) present challenges for real-time applications. To overcome these issues, a dual-branch fusion of a GCN and convolutional neural network (DFGCN) is proposed for HSIC tasks. The GCN branch uses an adaptive multi-scale superpixel segmentation method to build fusion adjacency matrices at various scales, which improves the graph convolution efficiency and node representations. Additionally, a spectral feature enhancement module (SFEM) enhances the transmission of crucial channel information between the two graph convolutions. Meanwhile, the CNN branch uses a convolutional network with an attention mechanism to focus on detailed features of local areas. By combining the multi-scale superpixel features from the GCN branch and the local pixel features from the CNN branch, this method leverages complementary features to fully learn rich spatial-spectral information. Our experimental results demonstrate that the proposed method outperforms existing advanced approaches in terms of classification efficiency and accuracy across three benchmark data sets.

Deep Network With Irregular Convolutional Kernels and Self-Expressive Property for Classification of Hyperspectral Images

This article presents a novel deep network with irregular convolutional kernels and self-expressive property (DIKS) for the classification of hyperspectral images (HSIs). Specifically, we use the principal component analysis (PCA) and superpixel segmentation to obtain a series of irregular patches, which are regarded as convolutional kernels of our network. With such kernels, the feature maps of HSIs can be adaptively computed to well describe the characteristics of each object class. After multiple convolutional layers, features exported by all convolution operations are combined into a stacked form with both shallow and deep features. These stacked features are then clustered by introducing the self-expression theory to produce final features. Unlike most traditional deep learning approaches, the DIKS method has the advantage of self-adaptability to the given HSI due to building irregular kernels. In addition, this proposed method does not require any training operations for feature extraction. Because of using both shallow and deep features, the DIKS has the advantage of being multiscale. Due to introducing self-expression, the DIKS method can export more discriminative features for HSI classification. Extensive experimental results are provided to validate that our method achieves better classification performance compared with state-of-the-art algorithms.

Statistic and Network Features of RGB and Hyperspectral Imaging for Determination of Black Root Mold Infection in Apples

Apples damaged by black root mold (BRM) lose moisture, vitamins, and minerals as well as carry dangerous toxins. Determination of the infection degree can allow for customized use of apples, reduce financial losses, and ensure food safety. In this study, red-green-blue (RGB) imaging and hyperspectral imaging (HSI) are combined to detect the infection degree of BRM in apple fruits. First, RGB and HSI images of healthy, mildly, moderately, and severely infected fruits are measured, and those with effective wavelengths (EWs) are screened from HSI by random frog. Second, the statistic and network features of images are extracted by using color moment and convolutional neural network. Meanwhile, random forest (RF), K-nearest neighbor, and support vector machine are used to construct classification models with the above two features of RGB and HSI images of EWs. Optimal results with the 100% accuracy of training set and 96% accuracy of prediction set are obtained by RF with the statistic and network features of the two images, outperforming the other cases. The proposed method furnishes an accurate and effective solution for determining the BRM infection degree in apples.

Construction and Analysis of Emotion Recognition and Psychotherapy System of College Students under Convolutional Neural Network and Interactive Technology

This study's aim is to effectively establish a psychological intervention and treatment system for college students and discover and correct their psychological problems encountered in a timely manner. From the perspectives of pedagogy and psychology, the college students majoring in physical education are selected as the research objects, and an interactive college student emotion recognition and psychological intervention system is established based on convolutional neural network (CNN). The system takes face recognition as the data source, adopts feature recognition algorithms to effectively classify the different students, and designs a psychological intervention platform based on interactive technology, and it is compared with existing systems and models to further verify its effectiveness. The results show that the deep learning CNN has better ability to recognize student emotions than backpropagation neural network (BPNN) and decision tree (DT) algorithm. The recognition accuracy (ACC) can be as high as 89.32%. Support vector machine (SVM) algorithm is adopted to classify the emotions, and the recognition ACC is increased by 20%. When the system's K value is 5 and d value is 8, the ACC of the model can reach 92.35%. The use of this system for psychotherapy has a significant effect, and 45% of the students are very satisfied with the human-computer interaction of the system. This study aims to guess the psychology of students through emotion recognition and reduce human participation based on the human-computer interaction, which can provide a new research idea for college psychotherapy. At present, the mental health problems of college students cannot be ignored; especially every year, there will be news reports of college students' extreme behaviors due to depression and other psychological problems. An interactive college student emotion recognition and psychological intervention system based on convolutional neural network (CNN) is established. This system uses face recognition as the basic support technology and uses feature recognition algorithms to effectively classify different students. An interaction technology-based psychological intervention platform is designed and compared with existing systems and models to further verify the effectiveness of the proposed system. The results show that deep learning has better student emotion recognition ability than backpropagation neural network (BPNN) and decision tree algorithm. The recognition accuracy is up to 89.32%. Support vector machine algorithm is employed to classify emotions, and the recognition acceptability rate increases by 20%. When K is 5 and d is 8, the acceptability rate of the model can reach 92.35%. The effect of this system in psychotherapy is remarkable, and 45% of students are very satisfied with the human-computer interaction of this system. This work aims to speculate students' psychology through emotion recognition, reduce people's participation via human-computer interaction, and provide a new research idea for university psychotherapy.

Potential of Snapshot-Type Hyperspectral Imagery Using Support Vector Classifier for the Classification of Tomatoes Maturity

It is necessary to convert to automation in a tomato hydroponic greenhouse because of the aging of farmers, the reduction in agricultural workers as a proportion of the population, COVID-19, and so on. In particular, agricultural robots are attractive as one of the ways for automation conversion in a hydroponic greenhouse. However, to develop agricultural robots, crop monitoring techniques will be necessary. In this study, therefore, we aimed to develop a maturity classification model for tomatoes using both support vector classifier (SVC) and snapshot-type hyperspectral imaging (VIS: 460–600 nm (16 bands) and Red-NIR: 600–860 nm (15 bands)). The spectral data, a total of 258 tomatoes harvested in January and February 2022, was obtained from the tomatoes’ surfaces. Spectral data that has a relationship with the maturity stages of tomatoes was selected by correlation analysis. In addition, the four different spectral data were prepared, such as VIS data (16 bands), Red-NIR data (15 bands), combination data of VIS and Red-NIR (31 bands), and selected spectral data (6 bands). These data were trained by SVC, respectively, and we evaluated the performance of trained classification models. As a result, the SVC based on VIS data achieved a classification accuracy of 79% and an F1-score of 88% to classify the tomato maturity into six stages (Green, Breaker, Turning, Pink, Light-red, and Red). In addition, the developed model was tested in a hydroponic greenhouse and was able to classify the maturity stages with a classification accuracy of 75% and an F1-score of 86%.

Hyperspectral Image Classification with a Multiscale Fusion-Evolution Graph Convolutional Network Based on a Feature-Spatial Attention Mechanism

Convolutional neural network (CNN) has achieved excellent performance in the classification of hyperspectral images (HSI) due to its ability to extract spectral and spatial feature information. However, the conventional CNN model does not perform well in regions with irregular geometric appearances. The recently proposed graph convolutional network (GCN) has been successfully applied to the analysis of non-Euclidean data and is suitable for irregular image regions. However, conventional GCN has problems such as very high computational cost on HSI data and cannot make full use of information in the image spatial domain. To this end, this paper proposes a multi-scale fusion-evolution graph convolutional network based on the feature-spatial attention mechanism (MFEGCN-FSAM). Our model enables the graph to be automatically evolved during the graph convolution process to produce more accurate embedding features. We have established multiple local and global input graphs to utilize the multiscale spectral and spatial information of the image. In addition, this paper designs a feature-spatial attention module to extract important features and structural information from the graph. The experimental results on four typical datasets show that the MFEGCN-FSAM proposed in this paper has better performance than most existing HSI classification methods.

Multiscale Feature Aggregation Capsule Neural Network for Hyperspectral Remote Sensing Image Classification

Models based on capsule neural network (CapsNet), a novel deep learning method, have recently made great achievements in hyperspectral remote sensing image (HSI) classification due to their excellent ability to implicitly model the spatial relationship knowledge embedded in HSIs. However, the number of labeled samples is a common bottleneck in HSI classification, limiting the performance of these deep learning models. To alleviate the problem of limited labeled samples and further explore the potential of CapsNet in the HSI classification field, this study proposes a multiscale feature aggregation capsule neural network (MS-CapsNet) based on CapsNet via the implementation of two branches that simultaneously extract spectral, local spatial, and global spatial features to integrate multiscale features and improve model robustness. Furthermore, because deep features are generally more discriminative than shallow features, two kinds of capsule residual (CapsRES) blocks based on 3D convolutional capsule (3D-ConvCaps) layers and residual connections are proposed to increase the depth of the network and solve the limited labeled sample problem in HSI classification. Moreover, a squeeze-and-excitation (SE) block is introduced in the shallow layers of MS-CapsNet to enhance its feature extraction ability. In addition, a reasonable initialization strategy that transfers parameters from two well-designed, pretrained deep convolutional capsule networks is introduced to help the model find a good set of initializing weight parameters and further improve the HSI classification accuracy of MS-CapsNet. Experimental results on four widely used HSI datasets demonstrate that the proposed method can provide results comparable to those of state-of-the-art methods.

Deep Learning Model for Enhanced Crop Identification from Landsat 8 Images

Deep learning is a powerful state-of-the-art technique for image processing, including remote sensing images. This paper describes a multilevel deep learning based crop type identification system that targets land cover and crop type classification from multi-temporal multisource satellite imagery. The proposed crop type identification is based on unsupervised neural network that is used for optical imagery segmentation and missing data restoration due to clouds and shadows, and an ensemble of supervised neural networks. The main part of this multilayer deep network with Self Organizing maps and atmospheric correction is an ensemble of CNNs. The proposed system is applied for crop identification using Landsat-8 time-series and implemented with different sized vector data, parcel boundary. Aided with self-organizing maps and atmospheric correction for pre-processing doing both pixel based and parcel based analysis, this proposed crop type identification system allowed us to achieve the overall classification accuracy of nearly 95% for three different time periods.

Adaptable Convolutional Network for Hyperspectral Image Classification

Nowadays, a large number of remote sensing instruments are providing a massive amount of data within the frame of different Earth Observation missions. These instruments are characterized by the wide variety of data they can collect, as well as the impressive volume of data and the speed at which it is acquired. In this sense, hyperspectral imaging data has certain properties that make it difficult to process, such as its large spectral dimension coupled with problematic data variability. To overcome these challenges, convolutional neural networks have been proposed as classification models because of their ability to extract relevant spectral–spatial features and learn hidden patterns, along their great architectural flexibility. Their high performance relies on the convolution kernels to exploit the spatial relationships. Thus, filter design is crucial for the correct performance of models. Nevertheless, hyperspectral data may contain objects with different shapes and orientations, preventing filters from “seeing everything possible” during the decision making. To overcome this limitation, this paper proposes a novel adaptable convolution model based on deforming kernels combined with deforming convolution layers to fit their effective receptive field to the input data. The proposed adaptable convolutional network (named DKDCNet) has been evaluated over two well-known hyperspectral scenes, demonstrating that it is able to achieve better results than traditional strategies with similar computational cost for HSI classification.

Hyperspectral image spectral-spatial classification via weighted Laplacian smoothing constraint-based sparse representation

As a powerful tool in hyperspectral image (HSI) classification, sparse representation has gained much attention in recent years owing to its detailed representation of features. In particular, the results of the joint use of spatial and spectral information has been widely applied to HSI classification. However, dealing with the spatial relationship between pixels is a nontrivial task. This paper proposes a new spatial-spectral combined classification method that considers the boundaries of adjacent features in the HSI. Based on the proposed method, a smoothing-constraint Laplacian vector is constructed, which consists of the interest pixel and its four nearest neighbors through their weighting factor. Then, a novel large-block sparse dictionary is developed for simultaneous orthogonal matching pursuit. Our proposed method can obtain a better accuracy of HSI classification on three real HSI datasets than the existing spectral-spatial HSI classifiers. Finally, the experimental results are presented to verify the effectiveness and superiority of the proposed method.

Application and Evaluation of a Deep Learning Architecture to Urban Tree Canopy Mapping

Urban forest is a dynamic urban ecosystem that provides critical benefits to urban residents and the environment. Accurate mapping of urban forest plays an important role in greenspace management. In this study, we apply a deep learning model, the U-net, to urban tree canopy mapping using high-resolution aerial photographs. We evaluate the feasibility and effectiveness of the U-net in tree canopy mapping through experiments at four spatial scales—16 cm, 32 cm, 50 cm, and 100 cm. The overall performance of all approaches is validated on the ISPRS Vaihingen 2D Semantic Labeling dataset using four quantitative metrics, Dice, Intersection over Union, Overall Accuracy, and Kappa Coefficient. Two evaluations are performed to assess the model performance. Experimental results show that the U-net with the 32-cm input images perform the best with an overall accuracy of 0.9914 and an Intersection over Union of 0.9638. The U-net achieves the state-of-the-art overall performance in comparison with object-based image analysis approach and other deep learning frameworks. The outstanding performance of the U-net indicates a possibility of applying it to urban tree segmentation at a wide range of spatial scales. The U-net accurately recognizes and delineates tree canopy for different land cover features and has great potential to be adopted as an effective tool for high-resolution land cover mapping.

A Lightweight 1-D Convolution Augmented Transformer with Metric Learning for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is the subject of intense research in remote sensing. The tremendous success of deep learning in computer vision has recently sparked the interest in applying deep learning in hyperspectral image classification. However, most deep learning methods for hyperspectral image classification are based on convolutional neural networks (CNN). Those methods require heavy GPU memory resources and run time. Recently, another deep learning model, the transformer, has been applied for image recognition, and the study result demonstrates the great potential of the transformer network for computer vision tasks. In this paper, we propose a model for hyperspectral image classification based on the transformer, which is widely used in natural language processing. Besides, we believe we are the first to combine the metric learning and the transformer model in hyperspectral image classification. Moreover, to improve the model classification performance when the available training samples are limited, we use the 1-D convolution and Mish activation function. The experimental results on three widely used hyperspectral image data sets demonstrate the proposed model’s advantages in accuracy, GPU memory cost, and running time.

A Neural Network Method for Classification of Sunlit and Shaded Components of Wheat Canopies in the Field Using High-Resolution Hyperspectral Imagery

(1) Background: Information rich hyperspectral sensing, together with robust image analysis, is providing new research pathways in plant phenotyping. This combination facilitates the acquisition of spectral signatures of individual plant organs as well as providing detailed information about the physiological status of plants. Despite the advances in hyperspectral technology in field-based plant phenotyping, little is known about the characteristic spectral signatures of shaded and sunlit components in wheat canopies. Non-imaging hyperspectral sensors cannot provide spatial information; thus, they are not able to distinguish the spectral reflectance differences between canopy components. On the other hand, the rapid development of high-resolution imaging spectroscopy sensors opens new opportunities to investigate the reflectance spectra of individual plant organs which lead to the understanding of canopy biophysical and chemical characteristics. (2) Method: This study reports the development of a computer vision pipeline to analyze ground-acquired imaging spectrometry with high spatial and spectral resolutions for plant phenotyping. The work focuses on the critical steps in the image analysis pipeline from pre-processing to the classification of hyperspectral images. In this paper, two convolutional neural networks (CNN) are employed to automatically map wheat canopy components in shaded and sunlit regions and to determine their specific spectral signatures. The first method uses pixel vectors of the full spectral features as inputs to the CNN model and the second method integrates the dimension reduction technique known as linear discriminate analysis (LDA) along with the CNN to increase the feature discrimination and improves computational efficiency. (3) Results: The proposed technique alleviates the limitations and lack of separability inherent in existing pre-defined hyperspectral classification methods. It optimizes the use of hyperspectral imaging and ensures that the data provide information about the spectral characteristics of the targeted plant organs, rather than the background. We demonstrated that high-resolution hyperspectral imagery along with the proposed CNN model can be powerful tools for characterizing sunlit and shaded components of wheat canopies in the field. The presented method will provide significant advances in the determination and relevance of spectral properties of shaded and sunlit canopy components under natural light conditions.

Data Augmentation and Spectral Structure Features for Limited Samples Hyperspectral Classification

For both traditional classification and current popular deep learning methods, the limited sample classification problem is very challenging, and the lack of samples is an important factor affecting the classification performance. Our work includes two aspects. First, the unsupervised data augmentation for all hyperspectral samples not only improves the classification accuracy greatly with the newly added training samples, but also further improves the classification accuracy of the classifier by optimizing the augmented test samples. Second, an effective spectral structure extraction method is designed, and the effective spectral structure features have a better classification accuracy than the true spectral features.

Hyperspectral Imaging for Minced Meat Classification Using Nonlinear Deep Features

Minced meat substitution is one of the most common forms of food fraud in the meat industry. Recently, Hyperspectral Imaging (HSI) has been used for the classification and identification of minced meat types. However, conventional methods are based only on spectral information and ignore the spatial variability of the data. Moreover, these methods first tend to reduce the size of the data, which to some extent ignores the abstract level information and does not preserve the spatial information. Therefore, this work proposes a novel Isos-bestic wavelength reduction method for the different minced meat types, by retaining only Myoglobin pigments (Mb) in the meat spectra. A total of 60 HSI cubes are acquired using Fx 10 Hyperspectral sensor. For each HSI cube, a set of preprocessing schemes is applied to extract the Region of Interest (ROI) and spectral preprocessing, i.e., Golay filtering. Later, these preprocessed HSI cubes are fed into a 3D-Convolutional Neural Network (3D-CNN) model for nonlinear feature extraction and classification. The proposed pipeline outperformed several state-of-the-art methods, with an overall accuracy of 94.0%.

Hyperspectral deep convolution anomaly detection based on weight adjustment strategy

Hyperspectral anomaly detection has garnered much research in recent years due to the excellent detection ability of hyperspectral remote sensing in agriculture, forestry, geological surveys, environmental monitoring, and battlefield target detection. The traditional anomaly detection method ignores the non-linearity and complexity of the hyperspectral image (HSI), while making use of the effectiveness of spatial information rarely. Besides, the anomalous pixels and the background are mixed, which causes a higher false alarm rate in the detection result. In this paper, a hyperspectral deep net-based anomaly detector using weight adjustment strategy (WAHyperDNet) is proposed to circumvent the above issues. We leverage three-dimensional convolution instead of the two-dimensional convolution to get a better way of handling high-dimensional data. In this study, the determinative spectrum-spatial features are extracted across the correlation between HSI pixels. Moreover, feature weights in the method are automatically generated based on absolute distance and the spectral similarity angle to describe the differences between the background pixels and the pixels to be tested. Experimental results on five public datasets show that the proposed approach outperforms the state-of-the-art baselines in both effectiveness and efficiency.

Understanding deep learning in land use classification based on Sentinel-2 time series

The use of deep learning (DL) approaches for the analysis of remote sensing (RS) data is rapidly increasing. DL techniques have provided excellent results in applications ranging from parameter estimation to image classification and anomaly detection. Although the vast majority of studies report precision indicators, there is a lack of studies dealing with the interpretability of the predictions. This shortcoming hampers a wider adoption of DL approaches by a wider users community, as model's decisions are not accountable. In applications that involve the management of public budgets or policy compliance, a better interpretability of predictions is strictly required. This work aims to deepen the understanding of a recurrent neural network for land use classification based on Sentinel-2 time series in the context of the European Common Agricultural Policy (CAP). This permits to address the relevance of predictors in the classification process leading to an improved understanding of the behaviour of the network. The conducted analysis demonstrates that the red and near infrared Sentinel-2 bands convey the most useful information. With respect to the temporal information, the features derived from summer acquisitions were the most influential. These results contribute to the understanding of models used for decision making in the CAP to accomplish the European Green Deal (EGD) designed in order to counteract climate change, to protect biodiversity and ecosystems, and to ensure a fair economic return for farmers.

MRI Brain Classification Using the Quantum Entropy LBP and Deep-Learning-Based Features

Brain tumor detection at early stages can increase the chances of the patient’s recovery after treatment. In the last decade, we have noticed a substantial development in the medical imaging technologies, and they are now becoming an integral part in the diagnosis and treatment processes. In this study, we generalize the concept of entropy difference defined in terms of Marsaglia formula (usually used to describe two different figures, statues, etc.) by using the quantum calculus. Then we employ the result to extend the local binary patterns (LBP) to get the quantum entropy LBP (QELBP). The proposed study consists of two approaches of features extractions of MRI brain scans, namely, the QELBP and the deep learning DL features. The classification of MRI brain scan is improved by exploiting the excellent performance of the QELBP–DL feature extraction of the brain in MRI brain scans. The combining all of the extracted features increase the classification accuracy of long short-term memory network when using it as the brain tumor classifier. The maximum accuracy achieved for classifying a dataset comprising 154 MRI brain scan is 98.80%. The experimental results demonstrate that combining the extracted features improves the performance of MRI brain tumor classification.

Shape Adaptive Neighborhood Information-Based Semi-Supervised Learning for Hyperspectral Image Classification

Hyperspectral image (HSI) classification is an important research topic in detailed analysis of the Earth’s surface. However, the performance of the classification is often hampered by the high-dimensionality features and limited training samples of the HSIs which has fostered research about semi-supervised learning (SSL). In this paper, we propose a shape adaptive neighborhood information (SANI) based SSL (SANI-SSL) method that takes full advantage of the adaptive spatial information to select valuable unlabeled samples in order to improve the classification ability. The improvement of the classification mainly relies on two aspects: (1) the improvement of the feature discriminability, which is accomplished by exploiting spectral-spatial information, and (2) the improvement of the training samples’ representativeness which is accomplished by exploiting the SANI for both labeled and unlabeled samples. First, the SANI of labeled samples is extracted, and the breaking ties (BT) method is used in order to select valuable unlabeled samples from the labeled samples’ neighborhood. Second, the SANI of unlabeled samples are also used to find more valuable samples, with the classifier combination method being used as a strategy to ensure confidence and the adaptive interval method used as a strategy to ensure informativeness. The experimental comparison results tested on three benchmark HSI datasets have demonstrated the significantly superior performance of our proposed method.

Application of Convolutional Neural Network-Based Feature Extraction and Data Fusion for Geographical Origin Identification of Radix Astragali by Visible/Short-Wave Near-Infrared and Near Infrared Hyperspectral Imaging

Radix Astragali is a prized traditional Chinese functional food that is used for both medicine and food purposes, with various benefits such as immunomodulation, anti-tumor, and anti-oxidation. The geographical origin of Radix Astragali has a significant impact on its quality attributes. Determining the geographical origins of Radix Astragali is essential for quality evaluation. Hyperspectral imaging covering the visible/short-wave near-infrared range (Vis-NIR, 380-1030 nm) and near-infrared range (NIR, 874-1734 nm) were applied to identify Radix Astragali from five different geographical origins. Principal component analysis (PCA) was utilized to form score images to achieve preliminary qualitative identification. PCA and convolutional neural network (CNN) were used for feature extraction. Measurement-level fusion and feature-level fusion were performed on the original spectra at different spectral ranges and the corresponding features. Support vector machine (SVM), logistic regression (LR), and CNN models based on full wavelengths, extracted features, and fusion datasets were established with excellent results; all the models obtained an accuracy of over 98% for different datasets. The results illustrate that hyperspectral imaging combined with CNN and fusion strategy could be an effective method for origin identification of Radix Astragali.

A Lightweight Spectral–Spatial Feature Extraction and Fusion Network for Hyperspectral Image Classification

Hyperspectral image (HSI) classification accuracy has been greatly improved by employing deep learning. The current research mainly focuses on how to build a deep network to improve the accuracy. However, these networks tend to be more complex and have more parameters, which makes the model difficult to train and easy to overfit. Therefore, we present a lightweight deep convolutional neural network (CNN) model called S2FEF-CNN. In this model, three S2FEF blocks are used for the joint spectral–spatial features extraction. Each S2FEF block uses 1D spectral convolution to extract spectral features and 2D spatial convolution to extract spatial features, respectively, and then fuses spectral and spatial features by multiplication. Instead of using the full connected layer, two pooling layers follow three blocks for dimension reduction, which further reduces the training parameters. We compared our method with some state-of-the-art HSI classification methods based on deep network on three commonly used hyperspectral datasets. The results show that our network can achieve a comparable classification accuracy with significantly reduced parameters compared to the above deep networks, which reflects its potential advantages in HSI classification.

Tumor tissue classification based on micro-hyperspectral technology and deep learning

In order to explore the application of hyperspectral technology in the pathological diagnosis of tumor tissue, we used microscopic hyperspectral imaging technology to establish a hyperspectral database of 30 patients with gastric cancer. Based on the difference in spectral-spatial features between gastric cancer tissue and normal tissue in the wavelength of 410-910 nm, we propose a deep-learning model-based analysis method for gastric cancer tissue. The microscopic hyperspectral feature and individual difference of gastric tissue, spatial-spectral joint feature and medical contact are studied. The experimental results show that the classification accuracy of proposed model for cancerous and normal gastric tissue is 97.57%, the sensitivity and specificity of gastric cancer tissue are 97.19% and 97.96% respectively. Compared with the shallow learning method, CNN can fully extract the deep spectral-spatial features of tumor tissue. The combination of deep learning model and micro-spectral analysis provides new ideas for the research of medical pathology.

Integrating the Continuous Wavelet Transform and a Convolutional Neural Network to Identify Vineyard Using Time Series Satellite Images

Grape is an economic crop of great importance and is widely cultivated in China. With the development of remote sensing, abundant data sources strongly guarantee that researchers can identify crop types and map their spatial distributions. However, to date, only a few studies have been conducted to identify vineyards using satellite image data. In this study, a vineyard is identified using satellite images, and a new approach is proposed that integrates the continuous wavelet transform (CWT) and a convolutional neural network (CNN). Specifically, the original time series of the normalized difference vegetation index (NDVI), enhanced vegetation index (EVI), and green chlorophyll vegetation index (GCVI) are reconstructed by applying an iterated Savitzky-Golay (S-G) method to form a daily time series for a full year; then, the CWT is applied to three reconstructed time series to generate corresponding scalograms; and finally, CNN technology is used to identify vineyards based on the stacked scalograms. In addition to our approach, a traditional and common approach that uses a random forest (RF) to identify crop types based on multi-temporal images is selected as the control group. The experimental results demonstrated the following: (i) the proposed approach was comprehensively superior to the RF approach; it improved the overall accuracy by 9.87% (up to 89.66%); (ii) the CWT had a stable and effective influence on the reconstructed time series, and the scalograms fully represented the unique time-related frequency pattern of each of the planting conditions; and (iii) the convolution and max pooling processing of the CNN captured the unique and subtle distribution patterns of the scalograms to distinguish vineyards from other crops. Additionally, the proposed approach is considered as able to be applied to other practical scenarios, such as using time series data to identify crop types, map landcover/land use, and is recommended to be tested in future practical applications.

Mapping Plastic Mulched Farmland for High Resolution Images of Unmanned Aerial Vehicle Using Deep Semantic Segmentation

With increasing consumption, plastic mulch benefits agriculture by promoting crop quality and yield, but the environmental and soil pollution is becoming increasingly serious. Therefore, research on the monitoring of plastic mulched farmland (PMF) has received increasing attention. Plastic mulched farmland in unmanned aerial vehicle (UAV) remote images due to the high resolution, shows a prominent spatial pattern, which brings difficulties to the task of monitoring PMF. In this paper, through a comparison between two deep semantic segmentation methods, SegNet and fully convolutional networks (FCN), and a traditional classification method, Support Vector Machine (SVM), we propose an end-to-end deep-learning method aimed at accurately recognizing PMF for UAV remote sensing images from Hetao Irrigation District, Inner Mongolia, China. After experiments with single-band, three-band and six-band image data, we found that deep semantic segmentation models built via single-band data which only use the texture pattern of PMF can identify it well; for example, SegNet reaching the highest accuracy of 88.68% in a 900 nm band. Furthermore, with three visual bands and six-band data (3 visible bands and 3 near-infrared bands), deep semantic segmentation models combining the texture and spectral features further improve the accuracy of PMF identification, whereas six-band data obtains an optimal performance for FCN and SegNet. In addition, deep semantic segmentation methods, FCN and SegNet, due to their strong feature extraction capability and direct pixel classification, clearly outperform the traditional SVM method in precision and speed. Among three classification methods, SegNet model built on three-band and six-band data obtains the optimal average accuracy of 89.62% and 90.6%, respectively. Therefore, the proposed deep semantic segmentation model, when tested against the traditional classification method, provides a promising path for mapping PMF in UAV remote sensing images.

Compressive spectral imaging system for soil classification with three-dimensional convolutional neural network

Compressive spectral imaging systems have promising applications in the field of object classification. However, for soil classification problem, conventional methods addressing this specific task often fail to produce satisfying results due to the tradeoff between the invariance and discrepancy of each soil. In this paper, we explore a liquid crystal tunable filters (LCTF)-based system and propose a three-dimensional convolutional neural network (3D-CNN) for soil classification. We first obtain a set of soil compressive measurements via a low spatial resolution detector, and soil hyperspectral images are reconstructed with improved resolution in spatial as well as spectral domains by a compressive sensing (CS) method. Furthermore, different from previous spectral-based object classification methods restricted to extract features from each type independently, on account of the potential of spectral property on individual solid, our method proposes to apply the principal component analysis(PCA) to achieve a dimensionality reduction in the spectral domain. Then, we explore a differential perception model for flexible feature extraction, and finally introduce a 3D-CNN framework to solve the multi-soil classification problem. Experimental results demonstrate that our algorithm not only is able to accelerate the ability of feature discriminability but also performs against conventional soil classification methods.

A Lightweight Hyperspectral Image Anomaly Detector for Real-Time Mission

In real-time onboard hyperspectral-image(HSI) anomalous targets detection, processing speed and accuracy are equivalently desirable which is hard to satisfy at the same time. To improve detection accuracy, deep learning based HSI anomaly detectors (ADs) are widely studied. However, their large scale network results in a massive computational burden. In this paper, to improve the detection throughput without sacrificing the accuracy, a pruning–quantization–anomaly–detector (P-Q-AD) is proposed by building an underlying constraint formulation to make a trade-off between accuracy and throughput. To solve this formulation, multi-objective optimization with nondominated sorting genetic algorithm II (NSGA-II) is employed to shrink the network. As a result, the redundant neurons are removed. A mixed precision network is implemented with a delicate customized fixed-point data expression to further improve the efficiency. In the experiments, the proposed P-Q-AD is implemented on two real HSI data sets and compared with three types of detectors. The results show that the performance of the proposed approach is no worse than those comparison detectors in terms of the receiver operating characteristic curve (ROC) and area under curve (AUC) value. For the onboard mission, the proposed P-Q-AD reaches over 4 . 5 × speedup with less than 0 . 5 % AUC loss compared with the floating-based detector. The pruning and the quantization approach in this paper can be referenced for designing the anomalous targets detectors for high efficiency.

Deep Learning Based Retrieval of Forest Aboveground Biomass from Combined LiDAR and Landsat 8 Data

Estimation of forest aboveground biomass (AGB) is crucial for various technical and scientific applications, ranging from regional carbon and bioenergy policies to sustainable forest management. However, passive optical remote sensing, which is the most widely used remote sensing data for retrieving vegetation parameters, is constrained by spectral saturation problems and cloud cover. On the other hand, LiDAR data, which have been extensively used to estimate forest structure attributes, cannot provide sufficient spectral information of vegetation canopies. Thus, this study aimed to develop a novel synergistic approach to estimating biomass by integrating LiDAR data with Landsat 8 imagery through a deep learning-based workflow. First the relationships between biomass and spectral vegetation indices (SVIs) and LiDAR metrics were separately investigated. Next, two groups of combined optical and LiDAR indices (i.e., COLI1 and COLI2) were designed and explored to identify their performances in biomass estimation. Finally, five prediction models, including K-nearest Neighbor, Random Forest, Support Vector Regression, the deep learning model, i.e., Stacked Sparse Autoencoder network (SSAE), and multiple stepwise linear regressions, were individually used to estimate biomass with input variables of different scenarios, i.e., (i) all the COLI1 (ACOLI1), (ii) all the COLI2 (ACOLI2), (iii) ACOLI1 and all the optical (AO) and LiDAR variables (AL), and (iv) ACOLI2, AO and AL. Results showed that univariate models with the combined optical and LiDAR indices as explanatory variables presented better modeling performance than those with either optical or LiDAR data alone, regardless of the combination mode. The SSAE model obtained the best performance compared to the other tested prediction algorithms for the forest biomass estimation. The best predictive accuracy was achieved by the SSAE model with inputs of combined optical and LiDAR variables (i.e., ACOLI1, AO and AL) that yielded an R2 of 0.935, root mean squared error (RMSE) of 15.67 Mg/ha, and relative root mean squared error (RMSEr) of 11.407%. It was concluded that the presented combined indices were simple and effective by integrating LiDAR-derived structure information with Landsat 8 spectral data for estimating forest biomass. Overall, the SSAE model with inputs of Landsat 8 and LiDAR integrated information resulted in accurate estimation of forest biomass. The presented modeling workflow will greatly facilitate future forest biomass estimation and carbon stock assessments.

Double-Branch Multi-Attention Mechanism Network for Hyperspectral Image Classification

Recently, Hyperspectral Image (HSI) classification has gradually been getting attention from more and more researchers. HSI has abundant spectral and spatial information; thus, how to fuse these two types of information is still a problem worth studying. In this paper, to extract spectral and spatial feature, we propose a Double-Branch Multi-Attention mechanism network (DBMA) for HSI classification. This network has two branches to extract spectral and spatial feature respectively which can reduce the interference between the two types of feature. Furthermore, with respect to the different characteristics of these two branches, two types of attention mechanism are applied in the two branches respectively, which ensures to extract more discriminative spectral and spatial feature. The extracted features are then fused for classification. A lot of experiment results on three hyperspectral datasets shows that the proposed method performs better than the state-of-the-art method.

A Spectral Feature Based Convolutional Neural Network for Classification of Sea Surface Oil Spill

Spectral characteristics play an important role in the classification of oil film, but the presence of too many bands can lead to information redundancy and reduced classification accuracy. In this study, a classification model that combines spectral indices-based band selection (SIs) and one-dimensional convolutional neural networks was proposed to realize automatic oil films classification using hyperspectral remote sensing images. Additionally, for comparison, the minimum Redundancy Maximum Relevance (mRMR) was tested for reducing the number of bands. The support vector machine (SVM), random forest (RF), and Hu’s convolutional neural networks (CNN) were trained and tested. The results show that the accuracy of classifications through the one dimensional convolutional neural network (1D CNN) models surpassed the accuracy of other machine learning algorithms such as SVM and RF. The model of SIs+1D CNN could produce a relatively higher accuracy oil film distribution map within less time than other models.

Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series

Latest remote sensing sensors are capable of acquiring high spatial and spectral Satellite Image Time Series (SITS) of the world. These image series are a key component of classification systems that aim at obtaining up-to-date and accurate land cover maps of the Earth’s surfaces. More specifically, current SITS combine high temporal, spectral and spatial resolutions, which makes it possible to closely monitor vegetation dynamics. Although traditional classification algorithms, such as Random Forest (RF), have been successfully applied to create land cover maps from SITS, these algorithms do not make the most of the temporal domain. This paper proposes a comprehensive study of Temporal Convolutional Neural Networks (TempCNNs), a deep learning approach which applies convolutions in the temporal dimension in order to automatically learn temporal (and spectral) features. The goal of this paper is to quantitatively and qualitatively evaluate the contribution of TempCNNs for SITS classification, as compared to RF and Recurrent Neural Networks (RNNs) —a standard deep learning approach that is particularly suited to temporal data. We carry out experiments on Formosat-2 scene with 46 images and one million labelled time series. The experimental results show that TempCNNs are more accurate than the current state of the art for SITS classification. We provide some general guidelines on the network architecture, common regularization mechanisms, and hyper-parameter values such as batch size; we also draw out some differences with standard results in computer vision (e.g., about pooling layers). Finally, we assess the visual quality of the land cover maps produced by TempCNNs.

Exploring Spatial Influence of Remotely Sensed PM2.5 Concentration Using a Developed Deep Convolutional Neural Network Model

Currently, more and more remotely sensed data are being accumulated, and the spatial analysis methods for remotely sensed data, especially big data, are desiderating innovation. A deep convolutional network (CNN) model is proposed in this paper for exploiting the spatial influence feature in remotely sensed data. The method was applied in investigating the magnitude of the spatial influence of four factors—population, gross domestic product (GDP), terrain, land-use and land-cover (LULC)—on remotely sensed PM2.5 concentration over China. Satisfactory results were produced by the method. It demonstrates that the deep CNN model can be well applied in the field of spatial analysing remotely sensed big data. And the accuracy of the deep CNN is much higher than of geographically weighted regression (GWR) based on comparation. The results showed that population spatial density, GDP spatial density, terrain, and LULC could together determine the spatial distribution of PM2.5 annual concentrations with an overall spatial influencing magnitude of 97.85%. Population, GDP, terrain, and LULC have individual spatial influencing magnitudes of 47.12% and 36.13%, 50.07% and 40.91% on PM2.5 annual concentrations respectively. Terrain and LULC are the dominating spatial influencing factors, and only these two factors together may approximately determine the spatial pattern of PM2.5 annual concentration over China with a high spatial influencing magnitude of 96.65%.

Deep Convolutional Capsule Network for Hyperspectral Image Spectral and Spectral-Spatial Classification

Capsule networks can be considered to be the next era of deep learning and have recently shown their advantages in supervised classification. Instead of using scalar values to represent features, the capsule networks use vectors to represent features, which enriches the feature presentation capability. This paper introduces a deep capsule network for hyperspectral image (HSI) classification to improve the performance of the conventional convolutional neural networks (CNNs). Furthermore, a modification of the capsule network named Conv-Capsule is proposed. Instead of using full connections, local connections and shared transform matrices, which are the core ideas of CNNs, are used in the Conv-Capsule network architecture. In Conv-Capsule, the number of trainable parameters is reduced compared to the original capsule, which potentially mitigates the overfitting issue when the number of available training samples is limited. Specifically, we propose two schemes: (1) A 1D deep capsule network is designed for spectral classification, as a combination of principal component analysis, CNN, and the Conv-Capsule network, and (2) a 3D deep capsule network is designed for spectral-spatial classification, as a combination of extended multi-attribute profiles, CNN, and the Conv-Capsule network. The proposed classifiers are tested on three widely-used hyperspectral data sets. The obtained results reveal that the proposed models provide competitive results compared to the state-of-the-art methods, including kernel support vector machines, CNNs, and recurrent neural network.

Spectral-Spatial Classification of Hyperspectral Images: Three Tricks and a New Learning Setting

Spectral-spatial classification of hyperspectral images has been the subject of many studies in recent years. When there are only a few labeled pixels for training and a skewed class label distribution, this task becomes very challenging because of the increased risk of overfitting when training a classifier. In this paper, we show that in this setting, a convolutional neural network with a single hidden layer can achieve state-of-the-art performance when three tricks are used: a spectral-locality-aware regularization term and smoothing- and label-based data augmentation. The shallow network architecture prevents overfitting in the presence of many features and few training samples. The locality-aware regularization forces neighboring wavelengths to have similar contributions to the features generated during training. The new data augmentation procedure favors the selection of pixels in smaller classes, which is beneficial for skewed class label distributions. The accuracy of the proposed method is assessed on five publicly available hyperspectral images, where it achieves state-of-the-art results. As other spectral-spatial classification methods, we use the entire image (labeled and unlabeled pixels) to infer the class of its unlabeled pixels. To investigate the positive bias induced by the use of the entire image, we propose a new learning setting where unlabeled pixels are not used for building the classifier. Results show the beneficial effect of the proposed tricks also in this setting and substantiate the advantages of using labeled and unlabeled pixels from the image for hyperspectral image classification.

Bidirectional-Convolutional LSTM Based Spectral-Spatial Feature Learning for Hyperspectral Image Classification

This paper proposes a novel deep learning framework named bidirectional-convolutional long short term memory (Bi-CLSTM) network to automatically learn the spectral-spatial features from hyperspectral images (HSIs). In the network, the issue of spectral feature extraction is considered as a sequence learning problem, and a recurrent connection operator across the spectral domain is used to address it. Meanwhile, inspired from the widely used convolutional neural network (CNN), a convolution operator across the spatial domain is incorporated into the network to extract the spatial feature. In addition, to sufficiently capture the spectral information, a bidirectional recurrent connection is proposed. In the classification phase, the learned features are concatenated into a vector and fed to a Softmax classifier via a fully-connected operator. To validate the effectiveness of the proposed Bi-CLSTM framework, we compare it with six state-of-the-art methods, including the popular 3D-CNN model, on three widely used HSIs (i.e., Indian Pines, Pavia University, and Kennedy Space Center). The obtained results show that Bi-CLSTM can improve the classification performance by almost 1.5 % as compared to 3D-CNN.

A Hyperspectral Image Classification Framework with Spatial Pixel Pair Features

During recent years, convolutional neural network (CNN)-based methods have been widely applied to hyperspectral image (HSI) classification by mostly mining the spectral variabilities. However, the spatial consistency in HSI is rarely discussed except as an extra convolutional channel. Very recently, the development of pixel pair features (PPF) for HSI classification offers a new way of incorporating spatial information. In this paper, we first propose an improved PPF-style feature, the spatial pixel pair feature (SPPF), that better exploits both the spatial/contextual information and spectral information. On top of the new SPPF, we further propose a flexible multi-stream CNN-based classification framework that is compatible with multiple in-stream sub-network designs. The proposed SPPF is different from the original PPF in its paring pixel selection strategy: only pixels immediately adjacent to the central one are eligible, therefore imposing stronger spatial regularization. Additionally, with off-the-shelf classification sub-network designs, the proposed multi-stream, late-fusion CNN-based framework outperforms competing ones without requiring extensive network configuration tuning. Experimental results on three publicly available datasets demonstrate the performance of the proposed SPPF-based HSI classification framework.