A new learning paradigm for random vector functional-link network: RVFL+

Quantized minimum error entropy with fiducial points for robust regression

Minimum error entropy with fiducial points (MEEF) has received a lot of attention, due to its outstanding performance to curb the negative influence caused by non-Gaussian noises in the fields of machine learning and signal processing. However, the estimate of the information potential of MEEF involves a double summation operator based on all available error samples, which can result in large computational burden in many practical scenarios. In this paper, an efficient quantization method is therefore adopted to represent the primary set of error samples with a smaller subset, generating a quantized MEEF (QMEEF). Some basic properties of QMEEF are presented and proved from theoretical perspectives. In addition, we have applied this new criterion to train a class of linear-in-parameters models, including the commonly used linear regression model, random vector functional link network, and broad learning system as special cases. Experimental results on various datasets are reported to demonstrate the desirable performance of the proposed methods to perform regression tasks with contaminated data.

Osteoarthritis Diagnosis Integrating Whole Joint Radiomics and Clinical Features for Robust Learning Models Using Biological Privileged Information

This paper proposes a machine learning model using privileged information (LUPI) and normalized mutual information feature selection method (NMIFS) to build a robust and accurate framework to diagnose patients with Temporomandibular Joint Osteoarthritis (TMJ OA). To build such a model, we employ clinical, quantitative imaging and additional biological markers as privileged information. We show that clinical features play a leading role in the TMJ OA diagnosis and quantitative imaging features, extracted from cone-beam computerized tomography (CBCT) scans, improve the model performance. As the proposed LUPI model employs biological data in the training phase (which boosted the model performance), this data is unnecessary for the testing stage, indicating the model can be widely used even when only clinical and imaging data are collected. The model was validated using 5-fold stratified cross-validation with hyperparameter tuning to avoid the bias of data splitting. Our method achieved an AUC, specificity and precision of 0.81, 0.79 and 0.77, respectively.

Online dynamic ensemble deep random vector functional link neural network for forecasting

This paper proposes a three-stage online deep learning model for time series based on the ensemble deep random vector functional link (edRVFL). The edRVFL stacks multiple randomized layers to enhance the single-layer RVFL's representation ability. Each hidden layer's representation is utilized for training an output layer, and the ensemble of all output layers forms the edRVFL's output. However, the original edRVFL is not designed for online learning, and the randomized nature of the features is harmful to extracting meaningful temporal features. In order to address the limitations and extend the edRVFL to an online learning mode, this paper proposes a dynamic edRVFL consisting of three online components, the online decomposition, the online training, and the online dynamic ensemble. First, an online decomposition is utilized as a feature engineering block for the edRVFL. Then, an online learning algorithm is designed to learn the edRVFL. Finally, an online dynamic ensemble method, which can measure the change in the distribution, is proposed for aggregating all layers' outputs. This paper evaluates and compares the proposed model with state-of-the-art methods on sixteen time series.

B-mode ultrasound based CAD for liver cancers via multi-view privileged information learning

B-mode ultrasound-based computer-aided diagnosis model can help sonologists improve the diagnostic performance for liver cancers, but it generally suffers from the bottleneck due to the limited structure and internal echogenicity information in B-mode ultrasound images. Contrast-enhanced ultrasound images provide additional diagnostic information on dynamic blood perfusion of liver lesions for B-mode ultrasound images with improved diagnostic accuracy. Since transfer learning has indicated its effectiveness in promoting the performance of target computer-aided diagnosis model by transferring knowledge from related imaging modalities, a multi-view privileged information learning framework is proposed to improve the diagnostic accuracy of the single-modal B-mode ultrasound-based diagnosis for liver cancers. This framework can make full use of the shared label information between the paired B-mode ultrasound images and contrast-enhanced ultrasound images to guide knowledge transfer It consists of a novel supervised dual-view deep Boltzmann machine and a new deep multi-view SVM algorithm. The former is developed to implement knowledge transfer from the multi-phase contrast-enhanced ultrasound images to the B-mode ultrasound-based diagnosis model via a feature-level learning using privileged information paradigm, which is totally different from the existing learning using privileged information paradigm that performs knowledge transfer in the classifier. The latter further fuses and enhances feature representation learned from three pre-trained supervised dual-view deep Boltzmann machine networks for the classification task. An experiment is conducted on a bimodal ultrasound liver cancer dataset. The experimental results show that the proposed framework outperforms all the compared algorithms with the best classification accuracy of 88.91 ± 1.52%, sensitivity of 88.31 ± 2.02%, and specificity of 89.50 ± 3.12%. It suggests the effectiveness of our proposed MPIL framework for the BUS-based CAD of liver cancers.

Tensor based stacked fuzzy neural network for efficient data regression

Random vector functional link and extreme learning machine have been extended by the type-2 fuzzy sets with vector stacked methods, this extension leads to a new way to use tensor to construct learning structure for the type-2 fuzzy sets-based learning framework. In this paper, type-2 fuzzy sets-based random vector functional link, type-2 fuzzy sets-based extreme learning machine and Tikhonov-regularized extreme learning machine are fused into one network, a tensor way of stacking data is used to incorporate the nonlinear mappings when using type-2 fuzzy sets. In this way, the network could learn the sub-structure by three sub-structures' algorithms, which are merged into one tensor structure via the type-2 fuzzy mapping results. To the stacked single fuzzy neural network, the consequent part parameters learning is implemented by unfolding tensor-based matrix regression. The newly proposed stacked single fuzzy neural network shows a new way to design the hybrid fuzzy neural network with the higher order fuzzy sets and higher order data structure. The effective of the proposed stacked single fuzzy neural network are verified by the classical testing benchmarks and several statistical testing methods.

Doubly Supervised Transfer Classifier for Computer-Aided Diagnosis With Imbalanced Modalities

Transfer learning (TL) can effectively improve diagnosis accuracy of single-modal-imaging-based computer-aided diagnosis (CAD) by transferring knowledge from other related imaging modalities, which offers a way to alleviate the small-sample-size problem. However, medical imaging data generally have the following characteristics for the TL-based CAD: 1) The source domain generally has limited data, which increases the difficulty to explore transferable information for the target domain; 2) Samples in both domains often have been labeled for training the CAD model, but the existing TL methods cannot make full use of label information to improve knowledge transfer. In this work, we propose a novel doubly supervised transfer classifier (DSTC) algorithm. In particular, DSTC integrates the support vector machine plus (SVM+) classifier and the low-rank representation (LRR) into a unified framework. The former makes full use of the shared labels to guide the knowledge transfer between the paired data, while the latter adopts the block-diagonal low-rank (BLR) to perform supervised TL between the unpaired data. Furthermore, we introduce the Schatten-p norm for BLR to obtain a tighter approximation to the rank function. The proposed DSTC algorithm is evaluated on the Alzheimer's disease neuroimaging initiative (ADNI) dataset and the bimodal breast ultrasound image (BBUI) dataset. The experimental results verify the effectiveness of the proposed DSTC algorithm.

1-Norm random vector functional link networks for classification problems

This paper presents a novel random vector functional link (RVFL) formulation called the 1-norm RVFL (1N RVFL) networks, for solving the binary classification problems. The solution to the optimization problem of 1N RVFL is obtained by solving its exterior dual penalty problem using a Newton technique. The 1-norm makes the model robust and delivers sparse outputs, which is the fundamental advantage of this model. The sparse output indicates that most of the elements in the output matrix are zero; hence, the decision function can be achieved by incorporating lesser hidden nodes compared to the conventional RVFL model. 1N RVFL produces a classifier that is based on a smaller number of input features. To put it another way, this method will suppress the neurons in the hidden layer. Statistical analyses have been carried out on several real-world benchmark datasets. The proposed 1N RVFL with two activation functions viz., ReLU and sine are used in this work. The classification accuracies of 1N RVFL are compared with the extreme learning machine (ELM), kernel ridge regression (KRR), RVFL, kernel RVFL (K-RVFL) and generalized Lagrangian twin RVFL (GLTRVFL) networks. The experimental results with comparable or better accuracy indicate the effectiveness and usability of 1N RVFL for solving binary classification problems.

Performance Enhancement of the Unbalanced Text Classification Problem Through a Modified Chi Square-Based Feature Selection Technique

This paper proposes a modified chi square-based feature selection algorithm in conjunction with a random vector functional link network-based text classifier for improving the classification performance of multi-labeled text documents with unbalanced class distributions. In the proposed feature selection method, maximum features are selected from classes that have a great deal of training and testing documents as an improvement towards original chi-square method. On two benchmark datasets that are multi-labeled, multi-class, and unbalanced, a comparison of the model with three conventional selection techniques such as chi-square, term frequency-inverse document frequency, and mutual information is accumulated for assessing its effectiveness. Additionally, the proposed model is compared with four different classifiers. In the study, it was found that the proposed model performs better in terms of precision, recall, f-measure, and hamming losses and is able to select the majority of true positive documents despite an unbalanced class distribution for both the datasets.

Self-Paced Learning and Privileged Information based Cascaded Multi-column Classification algorithm for ASD diagnosis

Autism spectrum disorder (ASD) is one of the most serious mental disorder in children. Machine learning based computer aided diagnosis (CAD) on resting-state functional magnetic resonance imaging (rs-fMRI) for ASD has attracted widespread attention. In recent years, learning using privileged information (LUPI), a supervised transfer learning method, has been generally used on multi-modality cases, which can transfer knowledge from source domain to target domain in order to improve the prediction capability on the target domain. However, multi-modality data is difficult to collect in clinical cases. LUPI method without introducing additional imaging modality images is worth further study. Random vector function link network plus (RVFL+) is a LUPI diagnosis algorithm, which has been proven to be effective for classification tasks. In this work, we proposed a self-paced learning based cascaded multi-column RVFL+ algorithm (SPL-cmcRVFL+) for ASD diagnosis. Initial classification model is trained using RVFL on the single-modal data (e.g. rs-fMRI). The output of the initial layer is then sent as privileged information (PI) to train the next layer of classification model. During this process, samples are selected using self-paced learning (SPL), which can adaptively select simple to difficult samples according to the loss value. The procedure is repeated until all samples are included. Experimental results show that our proposed method can accurately identify ASD and normal control, and outperforms other methods by a relatively higher classification accuracy.

Temporomandibular Joint Osteoarthritis Diagnosis Using Privileged Learning of Protein Markers

Diagnosis of temporomandibular joint (TMJ) Osteoarthritis (OA) before serious degradation of cartilage and subchondral bone occurs can help prevent chronic pain and disability. Clinical, radiomic, and protein markers collected from TMJ OA patients have been shown to be predictive of OA onset. Since protein data can often be unavailable for clinical diagnosis, we harnessed the learning using privileged information (LUPI) paradigm to make use of protein markers only during classifier training. Three different LUPI algorithms are compared with traditional machine learning models on a dataset extracted from 92 unique OA patients and controls. The best classifier performance of 0.80 AUC and 75.6 accuracy was obtained from the KRVFL+ model using privileged protein features. Results show that LUPI-based algorithms using privileged protein data can improve final diagnostic performance of TMJ OA classifiers without needing protein microarray data during classifier diagnosis.

A turning point prediction method of stock price based on RVFL-GMDH and chaotic time series analysis

Stock market prediction is extremely important for investors because knowing the future trend of stock prices will reduce the risk of investing capital for profit. Therefore, seeking an accurate, fast, and effective approach to identify the stock market movement is of great practical significance. This study proposes a novel turning point prediction method for the time series analysis of stock price. Through the chaos theory analysis and application, we put forward a new modeling approach for the nonlinear dynamic system. The turning indicator of time series is computed firstly; then, by applying the RVFL-GMDH model, we perform the turning point prediction of the stock price, which is based on the fractal characteristic of a strange attractor with an infinite self-similar structure. The experimental findings confirm the efficacy of the proposed procedure and have become successful for the intelligent decision support of the stock trading strategy.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10115-021-01602-3.

Improving MRI-based analysis of brain structural changes in patients with hypertension via a privileged information learning algorithm

Hypertension can lead to changes in the brain structure and function, and different blood pressure levels (2017ACC/AHA) have different effects on brain structure. It is important to analyze these changes by machine learning methods, and various characteristics can provide rich information for the analysis of these changes. However, multiple feature extraction involves complex data processing. How to make a single feature achieve the same diagnosis effect as multiple features do is worth of study. Kernel ridge regression (KRR) is a kind of machine learning method, which shows faster learning speed and generalization ability in classification tasks. In order to knowledge transfer, we use privileged information (PI) to transfer information of multiple types of feature to single feature. This allows only one feature type to be used during the test stage. In the process of feature fusion, we need to consider all the samples' attribution making the classifier better. In this work, we propose a multi-kernel KRR+ framework based on self-paced learning to analyze the changes of the brain structure in patients with different blood pressure levels. Specifically, one kind of a feature is taken as main feature, and other features are input into the multi-kernel KRR as PI. These two inputs are fed into the final KRR classifier together. In addition, a self-paced learning method is introduced into sample selecting to avoid training the classifier using samples with a large loss value firstly, which improves the generalization performance of the classifier. Experimental results show that the proposed method can make full use of the information of various features and achieve better classification performance. This shows self-paced learning based KRR can help analyze brain structure of patients with different blood pressure levels. The discriminative features may help clinicians to make judgments of hypertension degrees on brain MRI images.

A Novel Functional Link Network Stacking Ensemble with Fractal Features for Multichannel Fall Detection

Falls are a major health concern and result in high morbidity and mortality rates in older adults with high costs to health services. Automatic fall classification and detection systems can provide early detection of falls and timely medical aid. This paper proposes a novel Random Vector Functional Link (RVFL) stacking ensemble classifier with fractal features for classification of falls. The fractal Hurst exponent is used as a representative of fractal dimensionality for capturing irregularity of accelerometer signals for falls and other activities of daily life. The generalised Hurst exponents along with wavelet transform coefficients are leveraged as input feature space for a novel stacking ensemble of RVFLs composed with an RVFL neural network meta-learner. Novel fast selection criteria are presented for base classifiers founded on the proposed diversity indicator, obtained from the overall performance values during the training phase. The proposed features and the stacking ensemble provide the highest classification accuracy of 95.71% compared with other machine learning techniques, such as Random Forest (RF), Artificial Neural Network (ANN) and Support Vector Machine. The proposed ensemble classifier is 2.3× faster than a single Decision Tree and achieves the highest speedup in training time of 317.7× and 198.56× compared with a highly optimised ANN and RF ensemble, respectively. The significant improvements in training times of the order of 100× and high accuracy demonstrate that the proposed RVFL ensemble is a prime candidate for real-time, embedded wearable device–based fall detection systems.

Analyzing brain structural differences associated with categories of blood pressure in adults using empirical kernel mapping-based kernel ELM

Background

Hypertension increases the risk of angiocardiopathy and cognitive disorder. Blood pressure has four categories: normal, elevated, hypertension stage 1 and hypertension stage 2. The quantitative analysis of hypertension helps determine disease status, prognosis assessment, guidance and management, but is not well studied in the framework of machine learning.

Methods

We proposed empirical kernel mapping-based kernel extreme learning machine plus (EKM–KELM+) classifier to discriminate different blood pressure grades in adults from structural brain MR images. ELM+ is the extended version of ELM, which integrates the additional privileged information about training samples in ELM to help train a more effective classifier. In this work, we extracted gray matter volume (GMV), white matter volume, cerebrospinal fluid volume, cortical surface area, cortical thickness from structural brain MR images, and constructed brain network features based on thickness. After feature selection and EKM, the enhanced features are obtained. Then, we select one feature type as the main feature to feed into KELM+, and the rest of the feature types are PI to assist the main feature to train 5 KELM+ classifiers. Finally, the 5 KELM+ classifiers are ensemble to predict classification result in the test stage, while PI is not used during testing.

Results

We evaluated the performance of the proposed EKM–KELM+ method using four grades of hypertension data (73 samples for each grade). The experimental results show that the GMV performs observably better than any other feature types with a comparatively higher classification accuracy of 77.37% (Grade 1 vs. Grade 2), 93.19% (Grade 1 vs. Grade 3), and 95.15% (Grade 1 vs. Grade 4). The most discriminative brain regions found using our method are olfactory, orbitofrontal cortex (inferior), supplementary motor area, etc.

Conclusions

Using region of interest features and brain network features, EKM–KELM+ is proposed to study the most discriminative regions that have obvious structural changes in different blood pressure grades. The discriminative features that are selected using our method are consistent with the existing neuroimaging studies. Moreover, our study provides a potential approach to take effective interventions in the early period, when the blood pressure makes minor impacts on the brain structure and function.