An analytical approach for unsupervised learning rate estimation using rectified linear units

Unsupervised learning based on restricted Boltzmann machine or autoencoders has become an important research domain in the area of neural networks. In this paper mathematical expressions to adaptive learning step calculation for RBM with ReLU transfer function are proposed. As a result, we can automatically estimate the step size that minimizes the loss function of the neural network and correspondingly update the learning step in every iteration. We give a theoretical justification for the proposed adaptive learning rate approach, which is based on the steepest descent method. The proposed technique for adaptive learning rate estimation is compared with the existing constant step and Adam methods in terms of generalization ability and loss function. We demonstrate that the proposed approach provides better performance.

nPCA: a linear dimensionality reduction method using a multilayer perceptron

Background: Linear dimensionality reduction techniques are widely used in many applications. The goal of dimensionality reduction is to eliminate the noise of data and extract the main features of data. Several dimension reduction methods have been developed, such as linear-based principal component analysis (PCA), nonlinear-based t-distributed stochastic neighbor embedding (t-SNE), and deep-learning-based autoencoder (AE). However, PCA only determines the projection direction with the highest variance, t-SNE is sometimes only suitable for visualization, and AE and nonlinear methods discard the linear projection. Results: To retain the linear projection of raw data and generate a better result of dimension reduction either for visualization or downstream analysis, we present neural principal component analysis (nPCA), an unsupervised deep learning approach capable of retaining richer information of raw data as a promising improvement to PCA. To evaluate the performance of the nPCA algorithm, we compare the performance of 10 public datasets and 6 single-cell RNA sequencing (scRNA-seq) datasets of the pancreas, benchmarking our method with other classic linear dimensionality reduction methods. Conclusion: We concluded that the nPCA method is a competitive alternative method for dimensionality reduction tasks.

Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation

Medical image segmentation primarily utilizes a hybrid model consisting of a Convolutional Neural Network and sequential Transformers. The latter leverage multi-head self-attention mechanisms to achieve comprehensive global context modelling. However, despite their success in semantic segmentation, the feature extraction process is inefficient and demands more computational resources, which hinders the network's robustness. To address this issue, this study presents two innovative methods: PTransUNet (PT model) and C-PTransUNet (C-PT model). The C-PT module refines the Vision Transformer by substituting a sequential design with a parallel one. This boosts the feature extraction capabilities of Multi-Head Self-Attention via self-correlated feature attention and channel feature interaction, while also streamlining the Feed-Forward Network to lower computational demands. On the Synapse public dataset, the PT and C-PT models demonstrate improvements in DSC accuracy by 0.87% and 3.25%, respectively, in comparison with the baseline model. As for the parameter count and FLOPs, the PT model aligns with the baseline model. In contrast, the C-PT model shows a decrease in parameter count by 29% and FLOPs by 21.4% relative to the baseline model. The proposed segmentation models in this study exhibit benefits in both accuracy and efficiency.

Deep Learning Techniques for Vehicle Detection and Classification from Images/Videos: A Survey

Detecting and classifying vehicles as objects from images and videos is challenging in appearance-based representation, yet plays a significant role in the substantial real-time applications of Intelligent Transportation Systems (ITSs). The rapid development of Deep Learning (DL) has resulted in the computer-vision community demanding efficient, robust, and outstanding services to be built in various fields. This paper covers a wide range of vehicle detection and classification approaches and the application of these in estimating traffic density, real-time targets, toll management and other areas using DL architectures. Moreover, the paper also presents a detailed analysis of DL techniques, benchmark datasets, and preliminaries. A survey of some vital detection and classification applications, namely, vehicle detection and classification and performance, is conducted, with a detailed investigation of the challenges faced. The paper also addresses the promising technological advancements of the last few years.

An Optimized Convolutional Neural Network for the 3D Point-Cloud Compression

Due to the tremendous volume taken by the 3D point-cloud models, knowing how to achieve the balance between a high compression ratio, a low distortion rate, and computing cost in point-cloud compression is a significant issue in the field of virtual reality (VR). Convolutional neural networks have been used in numerous point-cloud compression research approaches during the past few years in an effort to progress the research state. In this work, we have evaluated the effects of different network parameters, including neural network depth, stride, and activation function on point-cloud compression, resulting in an optimized convolutional neural network for compression. We first have analyzed earlier research on point-cloud compression based on convolutional neural networks before designing our own convolutional neural network. Then, we have modified our model parameters using the experimental data to further enhance the effect of point-cloud compression. Based on the experimental results, we have found that the neural network with the 4 layers and 2 strides parameter configuration using the Sigmoid activation function outperforms the default configuration by 208% in terms of the compression-distortion rate. The experimental results show that our findings are effective and universal and make a great contribution to the research of point-cloud compression using convolutional neural networks.

DeepAProt: Deep learning based abiotic stress protein sequence classification and identification tool in cereals

The impact of climate change has been alarming for the crop growth. The extreme weather conditions can stress the crops and reduce the yield of major crops belonging to Poaceae family too, that sustains 50% of the world's food calorie and 20% of protein intake. Computational approaches, such as artificial intelligence-based techniques have become the forefront of prediction-based data interpretation and plant stress responses. In this study, we proposed a novel activation function, namely, Gaussian Error Linear Unit with Sigmoid (SIELU) which was implemented in the development of a Deep Learning (DL) model along with other hyper parameters for classification of unknown abiotic stress protein sequences from crops of Poaceae family. To develop this models, data pertaining to four different abiotic stress (namely, cold, drought, heat and salinity) responsive proteins of the crops belonging to poaceae family were retrieved from public domain. It was observed that efficiency of the DL models with our proposed novel SIELU activation function outperformed the models as compared to GeLU activation function, SVM and RF with 95.11%, 80.78%, 94.97%, and 81.69% accuracy for cold, drought, heat and salinity, respectively. Also, a web-based tool, named DeepAProt (http://login1.cabgrid.res.in:5500/) was developed using flask API, along with its mobile app. This server/App will provide researchers a convenient tool, which is rapid and economical in identification of proteins for abiotic stress management in crops Poaceae family, in endeavour of higher production for food security and combating hunger, ensuring UN SDG goal 2.0.

A robust zeroing neural network and its applications to dynamic complex matrix equation solving and robotic manipulator trajectory tracking

Dynamic complex matrix equation (DCME) is frequently encountered in the fields of mathematics and industry, and numerous recurrent neural network (RNN) models have been reported to effectively find the solution of DCME in no noise environment. However, noises are unavoidable in reality, and dynamic systems must be affected by noises. Thus, the invention of anti-noise neural network models becomes increasingly important to address this issue. By introducing a new activation function (NAF), a robust zeroing neural network (RZNN) model for solving DCME in noisy-polluted environment is proposed and investigated in this paper. The robustness and convergence of the proposed RZNN model are proved by strict mathematical proof and verified by comparative numerical simulation results. Furthermore, the proposed RZNN model is applied to manipulator trajectory tracking control, and it completes the trajectory tracking task successfully, which further validates its practical applied prospects.

Position Control of a Pneumatic Drive Using a Fuzzy Controller with an Analytic Activation Function

The fuzzy logic controller, which uses an analytic activation function for the defuzzification procedure, was applied to the position control of a servo pneumatic drive controlled by a proportional valve. The Gaussian shape of input fuzzy sets, with the possibility of their modification, was used to fuzzify the input signal. The control signal was determined by introducing an analytic function instead of defining the fuzzy rule base. In this way, a conventional 2-D fuzzy rule table base is modified into 1-D fuzzy defuzzification based on an analytic function to calculate the controller output. In this control algorithm, the problem of conventional fuzzy logic control, in terms of the exponential growth in rules as the number of input variables increases, is eliminated. The synthesis controller procedure is adjusted to the flow rate characteristic of the proportional valve. The developed control algorithms are verified by computer simulation and by testing on a real pneumatic rodless cylindrical drive.

Dynamic Perturbation of Weights for Improved Data Reconstruction in Unsupervised Learning

The concept of weight pruning has shown success in neural network model compression with marginal loss in classification performance. However, similar concepts have not been well recognized in improving unsupervised learning. To the best of our knowledge, this paper proposes one of the first studies on weight pruning in unsupervised autoencoder models using non-imaging data points. We adapt the weight pruning concept to investigate the dynamic behavior of weights while reconstructing data using an autoencoder and propose a deterministic model perturbation algorithm based on the weight statistics. The model perturbation at periodic intervals resets a percentage of weight values using a binary weight mask. Experiments across eight non-imaging data sets ranging from gene sequence to swarm behavior data show that only a few periodic perturbations of weights improve the data reconstruction accuracy of autoencoders and additionally introduce model compression. All data sets yield a small portion of (<5%) weights that are substantially higher than the mean weight value. These weights are found to be much more informative than a substantial portion (>90%) of the weights with negative values. In general, the perturbation of low or negative weight values at periodic intervals has improved the data reconstruction loss for most data sets when compared to the case without perturbation. The proposed approach may help explain and correct the dynamic behavior of neural network models in a deterministic way for data reconstruction and obtaining a more accurate representation of latent variables using autoencoders.

Development of pedestrian crash prediction model for a developing country using artificial neural network

Urban intersections in India constitute a significant share of pedestrian fatalities. However, model-based prediction of pedestrian fatalities is still in a nascent stage in India. This study proposes an artificial neural network (ANN) technique to develop a pedestrian fatal crash frequency model at the intersection level. In this study, three activation functions are used along with four different learning algorithms to build different combinations of ANN models. In each of these combinations, the number of neurons in the hidden layer is varied by trial and error method, and the best results are considered. In this way, 12 sets of pedestrian fatal crash predictive models are developed. Out of these, Bayesian Regularization Neural Network consisting of 13 neurons in the hidden layer with 'hyperbolic tangent-sigmoid' activation function is found to be the best-fit model. Finally, based on sensitivity analysis, it is found that the 'approaching speed' of the motorized vehicle has the most significant influence on the fatal pedestrian crashes. 'Logarithm of average daily traffic' (ADT) volume is found to be the second most sensitive variable. Pedestrian-vehicular interaction concerning 'pedestrian-vehicular volume ratio' and lack of 'accessibility of pedestrian cross-walk' are found to be approximately as sensible as 'logarithm of ADT'.

Neuromodulation influences synchronization and intrinsic read-out

Background: The roles of neuromodulation in a neural network, such as in a cortical microcolumn, are still incompletely understood. Neuromodulation influences neural processing by presynaptic and postsynaptic regulation of synaptic efficacy. Neuromodulation also affects ion channels and intrinsic excitability.

Methods: Synaptic efficacy modulation is an effective way to rapidly alter network density and topology. We alter network topology and density to measure the effect on spike synchronization. We also operate with differently parameterized neuron models which alter the neuron's intrinsic excitability, i.e., activation function.

Results: We find that (a) fast synaptic efficacy modulation influences the amount of correlated spiking in a network. Also, (b) synchronization in a network influences the read-out of intrinsic properties. Highly synchronous input drives neurons, such that differences in intrinsic properties disappear, while asynchronous input lets intrinsic properties determine output behavior. Thus, altering network topology can alter the balance between intrinsically vs. synaptically driven network activity.

Conclusion: We conclude that neuromodulation may allow a network to shift between a more synchronized transmission mode and a more asynchronous intrinsic read-out mode. This has significant implications for our understanding of the flexibility of cortical computations.

Algorithm Design for Edge Detection of High-Speed Moving Target Image under Noisy Environment

For some measurement and detection applications based on video (sequence images), if the exposure time of camera is not suitable with the motion speed of the photographed target, fuzzy edges will be produced in the image, and some poor lighting condition will aggravate this edge blur phenomena. Especially, the existence of noise in industrial field environment makes the extraction of fuzzy edges become a more difficult problem when analyzing the posture of a high-speed moving target. Because noise and edge are always both the kind of high-frequency information, it is difficult to make trade-offs only by frequency bands. In this paper, a noise-tolerant edge detection method based on the correlation relationship between layers of wavelet transform coefficients is proposed. The goal of the paper is not to recover a clean image from a noisy observation, but to make a trade-off judgment for noise and edge signal directly according to the characteristics of wavelet transform coefficients, to realize the extraction of edge information from a noisy image directly. According to the wavelet coefficients tree and the Lipschitz exponent property of noise, the idea of neural network activation function is adopted to design the activation judgment method of wavelet coefficients. Then the significant wavelet coefficients can be retained. At the same time, the non-significant coefficients were removed according to the method of judgment of isolated coefficients. On the other hand, based on the design of Daubechies orthogonal compactly-supported wavelet filter, rational coefficients wavelet filters can be designed by increasing free variables. By reducing the vanishing moments of wavelet filters, more high-frequency information can be retained in the wavelet transform fields, which is benefit to the application of edge detection. For a noisy image of high-speed moving targets with fuzzy edges, by using the length 8-4 rational coefficients biorthogonal wavelet filters and the algorithm proposed in this paper, edge information could be detected clearly. Results of multiple groups of comparative experiments have shown that the edge detection effect of the proposed algorithm in this paper has the obvious superiority.

Neuromodulation influences synchronization and intrinsic read-out

Background: The roles of neuromodulation in a neural network, such as in a cortical microcolumn, are still incompletely understood. Neuromodulation influences neural processing by presynaptic and postsynaptic regulation of synaptic efficacy. Neuromodulation also affects ion channels and intrinsic excitability. Methods: Synaptic efficacy modulation is an effective way to rapidly alter network density and topology. We alter network topology and density to measure the effect on spike synchronization. We also operate with differently parameterized neuron models which alter the neuron's intrinsic excitability, i.e., activation function. Results: We find that (a) fast synaptic efficacy modulation influences the amount of correlated spiking in a network. Also, (b) synchronization in a network influences the read-out of intrinsic properties. Highly synchronous input drives neurons, such that differences in intrinsic properties disappear, while asynchronous input lets intrinsic properties determine output behavior. Thus, altering network topology can alter the balance between intrinsically vs. synaptically driven network activity. Conclusion: We conclude that neuromodulation may allow a network to shift between a more synchronized transmission mode and a more asynchronous intrinsic read-out mode. This has significant implications for our understanding of the flexibility of cortical computations.

Neural networks: further insights into error function, generalized weights and others

The article is a continuum of a previous one providing further insights into the structure of neural network (NN). Key concepts of NN including activation function, error function, learning rate and generalized weights are introduced. NN topology can be visualized with generic plot() function by passing a "nn" class object. Generalized weights assist interpretation of NN model with respect to the independent effect of individual input variables. A large variance of generalized weights for a covariate indicates non-linearity of its independent effect. If generalized weights of a covariate are approximately zero, the covariate is considered to have no effect on outcome. Finally, prediction of new observations can be performed using compute() function. Make sure that the feature variables passed to the compute() function are in the same order to that in the training NN.

Multiple structurally distinct ERα mRNA variants in zebrafish are differentially expressed by tissue type, stage of development and estrogen exposure

It is well established that estrogen-like environmental chemicals interact with the ligand-binding site of estrogen receptors (ERs) to disrupt transcriptional control of estrogen responsive targets. Here we investigate the possibility that estrogens also impact splicing decisions on estrogen responsive genes, such as that encoding ERα itself. Targeted PCR cloning was applied to identify six ERα mRNA variants in zebrafish. Sequencing revealed alternate use of transcription and translation start sites, multiple exon deletions, intron retention and alternate polyadenylation. As determined by quantitative (q)PCR, N-terminal mRNA variants predicting long (ERαA(L)) and short (ERα(S)) isoforms were differentially expressed by tissue-type, sex, stage of development and estrogen exposure. Whereas ERα(L) mRNA was diffusely distributed in liver, brain, heart, eye, and gonads, ERα(S) mRNA was preferentially expressed in liver (female>male) and ovary. Neither ERα(L) nor ERα(S) transcripts varied significantly during development, but 17β-estradiol selectively increased accumulation of ERα(S) mRNA (∼170-fold by 120 hpf), an effect mimicked by bisphenol-A and diethylstilbestrol. Significantly, a C-truncated variant (ERα(S)-Cx) lacking most of the ligand binding and AF-2 domains was transcribed exclusively from the short isoform promoter and was similar to ERα(S) in its tissue-, stage- and estrogen inducible expression. These results support the idea that promoter choice and alternative splicing of the esr1 gene of zebrafish are part of the autoregulatory mechanism by which estrogen modulates subsequent ERα expression, and further suggest that environmental estrogens could exert some of their toxic effects by altering the relative abundance of structurally and functionally distinct ERα isoforms.

How glucocorticoid receptors modulate the activity of other transcription factors: a scope beyond tethering

The activity of the glucocorticoid receptor (GR), a nuclear receptor transcription factor belonging to subclass 3C of the steroid/thyroid hormone receptor superfamily, is typically triggered by glucocorticoid hormones. Apart from driving gene transcription via binding onto glucocorticoid response elements in regulatory regions of particular target genes, GR can also inhibit gene expression via transrepression, a mechanism largely based on protein:protein interactions. Hereby GR can influence the activity of other transcription factors, without contacting DNA itself. GR is known to inhibit the activity of a growing list of immune-regulating transcription factors. Hence, GCs still rule the clinic for treatments of inflammatory disorders, notwithstanding concomitant deleterious side effects. Although patience is a virtue when it comes to deciphering the many mechanisms GR uses to influence various signaling pathways, the current review is testimony of the fact that groundbreaking mechanistic work has been accumulating over the past years and steadily continues to grow.

Structural and functional relationships of the steroid hormone receptors' N-terminal transactivation domain

Steroid hormone receptors are members of a family of ligand inducible transcription factors, and regulate the transcriptional activation of target genes by recruiting coregulatory proteins to the pre-initiation machinery. The binding of these coregulatory proteins to the steroid hormone receptors is often mediated through their two activation functional domains, AF1, which resides in the N-terminal domain, and the ligand-dependent AF2, which is localized in the C-terminal ligand-binding domain. Compared to other important functional domains of the steroid hormone receptors, our understanding of the mechanisms of action of the AF1 are incomplete, in part, due to the fact that, in solution, AF1 is intrinsically disordered (ID). However, recent studies have shown that AF1 must adopt a functionally active and folded conformation for its optimal activity under physiological conditions. In this review, we summarize and discuss current knowledge regarding the molecular mechanisms of AF1-mediated gene activation, focusing on AF1 conformation and coactivator binding. We further propose models for the binding/folding of the AF1 domains of the steroid hormone receptors and their protein:protein interactions. The population of ID AF1 can be visualized as a collection of many different conformations, some of which may be assuming the proper functional folding for other critical target binding partners that result in the ultimate assembly of AF1:coactivator complexes and subsequent gene regulation. Knowledge of the mechanisms involved therein will significantly help in understanding how signals from a steroid to a specific target gene are conveyed.

Differential ligand-dependent protein-protein interactions between nuclear receptors and a neuronal-specific cofactor

Nuclear receptors are transcription factors that require multiple protein-protein interactions to regulate target gene expression. We have cloned a 27-kDa protein, termed NIX1 (neuronal interacting factor X 1), that directly binds nuclear receptors in vitro and in vivo. Protein-protein interaction between NIX1 and ligand-activated or constitutive active nuclear receptors, including retinoid-related orphan receptor beta (RORbeta) (NR1F2), strictly depends on the conserved receptor C-terminal activation function 2 (AF2-D). NIX1 selectively binds retinoic acid receptor (RAR) (NR1A) and thyroid hormone receptor (TR) (NR1B) in a ligand-dependent manner, but does not interact with retinoid X receptor (RXR) (NR2B) or steroid hormone receptors. Interestingly, NIX1 down-regulates transcriptional activation by binding to ligand-bound nuclear receptors. A 39-aa domain within NIX1 was found to be necessary and sufficient for protein-protein interactions with nuclear receptors. Northern blot analysis demonstrates low-abundance RNA messages only in brain and neuronal cells. In situ hybridization and immunohistochemistry revealed that NIX1 expression is restricted to the central nervous system and could be confined to neurons in the dentate gyrus of the hippocampus, the amygdala, thalamic, and hypothalamic regions. In summary, protein-protein interactions between the neuronal protein NIX1 and ligand-activated nuclear receptors are both specific and selective. By suppressing receptor-mediated transcription, NIX1 implements coregulation of nuclear receptor functions in brain.