Reinforcement learning and its connections with neuroscience and psychology

Evolution of cooperation with Q-learning: The impact of information perception

The inherent complexity of human beings manifests in a remarkable diversity of responses to intricate environments, enabling us to approach problems from varied perspectives. However, in the study of cooperation, existing research within the reinforcement learning framework often assumes that individuals have access to identical information when making decisions, which contrasts with the reality that individuals frequently perceive information differently. In this study, we employ the Q-learning algorithm to explore the impact of information perception on the evolution of cooperation in a two-person prisoner's dilemma game. We demonstrate that the evolutionary processes differ significantly across three distinct information perception scenarios, highlighting the critical role of information structure in the emergence of cooperation. Notably, the asymmetric information scenario reveals a complex dynamical process, including the emergence, breakdown, and reconstruction of cooperation, mirroring psychological shifts observed in human behavior. Our findings underscore the importance of information structure in fostering cooperation, offering new insights into the establishment of stable cooperative relationships among humans.

A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity

BACKGROUND

Due to the sparse encoding character of the human visual cortex and the scarcity of paired training samples for {images, fMRIs}, voxel selection is an effective means of reconstructing perceived images from fMRI. However, the existing data-driven voxel selection methods have not achieved satisfactory results.

NEW METHOD

Here, a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model is proposed to reconstruct perceived images from fMRI. We innovatively describe voxel selection as a Markov decision process (MDP), training agents to select voxels that are highly involved in specific visual encoding.

RESULTS

Experimental results on two public datasets verify the effectiveness of the proposed DRL-SV, which can accurately select voxels highly involved in neural encoding, thereby improving the quality of visual image reconstruction.

COMPARISON WITH EXISTING METHODS

We qualitatively and quantitatively compared our results with the state-of-the-art (SOTA) methods, getting better reconstruction results. We compared the proposed DRL-SV with traditional data-driven baseline methods, obtaining sparser voxel selection results, but better reconstruction performance.

CONCLUSIONS

DRL-SV can accurately select voxels involved in visual encoding on few-shot, compared to data-driven voxel selection methods. The proposed decoding model provides a new avenue to improving the image reconstruction quality of the primary visual cortex.

Nonlinear age-related differences in probabilistic learning in mice: A 5-armed bandit task study

This study explores the impact of aging on reinforcement learning in mice, focusing on changes in learning rates and behavioral strategies. A 5-armed bandit task (5-ABT) and a computational Q-learning model were used to evaluate the positive and negative learning rates and the inverse temperature across three age groups (3, 12, and 18 months). Results showed a significant decline in the negative learning rate of 18-month-old mice, which was not observed for the positive learning rate. This suggests that older mice maintain the ability to learn from successful experiences while decreasing the ability to learn from negative outcomes. We also observed a significant age-dependent variation in inverse temperature, reflecting a shift in action selection policy. Middle-aged mice (12 months) exhibited higher inverse temperature, indicating a higher reliance on previous rewarding experiences and reduced exploratory behaviors, when compared to both younger and older mice. This study provides new insights into aging research by demonstrating that there are age-related differences in specific components of reinforcement learning, which exhibit a non-linear pattern.

AI-Powered Microfluidics: Shaping the Future of Phenotypic Drug Discovery

Phenotypic drug discovery (PDD), which involves harnessing biological systems directly to uncover effective drugs, has undergone a resurgence in recent years. The rapid advancement of artificial intelligence (AI) over the past few years presents numerous opportunities for augmenting phenotypic drug screening on microfluidic platforms, leveraging its predictive capabilities, data analysis, efficient data processing, etc. Microfluidics coupled with AI is poised to revolutionize the landscape of phenotypic drug discovery. By integrating advanced microfluidic platforms with AI algorithms, researchers can rapidly screen large libraries of compounds, identify novel drug candidates, and elucidate complex biological pathways with unprecedented speed and efficiency. This review provides an overview of recent advances and challenges in AI-based microfluidics and their applications in drug discovery. We discuss the synergistic combination of microfluidic systems for high-throughput screening and AI-driven analysis for phenotype characterization, drug-target interactions, and predictive modeling. In addition, we highlight the potential of AI-powered microfluidics to achieve an automated drug screening system. Overall, AI-powered microfluidics represents a promising approach to shaping the future of phenotypic drug discovery by enabling rapid, cost-effective, and accurate identification of therapeutically relevant compounds.

Motivation as a Lens for Understanding Information-seeking Behaviors

Most prior research characterizes information-seeking behaviors as serving utilitarian purposes, such as whether the obtained information can help solve practical problems. However, information-seeking behaviors are sensitive to different contexts (i.e., threat vs. curiosity), despite having equivalent utility. Furthermore, these search behaviors can be modulated by individuals' life history and personality traits. Yet the emphasis on utilitarian utility has precluded the development of a unified model, which explains when and how individuals actively seek information. To account for this variability and flexibility, we propose a unified information-seeking framework that examines information-seeking through the lens of motivation. This unified model accounts for integration across individuals' internal goal states and the salient features of the environment to influence information-seeking behavior. We propose that information-seeking is determined by motivation for information, invigorated either by instrumental utility or hedonic utility, wherein one's personal or environmental context moderates this relationship. Furthermore, we speculate that the final common denominator in guiding information-seeking is the engagement of different neuromodulatory circuits centered on dopaminergic and noradrenergic tone. Our framework provides a unified framework for information-seeking behaviors and generates several testable predictions for future studies.

Machine Learning Methods for Small Data Challenges in Molecular Science

Small data are often used in scientific and engineering research due to the presence of various constraints, such as time, cost, ethics, privacy, security, and technical limitations in data acquisition. However, big data have been the focus for the past decade, small data and their challenges have received little attention, even though they are technically more severe in machine learning (ML) and deep learning (DL) studies. Overall, the small data challenge is often compounded by issues, such as data diversity, imputation, noise, imbalance, and high-dimensionality. Fortunately, the current big data era is characterized by technological breakthroughs in ML, DL, and artificial intelligence (AI), which enable data-driven scientific discovery, and many advanced ML and DL technologies developed for big data have inadvertently provided solutions for small data problems. As a result, significant progress has been made in ML and DL for small data challenges in the past decade. In this review, we summarize and analyze several emerging potential solutions to small data challenges in molecular science, including chemical and biological sciences. We review both basic machine learning algorithms, such as linear regression, logistic regression (LR), k-nearest neighbor (KNN), support vector machine (SVM), kernel learning (KL), random forest (RF), and gradient boosting trees (GBT), and more advanced techniques, including artificial neural network (ANN), convolutional neural network (CNN), U-Net, graph neural network (GNN), Generative Adversarial Network (GAN), long short-term memory (LSTM), autoencoder, transformer, transfer learning, active learning, graph-based semi-supervised learning, combining deep learning with traditional machine learning, and physical model-based data augmentation. We also briefly discuss the latest advances in these methods. Finally, we conclude the survey with a discussion of promising trends in small data challenges in molecular science.

Advanced Reinforcement Learning and Its Connections with Brain Neuroscience

In recent years, brain science and neuroscience have greatly propelled the innovation of computer science. In particular, knowledge from the neurobiology and neuropsychology of the brain revolutionized the development of reinforcement learning (RL) by providing novel interpretable mechanisms of how the brain achieves intelligent and efficient decision making. Triggered by this, there has been a boom in research about advanced RL algorithms that are built upon the inspirations of brain neuroscience. In this work, to further strengthen the bidirectional link between the 2 communities and especially promote the research on modern RL technology, we provide a comprehensive survey of recent advances in the area of brain-inspired/related RL algorithms. We start with basis theories of RL, and present a concise introduction to brain neuroscience related to RL. Then, we classify these advanced RL methodologies into 3 categories according to different connections of the brain, i.e., micro-neural activity, macro-brain structure, and cognitive function. Each category is further surveyed by presenting several modern RL algorithms along with their mathematical models, correlations with the brain, and open issues. Finally, we introduce several important applications of RL algorithms, followed by the discussions of challenges and opportunities for future research.

Convergence of Artificial Intelligence and Neuroscience towards the Diagnosis of Neurological Disorders-A Scoping Review

Artificial intelligence (AI) is a field of computer science that deals with the simulation of human intelligence using machines so that such machines gain problem-solving and decision-making capabilities similar to that of the human brain. Neuroscience is the scientific study of the struczture and cognitive functions of the brain. Neuroscience and AI are mutually interrelated. These two fields help each other in their advancements. The theory of neuroscience has brought many distinct improvisations into the AI field. The biological neural network has led to the realization of complex deep neural network architectures that are used to develop versatile applications, such as text processing, speech recognition, object detection, etc. Additionally, neuroscience helps to validate the existing AI-based models. Reinforcement learning in humans and animals has inspired computer scientists to develop algorithms for reinforcement learning in artificial systems, which enables those systems to learn complex strategies without explicit instruction. Such learning helps in building complex applications, like robot-based surgery, autonomous vehicles, gaming applications, etc. In turn, with its ability to intelligently analyze complex data and extract hidden patterns, AI fits as a perfect choice for analyzing neuroscience data that are very complex. Large-scale AI-based simulations help neuroscientists test their hypotheses. Through an interface with the brain, an AI-based system can extract the brain signals and commands that are generated according to the signals. These commands are fed into devices, such as a robotic arm, which helps in the movement of paralyzed muscles or other human parts. AI has several use cases in analyzing neuroimaging data and reducing the workload of radiologists. The study of neuroscience helps in the early detection and diagnosis of neurological disorders. In the same way, AI can effectively be applied to the prediction and detection of neurological disorders. Thus, in this paper, a scoping review has been carried out on the mutual relationship between AI and neuroscience, emphasizing the convergence between AI and neuroscience in order to detect and predict various neurological disorders.

CoBeL-RL: A neuroscience-oriented simulation framework for complex behavior and learning

Reinforcement learning (RL) has become a popular paradigm for modeling animal behavior, analyzing neuronal representations, and studying their emergence during learning. This development has been fueled by advances in understanding the role of RL in both the brain and artificial intelligence. However, while in machine learning a set of tools and standardized benchmarks facilitate the development of new methods and their comparison to existing ones, in neuroscience, the software infrastructure is much more fragmented. Even if sharing theoretical principles, computational studies rarely share software frameworks, thereby impeding the integration or comparison of different results. Machine learning tools are also difficult to port to computational neuroscience since the experimental requirements are usually not well aligned. To address these challenges we introduce CoBeL-RL, a closed-loop simulator of complex behavior and learning based on RL and deep neural networks. It provides a neuroscience-oriented framework for efficiently setting up and running simulations. CoBeL-RL offers a set of virtual environments, e.g., T-maze and Morris water maze, which can be simulated at different levels of abstraction, e.g., a simple gridworld or a 3D environment with complex visual stimuli, and set up using intuitive GUI tools. A range of RL algorithms, e.g., Dyna-Q and deep Q-network algorithms, is provided and can be easily extended. CoBeL-RL provides tools for monitoring and analyzing behavior and unit activity, and allows for fine-grained control of the simulation via interfaces to relevant points in its closed-loop. In summary, CoBeL-RL fills an important gap in the software toolbox of computational neuroscience.

Smart Scheduling of Electric Vehicles Based on Reinforcement Learning

As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs on a larger scale, starting from enhancing the network interoperability and accessibility and removing the uncertainty associated with the availability of charging stations. Another issue is of particular interest for EV drivers travelling longer distances and is related to scheduling a recharging operation at the estimated time of arrival, without long queuing times. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

A Novel Multi-Factor Three-Step Feature Selection and Deep Learning Framework for Regional GDP Prediction: Evidence from China

Gross domestic product (GDP) is an important index reflecting the economic development of a region. Accurate GDP prediction of developing regions can provide technical support for sustainable urban development and economic policy formulation. In this paper, a novel multi-factor three-step feature selection and deep learning framework are proposed for regional GDP prediction. The core modeling process is mainly composed of the following three steps: In Step I, the feature crossing algorithm is used to deeply excavate hidden feature information of original datasets and fully extract key information. In Step II, BorutaRF and Q-learning algorithms analyze the deep correlation between extracted features and targets from two different perspectives and determine the features with the highest quality. In Step III, selected features are used as the input of TCN (Temporal convolutional network) to build a GDP prediction model and obtain final prediction results. Based on the experimental analysis of three datasets, the following conclusions can be drawn: (1) The proposed three-stage feature selection method effectively improves the prediction accuracy of TCN by more than 10%. (2) The proposed GDP prediction framework proposed in the paper has achieved better forecasting performance than 14 benchmark models. In addition, the MAPE values of the models are lower than 5% in all cases.