Visualizing and Understanding Patch Interactions in Vision Transformer

Exploring the Impact of Additive Shortcuts in Neural Networks via Information Bottleneck-like Dynamics: From ResNet to Transformer

Deep learning has made significant strides, driving advances in areas like computer vision, natural language processing, and autonomous systems. In this paper, we further investigate the implications of the role of additive shortcut connections, focusing on models such as ResNet, Vision Transformers (ViTs), and MLP-Mixers, given that they are essential in enabling efficient information flow and mitigating optimization challenges such as vanishing gradients. In particular, capitalizing on our recent information bottleneck approach, we analyze how additive shortcuts influence the fitting and compression phases of training, crucial for generalization. We leverage Z-X and Z-Y measures as practical alternatives to mutual information for observing these dynamics in high-dimensional spaces. Our empirical results demonstrate that models with identity shortcuts (ISs) often skip the initial fitting phase and move directly into the compression phase, while non-identity shortcut (NIS) models follow the conventional two-phase process. Furthermore, we explore how IS models are still able to compress effectively, maintaining their generalization capacity despite bypassing the early fitting stages. These findings offer new insights into the dynamics of shortcut connections in neural networks, contributing to the optimization of modern deep learning architectures.

Universal Deoxidation of Semiconductor Substrates Assisted by Machine Learning and Real-Time Feedback Control

Substrate oxidation is inevitable when exposed to ambient atmosphere during semiconductor manufacturing, which is detrimental to the fabrication of state-of-the-art devices. Optimizing the deoxidation process in molecular beam epitaxy (MBE) for random substrates poses a multidimensional challenge and is sometimes controversial. Due to variations in substrates and growth processes, the determination of the deoxidation condition heavily relies on the individual's expertise, yielding inconsistent results. This study employs a machine learning model that integrates interpolation and vision transformer (Interpolation-ViT) techniques. The model utilizes reflection high-energy electron diffraction videos as input to predict the status of the substrate, enabling automated deoxidation within a controlled architecture for various substrates. Furthermore, we highlight the potential of models trained on data from specific MBE equipment to achieve high-accuracy deployment on different pieces of equipment. In contrast to traditional methods, our approach holds exceptional value, as it standardizes deoxidation temperatures across diverse equipment and substrates. This significantly advances the standardization of the semiconductor process. The concepts and methods presented are expected to revolutionize semiconductor manufacturing processes in the optoelectronic and microelectronic industries.