Quantization-aware training for low precision photonic neural networks

ADFQ-ViT: Activation-Distribution-Friendly post-training Quantization for Vision Transformers

Vision Transformers (ViTs) have exhibited exceptional performance across diverse computer vision tasks, while their substantial parameter size incurs significantly increased memory and computational demands, impeding effective inference on resource-constrained devices. Quantization has emerged as a promising solution to mitigate these challenges, yet existing methods still suffer from significant accuracy loss at low-bit. We attribute this issue to the distinctive distributions of post-LayerNorm and post-GELU activations within ViTs, rendering conventional hardware-friendly quantizers ineffective, particularly in low-bit scenarios. To address this issue, we propose a novel framework called Activation-Distribution-Friendly post-training Quantization for Vision Transformers, ADFQ-ViT. Concretely, we introduce the Per-Patch Outlier-aware Quantizer to tackle irregular outliers in post-LayerNorm activations. This quantizer refines the granularity of the uniform quantizer to a per-patch level while retaining a minimal subset of values exceeding a threshold at full-precision. To handle the non-uniform distributions of post-GELU activations between positive and negative regions, we design the Shift-Log2 Quantizer, which shifts all elements to the positive region and then applies log2 quantization. Moreover, we present the Attention-score enhanced Module-wise Optimization which adjusts the parameters of each quantizer by reconstructing errors to further mitigate quantization error. Extensive experiments demonstrate ADFQ-ViT provides significant improvements over various baselines in image classification, object detection, and instance segmentation tasks at 4-bit. Specifically, when quantizing the ViT-B model to 4-bit, we achieve a 5.17% improvement in Top-1 accuracy on the ImageNet dataset. Our code is available at: https://github.com/llwx593/adfq-vit.git.

Progressive fine-to-coarse reconstruction for accurate low-bit post-training quantization in vision transformers

Due to its efficiency, Post-Training Quantization (PTQ) has been widely adopted for compressing Vision Transformers (ViTs). However, when quantized into low-bit representations, there is often a significant performance drop compared to their full-precision counterparts. To address this issue, reconstruction methods have been incorporated into the PTQ framework to improve performance in low-bit quantization settings. Nevertheless, existing related methods apply the single and fixed reconstruction granularity and seldom explore the progressive relationships between different reconstruction granularities, which leads to sub-optimal quantization results in ViTs. To this end, in this paper, we propose a Progressive Fine-to-Coarse Reconstruction (PFCR) method for accurate PTQ, which significantly improves the performance of low-bit quantized vision transformers. Specifically, we define multi-head self-attention and multi-layer perceptron modules along with their shortcuts as the finest reconstruction units. After reconstructing these two fine-grained units, we combine them to form coarser blocks and reconstruct them at a coarser granularity level. We iteratively perform this combination and reconstruction process, achieving progressive fine-to-coarse reconstruction. Additionally, we introduce a Progressive Optimization Strategy (POS) for PFCR to alleviate the difficulty of training, thereby further enhancing model performance. Experimental results on the ImageNet dataset demonstrate that our proposed method achieves the best Top-1 accuracy among state-of-the-art methods, particularly attaining 75.61% for 3-bit quantized ViT-B in PTQ. Besides, quantization results on the COCO dataset reveal the effectiveness and generalization of our proposed method on other computer vision tasks like object detection and instance segmentation.

Machine learning algorithms for FPGA Implementation in biomedical engineering applications: A review

Field Programmable Gate Arrays (FPGAs) are integrated circuits that can be configured by the user after manufacturing, making them suitable for customized hardware prototypes, a feature not available in general-purpose processors in Application Specific Integrated Circuits (ASIC). In this paper, we review the vast Machine Learning (ML) algorithms implemented on FPGAs to increase performance and capabilities in healthcare technology over 2001-2023. In particular, we focus on real-time ML algorithms targeted to FPGAs and hybrid System-on-a-chip (SoC) FPGA architectures for biomedical applications. We discuss how previous works have customized and optimized their ML algorithm and FPGA designs to address the putative embedded systems challenges of limited memory, hardware, and power resources while maintaining scalability to accommodate different network sizes and topologies. We provide a synthesis of articles implementing classifiers and regression algorithms, as they are significant algorithms that cover a wide range of ML algorithms used for biomedical applications. This article is written to inform the biomedical engineering and FPGA design communities to advance knowledge of FPGA-enabled ML accelerators for biomedical applications.