FedSim: Similarity guided model aggregation for Federated Learning

Federated Collaborative Learning with Sparse Gradients for Heterogeneous Data on Resource-Constrained Devices

Federated learning enables devices to train models collaboratively while protecting data privacy. However, the computing power, memory, and communication capabilities of IoT devices are limited, making it difficult to train large-scale models on these devices. To train large models on resource-constrained devices, federated split learning allows for parallel training of multiple devices by dividing the model into different devices. However, under this framework, the client is heavily dependent on the server's computing resources, and a large number of model parameters must be transmitted during communication, which leads to low training efficiency. In addition, due to the heterogeneous distribution among clients, it is difficult for the trained global model to apply to all clients. To address these challenges, this paper designs a sparse gradient collaborative federated learning model for heterogeneous data on resource-constrained devices. First, the sparse gradient strategy is designed by introducing the position Mask to reduce the traffic. To minimize accuracy loss, the dequantization strategy is applied to restore the original dense gradient tensor. Second, the influence of each client on the global model is measured by Euclidean distance, and based on this, the aggregation weight is assigned to each client, and an adaptive weight strategy is developed. Finally, the sparse gradient quantization method is combined with an adaptive weighting strategy, and a collaborative federated learning algorithm is designed for heterogeneous data distribution. Extensive experiments demonstrate that the proposed algorithm achieves high classification efficiency, effectively addressing the challenges posed by data heterogeneity.

FLCMC: Federated Learning Approach for Chinese Medicinal Text Classification

Addressing the privacy protection and data sharing issues in Chinese medical texts, this paper introduces a federated learning approach named FLCMC for Chinese medical text classification. The paper first discusses the data heterogeneity issue in federated language modeling. Then, it proposes two perturbed federated learning algorithms, FedPA and FedPAP, based on the self-attention mechanism. In these algorithms, the self-attention mechanism is incorporated within the model aggregation module, while a perturbation term, which measures the differences between the client and the server, is added to the local update module along with a customized PAdam optimizer. Secondly, to enable a fair comparison of algorithms' performance, existing federated algorithms are improved by integrating a customized Adam optimizer. Through experiments, this paper first conducts experimental analyses on hyperparameters, data heterogeneity, and validity on synthetic datasets, which proves that the proposed federated learning algorithm has significant advantages in classification performance and convergence stability when dealing with heterogeneous data. Then, the algorithm is applied to Chinese medical text datasets to verify its effectiveness on real datasets. The comparative analysis of algorithm performance and communication efficiency shows that the algorithm exhibits strong generalization ability on deep learning models for Chinese medical texts. As for the synthetic dataset, upon comparing with comparison algorithms FedAvg, FedProx, FedAtt, and their improved versions, the experimental results show that for data with general heterogeneity, both FedPA and FedPAP show significantly more accurate and stable convergence behavior. On the real Chinese medical dataset of doctor-patient conversations, IMCS-V2, with logistic regression and long short-term memory network as training models, the experiment results show that in comparison to the above three comparison algorithms and their improved versions, FedPA and FedPAP both possess the best accuracy performance and display significantly more stable and accurate convergence behavior, proving that the method in this paper has better classification effects for Chinese medical texts.