PFLlib

Benchmark Platform

To integrate all the algorithms, datasets, and scenarios, we standardize the experimental settings and create a unified benchmark platform for a fair comparison of these algorithms. Here, we present the benchmark results of 20 algorithms across two widely-used label skew scenarios. This is just one example. You can obtain different results by resetting all the configurations in main.py in our PFLlib.

Leaderboard

Our methods—FedCP, GPFL, and FedDBE—lead the way. Notably, FedDBE stands out with robust performance across varying data heterogeneity levels.

The test accuracy (%) on the CV and NLP tasks in label skew settings.

The test accuracy (%) under different heterogeneous degrees in the practical label skew setting using the 4-layer CNN.

The time cost (in minutes) on Tiny-ImageNet using ResNet-18 in the practical label skew setting.

Experimental Setup

We set up the experiments following our pFL algorithm GPFL, as it provides comprehensive evaluations. Here are the details:

Datasets and Models

• For the CV tasks, we use three popular datasets:
  • Fashion-MNIST (FMNIST) (generate_FashionMNIST.py) | 4-layer CNN (model code)
  • Cifar100 (generate_Cifar100.py) | 4-layer CNN (model code)
  • Tiny-ImageNet (generate_TinyImagenet.py) | 4-layer CNN (model code) and ResNet-18 (model code)
• For the NLP task, we use one popular dataset:
  • AG News (generate_AGNews.py) | fastText (model code)

We denote TINY as the 4-layer CNN on Tiny-ImageNet, and TINY* as the ResNet-18 on Tiny-ImageNet, respectively.

Two Widely-Used Label Skew Scenarios

• Pathological label skew: We sample data with label distribution of 2/10/20 for each client on FMNIST, Cifar100, and Tiny-ImageNet, drawn from a total of 10/100/200 categories. The data is disjoint, with varying numbers of samples across clients.
• Practical label skew: Data is sampled from FMNIST, CIFAR-100, Tiny-ImageNet, and AG News using Dirichlet distribution, denoted by \(Dir(\beta)\). Specifically, we sample \(q_{c, i} \sim Dir(\beta)\) (with \(\beta = 0.1\) or \(\beta = 1\) by default for CV/NLP tasks) and allocate a \(q_{c, i}\) proportion of samples with label \(c\) to client \(i\).

Other Implementation Details

Following pFedMe and FedRoD, we use 20 clients with a client joining ratio of \(\rho = 1\), with 75% of data for training and 25% for evaluation. We report the best global model performance for traditional FL and the best average performance across personalized models for pFL. The batch size is set to 10, and the number of local epochs is 1. We run 2000 iterations with three trials per method and report the mean and standard deviation.

References

If you're interested in experimental results (e.g., accuracy) for the algorithms mentioned above, you can find results in our accepted FL papers, which also utilize this library. These papers include:

• FedALA
• FedCP
• GPFL
• FedDBE

Here are the relevant papers for your reference:

@inproceedings{zhang2023fedala,
    title={Fedala: Adaptive local aggregation for personalized federated learning},
    author={Zhang, Jianqing and Hua, Yang and Wang, Hao and Song, Tao and Xue, Zhengui and Ma, Ruhui and Guan, Haibing},
    booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
    volume={37},
    number={9},
    pages={11237--11244},
    year={2023}
}

@inproceedings{Zhang2023fedcp,
    author = {Zhang, Jianqing and Hua, Yang and Wang, Hao and Song, Tao and Xue, Zhengui and Ma, Ruhui and Guan, Haibing},
    title = {FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policy},
    year = {2023},
    booktitle = {Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining}
}

@inproceedings{zhang2023gpfl,
    title={GPFL: Simultaneously Learning Global and Personalized Feature Information for Personalized Federated Learning},
    author={Zhang, Jianqing and Hua, Yang and Wang, Hao and Song, Tao and Xue, Zhengui and Ma, Ruhui and Cao, Jian and Guan, Haibing},
    booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
    pages={5041--5051},
    year={2023}
}

@inproceedings{zhang2023eliminating,
    title={Eliminating Domain Bias for Federated Learning in Representation Space},
    author={Jianqing Zhang and Yang Hua and Jian Cao and Hao Wang and Tao Song and Zhengui XUE and Ruhui Ma and Haibing Guan},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=nO5i1XdUS0}
}