Federated Learning vs. Centralized Learning: A Complete Benchmark Suite

This article introduces an open-source benchmark project based on the Flower.ai framework, which systematically compares the performance differences between federated learning (FL) and traditional centralized learning across three mainstream models: Random Forest, Neural Network (MLP), and XGBoost. The project supports dual-mode training comparison, Non-IID data simulation, round-by-round accuracy tracking, and CSV result export, aiming to help researchers and engineers understand the performance differences between FL and centralized learning and provide data support for technical decisions.

Traditional centralized learning requires collecting all data on a server for training, which is efficient but faces data privacy challenges (e.g., sensitive data in healthcare and finance cannot leave local devices). Federated learning allows multi-party collaborative training without sharing raw data, but can FL achieve the performance of centralized training? How do different models perform in an FL environment? These questions gave birth to this benchmark project.

The project is built on the Flower.ai framework and supports three models: Random Forest, Neural Network (MLP), and XGBoost. Core features include:

• Dual-mode comparison: Supports both centralized and federated training for direct comparison;
• Non-IID data: Simulates real-world scenarios where each client has a different data category distribution;
• Round-by-round accuracy tracking: Records model performance after each round of FL communication;
• Structured export: Automatically generates comparison reports in CSV format.

Data Distribution: Non-IID partitioning is used by default, with 3 clients each having a different category distribution; FL Process: Each round includes client local training → local evaluation → server metric aggregation → statistical calculation → next round; Aggregation Strategy: Neural networks use the FedAvg algorithm to aggregate parameters; tree models (Random Forest, XGBoost) aggregate prediction results after independent training; Configurable Parameters: Adjust the number of clients (default 3, max ~6), number of communication rounds (default 10), number of local training rounds, task type (classification/regression), etc., via config/__init__.py.

The project was tested on the Iris dataset, with results as follows: Centralized Learning Results

Model	Accuracy
Random Forest	90.0%
Neural Network	76.7%
XGBoost	93.3%

Federated Learning Results (After 10 Rounds)

Model	Average Accuracy	Minimum Accuracy	Maximum Accuracy
Random Forest	92.2%	86.7%	96.7%
Neural Network	84.4%	80.0%	90.0%
XGBoost	88.9%	80.0%	96.7%

Analysis: 1. Random Forest performs slightly better in FL than in centralized mode (due to ensemble learning effects); 2. Neural Network performance improves from 76.7% to 84.4% in FL, but Non-IID data has a significant impact; 3. XGBoost maintains high performance in both modes.

Application Value:

1. Technical selection reference: Helps teams choose suitable models for FL environments;
2. Performance expectation setting: Understand the possible range of accuracy loss in FL;
3. Rapid prototype verification: Quickly verify the feasibility of FL on specific datasets using existing code.

Future Directions:

• Support more UCI datasets;
• Improve FL aggregation strategies for tree models;
• Introduce advanced aggregation algorithms like FedProx and FedOptimize;
• Add differential privacy support;
• Optimize client sampling and communication efficiency.