6G-Bench: An Evaluation Benchmark for Large Model Semantic Communication and Network Reasoning Capabilities in AI-Native 6G Networks

6G-Bench is an open-source standardized evaluation framework specifically designed to assess the semantic communication and network-level reasoning capabilities of foundation models in AI-native 6G networks. This framework fills the gap in the systematic evaluation of large models in the network domain. It tests the decision-making quality of models in complex network environments through multi-dimensional test scenarios, covering typical 6G features such as network slicing, edge computing, mMTC, URLLC, as well as real-world application scenarios like drone swarm control and intelligent transportation.

Background: Deep Integration of 6G and AI

With the completion of 5G deployment, 6G has become the focus of the communication industry, and its core feature is "AI-native"—AI is embedded into all layers of the network from the initial architectural design. Traditional communication optimization relies on fixed mathematical models and heuristic algorithms, which are difficult to cope with the massive heterogeneous devices, dynamic service requirements, and complex wireless environments of 6G. Although large models have strong reasoning capabilities, there is a lack of systematic evaluation standards for network-level real-time decision-making. This gap gave birth to the 6G-Bench project.

Overview of the 6G-Bench Project

6G-Bench is an open-source project that addresses the above evaluation gap, focusing on two core capabilities:

1. Semantic communication capability: The model's ability to understand and generate network intents;
2. Network-level reasoning capability: The model's ability to make multi-objective trade-off decisions under complex constraints. The framework design considers typical 6G features (network slicing, edge computing, mMTC, URLLC, eMBB), and the test scenarios cover demanding applications such as drone swarm control, intelligent transportation, and industrial automation.

Core Evaluation Dimensions

6G-Bench is structured around three task dimensions:

1. Intent Feasibility Assessment

The model needs to determine whether a network intent is feasible in the current state, considering factors such as network slicing performance, edge load, and weather, and provide feasibility judgments and minimal adjustment suggestions.

2. Intent Conflict Resolution

Handle resource competition and priority conflicts between multiple services, such as the resource trade-off between drone video transmission (high bandwidth) and flight control (low latency), to find the optimal solution under limited network resources.

3. Intent Drift Detection

Identify subtle drifts in user intent during long-term tasks, distinguish between reasonable adaptive adjustments and strategy deviations, such as whether slice switching aligns with the original task goal when network status changes.

Technical Implementation and Dataset Features

• Data Format: Test data is organized in structured JSON, including scenario descriptions, time-series data of network metrics, multiple-choice options, and answer reasoning, supporting automated evaluation and diagnosis.
• Network Metrics: Covers latency, jitter, packet loss rate, throughput, edge load, etc., including uncertain ranges (e.g., "25±3ms") to simulate real-world environments.
• Difficulty Grading: Questions are divided into different difficulty levels, from basic state recognition to complex time-series reasoning, comprehensively evaluating the cognitive levels of models.

Significance and Outlook

• Industry Value: Fills the gap in the evaluation of foundation models in the network domain, provides objective selection criteria for operators and equipment manufacturers, and promotes industry technology improvement; reveals new challenges for AI researchers in professional domain applications (such as real-time multi-dimensional numerical processing and causal reasoning).
• Future Outlook: With the advancement of 6G standardization, it is expected to become an industry-standard test suite; its open-source nature supports community contributions of new scenarios, keeping it synchronized with cutting-edge technologies.