HybridGen: CPU-GPU Hybrid Computing Architecture Breaks Through Long Context Inference Bottlenecks of Large Models

HybridGen addresses the KV cache bottleneck in long-context LLM inference through an innovative CPU-GPU collaborative attention mechanism combined with CXL extended memory technology, achieving a performance improvement of 1.41x to 3.2x and providing a new direction for AI system optimization in heterogeneous computing environments.

As the context length of LLMs expands to millions of tokens, the size of KV cache grows linearly, far exceeding the memory capacity of a single GPU. Traditional KV cache pruning and offloading solutions have limitations: they do not fully utilize heterogeneous hardware capabilities, or rely on a single hardware leading to resource idleness, and fail to effectively use emerging memory expansion technologies.

HybridGen proposes a CPU-GPU hybrid attention framework designed for CXL hierarchical memory expansion systems. The core lies in CPU-GPU collaborative computing rather than simple offloading: attention computation is intelligently decomposed to be executed in parallel by both, leveraging the GPU's advantages in matrix operations and the CPU's large memory capacity and complex control flow processing capabilities, and completing the computation collaboratively through an efficient synchronization mechanism.

HybridGen addresses three key technical challenges:

1. Multi-dimensional attention dependency: Introduce an attention logit parallel mechanism, decompose attention score computation into independent subtasks, and assign them to CPU/GPU based on data locality and computational characteristics;
2. Load imbalance: A feedback-driven dynamic scheduler monitors status in real time and dynamically adjusts task allocation to balance loads;
3. NUMA penalty: A semantics-aware KV cache mapping strategy places frequently accessed and semantically important tokens in local memory, and the rest in CXL extended memory to reduce access latency.

The team tested 11 LLM models on 3 GPU platforms and compared them with 6 advanced methods:

• Average performance improvement of 1.41x to 3.2x;
• The accuracy difference in downstream tasks compared to the baseline is negligible;
• The advantages become more obvious as sequence length and model size increase, showing excellent scalability.

HybridGen marks that LLM inference optimization has entered a new stage of heterogeneous collaboration. Its practical application values include longer context support, lower inference costs, and better energy efficiency. In the future, it will explore applications in the training phase, support collaboration with more accelerators such as TPU/NPU, and has broad prospects with the popularization of CXL.