TriAttention: Compressing KV Cache with Trigonometric Series to Run Long-Inference Models on Consumer GPUs

Long-text inference reshapes the capability boundary of large language models, but KV cache memory explosion has become a deployment bottleneck. By exploring the concentration phenomenon of Q/K vectors in the pre-RoPE space and using trigonometric series to model distance preferences, TriAttention achieves a 10.7x KV memory compression and a 2.5x throughput improvement while maintaining full attention accuracy, enabling 32K token inference to run on a single consumer GPU for the first time.

Modern LLM inference consists of pre-filling and decoding stages. During decoding, the KV cache grows linearly with sequence length—32K token inference requires dozens of GB of VRAM, which exceeds the capacity of consumer GPUs. Existing compression methods rely on post-RoPE attention scores, but RoPE rotation causes query vectors to disperse, leading to sparse sampling, suboptimal key selection, and unstable inference.

The core insight of TriAttention comes from observations in the pre-RoPE space: Q/K vectors are highly concentrated around fixed non-zero centers, and the distribution pattern is stable across positions (Q/K concentration phenomenon). Mathematical analysis shows that this property makes queries prioritize keys at specific distances, and distance preferences can be accurately characterized by trigonometric series—each center corresponds to a specific frequency component.

TriAttention does not rely on post-RoPE attention scores; it directly leverages the concentration characteristics of Q/K in the pre-RoPE space: 1. Identify concentration centers (encoding distance preference patterns); 2. Decompose centers using trigonometric series to calculate distance preference scores for keys; 3. Combine Q/K norms to improve key selection accuracy. This mechanism computes in constant time with no additional sequence length overhead, making it suitable for ultra-long inference.

In the AIME25 benchmark test for 32K token inference, TriAttention's performance: 1. Accuracy is basically the same as full attention; 2. 10.7x KV memory compression; 3. 2.5x throughput improvement. Compared to baseline methods, it has about twice the accuracy at the same efficiency, and for the first time enables 32K inference to run on a single consumer GPU.

TriAttention demonstrates the value of deeply understanding the internal mechanisms of Transformers and triggers thinking about position encoding design (the value of information in the pre-RoPE space). In applications, it promotes the inclusive deployment of long-context LLMs, making it possible to run them on edge devices. The team plans to open-source the implementation and explore applications in scenarios such as multimodal long sequences and real-time dialogue.