TriAttention: Trigonometric KV Cache Compression to Eliminate Memory Anxiety in Long Text Reasoning

This article introduces the TriAttention technology, which addresses the KV cache memory explosion problem in long text reasoning for large language models. By leveraging the concentration property of Q/K vectors in the pre-RoPE space and using trigonometric series to estimate key-value importance, it achieves 10.7x KV memory compression in 32K token scenarios while maintaining full attention accuracy, along with a 2.5x throughput improvement. It also provides a GGUF implementation supporting deployment on consumer GPUs.

Long text reasoning (e.g., chain of thought) requires storing a large amount of KV cache, leading to memory overflow on consumer GPUs. Existing KV compression methods rely on attention scores in the post-RoPE space, but RoPE rotation limits the query window (only the latest 25 tokens), which easily misjudges early key tokens and impairs reasoning coherence.

The TriAttention team discovered that Q/K vectors in the pre-RoPE space (before positional encoding) are highly concentrated at fixed non-zero centers. This phenomenon has stability (across positions/sequences), predictability (not affected by RoPE rotation), and semantic relevance; moreover, when concentrated, attention scores can be accurately reconstructed using trigonometric series.

TriAttention strategies include: 1. Distance preference modeling: use Q/K center points to calculate the attention-distance curve, and quantify preferences via trigonometric series; 2. Dual-signal fusion scoring: combine the distance preference signal and Q/K norm signal, with weights automatically adjusted based on Q/K concentration; 3. Dynamic Top-K retention: only retain high-score key-value pairs.

Benchmark test results: In AIME25 (32K tokens), it achieves the same accuracy as full attention (40.8%), with a 2.5x throughput increase and 10.7x KV memory compression. Under a fixed memory budget, TriAttention's accuracy far surpasses R-KV (e.g., 32.9% vs R-KV's 17.5% in AIME25). It supports local deployment on consumer GPUs.

The GitHub repository g023/triattention provides a GGUF format implementation, compatible with the llama.cpp ecosystem. It supports CPU/GPU hybrid inference, quantization, and cross-platform operation (Windows/macOS/Linux), and can be integrated with frameworks like OpenClaw.

TriAttention's insights: the value of pre-encoding space, the power of mathematical priors, and hardware democratization. In the future, it will become a standard for LLM deployment, paving the way for models with longer contexts; enabling consumer hardware to handle advanced AI reasoning capabilities.