TriAttention: KV Cache Compression Using Trigonometric Functions for Long Text Inference Acceleration

This article introduces TriAttention, an innovative KV cache compression method that uses trigonometric functions and RoPE positional encoding features. It enables efficient long text inference on consumer GPUs, reducing memory usage by 5.8x while maintaining output quality, effectively addressing the memory bottleneck in long text inference for large language models.

In large language model inference, KV cache grows linearly with the generated sequence length, becoming a memory bottleneck for long text inference. Traditional solutions like quantization compression, sliding window attention, and StreamingLLM either sacrifice precision or have limitations in modeling long-distance dependencies.

TriAttention is based on the mathematical properties of RoPE positional encoding, and its attention pattern can be described as a trigonometric series. The core implementation includes:

1. Trigonometric scoring mechanism: Weighted approximate scoring function (emphasizing low-frequency bands) + geometric series future offset averaging;
2. Three-region retention strategy: Always retain the first few tokens (attention convergence area), the most recent few tokens (recent window area), and retain middle tokens based on their scores;
3. Windowed pruning: Trigger pruning every 128 tokens generated, operating on KV cache via the llama-cpp-python low-level API.

In tests on the Qwen3-1.77B Q8_0 GGUF model (RTX3060 12GB):

• Output quality fully preserved: The pruned version is consistent with the first 300 characters of the baseline;
• No speed loss: Pruning overhead is negligible, even slightly improved;
• Significant memory savings: When KV budget is 64, memory usage is reduced by 5.8x. Detailed data is shown in the table:

  KV Budget	Baseline tok/s	TriAttention tok/s	Final Cache	Pruning Count	Memory Reduction Multiple
  Full	17.7	-	542	0	1.0×
  256	17.7	17.7	286	2	1.9×
  128	17.7	17.9	158	6	3.4×
  64	17.8	17.8	94	14	5.8×

TriAttention uses a single-file Python design and only depends on the llama-cpp-python library. Its core components include:

• Frequency extraction module: Reads RoPE configuration to calculate rotation rates;
• Score calculation module: Implements trigonometric scoring and future offset averaging;
• Pruning execution module: Integrates the three-region strategy to perform cache pruning;
• Generation control module: Precisely controls KV cache using low-level APIs; It also provides a baseline test mode for easy performance comparison and verification.

Simplifications in the current implementation compared to the paper:

1. Restricted access to pre-RoPE vectors, using a general frequency-weighted cosine approximation score;
2. No use of value vector information;
3. Simplified head-level processing (unified scoring across heads);
4. Backend differences (targeting llama-cpp-python/GGUF vs. the paper's vLLM/FlashAttention-2); However, the core mathematical structure (RoPE frequency trigonometric series + geometric future offset averaging) is preserved.

Applicable scenarios:

• Long document generation, multi-turn dialogue systems, code generation, edge device deployment; Usage recommendations:
• Start with budget=256 and gradually reduce until memory constraints are met;
• If output quality decreases, increase the recent-tokens parameter;
• Use a CUDA-supported version of llama-cpp-python for optimal performance.