SinkRouter: A Long-Context Decoding Acceleration Framework Based on the Attention Sink Mechanism

SinkRouter is a training-agnostic selective routing framework. By deeply understanding the essence of the Attention Sink phenomenon (stable, reachable, and error-controllable fixed points), it detects sink signals and skips computations that produce near-zero outputs. Combined with hardware-aware Triton kernels, it achieves a 2.03x speedup at 512K context length while maintaining competitive accuracy, providing an efficient solution for the deployment of long-context large models.

Bottlenecks in Long-Context Inference

As the capabilities of LLMs and LMMs expand, the demand for long contexts increases. However, the memory access overhead of KV caching during decoding grows linearly or super-linearly with context length, becoming a bottleneck for inference speed—especially prominent in scenarios with hundreds of thousands of tokens.

Limitations of Existing Methods

• Efficiency vs. Accuracy Trade-off: Reliance on heuristic pruning easily loses useful information, sacrificing output quality;
• Misunderstanding of Attention Sinks: Indiscriminately retaining high-score tokens, mechanically treating early tokens as anchors, or relying on heuristic routing—lacking a mechanistic understanding.

Essence of Attention Sinks

The SinkRouter team reveals that Attention Sinks are stable, reachable, and error-controllable fixed points constructed during training. This elevates the understanding to a mathematical structural level, providing a theoretical foundation for optimization.

Core Mechanisms of the SinkRouter Framework

1. Sink Signal Detection: Real-time identification of sink positions and intensities during inference;
2. Selective Computation: Skipping computation steps that produce near-zero outputs;
3. Accuracy Preservation: Ensuring no significant accuracy loss via fixed-point theory.

Hardware-Aware Optimization

Development of Triton Kernels:

• Block-Level Branching: GPU block-level conditional branching reduces thread divergence;
• Split-K Parallelism: Optimizes parallel strategies for matrix computations, improving hardware utilization.

Experimental Setup

Test benchmarks include LongBench, InfiniteBench, CVBench, MileBench, and MMVP, covering pure text models (Llama-3.1-8B/70B, Yi-9B-200K) and multimodal models (LLaVA-1.5-7B/13B).

Performance Results

• Sustained improvement in decoding efficiency across all settings;
• Competitive accuracy maintained with no significant degradation;
• 2.03x speedup achieved at 512K context length.

Significance of the Methodology

• Theoretically Guided Design: Optimization strategies designed based on fixed-point theory, combining theoretical guarantees with practicality;
• Training-Agnostic Advantage: No need to modify weights or retrain—directly applicable to pre-trained models, lowering deployment barriers;
• Hardware Co-Optimization: Deep integration with Triton kernels to fully leverage GPU parallel capabilities.

Application Prospects

SinkRouter opens up new possibilities for the practical deployment of long-context large models. As context windows expand, such optimization methods based on mechanistic understanding will become increasingly important.