RIS-Kernel: A Sparse Attention Inference Engine for Running 64K+ Long Texts on Ordinary CPUs

RIS-Kernel is a model-agnostic sparse attention inference engine. It reduces self-attention complexity from O(N²) to O(N log N) using sparse random geometry methods, enabling long-text inference of 65536 tokens on ordinary CPUs without GPU acceleration, thus lowering the hardware threshold for long-text large model applications.

Long-text inference for large language models faces an O(N²) complexity bottleneck. When the context window expands from 4K to 64K tokens, the computational load and memory requirements surge by 256 times. Traditional solutions relying on expensive GPU clusters limit widespread applications. However, long-text capabilities are crucial for scenarios such as legal contract analysis, academic paper reviews, codebase understanding, and multi-turn dialogue management.

The core breakthroughs of RIS-Kernel include:

1. Sparse Random Sampling Strategy: 1% attention density + 70 seed ensembles, achieving 75% accuracy in 32K token evaluation, surpassing the dense baseline (71.88%);
2. Structured Sparse Pattern: 1% density +10 seeds reach 68.75% accuracy, recovering 75% of the context gap;
3. Memory Efficiency: No OOM (Out of Memory) in 65K token scenarios, achieving a 14.06 percentage point retrieval gain.

RIS-Kernel is designed specifically for ordinary CPUs:

• Runs with 16-128GB memory; pre-filling 65K tokens takes about 50 minutes (cacheable), generating at 5 seconds per token;
• Dual hash caching mechanism optimizes performance;
• Supports attention topology visualization (exports .dot files);
• Model-agnostic, validated the effectiveness of Qwen2-1.5B-Instruct.

Experimental Results:

• Controlled Evaluation (32K tokens): Sparse attention acts as a regularizer; low density filters noise, and 1% density outperforms the dense baseline;
• Extreme Evaluation (65K tokens): Dense attention leads to OOM, while RIS runs successfully, proving feasibility on ordinary hardware.

Application scenarios of RIS-Kernel include:

• Academic research: Long document analysis on local workstations;
• Enterprise applications: Contract review and knowledge base Q&A for small and medium enterprises;
• Edge computing: Running large models on offline/edge devices;
• Model evaluation: Comparing different sparse attention strategies.

Key insights from RIS-Kernel:

1. Sparsity can improve performance through noise filtering;
2. Algorithm innovation compensates for hardware limitations and promotes technological democratization;
3. Model-agnostic architecture has "plug-and-play" value; The project is open science, providing reproducibility capsules as a starting point for developers and researchers.