Helix-Lite: Long Context Inference Optimization Scheme on Dual RTX 3090

Helix-Lite is a long context inference project optimized for consumer-grade hardware. It enables inference of the Qwen2.5-7B-1M model with 128K context on two RTX 3090 GPUs and supports EM-LLM RAG retrieval augmentation for documents exceeding 128K tokens. This article will cover its background, technical solutions, performance, application scenarios, and other aspects.

Extending the context length of large language models brings capabilities like whole-book summarization and large codebase understanding, but memory consumption increases with sequence length and inference speed decreases. Even with quantization techniques, consumer-grade hardware like RTX 3090 (24GB memory) still faces memory bottlenecks when running 7B models to process 128K context. The Helix-Lite project explores an efficient solution for dual RTX 3090 GPUs to address this challenge.

Model Quantization: AWQ INT4

Adopts Activation-Aware Weight Quantization (AWQ) to compress the 7B model weights from FP16 (≈14GB) to INT4 (≈3.5GB), saving memory for KV cache and long context.

KV Cache Compression: nuq4

Compresses KV cache using a non-uniform quantization strategy, allocating more levels to frequent value ranges while preserving key attention information.

Attention Optimization: Quest top-K

Uses query-guided sparse attention, focusing only on the most relevant K historical positions, reducing computational complexity from O(n²) to O(n×K).

Ultra-Long Document Support: EM-LLM RAG

Splits ultra-long documents into chunks and builds a hierarchical index. During inference, it retrieves the most relevant chunks and handles cross-chunk dependencies via an evidence fusion mechanism.

Hot-Cold Data Exchange

Active context is kept in GPU memory, while historical context is swapped to CPU/disk and loaded on demand.

Custom Triton Kernels

Optimizes key operators like nuq4 dequantization, Quest attention, and EM-LLM retrieval to leverage Tensor Core performance.

In the 2x RTX 3090 configuration:

• Model: Qwen2.5-7B-1M @ AWQ INT4
• Maximum context: 128K tokens
• Memory usage: ~40-44GB (distributed across two GPUs)
• Documents exceeding 128K tokens can be processed via EM-LLM RAG mode, at the cost of retrieval and fusion overhead.

Applicable to:

• Long document Q&A (whole books, legal documents, etc.)
• Codebase analysis (cross-file dependencies, architecture review)
• Multi-turn conversation history (maintaining full context)
• Long video script analysis
• Scientific literature review (cross-literature comprehensive analysis)

• Quantization loss: INT4 quantization introduces precision loss; precision-sensitive scenarios need verification.
• Sparse attention limitations: Quest top-K may affect long-distance dependency capture.
• RAG overhead: EM-LLM mode has higher latency than direct inference.
• Hardware requirements: Dual RTX 3090 is a high-end configuration; single-card setups need to reduce context length.

• Support more long-context models (e.g., Llama3.1 405B's 128K version)
• Optimize single-card performance to lower hardware barriers
• Integrate technologies like FlashAttention-3 and Ring Attention
• Support multi-modal long context (images, videos)

Helix-Lite achieves long-sequence inference capabilities on consumer-grade hardware through a combination of quantization, compression, sparse attention, and RAG optimization. It provides valuable references for local deployment of long-context LLMs and is worth researching and trying by developers.