llama-tq: A Technical Breakthrough Enabling Large Models to Run 100k Token Long Contexts on Consumer GPUs

The llama-tq project, through two core innovations—TurboQuant KV cache quantization and sparse fine-tuning technology for hybrid MoE+SSM architectures—successfully enables running 35B-scale models with 100k token context on a single 12GB GPU while maintaining f16-level generation quality, solving the memory bottleneck issue of long contexts in large model deployment.

In large language model deployment, KV cache memory usage is a key bottleneck restricting long context capabilities. For example, a 35B-parameter MoE model in f16 precision requires tens of GB of memory for KV cache when processing 100k token contexts, which is unaffordable for consumer GPU users. Traditional KV cache quantization schemes face a dilemma: low-bit quantization loses quality, while high-bit quantization saves limited memory.

The TurboQuant technology of llama-tq decouples K and V caches and designs compression schemes based on their respective characteristics:

• K cache: Uses random Hadamard transform for decorrelation, combined with Lloyd-Max codebook, and Q·K dot product is performed directly in the Hadamard domain without dequantization, avoiding overhead and quality loss;
• V cache: Provides three sub-families: v1 (codebook), v2 (grouped Viterbi grid, default), v3 (outlier channel separation), using Trellis coding to intelligently allocate bits and adapt to the heavy-tailed distribution of V cache.

The upstream llama.cpp cannot directly fine-tune MoE+SSM models (such as Qwen3.5/3.6-A3B) due to issues like missing backward propagation for SSM operators and in-place operations being rejected by automatic differentiation. llama-tq solves these problems through:

1. The --train-skip-regex parameter to freeze specific tensors and prune the backward computation graph;
2. GGML_BACKWARD_SKIP_INPLACE=1 to bypass in-place operation assertions;
3. Implementing backward propagation for UNARY_OP_SIGMOID;
4. LLAMA_SAVER_ALLOW_UNTESTED=1 to allow saving unvalidated architectures.

Using the --cache-type-k ktq2 --cache-type-v vtq2 parameters, llama-tq achieves a KV cache compression rate of 2.78 bits-per-weight, reducing memory usage by approximately 83%. On the wikitext-2 test set, the perplexity drift is only -0.33% (within statistical error), reaching f16-equivalent quality. Actual benefits: A single 12GB GPU can run a 35B-scale MoE model with 100k token context; dual 12GB GPUs support 200k token parallel slots; a single 12GB GPU can run 4 20B-scale MoE models with 65k token slots simultaneously.

Fine-Tuning Practice: Dual RTX2060 12GB GPUs can complete sparse fine-tuning of a 35B MoE+Mamba model in 6 hours (only training embedding, output layer, and normalization layer; loss drops from 5.44 to 1.40). Usage: As a llama.cpp branch, TurboQuant requires adding the --cache-type-k ktq2 --cache-type-v vtq2 parameters. It only supports CUDA (sm_75+), and the Vulkan backend is under development. License: MIT, inherited from llama.cpp.

llama-tq marks a new stage in local deployment of large models: consumer-grade hardware can access long context scenarios, and hybrid architecture fine-tuning opens up possibilities for personalized adaptation. Its pragmatic engineering thinking (such as designing coding for K/V characteristics, pruning computation graphs to bypass limitations) is worth learning. In the future, it will continue to integrate upstream improvements and advance backend support.