K-Token Merging: An Efficient Inference Scheme for Large Models by Compressing Sequences in Latent Space

K-Token Merging is an efficient inference scheme for long text processing in Large Language Models (LLMs). Its core idea is to merge the embedding vectors of consecutive tokens in the latent embedding space, achieving up to 75% input length compression while maintaining almost no loss in model performance. This scheme breaks through the limitations of traditional token-space compression, addresses the quadratic computational bottleneck of LLM's self-attention mechanism, and provides a new direction for efficient inference.

Computational Bottleneck

The computational complexity of LLM's self-attention mechanism is quadratic with input length. When the input increases from 1000 to 10000 tokens, the overhead may grow by 100 times, restricting applications in scenarios like long documents and codebases.

Limitations of Existing Methods

Mainstream strategies (selective retention, summary generation, hierarchical processing) are all limited to the token space, failing to leverage the semantic redundancy of adjacent tokens in the latent embedding space and treating tokens as indivisible atomic units.

Core Idea

Merge the embedding vectors of consecutive K tokens directly in the latent embedding space, rather than operating in the token space.

Technical Architecture

1. Lightweight Encoder: Fuses every K consecutive token embeddings into a single vector with low compression overhead;
2. LoRA Adaptation for LLMs: Fine-tunes the model via Low-Rank Adaptation (LoRA) to adapt to compressed representations, training only a small number of parameters;
3. Original Vocabulary Generation: Retains the original token vocabulary at the output end, so the generated results are not restricted.

The research team validated its effectiveness across three tasks:

1. Structured Reasoning (Textualized Tree): Maintains reasoning accuracy without breaking hierarchical relationships;
2. Sentiment Classification (Amazon Reviews): Preserves semantic information to support accurate sentiment judgment;
3. Code Editing (CommitPackFT): Reliably handles technical content, verifying applicability in precision scenarios.

Balance Between Compression and Performance

• Up to 75% input compression (4000→1000 tokens);
• Almost no significant performance drop compared to the uncompressed version;
• Lies on the Pareto frontier of performance-compression ratio.

Technical Advantages

• Computational Efficiency: The self-attention computation is theoretically reduced to 1/16;
• Memory Optimization: Smaller activation memory, supporting longer contexts or larger batch sizes;
• Versatility: Seamlessly integrates with downstream tasks without modifying generation logic;
• Scalability: Flexible adjustment of K value to balance compression ratio and performance.

Application Prospects

Applicable to scenarios like long document processing (legal/academic/technical manuals), codebase understanding, multi-turn dialogue memory, and RAG systems.

Summary

K-Token Merging represents an important advancement in prompt compression technology. It breaks through traditional limitations, achieves efficient compression while maintaining performance, opens up a new direction for efficient LLM inference, and will play a key role in practical deployments.