Orthrus: A Large Language Model Inference Framework for Lossless Acceleration via Dual-View Diffusion Decoding

Orthrus is an innovative dual-view diffusion decoding framework for large language model (LLM) inference. It combines the precise generation quality of autoregressive models with the high-speed parallel decoding capability of diffusion models, achieving up to 7.8x inference acceleration while maintaining completely lossless output. Built on the Qwen3 series models, it adopts a parameter-efficient fine-tuning strategy with negligible memory overhead, providing a new path for optimizing LLM inference efficiency.

Most current mainstream large language models (LLMs) use autoregressive architectures, which require sequential decoding of tokens one by one to generate text. While this ensures quality and coherence, it cannot fully utilize the parallel computing capabilities of modern GPUs, leading to efficiency bottlenecks. Diffusion models have shown advantages in parallel generation in the image domain, but applying them to language models—how to achieve true lossless acceleration while maintaining generation quality—is a major challenge for academia and industry.

Orthrus proposes a dual-view diffusion decoding scheme, maintaining two working modes within a single model: the autoregressive view ensures the precision of generation quality, while the diffusion view is responsible for high-speed parallel token prediction. The two views share the same key-value cache (KV Cache), with memory overhead at the O(1) level—almost negligible—allowing it to deliver excellent acceleration even in resource-constrained environments.

Orthrus adopts a parameter-efficient fine-tuning strategy, requiring only about 16% of the base model's parameters to be fine-tuned, while the core weights of the base LLM are completely frozen. This ensures the integrity of the original capabilities and lowers the threshold for training and deployment. Based on the 1.7B, 4B, and 8B versions of Qwen3 models, while maintaining consistency with the original model's prediction distribution, it achieves average inference acceleration of 4.25x, 5.20x, and 5.36x respectively, with an acceleration ratio of up to 7.8x for specific tasks.

1. Strictly lossless generation: Ensures that the output is completely consistent with the original base model's prediction distribution through an in-model consensus mechanism; 2. Zero redundant memory overhead: The dual views share a high-fidelity KV Cache with no additional video memory usage; 3. Production-ready deployment: Native integration support for mainstream inference frameworks such as vLLM and SGLang is under development, facilitating easy integration into existing LLM service infrastructures.

Applicable to real-time interactive AI systems (intelligent customer service, code completion, real-time translation) to reduce user waiting time; For enterprise-level text tasks (content creation platforms, automatic report generation, data summarization systems), it can reduce computing costs without sacrificing quality; For edge device deployment, its efficient memory characteristics make it possible to run high-performance LLMs on a single card or even consumer-grade GPUs.

The research results of Orthrus have been published on arXiv (paper number: 2605.12825) with the title "Orthrus: Memory-Efficient Parallel Token Generation via Dual-View Diffusion", proving that the autoregressive and diffusion paradigms can complement each other's advantages. After completing the integration with vLLM and SGLang in the future, it is expected to become an important infrastructure for the next generation of efficient LLM services, which is worth the attention and trial of developers and researchers.