big-vllm: A High-Performance Inference Engine Built for Qwen Series Models

big-vllm is a high-performance inference engine optimized for Alibaba's Qwen2/3/3.5 series large language models. Forked from nano-vLLM, it integrates advanced technologies like hybrid attention mechanism, CUDA graph optimization, asynchronous streaming, and compressed tensor quantization. It aims to address the inference performance bottlenecks of Qwen series models, balancing high throughput, low latency, and memory efficiency.

big-vllm was initiated by developer duchengyao. It is an open-source inference engine project deeply optimized for Qwen2, Qwen3, and Qwen3.5 series models. Forked from nano-vLLM, it inherits the advantages of a lightweight architecture while introducing advanced features required for production environments. Unlike general-purpose inference frameworks, big-vllm adopts a 'deep vertical optimization' approach—it does not aim to support all model architectures but instead concentrates resources on exploring the performance limits of Qwen series models, resulting in significant efficiency improvements.

Native Hybrid Attention Mechanism

Traditional attention computation incurs high overhead in long-sequence scenarios. big-vllm implements a native hybrid attention mechanism that dynamically selects sparse attention, sliding window attention, or full attention strategies based on sequence characteristics, significantly reducing computational complexity while ensuring model quality.

CUDA Graph Optimization

CPU-GPU synchronization overhead in inference is a major source of latency. big-vllm minimizes kernel launch overhead via CUDA graph technology, enabling near-zero-overhead GPU task submission—this is particularly critical for interactive applications requiring low first-token latency.

Asynchronous Streaming

In generative model deployment, the speed of token streaming return affects user experience. big-vllm implements a true asynchronous streaming architecture where generation and transmission are parallelized, avoiding blocking waits and improving response smoothness and real-time performance.

Compressed Tensor Quantization Support

Model quantization can reduce memory usage and improve inference speed. big-vllm has built-in native support for the compressed-tensors format, allowing models to be compressed to INT8 or even lower precision with almost no loss of accuracy, making it possible for consumer-grade hardware to run large-parameter models.

For enterprises and developers building their own LLM services, big-vllm provides a battle-tested inference foundation:

• Lower hardware costs: Through quantization and efficient memory management, the same hardware can support larger models or more concurrent users
• Better user experience: CUDA graph optimization and asynchronous streaming ensure smooth interactive responses
• Simpler deployment: Focused design reduces the complexity of configuration tuning

big-vllm is an actively maintained open-source project that continuously follows the updates and iterations of the Qwen series. Developers can contribute via GitHub, including performance optimization, new feature development, or documentation improvement, to inject vitality into the project.

big-vllm represents a successful practice of deep optimization for a specific model family in the open-source community. In the field of LLM inference, focus and depth are often more practically valuable than being broad but shallow. For teams using Qwen series models, it is a tool worth paying attention to and trying.