Mix-Quant: A Phased Hybrid Quantization Inference Framework for Agentic LLMs

Mix-Quant is a phased hybrid quantization inference framework for Agentic LLMs. Addressing the prefill phase bottleneck caused by long contexts and multi-turn interactions in Agentic workflows, it proposes a phase-aware strategy: using FP4 (NVFP4) quantization to accelerate computation during the prefill phase while maintaining BF16 precision in the decoding phase. This achieves up to 3x prefill acceleration with almost no loss in task performance, providing a new paradigm for LLM agent inference optimization.

Agentic LLMs solve complex tasks through planning and tool use, but face challenges like long context maintenance, multi-turn interactions, and high input-side overhead. The prefill phase (processing the entire input context) becomes a key bottleneck. Quantization is a common method to accelerate inference, but global FP4 quantization leads to significant performance loss. However, studies show that the prefill phase has quantization redundancy and low precision sensitivity, allowing more aggressive quantization.

Mix-Quant adopts phase-aware hybrid quantization: 1. NVFP4 quantization in the prefill phase, leveraging native NVIDIA hardware support to accelerate matrix multiplication and reduce memory bandwidth requirements; 2. Maintaining BF16 precision in the decoding phase to ensure token generation accuracy and avoid semantic drift; 3. Phase decoupling achieves algorithm-level optimization, hardware-level efficiency, and end-to-end performance balance.

Experiments validate Mix-Quant in long-context and Agent benchmark tests: 1. Performance preservation: Almost fully maintains the original model's performance in long-context tests like RULER and Needle-in-Haystack, Agent tasks such as multi-step tool calls and complex planning, and multi-turn dialogues; 2. Speed improvement: Up to 3x acceleration in the prefill phase (e.g., processing a 100K token context reduced from 30 seconds to 10 seconds); 3. Memory efficiency: FP4 quantization significantly reduces memory usage, supporting larger models or longer contexts.

Mix-Quant is suitable for: 1. Enterprise-level Agent systems: Scenarios with long contexts such as processing large volumes of documents and historical dialogues; 2. Real-time interactive applications: Scenarios requiring fast responses like customer service robots and programming assistants; 3. Edge deployment: Improving memory efficiency on resource-constrained devices to support larger-scale Agent deployments.

Current limitations: Hardware dependency (requires NVIDIA Blackwell and subsequent architectures to support NVFP4), minor phase switching latency, and task-specific tuning needs. Future directions: Adaptive quantization (dynamically adjusting strategies), multi-hardware support, and exploring the feasibility of lower precision in the decoding phase.

Mix-Quant addresses the prefill bottleneck of Agentic LLMs through phase-aware hybrid quantization, combining FP4 acceleration in the prefill phase with BF16 precision in the decoding phase to achieve a balance between efficiency and quality. As Agent applications become widespread, phase-aware optimization will become a key technology for improving LLM inference efficiency.