AMD RDNA2 Graphics Card Local Large Model Inference Practice: Optimization Scheme Based on ROCm and TurboQuant

This project shows how to implement efficient local large model inference on AMD RDNA2 architecture graphics cards (e.g., RX 6800 XT) using the ROCm platform and the llama.cpp TurboQuant branch. It provides complete configuration scripts and multiple preset running modes, bringing AMD users a local AI development experience comparable to NVIDIA's, and supports backends for AI programming assistants like OpenCode.

For a long time, NVIDIA has dominated the AI training and inference fields with its CUDA ecosystem. With the maturity of AMD's ROCm platform and the efforts of the open-source community, AMD graphics card users can now run high-performance local large language models. This project provides a complete LLM inference solution for the RDNA2 architecture, based on the TurboQuant branch and ROCm platform, addressing the local AI development needs of AMD users.

Hardware Configuration: GPU is AMD Radeon RX 6800 XT (16GB VRAM, gfx1030 architecture), CPU is Ryzen 7 7700X, memory 64GB, operating system is Arch Linux and its derivatives. Software Dependencies: Core components of the ROCm SDK (llvm, hip-runtime-amd, hipblas, rocblas, etc.). You need to add /opt/rocm/bin to the PATH environment variable.

Adopts the non-uniform dynamic quantization strategy (Unsloth Dynamic 2.0) from the llama.cpp TurboQuant branch, keeping high precision in key layers and aggressively compressing non-key layers. Supported quantization levels:

• UD-Q2_K_XL: 10GB VRAM, 92% BF16 quality
• UD-Q3_K_XL: 13.5GB VRAM, 99% BF16 quality
• UD-Q4_K_XL: 16.5GB VRAM, 99.5% BF16 quality
• UD-Q6_K: 22GB VRAM, close to BF16 quality (requires memory offloading)

Running Modes:

1. Fast mode (default): Qwen3.6-35B-A3B MoE, 32k context, thinking mode disabled, 28 CPU MoE experts—ideal for daily Agent/code completion.
2. Smart mode: Qwen3.6-27B dense, 32k context, thinking mode enabled (2048 token budget)—suitable for complex reasoning/code review.
3. Bigctx mode: Qwen3.6-27B dense, 100k context—fit for long document/codebase analysis.
4. Custom mode: User-defined configuration. Key Parameters: CTX (context window), B/UB (batch processing), THINKING (thinking mode), N_CPU_MOE (number of CPU MoE experts), KV cache precision, etc.

• Fast mode: The 35B-A3B MoE model achieves a generation speed of 15-20 tokens/s, meeting real-time code completion needs.
• Smart mode: The 27B model's quality is significantly improved, with higher accuracy for complex programming tasks.
• Bigctx mode: The 100k context can load large codebases and support cross-file analysis.

This project demonstrates the potential of AMD graphics cards for local AI inference. Through the efforts of ROCm and the open-source community, AMD users gain a local large model experience similar to NVIDIA's, promoting a diversified AI hardware ecosystem and reducing dependence on a single supplier. Summary: A consumer-grade graphics card with 16GB VRAM can run a high-quality 35B model.

It is recommended to choose the mode based on the scenario: use Fast mode for daily coding (best response speed), Smart mode for complex tasks (high-quality answers), and Bigctx mode for large projects (long context support); balance performance and quality by adjusting parameters.