Exploring the Limits of Framework Desktop Inference: Practical Large Model Optimization on the Strix Halo Platform

This research project focuses on the Framework Desktop platform with AMD Strix Halo architecture, combining RTX 3090 to optimize large model inference via llama.cpp RPC. It completed 34 tasks covering cutting-edge technologies like KV cache compression, speculative decoding, and heterogeneous RPC inference, exploring the limits of desktop-level LLM inference and challenging the traditional reliance on data center GPUs.

Research Background

As LLM scales grow, inference efficiency has become a bottleneck for deployment, with traditional reliance on expensive data center GPUs. The Framework Desktop with AMD Strix Halo architecture (Ryzen AI MAX+395, Radeon 8060S iGPU, 128GB unified memory) provides an ideal platform for desktop-level inference.

Test Environment

• Main Node: Framework Desktop (Ryzen AI MAX+395, Radeon 8060S, 128GB LPDDR5X, Vulkan/ROCm backend)
• Companion Node: RTX 3090 (24GB GDDR6X, CUDA 12.8)
• Software Stack: llama.cpp (b8775/b8779), RPC over Wi-Fi

Key Task Exploration

1. KV Cache: Tested 14 Pareto-optimal configurations to balance context length and speed
2. Speculative Decoding: Used a 0.8B draft model to accelerate the 122B target model, increasing decoding speed by 1.98x
3. Parallel Throughput: Aggregate throughput increased by 2.21x when npl=8
4. Comprehensive Optimization: Q4_K_M quantization + ubatch=2048 + parallel slots achieved an aggregate throughput of 60.54 tok/s
5. Thermal Sustainability: Throughput drift was only -0.08% after 60 minutes of operation
6. Heterogeneous RPC: Split the Qwen3.5-122B model across AMD + NVIDIA GPUs, with only a 4.3% decrease in decoding speed

Technical Depth

• Unified Memory Architecture: Shared 128GB memory supports larger models and zero-copy transfer
• rocWMMA Flash Attention: Reduces memory bandwidth requirements
• Mixed-Precision Quantization: Established a trade-off curve between quantization levels and quality
• NPU Experiments: Explored the potential of Neural Processing Units (NPUs) in LLM inference

Core Data

• Phase0: ROCm + MMQ prefill at 406 tok/s, decoding at 40.1 tok/s; chat load improved by 47% compared to Vulkan
• Mission01: f16/f16 KV precision supports 131K token context, prefill at 152.76 tok/s
• Mission34: Successfully loaded the 129GB MiniMax-M2.5 model (RTX3090 uses 22.1GB, Radeon8060S uses 109.5GB)

Reproducibility Design

• Environment variable-driven configuration
• Task-level detailed documentation
• Raw data (JSON/CSV) made public
• Runnable test scripts open-sourced (MIT license)

Core Conclusions

1. Desktop integrated GPU platforms can handle serious large model inference; 128GB unified memory supports models with over 100B parameters
2. Heterogeneous RPC inference validates the feasibility of cross-vendor GPU collaboration
3. Submitted fixes and optimization suggestions to the llama.cpp upstream

Industry Significance

• Promotes AI democratization: Reduces local inference costs and supports privacy-sensitive/offline scenarios
• Demonstrates heterogeneous computing: Provides new ideas for ultra-large-scale model inference
• Open-source contributions: Publishes data and scripts to support community development

Current Limitations

1. Wi-Fi RPC introduces latency; wired connections may improve performance
2. ROCm ecosystem maturity lags behind CUDA
3. Long-term high load poses challenges to heat dissipation

Future Directions

1. Expand testing to latest models like Llama3 and Qwen3
2. Explore new GGUF quantization schemes
3. Try multi-node RPC clusters
4. Develop a dedicated deployment toolchain for Strix Halo