Gemma4 on DGX Spark: Quantization Practice and Performance Analysis for ARM64 Edge Inference

This article focuses on the integration of Google Gemma4 series models with NVIDIA DGX Spark (GB10) hardware. Through the open-source project gemma4-llama-dgx-spark, it explains how to achieve efficient quantized inference on the ARM64 architecture using llama.cpp, explores the secrets of activation parameters in MoE models, conducts multi-dimensional performance benchmarking, and finally provides deployment recommendations and best practices.

Positioning of the Gemma4 Family

The Gemma4 series includes four models: E2B/E4B (efficient and lightweight, no chain-of-thought capability), 26B-A4B (MoE architecture with 25.23 billion total parameters but only 4 billion activated), and 31B (fully dense with all 30.7 billion parameters computed).

Hardware Features of DGX Spark

DGX Spark (ASUS Ascent GX10) is equipped with the Grace Blackwell SoC and uses the ARM64 architecture. It faces challenges such as binary incompatibility and complex source code compilation, but its unified memory architecture eliminates PCIe transmission bottlenecks. The project provides a Dockerized solution to adapt to ARM64.

Quantization Format Selection

• E2B/E4B: Q4_K_M is recommended (balances speed and quality)
• 26B-A4B: Q5_K_M is recommended (balances quality and speed)
• 31B: Q6_K/Q8_0 are recommended (for high quality)

Docker Containerized Deployment

Compile llama.cpp (with CUDA enabled) based on the ARM64 CUDA 13 image. The container provides OpenAI-compatible API endpoints, supporting chat.completions and completions interfaces.

Benchmarking Dimensions

1. Single-sequence throughput: E2B/E4B reach dozens of tokens per second (t/s), while 31B drops to single digits
2. Context window expansion: Performance decreases as length increases
3. Multi-user concurrency: Unified memory architecture reduces switching overhead
4. Chain-of-thought timing: Measure first-token latency, chain length, and transition time

MoE Model Performance

26B-A4B activates 8 out of 128 experts. It needs to load all parameters into memory but only computes 4 billion. Its latency is lower than E4B, throughput is higher than 31B, and quality is close to 31B—making it the best overall choice.

Model Selection Decision Tree

• Embedded/Edge: E2B
• Low-latency interaction: E4B
• General production: 26B-A4B
• High-quality offline: 31B

Quantization Configuration Table

Model	Recommended Quantization	VRAM Usage	Expected Speed
E2B	Q4_K_M	~1.5GB	30-50 t/s
E4B	Q4_K_M	~2.5GB	20-35 t/s
26B-A4B	Q5_K_M	~16GB	10-20 t/s
31B	Q6_K	~24GB	5-10 t/s

Docker Resource Limits

Set container memory limits appropriately to avoid a single instance occupying too many resources.

The gemma4-llama-dgx-spark project demonstrates a complete technical path for edge deployment of large models (ARM64 adaptation, quantization compression, performance testing). As edge AI devices become more popular, large models will move from the cloud to the terminal, spawning scenarios such as offline assistants and local knowledge bases. Mastering edge deployment technology will become an essential skill for AI engineers.