Jetson Orin Nano Super 8GB Local Large Model Inference Practice: In-Depth Analysis of the Rimrock-Runtimes Project

Rimrock-Runtimes is an open-source practical project based on the Jetson Orin Nano Super 8GB, providing a guide to large model deployment on edge devices. It covers measured data, performance bottleneck analysis, and production-level configuration schemes of mainstream frameworks such as llama.cpp, ONNX Runtime, and MLC-LLM, helping developers solve LLM deployment issues on resource-constrained edge devices.

With the development of LLM technology, efficiently running LLMs on edge devices has become a focus. The Jetson Orin Nano Super 8GB has become the preferred platform due to its compact size and computing power, and this project was born based on this hardware. Hardware configuration: SM87 architecture SoC, 8GB LPDDR5 (about 7.43GB for CUDA), 915GB NVMe storage; Software stack: JetPack 6.2.2, CUDA 12.6, cuDNN9.3, TensorRT10.3.

Running LLMs on edge devices requires tuning power and clock frequencies. The project developed the RIMROCK_TOKENS configuration: CPU locked at 1728MHz, GPU locked at approximately 1020MHz, and EMC frequency increased to 3199MHz (to solve memory bandwidth bottlenecks). Maximize hardware performance through operations such as nvpmodel mode setting, jetson_clocks locking, and EMC state control.

• llama.cpp: Preferred for production. Version build8664 supports GGUF and multimodality. Gemma4 E2B Q4_K_M reaches 26.3 tok/s (4.6/5 points), Nemotron-3-Nano-4B Q4_K_M at 14.9 tok/s (5/5 points);
• ONNX Runtime: Peak at 33.0 tok/s but has MatMulNBits operator bottleneck;
• MLC-LLM: Qwen2.5-3B q4f16 only gets 3.8/5 points, not production-ready;
• vLLM: Cannot run due to memory constraints (version 0.19.0).

• Balance of quality and speed: Gemma4 E2B Q4_K_M (4.6 points/26.3 tok/s), IQ4_XS (4.4 points/28.7 tok/s);
• Extreme quality: Nemotron-3-Nano-4B (5 points/14.9 tok/s, suitable for code generation/professional creation);
• Choose carefully: Phi-4-mini (3.4/5 points).

Project structure includes runtimes (configuration), benchmarks (test results), models (management); Production deployment recommends fixed IP (e.g., 172.16.0.248) and port 8424, with startup scripts provided; Quantization strategy recommends Q4_K_M (balance of quality and speed), and aggressive schemes need to balance quality tolerance.

Core conclusions: llama.cpp is the most mature choice for edge production; ONNX Runtime has potential but needs optimization; vLLM is not suitable for edge; Gemma4/Nemotron-3-Nano are high-quality models. The project provides configuration scripts and tuning ideas, offering practical references for edge LLM deployment. The value of edge AI practical projects will become more prominent in the future.