Jetson LLM: Performance Benchmarking of Large Language Models on NVIDIA Edge Devices

The Jetson LLM project provides detailed performance benchmark data for running large language models using llama.cpp on the NVIDIA Jetson AGX Xavier 32GB edge device, aiming to serve as a reference for edge AI deployment. This project focuses on the efficient operation of LLMs in resource-constrained environments, verifies the feasibility of llama.cpp on the Jetson platform, and has important guiding significance for edge AI developers.

Edge AI has become an essential need due to advantages such as low latency and privacy protection, but edge devices have limited computing power—how to run LLMs efficiently is a core challenge. NVIDIA Jetson is an embedded platform specifically designed for edge AI; as a high-end model, the Jetson AGX Xavier is equipped with 32GB memory and GPU units, compatible with the CUDA ecosystem, providing a hardware foundation for edge LLM deployment.

The project uses llama.cpp (an open-source tool developed by Georgi Gerganov, optimized for CPU inference and supporting multiple quantization formats) for testing on the Jetson AGX Xavier. The tests cover models such as LLaMA, Mistral, and Llama2, with metrics including inference speed (tokens per second), memory usage, quantization effects (Q4/Q5/Q8), and batch processing performance.

Tests show: For a 7B parameter model with 4-bit quantization, the Jetson AGX Xavier can reach several to more than ten tokens per second (slightly slower for interactive use, usable for batch processing); the speed of 13B/30B models decreases but is still acceptable; the 32GB memory supports 4-bit quantized 7B models (4-5GB) and 13B models (8-10GB), allowing simultaneous operation of multiple instances or larger models.

The project helps developers select models, optimize configurations, and migrate llama.cpp workflows. Application scenarios include industrial IoT (device diagnosis), field operations (offline assistants), privacy-sensitive scenarios (medical/finance), real-time interaction (robots/drones), etc.

Edge LLMs are penetrating into device endpoints; future directions include better quantization, edge-specific hardware accelerators, model distillation, etc. The benchmark data from Jetson LLM will serve as a reference for evaluating technological progress.

Jetson LLM is a practical benchmarking project that provides valuable references for edge LLM deployment. As edge AI technology matures, more optimization work will promote the implementation of LLMs in edge scenarios.