LIOB: An Automated Benchmarking Framework for Quantized Inference of Local LLMs

An automated local framework for systematically evaluating the performance, memory usage, and response quality of quantized large language models (LLMs) on edge devices. It supports multiple quantization schemes such as INT8, INT4, and GGUF, helping developers find the optimal deployment precision.

• Original Author/Maintainer: ADM1SH
• Source Platform: GitHub
• Original Title: LLM-Inference-Quantization-Benchmarker (LIOB)
• Original Link: https://github.com/ADM1SH/LLM-Inference-Quantization-Benchmarker
• Publication Date: 2026-06-04

---

With the exponential growth in the parameter scale of large language models, local inference environments face a severe challenge: memory demand grows exponentially, while the improvement in computational throughput is linear or sublinear. This asymmetric development makes deploying large models on edge devices a complex art of trade-offs.

Post-Training Quantization (PTQ) technology reduces memory usage by lowering the numerical precision of model parameters, allowing larger models to run on resource-constrained devices. However, quantization is not without cost—it may lead to a decline in inference quality. Developers need to find the optimal balance between memory efficiency, inference speed, and output quality, but the lack of systematic evaluation tools makes this decision difficult.

The LIOB (LLM Inference & Quantization Benchmarker) framework is designed to address this "precision prisoner's dilemma". It provides a unified automated benchmarking system that can systematically evaluate the trade-offs between memory usage, inference speed, and model quality under different quantization paradigms.

---

LIOB adopts a modular architecture design, breaking down the complex benchmarking process into clear stages. The entire system is built around the Ollama local inference engine and interacts with models through standardized API interfaces.

The execution process of benchmarking starts with environment preparation: first, set up a Python virtual environment and install dependencies, then start the Ollama service. The system checks if the target GGUF model exists locally; if not, it automatically downloads it from the HuggingFace Hub. After the model is registered with Ollama, a warm-up inference call is performed to stabilize performance.

Next, it enters the core testing phase: the system executes a unified prompt test suite at multiple quantization precisions (e.g., Q4, Q8, FP16), while starting a system resource monitoring thread to collect VRAM, RAM, and CPU usage data. The response of each test case is submitted to a judge model (llama3.2:3b) for quality scoring. The final results are exported in JSON and CSV formats, static visualization charts are generated, and a local web dashboard is launched for interactive analysis.

The innovation of LIOB lies in the introduction of an LLM-as-a-Judge quality evaluation mechanism. Unlike the traditional Perplexity metric, which only measures the model's confidence in its own output, LIOB uses an independent judge model to evaluate the actual quality of the quantized model's output. This method is closer to human perception of response quality, making the evaluation results more practical.

---

Experiments conducted on the Qwen2.5-0.5B-Instruct model and Apple M4 Pro hardware revealed some interesting findings:

Experimental data shows that 4-bit quantization (Q4_K_M) achieves a 31.75% throughput improvement and a 44.12% reduction in VRAM usage compared to the FP16 baseline, while the response quality only decreases by 12.20%. This data indicates that 4-bit quantization is a highly attractive option in resource-constrained scenarios.