inference_engine_rust: A GGUF Format LLM Inference Engine Implemented in Rust

inference_engine_rust is a GGUF format LLM inference engine implemented from scratch in Rust, combining educational and practical value. It supports model loading, tokenizer processing, embedding calculation, and generation capabilities. It provides performance benchmarking and comparative validation with llama.cpp, serving both as a practical inference tool and a learning resource for understanding LLM inference mechanisms.

The project is positioned as an "educational implementation", with core components written natively in Rust instead of calling mature libraries. Built on Rust 2024 Edition, it requires rustc 1.85+. It supports the GGUF format and can directly load quantized models (e.g., Q4_K_M), making it suitable for resource-constrained environments.

It includes four core modules:

1. Model Loading and Parsing: A complete GGUF file parser that handles key-value metadata and chunked tensor storage;
2. Tokenizer Support: Compatible with SentencePiece (for Mistral, etc.) and Hugging Face Tokenizers (for Gemma, etc.);
3. Embedding Calculation and Inference: Implements a complete forward propagation path including attention, layer normalization, etc., and compares logits and hidden states with llama.cpp to ensure correctness;
4. Greedy Generation: Supports basic greedy decoding, laying the foundation for complex sampling strategies.

It includes a built-in bench_compare tool that measures cold/hot start TTFT (Time To First Token), decoding throughput, and can directly compare with llama.cpp. Validation strategies include embedding layer validation, logits comparison, hidden state validation, and generation smoke tests to ensure implementation correctness.

Currently in the early stage, the TTFT for a 6-token prompt on an M1 MacBook is about 64 seconds, which is 40 times slower than llama.cpp. Optimization directions include memory layout optimization, computation kernel optimization (SIMD/matrix acceleration), Metal GPU support, and quantized operator optimization.

It uses the MIT OR Apache-2.0 dual license, in line with Rust ecosystem conventions. The educational value is reflected in: manageable code size, pure Rust implementation, modular design, and a complete test suite, making it easy for developers to deeply understand LLM inference systems.

inference_engine_rust deepens the understanding of LLM inference through implementation from scratch, providing a unique learning resource for the community. Although its current performance is not as good as mature solutions, with continuous optimization, it is expected to have both educational value and practical capabilities, making it a beneficial addition to the Rust ecosystem and the LLM inference field.