LLM-Engine: A Local LLM Inference Desktop App Running Purely on CPU

LLM-Engine is a local LLM inference engine implemented from scratch in C++, supporting GGUF format models and running entirely on CPU (no GPU/cloud API needed). It features a complete Transformer inference stack (tokenizer, attention, KV cache, sampling) and a desktop chat interface built with Dear ImGui. Key advantages include privacy protection (local inference), no internet dependency, and educational value as a hands-on resource for understanding LLM inference mechanisms.

Most LLM users/developers rely on cloud APIs or expensive GPUs, leading to privacy risks, network latency, and high costs. LLM-Engine addresses these issues by providing a CPU-only local alternative, allowing users to run models on their own devices without internet, API keys, or dedicated graphics cards. Unlike projects based on existing libraries like llama.cpp, it implements all core components from scratch, making it both a usable app and a learning tool.

LLM-Engine covers the full LLM inference pipeline:

1. GGUF Model Loading: Uses memory mapping to load GGUF format models (supports quantized types), parsing metadata and dynamically reading weights during inference.
2. BPE Tokenizer: Implements full BPE (Byte Pair Encoding) compatible with SentencePiece, including byte-level fallback for models like Llama 3.
3. Transformer Forward Pass: Includes embedding layer (with weight tying support), multi-layer processing (RMSNorm, Grouped-Query Attention with RoPE, SwiGLU FFN), and output layer.
4. KV Cache: Reduces computation complexity from O(n²) to O(n) by storing previous K/V pairs for efficient autoregressive generation.
5. Sampling: Applies repeat penalty, temperature scaling, Top-K/Top-P sampling (or greedy decoding) to generate coherent outputs.

The UI is built with Dear ImGui (immediate-mode GUI library) for low resource usage and smooth interaction:

• Left Sidebar: Displays model architecture info (layers, heads, embedding dim, vocab size) and adjustable sampling parameters (temperature, top-k, top-p, repeat penalty), plus real-time stats (tokens per second, context usage).
• Main Panel: Scrollable chat list with streaming assistant responses (逐token display) and a blinking cursor for active messages.
• Bottom Controls: Text input box and send/stop buttons; generation runs in a background thread to keep the interface responsive (users can interrupt anytime).

Tested on Apple Silicon devices, here are key performance metrics (Release build, scalar math path):

Model	Size	Quantization	Memory Usage	Speed (Apple M Series)
Llama3.2 1B Instruct	808MB	Q4_K_M	~1.2GB	~3 tok/s
Llama3.2 3B Instruct	~2GB	Q4_K_M	~2.5GB	~1 tok/s
Mistral7B Instruct	~4.1GB	Q4_K_M	~5GB	<1 tok/s (estimated)

While slower than GPU inference, these speeds are practical for local CPU use, especially for 1B/3B models.

LLM-Engine is ideal for:

• Privacy-sensitive apps: All data stays local, no external leaks.
• Offline environments: Works without internet (great for network-limited areas).
• Learning & research: Clear code structure helps understand LLM inference core concepts.
• Edge devices: Runs on CPU-only edge devices (industrial PCs, embedded systems).
• Low-latency interaction: Local inference eliminates network delays for fast responses.

LLM-Engine represents a key direction in open-source AI—providing a feasible local alternative to cloud/GPU-dependent LLMs for ordinary users. It proves consumer CPUs can run modern LLMs (though slower than professional hardware) for practical scenarios.

For developers, it's an educational resource: code covers GGUF parsing, BPE, Transformer computation, KV cache, and sampling. The project is open-source under MIT license, with clear structure and documentation—welcome to try, learn, and contribute.