LeanLLM: A Concise and Correct Layer-wise Reference Implementation for Large Model Inference

LeanLLM is a lightweight inference engine for Google's Gemma4 model. Its core feature is achieving extremely low memory usage through a layer-wise loading strategy, solving compatibility issues that mainstream inference engines (such as vLLM and llama.cpp) encounter when adapting to Gemma4's new architecture. It provides a 'concise and correct' reference implementation that balances educational value and practicality.

Gemma4, released by Google in April 2026, adopts new designs such as the MatFormer architecture, layer-wise embedding, and dual attention head mechanism. However, mainstream inference engines lag in adaptation: vLLM's performance drops sharply (only 9 tokens/s on RTX4090) due to its inability to handle heterogeneous attention head dimensions; llama.cpp hardcodes the final_logit_softcapping parameter, leading to degenerative token loops (e.g., repeated <unused24>). LeanLLM was born to provide a correct implementation of Gemma4's features.

LeanLLM is positioned as an educational reference implementation with concise code (fewer than 2000 lines), single-responsibility modules, and test-driven development (all 67 test cases passed). It prioritizes correctness in design, handling Gemma4's features carefully: implementing dual-path attention mechanism, dynamically reading configuration parameters, filtering unused tokens, and avoiding hardcoding issues.

LeanLLM uses a layer-wise inference strategy: during each forward pass, it loads, computes, and evicts layers one by one, with peak memory usage being only the size of a single layer plus activation values. The cost is disk I/O overhead, but this is mitigated via background prefetching. A real test on MacBook Air M1 (8GB RAM) with the SmolLM2-135M model shows: peak memory 124MB, throughput 1.4 tokens/s—this trade-off is reasonable.

For Gemma4's heterogeneous attention heads (local 256D/global 512D), LeanLLM implements dual-path attention, dynamically selecting local/global paths; it dynamically reads the final_logit_softcapping parameter in sampler.py to avoid hardcoding pitfalls; it also supports thought token budget configuration for fine-grained control of model behavior.

LeanLLM uses a layered architecture (core/models/server/cli) with clear module responsibilities for easy maintenance and comprehensive test coverage (unit + integration tests). It provides multiple interfaces (CLI/Python API/OpenAI-compatible REST API). A real test on MacBook M1 with SmolLM2-135M shows: throughput of 1.4 tokens/s, peak memory of 124MB, and coherent generated text quality—verifying the correctness of key technologies.

Current limitations of LeanLLM: No KV caching leads to linear growth in long-sequence generation costs; the layer-wise strategy limits multi-GPU parallelism. The future roadmap includes exploring cutting-edge compression research, optimizing efficiency while maintaining low memory usage, and gradually improving performance.

The value of LeanLLM lies in providing a 'correct' reference implementation, filling the gap in mainstream engines' adaptation to Gemma4. For developers, the concise code makes it easy to understand LLM inference principles; for resource-constrained scenarios, the layer-wise strategy provides a feasible solution; its 'correct first, optimize later' philosophy sets a pragmatic example for AI engineering.