Lumen Core Guide

Lumen is a compiler and runtime system designed specifically for large language model (LLM) inference. It enables automatic synthesis of quantization kernels through self-developed DSL, IR, and code generators, while prioritizing inference optimization for Korean LLMs. Its core goal is to address the pain point of manually writing quantization kernels in existing solutions, improving inference efficiency and the iteration speed of new quantization technologies.

Project Background and Motivation

Existing LLM inference solutions like llama.cpp have significant pain points: when introducing new quantization formats or data type combinations, corresponding computation kernels (e.g., matrix multiplication functions) need to be written manually, which is time-consuming and labor-intensive, and limits the iteration of new quantization technologies (taking weeks/months from lab to production). Lumen, as a complete from-scratch compiler and runtime system, aims to solve this problem.

Core Technical Architecture

Lumen uses a fully self-developed tech stack to implement a complete compilation chain from high-level language to machine code:

1. Self-developed Tensor DSL: Optimized for LLM inference operations, concisely expressing complex tensor transformations and computation graphs.
2. SSA-form IR: Tensor shapes are encoded in the type system, allowing precise dimension information to be obtained during the optimization phase.
3. Multi-backend Code Generation: Supports hardware architectures such as x86_64 (AVX2/AVX-512), ARM64 (NEON/SVE), and CUDA.
4. JIT Just-In-Time Compilation: Generates specialized kernels based on input shapes at runtime, avoiding the overhead of unknown shapes in static compilation.

Automatic Quantization Kernel Synthesis

Lumen can automatically synthesize quantization kernels. When encountering quantization operations, it performs four-step fusion optimization:

1. Unpacking: Extract compressed quantization data
2. Dequantization: Convert low-precision integers to floating-point numbers
3. Matrix Multiplication: Core computation
4. Requantization: Recompress results into quantization format Fusion eliminates intermediate memory round-trips to improve efficiency; adding a new quantization format only requires adding IR-layer type definitions and conversion rules, which are automatically supported by all backends.

First-Class Support for Korean LLMs

Lumen provides targeted optimization for Korean LLMs:

• Tokenizer Efficiency: Optimized encoding efficiency for the syllabic character characteristics of Korean Hangul.
• RoPE Variants: Natively supports modified Rotary Position Embedding (RoPE) commonly used in Korean models. Currently explicitly supported Korean models include EXAONE (LG AI), HyperCLOVA-X (NAVER), and the A.X series, while also being compatible with the Chinese Qwen series models.

Development Roadmap and Technical Positioning

Development Roadmap:

Phase	Goal	Status
Phase1	DSL and parser (Pratt parser, AST, type system)	To be started
Phase2	IR and code generation (basic matrix operations for x86_64/ARM64)	To be started
Phase3	SIMD optimization (AVX2/NEON, target 90% peak GEMM performance)	To be started
Phase4	JIT engine (runtime compilation)	To be started
Phase5	Quantization support (INT8/INT4, GGUF format)	To be started
Phase6	Complete LLM inference functions (Tokenizer, KV Cache, sampling)	To be started
Phase7	Benchmarking and performance comparison (vs llama.cpp)	To be started

Non-goals: Does not support training; no built-in visualization/debugger; limited model support (prioritizing 6 Korean models + Qwen series).

Open Source and License

Lumen is open-sourced under the Apache-2.0 license and can be freely used in commercial projects. The project is developed using the Rust language (requires version 1.78+), leveraging its memory safety features and zero-cost abstraction capabilities.

Conclusion

Lumen represents a new idea for LLM inference optimization: building an inference-specific compiler from scratch, achieving dual breakthroughs in inference efficiency and development iteration speed through automatic quantization kernel synthesis and deep optimization for specific language models. For teams deploying Korean LLMs or pursuing extreme inference performance, it is an emerging project worth paying attention to.