tiny-llm: A Systems Engineer Course to Master LLM Inference Services in One Week

tiny-llm is an LLM inference service course for systems engineers. It aims to build a vLLM-like inference system from scratch on Apple Silicon using the MLX framework. The course covers core concepts such as attention mechanisms, KV caching, continuous batching, and Flash Attention, all implemented using low-level array APIs, helping learners master the key technologies of LLM inference services.

Existing LLM resources either are too high-level abstract or rely on expensive NVIDIA GPUs, which is a pain point. The tiny-llm course uses pure MLX array/matrix APIs, avoiding high-level neural network APIs, allowing learners to deeply understand optimization details. MLX is chosen because Mac environments are easily accessible, and Qwen3 is selected as the target model due to its small size and inclusion of modern details (such as QK norm, bfloat16 weights). The official MLX 4-bit model files are provided to facilitate deployment on Apple Silicon.

First Week: Basic Model Components

Implement core components required for Qwen3 generation (attention, RoPE, GQA, RMSNorm, model loading, response generation, sampling strategies, etc.)

Second Week: Inference System Optimization

Build a simplified version of vLLM, covering KV caching, quantized matrix multiplication (CPU/GPU), Flash Attention 2 (CPU/GPU), continuous batching, chunked pre-filling, etc.

Third Week: Advanced Topics (In Progress)

Include Paged Attention, MoE (Mixture of Experts) models, speculative decoding, RAG pipelines, AI Agent/tool calling, long context processing, etc.

• Pure Low-Level Implementation: Directly use MLX array operation APIs to demonstrate matrix multiplication details, memory layout impacts, and the underlying working principles of quantization.
• Cross-Platform Optimization: Cover CPU and GPU implementations to help understand the differences in optimization strategies for different hardware.
• Production-Grade Technologies: Include key technologies used in actual production environments such as Flash Attention, KV caching, and continuous batching.

Target Audience

• Systems engineers: Optimize LLM deployment solutions
• Algorithm engineers: Understand low-level implementation details
• Researchers: Customize inference processes or model analysis
• Students: Systematically learn LLM technologies

Learning Recommendations

• Have basic Python skills, linear algebra, and calculus foundations
• Have a preliminary understanding of the Transformer architecture
• Own an Apple Silicon Mac (M1 or above)
• Hands-on implement each component and verify correctness through testing

tiny-llm fills the gap in LLM educational resources, as it both demonstrates underlying working principles and has production practicality. It is a rare opportunity for systems engineers who want to deeply understand LLM inference rather than just calling APIs. As the third-week content is improved, the course will become a complete reference for LLM system education.