Building an LLM Inference Server from Scratch: A Deep Dive into Static Batching and Continuous Batching

This article provides an in-depth analysis of the minibatch-llm project, an LLM inference server built from scratch, focusing on the technical principles, implementation methods, and trade-offs between throughput and latency of static batching and continuous batching (iteration-level batching).

The original author/maintainer of the project is lmnst, hosted on GitHub. Original link: https://github.com/lmnst/minibatch-llm. Release/update time: 2026-06-02T14:15:01Z.

The following floors will cover background, technical details, trade-off analysis, and other content.

minibatch-llm is an LLM inference server project built from scratch, focusing on implementing efficient and scalable batching mechanisms, with complete code implementation and detailed performance benchmarks.

In current large model deployment, inference efficiency is a core challenge: as model scales grow, how to maximize throughput and minimize latency under limited computing resources is a key issue for AI infrastructure engineers. This project was created to address this problem.

Static batching is the most intuitive batching strategy: the system waits to collect a certain number of requests, then combines them into a batch and sends them to the model for inference at once. Its advantage lies in simple implementation and full utilization of GPU parallel computing capabilities.

Limitations: Waiting for the batch to fill increases the Time To First Token (TTFT); if request lengths vary greatly, short requests have to wait for long ones to complete, leading to wasted computing resources (padding overhead is particularly obvious in scenarios with large sequence length variations).

Continuous batching (iteration-level batching) is an important breakthrough in LLM inference optimization. Unlike static batching, it re-evaluates and schedules requests at each iteration step: when a request completes generation in the current iteration, a new request is immediately taken from the waiting queue and added to the batch, instead of waiting for the entire batch to finish.

Advantages: Significantly improves GPU utilization and reduces idle waiting time; better handles sequences of different lengths—short sequences complete faster and release resources, and dynamic scheduling makes the system more stable and efficient when facing mixed-length requests.

A highlight of minibatch-llm is its 'honest' performance benchmarks. In LLM inference, throughput and latency are conflicting goals: increasing batch size improves throughput but increases latency; reducing batch size lowers latency but sacrifices throughput.

The project uses experimental data to demonstrate the trade-off relationship: in online service scenarios, latency should be prioritized; for offline batch processing tasks, maximizing throughput is more reasonable. The benchmarks help developers choose appropriate strategies and parameters based on their scenarios.

minibatch-llm demonstrates key components of a production-grade LLM inference server: request queue management, memory pool allocation, KV cache optimization, and efficient CUDA kernel calls.

KV cache management is a core optimization point: by caching previously computed key-value pairs to avoid redundant calculations, it accelerates the generation process. The project implements an efficient caching strategy that supports dynamic expansion and recycling to meet the needs of sequences of different lengths.

For developers who want to deeply understand the underlying mechanisms of LLM inference, minibatch-llm is an excellent learning resource: it shows the concrete implementation of theoretical knowledge, and provides runnable code and reproducible experimental results.

In actual production, these batching optimization techniques have been widely used in mainstream inference frameworks such as vLLM and TensorRT-LLM. Understanding their implementation principles helps in tuning and troubleshooting performance issues of these frameworks.

minibatch-llm provides a clear and concise reference implementation for LLM inference optimization. By comparing static and continuous batching, it reveals the key impact of batching strategies on system performance.

As LLMs are widely applied, inference efficiency optimization remains an active research direction. The iteration-level batching ideas and throughput-latency trade-off analysis demonstrated in the project lay a foundation for further development in the field. It is recommended that engineers and researchers who wish to build efficient LLM services carefully study and draw inspiration from this project.