llm-d-inference-sim: A GPU-free vLLM Behavior Simulator to Lighten LLM Inference Testing

llm-d-inference-sim is a lightweight, configurable real-time simulator. It can simulate the core behavioral characteristics of vLLM without needing a GPU or real large models, addressing the pain point of relying on expensive resources in LLM inference system development and testing, allowing developers to complete most development and testing tasks on ordinary devices.

Modern LLM inference engines like vLLM use technologies such as PagedAttention to optimize memory and throughput. However, development and debugging require high-performance GPUs, large model weight files, and complex environment configurations, which increase costs and limit flexibility. Especially in CI/CD, automated testing, and algorithm prototype verification stages, lightweight simulation tools are urgently needed.

The design philosophy is 'behavioral equivalence rather than result equivalence'. It does not need to load real models or perform neural network computations; instead, it simulates vLLM's key behaviors through mathematical models and statistics:

• Request scheduling: Reproduce logic such as continuous batching, preemption, and recomputation
• Memory management: Simulate PagedAttention memory allocation and show KV cache fragmentation, etc.
• Performance metrics: Generate realistic indicators like TTFT and TPOT It also provides rich configuration options (model parameters, hardware characteristics, workloads, scheduling strategies, etc.).

Supports practice in multiple scenarios:

1. Scheduling algorithm development: A safe experimental environment for rapid strategy iteration and performance observation
2. System behavior verification: Regression testing to compare simulations with expectations and identify logical errors
3. Capacity planning: Evaluate the performance of different hardware configurations to assist procurement decisions
4. Education and training: Help learners understand the core design and scheduling logic of LLM inference systems.

It should be clear: The simulator is not a production alternative; its value lies in the development and testing phase. Performance data is a statistical approximation, not an exact prediction. Verification on real hardware and models is required before actual deployment.

As an open-source project, community contributions are welcome. Future directions include:

• Supporting more inference engines (TensorRT-LLM, llama.cpp, etc.)
• Adding a visual interface to display the scheduling process
• Integrating performance analysis tools to automatically identify bottlenecks
• Supporting distributed inference scenario simulation.

llm-d-inference-sim is an important addition to the LLM infrastructure toolchain. It lowers development barriers, improves efficiency, and provides a friendly experimental environment for innovation in the LLM inference field. As large model technology evolves, its value will become increasingly prominent.