When Classical Control Theory Meets LLM Inference: An 11-Chapter Exploration of Latency, Queues, and GPUs

This project applies classical control theory to LLM inference service systems. Through 11 chapters of iterative experiments (3 successful controllers, 8 failure cases) and a real GPU closed-loop TTFT regulator, it reveals the core truth: understanding the controlled object is more important than the control law itself. The project covers the entire process from simulation to the Modal cloud platform's NVIDIA T4 GPU.

The project starts with the problem of using classical control to manage LLM inference latency and queues, with tools chosen as MATLAB and Python. It gradually transitions from pure simulation to local Ollama deployment, vLLM Apple Silicon version, and finally to the Modal cloud platform's NVIDIA T4 GPU. Each chapter's structure: specific control architecture design → implementation in a specific environment → experimental result analysis → failure cause dissection.

Successful chapters include: 1. Single-loop LQR + pole placement (simulation verifies feasibility); 2. Cascade control architecture (in simulation, the inner loop controls batch size to queue depth, and the outer loop controls queue reference value to p95 latency); 4. Ollama single-loop integral control (first success on real hardware; batch size controls TTFT via GPU concurrency); 9. Cascade control in low-level GPU batch processing factory (found correct abstraction layers on Modal T4: precise batch executor, real backlog queue, measured GPU service time); 11. Closed-loop TTFT controller (velocity-type PI; set points 200/350/500ms with deviation within ±5ms, suppressing load step disturbances).

Core lessons from the 8 failure chapters: Chapter 2a (integration should be based on measured values rather than model predictions); Chapter 3 (cascade inner loop adjusts non-existent queue variables); Chapter5 (verify correctness before relying on metrics); Chapter6 (hardware scheduling behavior affects control architecture effectiveness); Chapter7 (cloud environment abstraction layers introduce signal noise); Chapter8 (control law applied to wrong abstraction layer); Chapter10 (correct controlled variable is TTFT, actuator is arrival rate metering).

Architecture evolution: Chapters1-2 pure simulation (MATLAB→llm_plant.m); Chapters3-4 real hardware (MATLAB/Simulink→Ollama HTTP→Apple Silicon GPU); Chapter6 real queue server (MATLAB→queue_server.py→Ollama→CPU); Chapters7-9 Modal remote GPU (MATLAB→Modal wrapper/vLLM→NVIDIA GPU); Chapter11 closed-loop TTFT controller (run_load_step.py→Modal wrapper→vLLM/Qwen→T4; PI parameters kp=0.03, ki=0.002; cycle 0.1s). Toolchain: MATLAB R2024b+, Control System Toolbox, Python3.11+, Modal, Ollama, vLLM.

Core insight: The control law itself is never wrong; the problem lies in applying it to the wrong abstraction layer. High-level API abstractions of modern LLM services hide underlying signals (e.g., aggregated latency, invalid queue metrics). Key conclusions: Measurement precedes modeling—need to understand the system's real behavior through experiments; control engineering is a combination of science (mathematical models, theory) and art (abstraction layers, variable selection).

For LLM service developers: Expose underlying signals, provide backpressure mechanisms, verify metric correctness; For control engineers: First characterize the controlled object, select the correct abstraction layer, start with simple control; For researchers: Value of failure cases, interdisciplinary thinking, experiment-driven. Project code is reproducible, link: https://github.com/hari-vasudevan/llm-serving-control, blog: https://vasudevanhari.substack.com/.