Warply：面向LLM推理的开源Python控制平面

Warply is an open-source Python control plane designed to simplify the deployment, routing, scaling, and observation of large language model (LLM) inference services. It converts Python service intents into executable deployment plans via declarative configuration, supports backends like SGLang and NIXL, and provides an OpenAI-compatible HTTP client interface. This post will break down its background, features, architecture, usage, limitations, and value.

With the explosive growth of LLM inference demands, enterprises and research teams face rising deployment complexity. Traditional LLM service stacks require significant infrastructure work to integrate GPUs, cloud services, engine configurations, KV transfer settings, routing, health checks, and client endpoints. Warply addresses this pain point by offering a user-oriented control plane, allowing researchers or small teams to describe their desired inference service system in Python, launch it on controlled infrastructure, check deployment status, and iterate without becoming full-time inference infrastructure engineers.

Warply is an open-source Python control plane for full-lifecycle management of LLM inference endpoints, with a core design philosophy of "Python-first". Its core capabilities include:

• SDK: DisaggEngine, Pool, up(), down(), local scale(), status(), stats(), events(), client(), generate()
• Compiler: Deterministic DeploymentPlan, engine.plan(), engine.export_yaml()
• Inference engines: SGLang adapter (supports prefill, decoding, routing process config)
• KV transfer: NIXL for CUDA (kv_transfer="auto" auto-resolves to NIXL on known CUDA GPUs)
• Cloud services: SkyPilot Lambda task rendering and dry-run (CoreWeave in plan)
• Deployment topology: 1 prefill node + N decode nodes in a SkyPilot multi-node cluster
• Client: Local mock client + OpenAI-compatible HTTP client for deployed routers
• Hardware planning: CUDA/ROCm accelerator profiles (ROCm launch disabled pending validation)
• Speculative decoding: Supports engine-native, MTP, EAGLE, DFlash, and draft model modes (config/plan export)

Warply uses a layered architecture separating the control plane from execution engines: Python serving intent → compiler → DeploymentPlan → provider adapter (SkyPilot, local mock, future direct providers) → engine adapter (SGLang now; vLLM/TensorRT-LLM later) → KV adapter (NIXL now; MORI/Mooncake/LMCache candidates later) → router + client (OpenAI-compatible endpoint). This separation ensures a stable control plane while allowing providers to supply infrastructure and engines to handle token generation, enabling the same deployment intent to grow with new clouds, hardware, engines, observability, and serving strategies without changing application-facing endpoint contracts.

Installation: git clone https://github.com/afifi-yusuf/warply.git
cd warply
pip install -e ".[dev]"

Local GPU-free lifecycle test: import warply as wp
engine = wp.DisaggEngine(
model="meta-llama/Llama-3.1-8B",
prefill=wp.Pool("1xH100", replicas=1),
decode=wp.Pool("1xH100", replicas=1),
backend="sglang",
kv_transfer="nixl",
cloud="local",
)
engine.up()
print(engine.generate("hello"))
print(engine.status())
print(engine.stats())
print(engine.events())
print(wp.doctor(engine).to_dict())
engine.down()

Cloud Dry-Run: Set WARPLY_SKYPILOT_DRY_RUN=1 to test without GPU/cloud costs: WARPLY_SKYPILOT_DRY_RUN=1 python - <<'PY'
import warply as wp
engine = wp.DisaggEngine(
model="meta-llama/Llama-3.1-8B",
prefill=wp.Pool("1xH100", replicas=1),
decode=wp.Pool("1xH100", replicas=2),
cloud="lambda",
)
engine.up()
print(engine.status().endpoint)
engine.down()
PY

Offline Pre-check: Use wp.doctor() to check compatibility: report = wp.doctor(engine)
print(report.overall, report.ready)
lambda_dry_run = wp.doctor(
wp.DisaggEngine(
model="meta-llama/Llama-3.1-8B",
prefill=wp.Pool("1xH100"),
decode=wp.Pool("1xH100", replicas=2),
cloud="lambda",
),
dry_run=True,
)
print(lambda_dry_run.to_dict())

Known Limitations:

• cloud="local" is a mock runtime (no actual SGLang process启动)
• Real-time cloud scale() not implemented (needs restart with new specs)
• Cloud disaggregation supports prefill.replicas ==1 and decode.replicas >=1
• CUDA/SGLang/NIXL is the only actively validated real-time target
• engine.stats() reports control plane facts (backend metrics empty until runtime exposure)
• AMD Instinct specs compile to ROCm-aware plans but real-time launch fails (pending ROCm image/MORI validation)
• Speculative decoding modes are compiled/exported but backend flags not enabled (pending SGLang/vLLM validation)
• KV-aware routing, runtime stats, vLLM/TensorRT-LLM adapters, Dynamo runtime integration, and policy-driven optimization are roadmap items.

Roadmap: Validating real-time SGLang/NIXL Lambda services, runtime stats, and 1:N prefill/decode placement; expanding via provider/engine/KV adapters; improving topology/scaling/routing/decoding strategies with production signals.

Warply represents a key trend in LLM inference infrastructure: separation of control plane and execution engine. Key advantages:

1. Lower barrier: Researchers don’t need Kubernetes/cloud expertise to deploy production-level LLM services.
2. Portability: Same Python code runs in local mock, Lambda, CoreWeave, etc.
3. Observability: Built-in doctor() tool and stats interface for clear system status.
4. Ecosystem compatibility: OpenAI-compatible client interface works with existing tools/frameworks.
5. Extensibility: Adapter pattern allows easy integration of new providers, engines, and KV transfer backends.

Warply provides a Python-native, declarative, observable solution for LLM inference service deployment and management. It abstracts complex distributed system config into intuitive Python code while remaining flexible for various deployment scenarios. For research teams and small enterprises wanting to quickly build LLM inference infrastructure, Warply offers an attractive middle ground—simpler than manual cloud config, more controllable than fully managed API services. As it matures (real-time cloud integration, runtime stats, policy optimization), Warply is poised to become an important tool in the LLM inference infrastructure space.