FastAPI + Celery + LangChain: Best Practices for Building Production-Grade LLM Inference Services

This article introduces the inference-core project—a backend template for LLM inference services built with FastAPI, Celery, and LangChain. The project aims to address engineering challenges of LLM services (such as long inference times, complex context management, etc.) and provides a production-ready inference service solution through asynchronous processing, task queues, and modular LLM integration.

LLM inference services are fundamentally different from traditional web services: a single call takes a long time (seconds to tens of seconds), requiring handling of complex context management, multi-turn conversation states, and interactions with external data sources (such as vector databases, knowledge graphs). These characteristics demand the use of asynchronous processing, task queues, and modular integration solutions. The inference-core project is exactly a backend template designed to address these challenges.

Asynchronous First

The project takes asynchronous processing as the core principle: non-blocking I/O, high concurrency processing, resource efficiency, avoiding server resource exhaustion from synchronous processing.

Task Separation

Clearly distinguish between synchronous tasks (health checks, status queries, etc.) and asynchronous tasks (long text generation, batch processing, etc.), offloading time-consuming tasks to the background via Celery.

Modular LLM Integration

Implemented based on LangChain: vendor independence (switching between OpenAI/Anthropic/local models), capability combination (retrieval/memory/tool usage), prompt management (version control/A/B testing).

FastAPI Application Layer

• Dependency injection system: reuse resources and avoid repeated initialization;
• Request validation: Pydantic models define API contracts (input restrictions, parameter validation);
• Streaming response: support SSE output for long text generation results.

Celery Task Queue

• Task definition: asynchronous tasks with retry mechanisms;
• Status tracking: maintain task lifecycle (PENDING/STARTED/SUCCESS, etc.);
• Priority queues: implement high/low priority task distribution via routing keys.

LangChain Integration Layer

• Chain abstraction: encapsulate complex processes like conversation chains and RAG chains;
• Tool usage: support LLM calling external tools (search, calculation, etc.).

State Management Strategy

• In-memory storage: suitable for single-instance development environments;
• Redis storage: production multi-instance deployment, supporting persistence and TTL;
• Database storage: long-term conversation history scenarios, supporting structured queries.

Error Handling and Degradation

• Model-level fault tolerance: switch to a backup model when the main model fails;
• Rate limiting: exponential backoff retries, request queue peak shaving;
• Partial failure: return generated content when streaming is interrupted; record successful/failed sub-items in batch processing.

Observability Design

• Structured logging: record information like model, latency, token usage;
• Performance metrics: latency distribution, throughput, queue depth;
• Distributed tracing: OpenTelemetry to trace request links.

Deployment Architecture

• Docker Compose development environment: includes API, Worker, Redis services;
• Kubernetes production deployment: API auto-scaling, independent Worker strategies, configuration management (ConfigMap/Secret).

Performance Optimization

• Model inference: batch processing, KV cache reuse, quantization and distillation;
• System-level: connection pool reuse, semantic caching, load balancing (round-robin/latency priority).

Adding New LLM Providers

Implement LangChain's LLM base class and customize model calling logic.

Custom Task Types

Define domain-specific tasks via Celery's shared_task decorator.

Middleware Extension

Add request/response processing logic using FastAPI's middleware decorator.

The inference-core project provides a collection of engineering practices for production-grade LLM services, combining three key technologies: FastAPI (high-performance development), Celery (asynchronous tasks), and LangChain (LLM integration) to solve infrastructure problems. Future LLM service architectures will continue to evolve, but core principles like asynchronous processing and task queues will remain applicable—mastering these fundamentals will keep you competitive.