Production-Grade AI Backend Service: Engineering Practice of LLM Inference and Vector Retrieval

A production-mode backend service demonstrating how to integrate LLM inference APIs and vector similarity search into scalable REST endpoints, with complete implementations including RAG, retry mechanisms, streaming responses, etc.

With the popularization of large language models, more and more teams want to integrate AI capabilities into existing products. However, there is a huge engineering gap between prototype and production: How to handle API rate limits? How to implement reliable error recovery? How to integrate vector databases with business data?

This open-source project provides a proven production-mode backend service architecture, demonstrating engineering practices adopted by companies like Mastercard, Google, and Amazon when building AI features.

The system adopts a layered architecture design, with core components including:

• FastAPI Application Layer: Provides high-performance asynchronous REST API endpoints
• LLM Service Layer: Encapsulates OpenAI API calls, including features like retry, streaming, JSON output, etc.
• Vector Service Layer: Implements document embedding and similarity search based on ChromaDB
• Data Model Layer: Uses Pydantic to define type-safe request/response contracts

This layered design follows the Separation of Concerns principle, allowing each component to be independently tested, extended, and replaced.

The llm_service.py module demonstrates how to correctly call LLM inference APIs, with the following key features:

Intelligent Retry Mechanism: Uses the Tenacity library to implement exponential backoff retries, automatically handling rate limits and temporary failures. When the API returns a 429 error, the system waits 1 second, 2 seconds, 4 seconds, increasing the interval gradually to avoid overwhelming upstream services.

Streaming Response Support: For real-time scenarios like chat interfaces, supports streaming token output to enhance user experience.

Structured JSON Output: Through carefully designed prompt engineering, ensures that the LLM returns parsable JSON format, avoiding fragile string parsing.

Token Usage Tracking: Records token consumption for each call, facilitating cost analysis and usage control.

The vector_service.py module encapsulates ChromaDB operations and implements the core capabilities of RAG (Retrieval-Augmented Generation):

Text Embedding: Converts documents into high-dimensional vectors to capture semantic meaning. Texts with similar meanings are closer in vector space, enabling semantic-based search instead of simple keyword matching.

Similarity Search: Receives query text and returns the most relevant documents from the knowledge base. Supports adjusting the top_k parameter to control the number of returned results.

CRUD Operations: Complete document create, read, update, delete interfaces, supporting batch operations to improve efficiency for large-scale data import.

RAG (Retrieval-Augmented Generation) solves two fundamental problems of LLMs: knowledge cutoff and hallucinations. Its workflow is as follows:

1. Retrieval Phase: Convert the user's question into a vector and search for the most relevant documents in the knowledge base
2. Augmentation Phase: Inject the retrieved documents into the LLM prompt as context
3. Generation Phase: The LLM generates answers based on the provided context instead of relying on its training memory

This method ensures that answers are based on the enterprise's real data and can reference specific sources.

The system provides the following REST endpoints:

• /health: Health check, including dependency service status
• /api/analyze: General text analysis, supporting custom instructions
• /api/documents: Document management (create, update, delete)
• /api/search: Vector similarity search
• /api/rag: Complete RAG question-answering flow

Each endpoint has a corresponding Pydantic model defining the request and response formats, automatically generating OpenAPI documentation.

The entire system uses Pydantic for data validation and serialization, capturing type errors during development and validating input data during runtime. This is more robust than the typical practices of dynamic languages.