Kiln: A Single-Model LLM Inference Server Supporting Real-Time Online Learning

Kiln is an innovative LLM inference server. It enables parallel training and serving via LoRA hot-swapping technology, allowing models to perform real-time fine-tuning while continuously providing services. This solves the dilemma of separated training and deployment in traditional model services and supports continuous learning.

Traditional LLM applications face a dilemma between general models and fine-tuned models: general models are flexible but perform poorly in specific domains; fine-tuned models are accurate but require downtime for retraining and redeployment, affecting business continuity. With continuous data generation in production environments, the traditional "train-deploy" separated architecture cannot meet the needs of continuous learning. How to enable model evolution without downtime is a major challenge.

Kiln proposes to achieve single-model real-time online learning via LoRA hot-swapping technology. Its core is the "Train while you serve" concept, which breaks the mindset of separated training and serving. It allows models to continuously provide services while receiving new data for fine-tuning, and after completion, the hot-swapping update capability eliminates the need for downtime.

LoRA is a parameter-efficient fine-tuning method that does not change the base model weights. It trains a small number of low-rank matrices to adapt to tasks, featuring efficient storage, flexible switching, and combinability. Kiln maintains a base model and multiple LoRA adapters. During serving, it dynamically loads/unloads LoRAs, and after training new LoRAs in the background, it hot-swaps them into the service.

Kiln adopts a single-model architecture with high resource utilization. All requests share the base model, and capability customization is achieved through different LoRA adapters. It is suitable for scenarios such as multi-tenant SaaS (independent LoRAs for customers with data isolation), A/B testing (small traffic verification of new LoRAs), and progressive learning (continuous fine-tuning to improve domain performance).

Implementing real-time online learning requires solving three major problems: 1. Data flow management: efficiently collecting and preprocessing production data, including cleaning, deduplication, and quality screening; 2. Training-serving resource balance: resource scheduling for inference and training on the same server to ensure training does not affect inference latency; 3. Versioning and rollback: version management for frequent LoRA updates, supporting quick rollback to address model degradation.

Kiln is applicable to various scenarios: customer service robots (daily dialogue fine-tuning to make responses more aligned with the enterprise style and customer expectations), code assistants (learning team coding standards and API patterns to provide accurate code suggestions), and content moderation (optimizing judgment criteria through human feedback to adapt to policy changes).

Kiln demonstrates a service model for LLM continuous learning, transforming models from static assets into continuously evolving services, similar to the DevOps concept (automation shortens the feedback and improvement cycle). As LLMs move from experiments to production, such infrastructure will become more important, representing a key direction in LLM engineering: making models continuously better.