Building a Complete AI Image Generation System from Scratch: Stable Diffusion v1.5 + LoRA Fine-Tuning Practice

This article introduces an end-to-end image generation system project based on Stable Diffusion v1.5, covering LoRA fine-tuning, FastAPI deployment, Gradio interface, and Android client integration, demonstrating the entire process from training to multi-end deployment. The project uses LoRA technology for efficient fine-tuning—only 0.185% of the parameters need to be trained to achieve customized generation capabilities, and it supports running on consumer-grade GPUs (such as NVIDIA RTX3060 6GB), providing a complete reference for developers.

Generative AI is reshaping the creative industry, and Stable Diffusion, as an open-source representative, provides strong customization capabilities. This project implements an end-to-end system, with the core highlight being coverage of the entire link from data preparation to mobile deployment. The tech stack includes: model layer (SD v1.5 + LoRA), training optimization (FP16 mixed precision, gradient checkpointing, XFormers), service layer (FastAPI + Uvicorn), interface layer (Gradio), and mobile end (Android native app). The layered architecture adapts to consumer-grade hardware and supports multi-terminal access.

LoRA (Low-Rank Adaptation) is the core technology. Compared to full-parameter fine-tuning (861 million parameters), only 1.59 million parameters (0.185%) need to be trained to achieve style transfer. Training process: 1. Data preparation: Use build_subset.py to build subsets, and cache_latents.py to pre-encode images into latent space representations (to avoid repeated VAE computations); 2. Training configuration: Learning rate of 1e-4 (cosine annealing schedule), batch size of 1 (adapted to 6GB VRAM), enable FP16 mixed precision, gradient checkpointing, attention slicing, and XFormers optimization, supporting 4 epochs of training.

After training, the model is served in three ways: 1. Gradio Web interface (app.py): Supports text-to-image generation, guidance scale adjustment, inference step control, negative prompts, seed fixing, size selection, and result saving; 2. FastAPI backend (api.py): Encapsulated as a REST API with automatically generated documentation (at the /docs endpoint); 3. Android client: Uses Kotlin + Jetpack Compose to build the UI, Retrofit to communicate with FastAPI, the server handles computations, and the mobile end is responsible for interactive display.

The project adopts multiple optimization strategies: 1. Memory optimization: FP16 mixed precision (halves VRAM usage), gradient checkpointing (trades computation for memory), XFormers efficient attention; 2. Latent space caching: Encode images into latent variables and persist them before training, saving training time; 3. Parameter efficiency: LoRA training parameters account for 0.185%, bringing advantages such as faster training speed, low storage cost, dynamic weight switching, and stable base model. These optimizations enable the system to run on RTX3060.

Application scenarios include e-commerce product image generation, game asset prototyping, advertising material production, personalized avatar generation, etc. (by injecting domain knowledge via LoRA). Future expansion directions: Functionally support image-to-image, inpainting, ControlNet pose control, SDXL model, and DreamBooth training; Engineering-wise, improve cloud deployment, user authentication, generation history gallery, and model version management.

This project provides a complete reference for generative AI developers, with core values: end-to-end coverage of the entire lifecycle, hardware-friendly (runs on RTX3060), clear architecture (layered decoupling), and rich engineering details. It is recommended that developers start by understanding the principles of LoRA and the Diffusers library, gradually tune hyperparameters, and choose deployment solutions based on scenarios. Such open-source projects promote the democratization of generative AI technology.