Building a Local LLM API Service with FastAPI and Ollama: Zero-Cost Large Model Inference

This article introduces the open-source project local-llm-api, which is built on FastAPI and Ollama. It enables zero-cost local large model inference and provides REST API services. It supports multi-turn conversations, image description, text classification, and other features, without calling paid APIs, making it suitable for private deployment needs.

With the popularization of LLM technology, developers face issues like costs, latency, and data privacy when integrating AI capabilities via third-party APIs. Local deployment can solve these problems, but it has high barriers (model loading, inference optimization, API encapsulation, etc.). local-llm-api provides an out-of-the-box solution to simplify the setup of local LLM services.

local-llm-api is an open-source project based on Python FastAPI, with the core goal of simplifying the APIization of local LLMs. It uses Ollama as the underlying model runtime engine and integrates Alibaba's Qwen2.5-VL 3B multimodal model by default. The tech stack is all open-source and business-friendly: FastAPI (MIT), Uvicorn (BSD), Ollama (MIT), Qwen2.5-VL (Apache 2.0), Streamlit (Apache 2.0), which can be used in commercial projects.

The project provides 7 main API endpoints:

1. /health: Health check to verify the status of the service and Ollama backend
2. /chat: Multi-turn conversation with conversation history maintenance
3. /generate: One-time text generation, supporting parameters like temperature and max tokens
4. /describe-image: Image description (based on Qwen2.5-VL's multimodal capabilities)
5. /summarize: Long text summarization
6. /classify: Text classification into predefined categories
7. /extract-keywords: Keyword extraction These endpoints cover common LLM application scenarios.

The project also has several advanced features:

• Streaming response (SSE): Real-time output to enhance interaction experience
• Dynamic model switching: Each request can override the default model via the model parameter
• Full parameter control: Supports generation parameters like temperature, max_tokens, top_p, etc.
• Request logging: Recorded to SQLite and log files
• Streamlit visual interface: Supports PDF/image uploads for non-technical users
• Docker support: One-click containerized service building to simplify deployment

Steps are as follows:

1. Install Python 3.10+ and Ollama (download from official website is recommended)
2. Pull the model: ollama pull qwen2.5vl:3b
3. Clone the project: git clone https://github.com/sfc38/local-llm-api.git
4. Install dependencies: Create a virtual environment, activate it, then run pip install -r requirements.txt
5. Start the service: uvicorn app.main:app --reload
6. Test: Visit http://127.0.0.1:8000/docs to view the Swagger UI, or use curl to test the generation endpoint.

This project is suitable for multiple scenarios:

• Enterprise internal tools: Deployed in private networks to handle sensitive documents
• Development and testing environments: Prototype verification, replacing paid APIs
• Edge computing scenarios: Running lightweight models in resource-constrained environments
• Learning and research: Learning FastAPI design, LLM integration, etc.
• Customized AI services: Extending business logic to build vertical domain applications

Limitations and Future: The project plans to add features like Oracle Cloud deployment guide, conversation history limits, file upload endpoints, rate limits, API key authentication, etc. Summary: local-llm-api is well-designed and has comprehensive documentation, lowering the threshold for local LLM deployment and providing a complete solution. It is suitable for developers exploring local LLM applications, with high code quality that can serve as a foundation for learning or secondary development. It is recommended to try this project, especially for scenarios with zero-cost and private deployment needs.