floship-llm: Engineering Practice for Building Reusable LLM Client Libraries

A reusable LLM client library for OpenAI-compatible inference endpoints, providing production-grade features like standardized interfaces, error handling, streaming responses, and retry mechanisms to simplify multi-model integration development.

With the booming development of the large language model (LLM) ecosystem, developers face an awkward reality: every time they integrate a new model provider, they have to repeatedly write similar HTTP client code. OpenAI, Anthropic, Google, Cohere, locally deployed vLLM... Each endpoint has subtle differences—different authentication methods, different request formats, different error codes, different streaming response protocols.

This repetitive work not only wastes time but also introduces inconsistencies. Having multiple LLM clients with distinct styles in one project means doubled maintenance costs and accumulated security risks. When needing to switch models or add new providers, developers often have to modify various parts of the codebase.

floship-llm was born to solve this pain point. It is a reusable LLM client library that provides a unified, robust, production-ready interface abstraction for OpenAI-compatible inference endpoints.

Building a general-purpose LLM client library faces a core tension: on one hand, it needs to provide a unified interface to simplify usage; on the other hand, it needs to retain sufficient flexibility to adapt to the characteristics of different providers. floship-llm's design strikes a balance between the two.

The project chooses the OpenAI API as the compatibility baseline, which is a pragmatic decision. OpenAI's API design has become a de facto industry standard—from open-source vLLM and TGI to commercial Azure OpenAI and Anthropic's compatibility mode, all follow this specification. Using OpenAI as the baseline means maximum ecosystem compatibility.

Although based on OpenAI, floship-llm does not assume all endpoints fully comply with this specification. The library's design supports provider-specific adapters to handle authentication differences, endpoint path differences, response format differences, etc. This adapter pattern keeps the core code concise while leaving room for expansion for special needs.

Modern Python development increasingly values type safety. floship-llm provides complete type annotations, allowing IDEs to offer accurate auto-completion and type checking. Request parameters, response structures, and error types all have clear type definitions, reducing runtime errors and improving the development experience.

floship-llm is not just an HTTP wrapper; it provides a range of features essential for production environments.

The core interfaces exposed by the library follow the conventions of the OpenAI SDK, including:

• Chat Completions: Dialogue completion, supporting multi-turn conversations and tool calls
• Embeddings: Text vectorization for RAG and semantic search
• Completions: Text completion (traditional interface, backward compatible)

A unified method signature means developers can switch seamlessly between different providers by changing configurations instead of code.