Modelito: Design and Practice of a Lightweight Multi-Provider LLM Abstraction Library

Modelito is a lightweight LLM abstraction library designed for Python developers, aiming to solve the pain point of switching between multiple LLM providers. Through a streamlined abstraction layer and optional dependency design, it supports seamless switching from local Ollama to cloud-based services like OpenAI, Claude, and Gemini. It also provides a test-friendly stub mechanism, helping developers achieve flexible LLM integration with minimal code changes and dependency overhead.

In LLM application development, multi-provider integration faces four major challenges:

1. Dependency Bloat: Introducing multiple official SDKs leads to rapid dependency growth and increased maintenance complexity;
2. Interface Differences: Different providers have significant differences in API parameter names, calling methods, etc., requiring a lot of adaptation code;
3. Testing Environment Issues: Real LLM calls are impractical in CI/CD or offline environments, and SDKs cannot work without API keys;
4. Switching Cost: Hardcoding specific SDK logic results in high migration costs when changing providers.

Design Philosophy

Modelito adopts a lightweight strategy with core principles including:

• Minimal Dependencies: Basic installation does not force any SDK dependencies; optional dependencies are loaded on demand;
• Test-Friendly Stubs: When SDKs are not installed or APIs are unavailable, deterministic stubs return preset responses;
• Progressive Enhancement: Basic functions are ready out of the box; installing optional dependencies upgrades to real SDK clients.

Core Components

• OllamaProvider: Supports a layered degradation strategy with HTTP API priority, CLI fallback, and stub as the last resort;
• Cloud Adaptation: OpenAIProvider, ClaudeProvider, GeminiProvider, etc., follow a unified interface—switching only requires configuration changes.

Installation Methods

• Basic Installation: pip install modelito (only core abstractions and stubs);
• Development Mode: pip install -e .[dev] + install dev-requirements.txt;
• On-Demand Installation: e.g., pip install -e .[ollama,tokenization] or pip install -e .[openai,anthropic].

Typical Scenarios

1. Development-Production Separation: Use Ollama for development, switch to OpenAI in production without modifying business code;
2. CI/CD Testing: Stub mechanism supports offline testing;
3. Multi-Model Comparison: Unified interface simplifies evaluation of outputs from different models.

Technical Implementation Highlights

• Type Safety: Uses type annotations and mypy static checks;
• CI Guarantee: GitHub Actions automatically runs type checks and unit tests; Ollama tests can be triggered optionally;
• Version Management: Follows semantic versioning and supports local wheel package installation.

Downstream Applications

Modelito has been used in:

• BatLLM: A local model batch processing tool;
• mail_summariser: An email summary generation service.

Solution Comparison

• LangChain: Comprehensive but heavy, suitable for complex Agent systems;
• LiteLLM: Focuses on multi-provider routing and provides proxy mode;
• Modelito: Lightweight client abstraction, suitable for underlying dependency integration.

Application Recommendations

Suitable Scenarios: Controlling dependency size, simulating LLM tests, serving as an underlying library, switching between local Ollama and cloud services; Unsuitable Scenarios: Complex Agent orchestration, advanced features (streaming/function calling), enterprise-level features (routing/caching).

Modelito is a concise, dependency-controllable, test-friendly lightweight choice, suitable for pragmatic developers.