AI Lab: End-to-End Comparative Experiment of Four Local Large Language Model Inference Tech Stacks

AI Lab is an open-source experimental sandbox designed to resolve the technical selection dilemma in local large language model (LLM) deployment. By using the same model (Qwen 2.5 0.5B Instruct, Q4_K_M GGUF format) and prompts, the project compares four local LLM inference solutions—llama-cpp-python, OllamaSharp, LLamaSharp, and Blazor Server—helping developers intuitively understand the trade-offs between different deployment and abstraction levels, rather than providing performance benchmarks.

With the rapid development of LLMs, local deployment has become a popular option due to advantages like data privacy, cost control, and low latency. However, numerous inference frameworks (e.g., llama.cpp, Ollama, LLamaSharp) have left developers confused about selection. As a comparative reading lab, AI Lab helps resolve this pain point by providing side-by-side code implementations, enabling developers to understand the design philosophies and trade-offs of different tech stacks.

The core of the project is "one problem, four solutions" to ensure fair comparison. The four tech stacks cover dimensions such as programming language (Python vs .NET), inference location (in-process vs external service), communication protocol (binding/interop vs HTTP), and interaction mode:

Tech Stack	Programming Language	Inference Method	Communication Mechanism	Interaction Interface
smoke_llama_cpp.py	Python	In-process with llama.cpp	Python binding	One-time completion
dotnet-client	.NET 10	External Ollama service	HTTP	Interactive console chat
dotnet-llamasharp	.NET 10	In-process with llama.cpp	Native interop	One-time streaming output
dotnet-blazor	.NET 10	External Ollama service	HTTP + SignalR	Blazor Server web UI
The project adopts an "anti-DRY" design, where each tech stack is self-contained to avoid shared libraries breaking the comparative reading experience, allowing developers to fully understand the complete picture of each solution.

smoke_llama_cpp.py (Python Direct Binding)

Closest to bare-metal implementation, directly interacts with llama.cpp with zero intermediate layers, self-contained (auto-downloads model), minimal dependencies, and optional GPU support. Suitable for Python data science workflows but has weak interactivity.

dotnet-client (Ollama HTTP Client)

Service-oriented architecture, separating model inference from the client, integrates Microsoft.Extensions.AI, supports interactive chat and streaming responses. Suitable for rapid prototyping and leveraging the Ollama ecosystem.

dotnet-llamasharp (.NET In-Process Inference)

Pure .NET solution with no external dependencies, manually manages ChatML templates, supports token-level streaming. Suitable for integrating LLMs into .NET applications.

dotnet-blazor (Web Chat Interface)

Blazor Server architecture, modern UI, full features (Markdown rendering, image attachments, etc.), SignalR streaming communication. Suitable as a reference for production-ready applications.

Runtime Environment:

• Python: 3.11+, venv, depends on llama-cpp-python;
• .NET: 10 SDK, .slnx format, dotnet CLI;
• External Service: Ollama (default endpoint http://127.0.0.1:11434);
• Hardware: ~400MB disk space, default CPU inference.

Model Cache Contract:

• Shared cache path: ~/.cache/ai-lab/gguf/qwen2.5-0.5b-instruct-q4_k_m.gguf;
• Only smoke_llama_cpp.py populates the cache; Ollama-related stacks use independent model management.

Technical Selection Reference: Helps teams understand trade-offs like in-process vs service-oriented, Python vs .NET, and abstraction level complexity; Learning Resource: Moderate code volume, progressive from simple to complex, showing production details; Architecture Decision Reference: Multi-tech-stack comparison organization, balance between code readability and engineering practices, shared resource management.

Limitations: No tests/CI/CD, simple prompts, default CPU inference; Expansion Directions: Add Rust/Go/Node.js tech stacks, performance benchmarks, GPU comparison, multimodal extensions.

By providing side-by-side implementation comparisons, AI Lab helps developers understand the design trade-offs in local LLM deployment, which is an effective model for technical learning. As a stable reference point, it not only shows "how to do it" but also explains "why it's designed this way", providing a foundation for deep understanding in local LLM deployment selection.