local-llms: Production-Grade Local LLM Deployment and Evaluation Toolchain

local-llms is a production deployment solution for local large language models based on llama.cpp, optimized specifically for NVIDIA CUDA environments. It provides systemd service management, OpenAI-compatible API, multi-backend support, and a complete evaluation framework, solving engineering problems from experimental to production environments.

With the improvement of large language model capabilities, enterprises consider local deployment due to data privacy, cost control, and low-latency requirements, but face engineering issues such as service persistence, API compatibility, model management, and performance evaluation. local-llms addresses these problems and provides a production-grade toolchain for NVIDIA GPU environments.

1. Configuration System: Uses YAML layered configuration (hardware/providers/profiles/endpoints) with priority order: endpoint > profile > hardware default, and performs capability checks during the configuration phase; 2. Multi-Backend Support: Can switch between inference backends like llama.cpp and ik_llama.cpp; 3. Production Service: Implements features such as automatic startup, process daemon, and log integration via systemd.

Quick Deployment: Clone the repository → run setup.sh (initialize dependencies, compile binaries, install systemd service); Daily Operations: CLI tools to manage endpoints and models; Evaluation System: Built-in adapters like local_smoke/mmlu/gsm8k/niah/frontend_agentic, supporting flexible execution and report generation.

Conclusion: local-llms is a practical local LLM deployment solution, focusing on NVIDIA environments, providing modular configuration, comprehensive evaluation, and production features; Limitations: Only supports CUDA, complex configuration; Recommendations: Learn dependencies from SETUP.md → understand configuration from CONFIGURATION.md → establish benchmarks from BENCHMARKING.md → select models for experiments from MODELS.md.