LMRunner: Lightweight Local LLM Inference Endpoint Management Tool

LMRunner is a concise CLI tool designed to start and manage local large language model inference endpoints based on llama.cpp. It supports interactive configuration management and multi-endpoint concurrency control, aiming to simplify the management of local LLM deployment.

Key highlights:

• Acts as a friendly frontend for llama.cpp (not a replacement)
• Solves pain points like complex parameter memorization and manual process management
• Offers features like interactive commands, unified config management, and full endpoint lifecycle control

Background: Local LLM Deployment Pain Points

With the miniaturization trend of large language models, more developers run LLMs locally. llama.cpp is a popular local inference engine with great performance and cross-platform support, but directly using its CLI has drawbacks:

• Need to remember complex startup parameters
• Manual process management
• Lack of unified configuration management

LMRunner was created to address these issues.

Installation & Configuration

Installation

• From source: Clone the repo (git clone https://github.com/jschw/LMRunner.git), cd into it, then python -m pip install -e .
• Standard pip install (precompiled llama-server): Use standard pip, then specify llama.cpp path in config (default: /lmrunner/Llamacpp/llama.cpp/build/bin/llama-server)
• Optional bindings: pip install --upgrade lmrunner[llamacppbindings] (convenient but may not be latest)

Configuration

• Uses two JSON files: llm_config.json (model config) and llm_server_config.json (server config)
• Commands: /editlmconf (open model config), /editserverconf (open server config), /refreshconf (reload config)

These enable flexible, transparent management with familiar editors.

Core Functionalities

Interactive Command Design

• IRC-like / commands (e.g., /startendpoint, /stopendpoint)
• Unified interactive prompt to avoid complex parameter memorization

Endpoint Lifecycle Management

• Start: /startendpoint <name> (by config name)
• Restart/Stop: /restartendpoint, /stopendpoint, /stopallendpnts (stop all)
• Status: /llmstatus (show all endpoint states)

Additional Features

• Model directory update: /updatemodels (update model list from GitHub)
• Auto-start: /setautostartendpoint <name> (auto-launch on next tool start)

These cover full management of local LLM endpoints.

Use Cases & Value

LMRunner is ideal for:

1. Multi-model development: Switch between models easily without remembering parameters
2. Local API service: Stable endpoint management with auto-start for quick recovery after system restart
3. Quick prototype validation: Fast model testing via simple startup and model directory updates

It significantly improves daily development efficiency for llama.cpp users.

Design Philosophy & Conclusion

Design Philosophy

• Follows Unix's "do one thing well" principle: Focuses on simplifying llama.cpp endpoint management (not replacing it)
• Lightweight and easy to use, with loose coupling to llama.cpp updates

Conclusion

LMRunner is a practical choice for developers using llama.cpp who want to simplify endpoint management. As local AI apps become more popular, such tooling will grow in importance.

It offers a clean, efficient way to manage local LLM endpoints, making daily workflows smoother.