KrillLM：专为Apple Silicon打造的高性能本地LLM推理引擎

KrillLM is a CLI tool built on Apple's MLX framework, specifically optimized for Apple Silicon (M-series chips). It delivers 1.57x speed improvement and 58% memory savings compared to Ollama, supports multimodal推理 (text, image, audio for Gemma 4 series), and features a complete benchmark system.

Project Overview

KrillLM is a local LLM inference CLI tool designed for Apple Silicon, released as a single binary to provide macOS users with faster, more efficient local AI experiences.

MLX Framework Integration

The tool deeply integrates Apple's MLX framework, leveraging Apple Silicon's unified memory architecture and neural engine for hardware-level optimization—outperforming cross-platform alternatives.

Multimodal Support

• Gemma 4 series: CLI native (text/image), bridge (audio via mlx-vlm), server (full text/image/audio).
• Other models: Llama, Qwen, Mistral, etc., support text-only in CLI/server modes.

Server Mode & API

Offers OpenAI-compatible API via krillm serve command, eliminating CLI overhead, supporting concurrency, and integrating with OpenAI tools.

Key Optimizations

• Native Swift implementation (no Python overhead).
• Unified memory architecture usage (reduces CPU-GPU data transfer).
• Default 4-bit quantization (balances quality and memory).

Core Metrics

• Throughput: 1.6-1.7x vs Ollama.
• Memory: 58% reduction.
• Speed: 1.57x end-to-end latency improvement.

Release Gate Metrics

• Text prefill: 3% below target (acceptable).
• Image prefill: Limited by visual cache.
• Audio: Waiting native support.

Benchmark System

• Compare with Ollama via make bench-compare.
• Reports include model config, test params, performance, env info.
• Gemma4 multimodal tests use fair 4-bit quantization.

1. Developer Testing: Lightweight alternative to Docker, ideal for 16GB MacBooks.
2. Edge Deployment: Single binary, minimal dependencies, fits low-power edge use.
3. Privacy: Local inference avoids sending sensitive data to cloud APIs.

Current State

Pre-release stage with core features complete; open-source on GitHub, accepting community contributions.

Future Plans

• Native audio support for Gemma4.
• Optimize prefill performance (1.5-3x target).
• Expand model family support.

KrillLM represents a trend toward platform-native local LLM optimization. It's a strong Ollama alternative for Apple Silicon, with robust engineering (benchmark system, release gates) and value for developers as both a tool and reference for local AI evolution.