Ferrum: A High-Performance LLM Inference Engine Written in Pure Rust

Ferrum is a Rust-native LLM inference engine designed to address Python's runtime dependencies, performance bottlenecks, and deployment complexity. Key features include zero Python dependency, single binary deployment, support for text generation, speech recognition/synthesis, embedding vectors, OpenAI-compatible API, and hardware optimizations (CUDA/Metal). It aims to provide a lightweight, efficient alternative for LLM deployment in production and edge environments.

Python has long dominated LLM deployment but faces issues like runtime dependencies and deployment complexity. Ferrum (ferrum-infer-rs) is a Rust reimplementation of an LLM inference engine. Core selling points: single binary, no Python/runtime dependencies. Installation via cargo: cargo install ferrum-cli or source build. For NVIDIA GPUs, add CUDA feature flag with CUDA_HOME=/usr/local/cuda cargo build --release --features cuda ....

Ferrum supports diverse AI capabilities:

• Text generation: LLaMA series (Llama3.x, TinyLlama), Qwen3/Qwen2 series (0.6B-4B), with CUDA/INT4/tensor parallel.
• Speech recognition: OpenAI Whisper (all models) with Metal acceleration, supports multiple audio formats.
• Speech synthesis: Qwen3-TTS with voice cloning (5s reference), streaming (2.5s first output), multi-language.
• Embedding vectors: CLIP/Chinese-CLIP, SigLIP, BERT (Chinese included).

CUDA Optimizations: Custom CUDA decoders (2x speedup for Qwen3/LLaMA), INT4 quantization (69% memory reduction), CUDA Graph (+18% speed), tensor parallel, batch decoding, paged KV cache, Flash Decoding. Metal Optimizations: Custom GEMM kernels, fused layers, Flash Attention, zero-copy memory on Apple Silicon (M4 Max: Qwen3-TTS 2.8x real-time). Benchmarks: RTX PRO6000 (Blackwell):

Mode	FP16 (eager)	FP16+CUDA Graph	INT4 (GPTQ+Marlin)
Single req	70.3 tok/s	82.9 tok/s (+18%)	130.4 tok/s
4 concurrent	109.4 tok/s	—	124.2 tok/s
Memory	~8GB	—	~2.5GB (-69%)
Whisper: large-v3-turbo (72s for5min audio,4.2x real-time); tiny (20s,15x real-time).

API: Supports OpenAI-compatible endpoints: /v1/chat/completions (streaming), /v1/audio/transcriptions, /v1/audio/speech, /v1/embeddings, /v1/models—drop-in replacement for OpenAI API. Architecture: Modular Rust workspace: ferrum-types (shared types), ferrum-interfaces (core traits), ferrum-runtime (backends), ferrum-engine (Metal kernels), ferrum-models (model architectures), ferrum-kernels (CUDA), ferrum-server (HTTP API), etc.

• Edge deployment: Single binary, no Python—ideal for IoT/embedded/edge servers.
• Privacy-first: Runs locally, data never leaves the machine.
• High-performance production: CUDA/INT4 optimizations, batch processing—good for consumer GPUs.
• Multi-modal apps: Integrates text, speech, embedding—build voice interaction or RAG systems without multiple tools.

Roadmap: Speculative decoding, more models (Mistral, Phi, DeepSeek), Qwen2 CUDA runner. Conclusion: Ferrum redefines LLM deployment with Rust—zero dependency, high performance, easy deployment. It’s a great choice for developers prioritizing simplicity, performance, privacy. Open-source (MIT license) for community contribution. Ferrum paves a new path for efficient, deployable AI engines.