QORA-4B: A Multi-Modal Inference Engine Built Entirely in Rust—A New AI Choice Free from Python Dependencies

QORA-4B is a fully Rust-developed multi-modal large model inference engine that eliminates dependencies on Python and CUDA. It runs as a single executable, supports Vulkan (Windows/Linux) and Metal (macOS) GPU acceleration, and is based on the Qwen3.5-4B architecture. This极简 deployment模式 addresses key pain points in current LLM deployment, enabling edge device usage and portable AI applications.

Mainstream LLM deployment relies heavily on Python ecosystems and CUDA toolchains, leading to tedious environment setup for developers and compatibility issues for end-users. These dependencies limit portability, making it nearly impossible to deploy on resource-constrained edge devices.

• Pure Rust Implementation: All components (matrix operations, attention mechanisms, image/text processing) are written in Rust, ensuring memory safety and zero-cost abstractions.
• Zero External ML Frameworks: No reliance on PyTorch/TensorFlow; all operators are handwritten for full control.
• Cross-Platform GPU Acceleration: Uses Burn framework's wgpu backend to auto-detect GPUs (Vulkan/Metal) and fallback to CPU if needed.
• Smart System Sensing: Auto-detects RAM/CPU cores to adjust generation parameters dynamically.

QORA-4B uses a hybrid architecture (24 DeltaNet layers +8 full attention layers, repeated in 3+1 cycles).

• DeltaNet: Gated Linear Attention with O(1) memory complexity (constant per-token memory), causal convolution, and multi-head design (16 QK heads +32 V heads).
• Full Attention: Group Query Attention (16 Query heads →4 KV heads), partial RoPE (64/256 dims), and output gating.
• Visual Capabilities: 24-layer ViT encoder (supports image/video input via Conv3d embedding, 2D spatial RoPE for spatial relations).

Speed: GPU (3.3 tok/s decode, ~4.5 tok/s prefill) vs CPU (1.3/1.9 tok/s). VRAM需求: ~2GB (Q4 quantized). Quantization: Q4 (3.5GB, good quality, fast) vs F16 (7.5GB, best quality, slower CPU). System Adaptation: Adjusts think budget/max tokens based on available memory: <4GB (minimal),4-8GB (restricted),8-12GB (normal),≥12GB (full capacity).

Command Line Examples:

• Text generation: qor4b --prompt "Explain quantum computing" --max-tokens 500
• Image processing: qor4b --prompt "What's in this image?" --image photo.jpg
• Video processing: Use frame directory (extract via ffmpeg: ffmpeg -i video.mp4 -vf \"select=not(mod(n\\,30))\" -frames:v 4 frames/frame_%02d.png) Platforms: Precompiled binaries for Windows x86_64 (Vulkan), Linux x86_64 (Vulkan), macOS aarch64 (Metal). Build: cargo build --release (CPU) or with --features gpu (Vulkan) / --features gpu-metal (Metal).

Use Cases: Edge devices (industrial controllers, IoT), offline privacy (medical/financial docs), fast prototyping, cross-platform apps. License: Apache 2.0 (same as Qwen3.5-4B), allowing commercial use and secondary development. Summary: QORA-4B offers unique advantages in portability and deployment ease, suitable for developers focusing on edge/cross-platform AI solutions.