Synapse: Technical Architecture and Edge Deployment Practice of a Cross-Platform Modular LLM Inference Engine

Synapse is a modular LLM inference engine built with Rust and Zig SIMD kernels, supporting full-platform deployment from desktop to browser and embedded devices. Key features include: modular pluggable design (config-driven, easy to add new models), multi-language tech stack (Rust for safety/abstraction, Zig for optimized SIMD kernels, Metal for Apple GPU), comprehensive quantization support (f32 to Q4_K), and support for emerging architectures like LEWM (world model) and state space models (Mamba/RWKV). It also enables edge/IoT deployment (e.g., ESP32-P4) and lightweight browser runtime via WASM.

In LLM inference, developers face a dilemma: high-performance local frameworks rely on complex C++ codebases, while easy-to-use solutions struggle in resource-limited environments. Synapse aims to break this by combining Rust and Zig to build a full-stack engine that balances native performance and cross-platform compatibility. Its core design philosophy is modularity and configurability—each component is a pluggable trait, allowing new models to be added via JSON config and weight mapper without modifying engine code, keeping the codebase lean and maintainable.

Multi-Language Tech Stack

Synapse uses Rust (inference engine, auto-diff, training framework), Zig (SIMD kernels for matrix multiplication, attention, RoPE, RMSNorm—optimized for ARM NEON/AVX2), and Metal (Apple Silicon GPU compute). Cross-language calls via C FFI ensure performance and modularity.

Pluggable Components

Transformer components are configurable: attention (GQA/MHA/MQA/SlidingWindow), normalization (RMSNorm/LayerNorm), feedforward (SwiGLU/GELU/GeGLU), position encoding (RoPE/learned/sine). Quantization supports f32→f16/INT8/Q4_0/Q4_K/Q6_K/Q8_0; weight formats: safetensors/GGUF.

Development Experience

• Build: cargo build --release (Zig kernels auto-recompile via build.rs).
• Examples: cargo run --example qwen3_chat --release -- --model-dir /tmp/qwen3-0.6b (Metal via --features metal).
• WASM build: wasm-pack build -p synapse-wasm --release.
• Supported models: Qwen3, LLaMA3.2, Mistral7B, Phi-3 (in progress), Gemma, ViT/CLIP/DINOv2. Adding new models: JSON config + weight mapper (no engine code changes).

Cross-Platform Performance

On Apple Silicon:

• Qwen3: f32 (11 tok/s prefill,7.3 tok/s decode); INT8 (23 tok/s prefill,27.3 tok/s decode) →3-4x improvement.
• LLaMA3.2: f32 (1 tok/s prefill,2.1 tok/s decode); INT8 (8 tok/s prefill,9.7 tok/s decode). Note: While slower than llama.cpp (Q4_K_M:5518 tok/s prefill,173 tok/s decode), Synapse's multi-platform support makes this competitive.

WASM Advantages

• WASM core: ~519KB (target 160KB), JS wrapper ~43KB (target32KB) → much smaller than Candle's 2-5MB.
• Brotli-compressed:133KB → one of the lightest browser LLM inference solutions, enabling full experience with hundreds of KB downloads.

LEWM (Latent Emergent World Model)

Synapse supports JEPA-style LEWM (ViT encoder + DiT predictor). On Apple Silicon: 224x224 image encoding (26.9ms), single-step prediction (12.8ms),50-step trajectory (609ms). It's the first public work on JEPA quantization: Q4 (9.4MB, cos sim0.93), INT8 (21.4MB, cos sim0.9998).

State Space Models (SSM)

Supports Mamba (130M/370M params, INT8/Q4, WASM-compatible) and RWKV-7 (0.1B/0.4B params, value residual, pre-LayerNorm). Early support for these models makes Synapse ideal for research.

Synapse supports ESP32-P4: it runs a WiFi HTTP server to receive images from phone cameras, performs LEWM inference locally, and returns JSON results. 25 tests passed, waiting for hardware for full video demo. Quantization is critical here: Q4 compression (6.4x) reduces 52.1MB f32 model to9.4MB, enabling complex models on ESP32's limited resources. Ongoing work: structured pruning, mixed Q4/Q8, Hadamard rotation to get LEWM <8MB with cos sim>0.95.

Synapse's roadmap includes:

1. Compress LEWM to <8MB (cos sim>0.95).
2. ESP32-P4 hardware validation and video demo.
3. WASM pre-quantized binary (skip 69MB f32 download, load ~10MB Q4 model directly).
4. NPM package for synapse-wasm.
5. Model surgery: Wanda pruning, channel pruning, layer pruning.

Synapse represents an important evolution in local LLM inference—balancing high performance with cross-platform flexibility (desktop/browser/embedded). Its modular design, multi-language stack, and support for emerging models (LEWM/SSM) make it a powerful tool for production and research. For resource-limited environments, its quantization and ESP32 support open new possibilities; for web devs, lightweight WASM enables browser AI; for researchers, clear code aids learning. As edge AI grows, Synapse's value in data privacy, low latency, and cost efficiency will become more prominent.