Cadence: An Experimental LLM Inference Framework for Apple Silicon Based on MPSGraph

Cadence is an experimental project by Ostinato Labs, built with Swift, SwiftUI, and Metal Performance Shaders Graph (MPSGraph) for macOS native LLM inference. It focuses on verifying core Transformer operators on GPU, currently in early R&D phase (not a ready chat app). It serves as an operator testbed, CPU-GPU validation tool, and prototype for future local inference engines.

With With Apple Silicon's growing power, leveraging Metal for LLM inference is a key focus for developers. Cadence is an early-stage R&D project, not a production-ready app. Its roles include:

1. Metal/MPSGraph operator experimentment field;
2. CPU-GPU output comparison validation;
3. Skeleton prototype for future local inference engines. It is not a complete chat app, Qwen runtime, or mature tested project.

Cadence uses Apple native tech stack: Swift5 (language), SwiftUI (UI), MPSGraph (GPU acceleration). Key components:

• Device management (MTLDevice, command queue, MPSGraphDevice) in Device.swift;
• Tensor utils (data conversion) in TensorUtils.swift; Implemented Transformer operators:
• Attention: single-head, multi-head (with causal mask), GQA;
• RoPE: precompute cos/sin tables and apply;
• Normalization: RMSNorm (with debug values), LayerNorm;
• Activation: SWiGLU; Tokenizer: ByteShadowMap (byte-level reversible encoding, foundation for BPE).

Cadence uses manual test runners (not XCTest) compiled into the app, invoked via CadenceApp.init(). Current tests:

• MatmulTest: CPU vs GPU consistency;
• RMSNormTest, RoPETest (numeric/property), LayerNormTest, SWiGLUTest;
• AttentionTest (single/multi/GQA), AttentionPerfTest (CPU-GPU performance);
• ByteShadowMapTest (round-trip encoding). This approach offers flexibility in early R&D.

Model assets: Qwen3.5-4B files (tokenizer config, vocab, merges, partial safetensors) exist but are not loaded/used (missing first safetensors shard). Unimplemented features: default Hello World UI, no safetensors reader, no tokenizer parsing, no end-to-end Transformer block, no logits sampling/text generation, tests not in XCTest.

Next steps for Cadence:

1. Add safetensors weight loading;
2. Parse tokenizer vocab and BPE rules;
3. Combine operators into full Transformer blocks;
4. Add embedding layer, LM head, KV cache;
5. Build end-to-end pipeline (prompt → tokens → logits → sampling → text);
6. Migrate tests to XCTest and add benchmarks.

Cadence's value:

• Proves Swift+MPSGraph can implement Transformer core operators for Apple platform end-to-end AI;
• Lightweight CPU-GPU validation method for operator correctness;
• Open-source resource for learning Metal/MPSGraph and LLM inference on Apple Silicon;
• Clear code structure and complete operator implementations make it a great learning material for developers interested in Apple native LLM inference.