Replicating ZAYA1-8B's Reasoning Capabilities on Consumer GPUs: A Technical Breakthrough with the 340M-Parameter MoE Model

The open-source project nano-zaya340M successfully compresses the core innovative technologies of Zyphra ZAYA1-8B into a 340M-parameter MoE model, which runs on only 8-10GB of VRAM. Using the CCA attention mechanism, MLP router, and Markovian RSA reasoning algorithm, it enables small models to achieve deep thinking capabilities, lowering the hardware barrier for powerful reasoning models.

In recent years, large language models (e.g., DeepSeek-R1, Gemini-2.5 Pro) have strong reasoning capabilities but require hundreds of gigabytes of VRAM, making them inaccessible to ordinary developers. Zyphra ZAYA1-8B, with 700 million active parameters (8 billion total), outperforms DeepSeek-R1-0528, but its hardware requirements are still high. The nano-zaya340M project aims to solve this problem by replicating its core technologies on consumer GPUs.

Compressed Convolutional Attention (CCA)

Traditional self-attention is computationally complex. CCA reduces computational and memory requirements by performing sequence mixing in a compressed latent space while retaining the ability to model long-range dependencies.

MLP Router and MoE Architecture

Using an MLP network as a router instead of traditional linear layers, it learns more complex expert selection strategies. Combined with the MoE++ architecture, the 340M active parameters achieve reasoning capabilities superior to those of equivalent dense models.

Markovian RSA Reasoning Algorithm

RSA improves performance by recursively combining reasoning chains. Markovian RSA retains the tail of a finite-length reasoning chain, enabling multi-round deep reasoning within a limited context window, helping ZAYA1-8B achieve top-level performance on math competition benchmarks.

1. Logical Warm-up: Train on logical reasoning questions and puzzles to build basic reasoning capabilities.
2. RLVE-Gym Curriculum: Train using a curriculum of 400 questions, covering various reasoning patterns.
3. Math and Code Training: Train using computation traces and synthetic programming environments, including computation traces during testing, to enable the model to learn to think rather than memorize.
4. Behavioral Reinforcement Learning: Focus on dialogue style and instruction following to ensure friendly expression of the thinking process.

• Low hardware requirements: Runs on a single RTX3070/4060 or laptop.
• Scenario applicability: Suitable for education and research scenarios, with significantly reduced deployment costs.
• Open-source contribution: Fully open-sourced training code, configuration files, and translated technical reports provide a foundation for community research.

Limitations: The 340M-parameter model has a gap compared to ZAYA1-8B; it is a small-scale replication to verify the feasibility of core technologies.

Future directions:

• Expand model scale to adapt to larger VRAM.
• Explore applications in more task domains.
• Optimize the reasoning efficiency of Markovian RSA.
• Compare capabilities with other open-source models.

nano-zaya340M demonstrates the trend of improving model capabilities through algorithmic innovation rather than parameter stacking, which is of practical significance in the context of computing power constraints. It provides developers with an experimental platform to understand MoE architectures and test-time reasoning algorithms, proving that consumer hardware can participate in cutting-edge AI research.

Project address: https://github.com/korziner/nano-zaya340M-cca-markov-moe