Reverse Thinking in Small-Parameter Reasoning Models: From 'Large Model Distillation' to 'Native Small Model Design'

An open-source project named small-reasoning-model proposes a reverse approach: instead of quantizing and compressing large models, it designs native small-parameter reasoning models (within 1B parameters) from scratch to explore the possibility of efficient reasoning. The core insight comes from DeepSeek R1's experience—reasoning ability stems from training recipes rather than architecture. The goal is to outperform quantized large models with double the parameter count on math/code reasoning tasks while reducing inference costs.

The current mainstream in AI focuses on large model parameters, but traditional compression paths (quantization/pruning/distillation) lose performance because the architecture is designed for large capacity. The project adopts the "Small first" principle: designing from the first line of code with the target parameter scale in mind, prioritizing inference efficiency. For example, all dimensions are multiples of 128 to fit the systolic array of AWS Trainium2 chips, avoiding zero-padding waste; the architecture uses 2024-2025 consensus configurations (Pre-norm RMSNorm, GQA, QK-Norm, etc.) with no experimental designs.

Take the 1B parameter configuration (Config B) as an example: d_model=2048, Layers=20, Q heads=16/KV heads=4 (GQA reduces KV cache), FFN dim=5504, Max seq=16384 (supports long chain-of-thought). Key designs: QK-Norm solves the numerical explosion problem of attention logits in small models; head dimension=128 aligns with the GGUF quantization block layout of llama.cpp for efficient quantization.

Three-stage training process:

1. Pre-training: Standard next-token prediction. The 1B model plans to use 50 billion tokens (50x Chinchilla's optimal value, intentionally over-trained);
2. SFT: Loss is calculated only on assistant responses to avoid overfitting to formats;
3. GRPO reinforcement learning: 8 results sampled per group, binary reward + group mean baseline, no value model needed. Integrated DAPO improvements: Clip-higher (prevents entropy collapse), token-level policy gradient (does not penalize long correct chains), dynamic sampling (avoids waste), length-debiased advantage (prevents short but incorrect responses).

Tokenizer details: BPE vocabulary size of 32768 (128×256), byte-level fallback, separate digit tokenization ("142" split into ["1","4","2"]), <think>/</think> set as the 4th/5th tokens (reinforce chain-of-thought mode). Deployment supports GGUF quantization: BF16 (2GB), Q8_0 (1GB), Q4_K_M (700MB recommended), Q4_0 (550MB, runnable on Raspberry Pi 5). Cost estimate: Q4_K_M model on Graviton4 achieves 25-35 tokens/second, $0.68 per hour, cost per 1000 tokens is less than 1 cent.

Open question: Can native small models outperform quantized 1.7B models on math/code tasks with lower cost? If yes, it will change the deployment paradigm and bring reasoning capabilities to edge devices. Challenges: Pre-training not started, need for high-quality validation datasets, limited generalization (focused only on specific tasks).

This project represents the "small but specialized" path, challenging the parameter arms race and pursuing extreme efficiency. The current architecture is complete and awaits pre-training. Regardless of the outcome, the reverse thinking is worth attention—when the mainstream moves right, moving left may discover new lands and promote AI inclusiveness.