ExTrm: A Journey of Exploring Experimental Miniature Reasoning Models Built with Elixir

ExTrm is an experimental miniature reasoning model project entirely written in Elixir, aiming to provide a lightweight experimental platform for research-oriented AI development. It explores two core directions: grid reasoning (to solve ARC-style tasks) and text/code reasoning (supporting long contexts and code generation), using Nx as the tensor computation backend. The project is positioned as a research codebase rather than a production-grade framework, focusing on rapid iteration to validate architectural ideas.

In the AI field, most reasoning models are built based on the Python ecosystem, but Elixir has become a new choice due to its concurrency model and functional programming features. ExTrm is not a mature production framework but a research codebase, aiming to provide an experimental platform for rapid iteration, trial-and-error, and validation of new architectures. Its code style is intentionally kept rough to prioritize idea validation, making it suitable for developers willing to experiment hands-on.

Grid reasoning was the initial core of the project, focusing on solving ARC (Abstraction and Reasoning Corpus) style tasks—pattern recognition and reasoning on colored grids. Key architectural elements include: recursive block structure (supporting multi-step reasoning), repeated thinking steps (simulating human reasoning), and colored grid representation. The default parameter count is about 100 million; running on CPU requires passing smaller configuration parameters.

The text/code reasoning direction is more practical, with components including: byte-level tokenizer (no pre-trained vocabulary needed), Hugging Face dataset integration, long-context text model foundation, and a complete training/saving/inference pipeline. The project uses the karti06k/Qwen-59k-Python-Instruct dataset for training, which contains instructions, reasoning processes, and code, suitable for code generation models.

Long Context Support: The text model supports a context length of 128K at the architectural level, with configuration parameters such as context_length:131072 and attention_window:2048. It uses techniques like chunked pre-filling, sliding attention, RoPE-style positional encoding, and memory tokens, but its actual capabilities need to be verified through training. Nx Backend: Uses Nx (Numerical Elixir) as the tensor computation backend, providing a NumPy-like API while maintaining a functional style, making it easy for Elixir developers to conduct deep learning experiments.

Code Structure: Modules are clearly divided. The core of the grid model is in lib/ex_trm/model.ex, the core of the text model is in lib/ex_trm/text/model.ex, and there are also modules for dataset loading, training, inference, and command-line tools. Quick Start:

1. Environment preparation: mix deps.get → mix test
2. Small-scale grid experiment: Need to pass small configuration parameters (e.g., vocab_size:8, d_model:16, etc.)
3. Text model operations: Download dataset (mix ex_trm.text.dataset.download), small-scale training (mix ex_trm.text.train --rows64 --steps10), inference test (mix ex_trm.text.generate --prompt ...)

ExTrm is suitable for: AI enthusiasts in the Elixir ecosystem, model architecture researchers (for rapid idea validation), educational purposes (learning reasoning model mechanisms), and ARC task researchers. It is not suitable for developers expecting an out-of-the-box production framework. It is recommended to treat it as a laboratory workbench to freely experiment with architectural ideas without pursuing a perfect API.

ExTrm proves that AI development does not have to be limited to the Python ecosystem. Elixir's concurrency features and fault-tolerant design are theoretically suitable for distributed training. Although it is in the early experimental stage, it provides a valuable starting point for the Elixir community. As the author said: "Some ideas may stay, some may get deleted, some may become separate models. That's fine." This open experimental mindset drives the development of AI frontiers.