Noema: Exploring Latent Space Reasoning of Language Models on Consumer GPUs

The Noema project focuses on exploring the reasoning capabilities of small language models (≤300 million parameters) in continuous latent spaces, aiming to replace the traditional discrete Chain-of-Thought (CoT) token approach to improve sample efficiency, reasoning depth, and speed. The core goal of the project is to verify whether small models can achieve efficient reasoning through continuous latent spaces, with an emphasis on hardware friendliness—all experiments can be reproduced on a single RTX 3060 (8GB VRAM), promoting the democratization of AI research.

Traditional large language models (LLMs) rely on discrete Chain-of-Thought (CoT) to generate text tokens for reasoning. However, Meta's 2024 research on Chain of Continuous Thought (Coconut) shows that models can reason in continuous latent spaces. Inspired by this, Noema's core question is: Can small language models reason in continuous latent spaces instead of discrete tokens?

Discrete CoT has limitations such as high reasoning latency, high computational cost, and suboptimal text representation. Latent space reasoning encodes semantics through continuous vectors, which can capture more fine-grained conceptual relationships. Noema focuses on consumer-grade hardware, with the philosophy that 'cutting-edge mechanisms often start at toy scales', allowing more researchers to participate in exploration without expensive resources.

Noema plans five phases: Phase 0 establishes a nanoGPT-style baseline model verification process; Phase 1 introduces the core innovation of 'continuous thought heads', allowing the output of latent vectors and feedback; Phase 2 uses curriculum learning to train on math/logic puzzles; Phase 3 compares the performance of latent space CoT, discrete CoT, and no CoT; Phase 4 will open-source the paper and invite collaboration if successful.

The project requires all experiments to be reproducible on a single RTX 3060 (8GB VRAM). The minimum configuration is RTX3060 8GB, 16GB RAM, and 50GB disk; the recommended configuration is 12GB+ VRAM, 32GB RAM, and 200GB disk. CPU-only training is only theoretically feasible for models ≤10 million parameters and is not recommended. This design promotes the democratization of AI research.

If successful, it will update the understanding of small model capabilities (parameter count is not the only determining factor); spawn efficient reasoning models that can run on edge devices, suitable for mobile AI and IoT; and its methodology of 'verifying new architectures on consumer-grade hardware' provides a reference for the community.

Noema represents a healthy trend in AI research: returning to the essence of experiments, emphasizing reproducibility, and embracing hardware constraints. The project does not chase parameter competitions but explores the essence of representation learning, providing an open-source focus for researchers in efficient AI and reasoning mechanisms. The community can track progress and participate.