Raw Weights: Understanding the Training and Inference of Large Language Models from the Perspective of a Robot Student

Raw Weights is an open-source project that helps learners intuitively understand the training and inference mechanisms of large language models through the metaphor of "a robot student learning to write" and interactive visualizations. The project's core philosophy is "No hype, just architecture"—it rejects AI hype and returns to the essence of technology, making it suitable for technical personnel, product managers, and others who want to deeply understand the underlying mechanisms of AI.

Raw Weights was created by developer Schikkeg, hosted on GitHub, and deployed at rawweights.com. The project is positioned to strip away the excessive marketing and mystery of AI, analyze the underlying components of the AI revolution from the perspective of scalable system design, focus on explaining core concepts thoroughly, and does not aim to cover all cutting-edge papers. It is suitable for technical personnel who already know about AI news and want to break through the "black box".

The project's core teaching methodology compares AI models to "robot students learning to write". The reasons this metaphor is effective include: 1. It concretizes abstract concepts (weight matrix → brain, loss function → score); 2. It emphasizes that training is a gradual process rather than magic; 3. It demonstrates the value of models learning from mistakes, building intuition first before introducing technical details.

The tutorial unpacks five core concepts:

1. Future-blind reading: The model is like a "future-blind reading" student, predicting the next letter based only on the current one, embodying autoregressive generation (the core of GPT-like models);
2. Voting box: Logits are compared to a voting box—after scoring letters, they are converted into a probability distribution, explaining the diversity of model outputs;
3. Teacher's score: The loss function is like a teacher's score—supervised learning adjusts parameters by minimizing the gap between predictions and reality;
4. Intelligent climber: The Adam optimizer is like an intelligent climber, optimizing parameters by combining momentum and adaptive learning rates;
5. Formal performance: In the inference phase, autonomous generation occurs—each token serves as input for the next step, and the recursive mechanism creates new content.

In terms of technical implementation, the project uses a web-based visualization interface that supports users to manipulate parameters and get real-time feedback; it uses a simplified character-level language model for demonstration; the playground design is similar to nanoGPT, simplifying complex systems to their core mechanisms to allow learners to experiment hands-on.

Learning value: Suitable for software engineers transitioning to AI (those with programming foundations who need to understand the training process), product managers (to understand the boundaries of AI capabilities), and AI beginners (to build intuition). Note: This project is a tool for building intuition; it is necessary to supplement basic knowledge such as linear algebra, probability theory, and deep learning frameworks.

Future outlook: The author plans to update content such as Transformer architecture unpacking, attention mechanism visualization, and agent workflows; the community can participate in the project's development by contributing code or front-end components via GitHub.