Peek: An Interactive Visualization Tool to 'See' the Inner Workings of Large Language Models

Peek is an open-source project that provides a Transformer model with only 825,000 parameters, trained on Shakespeare's texts. By making every weight clearly visible, it helps developers intuitively understand the working principles of Large Language Models (LLMs), aiming to solve the "black box" problem of LLMs and provide an intuitive entry point for learning and research.

As the scale of LLMs expands (from GPT-2's 1.5 billion parameters to GPT-4's rumored over 1 trillion parameters), the internal computation process of models is hidden in massive parameters and difficult to understand. The core concept of Peek is interpretability—by building a small-scale but fully functional Transformer model, it allows learners to observe the process of text processing, attention calculation, and word generation layer by layer and weight by weight, which has extremely high educational value.

Model Specifications

• Parameter count: Approximately 825,000
• Training data: Complete works of Shakespeare
• Architecture: Standard Transformer decoder
• Visualization granularity: Every weight and activation value is visible

Reasons for Choosing Shakespeare's Texts

• Unique and consistent language style, conducive to learning patterns
• Moderate text volume, suitable for small model training
• Widely known works, easy to judge the compliance of generated text

Peek provides a fully interactive visualization interface, supporting:

1. Embedding layer visualization: View word vector representations, observe the spatial positions of semantically similar words via t-SNE/PCA
2. Attention heatmap: Real-time display of other words the model focuses on when processing each word
3. Feedforward network activation: Observe information flow in hidden layers
4. Word-by-word generation process: Slow-motion replay of probability calculation and sampling process for word selection

• Beginners: Lower the learning threshold of Transformers through intuitive observation
• Advanced developers: Verify understanding of Transformers and discover knowledge gaps
• Researchers: Provide a controllable experimental platform to test hypotheses and the impact of component modifications

It uses the Next.js front-end framework, Geist font optimization, and Vercel deployment. It supports running on any modern browser without the need to install software or configure complex environments.

Limitations

• Scale limitation: The 825,000-parameter model does not have the emergent capabilities of large LLMs
• Single data source: Trained only on Shakespeare's texts, with limited knowledge and style

Future Directions

1. Multi-model comparison: Compare models of different scales/architectures
2. Custom training: Allow uploading small datasets for training
3. Interactive editing: Manually modify weights to observe output impacts
4. More visualizations: Integrate tools like neuron activation pattern analysis

Through exquisite design and intuitive visualization, Peek demystifies LLMs and provides an excellent entry point for understanding AI technology. It is worth trying for students, developers, and researchers alike.

Project Address: https://github.com/shawn14/peek