In-depth Analysis of GPT-2's "Syntax Circuit": Comprehensive Application of Three Mechanistic Interpretability Techniques

This article focuses on the GPT-2 Small model, comprehensively applying three core techniques—linear probing, causal activation patching, and sparse autoencoders—to systematically explore how it encodes and utilizes part-of-speech information. It aims to uncover the internal "black box" working mechanism of large language models (LLMs) and provide a practical path for the development of explainable AI.

Part-of-speech is a fundamental linguistic concept that identifies the grammatical role of words (e.g., nouns, verbs). It is crucial for humans to understand sentence structure and for LLMs' grammatical processing and word prediction. Although GPT-2 Small is relatively small (about 117 million parameters), it already demonstrates strong language capabilities. Understanding how it handles part-of-speech information helps promote research on large models and lays the foundation for explainable and controllable AI systems.

Linear probing verifies whether the model's internal representations contain target information by training a simple classifier. This study extracts residual stream activations from each layer of GPT-2, trains linear/MLP detectors, and uses the CoNLL-2003 part-of-speech dataset. The results show that part-of-speech information is distributed across multiple layers, with the middle layers having the best linear separability; nonlinear detectors only provide marginal improvements, indicating that most part-of-speech information is encoded linearly, which is beneficial for knowledge extraction and model compression.

This technique addresses the question of "which components have causal responsibility": it compares the model's behavior between "clean" and "damaged" inputs, patches the activations of specific attention heads, and observes the recovery of outputs. The evaluation metric is the logit difference of key tokens, and a layer-head heatmap is generated. The results show that a small number of attention heads have a significant impact on verb selection, and their contributions are distributed, which aligns with the design concept of Transformer's multi-head attention.

SAE decomposes high-dimensional activations into sparse and interpretable features, with an architecture including an encoder, decoder, and sparsity penalty term. This study uses SAE to analyze the feature activations of verb tokens and finds that the residual stream can be decomposed into syntax-related sparse features, some of which are consistently activated for verbs, providing a window into understanding the internal concept organization of the model.

Linear probing answers "whether the information exists", causal patching locates "responsible components", and SAE parses "representation composition". The three complement each other to form a complete analysis path: from representation analysis to causal explanation and then to feature decomposition, which can be extended to other linguistic information such as syntactic dependencies and semantic roles, as well as non-linguistic tasks.

The insights of this study for the AI field include: model safety alignment (precise guidance of behavior), model editing and repair (targeted correction of errors), efficient fine-tuning (linear transformation to adapt to new tasks), and large-scale model interpretability (methodology expansion).

Mechanistic interpretability is shifting from academic curiosity to a practical tool. This study demonstrates a systematic method for dissecting the syntax circuit of LLMs. Although there is still a long way to go to fully understand large models, this project provides complete code and an integrated process, offering an excellent starting point for researchers to get into mechanistic interpretability.