CircuitLasso: Scalable LLM Circuit Learning via Sparse Linear Regression

CircuitLasso is a scalable circuit learning method based on sparse linear regression, designed to address core challenges in the mechanistic interpretability of large language models (LLMs). It transforms the circuit learning problem into a sparse linear regression task, significantly reducing computational costs while recovering circuits with structural accuracy comparable to state-of-the-art intervention methods, and revealing the propagation paths of semantic features within models. This method provides a feasible solution for handling the high-dimensional feature spaces generated by sparse autoencoders (SAEs), advancing the understanding of the internal working mechanisms of LLMs.

The "black box" nature of LLMs hinders understanding of their internal working mechanisms, posing safety and controllability risks. The field of mechanistic interpretability reveals model behavior by learning sparse circuits (collaborative combinations of key neurons/features), but traditional methods face two major challenges:

1. Multi-semantic neuron problem: Original neurons often correspond to multiple concepts; while SAEs decompose them into single-semantic features, this leads to dimensional explosion of the feature space;
2. Excessive computational cost: Intervention-based methods require a large number of experiments, and costs grow exponentially with the number of components, making it difficult to handle the high-dimensional spaces of SAEs.

The core innovation of CircuitLasso is reframing circuit learning as a sparse linear regression problem. Its advantages include:

• Utilizing mature sparse regression algorithms without the need for explicit intervention experiments;
• Controlling circuit sparsity via regularization parameters to balance interpretability and coverage;
• Possibly adopting LASSO or its variants, using L1 regularization to encourage selection of a compact subset of features.

Experimental results show the advantages of CircuitLasso:

• Structural accuracy: Comparable to state-of-the-art intervention methods, reliably identifying important model components;
• Computational efficiency: Significantly reduces costs, supporting large-scale models and complex tasks;
• Scalability: The solution can be highly parallelized, adapting to modern hardware;
• Propagation path revelation: Tracks the transfer of semantic features between model layers (e.g., shallow layers identify lexical features, middle layers combine phrases, deep layers focus on global semantics);
• Domain generalization: The learned circuits capture the core mechanisms of tasks and maintain good performance in new domains.

The value of CircuitLasso for AI safety includes:

• Failure mode diagnosis: Locating the root cause of unexpected behaviors;
• Adversarial robustness analysis: Assisting in designing attack and defense strategies;
• Model editing and correction: Correcting behaviors by editing circuits without retraining;
• Value alignment verification: Verifying whether the model internalizes human values rather than just imitating them superficially.

CircuitLasso still faces challenges:

• Trade-off between completeness and sparsity: Need to balance circuit sparsity and information completeness;
• Dynamic behavior capture: Static analysis struggles to capture context-dependent dynamic changes;
• Cross-model transfer: The generalization of circuits across models of different architectures/scales needs further research;
• Causal relationship confirmation: Sparse regression identifies statistical correlations, requiring combination with intervention experiments to establish causality.

CircuitLasso, through its sparse linear regression framework, improves computational efficiency while maintaining accuracy, making circuit learning in the high-dimensional feature spaces of SAEs possible. As LLM capabilities advance, such tools will help make AI systems more transparent, controllable, and trustworthy, providing key support for mechanistic interpretability research.