SAE Interpretability Intervention: A Groundbreaking Study That Boosts Small-Model Browser Agent Performance by 7.5x

The Stanford University CS153 course project demonstrates how sparse autoencoder (SAE) feature intervention technology increases the success rate of the Llama-3.1-8B browser agent from 10% to 75%, narrowing the 72% performance gap with the 70B large model at approximately 1/8 the inference cost. This project is maintained by kalyvask, released on May 24, 2026, and its GitHub repository is named inside-the-agent (link: https://github.com/kalyvask/inside-the-agent).

Current large language model agent evaluation faces a black-box problem: we only know about failures but cannot explain the reasons or fix them. Traditional improvement paths (prompt engineering, evaluation-driven retraining, purchasing larger models) have limitations: prompts are constrained by the model's understanding ability, retraining is costly and time-consuming, and large models have significant inference costs and latency. Sparse autoencoders (SAE) provide a fourth path: directly intervene in model behavior at the representation level to fix specific failure modes without retraining.

Baseline Result Comparison

Strategy	Success Rate	95% Confidence Interval	Improvement Over Baseline
Baseline (no intervention)	10.0%	[4.7%,20.1%]	—
SAE feature intervention only	56.7%	[44.1%,68.4%]	+47 percentage points
System prompt optimization only	73.3%	[61.0%,82.9%]	+63 percentage points
SAE + prompt combination	75.0%	[62.8%,84.2%]	**+65 percentage points
Llama-3.3-70B baseline	100%	[94.0%,100%]	Cross-model reference

Key Insight: Two SAE feature edits (f26737 reduced by 6, f23803 increased by 6) plus one line of system prompt increased the success rate of the 8B model from 10% to 75%, narrowing the 72% gap with the 70B model at 1/8 the cost.

Fine-Grained Task Performance: SAE intervention performed best in promotion trap tasks (79% vs baseline's 0%), prompt optimization was more effective in planning tasks (67% vs baseline's 33%), and the combined strategy covered more failure modes.

Strict vs Loose Standards: The combined strategy achieved a loose success rate of 75%, but only 8.3% under strict standards (requiring only target items in the cart), which is still behind the 70B model's 90% strict success rate.

What is SAE: A sparse autoencoder is a neural network that learns efficient sparse representations, decomposing LLM activation vectors into a small number of interpretable features (e.g., UI selection vocabulary, search patterns, domain knowledge).

Intervention Mechanism: 1. Feature identification (analyze failure cases to find erroneous features); 2. Direction determination (enhance/inhibit features—e.g., f26737 corresponds to overactivated UI selection vocabulary and needs to be inhibited); 3. Real-time intervention (modify residual stream representations at specific inference steps). SAE intervention directly adjusts the model's internal activations, a level unreachable by prompt engineering.

Practical Significance: 1. Cost-effectiveness: The inference cost of 8B + SAE intervention is approximately 1/8 that of the 70B model, suitable for high-concurrency/cost-sensitive scenarios; 2. Interpretable error repair: Dynamic adjustment at runtime without modifying model weights; 3. Mechanism-driven evaluation: SAE features provide richer diagnostic information.

Limitations: 1. Execution rate issue: The intervention group had a 100% effective action rate but only a 36.3% actual execution rate; 2. Category specificity: Different tasks require different intervention strategies; 3. Strict standard gap: Small models + intervention still lag behind large models in precise execution scenarios.

Future Directions: Train models specifically designed for interpretability intervention, including training dedicated SAEs on agent task residuals and incorporating interpretability goals into joint training of foundation models.

Conclusion: The Inside the Agent project demonstrates the potential of interpretability research to shift from analytical tools to intervention layers. Through SAE intervention, small models' performance on specific tasks can be improved without increasing model size, representing a new optimization paradigm: instead of training larger models, we should deeply understand and manipulate the internal representations of existing models.