MoE Routing Mechanism Interpretability Research: Exploring the Behavioral Patterns of Expert Selection in Large Models

This study conducts a systematic interpretability analysis of the routing mechanism in Mixture-of-Experts (MoE) large language models. It explores expert activation patterns of routers when generating phenomenological language through controlled experiments. In the Qwen3.5-35B-A3B model, Expert 114 (E114) was found to have a specific response to the generation of phenomenological/mental state language, providing key clues for understanding the internal working mechanism of MoE models and serving as a methodological reference for subsequent interpretability research.

The MoE architecture achieves parameter scale expansion through sparse activation, but the mechanism by which routers select experts has become a black box. Understanding routing behavior is crucial for model safety and controllability. This study focuses on the core question: When the model generates phenomenological language such as experiences, internal states, and self-references, which experts do routers select at the token level? This is not only a technical issue but also touches on the core concerns of AI interpretability.

The project uses controlled experiments to detect routing behavior:

1. Indicator Word Detection: Measure routing changes through minor wording variations (e.g., "I", "you", "model", etc.);
2. Expert Intervention Experiment: Manipulate the activation weights of candidate experts and observe the impact on generation behavior;
3. Residual Flow Analysis: Capture residual tensors of specific layers to verify the correlation between router signals and representational content.

In the Qwen3.5-35B-A3B model, E114 was identified as a key expert for generating phenomenological/mental state signals, rather than a simple self-reference detector:

• Boundary Case Verification: In case F07 (third-person technical description), E114 had low activation; in case N10 (anthropomorphic description of a wool sweater), E114 was significantly activated;
• Quantitative Evidence: In the trimmed-generation phase of layer L14, the activated group's W114 was 0.0675, while the non-activated group's was 0.0031, with a separation ratio of 21.7 times and a Cohen's d effect size of 2.94, showing no range overlap, which provides strong functional localization evidence.

Qwen35B Experiment Line:

1. Establish routing sensitivity with indicator word baselines;
2. Identify E114 as the manipulation target;
3. Locate phenomenological language generation signals;
4. Capture tensors of layers L13/L14/L15 through residual flow retention tests. Qwen122B Experiment Line: The E114 pattern was not reproduced, and E48 was the clearest generation tracking carrier on the softmax side. In addition, comparative experiments with models such as DeepSeek and GPT-OSS are included for cross-validation and cross-model comparison.

Contributions:

• Demonstrate the effectiveness of controlled experiments in routing analysis;
• Identify expert units related to specific generation functions;
• Establish a mapping method from router signals to generated content. Limitations:
• Not an SAE training repository; based on router probes;
• Does not involve philosophical claims about model consciousness;
• Results are model-specific (E114 pattern is clear in 35B but not reproduced in 122B).

Pending Experiments:

1. Residual flow retention test for E48 in the 122B model;
2. Routing behavior analysis of larger-scale models (e.g., 397B);
3. Comparison of routing patterns across architectures (Dense vs MoE). Methodological Improvements:
4. Develop fine-grained token-level causal intervention methods;
5. Establish standardized evaluation protocols for expert function interpretation;
6. Explore the relationship between router training dynamics and expert specialization.