Hybrid Signal Lab: A Tool for Geometric Exploration of Attention Layers in Hybrid Architecture Large Language Models

Hybrid Signal Lab is a research tool for exploring the geometric properties of attention layers in hybrid architecture large language models (LLMs). Its core mechanism dynamically adjusts the contribution ratio between Gated DeltaNet (GDN) layers and attention layers during inference, enabling fine-grained control of model behavior. The tool supports hybrid architecture models such as Qwen3.5 and OLMo-Hybrid, allowing exploration of the model behavior space without retraining, and provides an experimental framework for understanding the internal mechanisms of hybrid architectures.

Project Background: Hybrid Signal Lab is a research project from the ASU CAS Capstone course, supervised by Professor Bryan Daniels, aiming to explore the internal working mechanisms of hybrid architecture LLMs, especially the dynamic relationship between attention layers and recurrent layers. Core Concepts of Hybrid Architecture: Hybrid architecture LLMs alternately stack attention layers (good at long-range dependencies, high complexity) and GDN layers (efficient recurrent structure, linear complexity). The target models Qwen3.5 and OLMo-Hybrid use a 3:1 interleaved stacking ratio to balance efficiency and performance.

The core of the technical principle is the inference-time intervention mechanism: by inserting hooks into the model's forward propagation, dynamically adjust the residual contribution ratio of attention layers. Role of adjustment parameter g: When g→0, GDN layers dominate; when g→1, attention layers dominate; when 0<g<1, explore synergistic effects. This mechanism does not require retraining the model, can explore the complete response surface, and reduces experimental costs.

The tool includes two core components:

1. Signal Lab: A single forward propagation diagnostic tool that reports metrics such as top-k logits, entropy, and attention statistics. Usage example: uv run python -m signal_lab.signal_lab --prompt "The color with the shortest wavelength is" --g-function constant --g 1.0
2. Sweep Tool: Automates experiments with combinations of multiple prompts and g configurations, collects metrics, and organizes outputs. Usage example: uv run python -m signal_lab.sweep --cartridge uniform_check_lite (For detailed parameters, refer to the original documentation.)

The experimental design uses a short prompt test set covering dimensions such as factual knowledge (e.g., the capital of Mongolia), mathematical reasoning (Fibonacci sequence), and code generation. Key metrics include target rank, target probability, final entropy, and KL divergence. Output file structure: main.jsonl (main results), _meta.json (metadata), verbose.jsonl (detailed logs).

Environment configuration requires Python ≥3.13, a Hugging Face account (to access Qwen models), and CUDA/MPS/CPU support. Installation steps:

1. Install dependencies via uv sync
2. Create a .env.development file to set HF_TOKEN
3. Verify: uv run python -m signal_lab.signal_lab --help

Research Significance: Provides an experimental framework for hybrid architecture models, enabling quantification of architectural trade-offs, analysis of dynamic behavior, and optimization of intervention strategies. Potential Applications: Model compression, inference optimization, and interpretability research. Future Directions: Signal Lab is the first step toward the "Colony" vision, which will implement a collective signal layer that automatically generates/adaptively adjusts intervention strategies.