Oryx: A New Hybrid Model Architecture with Dynamic Attention Mechanism Switching in Sequences

Researchers propose the Oryx architecture, which breaks through the static alternation design paradigm of traditional hybrid models, enabling sequence-level dynamic mixer switching with over 90% parameter sharing. At the 1.4B scale, it outperforms single-mixer baselines, providing new ideas for long-sequence modeling. The original authors are the Oryx research team, and the source is arXiv (published on 2026-05-27, link: http://arxiv.org/abs/2605.28769v1).

The Softmax attention mechanism is the cornerstone of large models, but its computational complexity grows quadratically, leading to high costs for long-sequence processing. Linear recurrent models (e.g., Mamba) are efficient but lag behind Transformers in long-context retrieval/learning tasks. Existing hybrid architectures are mostly static designs (inter-layer alternation or fixed ratios), assuming all tokens have the same needs, which is inconsistent with real-world scenarios.

Oryx dynamically switches mixers (e.g., attention/linear recurrent mechanisms) at the sequence dimension. Its core innovation is over 90% parameter sharing—different mixers operate on the same internal representations instead of independent spaces, which not only reduces the total number of parameters but also allows selecting the optimal mechanism based on token needs.

At the 1.4B scale, Oryx instances outperform single-mixer baselines on average language modeling tasks (improvement ≥0.7 percentage points). In retrieval tasks, using attention mode for less than 10% of tokens is sufficient to achieve Transformer baseline performance, enabling context understanding with low overhead.

Oryx reveals that attention and linear recurrent models can share representations, breaking traditional perceptions. Sequence-level mixing allocates resources more finely than static inter-layer mixing, reducing costs while maintaining performance. It provides a path for large model practitioners: reducing inference costs without sacrificing long-context capabilities, suitable for scenarios like long document processing and code generation.

Dynamic switching introduces routing decision overhead (needs actual deployment evaluation); 90% parameter sharing may limit expressive power for specific tasks; the optimal ratio and scheduling of hybrid training strategies still need to be explored.