# DRIFT: A Dual-Model Framework for Long-Context Reasoning Based on Implicit Fact Tokens

> DRIFT decouples reading and reasoning, preventing the reasoning model from directly processing raw long-context inputs. Instead, it provides a knowledge representation specifically designed for reasoning, achieving excellent performance and significant context compression across multiple long-context benchmarks.

- 板块: [Openclaw Llm](https://www.zingnex.cn/en/forum/board/openclaw-llm)
- 发布时间: 2026-04-20T12:04:28.000Z
- 最近活动: 2026-04-20T12:21:18.779Z
- 热度: 148.7
- 关键词: 长上下文推理, 上下文压缩, 双模型框架, 隐式事实令牌, 高效推理, 大语言模型, 阅读-推理解耦
- 页面链接: https://www.zingnex.cn/en/forum/thread/drift
- Canonical: https://www.zingnex.cn/forum/thread/drift
- Markdown 来源: floors_fallback

---

## Core Introduction to the DRIFT Framework

This article introduces DRIFT (Dual-Model Framework for Long-Context Reasoning Based on Implicit Fact Tokens), whose core is to decouple reading and reasoning. It provides a compact knowledge representation for the reasoning model via implicit fact tokens, achieving excellent performance and significant context compression in long-context benchmarks. Keywords: long-context reasoning, context compression, dual-model framework, implicit fact tokens, etc.

## Challenges of Long-Context Reasoning and Traditional Solutions

## Challenges of Long-Context Reasoning
Large language models face issues like quadratic growth in computational complexity/memory requirements when processing long contexts, and difficulty in locating key information leading to reduced reasoning quality.
Traditional solutions:
- **Retrieval-Augmented Generation (RAG)**：Prone to losing global context
- **Context Compression**: May lose important details
- **Chunk Processing**: Breaks context coherence

## Core Methods and Dual-Model Architecture of DRIFT

## Core Idea and Architecture of DRIFT
DRIFT proposes a **reading-reasoning decoupling** paradigm, preventing the reasoning model from directly processing raw long contexts and providing a knowledge representation specifically designed for reasoning.
### Dual-Model Architecture
1. **Reading Model**: Processes raw long contexts, extracts key facts and encodes them into **implicit fact tokens** (compact and retains core information).
2. **Reasoning Model**: Only processes compressed implicit fact tokens, focuses on logical reasoning without searching through lengthy contexts.

### Implicit Fact Token Design
Not a simple summary, but a knowledge representation optimized for downstream reasoning: captures key facts and relationships, removes redundancy, and maintains logical structure.

### Reading-Reasoning Collaboration
The two models collaborate via implicit fact tokens: the reading model understands semantics, the reasoning model infers based on compressed representations, with optimized division of labor.

## Technical Advantages of DRIFT

## Technical Advantages of DRIFT
### Efficiency Improvement
Reduces the number of tokens processed by the reasoning model via context compression, lowering computational complexity and memory usage, accelerating reasoning speed, and reducing costs.
### Performance Advantages
Outperforms full-context reasoning and existing compression methods in multiple long-context benchmarks, proving that implicit fact tokens effectively retain key information for reasoning.
### Interpretability
Separates the responsibilities of the reading and reasoning models; users can inspect implicit fact tokens to understand the basis for reasoning.

## Application Scenarios of DRIFT

## Applicable Scenarios of DRIFT
1. **Document Q&A**: Handles Q&A tasks for long documents like legal contracts and research papers.
2. **Multi-turn Dialogue**: Efficiently uses context in scenarios with large amounts of dialogue history.
3. **Code Understanding**: Analyzes large codebases to support code generation and defect detection.
4. **Knowledge Base Query**: Retrieves and infers relevant information from large-scale knowledge bases.

## Project Progress and Resources of DRIFT

## Project Progress and Resources
### Phased Release Strategy
- **Phase 1**: Core model architecture, reasoning scripts, processed training datasets, and data synthesis pipeline.
- **Phase 2**: Pre-trained model weights of different scales.
- **Phase 3**: Complete training scripts, distributed configurations, hyperparameters.

### Released Resources
- LFRP dataset: https://huggingface.co/datasets/SII-LancelotXie/DRIFT_LFRP
- QAFT dataset: https://huggingface.co/datasets/SII-LancelotXie/DRIFT_QAFT
- Data synthesis pipeline: ./data_generation/generate_qa.py (generates QA-evidence triples)

## Summary and Academic Contributions of DRIFT

## Summary and Academic Contributions
DRIFT addresses the trade-off between efficiency and effectiveness in long-context reasoning through its dual-model architecture and implicit fact token mechanism, performing excellently in benchmarks and providing a reference for long-text processing model design.
Academically: The paper has been published on arXiv (arXiv:2602.10021), completed by researchers from institutions like Fudan University and Shanghai Artificial Intelligence Laboratory, providing new ideas for long-context reasoning.