OAKS: Evaluating Large Language Models' Online Adaptation Capabilities in Continuous Knowledge Streams

The KAIST AI team released the OAKS benchmark (accepted to ACL 2026 Main), which is the first framework specifically designed to evaluate large language models (LLMs) online adaptation capabilities in dynamic, continuously updated knowledge streams. This benchmark simulates continuously incoming knowledge streams, testing whether models can track knowledge evolution in real time and adjust their responses. It includes both synthetic and real-world datasets, and open-source resources (datasets, evaluation code, etc.) to drive industry progress.

Traditional LLM benchmarks assume a static knowledge base, which creates a gap with real-world scenarios. In reality, knowledge is dynamically updated (new events, fact corrections). When models are deployed in real-time interactive environments (search engines, intelligent customer service), they need to adjust their responses immediately without retraining—this capability is called "online adaptation", and OAKS is designed for this purpose.

Core Design Philosophy

Based on the temporal evolution characteristics of real knowledge, evaluation needs to be conducted at each stage of the knowledge stream to capture issues not found in traditional benchmarks (such as forgetting early information, handling contradictory information, etc.).

Evaluation Mechanism

Adopts step-by-step online evaluation: after receiving each knowledge chunk, test the same set of questions to measure immediate adaptability, cumulative accuracy, forgetting patterns, and error propagation. The data structure includes fine-grained annotations (e.g., chunk_to_answer) to precisely locate error points.

OAKS-BABI (Synthetic Dataset)

Built on BABILong, it tests structured knowledge evolution: context length of 128k tokens, 65 chunks, 1200 questions. Question types include simple fact tracking, counting, bridging, and comparison, with an average of 4.7 answer changes.

OAKS-Novel (Real Dataset)

Uses 19 public domain novels as sources: context length of approximately 150k tokens,78 chunks,870 multiple-choice questions with an average of 5.5 options each. It includes complex character relationships and plots, with annotated evidence sources.

Basic Settings

Receive concatenated historical context; if it exceeds the length limit, truncate the earliest content: maximum document length of128k tokens, generation length of4096, temperature of0.7, Top-p of0.8, Top-k of20.

RAG Settings

Use dense retrievers (e.g., Qwen3-Embedding) to retrieve relevant chunks: build index → precompute retrieval results → answer based on retrieved chunks, simulating actual retrieval-augmented architectures.

Model Development Guidance

Identify weaknesses in temporal processing, optimize context management, improve training data, and evaluate the effectiveness of architectures.

Practical Application Scenarios

Real-time question-answering systems, financial intelligence analysis, medical diagnosis assistance, legal document analysis, and other fields that require dynamic knowledge adaptation.

Limitations

Language coverage (mainly English), knowledge types (focus on factual), interaction mode (passive reception).

Future Plans

Add multilingual support, expand coverage of knowledge types, and introduce active query and interactive learning scenarios.

OAKS shifts LLM evaluation from static to dynamic, which is a key indicator of a model's practical value. As a diagnostic tool, it helps developers improve model capabilities; open-source resources provide support to the community and are expected to drive the development of online adaptation technology.