OAKS: Can Large Language Models Keep Up with Knowledge Updates? A New Benchmark for Online Continual Learning

The KAIST team proposes the OAKS (Online Adaptation to Continual Knowledge Streams) benchmark to evaluate large language models (LLMs) online adaptation ability in continuous knowledge streams. This research has been accepted by the ACL 2026 main conference, providing a standardized evaluation framework for the field of LLM dynamic knowledge update.

Real-world knowledge is continuously updated (e.g., news, scientific discoveries), but traditional LLMs use a one-time pre-training approach followed by parameter freezing, which cannot adapt to dynamic knowledge changes. Although there are studies on model editing and continual learning, there is a lack of systematic benchmarks to evaluate their effectiveness. OAKS is proposed to address this gap.

OAKS is designed around the question "Can LLMs answer questions accurately as knowledge streams update?":

1. Continuous Knowledge Stream Simulation: Build two datasets—OAKS-BABI (simplified scenario) and OAKS-Wiki (real Wikipedia scenario), each containing multiple time segments. The same question may have different answers at different times.
2. Dense Evaluation Mechanism: Test with the same set of questions at each time interval to track the evolution of the model's knowledge state (whether it learns new knowledge, forgets old knowledge, or distinguishes knowledge at different time points).
3. Online Adaptation Protocol: The model can only access current and past data, not future data, simulating real deployment constraints.

The OAKS open-source codebase has a clear structure:

• data/: Datasets and prompt templates;
• src/: Core algorithm implementation;
• assets/: Visualization resources;
• installation.txt: Installation guide. The code supports: reproducing paper results, testing custom online adaptation methods, and building new continual learning datasets.

The OAKS design reveals key insights:

1. Importance of Knowledge Tracking: Models need to have time awareness to understand "a fact holds at a certain time point".
2. Balance Between Forgetting and Updating: Need to avoid catastrophic forgetting while correctly overwriting outdated knowledge.
3. Necessity of Fine-Grained Evaluation: Coarse-grained final evaluation may mask problems in the learning process; fine-grained tracking is more critical.

OAKS has important value for both academia and industry:

• Academia: Provides standardized tools for directions like continual learning and model editing, promoting research on LLM dynamic adaptation.
• Industry: Evaluates the knowledge freshness of production models, guides incremental update strategies and time-sensitive design of RAG systems.
• Differences from Other Benchmarks: Focuses on dynamic knowledge (vs static QA), large-scale continuous updates (vs single-point model editing), and knowledge-intensive tasks for language models (vs general continual learning).

OAKS is an important progress in evaluating LLM dynamic adaptation ability, solving the core problem of models "keeping up" with the continuous evolution of knowledge in reality. It is a practical tool for researchers, can guide system design for engineers, and also reminds us that AI needs to have both knowledge breadth and the ability to adapt to changes.