AGI-Genkai: Extreme Experiments Exploring the Boundaries of Large Language Model Capabilities

This article introduces the AGI-Genkai project, a series of extreme testing experiments targeting state-of-the-art large language models, aimed at systematically evaluating and exploring the capability boundaries and potential limitations of current AI systems. The term 'Genkai' in the project name means 'limit' in Japanese, and its core goal is to clarify the capability ceiling of current state-of-the-art large language models through systematic experiments. The answers to these questions are not only about technical evaluation but also relate to how to safely and effectively integrate AI into all aspects of social operation.

With the rapid development of artificial intelligence today, questions such as the capability boundaries, true understanding level, and failure scenarios of large language models (LLMs) urgently need answers. Traditional benchmark tests focus on the average performance of specific tasks, while extreme tests focus on boundary cases: the systematic failure points of models when task difficulty increases, input is complex, or cross-domain knowledge is integrated. The scientific value of such tests includes: helping to understand the real capability range of models and avoid expectation deviations; revealing failure modes to provide directions for algorithm improvement; understanding capability boundaries is crucial for AI safety, facilitating the design of protective measures.

AGI-Genkai has designed a multi-dimensional testing framework covering different aspects of cognitive abilities:

1. Logical Reasoning Ability: Basic formal logic, mathematical reasoning, and complex inductive/deductive reasoning; gradually increase difficulty to observe systematic errors.
2. Knowledge Coverage Breadth: Factual, procedural, and metacognitive knowledge, involving different fields, periods, and abstract levels.
3. Long Context Processing: Information retrieval, summary generation, and cross-paragraph reasoning; observe performance degradation as context length increases.
4. Multimodal Understanding (if supported by the model): Cross-modal association and information conversion capabilities.
5. Creativity and Generalization Ability: Open-ended creation beyond training data, novel problem solutions, and response to unknown issues.

AGI-Genkai adopts a hybrid methodology:

• Quantitative Analysis: Use standardized metrics such as accuracy, F1 score, and BLEU score to compare model performance horizontally.
• Qualitative Analysis: Focus on output quality, reasoning rationality, and error type characteristics.
• Adversarial Testing: Detect model vulnerability through interference information, expression changes, etc., to verify whether it relies on surface pattern matching.
• Progressive Difficulty Increase: Gradually increase complexity from the basic level, record performance change curves, and locate capability thresholds and critical point characteristics.

Based on current research, typical test scenarios and findings include:

• Mathematical Reasoning: Performs well in basic arithmetic, but prone to errors in multi-step reasoning, with error accumulation and 'hallucination' phenomena (seemingly reasonable but wrong intermediate steps).
• Common Sense Reasoning: Answers direct common sense questions well, but performs poorly in indirect reasoning (requiring integration of implicit common sense).
• Adversarial Robustness: Sensitive to input perturbations (synonym replacement, word order adjustment, etc., leading to answer changes), over-reliant on statistical patterns.
• Long Context Processing: Although supporting ultra-long windows, information extraction ability decays with distance, with the 'lost in the middle' phenomenon (middle information is easily ignored).

Challenges faced by AGI-Genkai include:

• Evaluation Subjectivity: The standard for 'correct' answers in open-ended tasks is unclear.
• Incomplete Test Coverage: Cannot exhaust all problem types; models may perform differently in untested fields.
• Dynamic Nature: LLMs iterate rapidly; current limitations may be solved by the next generation of models, requiring continuous test updates.
• Test Impact on Systems: Developers optimizing for public test sets may reduce differentiation, requiring constant innovation in evaluation schemes.

The value of AGI-Genkai to the AI field:

• Helps developers clarify improvement directions, users establish reasonable expectations, and policy makers provide an empirical basis for governance.
• Promotes thinking about the essence of intelligence: exploring the similarities and differences between machine intelligence and human intelligence, and whether the current technical path leads to artificial general intelligence. Although this project does not provide final answers, it offers valuable experimental data and thinking materials, and is an indispensable link to more powerful, reliable, and safe AI systems.