LiveK12Bench: Can Large Models Really Pass Real High School Exams?

LiveK12Bench: Can Large Models Really Pass Real High School Exams?

A new study reveals the performance gap of multimodal large models in real exam environments: GPT-5 scored 79 under ideal conditions but dropped sharply to 53 when switched to real exam constraint environments, exposing the limitations of current educational benchmark tests. The study built the dynamic, interdisciplinary LiveK12Bench benchmark platform, aiming to bridge the gap between laboratory evaluations and real teaching scenarios.

Current educational benchmark tests have three flaws:

1. Static datasets easily lead to data contamination; models may get high scores by memorizing and brushing questions;
2. Limited to single modality and subject, unable to fully reflect real capabilities;
3. Unable to simulate constraints like time and accuracy in real exams. These flaws create a huge gap between high lab scores and real classroom abilities.

LiveK12Bench Benchmark Platform

• Dynamic dataset: Contains over 2000 real high school exam questions (covering math, physics, chemistry, biology), continuously updated via automated processes to eliminate data leakage and model brushing.
• Simulated exam evaluation: Requires models to complete end-to-end problem-solving in a full exam environment (understand the question → choose strategy → perform calculations → give answer), constrained by time and accuracy, closer to real exam scenarios.

Testing of 12 mainstream multimodal models found:

• GPT-5 scored 79 under ideal conditions but dropped to 53 in real exam environments, its effective ability almost halved;
• All tested models showed similar patterns: excellent performance under loose lab conditions, but problems exposed in strict exam environments.

The main weaknesses of models include:

1. Visual understanding: Insufficient sensitivity to complex charts and layouts, prone to comprehension deviations;
2. Reasoning coherence: Prone to logical jumps or contradictions during long-chain reasoning;
3. Trade-off between efficiency and accuracy: Hard to balance fast answering and careful reasoning under time constraints.

Research Implications

1. Evaluating educational AI requires test methods close to real scenarios; high lab scores do not represent actual effectiveness;
2. Models need to improve visual reasoning and complex layout understanding abilities (basic requirements for educational applications);
3. Need to establish continuous dynamic evaluation platforms (such as LiveK12Bench) to avoid static benchmarks becoming outdated or contaminated.

Conclusion

Multimodal large models have great potential in the education field, but there is still a long way to go before conquering high school exams. Developers and educators need to adopt stricter and more comprehensive evaluation standards to ensure AI tools truly serve educational goals.