NCERTQABench: A Large-Scale Bilingual QA Evaluation Dataset Based on Indian School Curricula

Large language models are increasingly widely used globally, but how do they perform in non-English environments and on domain-specific knowledge? The education sector is an important test scenario—it not only tests the model's knowledge reserve but also its depth of understanding of curriculum content. The NCERTQABench project was built to address this need: a large-scale bilingual QA dataset rooted in India's school curriculum system, providing valuable resources for evaluating and optimizing educational AI.

Current mainstream large language model evaluation benchmarks (such as MMLU, GSM8K, HumanEval, etc.) have shortcomings:

• Cultural bias: Dominated by Western education systems, insufficient coverage of curricula from other regions
• Language limitations: English-centric, lack of systematic evaluation for educational languages like Hindi
• Knowledge granularity: Differences in depth and structure between general knowledge and educational curriculum knowledge
• Poor practicality: Evaluation results have low direct relevance to actual teaching scenarios

As a populous country, India has a huge education market and a unique curriculum system. The curriculum framework developed by NCERT is the core basis for India's school education. Building a dataset based on this framework can accurately evaluate the applicability of models in the Indian education market and provide references for localized research on educational AI.

NCERTQABench is built following a strict methodology:

Curriculum Content Anchoring

Questions and answers are all sourced from official NCERT textbooks, covering core subjects from middle school to high school: Science (Physics, Chemistry, Biology), Mathematics (Algebra, Geometry, Basic Calculus), Social Sciences (History, Geography, Politics, Economics), and Languages (English, Hindi Literature), ensuring high alignment with the syllabus.

Bilingual Parallel Design

• English version: For English-medium schools and international evaluation needs
• Hindi version: Covers regions where Hindi is the main teaching language

Diversified Question Types

Includes factual, reasoning, explanatory, and application-based questions, comprehensively evaluating different ability dimensions of models.

Scale features of NCERTQABench:

• Number of questions: Tens of thousands of QA pairs, covering all subjects and grades
• Subject coverage: Complete K-12 core curriculum system
• Difficulty distribution: Gradient design from basic concepts to high-order thinking skills
• Answer quality: Standard answers reviewed by education experts

The scale ensures the statistical significance and reliability of evaluation results.

NCERTQABench supports fine-tuning experiments:

Baseline Model Selection

Evaluated mainstream open-source models such as Llama series, Mistral series, Gemma series, Qwen series, etc.

Fine-tuning Strategies

Explored methods like full-parameter fine-tuning, LoRA fine-tuning, instruction fine-tuning, etc.

Key Findings

1. Scale effect: Larger models have better knowledge coverage
2. Language difference: Some models perform worse in Hindi than in English
3. Fine-tuning benefits: Fine-tuning with educational data significantly improves performance on related tasks
4. Subject difference: Models show varying performance across different subjects; science questions are more challenging.

Application Scenarios

• Model selection reference: Helps AI companies choose technical solutions suitable for the Indian education market
• Localization optimization: Identifies knowledge gaps for targeted data augmentation and model optimization
• Educational product evaluation: Used for internal evaluation of AI tutoring applications and intelligent question bank systems
• Academic research: Explores topics like multilingual educational AI, knowledge distillation, curriculum adaptation, etc.

Global Insights

• Importance of localized evaluation: General benchmarks struggle to capture the educational needs of specific regions
• Necessity of multilingual capabilities: Educational AI in non-English countries needs to have local language capabilities
• Value of curriculum alignment: Aligning evaluation content with the syllabus ensures practicality and interpretability.

Future Directions

• Subject expansion: Incorporate vocational education and skill training curricula
• Grade extension: Expand to primary school and higher education stages
• Question type innovation: Introduce open-ended questions and multimodal content
• Dynamic updates: Follow NCERT curriculum syllabus revisions

Conclusion

NCERTQABench represents progress in the field of educational AI evaluation: moving from general benchmarks to specific domains, from single language to multilingual, and from abstract knowledge to curriculum alignment. It provides valuable resources and insights for educational AI researchers, developers, and decision-makers. In the context of global AI competition, understanding and serving local educational needs is key to the success of educational AI products, and NCERTQABench provides a solid evaluation foundation for this.