IndicGovBench: A Multilingual Large Model Evaluation Benchmark for Indian Government Scenarios

Key Takeaways: IndicGovBench is a multilingual evaluation benchmark designed for Indian government service scenarios, covering three languages—English, Hindi, and Marathi. It aims to assess large models' capabilities in legal, civic, and government process reasoning, filling the gap in professional evaluation tools for government scenarios. Its core value lies in focusing on India's unique multilingual government needs, emphasizing accuracy, multilingual consistency, and anti-hallucination ability, providing a standardized evaluation tool for government AI systems.

With the widespread application of large language models in global government scenarios, existing evaluation benchmarks mostly focus on general knowledge or academic abilities, lacking professional tools targeting specific countries' government processes and legal provisions. As a multilingual country with 1.4 billion people and 22 official languages, India's high-frequency government scenarios (such as PAN card correction, EPFO withdrawal, GST registration, etc.) have extremely high requirements for AI systems' accuracy, multilingual consistency, and anti-hallucination ability. IndicGovBench emerged to fill this gap.

IndicGovBench evaluates model capabilities from six core dimensions:

1. Procedural Reasoning: Understanding the steps of government processes (e.g., documents required for PAN card address correction);
2. Government Workflow Understanding: Grasping cross-departmental collaboration relationships and cross-material requirements;
3. Interpretation of Legal and Administrative Instructions: Accurately understanding professional terms and conditions in official documents;
4. Multilingual Consistency: Ensuring factually consistent answers to the same question across different languages;
5. Anti-Hallucination Ability: Detecting whether false policies or processes are fabricated;
6. Citizen Service Reliability: Comprehensive evaluation of answer completeness, operability, etc.

Typical Examples:

• Government Process Category (English): "What documents are typically required for PAN card address correction in India?" (Reference answer: identity proof, address proof, PAN card copy, and supporting address documents);
• Multilingual Reasoning Category (Hindi): "EPFO claim status kaise check kare?" (How to check EPFO claim status; reference answer: via the EPFO portal or UMANG app);
• Hallucination Detection Category (Marathi): "रेशन कार्ड अपडेट करण्यासाठी कोणती कागदपत्रे आवश्यक असतात?" (Documents required for ration card update; reference answer: identity proof, address proof, and family member information). Data Sources: All come from official Indian public channels, including india.gov.in, epfindia.gov.in, incometax.gov.in, etc., ensuring authority and compliance with no sensitive information.

Evaluation Metrics: Multi-dimensional metrics are used, including Exact Match (EM), Accuracy, F1 Score, Hallucination Rate, Multilingual Consistency Score, and LLM-as-Judge scoring. Technical Architecture: The code repository has a clear layered structure, including modules such as data (dataset), evaluation (scoring scripts), notebooks (demonstrations), docs (documents), etc. It is compatible with the Kaggle Benchmarks SDK, supporting reproducible evaluation and batch assessment.

IndicGovBench is currently in the early stage, with the following roadmap:

• Phase 1 (Current): Basic framework, pilot multilingual dataset, basic metrics;
• Phase 2 (Near-term): Expand data scale, introduce manual review, improve hallucination evaluation;
• Phase 3 (Mid-term): Release public leaderboard, accept community contributions; Long-term goal: Become an authoritative evaluation standard in India's government AI field, improving the reliability and transparency of AI systems.

Industry Significance: It demonstrates the necessity of domain-specific evaluation benchmarks and provides a reference for AI deployment in multilingual countries. Insights for other regions: Need to start from real needs, design core scenario tasks, and focus on safety indicators such as anti-hallucination. Conclusion: IndicGovBench represents a new direction in vertical domain large model evaluation. It provides an assessment tool for Indian government AI solutions, helping to identify model capability boundaries and risks, and promoting the development of government AI to a higher level.