CARTE Benchmark: Exposing Systemic Blind Spots of Large Models in French Regional Knowledge

The CARTE benchmark uses 2,431 questions covering 13 regions and 14 thematic areas in France to reveal significant performance differences and pre-training coverage gaps of large models in regional-level geographic knowledge. The study evaluated 27 LLMs and found that models perform well in knowledge of mainstream regions (e.g., Paris), but poorly in remote/culturally unique regions (e.g., Corsica, Brittany) as well as dialects and fine-grained cultural knowledge, reflecting systemic biases in pre-training data.

Large language models have made significant progress in cultural understanding at the national level, but lack sufficient understanding of subtle differences at the regional level (e.g., Provence vs. Brittany in France). Existing benchmarks mostly focus on cross-country comparisons, language proficiency, or general knowledge, and generally ignore domestic regional differences, making it impossible to evaluate models' ability to understand internal national diversity.

CARTE (Culturally Anchored Regional-Territorial Evaluation) is specifically designed to assess LLMs' knowledge of French regions. France was chosen due to its long history, linguistic diversity (including Breton, etc.), and clear geographical/administrative divisions. It contains 2,431 multiple-choice questions covering 13 metropolitan regions and 14 themes (culture, language, economy, etc.); the CARTE-LV subset focuses on language variants (dialects, regional terms, language policies).

Evaluating 27 LLMs with 1B-12B parameters (in few-shot settings) revealed: 1. The scale effect exists but has diminishing marginal returns, and the largest models have not reached saturation; 2. Significant regional differences: high accuracy in the Paris region, low in remote regions like Corsica; 3. Thematic differences: good performance in general knowledge (geography/history), poor in fine-grained culture (dialects/traditions); 4. CARTE-LV shows models struggle to recognize dialects, regional terms, and language policies.

The results point to gaps in pre-training data coverage: 1. Data is biased towards mainstream regions (capital/economic centers), standard languages, and popular topics; 2. Models cannot learn missing knowledge, amplifying biases, and struggle to grasp long-tail knowledge (regional details); 3. Limited robustness to domestic regional variations, easily confusing similar regions.

CARTE question design principles: geographically anchored, discriminative, multi-grained, and objective. Quality control includes expert validation, multiple rounds of proofreading, and balanced coverage. Evaluation metrics include overall/regional/thematic accuracy and confusion matrices.

For developers: Need to improve geographic diversity, regional balance, linguistic inclusivity, and long-tail knowledge coverage in pre-training data; For the evaluation community: Provides a new dimension of regional granularity evaluation, which can be extended to other countries; For society: Lack of regional knowledge may lead to cultural neglect, representational bias, and fairness issues.

Core conclusion: Current LLMs have systemic gaps in pre-training coverage, with insufficient knowledge of non-mainstream regions, dialects, and fine-grained cultural knowledge. Limitations: Only covers France and French; Future directions: Expand to other countries/languages, dynamic updates, adversarial testing. CARTE provides an example for regional cultural evaluation, and we look forward to more similar work to promote balanced development.