Northern Thai LLM: Evaluation Framework for Dialect Understanding Capabilities of Large Language Models

For the translation task between Northern Thai dialect (Lanna language) and Standard Thai, this project constructs a complete evaluation framework for large language models, and significantly improves the models' performance on minority languages through LoRA fine-tuning.

Lanna language (ISO code: nod/nort2740) is a dialect used by millions of people in Northern Thailand, with significant differences from Standard Thai (tha/thai1261). Although it has a writing system (Lanna script), it is severely lacking in digital resources and internet content. This data scarcity makes Lanna a typical low-resource language scenario, which is ideal for testing the capability boundaries of large language models in handling non-mainstream languages.

The project adopts a clear three-layer architecture, with each layer named after a Lanna cultural item:

"Khuang" means container in Lanna culture; this layer is responsible for containerized data management:

• Convert raw corpus in Excel format to JSONL
• Perform stratified division of training/development/test sets
• Manage the alt-translation flow
• Support bidirectional translation: Lanna → Standard Thai, Standard Thai → Lanna

"Kuafai" means bamboo tray, symbolizing bearing and transmission. This layer is responsible for the actual operation of the model:

• Support cutting-edge API calls (GPT-4o, Claude, Gemini, DeepSeek-V3)
• Inference for open-source weight models (Typhoon2, SeaLLM, Qwen2.5, LLaMA-3.1-8B)
• LoRA fine-tuning (PEFT r=8)
• Provide the lanna-kuafai command-line tool

"Jorfa" means offering, representing the examination and inspection of the model. This layer focuses on evaluation and analysis:

• Triple-ChrF scoring (supports variable N-grams 1-4)
• G-statistic calculation
• Multi-dimensional facet slicing
• Error typology analysis
• Manual scoring form (BaiLan)
• Krippendorff's α consistency test (HomPoi)

The project adopts an improved ChrF (character-level F-score) evaluation method, calculating scores in three dimensions simultaneously:

1. ChrF_avg: Average F-score
2. ChrF_max: Best performance
3. ChrF_diff: Score difference (reflects the instability of model output)

This triple evaluation mechanism can capture the overall level and fluctuation degree of model performance.

The project establishes a five-category error classification system to help deeply understand model failure patterns:

• Lexical-level errors
• Syntactic-level errors
• Semantic-level errors
• Cultural-specific item errors
• Transcription errors