Maistros: Innovative Practice of Building a Greek Large Language Model via Knowledge Distillation

The Maistros project uses knowledge distillation technology to transfer the capabilities of large reasoning models to a Greek-specific model, providing a reproducible technical path for the development of large models for low-resource languages. It addresses the shortcomings of Greek users relying on general multilingual models in terms of cultural understanding, grammatical accuracy, and other aspects.

Global large language models (LLMs) are dominated by English. Greek, a language with approximately 13 million speakers, has long faced the dilemma of lacking high-quality training data and scarce dedicated models. Although general multilingual models support Greek, they perform poorly in cultural understanding, grammatical accuracy, and local knowledge.

Knowledge distillation is a model compression technique proposed by Geoffrey Hinton et al. in 2015. Its core is to use the soft labels (probability distribution) of a large teacher model to guide the learning of a small student model. Maistros built a culturally adapted Greek corpus covering various genres such as literature and news, optimized the vocabulary and tokenization strategy based on the Transformer architecture, and adopted a two-stage training approach: pre-training to master basic language rules, and a distillation stage to imitate the output of the teacher model to gain reasoning capabilities.

Maistros performed excellently in Greek grammatical correctness tests (verb conjugation, noun case changes) and cultural knowledge tests (mythology, history, geography); its reasoning capabilities (mathematics, logic, code generation) exceeded models of the same scale; compared with general multilingual models, its performance in Greek-specific tasks improved by 15-30%, especially with a significant gap in tasks involving cultural context and linguistic nuances.

Maistros proves that knowledge distillation can be a shortcut for building dedicated models for low-resource languages and can be extended to languages in Northern Europe, the Baltic region, Southeast Asia, etc. The key lies in high-quality local corpora, appropriate teacher models, and effective distillation strategies. It also raises thoughts on linguistic diversity and AI fairness, avoiding the marginalization of non-English cultures.

Greek large models still face challenges such as data scale limitations and ecosystem construction (toolchains, interfaces, communities). The team plans to open-source model weights and training code, call for more researchers of low-resource languages to participate, promote the progress of multilingual large models, and achieve technological democratization and linguistic equality.