KanEval: A Multi-Dimensional Evaluation Framework for Kannada LLM Summarization Tasks

KanEval is an open-source evaluation tool designed for low-resource languages. It helps researchers and developers objectively measure the performance of Kannada large language models (LLMs) in text summarization tasks through multi-metric comparison and semantic analysis. This framework addresses key challenges in evaluating low-resource language models and provides an intuitive interactive platform.

With the rapid spread of LLMs globally, mainstream languages like English have well-validated model capabilities. However, low-resource languages such as Kannada face unique evaluation challenges: lack of standardized benchmarks, single evaluation metrics, and difficulty in fair cross-model comparison. KanEval was created to solve these pain points by providing a systematic evaluation framework tailored to Kannada text summarization.

KanEval is built using Python and Streamlit, offering an intuitive web interface that allows users to perform model evaluation without writing complex code. Its core design goal is to lower technical barriers, enabling non-technical linguists and domain experts to participate in model testing. The framework supports loading multiple Kannada LLMs, performing parallel inference on a unified test set, and automatically generating comparison reports for fair model evaluation.

KanEval integrates traditional NLP metrics and semantic analysis methods:

• Traditional metrics: ROUGE-1, ROUGE-2, ROUGE-L to measure lexical matching between generated and reference summaries.
• Semantic analysis: Embedding-based semantic similarity to assess whether the model understands the core meaning of the original text.
• Readability evaluation: Kannada-specific readability scores to ensure generated summaries are natural and conform to language habits.

KanEval applies to various scenarios:

• Academic research: As a feedback loop tool to track performance changes during model iteration.
• Industrial use: Helps NLP engineers quickly screen candidate models for specific business needs.
• Extensibility: Modular architecture allows adaptation to other low-resource languages like Telugu and Marathi.

KanEval uses mature tools in the Python ecosystem: Streamlit for front-end interaction, Hugging Face Transformers for model loading and inference, NLTK and custom modules for metric calculation. It supports two deployment modes: local deployment (for sensitive data or large-scale batch evaluation) and cloud deployment (for team collaboration and result sharing).

KanEval has positive implications for the Kannada NLP community: it provides a practical tool and fosters a systematic evaluation culture, encouraging researchers to focus on multi-dimensional metrics. Looking ahead, as multi-language LLMs develop, language-specific frameworks like KanEval will play an increasingly important role in including underrepresented language groups.