KanEval: A Multi-Metric Framework for Summarization Evaluation of Kannada Large Language Models

KanEval is a Streamlit-based evaluation framework designed to address the gap in assessment tools for low-resource languages like Kannada. It enables researchers and developers to objectively compare the summary generation capabilities of Kannada large language models (LLMs) using a combination of NLP metrics and semantic analysis. Key goals include standardizing evaluation, multi-dimensional comparison, visualizing results, and providing an open-source tool for the Kannada NLP community.

While LLMs have advanced rapidly, benefits are unevenly distributed—high-resource languages like English lead, but thousands of low-resource languages lag. Kannada, with over 50 million users and a 1500-year literary tradition, faces a scarcity of digital resources and specialized evaluation tools. KanEval was created to fill this gap, focusing on standardized assessment of Kannada LLM summarization tasks.

System Architecture: Modular design with data layer (import test datasets: original text, reference summaries, model outputs), evaluation engine (NLP metric computation), visualization layer (Streamlit interface), and report generation (exportable results).

Metrics:

• Vocabulary level: ROUGE (1/2/L), BLEU (lexical overlap).
• Semantic level: BERTScore, MoverScore, Sentence-BERT similarity (deep semantic matching).
• Language-specific: Kannada character accuracy, grammar compliance, cultural adaptability.

Also supports multi-model comparison (side-by-side display, radar charts, significance tests, error analysis).

KanEval serves multiple use cases:

1. Model Selection: Helps enterprises choose optimal Kannada summary models (commercial/open-source).
2. Training Optimization: Assists researchers in monitoring model performance and refining training strategies.
3. Academic Benchmark: Provides a standard tool for comparable research results in Kannada NLP.
4. Teaching: Aids educators in demonstrating NLP evaluation concepts to students.

Three critical aspects:

1. Kannada Tokenization: Uses specialized tokenizers to handle the agglutinative nature of Kannada (root + affixes).
2. Reference Summary Quality: Includes data quality checks to filter low-quality references (avoiding distorted results).
3. Customizable Metric Weights: Allows users to adjust weights based on application needs (e.g., news vs. dialogue summaries).

Challenges:

• Data scarcity: Limited digital/text resources and annotated data for Kannada.
• Lack of evaluation standards: Makes model comparison difficult.

Solutions:

• Transfer learning: Leverage multi-language models (mBERT, XLM-R) for cross-language adaptation.
• Community-driven efforts: Open-source tools like KanEval help build shared benchmarks and datasets.

Outlook: Improved multi-language LLM capabilities, synthetic data, and community collaboration will drive progress.

Open Source Value:

• Tool contribution: Lowers technical barriers for Kannada LLM evaluation.
• Methodological reference: Serves as a template for other low-resource languages.
• Data/community: Shares datasets and fosters collaboration.

Future Plans:

1. Expand to other NLP tasks (translation, QA, text generation).
2. Integrate human evaluation workflows.
3. Add model interpretability features.
4. Support real-time production monitoring.
5. Extend to other Indian languages for a South Asian evaluation ecosystem.