Indian Low-Resource Language LLM Evaluation Platform: A Modular Framework Bridging the Multilingual AI Gap

This article introduces a professional-grade LLM evaluation framework for six Indian low-resource languages (Telugu, Tamil, Kannada, Malayalam, Marathi, Hindi). The framework adopts a modular design, integrating a FastAPI backend and Next.js visualization portal, supporting multi-model engines and in-depth linguistic analysis. It aims to address the marginalization of low-resource languages in AI capability assessment and promote balanced development of multilingual AI.

The current LLM evaluation system is highly focused on English. Six Indian low-resource languages (Telugu, Tamil, Kannada, Malayalam, Marathi, Hindi) have long been marginalized in AI capability assessment, leading to performance blind spots of models in multilingual scenarios and hindering the implementation of localized AI applications.

Backend: FastAPI High-Performance Service

Build RESTful APIs using FastAPI, providing low-latency model inference interfaces and metric calculation services. Deployed via Uvicorn to support asynchronous processing and high concurrency.

Frontend: Next.js Research Portal

Build an interactive dashboard based on Next.js, integrating Recharts and Primereact component libraries to display model comparisons, heatmaps, scatter plots, etc., in real time.

Multi-Model Inference Engine

Supports open-source models such as Llama3, Mistral, and Gemma. Reserves extension interfaces for Indian language family architectures, allowing model and task switching via YAML configuration.

The platform is designed with a three-stage automated process:

1. Data Seeding Phase: Generate a simulated research corpus via scripts/download_data.py;
2. Dataset Construction Phase: Use scripts/build_datasets.py with IndicNLP preprocessing to build JSONL shards;
3. Model Evaluation Phase: Execute inference and calculate ROUGE, BERTScore, and complexity metrics via src/evaluation/benchmark_runner.py. A single command can complete the entire process from raw data to visual reports.

Different from traditional evaluations, the framework deeply analyzes language complexity features:

• Sentence length distribution: Identify the robustness of models to inputs of different lengths;
• Token depth analysis: Track the impact of subword segmentation on comprehension ability;
• Semantic similarity correlation: Perform correlation analysis between linguistic complexity metrics and model performance; It helps researchers understand the reasons for poor model performance rather than just knowing the results.

The project adopts solid engineering practices:

• Reverse proxy configuration: Hide backend details to enhance security;
• JSON Schema validation: Ensure consistent data formats to avoid runtime errors;
• Modular directory structure: Separate configs, data, src, and scripts with clear responsibilities;
• Virtual environment management: Provide venv activation scripts for both Windows and Linux/Mac platforms.

Practical Application Value

• Provide AI researchers with a standardized baseline for low-resource language evaluation;
• Show developers how to implement the combination of academic research and engineering practice;
• Remind the AI community to pay attention to the technical inclusion needs of global language diversity.

Conclusion

This project represents a technical ethical stance: AI development should benefit all language users. Through professional evaluation tools, the capabilities of low-resource language models can be measured, compared, and improved, paving the way for balanced development of multilingual AI.