AI Evaluation App: Practice of Production-Level Large Model Evaluation and Selection Framework

This project presents a production-level LLM evaluation framework using NBA datasets to enable data-driven model selection. It addresses the problem of subjective 'vibes-based testing' by leveraging RAG pipelines, LLM-as-a-Judge scoring, and multi-dimensional KPIs to compare local (Qwen, Gemma via Ollama) and cloud (Gemini via Google API) models in sports analysis scenarios.

In the era of LLM applications, model selection is critical but often relies on subjective 'feel tests'. This approach is risky as software shifts from deterministic to probabilistic. The AI Evaluation App aims to replace this with a structured framework using golden datasets and automated scoring to ensure objective decisions.

The framework consists of four layers:

1. Data Acquisition: NBA official API (via nba_api) for golden dataset (nba_golden_dataset.csv) and expert-defined eval questions (eval_questions.json).
2. RAG Pipeline: Context injection from golden data, multi-model inference (local Ollama models + cloud Gemini API), and unified Python test runner.
3. Evaluation Layer: LLM-as-a-Judge for automated Pass/Fail and semantic similarity scoring.
4. Decision Layer: Streamlit dashboard for model selection matrix with ROI analysis.

Core innovations include:

• LLM-as-a-Judge: Automates scoring of open-ended answers against human standards, ensuring consistency and scalability.
• AI Safety: Tests instruction following (e.g., rejecting subjective questions like 'who is the best player?') to prevent hallucinations.
• Benchmarking: Compares open-source local models vs commercial cloud APIs, and different parameter sizes (e.g., Gemma 2B vs 26B) across scenarios.

Two main challenges were addressed:

1. Data Integrity: NBA game IDs (e.g., 0042500101) were truncated in Pandas—solved by forcing string dtype and zfill(10).
2. Hardware Constraints: Large models (e.g., Gemma 26B) caused GPU issues—solved by client timeouts (60s) and GPU cooling periods between model switches.

The 5-dimensional KPI system includes:

• Accuracy (pass rate)
• Semantic similarity (1-5 score)
• Inference latency
• Cost of correctness
• Instruction following (safety)

Practical value: Supports product selection, continuous regression testing, cost-benefit analysis, and team knowledge documentation. The Streamlit dashboard helps stakeholders balance accuracy vs latency for ROI-driven decisions.