Judge-Aware Ranking: A New Framework for Large Language Model Evaluation Without Ground Truth

This article introduces the Judge-Aware Ranking framework proposed by the TanXZfra team. Its core innovation lies in the introduction of a judge-aware mechanism, which enables reliable ranking of large language models without relying on ground truth. This addresses the limitations of traditional evaluation methods in open-domain tasks (such as creative writing and code generation) and provides a new methodological perspective for LLM evaluation. The framework is sourced from GitHub and was released on May 26, 2026.

Traditional LLM evaluation relies on ground truth, but in scenarios like open-domain Q&A, creative writing, and code generation, correct answers are hard to define, making metrics like BLEU/ROUGE ineffective. Additionally, manually annotating ground truth is costly and difficult to scale, creating an urgent need for reference-free evaluation methods.

The framework introduces a judge-aware mechanism that explicitly considers the characteristics of the judging model. Its core steps include: 1. Pairwise comparison: Let the judging model compare the answers of candidate models in pairs; 2. Judge modeling: Analyze and correct the uncertainty and bias of the judging model; 3. Reference-free ranking aggregation: Use ranking learning techniques to aggregate results, weighted by the judge's confidence.

Advantages: 1. Frees from ground truth dependency, suitable for open-domain tasks; 2. High scalability, no manual annotation required; 3. High interpretability, allowing understanding of the judging model's reliable and biased areas. Application scenarios: Model selection and deployment, fine-tuning effect evaluation, open-task evaluation, multi-dimensional evaluation (usefulness/safety/creativity, etc.).

Limitations: 1. The quality of the judging model affects the reliability of results; 2. The computational cost of pairwise comparisons grows quadratically with the number of models. Future directions: Develop efficient sampling strategies to reduce the number of comparisons; explore multi-judge integration to improve robustness; expand to multilingual and multimodal evaluation scenarios.

This framework provides an important methodological innovation for the field of LLM evaluation. Through the judge-aware mechanism, it achieves reference-free reliable ranking, opening up new possibilities for automatic evaluation of open-domain tasks. For LLM developers and researchers, it is a practical evaluation tool for handling complex tasks without ground truth.