Rank-ICL: Using In-Context Learning to Enhance Query-Document Relevance Explanation Generation

Rank-ICL is an open-source project that uses In-Context Learning (ICL) technology to enable large language models (LLMs) to generate high-quality query-document relevance explanations, supporting three settings: zero-shot, few-shot, and Rank-ICL. Maintained by ariflaksito, the project is open-sourced on GitHub (link: https://github.com/ariflaksito/rank-icl) and was released on June 10, 2026. It addresses the problem that traditional information retrieval only provides relevance scores without explanations, aiming to improve the interpretability of retrieval results.

In the field of information retrieval, judging the relevance between queries and documents is one of the core tasks. Traditional methods often only provide relevance scores but cannot explain why a document is relevant to a query. With the rise of large language models (LLMs), researchers have begun to explore how to enable models to not only judge relevance but also generate human-understandable natural language explanations. The Rank-ICL project was born in this context; it focuses on using In-Context Learning technology to guide the generation of high-quality explanations by showing examples to the model, without the need for fine-tuning the model.

Rank-ICL stands for "Ranking with In-Context Learning", which is a method combining retrieval-augmented generation and In-Context Learning. The core idea is: when given a query and candidate documents, use carefully designed prompts and examples to enable large language models to generate natural language descriptions explaining why the query and document are relevant. Its advantages include: 1. No fine-tuning required—leveraging the ICL capabilities of LLMs without expensive training; 2. Strong interpretability—the generated explanations help users understand retrieval results; 3. High flexibility—supporting multiple settings to adapt to different scenario needs.

The project supports three settings:

1. Zero-shot Setting

Does not rely on any examples; generates explanations directly based on queries and documents, requiring the model to have strong semantic understanding capabilities.

2. Few-shot Setting

Provides a small number (1-5) of labeled query-document-explanation triples as examples to guide the model to learn explanation patterns, suitable for scenarios with limited data.

3. Rank-ICL Setting

A core innovation that combines ranking tasks and In-Context Learning; the model not only generates explanations but also ranks candidate documents according to the relevance patterns in the examples, making the explanations more aligned with ranking needs.

The project code mainly focuses on the following aspects: Prompt Engineering: Design effective prompt templates to organize queries, documents, and optional examples into a format understandable by LLMs. Example Selection: Select representative examples in few-shot and Rank-ICL settings to improve explanation quality. Output Generation: Call LLM APIs to generate explanation text and perform post-processing to ensure format compliance. Evaluation Metrics: Includes methods for evaluating explanation quality, such as metrics in dimensions like fluency, relevance, and faithfulness.

Rank-ICL technology can be applied in multiple scenarios:

• Search engine optimization: Provide explanations for search results to enhance user trust;
• Recommendation systems: Explain the reasons for recommending specific content;
• Academic literature retrieval: Help researchers understand the relevance between papers and queries;
• Legal/medical retrieval: Provide interpretable retrieval results in high-risk fields.

The Rank-ICL project demonstrates the potential of In-Context Learning in the task of generating explanations for information retrieval. Through simple example guidance, LLMs can generate high-quality relevance explanations, providing new ideas for building more interpretable retrieval systems. In the future, with the improvement of LLM capabilities and the optimization of example selection strategies, Rank-ICL is expected to be deployed in more practical applications, helping users better understand and utilize retrieval results.