LLM Selection Tool: A Model Comparison Tool Based on Multi-Dimensional Percentile Scoring

This article introduces the open-source project llm-comparison, which is based on authoritative evaluation data from Artificial Analysis. It solves the multi-dimensional trade-off problem in LLM selection, supports custom metric weights and real-time reordering, and intuitively displays the relationships between models in dimensions like intelligence, price, speed, and latency through 2D/3D Pareto frontier charts.

The current LLM market is growing explosively, and traditional single-metric comparisons cannot reflect real-world scenarios. Production-level applications need to consider dimensions such as intelligence level, cost-effectiveness, response speed, first-token latency, and context window simultaneously. There are trade-offs between these metrics (improvement in one may lead to decline in another), and this tool aims to solve the problem of multi-dimensional optimal selection.

The project is implemented using pure Python standard libraries with no external dependencies; the code structure is separated (command-line entry, core logic, HTML templates). The core algorithm is direction-aware percentile scoring: for metrics where higher values are better, original percentiles are used; for those where lower values are better, inverse percentiles are used. The composite score is the average of selected dimensions, ensuring dimensionless, consistent direction, and fairness.

Visualizations include: basic table view (sortable), 2D Pareto scatter plot (trade-off between two dimensions), 3D interactive scatter plot (trade-off between three dimensions). Use case examples: budget-sensitive (intelligence + price), real-time interaction (speed + latency + intelligence), comprehensive comparison (multiple metrics).

Data source: Artificial Analysis. Update steps: 1. Copy the ranking data to input.txt; 2. Run convert_results.py to generate results.csv. A semi-automated mechanism ensures data accuracy.

Limitations: manual data acquisition, equal weights, no historical trends, limited visualization forms. Improvement directions: automated data acquisition, custom weights, historical trend analysis, more visualization types (radar charts, heatmaps, etc.).

The project solves practical selection problems, and its design philosophy is worth learning (single dependency, data-driven, flexible and easy to use, visualization-first). It provides a lightweight and complete starting point for LLM selection teams, which can adjust the logic or integrate into CI/CD processes.