PPB-MCP: Transforming GPU Benchmark Data into Queryable MCP Services

PPB-MCP is an open-source Model Context Protocol server developed by paulplee. It encapsulates over 30,000 real-world records from Poor Paul's Benchmark (PPB) GPU inference data (including quantization schemes, throughput, memory usage, concurrent user count, etc.) into queryable services, supporting mainstream AI clients like Claude Desktop and Cursor. Guided by the principle of "evidence first", this project helps developers solve decision-making challenges in LLM deployment such as quantization scheme selection and memory planning, providing data-driven reliable recommendations.

During LLM deployment, developers often face complex issues like quantization scheme selection and hardware configuration matching (e.g., "What quantization scheme should I use for a 32GB GPU running Qwen3.5-9B with 8 concurrent users?"). The PPB dataset contains a large number of real benchmark records but lacks a convenient query method. PPB-MCP was created to transform PPB data into MCP services, allowing AI clients to query directly.

PPB-MCP provides 9 query tools, divided into three categories:

1. Basic Queries: list_tested_configs (lists all tested GPU/model/quantization schemes), query_ppb_results (filters raw benchmark data);
2. Intelligent Recommendations: recommend_quantization (three-level confidence quantization recommendation), get_gpu_headroom (verifies memory headroom);
3. Quality Assessment: get_qualitative_summary (obtains quality scores), query_qualitative_results (filters quality data), get_context_rot_breakdown (long context recall analysis), get_tool_accuracy_breakdown (tool call accuracy breakdown), compare_quants_qualitative (quantization scheme quality comparison).

The recommendation engine has three confidence levels: High (≥3 actual tests on the same GPU), Medium (scaled conversion across different GPUs), and Low (formula extrapolation).

PPB-MCP uses an SQLite local caching strategy: it loads the local database at startup and only updates when the dataset's git commit SHA changes, supporting offline use and reducing HuggingFace dependencies. It supports two MCP transport protocols: stdio (local integration) and streamable-http (remote deployment). The official hosted endpoint is https://mcp.poorpaul.dev/.

Integration: Supports clients like Claude Desktop, Cursor, Windsurf, and VS Code. Configuration involves modifying the corresponding JSON files (e.g., Claude requires editing claude_desktop_config.json to add the MCP server address). Deployment:

• pip installation: pip install ppb-mcp, startup command example: MCP_TRANSPORT=stdio ppb-mcp;
• Docker deployment: Run the official image and map ports;
• Development/production deployment: Provides git cloning, dev dependency installation, and one-click deployment scripts (supports Docker, systemd, etc.).

PPB-MCP addresses four key pain points in LLM deployment:

1. Difficulty in quantization scheme selection: Recommendations based on actual test data;
2. Memory planning risks: The get_gpu_headroom tool avoids OOM;
3. Inaccurate performance estimation: Provides throughput estimates using real data;
4. Trade-off between quality and speed: compare_quants_qualitative allows intuitive comparison. Example: Querying the quantization scheme for a 32GB GPU running Qwen3.5-9B (8 concurrent users) returns Q5_K_M (high confidence) along with memory/performance data.

PPB-MCP transforms the static PPB dataset into a dynamic query service, helping developers make data-driven decisions in LLM deployment. Project advantages include data-driven approach, multi-level confidence, wide compatibility, flexible deployment, offline-friendliness, etc. In the future, we will continue to update the PPB dataset and enhance recommendation capabilities; community contributions are welcome. It uses the MIT license and is suitable for local/private cloud LLM deployment optimization.