GPUguesstimator: A Physics-Based GPU Selection Tool for LLM Inference

GPUguesstimator is an open-source tool that uses physical modeling to help developers accurately estimate the GPU memory and computing resources required to run large language models (LLMs). It solves the hardware selection challenge in model deployment and provides a scientific basis for hardware planning in LLM inference.

As the parameter scale of LLMs expands from billions to hundreds of billions, the hardware requirements for model inference have become a core challenge in AI engineering. Developers face issues like insufficient memory to load models or resource waste from over-configuration during deployment. Traditional rule-of-thumb methods are too rough to handle the complex needs of different model architectures, quantization strategies, and batch processing scenarios.

Memory Usage Estimation

• Model Weights: Calculate based on parameter count and precision (FP16, INT8, INT4, etc.). For example, a 70B parameter model in FP16 requires ~140GB, while INT4 reduces it to ~35GB
• KV Cache: Calculate based on sequence length, batch size, and attention head dimension. It may account for more than the model weights in long-context tasks
• Activations and Working Memory: Consider intermediate activations during forward propagation and framework overhead

Computational Throughput Modeling

• Prefill Phase: Limited by GPU matrix multiplication throughput (TFLOPS)
• Decoding Phase: Memory bandwidth is the bottleneck; estimate token generation speed based on memory bandwidth
• Batch Processing Optimization: Analyze the trade-off between batch size and throughput/latency

Scenario 1: Individual Developer Selection

Local running of Llama-3-70B: FP16 requires 140GB (exceeding RTX4090's 24GB), INT4 requires ~35GB (two 24GB cards or A100-40GB). Inference speed on A100 is about 15-20 tokens per second

Scenario 2: Enterprise Service Deployment

Supporting 1000 concurrent dialogue services: Estimate GPU cluster size, evaluate cost-performance trade-offs between A100 and H100, predict peak load latency

Scenario 3: Edge Device Deployment

Running small models on edge devices (e.g., NVIDIA Jetson): Evaluate whether quantized 7B/13B models meet memory limits, estimate context length, analyze CPU offloading feasibility

1. Physical Interpretability: Each estimation has a clear physical meaning, allowing users to understand the decision-making principles
2. Multi-Precision Support: Covers quantization schemes from FP32 to INT2, adapting to precision-efficiency trade-offs
3. Attention Optimization Awareness: Considers the impact of efficient implementations like FlashAttention on memory and computation
4. Hardware Database: Built-in specifications of mainstream GPUs, supporting quick configuration comparisons

• Actual performance is affected by inference frameworks (vLLM, TensorRT-LLM, etc.)
• System-level overhead (OS, drivers, other processes) may add extra memory usage
• Estimations are based on theoretical models; it is recommended to reserve a 10-20% safety margin

GPUguesstimator provides a scientific and transparent decision-making tool for hardware selection in LLM deployment, lowering deployment barriers and optimizing resource utilization. In the future, it will expand support for new model architectures (e.g., MoE) and dedicated AI accelerators to enhance AI infrastructure planning capabilities.