Practical Guide to LLM Inference Optimization on Consumer-Grade GPUs: Quantization, Concurrency, and Cloud Platform Comparison

This study focuses on LLM inference optimization on consumer-grade GPUs (RTX 2080 8GB). It tests the effects of FP16/INT8/INT4 quantization and concurrency performance using the vLLM framework, and compares the deployment cost-effectiveness of AWS SageMaker and Google Vertex AI cloud platforms. It aims to answer two core questions: How to maximize inference performance on resource-constrained consumer hardware? Which platform offers better cost-effectiveness for cloud deployment? This provides a practical deployment guide for developers.

With the popularity of LLMs, efficient deployment in resource-constrained environments has become a challenge. Most developers and small-to-medium enterprises lack high-end GPUs and need to improve inference efficiency on consumer-grade hardware. This study focuses on two questions: 1. How to maximize LLM inference performance on the RTX 2080 (8GB) through quantization and concurrency control? 2. When deploying the optimal configuration to the cloud, which platform (AWS SageMaker or Google Vertex AI) offers better cost-effectiveness?

The experiment is divided into two parts: local optimization and cloud platform comparison. Local Optimization: Using the vLLM framework to test the meta-llama/Llama-3.2-3B-Instruct model. Variables include precision (FP16/INT8/GPTQ/INT4/AWQ) and number of concurrent users (1/4/8/16). The baseline is HuggingFace Transformers + FastAPI, and the dataset is ShareGPT (median input: 200 tokens, output:150 tokens). Cloud Platform Comparison: Deploy the optimal local configuration (INT4 AWQ) to AWS SageMaker (ml.g5.xlarge, A10G 24GB, $1.41/hour) and Google Vertex AI (g2-standard-4, L4 24GB, $0.98/hour). Compare latency, throughput, tokens per dollar, cold start time, and auto-scaling performance.

Core Advantages of vLLM:

1. PagedAttention: Draws on virtual memory management, splits KV cache into fixed blocks, eliminates fragmentation, and improves memory reuse.
2. Continuous Batching: Dynamically adds new requests to improve GPU utilization and throughput. Trade-offs of Quantization Technologies:

   Precision	VRAM Usage	Max Sequence Length	CUDA Graph	Application Scenario
   FP16	~6GB	1024	Disabled	High-quality short text
   INT8	~3-4GB	2048	Enabled	Balanced quality and efficiency
   INT4	~2GB	4096	Enabled	Resource-constrained high concurrency
   Note: The actual available VRAM of RTX2080 is about 6.9GB (Windows WDDM reserves 1GB), so FP16 requires disabling CUDA Graph and limiting sequence length.

Baseline Comparison: vLLM vs HuggingFace (single request): average latency reduced by 33.2%, P95 latency reduced by36.3%, token generation speed increased by49.4%, total throughput increased by57.1%. Synergy Between Quantization and Concurrency: Under high concurrency, INT4 throughput exceeds FP16 (reasons: memory release supports larger batches, CUDA Graph enabled, better concurrency scalability); INT8 is the sweet spot for most scenarios (close to INT4 performance with minimal quality loss). Cloud Platform Comparison: The INT8 throughput of Google Vertex AI's L4 GPU is about twice that of AWS A10G (485 TOPS vs250 TOPS), and the cost is 30% lower—this is important for cost-sensitive applications.

Monitoring and Observability: Use Prometheus + Grafana to monitor metrics such as KV cache utilization, request queue depth, latency distribution (P50/P95/P99), time to first token (TTFT), and throughput. Deployment Process: Provide Docker Compose configuration to start vLLM, Prometheus, and Grafana with one click; inject HuggingFace tokens via environment variables to support models from private repositories. Cost Control Recommendations: Delete endpoints promptly after cloud deployment; use auto-scaling; consider tokens per dollar instead of unit price comprehensively.

Insights: 1. Quantization is a strategy rather than a compromise—INT4 throughput exceeds FP16 in specific scenarios; 2. Concurrency design should fully leverage vLLM's continuous batching;3. Cloud platform selection needs to consider hardware performance, unit price, cold start, etc., comprehensively. Future Directions: Multi-tenant isolation optimization, dynamic precision switching, benchmark testing of more open-source models on different hardware.