LLM Inference Optimization in Practice: A Complete Performance Tuning Solution from GPU to CPU

This open-source project shows how to optimize LLM inference performance in Google Colab T4 GPU and local CPU environments. Based on Microsoft's Phi-2 model (2.7B parameters), it uses techniques like quantization, batching, KV caching, and streaming generation to achieve a 67% reduction in memory usage and significant inference speedup, along with an engineering deployment solution.

With the popularization of LLM applications, inference performance optimization has become a core challenge for developers, directly affecting user experience and cost control. This project, open-sourced by akolkaryash01, covers two deployment scenarios: GPU (Colab T4) and local CPU (Windows). It uses the Phi-2 model as a benchmark to systematically compare the effects of various optimization techniques.

The project uses a combination of optimization methods:

1. Model Quantization: Compare FP16 baseline with 4-bit NF4 quantization;
2. Batch Inference: Merge multiple requests to improve hardware utilization;
3. KV Cache Preheating and Prompt Caching: Reduce first-token latency and avoid redundant computations;
4. Streaming Generation: Output tokens in real time to improve perceived response;
5. CPU Inference Optimization: Implement local CPU inference based on llama-cpp-python.

Performance Data:

• FP16 Baseline: 14.5 tokens/sec, memory usage 5.57GB;
• 4-bit NF4 Quantization: 7.3 tokens/sec, memory usage 1.84GB (67% reduction);
• 4-bit Batch x4: 12.5 tokens/sec, memory usage 1.84GB (throughput close to baseline). Quality Evaluation: Use ROUGE and BERTScore to ensure performance improvement does not sacrifice output quality.

Deployment Methods:

• FastAPI REST Interface: Standardized HTTP API for easy integration;
• Gradio Interactive Demo: Visual interface for quick validation. Tech Stack: Model (Phi-2), Quantization (Transformers + bitsandbytes), CPU Inference (llama-cpp-python), Web Framework (FastAPI), Visualization (Gradio), Evaluation Metrics (ROUGE/BERTScore).

The project provides a validated optimization checklist. The synergy between quantization and batching is significant (quantization alone reduces throughput, but combining with batching restores performance). It is suitable for edge devices and cost-sensitive scenarios. Developers are advised to flexibly combine optimization techniques based on hardware constraints and latency requirements.