Personal LLM Inference Benchmark on RTX 3090: Large Model Practice on Consumer Hardware

This project is maintained by mkhasykov on GitHub (Project link: https://github.com/mkhasykov/llm-inference, updated on 2026-05-24). Its core is to conduct personal LLM inference benchmark tests in the WSL2 Ubuntu environment with a single RTX3090 graphics card, explore the performance and optimization strategies of running large language models on consumer hardware, and provide practical local deployment references for individual developers, researchers, students, and enthusiasts.

Large language models are usually considered to require professional hardware, but advances in model compression technology and inference frameworks have made it possible to run LLMs on consumer hardware. As a consumer graphics card with large memory (24GB), advanced architecture (Ampere), affordable price, and high ownership, RTX3090 is an ideal test hardware; WSL2 is the practical choice for many Windows users, and this project explores under this configuration.

RTX3090 Hardware Features: Based on the Ampere architecture, it includes 10496 CUDA cores, 24GB GDDR6X memory (936GB/s bandwidth), third-generation Tensor Cores, etc. The 24GB memory can support quantized models with 30-40B parameters, and models of 13B or below have sufficient KV cache space. WSL2 Environment Considerations: Shares Windows drivers, requires installation of CUDA Toolkit, may have slight performance loss, and needs attention to memory configuration and file system optimization. Inference Framework Selection: The project may test frameworks such as llama.cpp (highly optimized, low memory usage), vLLM (high throughput), Hugging Face Transformers (general and easy to use), TensorRT-LLM (extreme performance), etc.

Benchmark Test Dimensions: Covers latency (first token/ per token/ end-to-end), throughput (tokens per second, concurrency, batch processing), resource usage (memory, GPU utilization, power consumption), model coverage (different scales, quantization, architectures). Optimization Strategies: Quantization techniques (INT8/INT4/GPTQ/GGUF, etc.), memory optimization (KV cache management, chunk loading, CPU offloading), inference optimization (FlashAttention, continuous batching, speculative decoding).

WSL2 Configuration Recommendations: Set sufficient memory (e.g., 24GB), store models in WSL ext4 partition, match CUDA version, consider Docker deployment. Common Problem Solutions: OOM errors can be resolved by restarting or adjusting batch size; performance fluctuations need to monitor I/O bottlenecks; ensure driver and CUDA compatibility. Performance Expectations: Llama-2-7B (INT4: ~100-150 tokens/sec), 13B (INT4: ~60-90), 70B (INT4: ~10-20, needs optimization). Actual performance depends on implementation and input/output length.

Project Value: Lowers the threshold for local LLM operation, provides real performance references and configuration experience, and guides hardware selection (memory priority, quantization is a must). Limitations: Results are affected by personal configuration and software versions, and the test scope is limited. Future Outlook: Directions such as hardware (larger memory, new architectures), software (better quantization, efficient attention), and models (MoE, distilled models) will continue to be optimized.

This project is a pragmatic engineering exploration, focusing on the actual performance of LLM inference on personal hardware, and provides valuable resources for ordinary developers on "what can be done" and "how to do it". Its down-to-earth approach and "give it a try" spirit are particularly precious in today's rapid AI development, and it is a good starting point for running LLMs locally.