1Cat-vLLM: An AWQ 4-bit Inference Engine Optimized for Tesla V100 GPUs

1Cat-vLLM is an optimization solution based on the vLLM inference engine, specifically customized for Tesla V100 GPUs. Its core features include support for AWQ 4-bit quantization precision, compatibility with CUDA 12.8, verified support for large models like Qwen3.5 27B/35B, and suitability for multi-GPU deployment environments. This project aims to help users with V100 hardware fully unleash its potential, enabling them to run modern large language models without upgrading to new hardware.

With the growing demand for AI computing power, new flagship GPUs (such as A100 and H100) are expensive. As a previous-generation data center GPU, Tesla V100 may not match the specifications of new models on paper, but it is affordable in the second-hand market and still widely used. The main limitations of V100 are its memory capacity and lack of new architectural features (like sparsity acceleration), but quantization technology can alleviate these issues. 1Cat-vLLM was developed precisely to address this need.

AWQ 4-bit Quantization: An activation-aware weight quantization method that protects important weight channels. It reduces the model's memory usage to 1/8 (e.g., a 27B model from 54GB to 13.5GB) while lowering memory bandwidth requirements. When combined with vLLM's PagedAttention, it improves inference speed.

CUDA 12.8 Support: Brings the latest driver optimizations and library improvements (such as cuBLAS/cuDNN), enhancing inference performance and maintaining compatibility with frameworks like PyTorch 2.x.

1Cat-vLLM has verified support for Qwen3.5 27B/35B models. Qwen3.5 is the latest model from Alibaba Cloud's Tongyi Qianwen team, which performs excellently in benchmark tests for Chinese understanding and code generation. Through AWQ quantization, these models can run on V100, allowing users to enjoy modern AI capabilities without new hardware—especially suitable for Chinese scenario requirements.

1Cat-vLLM supports multi-GPU deployment, with advantages including: 1) Model parallelism for handling larger models; 2) Data parallelism for improving throughput; 3) Enhanced system availability. Targeting V100's PCIe/NVLink connections, the project has optimized tensor parallelism and pipeline parallelism to maximize the collaborative efficiency of multiple GPUs.

Target Users: Research institutions/universities with V100, small and medium-sized enterprises (SMEs) with limited budgets, and organizations sensitive to data privacy.

Typical Scenarios: Internal knowledge base Q&A, document analysis and summarization, code assistance and review, customer service chat systems, etc. (These scenarios are sensitive to throughput and cost, and do not require extremely low single-request latency.)

Deployment Notes: 1) Install a driver supporting CUDA 12.8 (version 535+); 2) Prepare AWQ quantization files for the corresponding models; 3) Ensure sufficient system memory and CPU cores.

Tuning Recommendations: Adjust batch size (max_num_seqs) and KV cache ratio; enable continuous batching to improve throughput; conduct benchmark tests based on actual scenarios to find the optimal configuration.

Limitations: V100 does not support new features like FP8 computing and Transformer Engine, and its memory bandwidth (900GB/s for the 32GB version) is lower than that of A100 (2039GB/s), leading to performance constraints in some scenarios.

Future Outlook: Better quantization methods and efficient attention mechanisms can extend the lifespan of legacy hardware; hybrid deployment (using V100 for batch processing and new hardware for real-time tasks) may become a resource optimization direction.