Long Context LLM Inference Performance Benchmark: Memory and Latency Analysis from 8K to 128K+

This project is a systematic open-source benchmark framework designed to measure the impact of long-context workloads on large language model (LLM) inference performance, covering comparative analysis of various model architectures, hardware configurations, and inference frameworks. Its core goal is to reveal performance bottlenecks in long-context scenarios (such as attention computation complexity, KV cache memory usage, batch processing efficiency, etc.), provide objective data support for developers and researchers, and assist in decisions related to model selection, hardware configuration, and deployment frameworks.

As LLM context windows expand from 8K to 128K+, traditional short-text inference optimization strategies can no longer address new challenges. The LLM_Inference open-source project emerged to answer key performance bottleneck questions when long contexts grow by orders of magnitude through systematic, reproducible benchmarks, providing a standardized measurement system for decisions across models, hardware, and frameworks.

The project establishes a comprehensive performance evaluation indicator system:

• Time dimension: TTFT (Time to First Token), TPOT (Average Time per Output Token), total latency;
• Throughput and resources: Tokens generated per second, peak GPU memory, KV cache memory estimation, success/failure status (e.g., OOM). All results are accompanied by metadata (model, backend, hardware, context length, batch size, etc.) to ensure cross-platform comparability.

Different attention mechanisms show significant performance differences:

• MHA: Strong expressive power, but KV cache grows linearly with the number of heads, leading to high memory pressure for long texts;
• GQA/MQA: Reduce cache usage by sharing KV, which are memory optimization solutions. The project quantifies the impact of these architectures on latency and throughput, helping to understand the trade-offs between 'memory for speed' or 'speed for memory', which is valuable for deployment in resource-constrained environments.

Comparison of mainstream frameworks:

• Hugging Face Transformers: Baseline reference with strong direct inference capabilities, but may have bottlenecks in long-context high-throughput scenarios;
• vLLM: Continuous batch processing + paged KV cache to improve throughput, suitable for high-concurrency services;
• TensorRT-LLM: NVIDIA compilation optimization, operator fusion + quantization to maximize GPU utilization, pursuing extreme single-run performance. Comparisons under the same hardware and workload will reveal the applicable boundaries of each optimization strategy.

Two core test modes are supported:

• Context length scan: Fix batch size =1, gradually increase input length (8K→16K→32K→64K) to identify performance degradation thresholds or OOM critical lengths;
• Batch size scan: Fix context length, change batch size (1→2→4→8) to study throughput-latency trade-offs. Results are stored in JSONL format, and summary scripts are provided to generate statistical reports.

Modular design:

• Benchmark module: Backend-agnostic experiment configuration, prompt generation, indicator collection, result storage;
• Backends module: Independent implementation of each inference framework, following a unified interface;
• Analysis module: Aggregation analysis and visualization tools. Adding a new backend only requires implementing the standard interface; support for vLLM and TensorRT-LLM is already in planning.

Future plans: Introduce streaming generation paths to directly measure TTFT, support inference/vision-language models, and improve batch processing runs. Community value: Fill the gap in long-context benchmarking, promote domain standardization, facilitate experience sharing, and provide systematic performance analysis infrastructure for scenarios with continuously growing context lengths.