infer-check: Catching Correctness Defects in LLM Inference Engines Missed by Benchmarks

infer-check is a tool specifically designed to detect correctness defects in LLM inference engines. It can identify hidden errors that traditional benchmarks fail to catch, helping developers improve the reliability of inference engines. In large language model (LLM) deployments, performance optimizations of inference engines (such as quantization, pruning, etc.) often introduce hard-to-detect correctness defects. However, traditional benchmarks cannot find these issues due to limitations like focusing on final output metrics. infer-check aims to fill this gap and help improve the reliability of inference engines.

Modern LLM inference engines use various technologies to improve efficiency. Common optimization methods include quantization, KV cache optimization, speculative decoding, operator fusion, etc. While these optimizations enhance performance, they may alter computational behavior, leading to subtle differences from the original model's output in edge cases. Traditional benchmarks usually only focus on the similarity or perplexity of the final output, ignoring correctness during the computation process, so they cannot capture these differences.

The core goal of infer-check is to fill the detection gap of traditional benchmarks, focusing on verifying the correctness of inference engine implementations. It systematically detects the following errors: numerical precision issues (precision loss caused by optimizations like quantization), memory management errors (KV cache boundary errors), operator implementation defects (custom kernel logic errors), and concurrency/race conditions (non-deterministic errors in high-throughput scenarios).

infer-check uses multiple technologies to achieve comprehensive verification: 1. Reference Implementation Comparison: Using a high-precision PyTorch implementation as the gold standard, compare the output of the tested engine and mark differences beyond the tolerance; 2. Edge Case Testing: Design test cases for extremely short/long sequences, special tokens, precise numerical edge cases, KV cache boundaries, etc.; 3. Randomness Control: Control random seeds to ensure repeatable testing of non-deterministic optimizations; 4. Stress Testing: Verify correctness in high-concurrency scenarios and capture race conditions and memory corruption issues.

Traditional benchmarks have limitations: 1. Insensitivity of End-to-End Metrics: Metrics like BLEU are not sensitive to minor token-level errors; 2. Insufficient Test Coverage: It's hard to cover all input patterns and computation paths; 3. Lack of Fine-Grained Verification: Only compare final outputs, ignoring correctness of intermediate steps; 4. Neglect of Numerical Stability: Do not pay attention to numerical stability issues after optimization.

infer-check is valuable to multiple types of users: inference engine developers (CI integration to prevent regressions), model deployment engineers (verify configuration correctness before deployment), optimization technology researchers (verify optimization correctness), and model users (evaluate engine reliability for selection).

To fully leverage infer-check, it is recommended: 1. Integrate into CI/CD pipelines; 2. Run full tests regularly; 3. Set reasonable numerical tolerance thresholds; 4. Combine with performance testing; 5. Record and track defects.

Possible future development directions for infer-check: support more inference engines and optimization technologies, develop dedicated modules for specific model architectures, introduce formal verification, and build a community defect database. In conclusion, infer-check is a key tool to ensure the balance between performance and correctness in LLM deployments, and it is indispensable for teams that value deployment quality.