Ekka: Automated Diagnosis of Silent Errors in Large Language Model Inference

Ekka is an automated diagnosis system that identifies the root causes of silent errors in large language model (LLM) inference by systematically aligning and comparing the intermediate execution states of the target framework and the reference framework. Its core idea is to transform silent error diagnosis into a differential debugging problem. It achieves an 80% pass@1 diagnostic accuracy in real-world benchmark tests and has successfully discovered 4 previously unknown silent errors, providing key support for LLM inference optimization.

The rapid evolution of LLM inference frameworks has brought about the problem of silent errors—hidden issues that do not trigger explicit error signals but cause a decline in output quality. Typical scenarios include numerical precision, memory optimization, parallelization, operator implementation differences, etc. Diagnosis is difficult due to the semantic gap between high-level symptoms and low-level root causes, and traditional debugging methods are inefficient.

The research team reframes silent error diagnosis as a differential debugging problem, using a known correct reference framework to compare with the target framework to be diagnosed. The Ekka system consists of three steps: 1. Execution state capture (instrumentation to obtain tensor values, attention weights, etc.); 2. State alignment and comparison (semantic-level alignment algorithm to establish correspondence and detect differences); 3. Root cause localization and reporting (trace the earliest difference point and generate a detailed report).

Ekka performs excellently in real silent error benchmark tests: pass@1 accuracy reaches 80%, pass@5 reaches 88%; it successfully diagnosed 4 unknown silent errors and obtained confirmation from developers. Compared with traditional methods, Ekka has high automation, accurate positioning, wide applicability, and strong interpretability.

Ekka faces three major challenges and corresponding solutions: 1. State space explosion—adaptive sampling strategy to prioritize capturing key checkpoints; 2. Numerical tolerance handling—semantic-aware tolerance mechanism applies different standards according to tensor roles; 3. Complex control flow—execution path normalization to map to a unified semantic space.

Ekka has significant value to the ecosystem: for developers, it accelerates the debugging cycle and improves code quality; for deployers, it ensures service quality and reduces operation and maintenance costs; for the open-source community, it promotes innovation, improves transparency, and accumulates knowledge of error patterns.

Ekka has limitations such as dependency on reference implementations, performance overhead, difficulty in handling non-deterministic errors, and insufficient multi-modal support. Future directions include developing self-consistent checks without reference implementations, lightweight online monitoring, error detection during training, and establishing a community error database.

Ekka provides a fully automated silent error diagnosis solution through the differential debugging approach. The 80% pass@1 accuracy and the discovery of new errors prove its practical value. As LLM inference optimization deepens, Ekka will become a key infrastructure to ensure service quality.