Event Tensor: A Unified Abstraction for Dynamic Large Kernel Compilation

This paper proposes Event Tensor—a unified compiler abstraction that supports dynamic shapes and data-dependent computations. By generating high-performance persistent kernels through static and dynamic scheduling transformations, it aims to address bottlenecks in LLM inference such as kernel launch overhead and coarse-grained synchronization, significantly reducing inference latency and system warm-up overhead.

LLM inference faces issues like accumulated kernel launch overhead, memory bandwidth pressure, and limited parallelism. In traditional kernel decomposition models, fine-grained kernel launches and global synchronization restrict efficiency. Large kernel technology reduces memory access and synchronization overhead by fusing multiple operators into persistent kernels, but existing solutions struggle to handle dynamic shape and data-dependent computation scenarios in LLM inference.

Core Abstraction of Event Tensor: Event-based dependency encoding (static/dynamic dependencies), tile-level task representation (load balancing, pipeline parallelism, locality optimization), and unified support for shape and data-dependent dynamics.

ETC Compilation Flow: The front-end converts the computation graph into Event Tensor (operator decomposition, dependency analysis, dynamicity annotation); the middle layer performs hybrid scheduling of static (loop transformation, memory optimization) and dynamic (load balancing, pipeline scheduling); the back-end generates target GPU code (memory hierarchy optimization, synchronization code generation, instruction-level optimization).

ETC performs significantly in LLM inference scenarios:

1. Inference Latency: Lower latency in small batches, short sequences, and decoding phases, outperforming current state-of-the-art systems;
2. Warm-up Overhead: Significantly reduces system warm-up time and improves elastic scaling capabilities;
3. Dynamic Shape Adaptability: Adapts to input shape changes without recompilation, offering stronger generality.

Core optimizations enabling ETC's high performance:

1. Dependency-Driven Scheduling: Event execution is triggered by dependency satisfaction, maximizing parallelism;
2. Hierarchical Synchronization Mechanism: Selects warp-level/block-level/global synchronization on demand, reducing overhead;
3. Dynamic Load Balancing: Work stealing mechanism balances uneven computational loads;
4. Memory Access Optimization: Automatically selects optimal memory layout and access strategies.

Insights for Deep Learning Compilation: Value of unified abstraction (avoids multi-path maintenance), importance of runtime scheduling (static + dynamic hybrid), and continuous criticality of hardware-aware optimization.

Impact on AI Infrastructure: Reduces inference costs (direct economic value), improves user experience (low latency + fast response), and supports flexible deployment (dynamic shape adaptation simplifies processes).

Current Limitations: Limited range of supported operators (mainly LLM inference operators), no expansion to multi-GPU scenarios, no deep integration with techniques like quantization/pruning, and lack of automatic tuning mechanisms.

Future Directions: Expand operator support, adapt to multi-GPU distributed scenarios, collaborate with other inference optimization techniques, and introduce automatic tuning mechanisms.

Event Tensor is a significant advancement in the field of deep learning compilers. By unifying dynamic abstraction and an efficient compilation flow, it extends the advantages of large kernel technology to dynamic scenarios, significantly improving LLM inference efficiency. Against the backdrop of growing AI computing demands, such compilation technologies will play a key role in AI infrastructure and provide new ideas for future compiler design.