Muduo Lock-Free Work Stealing Engine: Hardware-Aware Concurrent Task Scheduling Optimized for LLM Inference

This project is a hardware-aware concurrent task engine designed specifically for the Muduo network library. It optimizes performance for asymmetric workloads such as LLM inference using techniques like lock-free work stealing and cache line alignment, addressing the performance bottlenecks of traditional thread pools under heterogeneous requests.

Muduo Network Library

Muduo is a C++ network library based on the Reactor pattern, using a one-loop-per-thread model where each thread maintains its own event loop.

Work Stealing Scheduling

Work stealing is a dynamic load balancing technique where each thread maintains its own task queue, and idle threads steal tasks from other queues to reduce synchronization overhead.

Challenges in Lock-Free Programming

Implementing lock-free data structures requires solving issues like memory ordering, ABA problem, and cache consistency.

1. Lock-Free Work Stealing Queue

Thread-local operations require no synchronization; stealing operations are atomically safe; cache line alignment reduces false sharing.

Hardware-Aware Optimization

• Cache line padding: Ensure queue head/tail pointers, metadata, and task data are separated; key counters are aligned by cache lines.
• NUMA awareness: Prioritize memory allocation from local nodes; stealing strategies consider topology to reduce cross-node access.

A hardware-aware concurrent task engine for Muduo servers, optimized for asymmetric workloads (e.g., LLM inference) using lock-free work stealing and cache line alignment techniques.