E2LLM: An Efficient LLM Inference Framework for Heterogeneous Edge/Fog Environments

E2LLM is an efficient LLM inference framework for heterogeneous edge/fog environments. Its core strategies include model replication and role separation, combined with genetic algorithms and dynamic programming to optimize resource allocation, enabling efficient inference on resource-constrained devices and reducing waiting time by over 50% compared to baselines.

Large Language Models (LLMs) face three major challenges when deployed in edge/fog environments:

• Resource heterogeneity: Significant differences in device computing power and memory
• Complex network topology: Unstable communication bandwidth and latency
• Dynamic load: Dramatic changes in request volume and input/output length The assumption of traditional single-device deployment is no longer valid, making efficient distributed deployment a key issue.

Dual Replica Architecture

Copy the complete model to multiple device groups (replicas), and assign specialized roles to each replica:

• PREFILL Role: Handles input token pre-filling (computation-intensive)
• DECODER Role: Handles output token decoding (memory-intensive)

Intelligent Cluster Organization

Use genetic algorithms for device clustering to maximize overall system performance

Dynamic Partitioning Strategy

Use dynamic programming to determine the optimal model partitioning within clusters, minimizing execution bottlenecks These strategies leverage differences in the characteristics of inference stages to optimize resource utilization.

Experimental results show:

• E2LLM can adapt to workloads with varying input/output token lengths
• Under high pressure, compared to the Splitwise baseline, the average waiting time is reduced by over 50% Reasons for performance improvement:

1. Specialized division of labor improves efficiency
2. Genetic algorithms and dynamic programming achieve load balancing
3. Fine-grained partitioning reduces inter-device waiting and synchronization overhead

Technical insights and application prospects:

1. Specialized division of labor: In resource-constrained environments, shifting from unified processing to specialized division of labor can improve performance
2. Algorithm-system collaboration: Deep integration of genetic algorithms, dynamic programming, and system architecture to address heterogeneous complexity
3. Feasibility of edge intelligence: Proves that LLMs can be efficiently deployed in edge environments, paving the way for edge AI applications

E2LLM addresses the core challenges of LLM deployment in heterogeneous edge/fog environments through its model replication and role separation architecture, combined with genetic algorithm and dynamic programming optimizations. It achieves a reduction of over 50% in waiting time, marking a major breakthrough in edge LLM inference efficiency and providing a feasible path for AI deployment in resource-constrained scenarios.