Panoramic Research on Large Language Model Systems: A Systematic Paper Resource Library from Training to Deployment

The GitHub repository LLMSys-PaperList, maintained by AmberLJC, continuously updates academic papers, tutorials, and project resources related to large language model systems. It covers key areas such as pre-training, post-training, inference services, multimodal systems, and Agent systems, serving as a must-read guide for LLM system researchers and systematically outlining the evolution of this field from 2019 to the present.

With the explosive development of large language models like ChatGPT and Claude, the AI field faces stringent requirements for underlying system architectures. From optimizing distributed training parallelism strategies to balancing inference service latency and throughput, and improving multimodal data processing efficiency, system-level innovations are needed. As an academic literature map, LLMSys-PaperList provides a systematic overview for researchers in this field.

The resource library divides core areas by technology stack and application scenarios:

1. Pre-training systems: e.g., Megatron-LM model parallelism scheme, ByteDance's MegaScale 10k-GPU training practice;
2. Post-training and RLHF systems: efficient fine-tuning strategies, reward model training, and distributed implementation of PPO;
3. Inference service systems: memory management, KV cache optimization, dynamic batching, and edge device operation;
4. Agent systems: tool call optimization, multi-turn dialogue state management, and secure interaction with external systems;
5. Multimodal systems: data loading related to GPT-4V/Gemini, feature extraction, and challenges in cross-modal attention computation.

2024: ByteDance's MegaScale 10k-GPU training, DeepSeek-V3 MoE training optimization, Alibaba's HPN data center network architecture; 2025: ByteScale training of 2048K context models with 12000+ GPUs, TileLink programming primitives, Flex Attention optimization; 2026: AXLearn modular training for heterogeneous infrastructure, Arena dynamic scheduling and adaptive parallelism co-design.

System researchers: Obtain a complete literature map to quickly locate classic and top conference frontier results; Engineering practitioners: Learn industry practice experience (fault recovery, load balancing, network optimization, etc.); Learners: Get a systematic learning path through resources like MLSys courses.

LLM system research is in a period of rapid development, with new hardware, algorithms, and system innovations emerging one after another. This resource library not only organizes existing achievements but also serves as a continuously updated knowledge hub, providing a reliable guide for researchers, engineers, and learners to explore the frontiers of LLM systems.