AI Systems Engineering: A Knowledge Graph for Large Model Engineers in Production Environments

The original author amikumar91 maintains the open-source knowledge base 'AI Systems Engineering' on GitHub, which systematically organizes 136 topics from core model inference to agent orchestration, RAG to evaluation and governance. It provides a structured learning path for engineers building AI systems, addressing the knowledge gap and fragmentation issues in LLM production deployment.

LLM technology is evolving rapidly, but there exists a knowledge gap where 'those who understand models don't understand engineering, and those who understand engineering don't understand models'. Teams face systemic issues like inference optimization and RAG design during production deployment. Online knowledge is fragmented, and there's a lack of systematic production-oriented guides for engineers—this is the starting point for creating this knowledge base.

The knowledge base includes 10 core modules:

1. Core Model Inference (17 topics: Transformer architecture, KV Cache, etc.)
2. Prompt Engineering and Control (10 topics: System prompt design, etc.)
3. Service Infrastructure (15 topics: vLLM, TensorRT-LLM, etc.)
4. Model Optimization and Formats (11 topics: Quantization, LoRA fine-tuning, etc.)
5. Retrieval and Memory (12 topics: RAG architecture, vector databases, etc.)
6. Agents and Orchestration (15 topics: ReAct pattern, LangGraph, etc.)
7. Safety Alignment and Governance (15 topics: Prompt injection defense, etc.)
8. Evaluation and Quality (13 topics: LLM-as-Judge, etc.)
9. Observability and Operations (14 topics: Logging, model version management, etc.)
10. Integration and Cloud Native (15 topics: REST API, Kubernetes, etc.)

Four learning paths:

• Quick Start (7 topics, 2 days): Understand the overall picture of AI systems
• Basic Compulsory (22 topics, 2 weeks): Build a solid foundation
• Builder Path (28 topics, 3 weeks): For practical developers
• System Deep Dive Path (79 topics, continuous learning): Advanced technologies Currently, 10 topics are completed (🟢), and the rest are under development (🔴), covering core inference and basic prompt engineering.

1. Systematic and Structured: Modular design reduces the learning curve
2. Production-Oriented: Focuses on practical technologies like KV Cache and continuous batching
3. Continuous Updates: Updated in May 2026, supports community contributions
4. Technology-Neutral: Covers open-source (vLLM) and commercial (OpenAI API) solutions, no framework lock-in

• KV Cache: Caches Key/Value pairs to avoid redundant computations and reduce inference complexity
• Paged Attention: Draws on virtual memory to improve GPU memory utilization
• Speculative Decoding: Uses a lightweight draft model to generate candidates, verified by the main model, increasing throughput by 2-3 times
• RAG vs Long Context: Complementary relationship—RAG is precise and low-cost, while long context excels at global understanding

This knowledge base embodies the methodology of AI systems engineering moving from research to production, providing teams with a knowledge map to help build a complete cognitive framework. It is recommended that readers choose a path based on their background and practice; it is expected to become an authoritative reference in the field in the future. Note: The content is based on the current state of 10/136 topics—please follow updates in the original repository.