WFGY: Open-Source Troubleshooting Atlas for RAG and Agent Systems

WFGY is an open-source troubleshooting atlas for RAG, agent systems, and real-world AI workflows. It includes 16 types of problem maps, a global debugging card, and the WFGY 4.0 framework, aiming to help developers systematically diagnose and resolve AI system issues and tackle the challenges of debugging complex AI systems.

With the widespread application of RAG and agent systems in production scenarios, their failures are hidden, multi-dimensional, and have intertwined symptoms (e.g., poor retrieval leading to hallucinations, prompt issues masking retrieval defects), making developers prone to partial solutions. WFGY emerged as an open-source troubleshooting atlas, providing structured diagnostic methodologies and tools to build a comprehensive knowledge graph for classifying AI system issues.

WFGY summarizes 16 types of AI system failure modes, covering multiple dimensions:

• Data and Retrieval Layer: Document parsing errors, inappropriate chunking strategies, wrong embedding model selection, vector database bottlenecks, etc.;
• Model and Generation Layer: Prompt defects, improper context management, mismatched models, failed output format control, etc.;
• Agent Orchestration Layer: Unclear tool definitions, incorrect call sequences, chaotic state management, invalid loop control, etc.;
• Integration and Operation Layer: API rate limit handling, missing error recovery mechanisms, insufficient monitoring and alerts, version compatibility issues, etc.

The global debugging card is a structured checklist that follows the concept of 'from surface to depth, layer by layer': starting from symptoms, narrowing down the scope through diagnostic questions to locate the root cause. It includes diagnostic commands and tool recommendations, such as vector similarity analysis for retrieval quality issues, query rewriting evaluation; practical tips like prompt version comparison for model output issues, temperature parameter tuning, etc.

The WFGY 4.0 framework is the latest version, expanding the coverage of issues, introducing quantitative diagnostic indicators and automated detection tools. It enhances integration with mainstream AI development frameworks: adapting to RAG architectures like LangChain and LlamaIndex, and providing diagnostic solutions for agent frameworks like AutoGPT and LangGraph.

The methodological value of WFGY includes:

• Layered Diagnosis Thinking: Analyze AI systems layer by layer (data layer, model layer, orchestration layer, application layer);
• Hypothesis-Driven Debugging: Propose hypotheses and verify them through experiments to avoid blind attempts;
• Observability First: Emphasize the importance of logs, monitoring, and tracing, and provide observability best practices.

WFGY is applicable to multiple scenarios:

• RAG System Optimization: Troubleshooting guides from retrieval recall rate to document parsing anomalies;
• Agent Debugging: Identify tool selection, prompt design, or state management defects;
• Production Failure Response: Serve as an emergency manual for quick troubleshooting to shorten recovery time;
• Team Knowledge Precipitation: Organize issues and solutions according to the classification system to form organizational assets.

Open-Source Contributions: The community can submit new problem cases, improve diagnostic guides, develop auxiliary tools, and perform translation localization. Limitations and Improvements: Insufficient quantitative indicators, limited automation, need to supplement coverage in specific fields (medical/legal/financial), and continuous updates to keep up with AI technology development.