ai-diag-nose: AI Agent-based Health Detection and Auto-Repair System for Microservice Architecture

ai-diag-nose is an open-source AIOps (Artificial Intelligence for IT Operations) tool developed by rgr-dev, specifically designed for health monitoring and fault handling of distributed microservice architectures. This project deeply integrates AI Agent technology with traditional operation and maintenance scenarios, realizing a complete closed loop from fault detection, root cause analysis to automatic repair, representing the latest exploration direction in the field of intelligent operation and maintenance.

Complexity Explosion

Modern microservice architectures usually consist of dozens or even hundreds of service instances, with intricate call relationships between services:

• Complex topology: Dependencies in the service mesh are difficult to understand intuitively
• Fast fault propagation: A single service failure may trigger cascading reactions
• Scattered logs: Troubleshooting requires jumping between multiple services to view logs
• Numerous metrics: CPU, memory, latency, error rate and other metrics need comprehensive analysis

Limitations of Traditional Monitoring

Traditional monitoring tools mainly have the following problems:

• Passive response: Problems are often discovered only after user complaints
• Rigid thresholds: Fixed thresholds are difficult to adapt to business changes
• Information silos: Monitoring, log, and tracing data are scattered across different systems
• Manual bottleneck: Fault troubleshooting highly relies on expert experience

AI Agent-driven Workflow

ai-diag-nose adopts a multi-Agent collaboration architecture, where each Agent is responsible for specific operation and maintenance tasks:

Health Detection Agent

• Multi-dimensional collection: Continuously collect various metric data of services
• Anomaly detection: Use machine learning to identify behaviors deviating from normal patterns
• Intelligent noise reduction: Filter occasional fluctuations and focus on real problems

Diagnostic Analysis Agent

• Root cause localization: Analyze fault propagation paths and locate the source
• Correlation analysis: Correlate scattered abnormal signals into complete events
• Knowledge reasoning: Reason based on historical cases and operation and maintenance knowledge bases

Repair Execution Agent

• Automatic repair: Execute predefined repair operations (restart, scale up, rate limiting, etc.)
• Gray verification: Verify the effect after repair and roll back if necessary
• Experience learning: Record repair effects and optimize repair strategies

Key Technical Features

Distributed Tracing Integration

Deeply integrate with distributed tracing systems (e.g., Jaeger, Zipkin):

• Call chain analysis: Visually display the flow path of requests between microservices
• Latency attribution: Precisely locate the service where the performance bottleneck lies
• Error propagation tracking: Track how errors propagate between services

Intelligent Anomaly Detection

Compared with traditional fixed-threshold alerts, adopt more intelligent detection strategies:

Dynamic Baseline

• Establish a normal operation baseline for services based on historical data
• Baseline automatically adjusts with business cycles (e.g., daily/weekly/seasonal patterns)
• Support different baseline strategies for different services

Multi-metric Correlation

• Do not view a single metric in isolation
• Analyze correlations between metrics (e.g., whether error rate changes synchronously when latency increases)
• Identify composite abnormal patterns

Natural Language Interaction

Support natural language queries, allowing operation and maintenance personnel to ask about system status in daily language:

• "Which services had anomalies in the past hour?"
• "Why has the response time of the payment service slowed down?"
• "Compare CPU usage with the same period yesterday" This interaction method greatly reduces the threshold for use, enabling non-experts to gain system insights.

Repair Strategy Library

Built-in repair strategies for common faults:

Fault Type	Detection Indicator	Repair Operation
Memory leak	Sustained growth in memory usage	Service restart
Thread pool exhaustion	Active threads approaching upper limit	Temporary scaling
Database connection pool exhaustion	Surge in waiting connections	Connection pool scaling
Downstream service failure	Sudden increase in error rate	Circuit breaking and degradation
High load	Simultaneous rise in CPU usage and latency	Auto-scaling

Secure Repair Mechanism

Automatic repair involves production environment operations, so security is crucial:

• Impact assessment: Evaluate the scope of impact of repair operations before execution
• Approval process: Key operations can be configured with manual approval
• Fast rollback: Quickly roll back if repair is ineffective or has side effects
• Change audit: Fully record all automatic repair operations

Application Scenarios

E-commerce Promotion Guarantee

During large promotions like Double 11:

• Real-time monitoring of core links such as orders, payments, and inventory
• Auto-scaling to handle traffic peaks
• Quickly locate and repair faults to reduce losses

Financial System Operation and Maintenance

For financial systems with high availability requirements:

• 7x24 uninterrupted monitoring
• Second-level fault discovery and response
• Compliance audit and change tracking

SaaS Platform Operation

Challenges faced by multi-tenant SaaS platforms:

• Tenant-level performance isolation monitoring
• Resource usage anomaly detection
• Automated capacity planning recommendations

Comparison with Existing Solutions

Capability	ai-diag-nose	Traditional APM	Basic Monitoring
Anomaly detection	AI-driven, dynamic baseline	Rules/thresholds	Fixed thresholds
Root cause analysis	Automatic reasoning	Manual analysis	Manual analysis
Auto-repair	Built-in repair agent	Requires integration with external systems	None
Natural language	Natively supported	None	None
Learning and evolution	Continuous strategy optimization	Static configuration	Static configuration

Future Development Directions

Based on the current architecture, ai-diag-nose has multiple scalable directions:

• Chaos engineering integration: Proactively inject faults to verify system resilience
• Predictive operation and maintenance: Shift from passive response to active prevention
• Cost optimization: Combine cloud resource costs for optimization decisions
• Multimodal monitoring: Integrate logs, metrics, tracing, and performance profiling

Open-Source Value and Community Contributions

The open-source of ai-diag-nose brings the following to the intelligent operation and maintenance community:

1. Reusable Agent framework: Not limited to operation and maintenance, can be extended to other fields
2. Best practice reference: Demonstrates the application mode of AI Agent in production environments
3. Collaborative improvement opportunities: The community can contribute new detection algorithms and repair strategies