Flowertex: AI-Powered Data Pipeline Medallion Architecture and Conversational Operations Platform

Flowertex is an open-source project integrating the medallion data pipeline architecture (Bronze-Silver-Gold) with AI-powered conversational operations. It supports one-click deployment to AWS and Databricks, enables real-time pipeline monitoring, fault diagnosis, and automatic repair via Claude AI, and supports multi-channel interactions through WhatsApp, Telegram, and Discord, addressing the inefficiency issues of traditional data pipeline operations.

In modern data architectures, the medallion architecture has become a standard pattern for data lakehouse design, but increased pipeline complexity leads to time-consuming and inefficient traditional monitoring and operations methods: engineers need to sift through logs, check configurations, and analyze dependencies. Flowertex combines the medallion architecture with AI-powered conversational operations, enabling real-time monitoring, intelligent diagnosis, and automatic repair through natural language interactions.

Layered Architecture: Front-end layer (Nuxt4 + Vue3), back-end layer (FastAPI + SQLAlchemy2 + Pydantic), message gateway layer (Omni Gateway supporting multi-channels), data pipeline layer (Databricks PySpark implementing the medallion architecture). Three Medallion Layers: Bronze layer ingests raw data (maintaining integrity), Silver layer cleans and transforms (data quality checks + desensitization), Gold layer performs business aggregation (building business metrics). Observer Agent: Continuously monitors pipeline status; when anomalies occur, it calls Claude for diagnosis and creates a GitHub PR for repair, forming a closed loop of detection-diagnosis-repair-validation.

12 Real-Time Tool Integrations: Pipeline monitoring (list_databricks_jobs, etc.), data query (query_delta_table, etc.), code collaboration (list_recent_prs, etc.), operations (update_job_schedule, etc.). Unified Multi-Channel Sessions: Web/WhatsApp/Telegram/Discord maintain seamless session state. Intelligent Command System: Supports slash commands such as /pipelines, /resume, /status to enhance interaction efficiency.

Security Encryption: JWT authentication, Fernet encryption for sensitive credentials, Redis session caching. Chaos Testing: Inject controlled faults to verify the Observer Agent's detection and recovery capabilities. One-Click Deployment: Terraform configures AWS infrastructure, Docker Compose quickly sets up the environment. Test Coverage: 204 pytest tests (113 for the Observer framework, 91 for the pipeline library), front-end Vitest + Playwright tests.

Customer Service Data Analysis in Insurance Industry: Analyze WhatsApp chat records to optimize customer service processes. Real-Time Business Monitoring: Business teams query key metrics (e.g., number of new policyholders) via natural language. Rapid Fault Response: Query the root cause of faults via natural language; the system automatically analyzes logs and executes repairs.

Quick Start: Clone the repository → configure environment variables → start via docker compose → database migration → access the web interface to deploy pipelines. Future Outlook: Predictive maintenance, natural language ETL, cross-system correlation analysis, intelligent data governance.