RAG_for_AI: A Project-level Knowledge Operating System Designed for Telegram

This is an open-source RAG system based on Django, which corely addresses the pain point that Telegram chat records are difficult to retrieve and utilize effectively. It converts conversations into a structured knowledge base, supporting transparent traceability, hybrid search (semantic + keyword + time decay), and multi-signal ranking to provide AI assistants with reliable context-aware capabilities. The core concepts of the project are Telegram-native, project-centric, and transparent traceability, suitable for team collaboration and personal knowledge management scenarios.

Project Background and Core Positioning

Telegram is the preferred communication tool for many organizations and individuals, but massive chat records are scattered and hard to retrieve. This project is specifically designed for the Telegram native environment, using RAG technology to convert conversations into a structured knowledge base, providing AI bots with intelligent Q&A capabilities based on real context. Unlike general RAG solutions, it deeply integrates with the Telegram ecosystem, supporting multi-bot configuration, Webhook real-time message reception, and automatically organizing knowledge by Domain and Project levels.

Technical Architecture and Data Model

Tech Stack

• Web framework: Django 5.1+ (Admin backend, Web interface, API)
• Database: PostgreSQL 16 + pgvector (vector storage, full-text search)
• Cache and Queue: Redis 7 (Celery broker, cache)
• Task Queue: Celery 5 (asynchronous processing of embedding, import, summary)
• Object Storage: MinIO (attachments, exported files)
• LLM: OpenAI API (compatible with other providers)

Data Model

Adopts a four-layer structure to organize information:

1. Domain: Large knowledge categories (e.g., work, family)
2. Project: Actual work units (supports parent-child relationships, aliases)
3. Conversation Thread: Continuous topics reconstructed via time clustering
4. Message: Refined management with 15 role tags, 5 value levels, and 5 sensitivity levels

Additionally, it includes elements like Wiki space, context packs, agent profiles, and knowledge items.

RAG Retrieval Process and Transparent Traceability

Four-stage Retrieval Pipeline

1. Data Ingestion: Webhook receives messages → standardized processing → tagging → routing to Domain/Project/Thread → storage and trigger embedding task
2. Index Construction: Celery generates vector embeddings (stored in pgvector) + full-text search/fuzzy matching indexes
3. Retrieval and Recall: Hybrid search (semantic 50% + keyword 30% + time 20%) → multi-signal scoring and ranking (role, freshness, credibility, etc.) → assemble context
4. Generation and Traceability: LLM generates answers → attach complete sources (messages, Wiki, knowledge items) → record retrieval sessions; low-confidence ones automatically enter the review queue

Transparent Traceability Design

Each answer provides source proof, supporting traceback to specific original messages, Wiki versions, or knowledge items. A built-in retrieval quality evaluation framework allows quantifying improvement effects.

Security, Privacy, and Deployment Use Cases

Security Measures

• Encrypted storage: Fernet symmetric encryption protects keys and sensitive configurations
• Access audit: Records each key read operation
• Sensitivity classification: Five-level tags for fine-grained access control
• Review queue: Low-confidence sessions are automatically reviewed

Deployment Methods

• Docker Compose one-click deployment
• Local development environment support
• SQLite mode (limited functions) vs production stack (PostgreSQL + pgvector + Redis + MinIO)

Use Cases

• Team knowledge base: Automatically archive Telegram project group discussions
• Personal note assistant: Structured management of private chats and saved messages
• Customer support bot: Provide evidence-based answers based on historical conversations
• Project document center: Automatically generate Wiki that integrates discussions and decisions

Open-source Ecosystem and Summary Outlook

Open-source Extensibility

• AgentProfile: Custom bot profiles
• ContextPack: Inject domain rules and skills
• API interface: Django REST Framework provides Token authentication, supporting system integration
• Reserved re-ranker interface: Future integration with ML models or cross-encoders

Summary and Outlook

This project represents a practical RAG implementation approach, focusing on the Telegram scenario. It provides a usable knowledge management solution through refined data modeling, transparent traceability, and modular architecture. It is suitable for technical teams looking for open-source RAG, or individual users who want to convert Telegram chats into knowledge assets to try.