OpenEnv Email Classification System: An Intelligent Customer Service Decision Framework Based on Q-Learning

This project builds an email classification environment compliant with OpenEnv specifications, combining large language model (LLM) reasoning with Q-learning reinforcement learning agents to enable automated email processing decisions (supporting operations like reply, escalation, archiving, etc.). It aims to address the problems of high cost, large delays, and frequent misjudgments in traditional manual processing, as well as the limitation of rule/supervised learning methods that lack long-term decision-making considerations, providing an efficient decision framework for intelligent customer service.

In modern enterprise customer service systems, email remains an important communication channel, but manual classification and response have issues of high cost, delays, and misjudgments. Traditional rule-based methods struggle to handle content diversity and ambiguity; pure supervised learning can learn classification patterns but lacks long-term consideration of decision consequences—"correct" classification may lead to subsequent delays, while "suboptimal" choices may solve problems faster, which provides space for the application of reinforcement learning.

The project strictly follows OpenEnv specifications (emphasizing reproducibility, evaluability, and production compatibility), implementing core interfaces: reset() to initialize the environment, step(action) to execute decisions, state() to return the state; data models are defined using Pydantic to ensure consistency; FastAPI endpoints are provided for easy integration. Additionally, three levels of task difficulty are designed: simple (routine inquiries), medium (refund/billing issues), and difficult (system failure reports), simulating priority requirements in real scenarios.

The agent's action space includes reply, escalation, archiving, and request information; the reward mechanism balances efficiency and quality: correct action +1.0, partially correct +0.5, incorrect 0.0, step penalty (-0.1 × steps). Q-learning agent implementation: state representation (LLM-embedded semantics/keyword sparse features), epsilon-greedy strategy (reducing exploration rate with training), experience replay (breaking data correlation), reward shaping (accelerating early learning).

An LLM baseline is established (OpenAI-compatible API, supporting local/cloud switching, fixed random seeds to ensure reproducibility). The evaluation uses a deterministic scoring system (consistent output for the same input), with metrics including accuracy, average reward, average steps, and performance across difficulty levels, comprehensively assessing the agent's capability boundaries.

The project is not only a research prototype but also has commercial value: automating the processing of over 80% of routine inquiries to free up manual labor; intelligent routing to assign the most suitable team to reduce turnaround time; identifying edge cases requiring manual review through decision confidence; continuously optimizing based on actual data to adapt to business changes.

Evolution directions for the open-source project: multimodal expansion (supporting image/document attachments), multi-agent collaboration (sub-agents handling specific tasks), human-machine collaboration (seamless transfer to humans when the agent is uncertain), cross-language support (serving global enterprises), demonstrating the potential of reinforcement learning in real business scenarios.