Meta AI Hackathon Customer Service Simulator: An Agent Evaluation Environment Based on Deterministic Scoring

A realistic agent environment designed for the Meta AI Hackathon, which evaluates AI agents' ability to handle multi-turn conversations by simulating complex customer service scenarios and a deterministic scoring system.

MetaAIHackathon is a real-world agent environment specifically designed for the Meta AI Hackathon, aiming to evaluate AI agents' ability to handle complex customer service scenarios. Unlike traditional manual evaluation, this project uses a deterministic scoring system to provide an objective and repeatable measure of agent performance.

The project places AI agents in real customer service support roles, requiring them to solve multi-turn customer problems. This design breaks through the limitations of traditional benchmark tests—it is not just simple question-answer pairs, but a complete dialogue process that requires understanding context, showing empathy, and advancing problem-solving through multi-turn interactions.

The scoring system evaluates agent performance from two dimensions:

• Professionalism: Language style, politeness level, and standardization of response structure
• Task Completion: Whether the problem is accurately understood, effective solutions are provided, and escalation is done correctly when necessary

The score ranges from 0.0 to 1.0 as a floating-point number, ensuring results are comparable and traceable.

The project designs three difficulty levels of test scenarios:

Basic test scenario where the agent needs to:

• Verify the legitimacy of the refund request
• Process the refund according to standard procedures
• Confirm customer information and perform the operation

This is an entry-level test to evaluate the agent's basic ability to follow instructions.

Emotional management test where the agent needs to:

• Identify the customer's frustrated emotions
• Show a high level of empathy to soothe the customer
• Advance problem-solving only after the customer's emotions have calmed down
• Use appropriate de-escalation language

This scenario tests the agent's emotional intelligence and communication skills.

Complex case handling where the agent needs to:

• Determine if the problem is beyond their authority
• Prepare a complete case background summary
• Execute the formal manager handover process
• Ensure the customer understands the reason for escalation and the expected timeline

This is the ultimate test of the agent's judgment and ability to follow procedures.