GP-Stratz: A Racing Simulation Environment for Evaluating AI Agent Strategic Capabilities

GP-Stratz is a deterministic racing strategy simulation environment developed for the OpenEnv Hackathon. It aims to evaluate the performance of large language model (LLM) agents in high-pressure, multi-variable decision-making scenarios, covering complex tasks such as tire management, weather response, and real-time strategy adjustment. Through its quantifiable and repeatable design, it eliminates random noise and helps researchers systematically test AI's reasoning, planning, and adaptability.

Racing sports (e.g., F1) are the ultimate embodiment of strategic decision-making. Victory depends on the quality of decisions at critical moments: when to pit for tire changes, how to respond to weather changes, strategies during safety car deployments, etc. These decisions involve the interplay of multiple variables like tire wear, weather, safety cars, and fuel load. GP-Stratz abstracts this complexity into an evaluable environment, allowing researchers to systematically test AI's strategic capabilities.

Deterministic Design

GP-Stratz adopts a deterministic design: the same initial conditions and decision sequence produce the same results, eliminating random noise and enabling accurate attribution of performance differences.

State Space

It includes key information such as current lap number, tire wear (0-100%, critical when exceeding 86%), weather conditions (0: sunny /1: rain imminent /2: raining), gap to opponents, safety car status, traffic conditions, tire wear rate, and tire type.

Action Space

Agents can choose from 5 discrete actions: pit stop (resets tire wear), maintain, conserve tires (reduce speed to decrease wear), push (increase speed to increase wear), and switch to rain tires (force pit stop to change to rain tires).

Reward System

The total reward is normalized to [-2.0, +2.0] and includes four parts:

• Correctness reward (±1.2): Evaluate the decision's correctness based on golden rules
• Proactive reward (+0.4): Reward strategies like pitting during safety car periods or preparing for weather changes in advance
• Consistency reward (+0.3): Encourage maintaining the same strategy for more than 3 consecutive laps
• Inconsistency penalty (-0.3): Penalize erratic decisions

Three-Level Tasks

• Basic decision-making (easy): Single-step optimal decisions (e.g., tire selection based on weather, pitting due to tire wear)
• Contextual decision-making (medium): Multi-factor integrated decisions (e.g., adjusting strategies by predicting weather)
• Sequential strategy (hard): Multi-step planning (e.g., undercut overtaking, weather transition)

Tech Stack

• FastAPI Web Service: Provides RESTful API, supports OpenAI Gym-style interaction
• Docker Containerization: Ensures environment reproducibility, exposes port 8000 to comply with OpenEnv specifications
• LLM Inference Integration: Supports APIs like OpenAI/Groq, outputs structured formats
• Dataset Generation: Creates diverse test scenarios

OpenEnv Compliance

• Clear task grading (easy/medium/hard)
• Scores strictly fall within the (0.001, 0.999) range
• Standard output format (with [START]/[STEP]/[END] tags)
• Compliance with health check requirements

• Benchmark Testing: Standardizes LLM strategic capability evaluation, compares performance of different models
• Capability Analysis: Understands the capability boundaries of LLMs in complex reasoning
• Training Environment: Serves as a training tool for reinforcement learning/supervised learning
• Educational Tool: An intuitive and interesting AI practice environment, closer to real decision-making complexity than Atari

The idea of GP-Stratz can be extended to fields such as supply chain management (inventory/logistics), financial trading (risk/return), and medical resource scheduling (emergency triage/operating room arrangement), providing a reference paradigm for evaluating AI's multi-step decision-making under uncertainty.