Zero Trust SRE Gym: A New Reinforcement Learning Environment for Training AI Agents to Operate Compliantly in Zero Trust Security Architecture

Zero Trust SRE Gym is an OpenEnv-based reinforcement learning (RL) environment that encodes enterprise compliance requirements as hard constraints via Constrained Markov Decision Process (CMDP). It trains AI security agents to follow approval processes and avoid unauthorized operations in zero trust networks. The project was selected into the Top 100 at the Meta PyTorch OpenEnv Hackathon 2026. Key features include real-time microservice digital twin, adaptive threat generation, and LLM-driven approval evaluation.

Modern SOCs face a huge workload gap—human analysts handle only ~10 events per shift vs thousands daily. Autonomous AI agents are expected to bridge this gap, but zero trust architecture requires explicit authorization for any action. Existing RL security environments prioritize speed over compliance, leading to potential violations (audits, penalties, outages). This raises the "bounded autonomy" problem: balancing AI speed with human governance. Zero Trust SRE Gym addresses this by framing compliance as CMDP constraints.

The gym shifts from "doing the right thing" to "doing the right thing the right way" by using architectural constraints instead of just rewards. Unauthorized actions (e.g., isolating nodes without approved tickets) result in immediate -20 penalty and round termination. It models compliance as CMDP: state space (system status, alerts, ticket/approval status), action space (investigate, submit ticket, wait for approval, isolate), reward function (task completion + compliance), and hard preconditions (isolation requires approved ticket).

Key components:

1. Real-time Microservice Digital Twin: 3 Flask services (frontend, payment, HR DB) with persistent, causal states. Service breaches generate real SIEM logs, and failures propagate via dependency graphs (e.g., isolating frontend affects HR DB).
2. Adaptive Threat Generator: Uses real CVE data (score ≥7.5) + Groq LLM to synthesize unique SIEM logs, evolving with agent capabilities.
3. LLM-driven Approval Evaluator: Randomly assigns roles (junior SRE, senior SRE, CISO) to evaluate ticket justifications, requiring forensic-quality docs for approval.

Trained with GRPO, the agent showed significant improvements:

Metric	Untrained	Trained
Average Reward	-13.8	+9.7
Policy Compliance	Near 0	≥0.82
Trained agents learned that following processes is the only path to positive rewards.

Selected into Top100 at 2026 Meta Hackathon (31k+ teams,71k+ participants). It solves real industry pain points: AI speed vs governance, autonomy vs approval, efficiency vs compliance. It sets new standards for AI agent training by integrating compliance into the environment. Tech contributions: first zero-trust RL environment, CMDP formalization of compliance, real-time digital twin, adaptive learning, LLM-driven evaluation.

Resources available: HuggingFace Spaces (online experience), Colab notebooks (training code/tutorials), HuggingFace blog (tech解读). For customization: add new microservices, define new compliance constraints, integrate other LLM backends, modify threat generation strategies.

Zero Trust SRE Gym represents a new AI training paradigm—teaching agents to act correctly under complex constraints. It translates abstract governance into concrete training signals, emphasizing "doing the right thing the right way". As AI is used in critical infrastructure, this bounded autonomy approach will be key to safe, compliant AI operations, setting a technical benchmark for the field.