InterWhen: Microsoft's Open-Source Runtime Validation Framework for Reasoning Models

Microsoft Research has launched the InterWhen framework, which uses runtime validation mechanisms to check intermediate states in real time during reasoning, ensuring that language model outputs comply with preset policies and providing new ideas for reliable reasoning in high-risk scenarios (such as code generation, mathematical reasoning, and agent workflows). The framework is open-source, supporting automatic generation of validators from natural language policies and guiding model trajectories to be compliant during reasoning.

In high-risk AI application scenarios, traditional methods only validate after the model generates the final answer, which has two major flaws:

1. Early errors are hard to recover: Early violations or irreversible errors in agent workflows cannot be remedied when final validation is performed;
2. Insufficient instance-level reliability: Validating only the final output cannot guarantee the correctness of the reasoning process. The InterWhen framework solves this problem by real-time validating intermediate trajectories.

The core idea of InterWhen is validator-guided reasoning, using the "LLM-Process-Modulo" execution mode:

• Offline phase: Automatically generate code validators from natural language policies, even Lean specifications and machine-checkable proofs;
• Online phase: Generate reasoning trajectories in a streaming manner, determine the timing of checks through lightweight boundaries (paragraph separators, tool call events, etc.), and continuously monitor intermediate states.

Key technologies of InterWhen include:

1. State extraction and asynchronous validation: Extract variables (tool names, parameters, etc.) from trajectories; validators return True/False/Unknown states, and validation is parallel to generation;
2. Intervention and recovery: Interrupt generation when validation fails, roll back to the checkpoint and attach feedback to resume generation; perform blocking validation for irreversible operations (such as writing).

Core features of InterWhen:

1. Policy-compliant agent reasoning: Validate intermediate reasoning, tool usage, and responses to ensure agent actions are compliant;
2. Validation during generation: No external steps required, maintaining model reasoning flexibility;
3. Asynchronous efficiency: Validation is executed asynchronously, with negligible overhead when correct;
4. Unified interface: Supports multiple types of validators such as symbolic and neuro-symbolic, adapting to different domain needs.

InterWhen was validated on benchmarks such as Maze, Game of 24, and SpatialEval, using models like Qwen2 and Phi-4. The results show:

• Improved accuracy under a given computational budget;
• Or improved efficiency at a given accuracy. Typical scenario demonstrations include telecom agent compliance (guiding trajectory compliance), Maze path counting (color-marked validation steps), and ZebraLogic constraint assignment (intuitively showing the validation process).

Applicable scope and limitations of InterWhen:

• Applicable scenarios: Suitable for formalizable tasks such as mathematics and code reasoning; not suitable for subjective tasks (creative writing); currently mainly supports English;
• Usage limitations: In the research phase, commercial applications require further testing; may not be feasible in latency-sensitive scenarios; has training data bias; does not prevent indirect prompt injection;
• Recommendations: All decisions need human supervision; do not rely solely on system outputs.

The open-source of InterWhen provides tools for research on reasoning model reliability and opens up new paths for trustworthy AI systems. It reflects Microsoft Research's commitment to responsible AI and provides a reproducible foundation for academia and industry. As AI is deployed in key fields, such validation frameworks will become an important component to ensure reliability. The community is welcome to provide feedback and collaborate via GitHub Issues or email.