DeepInsight: A Unified Evaluation Infrastructure Across the Entire Stack of Physical AI

This article introduces the DeepInsight evaluation framework, which solves the problem of cross-layer regression localization in physical AI systems through unified runtime and diagnostic tracing capabilities, and has been put into production use in humanoid robot stacks. The framework aims to cover the full spectrum of physical AI evaluation and overcome the limitations of existing fragmented evaluation tools.

Physical AI systems consist of multiple layers, with large differences in time scales between layers (base models: tens of milliseconds vs. physical simulation: thousands of steps), diverse modal semantics (language/symbols → actions/states → physical forces/sensors), and heterogeneous resource requirements (GPU/physical engine/CPU). Existing federated solutions (splicing multiple tools) have problems such as lack of unified identity, inability to perform cross-layer diagnosis, and fragmented configurations.

DeepInsight achieves unification through three abstractions:

1. Unified Task Abstraction: Abstracts various evaluations (model decoding/physical interaction) into tasks for unified scheduling and management;
2. Unified Resource Abstraction: Uses a resource handle protocol to uniformly schedule resources such as GPU and physical engines;
3. Unified Tracing Identity: Assigns a unique identifier to each event, maintains causal relationships, and writes to shared storage. These abstractions enable new benchmarks to be integrated via configuration.

DeepInsight has been used in production in humanoid robot stacks, verifying its feasibility:

• Result Consistency: Reproduces reference results and readings from peer frameworks;
• Performance Advantage: Faster single-node runtime than federated solutions;
• Scalability: Near-linear scaling across nodes.

The core value of DeepInsight lies in cross-layer diagnosis: the unified tracing storage allows viewing the complete causal chain of events. For example, when adjustments to physical layer parameters lead to a drop in top-layer success rate, the root cause can be directly located via the tracing ID—this is end-to-end tracing that federated solutions cannot achieve.

Limitations: Unified abstraction may not adapt to special needs; tracing storage may become a performance bottleneck; currently mainly adapted to humanoid robot scenarios. Future Directions: Support more backends/evaluation modes; enhance tracing query and visualization; develop automated regression detection and root cause analysis tools.

Insights from DeepInsight for the industry:

1. Unification ≠ Homogeneity: Collaborate heterogeneous components via interface protocols;
2. Observability First: Unified tracing is key to diagnosing complex systems;
3. End-to-End Thinking: Evaluation should cover the complete process rather than individual components.