Clutter-Scorer: Integration Practice of Physical AI and Robot Vision

This article will conduct an in-depth analysis of how the Clutter-Scorer project integrates browser visual perception, edge computing, and the Gemini large model to achieve intelligent understanding of the physical world and robot operation planning. The project is maintained by ajaycyril, with source code available on GitHub (link: https://github.com/ajaycyril/clutter-scorer), and was released on 2026-06-13T08:44:01Z. Its core goal is to address the operational challenges of robots in cluttered scenarios and build a complete closed-loop system from visual perception to action execution.

Clutter-Scorer aims to address the operational challenges of robots in real cluttered scenarios (such as desktop stacking, warehouse goods placement). Traditional robot vision relies on predefined models and regularized scene assumptions, making it difficult to adapt to the uncertainties of the open world. This project adopts an innovative approach: combining real-time browser visual collection, edge video analysis, and Gemini large model reasoning to build an end-to-end intelligent system.

Clutter-Scorer adopts an edge-cloud collaborative architecture:

1. Perception Layer: Acquires camera video streams via browser WebRTC, supporting cross-device deployment;
2. Edge Video Analysis Layer: Performs local CV tasks such as object detection, segmentation, and tracking, with low latency and privacy protection;
3. Cloud Inference Layer: Uses the Gemini large model for world modeling and affordance reasoning, outputting high-level semantic understanding.

The visual perception module is based on browser WebRTC and Canvas API, enabling high-frame-rate image capture and preprocessing with deployment flexibility. The edge layer uses lightweight deep learning models to perform real-time object detection, instance segmentation, and pose estimation, reducing network latency. Scene understanding constructs 3D semantic representations through multi-frame fusion and spatial relationship reasoning, understanding the "existence", "position", and "association" of objects.

As the system's "brain", Gemini receives scene descriptions and builds a coherent world model, understanding the functional relationships and physical constraints of objects. Its core capability is affordance reasoning: based on the geometry, material, and function of objects, it infers operational possibilities (such as identifying graspable objects and avoiding tipping risks). Its analogical reasoning ability can handle unseen objects, demonstrating the potential for understanding physical common sense.

Action planning adopts a hierarchical strategy: the top layer determines the operation sequence, the middle layer refines parameters, and the bottom layer generates motion trajectories and force control commands. Closed-loop verification validates the effect of actions through visual feedback and dynamically adjusts strategies. The project also explores sim-to-real transfer technology, using simulation environments to accelerate strategy learning and accumulate failure experience to improve the reliability of real-world deployment.

The technology of Clutter-Scorer can be applied in multiple fields:

• Warehousing and Logistics: Identify and grasp cluttered goods;
• Home Services: Plan action sequences for organizing clutter;
• Industrial Quality Inspection: Detect part anomalies and plan corrective actions;
• AR/VR: Create natural interaction experiences by combining physical world understanding.

Current challenges include: perceptual uncertainty (occlusion, lighting, etc.), trade-off between real-time performance and accuracy, and safety constraints. Future directions: improvement of multimodal large model capabilities, optimization of edge computing hardware, advancement of robot hardware, to promote the transition of physical AI from the laboratory to real-world applications.