SceneCritic: A Symbolic Evaluator for 3D Indoor Scene Synthesis

This article introduces SceneCritic, a symbolic evaluator based on the structured spatial ontology SceneOnto. By jointly verifying semantic, directional, and geometric consistency, it provides stable and interpretable evaluations for 3D indoor scene layouts, significantly outperforming VLM-based assessment methods. It addresses the issues of view sensitivity, prompt sensitivity, and hallucination in existing LLM/VLM evaluations, offering a reliable assessment tool for the 3D scene generation field.

With the application of LLMs and VLMs in 3D scene generation, assessment methods rely on LLMs/VLMs to score rendered views, but they have fundamental flaws:

• View sensitivity: Different angles lead to score discrepancies
• Prompt sensitivity: Results depend on prompt wording
• Hallucination: VLM judgments do not match the actual scene These issues cause unstable evaluations and hinder scientific progress in the field.

SceneOnto Ontology Foundation

SceneCritic is based on the SceneOnto ontology, which aggregates prior knowledge from 3D-FRONT (professionally designed scenes), ScanNet (real scanned environments), and Visual Genome (visual relationship annotations), covering object categories, spatial relationships, directional constraints, and geometric rules.

Multi-dimensional Consistency Verification

1. Semantic consistency: Check the semantic rationality of objects and their relationships (e.g., a kitchen should have a stove, chairs should be around a dining table)
2. Directional consistency: Verify object orientations (e.g., sofas face the TV, bed heads are in a reasonable direction)
3. Geometric consistency: Detect collisions, spacing, and size proportions

Fine-grained Output

Provide object/relationship-level violation identification, success labeling, and interpretable feedback to help developers locate issues.

• Alignment: The consistency between SceneCritic's evaluation results and manual annotations is significantly better than that of VLM evaluators
• Cross-modal comparison: Pure-text LLMs sometimes outperform VLMs in semantic layout quality, challenging the perception that visual tasks require visual models
• Optimization effect: Image-based VLM critics perform best in semantic and directional corrections, highlighting the value of visual feedback

• Reliable evaluation benchmark: Stable and reproducible, ensuring comparability of different research results
• Debugging tool: Fine-grained feedback helps developers improve models in a targeted manner
• Hybrid strategy guidance: Experiments reveal the advantages of different critic modalities, providing a basis for hybrid evaluation

• Ontology coverage: Need to expand to special scenes (industrial spaces, outdoor-indoor hybrids)
• Cultural differences: Consider culture-specific variations in layout preferences between East and West
• Dynamic scenes: Extend the framework to support time-dimensional dynamic scene evaluation