SRUG: Shadow-Guided Relightable Urban Scene Generation Model—Solving the Challenge of Shadow Modeling for Invisible Areas

SRUG (Shadow-Guided Relightable Urban Scene with Generation Model) is an innovative framework that addresses the challenge of shadow modeling for invisible areas in urban scene relighting. It uses a shadow-guided 3D completion model to recover the geometry of invisible areas, combines iterative material decomposition and physical lighting models to achieve realistic relighting effects for urban scenes, and has broad application prospects in film and television, games, architectural visualization, and other fields.

Urban scene relighting faces three core challenges:

1. Unbounded Scenes and Invisible Areas: Urban environments extend beyond the visible range, and invisible areas cast shadows onto visible areas;
2. Severe Underdetermined Problem: Recovering 3D geometry, materials, and lighting from 2D observations is a highly underdetermined inverse problem;
3. Complexity of Material Decomposition: Real-world material reflection properties are complex, and accurate decomposition of attributes such as diffuse reflection and specular reflection is crucial.

The core innovation of SRUG lies in transforming shadows into valuable clues for recovering the geometry of invisible areas. Shadows contain rich geometric information: light source direction, height and shape of occluders, and spatial relationships. By analyzing shadow patterns in visible areas, SRUG can infer the geometric structure of invisible areas outside the frame.

SRUG consists of three closely collaborative components:

1. Shadow-Guided 3D Completion Model: Completes the geometry of invisible areas through shadow detection, light source estimation, geometry inference, and iterative optimization (combined with physical rendering verification);
2. Iterative Material Decomposition Scheme: Uses a Large Material Model (LMM) to provide prior knowledge, and achieves accurate material decomposition through an iterative process of initial estimation → LMM supervision → gradient update → convergence judgment;
3. Physical Lighting Model: Explicitly models direct lighting, indirect lighting, sky lighting, and artificial light sources, supports adjustment of various lighting conditions, and generates physically plausible relighting results.

SRUG performs excellently on benchmark datasets:

• Quantitative Metrics: Significant improvements in PSNR and SSIM, a notable reduction in LPIPS, and substantial enhancement in shadow consistency;
• Comparison with Existing Methods: More reasonable geometry completion, more realistic shadow synthesis, more accurate material decomposition, and higher quality new view synthesis;
• Visual Cases: Successfully recovers high-rise building shadows, handles complex materials, achieves dynamic lighting transitions, and preserves long-distance details.

SRUG's application scenarios include:

• Film Post-Production: Changing lighting conditions of real-shot scenes;
• Game Development: Quickly creating interactive urban scenes;
• Architectural Visualization: Evaluating design performance under different lighting conditions;
• Autonomous Driving Simulation: Generating diverse lighting variants for training data;
• Virtual Reality and Metaverse: Creating immersive virtual environments from ordinary photos.

Current Limitations: High computational cost, reduced completion quality under extreme occlusion, weak handling of dynamic scenes, challenges in processing special materials (transparent/luminous); Future Directions: Optimize real-time processing, extend to video sequences, integrate multi-modal data, enhance details with diffusion models, support interactive editing.

SRUG transforms shadows, which were traditionally regarded as "interferences", into "assets", reflecting the trend of computer vision from purely data-driven to a combination of physical constraints and data-driven approaches. It provides important technical support for the construction of digital twin cities and demonstrates the idea of using neglected constraints to solve underdetermined inverse problems.