FPGS: Feedforward Semantic-Aware Photorealistic Style Transfer for Large-Scale 3D Gaussian Splatting

The FPGS technology enables feedforward style transfer for large-scale 3D scenes. It can apply any artistic style to 3D scenes represented by Gaussian splatting in real time without per-scene training, while maintaining semantic consistency and rendering quality. This technology addresses the bottleneck of traditional methods that require per-scene optimization, achieving millisecond-level processing, supporting multi-reference style fusion and real-time rendering, and has wide application value in fields such as VR/AR and game development.

Extending style transfer from 2D images to 3D scenes faces unique challenges. Early 2D algorithms like AdaIN have significant effects on images but are difficult to adapt to the 3D domain. Compared to NeRF, 3D Gaussian Splatting (3DGS) technology achieves faster rendering speed and clearer quality. However, how to efficiently implement style transfer on 3DGS while maintaining multi-view consistency and semantic integrity was an unsolved problem before.

The core innovations of FPGS lie in its feedforward architecture and semantic-aware mechanism:

• Feedforward Architecture: No per-scene or per-style training is required. Stylization of a 3D scene is completed in a single forward pass, reducing processing time from minutes to milliseconds.
• Semantic Awareness: A semantic feature matching module is introduced to identify and protect the semantic structure of the scene, avoiding semantic distortion and artifacts in traditional methods (e.g., maintaining the spatial relationships between sky, walls, and vegetation in architectural scenes). In terms of technical architecture, FPGS integrates a pre-trained visual encoder to extract multi-scale style features, uses a lightweight style decomposition network to control style intensity, and designs primitive-level stylization operators for 3DGS (directly manipulating Gaussian primitive attributes).

FPGS supports multi-reference style fusion: Users provide multiple reference images, and the system automatically learns feature differences to generate fusion effects, which is suitable for complex scenes (e.g., applying different styles to different regions of urban streetscapes). For real-time rendering, through optimized architecture and CUDA implementation, it can achieve over 60fps on consumer GPUs. A block processing strategy is adopted for large-scale scenes, supporting large scenes with millions of Gaussian primitives (such as urban blocks and indoor spaces).

FPGS has a wide range of application scenarios:

• VR/AR: Real-time conversion of real scenes into specific styles to create immersive experiences.
• Game Development: Rapid prototyping of visual styles to shorten the art iteration cycle.
• Digital Creation: Providing 3D artists with instant interactive tools to replace manual adjustment of materials and lighting.
• Cultural Heritage Protection: Combining 3D cultural relic scanning with historical painting styles to achieve digital display.

Current Limitations: The effect on extremely abstract/surreal styles needs improvement; the temporal consistency issue for dynamic scenes has not been resolved. Future Directions: Improve the ability to adapt to extreme styles; introduce temporal consistency constraints to support dynamic scenes; develop more intuitive user interaction interfaces.

FPGS is an important milestone in the transition of 3D style transfer from the laboratory to practical application. By combining feedforward architecture, semantic awareness, and real-time rendering, it solves the efficiency and quality problems of stylizing large-scale 3D scenes, and has important reference value for the interdisciplinary fields of computer graphics, vision, and generative AI.