CBC-SLP: Robust Multispectral Semantic Segmentation via Structured Latent Projection

This article introduces the CBC-SLP method, which decomposes latent representations into shared and modality-specific components via structured latent projection to address the trade-off between modal missing and full-modal performance in multi-modal remote sensing image segmentation. It demonstrates superior robustness and performance compared to existing methods in experiments.

Multispectral data (RGB, infrared, radar, etc.) improves segmentation accuracy, but in reality, modal missing occurs due to sensor failures, weather, etc. Traditional shared representation learning is robust when modalities are missing, but fails to fully utilize the complementary information of each modality in full-modal scenarios, leading to a performance trade-off.

Studies have found that perfectly aligned multi-modal representations may lead to suboptimal downstream tasks because over-alignment discards valuable modality-specific information. For example, RGB is sensitive to color and texture, infrared reflects vegetation health, and SAR is not affected by illumination—forced alignment would lose these complementary features.

1. Explicit decomposition: Split latent representations into shared components (cross-modal invariant information) and modality-specific components (unique complementary information) as architectural inductive bias; 2. Adaptive transmission mechanism: Dynamically combine components based on modality availability; 3. Encoder-decoder structure with a core latent projection layer, avoiding complex gating to maintain simplicity and stability.

Evaluated on three datasets: Vaihingen, Potsdam, and MultiSpectral. Performance is higher in full-modal scenarios, decreases gently in missing-modal scenarios, and remains reasonable in single-modal scenarios. Ablation experiments show that removing specific components, shared components, or the adaptive mechanism all lead to significant performance degradation.

Qualitative analysis shows that shared components capture general semantics, specific components retain unique perspectives, and adaptive fusion adjusts dynamically. Insights include the importance of architecture as inductive bias, alignment not being the only goal, and dynamic adaptability.

Current limitations: Small number of modalities, mainly random missing, computational overhead; Future directions: Expand to multi-modality, missing prediction, end-to-end optimization, cross-domain transfer to tasks like medical imaging.