Research on Multimodal Tensor Connectivity: Exploring Robustness of Low-Rank Fusion and Geometric Conditioning

This project focuses on the tensor connectivity problem in multimodal AI, combining multi-kernel learning theory and low-rank multimodal fusion models to study the impact of geometric conditioning and rank constraints on generalization ability, robustness, and modal interaction. The project is maintained by ParthSinha19, with source code available on GitHub (https://github.com/ParthSinha19/Robustness-Of-Multimodal-Tensor-Connectivity), and was released on June 8, 2026.

Traditional multimodal systems face two core problems: geometric misalignment of data from different modalities in the latent space, making models vulnerable to distribution shifts and adversarial perturbations; high-dimensional fusion introduces over-parameterization, increasing computational costs and noise sensitivity. This project proposes a theoretical framework combining joint Wasserstein Autoencoder (jWAE) and Low-Rank Multimodal Fusion (LMF) to address these issues.

The project is based on three key hypotheses: 1. Low-rank constraints are an implicit spectral regularization mechanism that enables learning more compact and generalizable representations; 2. Geometric conditioning aligns embeddings of different modalities through shared Gaussian priors, reducing distribution mismatch; 3. Multimodal robustness depends on the balance of modal contributions; imbalance reduces system robustness.

The technical architecture integrates multi-kernel learning, tensor decomposition, and geometric latent modeling: 1. jWAE achieves modal alignment, manifold smoothing, and reduction of cross-modal distribution differences through shared Gaussian priors; 2. LMF uses low-rank decomposition (rank as capacity bottleneck, Hadamard element-wise interaction) to efficiently approximate high-order tensor interactions; 3. Prioritizes interpretability: rank factors provide explicit interaction paths, supporting modal contribution analysis (trading partial accuracy for transparency).

Evaluated on CMU-MOSI, MUSTARD, and Hateful Memes datasets: 1. Rank ablation experiments: Low ranks (r=2-4) yield optimal performance; at r=8, training loss is lowest but generalization decreases (overfitting), showing a non-monotonic relationship between rank and generalization; 2. jWAE vs. ordinary LMF: jWAE improves classification accuracy at low to medium ranks; at high ranks, LMF performance is comparable or better, and jWAE may worsen MAE (trade-off between separability and regression fidelity); 3. Audio dropout experiments: Performance decreases non-monotonically, with 30-50% dropout rate causing the most damage (modal interference exists).

Key conclusions: 1. Low-rank fusion is indeed an implicit spectral regularizer, limiting complexity and learning robust features; 2. Increasing rank does not guarantee performance improvement; there is an optimal range; 3. Geometric conditioning is a double-edged sword (improves classification but may harm regression); 4. The presence of weak modalities negatively affects fusion (modal selection and quality control need attention); 5. Multimodal learning has asymmetry; some modal combinations are more effective.

Research significance: Provides theoretical guidance and practical experience for multimodal AI design, revealing the roles and limitations of low-rank constraints and geometric conditioning. Application prospects: Provides benchmark implementations and experimental data for multimodal learning, tensor decomposition, and robustness research. Project structure: Includes modules such as lmf_module.py (low-rank fusion), jwae_module.py (jWAE), data loaders, and end-to-end training scripts.