IDMVAE: Implementation of Information-Disentangled Multimodal Variational Autoencoder

IDMVAE is the official PyTorch implementation of the ICLR 2026 paper Disentanglement of Variations with Multimodal Generative Modeling, focusing on disentangling variations via multimodal generative modeling. This project supports multimodal datasets including PolyMNIST, CUB-200-2011, CelebAMask-HQ, and TCGA, and provides training and evaluation code. It aims to solve the problem of entangled variation factors in multimodal data, enhancing model interpretability and controllability.

Multimodal learning is an important direction in artificial intelligence, but variation factors in multimodal data are often entangled, posing challenges to model interpretability and controllability. Unimodal VAEs have demonstrated the ability to learn disentangled representations, but extending this to multimodal scenarios remains an open problem. IDMVAE addresses this issue using information-theoretic guided methods to achieve variation disentanglement in multimodal generative modeling.

Core Concepts: Multimodal VAEs need to learn a shared latent space (capturing cross-modal common information + preserving modality-specific information); the goal of disentangled representation learning is to make latent variables correspond to independent variation factors.

Technical Design: The architecture includes multimodal encoders/decoders, with the latent space divided into shared variables and modality-specific variables; the training objective combines VAE loss (reconstruction loss + KL divergence) with information-theoretic regularization terms to maximize shared information while reducing redundancy in modality-specific information.

Supported Datasets: PolyMNIST (multimodal variant of MNIST), CUB-200-2011 (bird images + text descriptions), CelebAMask-HQ (face images + segmentation masks), TCGA (multimodal medical data for cancer).

Code Structure: The src/ directory contains core code (model definitions, training scripts, data loaders), src/commands/ contains experiment scripts, and src/baseline/ contains baseline reference implementations.

Usage Instructions: Dependencies are managed using pip-tools. Data preparation scripts (e.g., PolyMNIST generation, format conversion) are provided. Each dataset has corresponding training/evaluation scripts and supports multiple running modes.

Experiment Reproduction: Set environment variables pointing to the dataset path, then run the corresponding shell scripts under src/ (which automatically handle initialization, training, and checkpoint saving). Weights & Biases experiment tracking is supported.

Academic Contributions: The paper was accepted by ICLR 2026. It introduces an information disentanglement mechanism based on baselines like MMVAEplus and MMVAE; the open-source implementation facilitates reproduction, comparative research, and domain development.

Practical Applications: Controllable content generation (independent control of attributes), cross-modal retrieval (text-to-image search), data augmentation (synthetic data), medical image analysis (application on the TCGA dataset).

Future Directions: Extend to more modalities and datasets, improve disentanglement evaluation metrics, integrate new technologies like diffusion models, and apply to a wider range of scenarios.