Panoramic Map of Multimodal Models: The Evolution of Architectures from MLLM to NMM

Based on the Awesome Multimodal Modeling resource list maintained by OpenEnvision, this article systematically organizes the development of multimodal AI, covering four evolutionary stages—traditional multimodal models, multimodal large language models (MLLM), unified multimodal models (UMM), and native multimodal models (NMM)—as well as three core paradigms (MLLM, UMM, NMM). It provides researchers with a clear classification system and architecture comparison, helping to clarify the technical evolution path of the field.

The multimodal AI field has evolved rapidly from early image-text alignment to video understanding, audio generation, and cross-modal reasoning, but this progress is accompanied by conceptual confusion: the definitions and differences between MLLM, UMM, and NMM are unclear, and the technical considerations behind architecture design lack systematic organization. The Awesome Multimodal Modeling resource list emerged to address these issues—it is not just a collection of papers and projects, but also a knowledge graph for multimodal AI.

The resource list divides multimodal models into four stages:

1. Traditional Multimodal Models: Focus on representation learning and modal alignment, task-specific with no unified architecture;
2. MLLMs: Based on pre-trained LLMs, graft visual capabilities via visual adapters (e.g., Q-Former, cross-attention), essentially "language model + visual plug-ins", with limitations of modal asymmetry;
3. UMMs: Unified architecture for all modalities, divided into three generation paradigms: diffusion, autoregressive, and hybrid;
4. NMMs: Native multimodal pre-training, end-to-end unified architecture, divided into early fusion (e.g., Gemini) and late fusion strategies.

Dimension	MLLM	UMM	NMM
Training Cost	Low (frozen LLM)	Medium (multimodal pre-training)	High (native multimodal pre-training)
Modal Symmetry	Low (language-dominant)	High	High
Generation Capability	Limited (mainly text)	Strong (multimodal generation)	Strong (multimodal generation)
Inference Efficiency	High	Medium	Depends on architecture design
Applicable Scenarios	Visual understanding, VQA	Multimodal generation, editing	General-purpose multimodal assistant

The value of this resource list is reflected in:

1. Systematic Classification: A clear framework to help quickly locate research directions;
2. Evolution Timeline: Organized by stages to show technical context, aiding trend understanding;
3. Visualization Support: Architecture diagrams and comparison tables lower the threshold for technical understanding;
4. Continuous Updates: The open-source project follows the latest research to maintain timeliness.

Based on the resource list framework, researchers can refer to the following recommendations:

1. Establish a Panoramic Vision: Understand that the three paradigms are not substitutes but scenario-adapted solutions;
2. Focus on Trade-offs: Weigh design trade-offs such as computational efficiency vs. sufficiency of modal interaction;
3. Track NMM Progress: Native multimodal models are the future direction—follow open-source community dynamics;
4. Combine Practice with Theory: Deepen understanding through experiments with representative models (e.g., LLaVA, Stable Diffusion).