Lance: A Unified Multimodal Model with 3 Billion Parameters, Integrating Understanding, Generation, and Editing

Lance, an open-source model by ByteDance Research, unifies image understanding, generation, editing, and video generation with only 3 billion active parameters. This model challenges the inherent "bigger is better" perception in the multimodal field and provides new ideas for the inclusive application of multimodal AI, which is worth attention.

The current mainstream trend for large multimodal models (LMMs) is "bigger is better", with parameter counts often reaching billions or even hundreds of billions, leading to high training costs and huge inference resource requirements. The Lance project takes a different path: it unifies multiple tasks with 3 billion active parameters, providing new possibilities for resource-constrained scenarios.

Lance adopts a "natively unified" architecture, different from the scheme of simply concatenating visual encoders and language models. Its core features include: 1. Phased multi-task collaborative training to establish deep cross-modal associations; 2. Efficient parameter utilization, allowing inference to run on a single A100 GPU (40GB); 3. End-to-end workflow, where a single model handles the complete process from understanding to generation.

Lance supports four key scenarios:

1. Text-to-Video Generation: Generate 480p/12fps videos based on text descriptions, maintaining temporal coherence and visual quality;
2. Video Editing: Modify existing videos according to instructions (e.g., scene transitions, adding objects) while preserving temporal consistency;
3. Multi-round Consistent Editing: Avoid content "drift" during multiple iterations, suitable for creative scenarios requiring repeated adjustments;
4. Intelligent Video Generation: Generate style-consistent videos based on reference images, or generate subsequent frames from existing content.

Lance is positioned as a research project with a restrained training scale (up to 128 A100 GPUs), supporting 768x768 image generation and 480p/12fps video generation. The inference code and weights have been open-sourced (GitHub, Hugging Face), and a Gradio interface and online demo are provided. The team welcomes community feedback to optimize the model.

Lance has been officially supported by the vLLM-Omni high-performance inference framework, allowing users to enjoy more efficient inference acceleration and flexible deployment options. This integration reflects Lance's recognition in the community, and its architecture and interfaces align with industry consensus.

The emergence of Lance prompts the industry to rethink the relationship between model scale and practical value. In real-world applications, deployment cost, response speed, and accessibility are often more important than absolute performance. A 3-billion-parameter model can run on a single card, making it more practically valuable than 100-billion-parameter models, providing a new option for resource-constrained researchers and developers.

Lance represents an important exploration direction in the multimodal AI field—reducing resource thresholds while maintaining capabilities. For developers limited by computing resources, Lance is a worthy option to pay attention to. With community contributions and optimizations, this lightweight model is expected to show greater potential.