MCore-Bridge: A Model Definition Library to Make Megatron-Core Training as Simple as Transformers

MCore-Bridge, launched by the ModelScope community, provides Megatron-Core model definitions for over 300 large language models (LLMs) and 200+ multimodal large models (MLLMs). It supports LoRA and full-parameter training, is compatible with the PEFT ecosystem, bridges HuggingFace to Megatron-Core, and simplifies distributed large model training to make it efficient.

With the explosive growth of LLM and MLLM scales, training has extremely high requirements for engineering infrastructure. NVIDIA's Megatron-Core offers advanced parallel strategies like tensor parallelism and pipeline parallelism, which can efficiently utilize the computing power of multi-GPU clusters. However, its usage threshold is high: developers need to manually write complex model definition code, handle low-level details such as weight loading and distributed communication, and repeated wheel reinvention slows down research iteration speed.

MCore-Bridge is developed and maintained by the ModelScope community, released on March 30, 2026. It aims to solve the pain points of using Megatron-Core, providing out-of-the-box Megatron-Core model definitions. Its core goal is to make Megatron training as simple as Transformers, and it is a complete engineering solution.

Extensive Model Coverage

Supports over 300 pure-text LLMs (such as Qwen series, DeepSeek series, GLM series, etc.) and 200+ multimodal models (such as Qwen multimodal, Gemma4, GLM-4V, etc.).

Comprehensive Hardware Compatibility

Supports NVIDIA GPUs (A10/A100/H100/B200, etc.), domestic Ascend NPUs, compatible with CUDA 12.8/13.0 and PyTorch 2.0+.

Flexible Parallel Strategies

Inherits Megatron-Core's capabilities like tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, expert parallelism, virtual pipeline parallelism, etc.

Training Modes and Ecosystem Compatibility

Supports full-parameter training and LoRA training, fully compatible with the HuggingFace PEFT ecosystem, supports the safetensors weight format, and can seamlessly interface with inference frameworks like Transformers and vLLM.

Optimized for multimodal model requirements:

• FP8 training support: Uses NVIDIA Hopper architecture FP8 precision to accelerate training and improve throughput
• MTP (Multi-Token Prediction): Enhances model inference efficiency
• No sequence padding: Eliminates memory waste from sequence alignment within batches
• Packing feature: Packs multiple short sequences to improve GPU utilization

Basic Model Loading and Saving

Initialize the distributed environment via code, download the model, convert the configuration, create the model, and load/save weights (example code omitted).

LoRA Fine-Tuning Example

Integrate with PEFT, define LoRA configuration, wrap the model, and save LoRA weights (example code omitted).

Deep Integration with ms-swift

Combined with ModelScope's ms-swift training framework, it retains ease of use while gaining distributed training performance, supporting multiple task types.

Dependency Requirements

Component	Minimum Version	Recommended Version
Python	>=3.10	3.12
PyTorch	>=2.0	2.8.0/2.11.0
megatron-core	>=0.15,<0.18	0.17.0

Installation Methods

• pip installation: pip install mcore-bridge -U
• uv accelerated installation: uv pip install mcore-bridge -U --torch-backend=auto
• Source code installation: Clone the repository and run pip install -e .

MCore-Bridge combines the high performance of Megatron-Core with the ease of use of the Transformers ecosystem, allowing developers to focus on model innovation. Its extensive model support, hardware compatibility, and ecosystem integration make it a production-ready solution. In the future, it will continue to support new models (Day0 strategy) and play an important role in the field of large model infrastructure.