Arcadium: A Training Framework and Visualization Toolset for Large Language Models

Arcadium is a deep learning framework designed specifically for large language model (LLM) training. Its core features include a modular training ecosystem, support for ablation experiments, custom CUDA/Triton kernels, a configuration management system, rich visualization tools, and paper reproduction capabilities. It aims to improve the efficiency and reproducibility of LLM research and development.

In the wave of LLM research and development, an efficient and reproducible training framework is an essential need. As an emerging framework, Arcadium is not just a collection of simple training scripts but a complete modular training ecosystem. It calls itself "just another deep learning training framework" but actually has rich features and focuses on LLM training scenarios.

Modular Code Structure

Adopts a clear modular design, facilitating feature expansion, team collaboration, and code reuse testing.

Ablation Experiment Support

Includes the ablations/ directory and scripts, supporting comparative experiments on attention mechanisms, positional encoding, normalization layers, activation functions, etc., to help evaluate component performance.

Custom Kernels

The kernels/ directory provides custom CUDA/Triton kernels for fused operations, optimized attention computation (e.g., FlashAttention), etc., which can increase training speed by 20-50%.

Configuration Management System

The configs/ directory uses a configuration-driven approach, supporting version control of experiment configurations, hyperparameter grid search, and configuration inheritance for models of different scales.

Visualization Tools

Supports tracking of training metrics (loss curves, learning rates, etc.), attention visualization, activation distribution monitoring, and resource usage monitoring (GPU utilization, etc.), helping with training state monitoring and problem diagnosis.

Paper Reproduction Capabilities

Provides benchmark implementations, supporting result verification, technical learning, rapid experiment expansion, and fair method comparison, which is of significant value to the academic community.

Technology Stack

Mainly uses Python, with the uv package manager, including configuration files such as pyproject.toml and requirements.txt.

Application Scenarios

• Academic research: Reproduce papers, verify hypotheses through ablation experiments
• Industrial applications: Domain model pre-training, instruction fine-tuning
• Education and training: Learn LLM training principles and engineering practices

Comparison with Other Frameworks

Feature	Arcadium	Hugging Face Transformers	Megatron-LM	DeepSpeed
Focus area	Research + Visualization	General + Easy to use	Ultra-large-scale training	Training optimization
Ablation experiments	Built-in support	Need manual implementation	Need manual implementation	Need manual implementation
Visualization	Emphasized	Basic	Basic	Basic
Custom kernels	Yes	Limited	Yes	Yes
Paper reproduction	Emphasized	Community-driven	Little official support	Little official support

Limitations

• Documentation completeness needs improvement
• Small community size
• Production readiness needs evaluation
• Requires multi-GPU hardware environment support

Arcadium provides efficient tools for the LLM research community through modular design, support for ablation experiments, custom kernels, and visualization tools. Although it calls itself an ordinary framework, its emphasis on visualization and paper reproduction gives it a unique position. As LLM research deepens, such frameworks that focus on reproducibility and efficiency will play a more important role and are worth the attention of researchers and engineers.