Harmonia-LM: A Complete Training Pipeline for Generating MIDI Music Using Large Language Models

Harmonia-LM is an end-to-end MIDI music generation system based on PyTorch Lightning, supporting any Hugging Face CausalLM model, demonstrating the innovative application of LLMs in the field of music creation. This post will cover its background, architecture, key innovations, implementation details, application scenarios, limitations, and future directions.

Large Language Models (LLMs) have excelled in text generation, code writing, and other fields, but their application to music generation remains challenging. Traditional music generation models require specialized architectures, while Harmonia-LM proves that standard CausalLM models can handle music creation by converting MIDI data into token sequences. This project originated from a TIPE research topic in a French preparatory class, being a creative project where students combined academic requirements with cutting-edge technology.

Harmonia-LM adopts a modular three-stage architecture:

1. Data Preprocessing Layer: Uses the Miditok library to convert MIDI files into token sequences, supporting encoding strategies like REMI and CPWord while preserving musical structure information.
2. Data Chunking and Batching: An intelligent chunking mechanism splits long sequences into segments suitable for the model's context window, balancing the capture of long-range dependencies and training efficiency.
3. Training and Inference Engine: Built on PyTorch Lightning, supporting distributed training, automatic mixed precision, etc., and seamlessly integrated with Hugging Face Transformers, allowing switching between models like DistilGPT-2 and GPT-Neo.

The core innovations of Harmonia-LM include:

1. Cross-modal Unified Perspective: It considers music and text to be essentially similar at the sequence modeling level. By mapping MIDI events to tokens, it transfers the autoregressive generation capability of LLMs to the music domain.
2. Model-agnostic Flexibility: Supports plug-and-play models, enabling quick comparison of models of different scales, leveraging the latest pre-training advances, selecting model sizes based on hardware, and exploring different architectures (e.g., Mamba).
3. End-to-end Completeness: Provides a complete toolchain from raw MIDI to playable output, suitable for educational scenarios and rapid prototyping.

Tokenization Strategy Selection

Miditok supports multiple schemes: REMI (based on absolute time positions, suitable for beat structures), CPWord (compound tokens reduce sequence length), Octuple (octuple with high information density). The default configuration balances information retention and sequence length.

Training Strategy and Optimization

PyTorch Lightning brings advantages such as automatic device management, distributed training (DDP), checkpoint management, and log integration (TensorBoard, W&B).

Controllability of Inference Generation

Implements autoregressive sampling strategies like temperature sampling (controls randomness), Top-k/Top-p filtering (balances diversity and quality), and repetition penalty (avoids monotony).

Education Field

As a product of the TIPE project, it is suitable for teaching: helping understand sequence modeling, demonstrating the application of theory to creative tasks, and providing a reference structure for deep learning projects.

Music Creation Assistance

Can be used as an inspiration generator, style transfer tool, or arrangement assistant (generating accompaniment tracks).

Academic Research

Used to explore differences between different architectures in music tasks, study formal representations of music theory, and develop new evaluation metrics.

Current Limitations

1. Long-range structure preservation: May lose overall structure when generating long music pieces; 2. Multi-track coordination: Complex multi-instrument arrangements are challenging; 3. Music theory constraints: May generate sequences that do not conform to harmonic rules.

Future Improvement Directions

Introduce hierarchical generation strategies, combine diffusion model post-processing, integrate music theory-constrained decoding, and explore multi-modal extensions (text-to-music generation).

Harmonia-LM demonstrates the cross-domain potential of LLMs in the creative field. By transforming music generation into a sequence modeling problem, it lowers the entry barrier for the domain and lays the foundation for complex music AI systems. The project is open-source with complete documentation, and the community can continue to iterate to promote the development of music generation technology.