Metamorph.ml: A Machine Learning Framework Unifying Data Preprocessing and Hyperparameter Tuning in the Clojure Ecosystem

Metamorph.ml is a machine learning framework for the Clojure language, developed by the SciCloj organization. Its core innovation is unifying data preprocessing and model hyperparameter tuning into a single end-to-end optimization process, breaking the traditional separation of these stages. It supports integration with multiple mainstream ML libraries (Java's Smile/Tribuo, Python's scikit-learn, XGBoost) and leverages Clojure's functional programming features for a more efficient and maintainable workflow.

In traditional ML workflows, data preprocessing and model tuning are often treated as separate stages, leading to fragmented processes and efficiency losses. Metamorph.ml's core insight is that preprocessing decisions (e.g., PCA dimensions, text vocab size) are also uncertain and need systematic optimization. It aims to eliminate the artificial boundary between these stages, allowing joint optimization of preprocessing pipelines and model configurations.

Metamorph.ml's architecture is built on three core concepts:

1. Pipeline: A composable sequence of pure functions (transforms) that process data.
2. Transform: Basic operation unit that adapts to training/inference modes, avoiding code duplication.
3. Context: Carries state (dataset, model, configs) across pipeline steps, maintaining functional purity.

This design ensures pipelines are testable, reusable, and easy to modify.

Metamorph.ml enables optimization of various preprocessing and model parameters:

• Continuous parameters: PCA dimensions, TF-IDF vocab size, cluster counts.
• Discrete choices: Whether to enable a step (e.g., stemming), encoding method (one-hot vs label).
• Nested pipelines: Complex flows with multiple adjustable decision points.

The optimize-hyperparameter function handles cross-validation and returns the best configuration, decoupled from specific model types.

As a meta-framework, Metamorph.ml integrates with multiple ML libraries via plugins:

• Java: Smile (broad model support) and Tribuo (focus on interpretability/production readiness).
• Python: scikit-learn via sklearn-clj bridge.
• XGBoost: Direct support via xgboost4j bindings.

This plugin-based design allows easy expansion of supported models while keeping the core framework stable.

Metamorph.ml leverages Clojure's functional programming features:

• Immutability: Data is not modified in-place, ensuring safety and easier debugging.
• Composability: Build complex pipelines by combining simple transforms.
• Lazy evaluation: Efficient memory usage for large datasets (only needed columns are processed).
• REPL-driven development: Interactive experimentation with real-time feedback.

These features make ML workflows more robust and iterative.

For the Iris classification task, Metamorph.ml simplifies the workflow:

1. Define a pipeline with preprocessing and model training steps.
2. Split data into training/test sets.
3. Use optimize-hyperparameter to find the best configuration.

The declarative code style is concise, readable, and easy to maintain, covering the full ML lifecycle from data prep to prediction.

Metamorph.ml rethinks ML workflow abstraction by unifying preprocessing and tuning. It fills a gap in Clojure's ML ecosystem, offering a workflow aligned with the language's philosophy. While it has limitations (smaller community vs Python, need for extra integration for some deep learning tools), its innovative approach and growing ecosystem (part of SciCloj's data science toolchain) make it a promising tool for Clojure developers and a showcase of functional programming's value in ML.