Datamatic: A Structured Generation Tool for Orchestrating Multi-Step AI Workflows with YAML

Datamatic is a YAML-configured command-line tool for AI workflow orchestration. It supports multiple model backends (local ones like Ollama, cloud ones like OpenAI), JSON Schema structured output, step chaining, dataset loading, and CLI integration. It is suitable for scenarios like synthetic data generation and document classification. Its core value lies in lowering the barrier to structured generation and enabling complex workflow orchestration.

In large language model application development, the pain point is how to achieve reproducible and orchestratable multi-step processing workflows while ensuring output quality. Datamatic was designed to address this: it abstracts complex workflows into concise YAML configurations, allowing developers to focus on business logic rather than low-level API calls. Its core advantages are structured generation (via JSON Schema constraints) and step chaining (where the output of the previous step serves as the input for the next).

1. Multi-model Support: The same configuration can seamlessly switch between local (Ollama, LM Studio) and cloud (OpenAI, Gemini) models with just one line of configuration change;
2. JSON Schema Structured Output: Built-in validation ensures the model output complies with predefined formats (e.g., sentiment analysis needs to return sentiment and confidence);
3. Step Chaining: Steps are connected via template variables (e.g., generating a summary after extracting entities), with a declarative design that is intuitive and easy to version control;
4. Dataset and CLI Integration: Supports loading HuggingFace datasets for batch processing, integrates jq for data transformation, and can use any CLI tool as a workflow step.

1. Synthetic Data Generation: Batch generate training samples that comply with Schema (e.g., news title + sentiment label + clickbait score);
2. Document Classification and Analysis: Multi-step workflows (extract key info → classify → generate summary), using the most suitable model for each step;
3. SQL Query Generation: Combine Chain-of-Thought reasoning to generate executable SQL, with Schema constraints ensuring output correctness;
4. Multimodal Workflows: Supports image analysis steps to build text-image hybrid processing workflows.

The output uses JSON Lines format, where each line contains the complete execution context (id, prompt, response, values from previous steps, etc.), providing end-to-end traceability. Developers can clearly see the generation process of the output and its dependencies.

Installation Methods:

• Homebrew: brew tap mirpo/homebrew-tools && brew install datamatic;
• Go installation: go install github.com/mirpo/datamatic@latest;
• Source code compilation: Clone the repository and run make build. Basic Workflow: Write YAML configuration → Set environment variables (e.g., OPENAI_API_KEY) → Run datamatic -config config.yaml → Check results in the dataset/ directory. Dynamic Configuration: Supports environment variable injection (e.g., PROVIDER=ollama MODEL=llama3.2 datamatic -config config.yaml), suitable for multi-environment deployment.

Datamatic fills the gap between simple API calls and heavyweight MLOps platforms, providing a lightweight, declarative AI workflow orchestration solution. Its core values include lowering the barrier to structured generation (built-in JSON Schema), supporting complex workflows (step chaining), and a model-agnostic design (seamless switching between local/cloud). As LLM capabilities improve, efficient orchestration will become a key competitive advantage, and Datamatic is an elegant solution worth trying.