agent-opt: One-stop Solution for AI Agent Workflow Tuning Challenges with Six Prompt Optimization Algorithms

The agent-opt library open-sourced by the Future AGI team integrates six prompt optimization algorithms, supports any LLM and over 50 evaluation metrics, and transforms the trial-and-error process of manual prompt tuning into a systematic automatic optimization workflow, helping solve AI agent workflow tuning challenges.

In LLM-driven agent applications, prompts are crucial. A single prompt can be optimized via manual trial and error, but manual tuning becomes infeasible when there are dozens or hundreds of prompts in production systems; moreover, model updates can lead to 'prompt drift' (silent updates of third-party APIs may cause a sudden drop in agent performance), creating a demand for automated prompt optimization tools.

agent-opt is a Python library open-sourced by the Future AGI team. Its core idea is to automatically find better prompts after selecting optimization algorithms, evaluation metrics, and datasets. It uses the Apache 2.0 license and has nearly 60 stars on GitHub; it supports mainstream LLMs (such as OpenAI, Anthropic, etc.) via LiteLLM, and provides over 50 evaluation metrics in collaboration with the ai-evaluation library.

agent-opt provides six algorithms: 1. Random Search: A basic strategy that generates seed prompt variants for evaluation and establishes a performance baseline; 2. Bayesian Search: Based on Optuna TPE, it optimizes few-shot example selection and ordering; 3. ProTeGi: A text gradient method that analyzes error samples to generate modification feedback and selects the optimal one using beam search; 4. Meta-Prompt: Uses a teacher model to analyze failure cases and rewrite prompts; 5. PromptWizard: A mutation-criticism-refinement pipeline; 6. GEPA: Genetic Evolutionary Pareto Algorithm that finds multi-objective Pareto optimal solutions.

The agent-opt architecture revolves around three decoupled abstractions: 1. Generator: Executes prompts to get responses; LiteLLMGenerator supports multiple models and can be customized; 2. Evaluator: Outputs scores, supports rule-based heuristics (BLEU, etc.), LLM-as-Judge, and over 50 pre-built templates, and can be customized; 3. Data Mapper: Maps dataset fields to evaluator inputs, reducing adaptation costs.

agent-opt is the optimization link in the Future AGI open-source platform, forming a closed loop with other components: traceAI captures LLM call trace data → ai-evaluation scores → agent-opt converts to better prompts → Agent Command Center deploys to OpenAI-compatible endpoints. The loop design enables continuous automated optimization, and all components are independently open-sourced (Apache 2.0).

Applicable scenarios: Production systems managing a large number of prompts, teams that frequently switch/update models, enterprise applications with quantitative output quality requirements, teams hoping to turn prompt engineering into engineering practice. Limitations: Optimization effectiveness depends on the selection of evaluation metrics (inaccurate metrics lead to poor results); optimization iterations require calling LLM APIs, so large datasets plus high trial counts will incur costs.

agent-opt represents an important direction for prompt engineering from manual trial and error to systematic optimization. It encapsulates multiple algorithms into a unified API to lower technical barriers, making it suitable for teams building/maintaining LLM-driven applications—especially worth trying now when agent workflows are complex and prompt management costs are rising.