Hikyaku: The Super Agent and Intelligent Load Balancer for AI Inference

Hikyaku is an AI inference proxy and intelligent load balancer written in Go, supporting model virtualization, hybrid local and cloud backends, optimal caching, sampling parameter locking, message flow debugging, and OpenTelemetry metrics collection.

With the popularity of Large Language Models (LLMs), enterprises and developers face complex inference deployment challenges. On one hand, local deployment offers advantages in data privacy and cost control; on the other hand, cloud APIs (such as OpenAI, Anthropic) provide out-of-the-box convenience. How to flexibly switch between the two, how to optimize latency and cost, how to unify monitoring and debugging—these issues have spurred the demand for an intelligent proxy layer.

Hikyaku came into being. This is an open-source project written in Go, positioned as an "AI inference super agent and intelligent load balancer". It is not just a simple reverse proxy, but a feature-rich inference orchestration layer.

Hikyaku's design goal is very clear: to provide a unified entry point for AI inference workloads while solving the following key problems:

Hikyaku allows users to define virtual model names and map them to different backend providers. For example, you can define a virtual model named gpt-smart, which may actually route to OpenAI's GPT-4, a local Llama model, or other providers compatible with the OpenAI API based on configuration. This abstraction layer makes switching model providers extremely simple—just modify the configuration without changing application code.

Hikyaku supports configuring multiple backends simultaneously, including:

• Local Backends: Local models run via tools like Ollama, llama.cpp, vLLM
• Cloud Backends: Commercial APIs such as OpenAI, Anthropic, Azure OpenAI
• Hybrid Strategy: Intelligently select backends based on request characteristics, cost, latency, and other factors

This hybrid architecture enables enterprises to use local models in data-sensitive scenarios and cloud models in performance-critical scenarios, achieving the best balance between cost and performance.

Hikyaku has a built-in intelligent caching system that can cache responses to identical requests. For scenarios requiring deterministic outputs (such as code generation, structured data extraction), caching can significantly reduce costs and latency. The caching strategy supports classic algorithms like TTL (Time-to-Live) and LRU (Least Recently Used), and can be configured with fine granularity based on model and request characteristics.

In actual production environments, application developers may pass various sampling parameters (temperature, top_p, max_tokens, etc.), but these parameters may not be suitable for specific models or business scenarios. Hikyaku allows administrators to lock or override these parameters at the proxy layer, ensuring that downstream models always receive optimized parameter combinations. This is crucial for maintaining output quality and consistency.

One of the biggest challenges in debugging AI applications is understanding the complete request-response flow. Hikyaku provides detailed message flow logs that record the full lifecycle of each request: reception time, routing decision, backend selection, response time, token usage, etc. These logs are extremely valuable for performance optimization, troubleshooting, and cost analysis.