llm-d-batch-gateway: Open Source Batch Inference Gateway Implementing OpenAI Batch API

llm-d-batch-gateway is an open-source batch inference gateway that fully implements the OpenAI Batch Inference API, enabling developers to efficiently handle large-scale LLM inference tasks at lower costs.

• Original Author/Maintainer: llm-d
• Source Platform: github
• Original Title: llm-d-batch-gateway
• Original Link: https://github.com/llm-d/llm-d-batch-gateway
• Source Release Time/Update Time: 2026-06-10T20:12:35Z

In large-scale language model (LLM) application scenarios, batch inference is a common but costly requirement. Whether for data annotation, content generation, text analysis, or model evaluation, developers often need to process thousands of requests. OpenAI's batch inference API offers significant cost advantages—usually 50% cheaper than real-time APIs—but requires a compatible gateway to handle request queues, state management, and result callbacks.

llm-d-batch-gateway is an open-source project born to address this pain point. As part of the llm-d ecosystem, it provides a complete OpenAI Batch Inference API-compatible implementation, allowing developers to deploy batch processing services on their own infrastructure, enjoying cost advantages while maintaining data privacy and control.

The project implements the full specification of the OpenAI Batch Inference API, including:

• Batch Task Creation: Supports uploading JSONL-formatted request files, each containing up to 50,000 requests
• Task Status Management: Provides full lifecycle management, including states like validation, queued, processing, completed, and failed
• Result Retrieval: Supports downloading processing results via the file interface, including responses and metadata of original requests
• Cancellation & Error Handling: Allows canceling ongoing tasks and provides detailed error information and retry mechanisms

llm-d-batch-gateway adopts an asynchronous architecture design to efficiently handle large numbers of concurrent requests:

• Request Queue: Uses a persistent queue to store pending batch tasks, ensuring no task loss after system restart
• Worker Pool: Configurable adjustable worker pool that dynamically adjusts concurrency based on the capacity of backend LLM services
• Flow Control & Rate Limiting: Built-in rate limiting mechanism to prevent backend service overload, while supporting priority queues
• Resume on Interruption: Supports recovery mechanisms after task interruption to avoid reprocessing completed requests

As a component of the llm-d ecosystem, this gateway natively supports multiple LLM backends:

• OpenAI API: Directly connects to OpenAI's batch inference endpoints
• Compatible APIs: Supports any service implementing OpenAI-compatible interfaces, such as vLLM, TGI, Ollama, etc.
• Local Models: Configurable to use locally deployed open-source models for fully offline batch processing
• Hybrid Routing: Supports intelligent routing to different backends based on model type, cost, or latency requirements

llm-d-batch-gateway supports flexible deployment options:

Independent Deployment: Runs as an independent service, receiving batch tasks via REST API—suitable for teams with existing LLM infrastructure

Kubernetes Integration: Provides Helm Chart and Operator support, enabling elastic scaling in K8s clusters to handle large-scale batch processing loads

Edge Deployment: Lightweight configuration supports running on edge devices, ideal for scenarios with high data privacy requirements

Large-Scale Data Annotation: When needing to generate labels, classifications, or summaries for massive text data, the batch API can significantly reduce costs. For example, performing sentiment analysis or topic classification on millions of customer reviews.

Content Generation Workflows: When marketing teams need to generate large numbers of variant copy, product descriptions, or social media posts, they can submit templated requests in batches to obtain high-quality generated content with controlled costs.

Model Evaluation & Benchmarking: When researchers need to evaluate model performance on large test sets, batch processing can handle thousands of test cases in parallel, greatly shortening the evaluation cycle.

Historical Data Processing: When enterprises need to vectorize, summarize, or perform entity recognition on archived documents, batch processing is the most efficient choice.