OpenArc: A Local AI Inference Engine Built Exclusively for Intel Devices, One-Stop Support for Multimodal Models

OpenArc is an open-source inference engine based on OpenVINO, designed exclusively for Intel devices. It supports local private deployment of multimodal models such as LLM, VLM, speech processing, Embedding, and Reranker, and provides OpenAI-compatible API endpoints. It aims to solve the problem of insufficient AI toolchains for Intel device users, keeping data local while balancing performance and privacy.

In the AI inference field, NVIDIA GPUs have long dominated, leaving Intel device users facing the dilemma of insufficient toolchains. OpenArc emerged as a solution, built on OpenVINO, focusing on Intel devices to enable local private deployment of various AI models and provide services via OpenAI-compatible APIs, filling the gap in local AI deployment for the Intel ecosystem.

OpenArc covers mainstream AI scenarios:

• LLM: Supports text generation/conversation completion (compatible with OpenAI /v1/completions//v1/chat/completions endpoints). The latest version introduces speculative decoding to improve inference speed;
• VLM: Processes mixed image-text inputs to enable image understanding and generation;
• Speech Processing: ASR supports Whisper/Qwen3-ASR (/v1/audio/transcriptions), TTS integrates Kokoro-TTS/Qwen3-TTS (/v1/audio/speech);
• Text Embedding and Reranker: Supports Qwen3 models, providing a foundation for RAG (/v1/embeddings//v1/rerank endpoints).

• Multi-Device Support: Compatible with Intel CPU, GPU (multi-GPU parallelism), NPU, and supports CPU/GPU hybrid offloading to balance resources;
• Asynchronous Multi-Engine Architecture: Concurrent model loading/inference, streaming response/cancellation, automatic unloading on failure, OpenAI-compatible tool calls (streaming/parallel);
• Performance Monitoring: Records metrics such as TTFT, prefill throughput, decode throughput, TPOT, and model loading time. Built-in llama-bench style benchmarking and storage to SQLite.

• Local Installation: Quickly set up on Linux/Windows systems via the uv toolchain, supporting nightly wheels to install the latest OpenVINO and OpenVINO GenAI;
• Docker Containerization: Provides out-of-the-box configurations, supports environment variables for custom model paths, API keys, automatic model loading, etc., facilitating production deployment.

OpenArc draws on the concepts of open-source projects such as llama.cpp, vLLM, Transformers, and OpenVINO Model Server, with deep optimizations for Intel devices. It has an active Discord community, providing a communication platform for Intel AI users.

For users of Intel devices (such as Arc graphics cards, Core Ultra NPU), OpenArc fills a key gap in local AI deployment, compatible with OpenAI API to reduce migration costs, and keeps data local to meet privacy compliance. As Intel's next-generation hardware and the OpenVINO ecosystem mature, it is expected to become an important infrastructure for AI inference on the Intel platform.