Chimère ODO: A Unified Inference Orchestrator for Local LLMs with Adaptive Computing and Self-Improvement

Chimère ODO is a Python-based inference orchestration layer for local LLMs, positioned between user requests and inference servers. It provides intent classification, context enhancement, adaptive computing routing, quality assessment, and a self-improvement loop, and collaborates with an 8-step SOTA search pipeline to enable intelligent interactions.

Chimère ODO is an intelligent orchestration layer in the Chimère ecosystem, acting as an "intelligent intermediary between user requests and inference servers". It runs on the Python layer, listens on port 8084, and after receiving a user query, executes a series of preprocessing, routing decision-making, and postprocessing steps, finally forwarding the optimized request to the underlying chimere-server (Rust inference runtime).

This layered architectural design reflects a core trend in modern AI systems: separating "fast thinking" (inference execution) from "slow thinking" (orchestration decisions). The Rust layer focuses on extreme inference performance, while the Python layer handles flexible intent understanding, context management, and quality optimization. The two work together to deliver both fast and intelligent user experiences.

ODO executes a standardized five-stage process for each user request:

First Stage: Intent Classification . The system uses a three-level cascade strategy to identify user intent: first, quickly match common patterns via regular expressions, then make judgments based on file types, and finally call a local LLM for deep semantic analysis. This progressive strategy balances accuracy and efficiency.

Second Stage: Context Enhancement . Supplement the original query with relevant background information through ChromaDB vector retrieval, web search, tool injection, and SOUL.md integration. This stage transforms isolated user questions into information-rich structured requests.

Third Stage: Adaptive Routing . Dynamically select computing configurations based on entropy calculation, deciding whether to use deep thinking mode (think) or fast response mode (no-think). The system also selects appropriate routing configuration files based on query characteristics.

Fourth Stage: Forward Execution . Send the processed request to chimere-server, where the Rust runtime performs actual model inference.

Fifth Stage: Quality Assessment and Feedback . Score the model output, record training sample pairs, and provide data for the nightly self-improvement loop.

ODO's intent classification mechanism embodies engineering pragmatism: it does not blindly pursue the perfection of a single technology, but combines multiple methods to achieve reliable overall performance.

First Level: Regular Expression Matching . For high-frequency, fixed-pattern query types (e.g., code generation requests, file operation instructions), use precompiled regular expressions for millisecond-level identification.

Second Level: File Type Inference . Infer intent based on file extensions, MIME types, or path features involved in the request. For example, operations involving .py files are likely related to Python programming.

Third Level: LLM Semantic Analysis . When the first two levels cannot determine intent, call a lightweight local model for deep semantic understanding. Although this level has higher costs, it can handle complex and ambiguous queries.

This cascade design ensures most simple queries can be classified in a very short time, and only truly complex queries trigger expensive LLM calls.

ODO's context enhancement module integrates multiple information sources:

ChromaDB RAG Retrieval : Retrieve historically relevant documents, code snippets, or knowledge entries from the local vector database that are semantically related to the user query.

Web Search Integration : When local knowledge is insufficient to answer a question, automatically trigger web search. ODO implements an 8-step SOTA search pipeline: query expansion → parallel retrieval (ChromaDB + web) → reciprocal rank fusion → deep fetching → diversity handling → contrastive retrieval-augmented generation → contradiction detection → comprehensive synthesis.

Tool Injection : Dynamically inject relevant MCP tool descriptions based on identified intent, enabling the model to understand available external capabilities.

SOUL.md Integration : Read the user's or project's SOUL.md file and inject personalized background information, preference settings, and context memory.

These enhancement methods transform the user's short query into a structured prompt with rich context, significantly improving the relevance and accuracy of model outputs.

One of ODO's most innovative features is adaptive computing routing. The system dynamically adjusts computing resource configurations based on the "cognitive complexity" of the query:

Entropy Evaluation : Analyze the information entropy of the query to identify uncertainty, ambiguity, and parts requiring deep reasoning. High-entropy queries (e.g., open-ended creative tasks, complex problem-solving) require more computing resources.

Think vs No-Think Mode : For low-entropy simple queries, the system selects no-think mode, using faster sampling strategies and shorter chains of thought; for high-entropy complex queries, it enables think mode, allowing the model to perform deeper step-by-step reasoning.

Configuration File Routing : ODO predefines routing configurations for different scenarios (code, kine, cyber, research, default, vision, doc_qa, general), each specifying specific pipeline parameters, tool sets, and model behaviors.

This dynamic resource allocation strategy ensures computing power is used where it is most needed, avoiding resource waste on simple queries while providing sufficient reasoning depth for complex tasks.

ODO implements a complete quality feedback loop:

Output Scoring : Perform multi-dimensional scoring on each inference result, evaluating accuracy, completeness, relevance, and usefulness.

Training Pair Recording : Record (enhanced input, scored output) as training samples and store them in the local dataset.

Nightly LoRA Fine-Tuning : During idle system periods (nighttime), use accumulated training data to perform LoRA fine-tuning on the base model, gradually improving its performance in specific domains and tasks.

DSPy Optimization : Combine the DSPy framework for automatic optimization of prompts and pipeline parameters, continuously improving the system's overall performance.

Engram Memory Integration : Integrate high-quality interaction samples into the Engram memory system for subsequent semantic few-shot learning and n-gram log bias.

This self-improvement mechanism allows ODO to adapt to the user's specific needs and preferences over time, achieving a truly personalized AI assistant experience.

ODO's web search capability is not just a simple API call, but a complex multi-stage pipeline:

1. Query Expansion : Expand the user's short query into multiple related search terms, covering different expressions and related concepts.

2. Parallel Retrieval : Query the local ChromaDB and external web search engines simultaneously to obtain multi-source information.

3. Reciprocal Rank Fusion (RRF) : Use the RRF algorithm to merge search results from multiple sources and generate a unified ranking list.

4. Deep Fetching : Perform deep content crawling on top-ranked results instead of relying only on summaries.

5. Diversity Handling : Ensure the result set covers different aspects of the query to avoid information cocoons.

6. Contrastive Retrieval-Augmented Generation (CRAG) : Identify key information fragments in the retrieval results to guide the generation process.

7. Contradiction Detection : Analyze information from multiple sources to identify and mark conflicting statements.

8. Comprehensive Synthesis : Integrate all processed information into a coherent and accurate context summary.

This 8-step pipeline ensures the comprehensiveness, accuracy, and usability of web search results, providing high-quality external knowledge injection for the model.