EcoPrompt: An Energy-Efficient and High-Performance AI Prompt Routing System

EcoPrompt is an open-source layered AI prompt routing system. Its core idea is to intelligently assess query complexity: simple questions are assigned to low-cost local engines, while complex reasoning tasks are handled by large models, thus significantly reducing latency, costs, and energy consumption. The project is maintained by K Jayarama Das, and the source code and demos are available on GitHub.

Most current AI applications adopt a "one-size-fits-all" strategy, calling large models (e.g., GPT-4) regardless of query complexity, leading to resource waste. Specific issues include: rising costs (accumulated API fees), increased latency (slow response from large models), excessive energy consumption (unnecessary computations), and resource misallocation (simple queries occupying resources for complex tasks).

EcoPrompt uses a six-level layered routing (sorted from lowest to highest cost): 1. Rule/lookup engine (deterministic tasks); 2. Local knowledge base + lightweight RAG; 3. Code template responder; 4. Groq Llama3.1 8B; 5. Groq Llama3 70B; 6. Gemini web search. It also has an intelligent upgrade mechanism: prompt complexity scoring → quality check of low-cost answers (entity coverage, weak answer detection) → automatic level upgrade if not up to standard, balancing cost and quality.

Backend: Python + FastAPI + Uvicorn; model services use Groq (Llama3.1/70B); search uses Gemini web search; local engines include custom rules and RAG retrieval. Frontend: React + Vite + Tailwind; visualization uses Recharts; rendering uses react-markdown. Knowledge Base: The kb directory contains modules for geography, mathematics, science, etc., supporting semantic retrieval via rag_engine.py.

In sample tests, 96% of traffic was handled by local layers without needing paid cloud LLMs. Cost comparison: GPT-4o is about $4 per million tokens, while Groq Llama3 70B is about $0.7 per million tokens. Energy consumption is estimated based on latency × assumed power consumption (not hardware-measured). For transparency, the data is used to show relative savings, with the core metric being the cloud call avoidance rate.

Local Deployment: Backend requires installing dependencies, configuring .env (filling in API keys), and starting Uvicorn; frontend requires npm dependency installation and starting dev mode. API Endpoints: POST /generate (route prompts), POST /generate-stream (streaming return), GET /metrics (metrics). Testing: Offline unit tests cover routing logic, token management, energy consumption estimation, etc. No API calls ensure fast and reliable CI.

Significance: Proposes the concept of intelligent resource allocation, enlightening developers: not all queries need large models, quality checks are key, and transparency is important. Industry impact: Layered routing + quality checks + energy consumption tracking may become standard practices. Future: Support more pluggable model backends, configurable routing strategies, and per-user energy consumption cost reports.

EcoPrompt is a well-designed open-source project that solves the cost, latency, and energy consumption issues of AI applications through intelligent routing, providing AI application developers with ready-to-use reference implementations and architectural ideas.