STREAM: A Three-Tier LLM Inference Architecture Unifying Local, HPC, and Cloud Environments

STREAM is a three-tier architecture system addressing the resource fragmentation issue in LLM inference. It实现s unified scheduling of local, high-performance computing (HPC) center, and commercial cloud API resources via intelligent hierarchical routing and a dual-channel HPC streaming architecture. Its core value lies in reducing the first-token latency of HPC inference from 11.4 seconds to 0.54 seconds while ensuring data privacy, striking an optimal balance between cost, performance, and privacy.

Current LLM users face a triple dilemma:

• Local Deployment: Free and private, but hardware limitations prevent running large models or long contexts;
• Institutional HPC: Strong resources with data retained within the institution, but designed for batch processing jobs rather than interactive use;
• Commercial Cloud API: On-demand service but with high costs and privacy risks. The three types of resources each have their pros and cons, and there is no unified system allowing users to choose flexibly, forcing trade-offs between convenience, cost, and security.

The core of STREAM is the intelligent routing layer, integrating local, HPC, and cloud resources:

• Equipped with a local lightweight LLM complexity judge that analyzes query complexity in milliseconds;
• Simple queries → local, medium → HPC, complex → cloud;
• Avoids one-size-fits-all strategies to achieve optimal resource allocation.

To address HPC firewall issues, STREAM adopts a dual-channel design:

• Control Plane: Globus Compute handles authentication and scheduling;
• Data Plane: WebSocket relay transmits tokens without modifying network configurations;
• Effect: First-token latency reduced from 11.4 seconds to 0.54 seconds (21.1x improvement), with end-to-end AES-256-GCM encryption ensuring privacy.

Solves the problem of resource waste in long conversations:

• Context-Aware Level Retention: Intelligently compresses historical conversations to prevent simple queries from being moved to high-cost tiers;
• HPC-as-API: Encapsulates HPC into an OpenAI-compatible API, allowing users to call it without professional HPC knowledge and breaking the latency limits of traditional batch processing.

Benchmark test results (1200 queries across 10 domains):

• When using the Llama3.2 3B local model, 85.1% of queries are completed in the free tier;
• First-token latency comparison: Local (0.26s), HPC streaming (0.54s), commercial cloud API (1.68s);
• HPC mode latency is better than cloud, benefiting from high-performance hardware and optimized paths.

STREAM's value for academia and institutions:

• Compliance: Sensitive data stays in institutional HPC without third-party cloud involvement;
• Cost: 85% free queries save budget;
• Education Scenarios: HPC-as-API lowers the barrier, allowing students and teachers to use HPC like ChatGPT;
• Technical Paradigm: Demonstrates hybrid intelligent collaboration ideas, providing references for resource-constrained scenarios.

Current Limitations:

• Training data and generalization ability of the complexity judge are not detailed;
• WebSocket relay has single-point failure risk. Future Directions:
• Introduce more tiers like edge computing;
• Support tiered inference for multi-modal models;
• Develop adaptive complexity threshold adjustment mechanisms.