Client-Assisted LLM: Client-Side Assisted Inference Reduces Cloud LLM Costs and Latency

This project explores a hybrid inference model that involves client-side devices in the LLM inference process: using a local draft model to generate token candidates and a cloud-based validation model to confirm them. This reduces server GPU costs and network latency while fully leveraging the computing power of modern client devices.

Problems with Cloud Dependency

Completely relying on cloud API-based LLM services has two major pain points:

• High Server Costs: Cloud GPU resources are expensive, and each inference consumes a lot of computing resources;
• Network Latency: Clients have to wait for the cloud to complete all generation, leading to long response times that affect user experience.

Underutilized Client Computing Power

Modern laptop GPUs/NPUs have improved performance, but most LLM APIs still treat clients as terminals and do not fully utilize local computing power.

Project Goals

Resolve the above contradictions by having clients participate in the cloud generation process, share server load, and reduce costs and latency.

Basic Workflow

1. Local draft model generates a draft sequence of token IDs;
2. Cloud validation model checks the draft tokens;
3. Accept matching tokens without re-generation;
4. From the first mismatched position, the server takes over and continues generation.

Difference from Speculative Decoding

• Traditional Speculative Decoding: The draft model runs inside the server, and the client waits passively;
• Client-Assisted Inference: The draft model runs on the user's device, actively participates in generation, and fully leverages client computing power.

Model Combination Tests

Tested two cross-model combinations:

• Combination 1: SmolLM2 135M Instruct (draft) → SmolLM2 360M Instruct (validation)
• Combination 2: Qwen2.5 0.5B Instruct (draft) → Qwen2.5 1.5B Instruct (validation)

Acceptance Rate for Different Window Sizes

Model Combination	window=1	window=2	window=4	window=8
SmolLM2 135M→360M	76.2%	67.0%	51.7%	34.0%
Qwen2.5 0.5B→1.5B	59.1%	45.4%	29.8%	18.9%

Conclusion: The smaller the window, the higher the acceptance rate; when window=1, both exceed 50%.

Adaptive Window Strategy

Model Combination	Adaptive Acceptance Rate	Number of Accepted Tokens per Window
SmolLM2 135M→360M	55.2%	1.49
Qwen2.5 0.5B→1.5B	52.7%	0.87

Adaptive strategy maintains an acceptance rate of over 50%, which is practical.

Reliability of Validation Mechanism

Acceptance rate reaches 100% when validating with the same model, proving the measurement logic is correct:

Run Type	Draft Model	Validation Model	Weighted Acceptance Rate
Same Model Validation	SmolLM2-135M	SmolLM2-135M	100.0%

Window Size Trade-offs

• Small Window (1/2): High acceptance rate (50%-76%), but increased number of validation round trips, which is greatly affected by network RTT;
• Large Window (8): Reduces round trips, but acceptance rate drops significantly (19%-34%), and draft quality is unstable.

Practical Deployment Considerations

Need to consider comprehensively:

• Latency Factors: Network RTT, local generation time, cloud validation time;
• Efficiency Factors: Validator batch processing efficiency, client resource usage, server load balancing;
• Adaptive Strategy: Dynamically adjust window size, optimize parameters, real-time monitoring and feedback.

Edge Computing Optimization

Mobile devices use local NPUs to generate drafts, and the cloud only validates part of the generation, reducing response latency.

Cost-Sensitive Applications

Reduce the number of cloud GPU calls, lower API fees, and optimize cost structure.

Privacy Protection Scenarios

Complete most of the inference locally, only send necessary parts to the cloud, reducing data transmission and exposure risks.

Current Limitations

• Closed API Not Supported: Not a wrapper for closed APIs like OpenAI; requires open-source model stacks;
• Model Matching Requirements: Draft and validation models need to be compatible; cross-architecture/data combinations have poor results;
• Network Dependency: Still requires network connection for validation; cannot be fully offline.

Future Directions

• Larger-Scale Validation: Test larger model combinations (e.g., Qwen1.5B→3B/7B) and cross-family models;
• Adaptive Algorithm Optimization: Adjust strategies based on network conditions/input complexity, and learn user patterns;
• Productization Exploration: Develop end-to-end prototypes, measure latency and cost in real scenarios, and build SDKs.

Client-Assisted LLM demonstrates an innovative hybrid inference paradigm. By involving clients in token generation, it significantly reduces cloud costs and latency. Experiments show that small local models as draft generators have an acceptance rate of over 50%, which can halve server workload.

Although still in the experimental stage, the core concept and preliminary results prove its feasibility. With the improvement of edge computing power and network infrastructure, client-assisted inference is expected to become an important optimization direction for LLM deployment, opening up a more efficient and economical path for AI applications.