Browser LLM Lab: Technical Practice of Running Large Models Purely in the Browser

Browser LLM Lab is a technical project that demonstrates how to use Transformers.js and WebGPU to run open-source large models like Gemma, Qwen, and SmolLM directly in the browser, enabling zero-backend, fully local LLM inference. This project opens up new paths for privacy-first AI applications and addresses pain points of cloud-based inference such as privacy risks and network dependency. This post will cover aspects including background, tech stack, performance, features, deployment, optimization, and future outlook.

Traditional cloud-based LLM inference relies on GPU clusters but has issues like privacy risks (data upload), network dependency, high latency, and high operational costs. Edge AI has become a direction to solve these pain points, and Browser LLM Lab verifies that modern browsers have the ability to run billion-parameter models locally.

Core Technologies

• Transformers.js: A JS port of Hugging Face Transformers, supporting pre-trained models in browsers/Node.js. It uses ONNX Runtime Web as the backend and can convert PyTorch models to ONNX format.
• WebGPU: A next-generation browser graphics computing API that provides general computing capabilities close to native GPUs, which is key for LLM inference in browsers. Execution Paths:
• WebGPU Mode: Uses GPU parallel computing to provide usable inference speed;
• WASM Fallback: Used when WebGPU is unavailable, but almost unusable for models with >1B parameters.

Supported Models

Model	Quantized Size	Multimodal	Recommended Scenario
Qwen2.5 0.5B	~400MB	No	Fastest speed
Qwen2.5 1.5B	~1GB	No	Balance of speed and quality
SmolLM3 3B	~2GB	No	Multilingual inference
Phi-3.5 mini	~2.5GB	No	Structured inference
Gemma4 E2B	~3.4GB	Yes	High-quality multimodal

Performance: Token generation speeds vary significantly across different hardware:

Hardware	tok/s
Intel iGPU Gen-11	~1
Apple M1/M2	8-15
RTX3060/4060	25-40
RTX4090	60-80

Modern discrete GPUs and Apple Silicon can already provide a usable edge inference experience.

Core Features

1. Capability Detection: Detects WebGPU support, GPU model/memory, RAM, and storage before loading models to ensure compatibility.
2. Model Loading and Caching: Downloads ONNX weights from Hugging Face Hub, supports progress tracking, local caching (Cache API+OPFS), and cache cleaning.
3. Benchmark Testing: Built-in 32-token test to measure tok/s, first token time, and warm-up time.
4. Streaming Inference: Supports token-by-token generation, providing a real-time typewriter effect.

Deployment:

• Local: Zero dependencies. Host using python -m http.server or npx serve, then access localhost:8000.
• Cloud: Optimized for Cloudflare Pages deployment. Need to configure COOP/COEP headers to enable multi-threading and avoid a 2-4x drop in inference speed.

Optimization Tips

• Chrome Flags: Enabling #enable-unsafe-webgpu, #enable-webgpu-developer-features, etc., can improve performance by 1.5-2x.
• Model Mirroring: Mirror models to Cloudflare R2 (no egress fees, low storage cost).

Limitations:

• Long first load time (30-60 minutes on slow connections);
• Large storage usage (each model takes its full size);
• Poor mobile experience (easily overheats);
• WebGPU dependency (performance drops sharply without WebGPU).

Technical Significance

• Decentralized AI: Shifts inference to user devices, promotes AI democratization, and reduces reliance on giant APIs.
• Privacy-First: "Zero data leaves the browser" complies with regulations like GDPR.
• Edge Computing: Browsers become edge nodes. Future possibilities include smaller dedicated models, browser-built-in AI, and hybrid architectures (edge + cloud).

Applicable Scenarios: Privacy-sensitive applications (medical/legal), offline environments, low-latency interactions (typing assistance), cost-sensitive applications.

Summary: Browser LLM Lab demonstrates the feasibility of edge LLM inference. Despite its limitations, it opens new paths for privacy-first AI applications and is worth developers' attention.