Real-Time LLM User Persona at 1 Billion User Scale: Semantic Innovation in Video Recommendation Systems

Core Points: This article introduces the first real-time LLM user persona system deployed on a commercial video platform with 1 billion users. It addresses online LLM computing challenges via knowledge distillation and asynchronous inference, and significantly improves user viewing value in A/B tests.

Original Authors & Source:

• Original Authors: Paper author team (standard arXiv authorship)
• Source Platform: arXiv
• Original Title: LLM-Based User Personas for Recommendations at Scale
• Original Link: http://arxiv.org/abs/2606.12198v1
• Publication Date: 2026-06-10

Dilemmas of Traditional Recommendation Systems

Traditional recommendation systems rely on structured user-item interaction data (ID matrix) and have the following limitations:

1. Lack of Semantic Information: Cannot explain the reasons behind user preferences, limiting interpretability and precision;
2. Cold Start Problem: New users/items lack interaction history, making effective recommendations difficult;
3. Poor Dynamic Adaptability: Model updates require retraining, unable to respond to interest shifts in real time;
4. Exploration-Exploitation Dilemma: Tends to recommend known content, making it hard to explore potential interests.

Opportunities of LLM

LLMs have rich world knowledge, strong reasoning abilities, natural language generation, and context learning capabilities, which provide possibilities to solve the above problems, but face the challenge of real-time low-cost inference at 1 billion user scale.

Core Innovations

1. Natural Language User Interest Persona: Generate human-readable descriptions (e.g., "The user is interested in tech reviews and food exploration"), improving interpretability and semantic richness;
2. Exploitation-Exploration Balance Mechanism: Combine existing interest summaries with the introduction of novel topics to avoid content fatigue and experience degradation;
3. Cost-Efficient Inference Architecture: Reduce costs via knowledge distillation, asynchronous inference, and semantic clustering optimization.

Detailed Technical Architecture

• Layered Architecture: User behavior → Feature engineering → Lightweight persona model → [Optional] LLM refinement → Recommendation engine;
• Knowledge Distillation: Train a lightweight student model to mimic the LLM; 80% of requests are handled by the student model (cost is only 1/100);
• Asynchronous LLM Inference: Request queue + batch processing + caching + incremental updates, without blocking the main recommendation path;
• Semantic Clustering Input Optimization: Cluster user-interacted videos into topics (e.g., "35% tech reviews") to reduce LLM input length.

Experiment and Evaluation Results

Offline Evaluation

• Persona Quality: Manual evaluation shows higher accuracy and completeness; BERTScore verifies consistency with real behavior;
• Recommendation Effect: Outperforms baseline methods, with obvious advantages in long-tail content and cold start scenarios.

User Research

• 85% of users believe the persona accurately reflects their interests and can understand the reasons for recommendations;
• Satisfaction is significantly higher than traditional recommendations, with users praising content diversity and novelty.

Online A/B Test

• Core Metrics: Watch time and click-through rate increased significantly; user retention showed a positive trend; content diversity increased;
• Business Value: Supports real-time services for 1 billion users with controllable costs; both user engagement and platform value are improved.

Technical Challenges and Solutions

1. Latency Requirements: Precompute and cache frequently used personas, asynchronous updates, layered architecture (lightweight model for fast response);
2. Cost Constraints: Knowledge distillation reduces LLM call frequency; semantic clustering reduces input length; batch processing optimizes GPU utilization;
3. Persona Consistency: Structured templates, temperature parameter/random seed control, temporal smoothing mechanism;
4. Privacy Compliance: Differential privacy technology, user-controllable transparency/editing functions, data minimization principle.

Industry Significance

• Paradigm Shift: From ID-based to semantic understanding recommendations; LLMs become core components; interpretability and controllability become standard;
• Engineering Feasibility: Proves the cost controllability of LLM recommendation systems at 1 billion scale;
• User-Centered Design: Transparent recommendation process; users can understand and influence recommendations.

Limitations and Future Directions

• Limitations: Still resource-intensive; LLMs have hallucinations; cross-cultural migration needs optimization;
• Future: Multimodal personas, causal reasoning, federated learning, reinforcement learning to optimize exploitation-exploration balance.