Tencent Open-Sources POINTS-Seeker: Training a Multimodal Intelligent Search Agent Model from Scratch

Tencent's latest open-source POINTS-Seeker project aims to build a multimodal AI agent that can independently execute search tasks, demonstrating the technical path for training a dedicated search agent model from scratch. The project adopts an end-to-end training method and the "Agentic Search" paradigm, integrating visual understanding, text reasoning, and search behavior. It can actively plan search paths and is applicable to multiple scenarios such as intelligent customer service and e-commerce shopping guidance. Open-sourcing it provides technical references for the community and promotes the development of the multimodal Agentic Search field.

Traditional search engines are tools that passively respond to user queries. With the development of large language models and multimodal technologies, the industry is exploring more proactive intelligent search agents (which understand intentions, independently plan strategies, and integrate multi-source information). The Tencent team found that general-purpose large models have room for optimization in specialized search tasks, so they decided to train a multimodal agent model optimized for search scenarios from scratch.

POINTS-Seeker uses an end-to-end training method, unifying visual understanding, text reasoning, and search behavior, and can handle multimodal inputs such as images and text. The core innovation is the "Agentic Search" paradigm: unlike traditional RAG systems, it actively plans search paths, evaluates information quality, and iteratively optimizes query strategies, which gives it an advantage in handling complex open-ended problems.

As a multimodal model, POINTS-Seeker can handle various inputs. For example, when a user uploads a photo of a damaged electronic component, the model can identify its type, search technical documents, and provide repair suggestions without lengthy text descriptions. This capability has application value in scenarios such as e-commerce search, visual question answering, and multimedia analysis.

Training a multimodal search agent from scratch faces challenges in data construction (requiring a large amount of high-quality search trajectory data) and reward design (defining success criteria to optimize the model). The Tencent team adopts a strategy combining reinforcement learning and supervised learning: initially, it uses high-quality demonstration data for supervised fine-tuning, and later introduces reinforcement learning to allow the model to interact with a simulated environment to optimize strategies, improving stability and generalization ability.

The open-sourcing of POINTS-Seeker provides valuable technical references for the research community and promotes the standardization and rapid development of the multimodal Agentic Search field. For developers, it is both a usable tool and a learning case, helping them understand the architecture and training methods of building multimodal AI agent systems.

POINTS-Seeker is applicable to multiple scenarios:

• Intelligent customer service: Understand user screenshots and actively search the knowledge base to provide accurate answers
• E-commerce shopping guidance: Combine product images and descriptions to search for the best prices and reviews across platforms
• Academic research: Assist in literature retrieval and automatically track research progress
• Content creation: Collect materials, verify information, and generate references

As an early-stage project, POINTS-Seeker has limitations: its search capability is restricted by the coverage of training data, and it performs poorly in professional fields or emerging topics; the accuracy of multimodal understanding needs to be improved. In the future, the Tencent team plans to optimize search efficiency, expand multilingual support, and improve visual understanding accuracy, while welcoming community contributions to jointly promote technological development.