AI Agent-Based E-Commerce Product Information Crawling System: From Proof of Concept to Practice

This article introduces the AI-assisted e-commerce product crawling POC project developed by KevinSama6. Built with Playwright, OpenAI LLM, Pydantic, and MySQL, this project uses an intelligent agent architecture to achieve automatic extraction and storage of structured e-commerce data. The project was validated on two categories of the Safco Dental Supply website: sutures and surgical supplies, and dental examination gloves, demonstrating the application potential of AI Agents in the field of data crawling.

Traditional e-commerce crawlers face challenges such as complex page structures, strict anti-crawling mechanisms, and inconsistent data formats. This project proposes a solution combining an AI Agent architecture with browser automation and LLM, focusing on two core categories of the Safco Dental Supply website (sutures and surgical supplies, dental examination gloves) to verify the feasibility of an end-to-end workflow.

The project adopts a modular Agentic architecture with the following workflow: Seed category URL → MySQL queue → Get HTML → Page classifier → Navigation agent → Product URL → Queue → Extraction agent → Validation/deduplication → MySQL product table → Export. Core components include:

• Page Classifier: Uses rules to determine page type (category page/product page/unknown);
• Navigation Agent: Combines rule-based parsing and LLM to extract product links;
• Extraction Agent: Uses LLM to extract structured data (name, brand, specifications, etc.) from product pages;
• Validator/Deduplicator: Cleans SKUs, handles missing values, and deduplicates via MySQL primary keys.

The project uses Pydantic to define strict data models:

• ProductModel: Includes product name, brand, category hierarchy, URL, description, specifications, image URL, alternative products, and variant list;
• ProductVariant: Includes SKU, size/color, price, and inventory status. The database uses MySQL with core tables:
• urls_queue: Stores URL, type (category/product), status (pending/completed/failed), and update time;
• products: Stores product URL, JSON-formatted data, and update time.

The technology stack includes Playwright (dynamic page automation), OpenAI LLM (intelligent extraction), Pydantic (data validation), and MySQL (storage). Project significance:

1. Reduce maintenance costs: Minimize the writing of parsing rules;
2. Improve robustness: Adapt to page structure changes;
3. Structured output: Directly convert to JSON;
4. Extensible: Modular design supports multiple categories/websites.

Current limitations: Only supports two categories, single-threaded processing, limited error handling, and LLM call costs. Improvement directions:

• Expand full-site support;
• Introduce asynchronous/parallel processing;
• Improve monitoring and alerts;
• Optimize LLM call caching;
• Add distributed task queues (e.g., Celery);
• Implement incremental updates and data quality audits.

This POC proves that AI Agents combined with LLM and automation tools can build an intelligent and controllable crawling system. Insights for developers:

• Design a modular Agent architecture;
• Balance rule-based and AI methods;
• Build recoverable/extensible pipelines;
• Utilize modern Python toolchains. In the future, as LLM capabilities improve, such systems will become more popular and mature.