Safco Scraper：基于 AI Agent 的智能产品数据抓取系统

This post introduces Safco Scraper, an Agent-driven product data scraping system built with Playwright, OpenAI LLM, Pydantic, and MySQL. It demonstrates intelligent product page discovery, structured catalog data extraction, and a recoverable workflow. As a Proof of Concept (POC), it targets two product categories from Safco Dental Supply: Sutures & Surgical Products and Dental Exam Gloves. The system emphasizes modularity, cost-effectiveness, and data quality.

E-commerce data scraping is foundational for price monitoring, inventory management, and competitor analysis. However, modern e-commerce sites use complex dynamic rendering, making traditional static HTML parsing ineffective. Additionally, varying page structures across sites pose a challenge for adaptive data extraction. Safco Scraper addresses these issues with an Agent-driven architecture combining browser automation and AI capabilities.

Safco Scraper uses a clear Agent pipeline:

1. Seed category URLs → MySQL URL queue
2. Navigator Agent discovers product URLs → MySQL queue
3. Extractor Agent extracts structured data → MySQL product table
4. Export to CSV/JSON

Key components:

• pipeline.py: Orchestrates the workflow
• scraper.py: Uses Playwright for page automation
• agents.py: Implements Navigator and Extractor Agents
• models.py: Pydantic data models for structured output
• db.py: MySQL operations (queue, storage)
• export_sample.py: Exports data to CSV/JSON

Navigator Agent: Uses a hybrid strategy—rule-based HTML parsing first (low cost, fast, predictable) to find product links, with AI fallback for complex pages. This balances efficiency and coverage. Extractor Agent: Uses OpenAI LLM to extract structured data (product name, brand, specs, variants, etc.) and validates output via Pydantic models, ensuring data quality and structure.

• Playwright: Handles dynamic JS-rendered content, simulates user behavior, and bypasses simple anti-scraping measures.
• Pydantic: Ensures type safety, runtime validation, and easy serialization of structured data.
• MySQL: Manages URL queues (for recoverability) and stores structured product data.
• OpenAI LLM: Enables intelligent extraction from unstructured content and diverse page layouts.

The system supports断点续传 (resume from interruption) via:

• URL Queue State Tracking: Each URL has status (pending, processing, completed, failed) to avoid re-scraping and enable retries.
• Modular Design: Separated navigation, extraction, storage, and export components—single component failures don’t break the entire system, allowing independent fixes/retry.

Though a POC, Safco Scraper has production-level potential:

• Evolvable: Can extend to more product categories, multiple sites, and distributed scraping.
• Cost-Effective: Hybrid strategy (rules + AI fallback) controls operational costs.
• High Data Quality: Pydantic validation ensures consistent, usable data for downstream applications (analysis, ML).

Safco Scraper demonstrates AI Agent’s value in data scraping. Key takeaways:

1. Intelligent Layering: Use AI only when rules fail to reduce costs.
2. Structured Priority: Define clear data models (Pydantic) for quality output.
3. Recoverable Design: Critical for production systems to handle interruptions.
4. Modularity: Eases maintenance, testing, and scaling.

Agent-driven architectures like this will likely find broader applications in data engineering (web scraping, document processing, data cleaning) as AI capabilities advance.