Intelligent Price Monitoring System for Library Book Procurement: LLM-Based ETL Pipeline and Analysis Platform

This project constructs a complete ETL pipeline and analysis dashboard, leveraging large language models (LLM) and Google BigQuery to monitor e-commerce book market prices, extract structured data, classify literature, and generate procurement recommendations. It aims to provide data support for library procurement decisions, addressing challenges in traditional manual procurement processes.

Traditional library procurement relies on manual research, price comparison, and decision-making, which is time-consuming and struggles to grasp dynamic market changes. Key pain points include:

1. Difficulty in monitoring dynamic and dispersed e-commerce book prices (promotions, discounts, inventory changes affect costs).
2. Heavy workload and inconsistent standards in manual book classification and metadata organization. These issues drive the need for automated, data-driven solutions to optimize procurement efficiency and decision quality.

Intelligent ETL Pipeline: Multi-source data collection (distributed crawlers for multiple platforms), data cleaning/standardization (unify formats like price units, ISBN validation), incremental update mechanism (reduce storage/bandwidth costs). LLM Applications: Semantic-based intelligent classification (cross-category support), structured metadata extraction from unstructured descriptions, reader comment sentiment analysis. Google BigQuery: High-performance, scalable data warehouse for massive price data storage, real-time SQL analysis, cost-effective on-demand billing, and integration with visualization tools like Data Studio.

Price Trend Analysis: Track historical price changes, recommend optimal purchase timing (when prices hit historical lows), cross-platform price comparison (including shipping/member discounts). Intelligent Procurement Suggestions: Match馆藏 gaps and reader borrowing history to prioritize high-demand books, optimize budget under constraints, detect duplicate purchases. Classification & Theme Analysis: Dynamic topic clustering (identify emerging cross-disciplines), analyze collection structure (find over/under-represented areas), predict reader interest based on content and borrowing data.

Anti-Crawler Countermeasures: Request frequency control (random delays), proxy pool rotation, headless browser simulation for JS dynamic pages. Data Quality Assurance: Field validation rules (format/range checks), cross-platform data verification, manual review workflow for low-confidence data. Scalable Architecture: Plugin-based collectors (easy to add new platforms), configurable rules (classification, price thresholds), RESTful API for integration with library management systems.

Procurement Department: Real-time price monitoring, optimal timing recommendation, budget efficiency improvement, objective decision reports. Collection Development: Analyze collection structure gaps, track discipline trends, evaluate procurement effectiveness. Supplier Management: Leverage market price data for negotiation, assess supplier competitiveness, detect price anomalies.

1. Multi-modal Content Analysis: Extend LLM to analyze book covers/preview pages for richer information.
2. Predictive Procurement: Combine ML models to forecast future book demand (e.g., emerging academic hotspots).
3. Reader Behavior Integration: Deeply integrate borrowing data for personalized recommendations and precise procurement.
4. Open Data Contribution: Publish aggregated price data as open resources for publishing industry and library science research.

The intelligent price monitoring system demonstrates AI's potential in library applications. By combining LLM's semantic understanding with modern data engineering, it provides an intelligent, data-driven solution for procurement. This tool helps libraries fulfill their knowledge service mission more efficiently, offering better collection resources to readers in the digital transformation era.