Weather Data-Integrated New York Taxi Smart Analysis System: End-to-End Practice from ETL to Prediction

This article introduces the open-source project taxi-weather-analytics, which integrates New York taxi data and weather data to build an end-to-end analysis platform from ETL to prediction. The core tech stack includes Apache Airflow (ETL scheduling), PostgreSQL (data warehouse), Power BI (visualization), and Python machine learning libraries (demand prediction). It aims to solve the problem of ignoring weather factors in urban travel demand prediction and provide decision support for traffic planners and taxi operating companies.

Traditional taxi demand analysis only focuses on historical order data and ignores weather as a key influencing factor. Studies show that taxi demand increases by 30%-50% on rainy days; extreme temperatures reduce walking willingness leading to higher demand; seasonal changes and special weather events also alter travel patterns. Based on this insight, the taxi-weather-analytics project builds an analysis platform integrating both types of data to fill the gaps in traditional methods.

The project adopts an end-to-end layered architecture:

1. Core Components: Apache Airflow (task scheduling/orchestration), PostgreSQL (data storage), Power BI (visualization), Python ML libraries (prediction);
2. Data Flow: Data collection (taxi + weather data sources) → ETL processing (scheduled cleaning and transformation via Airflow) → Storage (PostgreSQL) → Analysis and display (Power BI) → Prediction (ML model);
3. ETL Details: Extraction (New York open data + weather API), Transformation (cleaning/standardization/feature engineering/association), Loading (writing to PostgreSQL and query optimization).

Value of Data Integration: Combining weather data can explain the reasons for demand changes (e.g., surge in demand on rainy days), but challenges such as time/space alignment, data quality, and real-time performance need to be addressed; Visualization Capabilities: Power BI supports time trends, geographic heatmaps, weather impact comparisons, and operational indicator monitoring; Typical Scenarios: Operational optimization (peak scheduling), pricing strategy (dynamic pricing), resource allocation (capacity deployment), anomaly detection (problem identification).

Prediction Objectives: Short-term regional demand prediction, weather impact quantification, abnormal travel pattern detection; Feature Engineering: Time features (hour/week/holiday), historical features (past demand trends), weather features (temperature/humidity/precipitation/weather conditions), geographic features (area code/POI density); Model Selection: Time series models (ARIMA/Prophet), ensemble learning (Random Forest/Gradient Boosting), deep learning (LSTM), etc.

Taxi Companies: Optimize scheduling to reduce empty driving rate, balance supply and demand via dynamic pricing, improve service satisfaction; Urban Planning: Identify congestion hotspots to optimize road design, complement public transportation, emergency deployment for extreme weather; Passengers: Get recommendations for optimal travel time, price expectations, and waiting time estimates.

Expansion and Customization: Adapt to other cities (modify data sources/geocoding), customize models (add new features/algorithms), customize visualization (add new charts/dashboards); Open-Source Ecosystem: Transparent and modifiable code, GitHub community support, continuous updates, free to use—suitable for data engineers, traffic planners, and related developers to learn and apply.