Turkey Daily Open Dataset: Providing Automated Data Infrastructure for AI and Time Series Prediction Research

acetinkaya/turkiye-daily-open-data is an automatically updated daily open data repository that provides rich data sources from Turkey for artificial intelligence, machine learning, predictive analytics, and time series research, covering multiple fields such as economy, energy, and meteorology. The project addresses pain points like scattered data acquisition, inconsistent formats, and delayed updates through end-to-end automation, greatly lowering the threshold for researchers to obtain high-quality time series data, making it a valuable resource for research in related fields.

In artificial intelligence and machine learning research, high-quality, continuously updated datasets are the foundation for model training and validation. However, common challenges include scattered data sources, inconsistent formats, delayed updates, and cumbersome acquisition processes. The acetinkaya/turkiye-daily-open-data project is an open-source data infrastructure created to address these issues.

Data Acquisition Layer

Uses a multi-source strategy to obtain data from Turkey's Statistical Institute, Central Bank, Energy Market Regulatory Authority, General Directorate of Meteorology, and other public APIs via API interfaces and web scraping. It uses Python's requests and selenium libraries with scheduled tasks to update daily.

Data Processing Layer

Responsible for format standardization (unifying date, value, and encoding formats), missing value handling (interpolation or marking), anomaly detection, and data validation to ensure data integrity and consistency.

Storage and Publication

Processed data is stored in CSV format in a GitHub repository, supporting version control, traceability, easy access, and open collaboration.

Economic Data

Includes USD/Turkish Lira, EUR/Turkish Lira exchange rates, gold prices, stock market indices, central bank policy rates, etc., suitable for exchange rate prediction and macroeconomic analysis.

Energy Data

Covers national and regional electricity consumption, natural gas consumption and import data, renewable energy generation data, supporting energy demand prediction and renewable energy output prediction.

Meteorological and Environmental Data

Provides daily temperatures (max/min/average), precipitation, and Air Quality Index (AQI) for major cities, used for weather prediction and climate change analysis.

Exchange Rate Prediction

Uses historical USD/TRY and EUR/TRY data to build traditional ARIMA/GARCH models or LSTM/Transformer deep learning models, combined with covariates like gold prices and stock market indices for multivariate prediction.

Energy Demand Prediction

Builds short-term (day-ahead), medium-to-long-term (monthly/annual) prediction models based on electricity consumption data, analyzing seasonal patterns, temperature correlations, and the impact of renewable energy penetration.

Climate Change Analysis

Uses long-term meteorological data to study temperature trends, changes in the frequency of extreme weather events, and the impact of urbanization on local climates.

Cross-domain Correlation

Supports interdisciplinary research such as economy-energy, meteorology-energy, and exchange rate-gold.

Acquisition Methods

1. Direct Download: Download individual CSV files (e.g., usdtry_daily.csv) from the GitHub repository;
2. Git Clone Synchronization: Clone the repository and pull updates regularly (git pull);
3. Programmatic Acquisition: Read GitHub raw files via Python (example: pd.read_csv("https://raw.githubusercontent.com/acetinkaya/turkiye-daily-open-data/main/usdtry_daily.csv")).

Quality Check Recommendations

Before use, perform missing value statistics, time continuity verification, anomaly detection, and cross-validation of data consistency.

1. Coverage: Mainly domestic data from Turkey; additional data sources are needed for cross-country comparisons;
2. Update Delay: Some indicators have a 1-2 day delay due to data source limitations;
3. Accuracy: Data comes from public channels; independent verification is required for critical scenarios.

This project provides a high-quality, automated data infrastructure for time series analysis and prediction research, lowering the threshold for data acquisition. As data accumulates, it will support more complex model training; its open-source nature encourages community participation, promoting improvements in data quality and coverage. It is not only a valuable resource for researchers in related fields but also provides a reference paradigm for open data operations.