Nextfish-HARENN: NNUE Training Data Generation System for Chess Engines

Nextfish-HARENN is a specialized data generation system designed for training NNUE evaluation functions integrated with HARE (History-Aware Residual Evaluation) neural networks. It addresses key challenges in the data-driven development of modern chess engines, supporting the full pipeline from data collection to format conversion. This tool plays a critical role in the open-source chess engine ecosystem, enabling researchers and developers to focus on neural network innovation rather than data engineering.

The past decade has seen a shift from manual evaluation functions to neural networks in computer chess. AlphaZero (2017) demonstrated deep learning's potential, while NNUE (2019) made neural evaluation practical on CPUs via incremental updates. NNUE's core innovation: it updates only affected features when pieces move, avoiding full forward passes. Its structure includes a feature conversion layer (sparse to dense vectors) and fully connected layers for non-linear transformations.

HARE extends NNUE with two key features: 1) Residual learning: focuses on deviations from a baseline, letting the network learn complex patterns. 2) History integration: uses context like repeat positions, piece development trends, and king safety changes. This allows nuanced judgments—e.g., distinguishing between a position from a sacrifice vs natural development.

Training data quality directly impacts engine performance. Key challenges:

• Source balance: self-play (large but style-single) vs human games (diverse but limited).
• Label accuracy: requires deep searches (depth 20+) or MCTS.
• Phase balance: opening/midgame/endgame.
• Diversity: covering tactics, strategy, quiet positions. Nextfish-HARENN addresses these via: data collection (PGN, UCI outputs), feature extraction (board + extra features like king safety), history encoding (path, exchange history), data augmentation (symmetry), validation, and format conversion (PyTorch/TensorFlow).

Implementation considerations:

• Performance: parallel processing/GPU acceleration for large datasets.
• Engine integration: UCI/Winboard protocol support.
• Memory: streaming/chunking for big data.
• Configurability: adjustable search depth, sampling.
• Reproducibility: seed recording. Ecosystem position: part of the NNUE engine pipeline: data generation → training → quantization → engine integration → testing.

Nextfish-HARENN contributes to the open-source community by:

• Standardizing data generation processes.
• Providing best practices (e.g., search depth, sampling).
• Lowering entry barriers for HARE-NNUE research. Summary: This tool fills a gap in the chess engine toolchain, supporting the advancement of data-driven neural networks in chess. As neural methods evolve, such data engineering tools will grow in importance.