Building a Polish Punctuation Restoration System from Scratch: An Educational NLP Project Implemented Purely in GNU Octave

This article introduces an educational project for Polish punctuation restoration implemented purely in GNU Octave. The core goal is to manually build models from scratch (covering n-gram baselines to neural networks), with all mathematical derivations and backpropagation completed manually without relying on external machine learning libraries. The project uses a sequence labeling task to predict the punctuation type (no punctuation, comma, period) after each Polish word, aiming to help learners deeply understand the principles of deep learning rather than just calling APIs.

Polish punctuation restoration is challenging due to its complex morphology and flexible word order. Most NLP solutions rely on frameworks like PyTorch/TensorFlow, but this project chooses to implement everything from scratch in pure GNU Octave. The aim is to allow developers to truly grasp the mathematical essence behind deep learning by manually deriving backpropagation formulas, performing matrix operations, and managing parameters—rather than staying at the level of calling high-level APIs.

The task is formalized as sequence labeling: given a Polish word, predict the punctuation after it (NONE/COMMA/PERIOD). Training data comes from free Polish literary texts from Wolne Lektury, with 90% for training and 10% for testing (split by document to avoid information leakage). Preprocessing includes text cleaning, tokenization, label generation, and vocabulary construction (using the top 5000 words from the training set).

The project is divided into multiple stages:

• Stage0: Preprocessing + bigram baseline (based on frequency statistics; test set Macro-F1=0.511; found that 80.6% of data is the no-punctuation class, indicating severe imbalance);
• Stage1: Single hidden layer MLP (embedding layer:5000×50, hidden layer:128 neurons with ReLU activation, output layer:3 classes; ~282k parameters; manually implemented backpropagation and numerical gradient checking);
• Future stages: Bi-LSTM, mini Transformer, expanding punctuation types, multi-task learning, and REST API deployment.

Technical highlights: Modular code structure (preprocessing, model initialization, forward/backward propagation, etc.); key decisions (weighted cross-entropy loss to balance classes, gradient clipping to prevent explosion, document-level data splitting). Evaluation uses Macro-F1 (primary metric), confusion matrix, etc. Baseline results show that the no-punctuation class has an F1 of 0.9255, but the period class only has 0.1463, reflecting the task's difficulty.

Educational value: Learners can deeply understand backpropagation, matrix operations, engineering details (e.g., handling class imbalance), and end-to-end workflows. This project is a model of educational NLP implementation, proving that deep learning systems can be built without external libraries, providing valuable resources for learners who want to master the essence of algorithms.