Implementing Advanced Recurrent Neural Networks from Scratch: Nonlinear System Identification and MIMO Control Modeling

This article introduces the GitHub project Nonlinear-System-Identification-Neural-Networks, which provides pure Python implementations of advanced neural networks like LSTM and RBFNN, focusing on nonlinear dynamic system identification and multi-input multi-output (MIMO) control modeling. It demonstrates a deep understanding of matrix calculus and custom backpropagation algorithms, serving as a valuable learning resource at the intersection of deep learning and control engineering.

• Original Author/Maintainer: Cata-tinola
• Source Platform: GitHub
• Original Project Title: Nonlinear-System-Identification-Neural-Networks
• Original Link: https://github.com/Cata-tinola/Nonlinear-System-Identification-Neural-Networks
• Release Date: June 14, 2026

The core goal of the project is to apply machine learning to complex control engineering problems, especially the identification, modeling, and tracking of nonlinear dynamic systems and MIMO devices. By building architectures from scratch, it demonstrates an understanding of matrix calculus, gradient flow, and custom backpropagation (such as Dynamic Backpropagation (DBP) and Backpropagation Through Time (BPTT)), providing learning resources for researchers and engineers who wish to deeply grasp the internal mechanisms of neural networks.

Five Core Modules

1. Pi-Sigma Network with Context Layer: Introduces a context layer to handle temporal dynamics, implements an asynchronous backpropagation engine, and captures nonlinear interactions of inputs.
2. Hermite Polynomial Delay Feedback Network: Uses Hermite polynomial expansion instead of standard hidden layers, combined with Dynamic Backpropagation (DBP) suitable for real-time control systems.
3. Radial Basis Function (RBF) Context Network: Enhances RBF networks, creates stable short-term memory structures, and enables simultaneous dynamic backpropagation updates for multiple parameters.
4. Improved Elman Network: Introduces internal self-feedback loops and implements custom Backpropagation Through Time (BPTT) using NumPy.
5. LSTM Mathematical Foundation Demonstration: Manually derives backpropagation, explicitly calculates gradients for each gate, and verifies equivalence with PyTorch's automatic differentiation.

Tech Stack

• Programming Language: Python
• Libraries: NumPy (core algorithms), PyTorch (validation), Matplotlib (visualization)
• Optimization Algorithms: Dynamic Backpropagation (DBP), Backpropagation Through Time (BPTT), Asynchronous Parameter Update
• Domains: Nonlinear Device Modeling, System Identification, Autoregressive Dynamic Systems

• Built from Scratch: No reliance on high-level APIs; each implementation includes complete mathematical derivations and gradient calculations to help understand framework principles.
• Backpropagation Validation: The LSTM module proves the correctness of derivations by comparing custom BPTT results with PyTorch's automatic differentiation, making it suitable for teaching and research.
• Control Engineering Oriented: All architectures are optimized for system identification and control tasks, handling time delays, MIMO scenarios, and real-time tracking requirements.

1. Clone the repository and install dependencies;
2. Enter any subfolder and run the training script;
3. Observe the process of the network dynamically learning and tracking simulated device data;
4. Automatically generate high-resolution PNG charts to visualize convergence loss and system tracking responses.

This project combines theory and practice, providing an ideal form for deep learning education. It is particularly valuable for the following groups:

• Graduate Students and Researchers: Deeply understand the internal mechanisms of RNN variants;
• Control Engineers: Learn the application of neural networks in system identification and control;
• Deep Learning Practitioners: Master custom backpropagation and gradient calculation;
• Educators: Use as teaching material for advanced neural network courses.

The project lays a solid foundation for the application of neural networks in control engineering and is a valuable resource for deeply understanding deep learning principles.