Reinforcement Learning in Browser with Pure JavaScript: Building a CartPole Policy Gradient Network from Scratch

Core Idea: This project is a reinforcement learning sandbox implemented with pure vanilla JavaScript, enabling training of neural networks to control the CartPole inverted pendulum balance in the browser without external machine learning libraries. The project has significant educational value, allowing beginners to explore core reinforcement learning concepts without complex environment setup.

Most reinforcement learning projects rely on frameworks like TensorFlow/PyTorch. This project chooses to implement the CartPole environment and policy gradient network with pure JS, allowing users to experience it by opening an HTML file. This design is beginner-friendly—no need to configure a Python environment, CUDA drivers, or handle dependency conflicts. With fewer than a thousand lines of code, it demonstrates key mechanisms such as environment simulation, neural network forward propagation, and policy gradient updates, making abstract algorithms accessible.

The core of the project is a feedforward neural network with a single hidden layer. The input layer receives 4 state variables (pole angle, angular velocity, cart position, velocity). The number of hidden layer nodes is configurable, and the output layer uses Softmax to generate a probability distribution for left/right actions. The REINFORCE algorithm is adopted—its core is to sample complete trajectories, calculate total rewards, and update weights in the direction that increases the probability of high-reward actions (gradient ascent).

The UI uses a dark glassmorphism style. Real-time neural network visualization shows dynamic changes in hidden layer weights; the interface displays action probabilities, confidence levels, and cumulative reward curves in real time. Interactive features include: controlling the training process (start/pause, adjust speed), switching test modes (exploration vs. exploitation), adjusting hyperparameters (learning rate, Gamma, number of hidden layer nodes), and saving/loading models.

Extension directions include: algorithm upgrades (migrating to DQN/PPO to improve efficiency), hardware acceleration (integrating TensorFlow.js to utilize GPU), environment expansion (supporting classic problems like MountainCar), interactive perturbation (mouse drag to test robustness), and evolutionary algorithms (genetic algorithms replacing gradient descent).

The project proves that complex concepts can be presented concisely without relying on large frameworks. Through clear code and intuitive visualization, core reinforcement learning mechanisms become easy to understand. For learners, it is the best starting point for policy gradient introduction; for developers, it demonstrates a lightweight AI deployment method; for educators, it is an out-of-the-box teaching tool. It reminds us that understanding the essence of algorithms is more important than proficiently using frameworks.