HardwareOne LLM Tool: A Complete Solution for Running Local Large Language Models on ESP32-S3 Microcontrollers

HardwareOne LLM Tool provides a complete toolchain for training miniature GPT-2-style language models on a PC and deploying them to ESP32-S3 microcontrollers via a browser-based INT8 quantization converter. This project enables local AI inference on just 8MB of PSRAM without cloud connectivity, opening up new possibilities for edge AI applications.

In the fields of IoT and edge computing, deploying large language models (LLMs) to resource-constrained microcontrollers has always been a highly challenging task. The HardwareOne LLM Tool project provides a complete solution that allows users to train miniature language models on a PC and convert them into a format that can run on ESP32-S3 microcontrollers, enabling fully offline local AI inference. This project is part of the Hardware One ecosystem—a self-contained IoT platform integrating WiFi, sensors, ESP-NOW mesh networks, MQTT, and local AI inference capabilities. The core highlight is: models are trained on a PC and run on ESP32; no training is performed on the device side.

To run within the 8MB PSRAM limit, the project uses a carefully designed miniature GPT-2 architecture:

• Vocabulary: 4K vocabulary size, covering common words while keeping the embedding matrix compact
• Layers: 12-22 layers (depending on preset configuration)
• Dimension: 128-192 dimensional hidden state
• Feedforward Network: 320-768 dimensions, balancing expressive power and memory usage
• Post-quantization size: Approximately 7.3-7.5MB, leaving about 733KB of headroom for runtime

The project provides multiple predefined configurations optimized for different application scenarios:

Preset Name	Vocabulary	Layers	Dimension	FFN	PSRAM Usage	Features
HW1HelpAgent192_deep	4K	18	192	320	~7.3MB	Recommended configuration, optimal balance between depth and width
HW1HelpAgent	4K	22	128	768	~7.5MB	Mature alternative with wide FFN
HW1HelpAgent192	4K	12	192	768	~7.5MB	Wider per layer but shallower
narrow3	4K	18	128	768	~6.9MB	Conservative configuration with maximum memory headroom

The project uses INT8 quantization technology to compress the model to the target size:

• Quantization Granularity: Supports group size 128 configuration
• Browser-based Conversion: No additional software installation required; conversion can be done via a webpage
• Output Format: A single model.bin file, easy to deploy to an SD card

The quantization process is implemented via JavaScript in the browser. Users only need to drag and drop the training output folder onto the webpage, select quantization parameters, and download the converted model file.

Training is done on a PC and requires the following environment:

• Python 3.8+
• PyTorch 2.0+ (CPU or CUDA; GPU is highly recommended)
• 8GB+ RAM (16GB recommended for GPU training)
• Modern browser (for quantization conversion)

GPU training can significantly reduce time: it takes about 30-60 minutes on a modern GPU, while CPU training may take several hours.

The project uses an innovative two-stage training method to improve model quality:

Stage 1 (about 150 epochs): Learn positive question-answer associations from hardwareone_rich.txt. This file contains complete question-answer pairs, paragraphs, and dialogue data.

Stage 2: Apply negative correction, learning to distinguish similar topics (such as ESP-NOW vs. WiFi, MQTT vs. direct connection, etc.) from hardwareone_qa_negatives.txt to prevent concept confusion.

A key technical improvement is boundary-aware training data packing:

Traditional fixed-length chunking cuts question-answer pairs into different chunks, leading to about 39% data corruption. The project implements training chunk packing of 128 tokens, ensuring that no question-answer pair crosses chunk boundaries, allowing the model to learn complete and clean question-answer associations.