ITLabAI: A High-Performance Neural Network Inference Library for Embedded Devices

ITLabAI is a lightweight C++17 neural network inference library optimized specifically for edge computing and embedded scenarios. It supports classic CNN architectures such as AlexNet, GoogLeNet, DenseNet, ResNet, and YOLO11x-cls. Its core goals include extreme performance, lightweight deployment, education-friendliness, and multi-architecture support. The project is maintained by embedded-dev-research and hosted on GitHub (link: https://github.com/embedded-dev-research/ITLabAI), with a release date of June 11, 2026.

With the development of artificial intelligence technology, the number of parameters in neural network models has grown from millions to billions or even trillions, and their memory usage, computation latency, and energy consumption far exceed the capacity of embedded devices. How to efficiently run inference in resource-constrained environments has become a key challenge, and ITLabAI is the solution to this problem.

ITLabAI is an inference library focused on classification tasks, implemented in C++17 and capable of running in bare-metal environments. Core goals:

1. Extreme performance (native C++ + parallel optimization)
2. Lightweight deployment (no bulky runtime)
3. Education-friendly (clear code with detailed comments)
4. Multi-architecture support

Supported models and accuracy (as of June 2026):

• AlexNet (MNIST): 98.01% (2026-04)
• GoogLeNet: Top1=43.84%, Top5=68.56%
• DenseNet-121: Top1=65.96%, Top5=86.41%
• ResNet: Top1=77.75%, Top5=93.93%
• YOLO11x-cls: Top1=54.90%, Top5=79.03%

1. Native C++17 implementation: Uses features like std::optional and structured bindings, compatible with GCC7+, Clang5+, MSVC2017+
2. Parallel acceleration: Integrates Intel OneTBB (OpenMP as an alternative) to improve efficiency of compute-intensive operations
3. Cross-platform support: Windows/Linux/macOS, with detailed build guides
4. Model format compatibility: Supports HDF5 (Keras), ONNX (PyTorch/TensorFlow), and PyTorch (YOLO .pt) formats

• Environment preparation: CMake3.10+, C++17 compiler, Python3.x, OpenMP/TBB
• Model conversion:
  • HDF5 (AlexNet): Run python app/converters/parser.py
  • ONNX/YOLO: Run python app/converters/parser_onnx.py Converted weights are stored in the docs folder
• Build (Linux/macOS): Clone the repository → Update submodules → Install OpenMP (macOS) → CMake configuration → Build
• Inference run: build/bin/Graph_Build --model [model name] --parallel (model names: alexnet_mnist/googlenet/densenet/resnet/yolo)

• Performance: The accuracy of each model reflects the correctness of migration from the original framework (see the Supported Models section)
• Application scenarios:
  • Industrial quality inspection: Real-time defect detection on embedded controllers
  • Smart cameras: Local face recognition/object detection (privacy protection + bandwidth saving)
  • Medical devices: Portable auxiliary diagnosis (fast preliminary analysis)
  • Educational research: Clear code for learning neural network inference implementation

• Current limitations: Only supports classification tasks, no low-precision quantization support, no GPU acceleration
• Future directions: Expand to object detection/segmentation tasks, introduce INT8/INT4 quantization, support NPU/TPU/GPU heterogeneous computing, integrate model pruning/knowledge distillation tools

ITLabAI provides a lightweight and powerful solution for embedded AI inference. For industrial developers, it is a directly deployable inference engine; for researchers/students, it is a high-quality teaching material for learning the underlying implementation of neural networks. As the edge AI market grows, such lightweight frameworks will become increasingly important, demonstrating the value of efficient and concise engineering design.