EdgeInfer: A Lightweight Deterministic Neural Network Inference Framework for ARM Embedded Devices

EdgeInfer is a bare-metal firmware framework written in pure C language, specifically designed for ARM-A architecture embedded devices, supporting the execution of ONNX-format neural networks. Its core features include static memory management (zero dynamic allocation), modular pipeline design, user-extensible hook mechanism, and QEMU simulation support. It addresses pain points in edge AI deployment such as resource constraints, high real-time requirements, and OS-less environments, enabling development and debugging before model deployment without physical hardware.

Edge AI deployment needs to address constraints like limited computing resources, limited memory capacity, high real-time requirements, power sensitivity, and bare-metal environments (without an operating system). Traditional frameworks such as TensorFlow Lite or PyTorch Mobile rely on dynamic memory allocation and complex runtimes, making them too bulky for strict embedded scenarios; moreover, there is a lack of lightweight simulation verification solutions before hardware is ready. EdgeInfer is designed specifically to address these pain points.

EdgeInfer follows three core design principles: 1. Zero dynamic memory allocation: All memory is pre-allocated at compile time, eliminating issues like heap fragmentation and leaks, with predictable memory usage; 2. Deterministic execution: The pipeline model and static memory ensure predictable inference latency, facilitating Worst-Case Execution Time (WCET) analysis; 3. Modular pipeline architecture: Inference is divided into three stages—preprocessing → inference → postprocessing—with clear interfaces for easy extension.

EdgeInfer adopts an offline conversion + online execution architecture: 1. ONNX to C: The development host uses Python tools to convert ONNX models into C header files (including weights and topology). The model is stored in Flash, eliminating the need for ONNX parsing on the device side; 2. User-extensible hooks: Customize preprocessing (data normalization, etc.), postprocessing (result parsing, etc.), and inference override (engine replacement) via function pointers; 3. ARM bare-metal support: Includes startup code, linker scripts, UART drivers, and supports QEMU simulation to accelerate early development.

EdgeInfer development workflow: 1. Model training and export: Train using PyTorch/TensorFlow and export to ONNX format; 2. Model conversion: Use scripts to convert ONNX to C header files; 3. User extension implementation: Write preprocessing/postprocessing hook functions; 4. Compilation and simulation: Cross-compile the firmware and run verification in QEMU; 5. Hardware deployment: Burn to the target ARM device; migration only requires adjusting the underlying drivers.

EdgeInfer is suitable for: 1. Early algorithm verification: QEMU simulation can verify model correctness and performance before hardware finalization; 2. Extremely resource-constrained devices: Zero dynamic allocation and streamlined code are suitable for KB-level memory and OS-less devices; 3. Functional safety-critical applications: Static memory design meets the requirements of scenarios like aviation, automotive, and industrial control; 4. Teaching and learning: Streamlined code makes it easy to understand the underlying implementation of neural network inference.

Current limitations of EdgeInfer: 1. Limited operator support: Mainly supports basic ONNX operators; complex structures (like Transformers, attention mechanisms) need additional implementation; 2. Single architecture support: Only supports ARM-A; ARM-M series requires further optimization and tailoring. Future improvements need to expand the operator library and architecture support.

EdgeInfer provides a lightweight, deterministic solution for edge AI deployment. Its features like static memory, modular pipeline, and QEMU simulation are suitable for resource-constrained scenarios and early verification. Comparison with existing solutions: It is lighter than TensorFlow Lite Micro (no complex runtime) and has a lower entry barrier than CMSIS-NN (direct ONNX conversion). It is positioned between the two, balancing flexibility and simplicity.