Deploying Multimodal Deep Learning on Microcontrollers: Edge AI Practice for CNC Tool Wear Prediction

This article introduces a study on compressing and deploying a multimodal deep learning model onto resource-constrained microcontrollers, aiming to solve the CNC tool wear prediction problem. By fusing image data (microscopic images of tool sides) and sensor data (time-frequency maps of multi-axis force/vibration signals), a dual-tower network architecture was built and compressed to INT8 precision. Finally, a prediction accuracy of 20.33 microns was achieved on the NXP FRDM-MCXN947 microcontroller, verifying the feasibility of edge AI in industrial predictive maintenance scenarios.

In the field of precision metal processing, CNC tool wear is a core challenge: premature replacement wastes resources, while delayed replacement leads to product defects or equipment damage. Traditional methods rely on operator visual inspection (subjective) or specialized sensors (high cost). The development of edge computing and TinyML technologies provides the possibility of real-time tool status monitoring on the device side, without relying on the cloud or expensive infrastructure.

The project aims to explore the feasibility of deploying deep learning models on resource-constrained microcontrollers, using the NXP FRDM-MCXN947 development board (Cortex-M33, 2MB flash/512KB SRAM). The dataset uses MATWI, which includes tool wear images, sensor records, and side wear measurement values. It is divided by tool groups: training set (7 groups, 647 samples), validation set (3 groups, 300 samples), test set (3 groups, 247 samples) to avoid data leakage.

Image Modality: Compressed ResNet18 (structured pruning + knowledge distillation + QAT), 1M parameter version with MAE of 29.46 microns, occupying 970KB flash; Sensor Modality: MultiScaleSensorCNN (input 5-channel CWT time-frequency map), MAE of 28.86 microns, occupying 238KB flash; Fusion Modality: Dual-tower architecture (image + sensor encoder), MAE of 20.33 microns under INT8 precision, occupying 1230KB flash, which is the main deployment target.

Deployment process (Path B): PyTorch QAT checkpoint → ONNX export → static quantization → static deduplication → NXP onnx2tflite conversion → boundary surgery → TFLite generation. Hardware validation includes modifying the linker script (extending flash to 2MB), MCUXpresso IDE configuration, TFLM operator parser setup, and realizing host-MCU data transmission via UART, verifying the feasibility of the end-to-end process.

Core results: Compressing the complex multimodal system to 1.2MB and achieving 20-micron precision on the MCU proves that deep learning can be used for industrial predictive maintenance under strict hardware constraints. Implications for manufacturing: Using low-cost MCUs and open-source toolchains to build on-device intelligent monitoring systems, reducing cloud dependency, and improving latency and data privacy.

Technical key points summary:

1. Multimodal fusion improves prediction accuracy;
2. Compression pipeline: structured pruning → knowledge distillation → QAT → static INT8 conversion;
3. Choosing NXP onnx2tflite tool to avoid precision loss;
4. Hardware optimization for Cortex-M33 and TFLM;
5. End-to-end process verification. Project code and documentation have been open-sourced on GitHub (https://github.com/DavidTrov/multimodal-tool-wear-prediction) for reference.