Communication-Efficient Splitting Inference in Networked MCUs

Abstract

The Tiny Machine Learning (TinyML) brings deep learning to Microcontroller Units (MCUs), but the few hundred kilobytes of Static Random-Access Memory (SRAM) available on these devices make deployment difficult. While model weights can reside in Flash, the intermediate activations generated at inference must fit in SRAM. For the efficient Convolutional Neural Networks (CNNs) that dominate TinyML, such as MobileNetV2, MCUNet, MnasNet, and Proxy lessNAS, it is the peak activation footprint rather than the model size that limits deployment. These networks are built from Inverted Residual Blocks (IRBs), whose expand–depthwise–project structure produces a large temporary activa tion inside each block that can exceed the SRAM of a single MCU. Distributing CNNs across several networked MCUs can aggregate their memory, but existing distributed runtimes communicate at layer or sub-layer boundaries. This turns the expanded activation inside each IRB into a network-level object that must traverse the network twice and be buffered in full at the coordinating device, creating both a communication and a memory bottleneck.

This thesis presents a framework that transforms a standard CNN into a communication-efficient distributed execution across a cluster of networked MCUs. An offline stage quantizes the model to an 8-bit integer and compiles it into a communication-aware execution plan and per-worker manifests. An online stage executes the plan on a coordinator–worker cluster. There are two novel mechanisms in the system: block-wise local execution, which keeps each expanded activation inside a worker, and cross-block data reuse, which exchanges only the thin boundary rows between consecutive blocks.

We implement the framework on a network of six MCUs and evaluate it on four representative IRB-based CNNs. Compared with naive layer-wise execution, our framework reduces early-block communication overhead by over 80%, end-to-end inference latency by up to 47%, and coordinator peak SRAM from 735 KB to 158 KB.