Policy-Decoupled Sub-Byte TinyML on Microcontrollers

Abstract

Sub-byte mixed precision helps a TinyML model fit the tight Flash budget of a microcontroller (MCU). Selected weights are stored in INT4 or INT2, while sensitive weights stay at INT8. Making such a per-tensor policy deployable on an MCU, however, takes more than a quantization choice. Current deployment stacks couple precision to the firmware across the model format, tensor metadata, and kernels. Thus, changing a per-tensor policy becomes firmware integration instead of simply regenerating an artifact, and each sub-byte model ends up reimplemented from scratch. Packed weights also cost latency. With no native sub-byte arithmetic on these cores, unpacking on the inference path can make sub-byte execution up to nearly 9× slower than INT8.

This thesis presents a manifest-guided deployment framework that decouples the precision policy from firmware execution. An offline packer turns a trained INT8 model and its per-tensor precision plan into a packed FlatBuffer plus a precision manifest, so one unchanged firmware serves different sub-byte policies. On the device, a two-phase Prepare/Eval contract routes each packed tensor to a backend path that keeps weight unpacking off the multiply-accumulate loop, trading a little SRAM for INT8-level latency. We implement the framework in TensorFlow Lite Micro and evaluate it on an Arm Cortex-M33 across five models and three sensing modalities. It reduces model Flash by 31% to 50% and keeps INT4, INT2, and mixed-precision inference within 0.91× to 1.24× of INT8 latency, recovering near-INT8 execution speed while retaining the storage savings of sub-byte weights.