ESP32-S3 Digit reconginiton in Tensor Flow Micro series 00: Intro

ESP32-S3 Digit reconginiton in Tensor Flow Micro series 00

Introduction

This series explores whether TensorFlow Lite Micro (TFLM) can reproduce or improve the digit recognition results I obtained with the PyTorch + ESP-DL workflow on ESP32-S3 hardware.

Goals for this post:

• Review the IDF example that ships with the TFLM component.
• Learn the minimal TFLM runtime flow (model → interpreter → invoke).
• Train a small model on PC/Google Colab, convert it to a TFLite model, and run it on the ESP32-S3.

Prerequisites

• Install the ESP-IDF according to the official guide.
• Google Colab environment or similar for model training and quantization.

Development Environment

• MCU: ESP32-S3 board + 2.8” TFT LCD
  • Hardware purchased from Taobao. More info: ESP32-LVGL开发板
  • ESP32-S3-N8R8
  • LCD Driver: ST7789, 320 x 240 pixels
  • Touch Driver: XPT2046
• IDE: VS Code with ESP-IDF extension
• IDF version: v5.4.2
• esp-tflite-micro package version: v1.3.4

Get started with the IDF tensor flow example code

1. In VS Code, open the ESP-IDF extension and fetch the TFLM example from the IDF component registry.
2. Set the build target to esp32s3, then build and flash the example to verify the runtime works on your hardware.

Learn from the code

1. There are 5 step for prepare the model
   • Get the model
   • Define the resolver
   • Define the interpreter
   • Allocate the memory
   • Get the pointer for input and output
2. There are 3 steps for using the inference
   • (Optional:If the input is the floating point) Quantize the input from floating point to integer ()
   • Run the inference using interpreter->Invoke() funciton
   • (Optional:If the input is the floating point)Convert the quantized output to floating point number.

How the example model is trained

• The IDF example uses a small fully-connected model (example: 1 → 16 → 16 → 1) trained in Python.
• The training script exports a SavedModel / TFLite file.
• A quantization script prepares the model for int8 inference (post-training quantization, PTQ).
• The xxd -i tool converts the .tflite file into a C array (model.cc) for embedding in firmware:

xxd -i model.tflite > model.cc

xxd is commonly available with Git for Windows.

So the fastest way to get the xxd is to install the Git

Workflow summary

1. Train a model on PC or Colab (TensorFlow).
2. Convert the trained model to a .tflite file.
3. Post-training quantize to int8 if targeting TFLM.
4. Convert .tflite to a C byte array (xxd -i).
5. Include the generated C source in the ESP-IDF project and flash.

Quick walkthrough

Train the model in PC side using TensorFlow

I will using the same model structure as the example but decrease the number of nero to 8. So the updated layer should be 1->8->8->1.

Let’s see how will it affect the interence result

Most of the task are using Google colab for the model creation and training process

Result

The test output of the tensor flow model:

The test output of the tenosrflow lite micro model:

Lesson learn

• Pay attension on the tensorflow version. There are bugs on v2.19 which can’t convert the model to tflite. Must use v2.19.1.
  Otherwise, you will have AttributeError: 'Sequential' object has no attribute '_get_save_spec'

• The example code is outdated. You will get the the error when you use the save_model API. save_format arguement is depreacted.
  So we need to udpate to use need API for save the model.

• The example code is wrong on plot the prediction. The example not covert the float to the int8 values as input to the tflite-micro model.
  You need to do the convertion as below in order to have correct ploting.

• TensorFlow framework is updated from time to time. Now, they have new name called LiteRT. For later, they also have updated version called LiteRT Next. For what I have seen, tensor flow lite micro still not be updated. Also, starting from version v2.20, tflite related API will be removed from TensorFLow

Next step

On the next step we will try to use LiteRT framework to make the same model to see if we can get the same result.