Automatic management of ESP32 flash partitions

[phoddie]

There's a very cool new feature in the mcconfig tool that automates management of flash partitions on ESP32 builds. Partitions are how the ESP-IDF divides flash storage for different uses including your project code, file system storage, NVS (used by Moddable SDK Preferences), and mods. There are also special partition requirements to support Over-the-Air (OTA) firmware updates.

Before these changes, developers had to manage partitions manually by creating a partitions.csv file for their project to override the defaults. The defaults reserved space for features, even if a particular project didn't use them. So, if your project didn't use a file system, for example, space was still reserved for files, reducing the flash space available for your project. Even a simple partition map is a bit complicated to understand, which discourages developers from modifying them. For example, here's our default ESP32 partition map before these changes:

nvs,      data, nvs,     0x9000,  0x006000,
phy_init, data, phy,     0xf000,  0x001000,
factory,  app,  factory, 0x10000, 0x390000,
xs,       0x40, 1,       0x3A0000, 0x040000,
settings, data, 1,       0x3E0000, 0x010000,
storage,  data, spiffs,  0x3F0000, 0x010000,

With the latest changes, mcconfig takes care of generating a custom partitions.csv automatically. You can still partition flash yourself, for special project requirements. But, for most developers, these changes will provide more flash space for their projects and eliminate the need to update the partition map themselves. It also simplifies porting to new ESP32 boards. Here's the ESP32 partition map after the changes:

nvs,      data, nvs,     0x9000,  0x006000,
phy_init, data, phy,     0xf000,  0x001000,
factory,  app,  factory, 0x10000, 0x3F0000,

The way mcconfig does this is by first looking for a "factory partition." For projects that don't use OTA, the factory partition holds the firmware image. If there is a factory partition, mcconfig can performa automatic partitioning. Modules like OTA and Files signal in their manifests that certain features are being used using the defines feature of manifests. Then mcconfig uses that information to divide up the factory partition as needed:

• If mod support is enabled, a partition for mods is created.
• If a file system is used (LittleFS, SPIFFS, or FAT32), a file system partition is created.
• If OTA updates are used, 8 KB is reserved for the "OTA Data" partition required by the ESP-IDF. The remaining space is divided into two parts to create the OTA partitions.

There are other rules, like OTA partitions must begin on 64 KB boundaries, that mcconfig automatically applies.

This feature is entirely transparent. By simply adding modules to your project, the partition map is automatically updated for you. The partition map now always make optimal use of the available space.

To learn more about how automatic partition for ESP32 works, check out the updated manifest documentation.

[phoddie]

I noticed that M5Stack Core2 port is only 4 MB of the 16 MB flash. That's because it is using the default partition map for ESP32 ports, which is 4 MB. I added a simple partitions.csv and sdkconfig.defaults and included them in the manifest. Now all 16 MB is available!

The extra space should be nice for Stack-chan. These changes should also be a good model for other ports to follow for ESP32 devices with larger flash memory.