Update MicroPython to v1.20.0

[brianredbeard]

Presently MicroPython is running version 1.18.0. I was previously able to upgrade to 1.19.x, but updating to 1.20.x is introducing some new challenges.

This issue is to track the progress of updating MicroPython to v1.20.0. Ultimately it may be used to produce a changelog of what needed to occur in the porting process.

While performing this work, there is a guiding principal:

Just keep in mind that we hope that badgepython will become usable on all the hardware we originally supported with the old platform thing, as long as that keeps working: great 😁👍

Errata:

• Presently there's a bug in PPP networking ( [ESP32 port] network_ppp does not manage pcb properly; can cause LWIP assertion or unmanaged pcb micropython/micropython#11419 ). The workaround is to comment out ${MICROPY_PORT_DIR}/network_ppp.c in components/micropython/CMakeLists.txt.

Todo:

Update `components/libopus`

Currently opus is very out of date. This isn't a problem except for the fact that not updating is also failing to stand on the shoulders of improvements in https://gitlab.xiph.org/xiph/opus/

After an initial check, i just pulled in the repo as a submodule (components/libopus/opus) and replaced components/libopus/CMakeFiles.txt with the following content:

include("${PROJECT_DIR}/components/libopus/opus/cmake/OpusFunctions.cmake")

set(IDF_COMPONENTS
    
    )

idf_component_register(SRCS "${srcs}"
                       INCLUDE_DIRS ${includes}
                       REQUIRES ${IDF_COMPONENTS})


idf_build_set_property(COMPILE_OPTIONS "-DVAR_ARRAYS" APPEND)
idf_build_set_property(COMPILE_OPTIONS "-DHAVE_CONFIG_H" APPEND)

In a perfect world, we may even be able to integrate some of the dark witchery demonstrated here.

That example shows how (for better or for worse) the ESP-IDF handles it's own dependency management (🤮 ) . At the same time, the demonstration of some (slightly beyond basics) configuration examples resulting in static compilation with the final comment explaining how to then consume the whole component is quite nice.

It leaves me inspired as to building a much lighter shim layer for MicroPython as opposed to the current state (i.e. components/micropython/CMakeLists.txt is largely the contents of components/micropython/micropython/ports/esp32/main/CMakeLists.txt). Being able to stand on the shoulders of as much of the upstream maintenance would be ideal.

Update `components/driver_framebuffer`

Some "warnings now become errors" in driver_framebuffer:

[195/265] Building C object esp-idf/driver_framebuffer/CMakeFiles/__idf_driver_framebuffer.dir/png/deflate_reader.c.obj
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c: In function 'lib_deflate_get_huffman':
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:140:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 8: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:141:4: note: here
    case 7: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:141:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 7: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:142:4: note: here
    case 6: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:142:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 6: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:143:4: note: here
    case 5: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:143:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 5: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:144:4: note: here
    case 4: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:144:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 4: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:145:4: note: here
    case 3: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:145:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 3: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:146:4: note: here
    case 2: value <<= 1; value += res & 1; res >>= 1;
    ^~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:146:47: warning: this statement may fall through [-Wimplicit-fallthrough=]
    case 2: value <<= 1; value += res & 1; res >>= 1;
                                           ~~~~^~~~~
~/Projects/hardware/badgePython/components/driver_framebuffer/png/deflate_reader.c:147:4: note: here
    case 1: value <<= 1; value += res;
    ^~~~
[198/265] Building C object esp-idf/driver_framebuffer/CMakeFiles/__idf_driver_framebuffer.dir/png/png_reader.c.obj
~/Projects/hardware/badgePython/components/driver_framebuffer/png/png_reader.c: In function 'lib_png_decode':
~/Projects/hardware/badgePython/components/driver_framebuffer/png/png_reader.c:292:13: warning: variable 'r' set but not used [-Wunused-but-set-variable]
    uint16_t r=0, g=0, b=0, a=0;
             ^

Update buses component

[220/743] Building C object esp-idf/buses/CMakeFiles/__idf_buses.dir/buses.c.obj
~/Projects/hardware/badgePython/components/buses/buses.c: In function 'start_buses':
~/Projects/hardware/badgePython/components/buses/buses.c:30:15: warning: unused variable 'res' [-Wunused-variable]
     esp_err_t res;
               ^~~

migrate from upip to mip

upip was finally nuked from MicroPython in commmit https://github.com/micropython/micropython/commit /924a3e03ec167c4417d89b531794c75ce5a631a3. As such, our various manifests will need to be updated. Of note, there's some new boilerplate here:

https://github.com/micropython/micropython/blob/924a3e03ec167c4417d89b531794c75ce5a631a3/ports/esp32/boards/manifest.py#L1-L15

Migrate FAT build process

Presently the VFS_FAT module is vendored within this code base. The upstream has on pulled in not just vfs_fat, but now littlefs support.

We should validate that using the upstream filesystem semantics are sufficient.

Define boilerplate BADGEPYTHON board

Unlike when the original project launched, today the mechanism exists for defining the customizations to be layered on top of MicroPython in a way which doesn't require code generation, copying of files, and other hi-jinx.

I've begun researching how to do this in a minimalist way utilizing the MicroPython BOARD variable which would also make out of tree development easier.

Implement some type of static analysis (e.g. `cppcheck`)

A cursory run of cppcheck found a memory leak and some macro generation which should be fixed:

❯ cppcheck main
Checking main/factory_reset.c ...
Checking main/factory_reset.c: CONFIG_DRIVER_DISPLAY_I2C_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/factory_reset.c: CONFIG_DRIVER_EINK_ENABLE;CONFIG_DRIVER_GXGDE0213B1_ENABLE;CONFIG_DRIVER_ILI9341_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/factory_reset.c: CONFIG_DRIVER_ERC12864_ENABLE;CONFIG_DRIVER_SSD1306_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/factory_reset.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/factory_reset.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE;CONFIG_DRIVER_HUB75_ENABLE...
Checking main/factory_reset.c: CONFIG_DRIVER_HUB75_ENABLE...
1/8 files checked 17% done
Checking main/main.c ...
Checking main/main.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
2/8 files checked 23% done
Checking main/nvs_init.c ...
3/8 files checked 27% done
Checking main/ota_update.c ...
Checking main/ota_update.c: CONFIG_DRIVER_DISPLAY_I2C_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/ota_update.c: CONFIG_DRIVER_EINK_ENABLE;CONFIG_DRIVER_GXGDE0213B1_ENABLE;CONFIG_DRIVER_ILI9341_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/ota_update.c: CONFIG_DRIVER_ERC12864_ENABLE;CONFIG_DRIVER_SSD1306_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/ota_update.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/ota_update.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE;CONFIG_DRIVER_HUB75_ENABLE...
Checking main/ota_update.c: CONFIG_DRIVER_HUB75_ENABLE...
Checking main/ota_update.c: CONFIG_MBEDTLS_DEBUG...
4/8 files checked 75% done
Checking main/platform.c ...
Checking main/platform.c: CONFIG_DRIVER_EINK_ENABLE;CONFIG_DRIVER_ILI9341_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/platform.c: CONFIG_DRIVER_ERC12846_ENABLE;CONFIG_DRIVER_SSD1306_ENABLE;CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
Checking main/platform.c: CONFIG_DRIVER_FRAMEBUFFER_ENABLE...
5/8 files checked 81% done
Checking main/platform_gen.c ...
main/platform_gen.c:5:5: error: There is an unknown macro here somewhere. Configuration is required. If INIT_DRIVER is a macro then please configure it. [unknownMacro]
    INIT_DRIVER(rtcmem, "rtcmem")
    ^
6/8 files checked 82% done
Checking main/system.c ...
Checking main/system.c: CONFIG_SHOW_CHIP_INFO...
7/8 files checked 85% done
Checking main/zip.c ...
main/zip.c:67:3: error: Memory leak: dr [memleak]
  return ESP_ERR_NO_MEM;
  ^
8/8 files checked 100% done

[brianredbeard]

The general idea was that I can then reference this issue with relevant branches to make sure thorough testing occurs for hardware I don't possess or can't emulate.

[renzenicolai]

One word: awesome! I'm looking forward to seeing BadgePython being worked on 😄

[brianredbeard]

#48 is a resolution for the buses component. Additionally, I'm eyeballing this issue on PPP and this PR I filed (micropython/micropython#11453) which is one minor blocker. In the end our BOARDS will be in our tree since this repository is the anchor of truth for gluing firmware together.

[brianredbeard]

Related to this MicroPython v1.21.0 is out.