Can't debug esp32s3 (DPTechnics Walter board) with GDB

[fatalkiller]

Hi,
I'm trying to debug a simple code flashed on the Walter board from DPTechnics (source here https://www.dptechnics.com/en/products/walter.html) which is based on an esp32s3 chip. This chip embeds a USB-JTAG to allow debugging without any additional hardware.

When I try to debug the code with GDB, I get the following errors after setting a breakpoint and trying to continue executing the code: Error writing instruction to memory: -4 and then Failed to add SW breakpoint !.

I created a test repo to reproduce the bug that you can find here https://github.com/fatalkiller/test_zephyr_esp32_s3.

What can I do to fix it?