benchmarks.adoc

Table of Contents

• Benchmarks
  • CMSIS / SWD
    • cp / OpenOCD / pyOCD
    • Comparison with picoprobe and automated_test.py of pyOCD
    • probe-rs: picoprobe vs yapicoprobe
• Network

Benchmarks

CMSIS / SWD

cp / OpenOCD / pyOCD

Benchmarking is done with an image with a size around 400KByte. Command lines are as follows:

• cp: time cp firmware.uf2 /media/picoprobe/

• OpenOCD 0.12.0-rc2 (CMSIS-DAP)v2: time openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "adapter speed 25000" -c "program {firmware.elf} verify reset; shutdown;"

• OpenOCD 0.12.0-rc2 (CMSIS-DAP)v1: time openocd -f interface/cmsis-dap.cfg -f target/rp2040.cfg -c "cmsis_dap_backend hid; adapter speed 25000" -c "program {firmware.elf} verify reset; shutdown;"

• pyOCD 0.34.3: time pyocd flash -f 25000000 -t rp2040 firmware.elf, pyOCD ignores silently "-O cmsis_dap.prefer_v1=true", except for the "list" option

Note that benchmarking takes place under Linux. Surprisingly OpenOCD and pyOCD behave differently under Windows. DAPv2 is always used, because DAPv1 does not run under Linux(?).

Table 1. CMSIS-DAP Benchmarks

command / version	cp	OpenOCD DAPv1	OpenOCD DAPv2	pyOCD DAPv2	comment
very early version	-	-	10.4s	-
v1.00	6.4s	-	8.1s	16.5s
git-3120a90	5.7s	-	7.8s	15.4s
- same but NDEBUG -	7.3s	-	9.5s	16.6s	a bad miracle…​ to make things worse, pyOCD is very instable
git-bd8c41f	5.7s	28.6s	7.7s	19.9s	there was a python update :-/
git-0d6c6a8	5.7s	28.5s	6.8s	20.2s
- same but optimized for OpenOCD	5.7s	28.5s	6.1s	-	pyOCD crashes
git-0eba8bf	4.9s	28.6s	6.5s	13.8s	cp shows sometimes 5.4s
- same but optimized for OpenOCD	4.9s	28.6s	5.8s	-	pyOCD crashes
git-e38fa52	4.8s	28.6s	6.6s	14.0s	cp shows sometimes 5.4s
- same but optimized for OpenOCD	4.8s	28.6s	5.9s	-	pyOCD crashes
git-28fd8db	4.1s	28.6s	6.2s	13.9s	cp shows sometimes 4.6s, SWCLK tuned to 25MHz
- same but optimized for OpenOCD	4.1s	28.6s	5.7s	-	pyOCD crashes

Comparison with picoprobe and automated_test.py of pyOCD

There is a good benchmark (actually the functional tests) in pyOCD: automated_test.py. For configuration check https://github.com/pyocd/pyOCD/blob/main/docs/automated_tests.md

Comparing the original picoprobe firmware (running on the official Raspi Pico Debug Probe) and YAPicoprobe shows the following picture:

Table 2. Benchmarking with automated_test.py

Probe HW	Probe FW	Target	1.1MHz	2MHz	4MHz	6MHz	8MHz
Debug Probe	Original Picoprobe (download)	nRF52840 dongle	586s	583s	580s	581s	(-)
Debug Probe	YAPicoprobe	nRF52840 dongle	327s	236s	192s	171s (*)	(-)
Pi Pico W	YAPicoprobe	nRF52840 dongle	332s	234s	192s	177s	(-)
Debug Probe	Original Picoprobe (download)	PCA10056	586s	582s	581s	580s	580s
Debug Probe	YAPicoprobe	PCA10056	322s	233s	189s	183s	244s
Pi Pico W	YAPicoprobe	PCA10056		237s	193s	178s	173s

(*) unexpected error only observed in conjuntion with the nRF52840 dongle.

Tests were performed with the following configuration:

probes:
  <probe-unid>:
    frequency:       [12468]000000
    target_override: nrf52840
    test_binary:     l1_nrf52840-dk.bin

Findings are not completely clear:

• the original probe firmware is very slow, no clue about the actual reason. Weird is that runtime with the original firmware is not frequency dependant

• there seems to be a frequency limit of the debug probe HW at around 6MHz due to its in-series resistors (runtime increase from 6 to 8MHz)

• the nRF52840 dongle has problems with >=6MHz as well. Either a PCB or a nRF52840 part problem. Max frequency of 4MHz should be used

• good setups are

  • YAPicoprobe + PCA10056 @ 6MHz

  • YAPicoprobe + nRF52840 dongle @ 4MHz

probe-rs: picoprobe vs yapicoprobe

Using probe-rs benchmark --address 0x20000000 --chip nrf52840_xxaa --speed 10000 as a simple benchmark shows the following result for maximum speed:

picoprobe 1.1: ~107KByte/s

yapicoprobe 1.20: ~175KByte/s

probe-rs is at version 0.23.

Network

This is a simple network performance measurement with for MSS in 100 800 1450; do iperf -c 192.168.14.1 -e -i 1 -M $MSS -l 8192 -P 1; sleep 2; done over CPU frequency. Network driver is NCM (new), application is running on one core.

Network throughput is almost identical from 72..192MHz and makes a jump by about 50% at 216MHz. Maximum iperf throughput is 5MBit/s.

Note	Benchmarks took place with YaPicoprobe v1.18 and Debian 6.4.4 x86_64 GNU/Linux. With the following kernel, the numbers were different: better iperf, but worse SystemView throughput.

CPU Frequency
72MHz
192MHz
216MHz
240MHz

More images here. CPU ran until 288MHz, @312MHz the CPU did not start.