Where can I find a performance testbed for comparing Toit to other language implementations?

[dsobotta]

Are there any open-source performance testbeds to compare Toit against C/Rust/Python/etc? The 30x perf over MicroPython on the ESP32 is really exciting. I'd love to run similar tests on RISC-V 64-bit hardware and see how it compares.

[kasperl]

Great question! We do have some standard benchmarks (also available in Python and JavaScript) ported to Toit that we should publish. Let me take a look at that.

[kasperl]

It is still early days, but we're starting to add relevant benchmarks here: https://github.com/toitlang/toit/tree/master/third_party/benchmarks. If you've got any benchmarks you'd like to see in there, do let me know!

[dsobotta]

Fantastic; thanks much!

[kasperl]

It would be interesting to see if the RISC-V port's startup performance is good (compared to Python, etc.). You might want to use snapshots for that comparison; see #4.

[dsobotta]

Toit is getting ~5x performance when compared to Python3.9 on 64-bit RISC-V/Linux on the SiFive Unmatched. Using a snapshot has a negligible effect over the toit source-probably since I'm running it off an m.2 NVMe drive with 3,500 MB/s read.

ubuntu@ubuntu: git/toit/third_party/benchmarks/python$ python3 ./deltablue.py
DeltaBlue (cpython) - time per iteration: 2154 ms
ubuntu@ubuntu: git/toit/third_party/benchmarks/python$ python3 ./richards.py
Richards (cpython) - time per iteration: 2123 ms

ubuntu@ubuntu: git/toit/third_party/benchmarks/toit$ git/toit/build/host/bin/toitvm ./deltablue.snapshot 
Deltablue - time per iteration: 561.30 ms (total 5.612976s)
Deltablue - allocated per iteration: 1196.135 kb
ubuntu@ubuntu: git/toit/third_party/benchmarks/toit$ git/toit/build/host/bin/toitvm ./richards.snapshot 
Richards - time per iteration: 465.55 ms (total 4.655549s)
Richards - allocated per iteration: 1.795 kb

[kasperl]

Thank you for providing these numbers, @dsobotta. It is a good start, but we can definitely do more with simple inlining, constant propagation and folding, and perhaps compilation to native code.

The snapshots only help with startup performance, so to measure that you probably should be running the simplest "Hello, World" kind of application.