Anonymous download performance enhancements

[egbertbouman]

I'm currently doing some performance enhancements to our core, in particular to the IPv8 socket/tunnel community. By moving some code to rust (Tribler/py-ipv8#1119), we're getting much better download speeds:

[synctext]

25.5 MByte/sec download speed 💥
Real results, after starting abandoned experiments in 2019: #4567
Prior 2019 code: https://github.com/ip-v8/rust-ipv8
🚀

[ichorid]

I wonder how much of increase that will be, if you move both sides of the tunnel (e.g. tunnel helpers too) to Rust... 🤩

[egbertbouman]

I don't really expect to see speeds beyond what I'm currently seeing. If I look at the speed with just a single circuit, it won't go beyond 3MB/s. So, while testing, I worked around the issue by creating 20 circuits.
At 25MB/s the most CPU intensive thread will use a full core at the downloader side. Of course, it all depends on which machine you use.

[synctext]

As discussed in Dev Meeting...this month, Jan 2024 all Rust matters on both IPv4 and IPv6 should come together. Stable performant code is the ambition.

[egbertbouman]

IPv8 has now been refactored to support a separate Rust endpoint (Tribler/py-ipv8#1250). There is also a release on PyPI.

[qstokkink]

For the occasion, I gave the Rust implementation a run on my bare metal machine (AMD Ryzen Threadripper 3970X 32-Core, 64 GB 2666 MHz RAM, from an M2 drive with a gigabit Internet connection).

As a benchmark, using the same torrent, I gathered three results that are more-or-less repeatable:

• A 0 hop download using the Libtorrent 2.0.7 "simple client" C implementation: the maximum we can achieve with Tribler.
• A 0 hop download using "Tribler Experimental" and Libtorrent 1.2.19 with everything switched off except the download manager, IPv8, and the tunnel community.
• A 1 hop download using "Tribler Experimental" and Libtorrent 1.2.19 with everything switched off except the download manager, IPv8, and the tunnel community.

Here are the screenshots:

Libtorrent 2.0.7 C implementation +- 110 MB/s

Tribler 0 hops +- 67.5 MB/s

Tribler Rust 1 hop anonymous +- 27.5 MB/s