Scaling to 100G and Beyond

[jakeb91]

Hi everyone

Bit of a preamble:
I've actually stumbled across LibreQOS by accident during my own foray into VPP and p4-dpdk-target based dataplanes - because what all soft-dataplane solutions sorely lack right now is almost any decent HQoS or queue management features. My work is currently in building a custom BNG that isn't bloated with unnecessary features, nor expects you to fill it with many 10'000's of customers to make it scale financially - VPP fits the bill for this quite nicely on all fronts other than QOS.

I have done alot of reading up on XDP, eBPF and functions around offloading tc-bpf using cls bpf, etc. The software development for such things is well over my head at my current skill level, but I am actively learning - I can at-least demonstrate that I understand why we can't just use XDP to do TC with the current setup for example (need the data from an sk_buffer to do QoS which just isn't available in an xdp_buffer).

I guess what I would like now is some guidance on the current issue(s) that LibreQoS faces that prevent it from being more performant per-core, or if there's any ideas around what might need to be looked at to start potentially offloading the work into SmartNICs (eg. Netronome, or Xilinx/AMD with OnLoad). I believe the former in that list is capable of offloading eBPF policies entirely, but it's been difficult finding any real-world data on them. They're affordable enough that it's still much more preferable to big-vendor-box-here.

I did find an email from @dtaht on the Spinics list recently saying he was looking for ways forward to 100gbps - I am running an ISP of a similar size here in Australia (15-20k subs), and in general I'm quite keen to work together to get figure that out too :).

[dtaht]

I am sorry but the vpp architecture is not very amiable to applying anything other than a policer. It hurts my head to think about it, actually. Pie, did appear recently for dpdk but I have not poked into it. I gave up when xdp started giving us dpdk-like read performance, and went all in on that approach. We are currently handling 10k subscribers at 10gbit total on 16 cores at 22% cpu use so we are certain we scale to at least 25Gbit today on a $1500 box, is that not enough? :)

All the devs can be easily found in the #libreqos:matrix.org chatroom, if you would like faster comms.

[jakeb91]

I have actually tried to join the matrix.org chat already but I get an error saying that I haven't been invited to it when I click the link to join it 🥲

[dtaht]

let me know your handle. It should be open... I thought it was open...

[jakeb91]

My handle is @jakeb91:matrix.org - below is the error I get when trying to join:

[dtaht]

https://matrix.to/#/#libreqos:matrix.org
perhaps? Otherwise I will invite you in, but I swore it was an open room...

[dtaht]

invited you in...

[lukastribus]

I think the per site/ap limit (which is per core) is the more pressing issue. 4 Gbps is probably enough for WISP deployments at this point, but FTTH or other Bitstream based access models will be limited by this. The workaround is to create "virtual" or partitioned sites/ap to be able to use more cores, but this very ugly and error prone.

[thebracket]

We've had individual cores pushing just shy of 10gbps on recent builds, so that problem is slowly going away.

…

On Wed, Feb 22, 2023 at 11:22 AM Lukas Tribus ***@***.***> wrote: I think the per site/ap limit (which is per core) is the more pressing issue. 4 Gbps is probably enough for WISP deployments at this point, but FTTH or other Bitstream based access models will be limited by this. The workaround is to create "virtual" or partitioned sites/ap to be able to use more cores, but this very ugly and error prone. — Reply to this email directly, view it on GitHub <#272 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADRU434D34JYD66GFDPWN4TWYZDL7ANCNFSM6AAAAAAVD64UCA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lukastribus]

Thanks, which CPU was that?

[dtaht]

Intel(R) Xeon(R) E-2378G CPU @ 2.80GHz

[dtaht]

We have a testbed setup here: https://payne.taht.net/ Click run bandwidth test, click on any of the plans. Please note that the tests we are running day to day vary, right now I am for example looking harder at the RTT estimator than the actual throughput, but here we are pushing 8Gbit in each direction, with headroom to spare.

[dtaht]

"I guess what I would like now is some guidance on the current issue(s) that LibreQoS faces that prevent it from being more performant per-core, or if there's any ideas around what might need to be looked at to start potentially offloading the work into SmartNICs (eg. Netronome, or Xilinx/AMD with OnLoad). I believe the former in that list is capable of offloading eBPF policies entirely, but it's been difficult finding any real-world data on them. They're affordable enough that it's still much more preferable to big-vendor-box-here."

We talked about our issues with moving forward we have on this recent podcast: https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/

0. We're at 10Gbit/core, but that is definitely not linear. We did run into scaling problems - using intel 20 cores, with an intel ethernet card, was vastly more performant than 64 AMD cores with a nvidia card, and we don't know why! and lack sufficient budget to explore it. What I am comfortable with saying is 10k subs 20Gbit at less than 50% of CPU, works great presently, based on the actual deployments we have. I would be very uncomfortable running the cores at much more than 60% utilization (it is irq response time that matters, not that mark), and we lack the ability to test anything higher than 25Gbit presently on the grant we got from equinix.

1. Despite our nerdy desire to do even better than we do, we imagine that 90% of the ISPs in the world are not pushing even 10Gbit, today. We are always of course interested in numbers that ISPs are willing to share, but so far it appears that the average bandwidth difference between a 25Mbit home subscriber and a 1gbit subscriber is going from 3Mbit to 5Mbit, tops. This of course differs for business and campuses.

2. We are certainly on a quest for a better network card! However, we have an unusual requirement in needing to do a decently sized LPM (longest prefix match) in order, not to find the route, but the right CPU to interrupt. This is what the @thebracket 's bifrost bridge does and what he's trimmed the most overhead out of, by leveraging xdp and ebpf to steer the data to the right cpu to be processed. This is where dpdk and vpp fall down in that they spray packets to all the cpus via RSS, and not into a definable queue to be shaped.

The onload card you referenced is a tcp offload engine, not helpful. Netronome was barely in business when last I checked. The tremendous expense of these cards in general lead me towards prefering to adopt a whitebox strategy - if you need more than X GB, just rack up another box. That said, if someone dropped a bunch on us and gave us enough development money to go somewhere with it (in a DPU multi-tenant scenario, perhaps), we'd probably pursue it. As it is, please give v1.4 alpha a shot (in monitor mode, at least) see our github sponsors page for how we presently keep the project alive.

https://github.com/LibreQoE/LibreQoS/wiki/v1.4

[dtaht]

We are also tracking and co-operating with the Linux XDP and eBPF communities closely. There are some major improvements in the next Linux release that we will adopt in the next 6 months or so, after the next stable release of ubuntu has time to stablize.

[jakeb91]

Ok this is excellent, lots of information to take in 😁 I've just woken up so I'll dig into the details of everything shared here soon.

I realised I forgot to explain one thing - the motivation behind wanting to scale the performance! We are facing the reality of needing to service up to 10gbps FTTH/FTTP tails on our government-backed national broadband network in the not too distant future. Currently 1gbps tails are the fastest tails we get and the rapidly increasingly popularity of these circuits is already becoming problematic, so you can imagine the impact of multiplying that workload by 10!

[dtaht]

Well... cynically... not a lot. What application would drive traffic to that level? Netflix isn't going to go to 32k video anytime soon. (I am serious, I would like to know what applications are going to actually drive traffic growth further. ChatGPT isn't it)

Anyway I am pleased to hear you want to shape, rather than police... I've heard a lot of complaints from down under about that. Are you using fq_codel or cake already elsewhere in your network?

[jakeb91]

At the moment everything is just RED on this network because it can't handle much more (take one guess as to the platform). Previously I was running a slightly bigger ISP+WISP that had ASR9K BNG's - the shaping mechanisms on them were a-lot better than policing, but I still wasn't overly happy with them in terms of their actual QoS features when compared to an AQM-style solution like CAKE and FQ_CODEL.

My interest in dealing with 10G tails doesn't actually come so much from the tails, I agree the utilisation of a single tail isn't going to be much different to a 1G tail. The obstacle that I am almost certain that most ISP's here will face is being forced to terminate a large number of those tails over a single 10G NNI with the wholesaler due to outrageously high fees and backhaul limitations.