Mode where nfd-worker updates the labels

[ivelichkovich]

What would you like to be added:

a mode where NFD worker can update labels without needing to run nfd-master with informer cache of entire nodefeatures, maybe some subset map of nodefeatures just for gc if this issue is implemented: #2021 i.e. just store nodefeatures by name/node.

Why is this needed:

nodefeature CRs can be a footgun if users list them i.e. k get nodefeatures in a high scale environment.

nfd master also uses a ton of memory at scale.

If nodefeature-worker just handled the labels for its own node then it would alleviate a lot of scale concerns

[marquiz]

One big issue with the nodefeature objects currently is their size. Quickly thinking, I can see two big culprits adding to that. One is the "managed fields metadata", basically every feature (like kernel.enabledmodule.e1000 is listed there. Not sure how we could avoid listing every possible member of the CRD there, or alternatively filtering out this metadata in listers. Second one is the kernel.config feature which lists every kernel config option, most of which nobody is interested in, and there's a ton of those. We could start building a deny list for filtering out the uninteresting ones or smth.

Second improvent, which I think we really need (and which I have thought for a long time) would be sharding of nfd-master. I.e. distribute the nodes across multiple instances of nfd-master. E.g. calculate a checksum of the nodename and do mod number-of-shards to get the instance (shard) which is responsible for that node.

Splitting the functionality to two daemons is deliberate, e.g. from the security considerations.

Thoughts?

[ivelichkovich]

Yeah I think the deny list as suggested here #2026 is a great idea and smaller change then the other suggestions. This would be a great place to start asap if possible.

Sharding is an interesting idea for sure, would love to discuss that further. The problem though is that even as we paginate the list calls or shard the master it helps automation work smoothly but it still leaves a footgun if a user gets curious and calls k get nodefeatures, although this is a bit mitigated by the allow/deny list. I wonder if the apiserver ListWatch/LIstStreaming work helps here though maybe?

Just for my understanding though, what are the security concerns for the split? Just giving the daemonset permissions to edit nodes?

[marquiz]

Yeah I think the deny list as suggested here #2026 is a great idea and smaller change then the other suggestions. This would be a great place to start asap if possible.

We can start there for sure

Sharding is an interesting idea for sure, would love to discuss that further. The problem though is that even as we paginate the list calls or shard the master it helps automation work smoothly but it still leaves a footgun if a user gets curious and calls k get nodefeatures, although this is a bit mitigated by the allow/deny list. I wonder if the apiserver ListWatch/LIstStreaming work helps here though maybe?

That might help, need to follow up the work more closely. One thing would be to try to make kubectl smarter, support pagination there, too. Another thing we could do is documentation in NFD, warn about the consequences in big clusters.

Just for my understanding though, what are the security concerns for the split? Just giving the daemonset permissions to edit nodes?

Yes. Run nfd-worker with smallest possible privileges.

We could also explore the NodeFeature-less nfd-worker-standalone option too, as an alternative operating mode. But then nfd-worker should replicate all the functionality of nfd-master (NodeFeatureRules, NodeFeatureGroups etc).

[marquiz]

We could/should create a separate issuea about sharding.