Move node-exporter to its own container

[zimnx]

Today, it is running part of the Scylla container. This is not good for multiple reasons:

• It could interfere with Scylla's work (resulting in higher latency), it consumes part of the memory allocated to it, etc.
• We don't, but we should be able to independently update them.

It's a process to get it done, but let's design first HOW we'd do it, then see what needs to be changed in the Scylla container.

[mykaul]

I don't know how to mark it as NOT priority/important-soon - it's not that important to be completed soon. Unless we can provide evidence it has a noticeable impact on latency.

[tnozicka]

I don't know how to mark it as NOT priority/important-soon

/priority important-longterm

unless we can provide evidence it has a noticeable impact on latency.

All the processes running in the same container impact the ScyllaDB latency by stealing CPU cycles from it. We have traced down an instance when it was the sidecar operator cache updating, I'd expect a similar ones occur as well, but this is quite hard to catch at the system level.

[tnozicka]

/remove-priority important-soon

[gecube]

@tnozicka Hi!

All the processes running in the same container impact the ScyllaDB latency by stealing CPU cycles from it. We have traced down an instance when it was the sidecar operator cache updating, I'd expect a similar ones occur as well, but this is quite hard to catch at the system level.

I think even if they would be running in the same pod (but in different containers) on the same host you will observe the latency issues... not?

[tnozicka]

I think even if they would be running in the same pod (but in different containers) on the same host you will observe the latency issues... not?

the cgroup limits are applied per container

[gecube]

from this perspective - I agree, but still we have shared CPU/Mem/disk on the same node.

[tnozicka]

with a guaranteed class, croup limits and pinned cores it's pretty much isolated

[mykaul]

BTW, we did recently removed some (many) of the default collectors from node-exporter, so there should be somewhat less noise. Perhaps when running on K8S we can use even less collectors? (but I think the path forward should be focused on splitting it to its own container).