How is vector able to keep up with multiple container logs?

[portante]

Hi folks,

How does vector handle scaling the number of log files for the various container use cases?

If I have an 40 CPU box (2 socket, 10 cores per socket, HT enabled), and perhaps have a ratio of 2:1 for containers to CPUs, where the containers are actively logging, how does vector handle the scale of reading from all those files?

Does it employ multiple CPUs to keep up?

Thanks!

[jszwedko]

Hi @portante !

Currently the file source only uses one thread for reading so will be bound to a single core with each file cooperatively scheduled. There is one knob, oldest_first that can be used to tell Vector to always read the oldest file first until there is no new data until moving on.

We have plans to revisit the file source soon with an eye towards improving performance. This will likely involve using multiple threads for reading.

[jcantrill]

Currently the file source only uses one thread for reading so will be bound to a single core with each file cooperatively scheduled. There is one knob, oldest_first that can be used to tell Vector to always read the oldest file first until there is no new data until moving on.

Does this imply I could define multiple file sources that mutually exclude file patterns and achieve multiple threads in that fashion?

[jszwedko]

That is correct, they would run independently and in parallel.