Metrics changes after 0.15.0 -> 0.16.0 upgrade

[mhratson]

Looking for pointers to explain new behaviour…

Vector Version

vector --version
vector 0.16.0 (x86_64-unknown-linux-gnu 9e4174d 2021-08-25)

Expected Behavior

Similar metrics behaviour

Actual Behavior

Metrics changed signifcantly

Example Data

Additional Context

We had no changes other than the version bump from 0.15.0 to 0.16.0.

It looks like the new behavior is the correct one but I'm wondering what have changed to cause it.

Coincidentally number of watched files dropped 10fold.

[jszwedko]

Hi @mhratson ! Apologies for the delay. I'm actually not seeing any relevant changes in 0.16.0 that would affect the events in and out metrics. Would you be able to share your config so we could try to reproduce and narrow it down?

[mhratson]

Looks related?

[bruceg]

Yes, that does look like it is closely related. Do you have a chance to try version 0.17.3?

[mhratson]

We're now running 0.17.3

1. watched files hasn't changed and remains similar
2. events_*_total looks more like before so likely fixed?

[bruceg]

1. I don't think there should be any changes to either the file or kubernetes sources that should cause a change in the number of files open. Are you seeing some logs missing from the sinks?
2. I would expect so, yes.

[mhratson]

oh, looks like #8616 may be related to drop of watched files