Tracking number of open file handles by vector

[hartfordfive]

Is there a metric that exposes the number of actively open file handles used? For example, I'd like to be able to track the number of open files by the file sink. Thank you.

[pront]

Hi @hartfordfive, are you using a file source which keeps reading new files?

There is a metric tracking files added here: https://vector.dev/docs/reference/configuration/sources/file/#files_added_total.

[hartfordfive]

I'm using a kafka source in this case. to write data to zstd compressed files. I need a way to track the total number of files being actively being written to.

[pront]

Hmm Kafka source metrics are specified here: https://vector.dev/docs/reference/configuration/sources/kafka/#telemetry-metrics. From a quick search, I don't think we provide such metric.

[pront]

to write data to zstd compressed files

Are you using a file sink for this? And this is the component that creates the files?

The file sink does emit an open_files metric. See https://github.com/vectordotdev/vector/blob/master/src/internal_events/file.rs#L33.

Unfortunately, this is missing from the docs: https://vector.dev/docs/reference/configuration/sinks/file/#instrumentation-telemetry-metrics-and-logs. We should add it.

[pront]

Does this answer you question @hartfordfive?

[hartfordfive]

Yes, the information relating to the file sink open_files metric does answer my question. It might be appropriate to have the documentation updated to include that one. Thank you.

[pront]

Thanks, created #21735 to track.