need to send data to clickhouse strongly every 10 seconds

[remort]

The clickhouse sink is configured and works well, though it sends every new chunk of data to clickhouse more than 10 seconds. It looks like it varies from 10 to 11 secs. I was unable to find how to configure this time for the certain sink in documentation. We rely on strong periodicity of incoming data in DB. Is there any way to configure Vector like this?

[spencergilbert]

We don't exactly have support for this, you could get close by using the batch configuration and have the batch flush every 10s, and have really high max events/bytes.

[remort]

Unfortunately it doesn't help. Still 11, sometimes 12 seconds.

[jszwedko]

Yeah, that's correct, you can control the flush interval, but it might still take a second for the request to be successfully processed by Clickhouse. How are you measuring the "every 10 seconds"? Is that after Clickhouse responds? Or just when the request is initiated by Vector?

[remort]

it's the clickhouse timestamp. it's when record has been held in DB.

[jszwedko]

Gotcha, yeah I see. I think it is going to be difficult for Vector to guarantee this since it depends on the request latency to Clickhouse.

[JeanMertz]

I'll mark this as resolved, as the provided solutions are as close as you're going to get when networking is involved. Vector cannot know how long a request will take to reach the destination, so even if it does emit events every 10 seconds, it'll still be between 10 and 10+latency duration until the event is stored in the database.

Technically, we could have a feature that tracks latency, and uses that to reduce the time, it waits to emit events, to get closer to the desired exact duration, but that still will be probabilistic in nature, and the complexity involved seems to outweigh the potential benefit of this feature.

I'll mark this as resolved, but you are welcome to leave a comment if you'd like to discuss this further.