Vector Sink - Batch & Buffer

[hemanthofficial3009]

Hello,
We are using vector as an agent and collector. In the agent the sink is vector and the source on the collector is vector.

As part of our use case, we would like to have the buffering and batching of log events. It would be really helpful if someone could explain in details few things related to Batching and Buffering of the log event provided by Vector.

Buffer:
Ref: Buffer

1. is buffer.max_size /bytes confined to each sink or does it apply for all the sinks we have in configuration?

Batch
Ref: Batch
What would be ceiling point for the-

1. maximum size per batch?
2. maximum batch.timeout_secs ?

Can we use Buffer and Batching together in the same sink? If yes please refer to the following config file :

[sinks.forward]
type = "vector"
inputs = ["validator" ]
address = "xyz"
batch.timeout_secs=10
buffer.type="disk"
buffer.max_size=104900000
buffer.when_full="drop_newest"

When tried to implement, we are facing the following error :

Any references to topics related to batching and buffering for vector sinks would be of great help.

Thanks alot,
Hemanth.

[jszwedko]

Hi @hemanthofficial3009 !

Good questions.

is buffer.max_size /bytes confined to each sink or does it apply for all the sinks we have in configuration?

Each sink has their own independent buffer which can be configured at the sink level with the options you linked to.

What would be ceiling point for the-
maximum size per batch?
maximum batch.timeout_secs ?

This depends on the amount of available memory and how many batches you expect to have in-flight.

Assuming version 2 of the vector sink (see below; you are currently using version 1 which does not support batching), there is no batch partitioning, so there will simply be 1 batch per outbound request. There can be concurrent outbound requests though. The default request concurrency for that sink is "adaptive" which means that Vector will dynamically adjust the number of concurrent requests based on response latency and other signals. You can also configure a static request concurrency by setting concurrency to an integer like 10.

Can we use Buffer and Batching together in the same sink? If yes please refer to the following config file :

You can, but you are running into an issue because the when using version 1 (the default) for the vector sink, it does not support batching so when you configure batch it fails to deserialize the configuration. The poor error messages when that happens are something we are aware of. I thought we had an issue for that, but I can't find it so I'll make a new one.

You can, however, opt into version 2 of the vector sources/sink which leverages gRPC over HTTP and supports batching. You do this by adding version = "2" to both the source and sink configuration.

[hemanthofficial3009]

Thanks @jszwedko for your quick response, that helped us.
We are planning to build agents based out of vector on various platforms.
We do have some relevant questions coming up when we tried to evaluate the buffering and batching concepts.
It would we be of great help if these are addressed.
Regarding Batching:

1. batch.timeout_secs where does the vector hold the events until this time interval is hit
2. Does batch.max_bytes has a ceiling point

Buffering :
Can you throw some light on the buffer.max_size,

1. how vector deals with respect to diskspace
2. how many buffers can be configured at maximum
3. What if the configured buffer size on each sink is greater than the diskspace

[jszwedko]

Hi @hemanthofficial3009 ,

Apologies for the delayed response.

Regarding Batching:

1. batch.timeout_secs where does the vector hold the events until this time interval is hit

It holds them in memory.

2. Does batch.max_bytes has a ceiling point

Vector itself applies no limit.

Buffering : Can you throw some light on the buffer.max_size,

1. how vector deals with respect to diskspace

Vector will use up to the configured max_bytes value on the disk buffer.

2. how many buffers can be configured at maximum

You can configure one buffer per sink.

3. What if the configured buffer size on each sink is greater than the diskspace

This will cause you to run out of disk space. Vector will try to write, and fail. I imagine Vector will crash, but I haven't tested it under these conditions.

Hope this helps! Let me know if you have any other questions.

[hemanthofficial3009]

@jszwedko thanks this helps.

[hemanthofficial3009]

Hello @jszwedko,

1. Is there any specific location which vector uses for buffering the data on the disk?
2. If yes, would that be platform agnostic (common for x86, arm)?
3. If no, is there any way that we can identify the location on the system where vector instance is running?

Thanks for your time.

[jszwedko]

Hi @hemanthofficial3009 !

It will use the configured data_dir. By default on *nix platforms this is /var/lib/vector.