Vector's Built-in-Redundancy Mechanism

[hemanthofficial3009]

Hello,
We run vector instances on different machines which acts as Agents and Collector.

http end-point -> Agent (source type="http") (sink type ="vector") -> Transform (type="remap") -> Collector (source type="vector") (sink type="file")

• Vector acting as an agent would received data from an Http end-point.
• The data received would be enriched (transform) and passed to a down stream vector object which acts as a collector running on another machine.

Can you let us know if there is an option for retries on the sink of the agent which can help to identify that the collector instance is not responding

[spencergilbert]

The vector sink will retry requests based on the provided configuration here: https://vector.dev/docs/reference/configuration/sinks/vector/#request

[hemanthofficial3009]

Thanks @spencergilbert for the quick response.
Can you throw some light on the redundancy mechanism provided by vector, in case any of the configured sink fails to receive the events.

[spencergilbert]

See:

https://vector.dev/docs/about/under-the-hood/networking/arc/
https://vector.dev/docs/about/under-the-hood/architecture/end-to-end-acknowledgements/
https://vector.dev/docs/about/concepts/#buffers

[hemanthofficial3009]

@spencergilbert that was informative, thanks for sharing that.
use case :
each of the below vector instances are running on different systems in a network.

• vector instance acting as an Agent (X)
• vector instance acting as primary Collector (Y)
• vector instance acting as secondary Collector (Z)
• Regular case where Y is running : X ------> Y
• Fail over case : if Y is down then only X ------> Z

X is configured to sink data to Y. for some reason the system where the primary Collector (Y) is present went down.

In this case is there any feature vector provides as part of configuration to configure the Agent(X) so that it would divert the events/logs to a backup/Secondary collector(Z)?

[spencergilbert]

I think you'd be looking for something like sink level circuit breakers, which aren't implemented but are a feature we'd like to add: #8545