Add ability to throttle streaming sinks

[s1msn]

Currently there seems to be no mechanism to limit the bandwidth used by vector when sending events.
In large environments, this could potentially lead to problems, especially when one or more vector instances start streaming buffered events or resume reading from a file source.

[LucioFranco]

@s1mpS Hi! I believe using a combination of the batch_size and rate_limit_* options to limit the amount of outgoing data so that you don't get the stampeding herd effect should work. This should limit the total amount of data able to leave that instance of vector via the configured sink.

[s1msn]

@LucioFranco Hi Lucio, thanks for the quick reply. I noticed the rate_limit options for the elasticsearch sink, but there does not seem to be one for the vector sink yet. As this is the recommended sink when deploying vector as an agent, it would be great to have this as an option in this sink too.

[LucioFranco]

@s1mpS yup, this is because sinks like the vector and tcp sink are "streaming" based and we would need to throttle. This should be possible though. I will update the issue to reflect this.

[LucioFranco]

I've added a couple more sinks to the list here but most likely what we will want to do is use some form of throttle https://docs.rs/tokio/0.2.0-alpha.6/tokio/stream/trait.StreamExt.html#method.throttle like this one. That said, the tokio throttle seems a bit limited and we might want to allow users to specify something similar to how our rate limits are configured by setting an amount of data that should be sent over a window and any more should apply back pressure.

[binarylogic]

by setting an amount of data that should be sent over a window

👍 , although our rate_limit* options do not do this. They're quantity based not bandwidth based. I think the latter would also be useful in batching sinks, I'm just unsure exactly how that would work. I think it would be worth agreeing on what the options look like before beginning work on this.

[ghost]

Is it possible to add a rate_limiter transform that would do the rate limiting before the events reach sinks? In that case it would be possible to keep sinks' configuration and code simple, yet do the rate limiting. It might also be necessary to do rate limiting before some transforms, especially if we implement transforms doing some external API requests (see #1041 for examples of such transforms).

[lukesteensen]

@a-rodin Yes, and we even have #258 already to represent it 😄

I think I'd prefer using the transform over adding options to every sink. The downside would be that we couldn't really integrate it with encoders for precise bytesize-based limits. Or we can at least implement the transform first, and then integrate into sinks/encoders in a second pass.

[binarylogic]

I agree, closing this in favor of #258.