Performance test to detect transforms code regressions

[anas-aso]

Hi 👋,
I want to build performance tests for the VRL code we write for our logging pipelines.
I looked into the Soak tests, but they require too much infra/setup to have them running. Also I am looking for something that I can run as a GitHub Workflow.
The setup I came up with looks like this :

1. replace all Vector sources with demo_logs
2. merge all sinks into a sink blackhole sink
3. run vector with that config and capture it's logs. Example log:

{"timestamp":"2022-02-16T17:10:15.290685Z","level":"INFO","message":"Total events collected","events":4000,"raw_bytes_collected":10058173,"target":"vector::sinks::blackhole::sink"}

Since the demo_logs source is "non-blocking" Vector will simply terminate gracefully once all events have been sent to the transforms. Now I can measure how long it took Vector to finish and I know how many bytes and events the blackhole sink used (from the last Vector log event).

I would appreciate it to get feedback about this setup or maybe share ideas about a better (yet simple) way of doing it.

[jszwedko]

Hi @anas-aso !

This seems like a reasonable way to do it. I'd recommend using the internal_metrics source with the prometheus_exporter sink to collect the metrics.

You could take another look at the soak test framework we have too. We've greatly simplified it recently to remove the use of Terraform, Kubernetes, etc. Now it just runs as a single process. See https://github.com/vectordotdev/vector/tree/master/soaks for details.