Attempting to upgrade from vector 0.27.0-distroless-libc to new versions

[mohdAkibUddin]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am attempting to go from vector 0.27.0-distroless-libc to 0.42.0-distroless-libc.

Taking an incremental approach from 0.27.0-distroless-libc to 0.33.0-distroless-libc to 0.38.0-distroless-libc to 0.42.0-distroless-libc we notice memory increases along the way.

On version 0.27.0-distroless-libc we saw an average use of 60-80 MiB

On version 0.33.0-distroless-libc we saw an average use of 105 MiB

On both versions 0.38.0-distroless-libc and 0.42.0-distroless-libc we saw an average use of 115-125 MiB.

Is it expected for memory usage to increase in this manner? Are there any VRL changes that come to mind between these changes that you think I’d have to make?

Version

0.27.0-distroless-libc, 0.33.0-distroless-libc, 0.38.0-distroless-libc, 0.42.0-distroless-libc

Example Data

On version 0.27.0-distroless-libc we saw an average use of 60-80 MiB

On version 0.33.0-distroless-libc we saw an average use of 105 MiB

Additional Context

Vector is running on Kubernetes

Here's what vector top shows on 0.42.0-distroless-libc

[pront]

Are there any VRL changes that come to mind between these changes that you think I’d have to make?

I would recommend smaller incremental changes. You can follow the upgrade guides (here).

[jszwedko]

Agreed. @mohdAkibUddin I'd suggest upgrading one minor release at a time (i.e. 0.27.x to 0.28.x to 0.29.x etc.). If you find a significant change between two specific minor releases, that'll make it easier to dig in to see what might be the cause (and could constitute a bug report; I converted this one to a discussion for now).

[mohdAkibUddin]

A change I've made was killing the processing of _unmatched logs by setting https://vector.dev/docs/reference/configuration/transforms/route/#reroute_unmatched to false. I haven't seen a huge reduction in the average but I do see less variance in the memory use.

[mehta-ankit]

@jszwedko long time Friend 👋🏼 (it was great working with you and spencer years ago)

we can try all the versions mentioned, but all versions we try higher than 0.27 have memory increase.
Especially on 0.42.0 going from 0.27.0 we see a huge increase at startup.

Do you have pointers on why at startup it uses so much memory ? We use auditd_logs, journald source and kubernetes_logs source.

[mehta-ankit]

Is is dependent in any way on the eks node size ? 🤔

[jszwedko]

Do you see an increase from 0.27.0 to 0.28.0? If so, by how much?