Goroutine spike when closing connections in v1.8.10 (regression)

[ash2k]

When we bumped the library from v1.8.7 to v1.8.10 we started seeing huge memory usage spikes when the process is shutting down. It turned out to be because the library starts lots of goroutines in some (?) case.

Blue graph is RAM usage in percentage of max allowed. Note how spikes start at some point - that's when the version with the library update was rolled out.

Screenshot from Google Profiler with the width of the flame graph showing the proportion of goroutines with a certain stacktrace. Note that at the moment of capture there was 153,983 goroutines, most of them from the library.

This is a regression. Code worked fine until the version bump and works fine again since we downgraded the library.

Original issue: https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/492

[nhooyr]

Related #429

Terrible code here on my part, will need to rewrite the close code to fix. Will try and do soon.

[nhooyr]

Thanks for reporting. I've fixed this in #427. Please review and let me know if you see anything else.

[nhooyr]

Sorry for the delay. Fixed by #427.