Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NFR Scale upstreams tests continue to report out of memory errors #2563

Open
bjee19 opened this issue Sep 16, 2024 · 0 comments
Open

NFR Scale upstreams tests continue to report out of memory errors #2563

bjee19 opened this issue Sep 16, 2024 · 0 comments
Labels
bug Something isn't working
Milestone

Comments

@bjee19
Copy link
Contributor

bjee19 commented Sep 16, 2024

In this pipeline run of the NFR test results on edge: https://github.com/nginxinc/nginx-gateway-fabric/actions/runs/10872403318, the scale test for UpstreamServers encountered an error when ran on Plus.

NGF Error:

{
  "level": "error",
  "ts": "2024-09-15T19:17:23Z",
  "logger": "eventLoop.eventHandler",
  "msg": "couldn't update upstream via the API, reloading configuration instead",
  "batchID": 231,
  "upstreamName": "scale_backend_80",
  "error": "failed to update servers of scale_backend_80 upstream: failed to add 10.120.10.168:8080 server to scale_backend_80 upstream: expected 201 response, got 500. error.status=500; error.text=upstream memory exhausted; error.code=UpstreamOutOfMemory; request_id=12c0924ef4cfa2569609b05e32b8261b; href=https://nginx.org/en/docs/http/ngx_http_api_module.html",
  "stacktrace": "github.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).updateUpstreamServers\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:374\ngithub.com/nginxinc/nginx-gateway-fabric/internal/mode/static.(*eventHandlerImpl).HandleEventBatch\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/mode/static/handler.go:200\ngithub.com/nginxinc/nginx-gateway-fabric/internal/framework/events.(*EventLoop).Start.func1.1\n\t/home/runner/work/nginx-gateway-fabric/nginx-gateway-fabric/internal/framework/events/loop.go:74"
}

Nginx Plus Error:

2024/09/15 19:17:23 [crit] 121#121: ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"
2024/09/15 19:17:23 [crit] 23#23: ngx_slab_alloc() failed: no memory in upstream zone "scale_backend_80"

The error is the same as the one described in #2023

#2439 should have fixed this issue.

The full NFR test results can be found in this PR: #2554

@kate-osborn kate-osborn added the bug Something isn't working label Sep 17, 2024
@kate-osborn kate-osborn added this to the v1.5.0 milestone Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 🆕 New
Development

No branches or pull requests

2 participants