Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENT bug] Query hits PostgreSQL 65535 parameters limit #1848

Closed
pxp928 opened this issue Apr 18, 2024 · 1 comment
Closed

[ENT bug] Query hits PostgreSQL 65535 parameters limit #1848

pxp928 opened this issue Apr 18, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@pxp928
Copy link
Collaborator

pxp928 commented Apr 18, 2024

Describe the bug
After a large dataset is ingested, the OSV certifier and others that queries for all packages hit the PostgreSQL 65535 parameters limit:

go run ./cmd/guacone certifier osv
{"level":"info","ts":1713447858.50912,"caller":"logging/logger.go:75","msg":"Logging at info level"}
{"level":"info","ts":1713447858.509234,"caller":"cli/init.go:69","msg":"Using config file: /Users/parth/Documents/pxp928/artifact-ff/guac.yaml"}
{"level":"error","ts":1713447861.422855,"caller":"cmd/osv.go:169","msg":"certifier ended with error: failed sources query: input: packages pq: got 563996 parameters but PostgreSQL only supports 65535 parameters\n","stacktrace":"github.com/guacsec/guac/cmd/guacone/cmd.init.func10.3\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:169\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify.func1\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:81\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:99\ngithub.com/guacsec/guac/cmd/guacone/cmd.init.func10.4\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:180"}
{"level":"info","ts":1713447861.4229681,"caller":"cmd/osv.go:192","msg":"All certifiers completed"}

This happens based on how the query for packages is setup where it eager loads the packageName edges to reconstruct model.Package as shown below (WithName does the eager loading):

pkgs, err := b.client.PackageVersion.Query().
Where(packageQueryPredicates(pkgSpec)).
WithName(func(q *ent.PackageNameQuery) {}).
Limit(MaxPageSize).
All(ctx)
if err != nil {
return nil, err
}

This similar issue may be faced by other nouns/verbs that do some type of eager loading or filtering.

To overcome this, we need to implement pagination in ENT: https://entgo.io/docs/tutorial-todo-gql-paginate/#add-pagination-support-for-query. More information found in the issue #1525

To Reproduce
Ingest a large amount of SBOM and try to query without a filter. This will result in:

{
  "errors": [
    {
      "message": "HasSBOM: pq: got 563996 parameters but PostgreSQL only supports 65535 parameters",
      "path": [
        "HasSBOM"
      ]
    }
  ],
  "data": null
}

Expected behavior
Return data back with pagination

GUAC version
main

@pxp928 pxp928 added bug Something isn't working trust issues describing providing additional information so that consumers can trust GUAC's results and removed trust issues describing providing additional information so that consumers can trust GUAC's results labels Apr 18, 2024
@pxp928 pxp928 self-assigned this May 1, 2024
@pxp928
Copy link
Collaborator Author

pxp928 commented Oct 28, 2024

fixed with #2184

@pxp928 pxp928 closed this as completed Oct 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant