You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
After a large dataset is ingested, the OSV certifier and others that queries for all packages hit the PostgreSQL 65535 parameters limit:
go run ./cmd/guacone certifier osv
{"level":"info","ts":1713447858.50912,"caller":"logging/logger.go:75","msg":"Logging at info level"}
{"level":"info","ts":1713447858.509234,"caller":"cli/init.go:69","msg":"Using config file: /Users/parth/Documents/pxp928/artifact-ff/guac.yaml"}
{"level":"error","ts":1713447861.422855,"caller":"cmd/osv.go:169","msg":"certifier ended with error: failed sources query: input: packages pq: got 563996 parameters but PostgreSQL only supports 65535 parameters\n","stacktrace":"github.com/guacsec/guac/cmd/guacone/cmd.init.func10.3\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:169\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify.func1\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:81\ngithub.com/guacsec/guac/pkg/certifier/certify.Certify\n\t/Users/parth/Documents/pxp928/artifact-ff/pkg/certifier/certify/certify.go:99\ngithub.com/guacsec/guac/cmd/guacone/cmd.init.func10.4\n\t/Users/parth/Documents/pxp928/artifact-ff/cmd/guacone/cmd/osv.go:180"}
{"level":"info","ts":1713447861.4229681,"caller":"cmd/osv.go:192","msg":"All certifiers completed"}
This happens based on how the query for packages is setup where it eager loads the packageName edges to reconstruct model.Package as shown below (WithName does the eager loading):
Expected behavior
Return data back with pagination
GUAC version
main
The text was updated successfully, but these errors were encountered:
pxp928
added
bug
Something isn't working
trust
issues describing providing additional information so that consumers can trust GUAC's results
and removed
trust
issues describing providing additional information so that consumers can trust GUAC's results
labels
Apr 18, 2024
Describe the bug
After a large dataset is ingested, the OSV certifier and others that queries for all packages hit the PostgreSQL 65535 parameters limit:
This happens based on how the query for packages is setup where it eager loads the
packageName
edges to reconstructmodel.Package
as shown below (WithName
does the eager loading):guac/pkg/assembler/backends/ent/backend/package.go
Lines 50 to 57 in 358205b
This similar issue may be faced by other nouns/verbs that do some type of eager loading or filtering.
To overcome this, we need to implement pagination in ENT: https://entgo.io/docs/tutorial-todo-gql-paginate/#add-pagination-support-for-query. More information found in the issue #1525
To Reproduce
Ingest a large amount of SBOM and try to query without a filter. This will result in:
Expected behavior
Return data back with pagination
GUAC version
main
The text was updated successfully, but these errors were encountered: