Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: store queries for content topics take longer with cursor #2963

Open
Tracked by #147
stubbsta opened this issue Aug 8, 2024 · 2 comments
Open
Tracked by #147

chore: store queries for content topics take longer with cursor #2963

stubbsta opened this issue Aug 8, 2024 · 2 comments
Assignees
Labels
effort/days Estimated to be completed in a few days, less than a week

Comments

@stubbsta
Copy link

stubbsta commented Aug 8, 2024

Background

When making a storev3 "get messages" request with arbitrary query criteria and no cursor, the response is received much faster than when a cursor is included in the query

Details

On average the query duration is at least twice as long for queries that include a cursor.
Here is an example of the query parameters:

"{'peerAddr': {'/dns4/store-02.do-ams3.status.prod.status.im/tcp/30303/p2p/16Uiu2HAm9aDJPkhGxc2SFcEACTFdZ91Q5TJjp76qZEhq9iF59x7R'}, 'pubsubTopic': {'/waku/2/rs/16/32'}, 'contentTopics': {'/waku/1/0x0ba4cd54/rfc26,/waku/1/0x2cd89333/rfc26,/waku/1/0x1d87a3cd/rfc26,/waku/1/0xf52f9686/rfc26,/waku/1/0x7a02f98c/rfc26,/waku/1/0x185bbe63/rfc26,/waku/1/0x3263a581/rfc26,/waku/1/0x07a60173/rfc26,/waku/1/0xbc0a6d4a/rfc26,/waku/1/0x0de4b636/rfc26,/waku/1/0x2f1b19f4/rfc26,/waku/1/0xa2987838/rfc26,/waku/1/0x90da9ebc/rfc26,/waku/1/0x273c8b02/rfc26,/waku/1/0xb74df338/rfc26,/waku/1/0xdced1732/rfc26,/waku/1/0x98919cb7/rfc26,/waku/1/0x40b24197/rfc26,/waku/1/0xffdd153c/rfc26,/waku/1/0x7aaf28e7/rfc26,/waku/1/0x6cd1c858/rfc26,/waku/1/0x44b029de/rfc26,/waku/1/0xb7b52398/rfc26,/waku/1/0x44db1d5e/rfc26,/waku/1/0x42c4e775/rfc26,/waku/1/0xd586b26d/rfc26,/waku/1/0xc4eeaf67/rfc26,/waku/1/0x5f720d24/rfc26,/waku/1/0xc804e40c/rfc26,/waku/1/0x14a93183/rfc26,/waku/1/0x692b412a/rfc26,/waku/1/0x236da80c/rfc26,/waku/1/0x4612b875/rfc26,/waku/1/0x4696ddf1/rfc26,/waku/1/0x9da54487/rfc26,/waku/1/0xd19ef198/rfc26,/waku/1/0xab5ffc85/rfc26,/waku/1/0x678be1ba/rfc26,/waku/1/0x68a5ccb2/rfc26,/waku/1/0xec639454/rfc26,/waku/1/0xe550dd44/rfc26,/waku/1/0xaff53976/rfc26,/waku/1/0xce498efb/rfc26,/waku/1/0x9bef037f/rfc26,/waku/1/0x019bcad8/rfc26,/waku/1/0x145bfd72/rfc26,/waku/1/0x538017c9/rfc26,/waku/1/0x4441abc9/rfc26,/waku/1/0x8de8f186/rfc26,/waku/1/0x929d3bc7/rfc26,/waku/1/0x9ba6609e/rfc26,/waku/1/0x55af6adb/rfc26,/waku/1/0xb5ddd12d/rfc26,/waku/1/0x91ddcfb6/rfc26,/waku/1/0xf3a9da50/rfc26,/waku/1/0x0080803f/rfc26,/waku/1/0x5fc435cb/rfc26,/waku/1/0x7b9dcfa5/rfc26,/waku/1/0x7d281fed/rfc26,/waku/1/0xd53f2af7/rfc26,/waku/1/0xe5f61b62/rfc26,/waku/1/0xdea01974/rfc26,/waku/1/0x0ccbb8ca/rfc26,/waku/1/0x8f19c9a2/rfc26,/waku/1/0xc08e94aa/rfc26,/waku/1/0xb6368e8a/rfc26,/waku/1/0x98de75de/rfc26,/waku/1/0xa0f174f3/rfc26,/waku/1/0x8bddaef2/rfc26,/waku/1/0xf13176e3/rfc26,/waku/1/0x1a381c17/rfc26,/waku/1/0x2fe862ca/rfc26,/waku/1/0xcc4f7f94/rfc26,/waku/1/0xfbe1741a/rfc26,/waku/1/0x3165779f/rfc26,/waku/1/0x2f97a551/rfc26,/waku/1/0xa133d621/rfc26,/waku/1/0x5c32bfbd/rfc26,/waku/1/0xfddf2992/rfc26,/waku/1/0x5e4d9b31/rfc26,/waku/1/0xa9f7eb66/rfc26,/waku/1/0x410c2bff/rfc26,/waku/1/0x7c9e67ab/rfc26,/waku/1/0x239ee8d3/rfc26,/waku/1/0x062dba0f/rfc26,/waku/1/0xfe9137ff/rfc26,/waku/1/0xa5130468/rfc26,/waku/1/0xd774ad51/rfc26,/waku/1/0xbdf8a655/rfc26,/waku/1/0x8714d0fa/rfc26,/waku/1/0xde88507c/rfc26,/waku/1/0x5f6cd757/rfc26,/waku/1/0x6f27e4b5/rfc26,/waku/1/0x50143bfc/rfc26,/waku/1/0x4b6ca0f6/rfc26,/waku/1/0xa4d9d91b/rfc26,/waku/1/0x42f4395f/rfc26,/waku/1/0x5b10759e/rfc26,/waku/1/0x1041bf52/rfc26,/waku/1/0xf5a7c078/rfc26'}, 'startTime': {1723068000000000000}, 'endTime': {1723111200000000000}, 'ascending': 'true', 'includeData': 'false', 'pageSize': 100, 'cursor': None}"

and the same can be used with this cursor: SFbykRpYfEnZfDxJdfb1z90OjHEaYWMI4FtpD/WzdS8=

Acceptance criteria

Investigate why the query with a cursor takes so much longer and see if it can be improved

@stubbsta
Copy link
Author

stubbsta commented Aug 8, 2024

The same behaviour is observed when only querying for 10 content topics

@gabrielmer gabrielmer added the effort/days Estimated to be completed in a few days, less than a week label Aug 19, 2024
@Ivansete-status
Copy link
Collaborator

Something interesting is that after applying ANALYZE messages the performance got better in status.prod ams db.

On the other hand, I don't see a clear evidence of being slower the queries in the cursor than the first query.
For example, while checking with the message-finder app:

image

@Ivansete-status Ivansete-status self-assigned this Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
effort/days Estimated to be completed in a few days, less than a week
Projects
Status: In Progress
Development

No branches or pull requests

3 participants