Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Search folders in batches to avoid 413 entity too large from drive api #228

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

tarekrached
Copy link
Contributor

@tarekrached tarekrached commented Oct 26, 2020

Description of Change

What it says on the tin - search folders in batches to avoid 413 entity too large from the Google Drive API.

Related Issue

Resolves #59

Motivation and Context

When we call drive.files.list() in fullSearch(), we pass all of the folderIds (if driveType === 'team'). When you have a lot of folders, the call to drive.files.list() returns 413 Entity too large because the query is too big.

In this PR I added a MAX_FOLDERS_TO_SEARCH parameter, and batched the folderIds to search into batches of that size or fewer, and then I search over the batches sequentially.

FWIW, we could search the batches simultaneously with a Promise.all().

In my testing with a 4-character search string, 128 folders worked, but 129 folders triggered the 413. Because the querystring contains other data besides the parents, you may hit the 418 with fewer folders, so I set MAX_FOLDERS_TO_SEARCH to 100 to give us some headroom.

Checklist

  • Ran npm run lint and updated code style accordingly
  • npm run test passes
  • PR has a description and all contributors/stakeholder are noted/cc'ed
  • tests are updated and/or added to cover new code
  • relevant documentation is changed and/or added
    • no user-facing changes

@tarekrached
Copy link
Contributor Author

I ended up implementing the concurrent searching over the batches of folderIds

@isaacwhite isaacwhite self-requested a review February 28, 2021 23:25
@isaacwhite isaacwhite self-assigned this Feb 28, 2021
Base automatically changed from master to main March 13, 2021 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Search Returns 500 Error
2 participants