Search folders in batches to avoid 413 entity too large from drive api #228
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Change
What it says on the tin - search folders in batches to avoid 413 entity too large from the Google Drive API.
Related Issue
Resolves #59
Motivation and Context
When we call
drive.files.list()
infullSearch()
, we pass all of the folderIds (ifdriveType === 'team'
). When you have a lot of folders, the call todrive.files.list()
returns413 Entity too large
because the query is too big.In this PR I added a
MAX_FOLDERS_TO_SEARCH
parameter, and batched thefolderIds
to search into batches of that size or fewer, and then I search over the batches sequentially.FWIW, we could search the batches simultaneously with a
Promise.all()
.In my testing with a 4-character search string, 128 folders worked, but 129 folders triggered the
413
. Because the querystring contains other data besides the parents, you may hit the 418 with fewer folders, so I setMAX_FOLDERS_TO_SEARCH
to 100 to give us some headroom.Checklist
npm run lint
and updated code style accordinglynpm run test
passesrelevant documentation is changed and/or added