Avoid infinite number of calls to listBlobs when doing prefix removals (e..g, gridset or layer removals) #1232
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aaime
commented
Apr 18, 2024
| @@ -154,7 +157,7 @@ public void testPutWithListener() throws MimeException, StorageException { | |||
| eq(tile.getLayerName()), | |||
| eq(tile.getGridSetId()), | |||
| eq(tile.getBlobFormat()), | |||
| anyString(), | |||
| anyStringOrNull(), | |||
There was a problem hiding this comment.
These are changes in Mockito behavior, previous version was matching null too, the current one does not.
aaime
commented
Apr 18, 2024
| break; | ||
| } | ||
| // marker will be empty if there is no next page | ||
| if (Strings.isNullOrEmpty(marker)) break; |
There was a problem hiding this comment.
This is the actual fix for the high number of listBlob. It would just not get out of this loop.
aaime
commented
Apr 18, 2024
| deletesToClear.add(prefix); | ||
| } | ||
| } | ||
| if (!deletesToClear.isEmpty()) { |
There was a problem hiding this comment.
Scheduled deletes stored in property files would not get removed if the folder was empty.
…s (e..g, gridset or layer removals)
aaime
force-pushed
the
azure_infinite_list
branch
from
0036973 to
8c3a051
Compare
This was referenced Apr 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I believe this one happened because of a Jackson behavioral change. The code use to check for the "next page" marker in a paged listing to be null, but it's getting an empty string instead. The object representing the response is being deserialized using jackson.
While I was at it, I've fixed also a couple of other issues in tests, which might be also due to behavioral changes, this time in Azure itself (a "a//b" path is not treated as "a/b" but as "a/<emtpyNameFolder/b").