Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update DTFx.AzureStorage v2 Queue to use Base64 Encoding #1154

Closed
wants to merge 8 commits into from

Conversation

nytian
Copy link
Collaborator

@nytian nytian commented Aug 7, 2024

DTFx.AzureStorage v1.x and v2.x with different version Azure Storage SDK has different default encoding style. This pull request updates the queue encoding mechanism in the DTFx.AzureStorage v2 to use Base 64 Encoding style. This change ensures that the queue messages created by DTFx.AzureStorage v2 are forward compatible with those created by DTFx.AzureStorage v1.

Unit test has been added.

@nytian nytian requested a review from davidmrdavid August 7, 2024 20:13
Copy link
Collaborator

@davidmrdavid davidmrdavid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some feedback. It would also be good to add a test here. Are we able to add an automated test that this is able to read messages generated by Azure Storage SDK v1?

Comment on lines 119 to 120
options = options ?? new QueueClientOptions();
options.MessageEncoding = QueueMessageEncoding.Base64;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nytian: I usually like to follow the "rule" that, if we're doing the same work in 3 places or more, we should probably refactor that logic into a common helper function.

Otherwise, I see the following risk: we could end up adding a new method to create a queue client, and then forget to add this "base64" encoding trick.

Can you look into a refactoring of these 3 methods where they call common logic to ensure the client options is always populated with base64 encoding?

Copy link
Collaborator

@davidmrdavid davidmrdavid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there. Left some feedback on the test (thank you for adding it)

@@ -2312,6 +2314,40 @@ public async Task ScheduledStart_Activity_GetStatus_Returns_ScheduledStart(bool
}
}

/// <summary>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please also add a test that this can still read messages sent by DTFX.AS V2.0.0 (which doesn't specify encoding)? Or do we already have that test?

Comment on lines 2346 to 2348
// Check the message can be successfully read without any error
Assert.IsNotNull(content);
Assert.IsInstanceOfType<string>(content);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please also check the contents of the message? Want to make sure it's de-serialized to the right string, and not a random string (which could happen under the wrong encoding)

@nytian
Copy link
Collaborator Author

nytian commented Oct 2, 2024

Close as these changes are no longer needed.

@nytian nytian closed this Oct 2, 2024
@nytian nytian deleted the nytian/update-queue-option branch October 18, 2024 17:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants