Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add opensearch version info while deserialization #16494

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

soosinha
Copy link
Member

@soosinha soosinha commented Oct 27, 2024

Description

Custom Metadata is serialized writeTo and deserialized using the readFrom method or a constructor with stream input.
For example - ComposableIndexTemplate is serialized using ComposableIndexTemplate.writeTo and deserialized using the stream input constructor.
Consider a case where a new attribute is added in a version. For example context is added in ComposableIndexTemplate from 2.16 onwards. When it serialized from cluster manager of 2.15 version, the context field will not be present. But when the 2.16 cluster manager deserializes the object, it expects context field to be present and breaks.
The fix is to set the opensearch version in the stream input so that the deserialization will not look for new fields.

Related Issues

NA

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

❌ Gradle check result for f98da36: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

✅ Gradle check result for 7a49332: SUCCESS

Copy link

codecov bot commented Oct 27, 2024

Codecov Report

Attention: Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 72.18%. Comparing base (6f1b59e) to head (4bd03a7).

Files with missing lines Patch % Lines
...ch/gateway/remote/RemoteGlobalMetadataManager.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #16494      +/-   ##
============================================
+ Coverage     72.02%   72.18%   +0.15%     
- Complexity    65000    65117     +117     
============================================
  Files          5313     5313              
  Lines        303397   303400       +3     
  Branches      43902    43902              
============================================
+ Hits         218528   219010     +482     
+ Misses        66982    66429     -553     
- Partials      17887    17961      +74     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

❌ Gradle check result for e448dd3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

✅ Gradle check result for 64c813f: SUCCESS

Copy link
Collaborator

@Bukhtawar Bukhtawar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use a specific version rather than CURRENT. How does it work for 2.16 given it is already released and the version might not have been added at the time of serialisation.

@soosinha
Copy link
Member Author

How does it work for 2.16 given it is already released and the version might not have been added at the time of serialisation.

It would be broken for 2.16 and 2.17 as well.

@soosinha
Copy link
Member Author

I have used Version.CURRENT because the serialization is always done using the current version. So we have to deserialize using the current version as well in the unit tests.
I am thinking if we can write a BWC integ test where we can verify deserialization from older version.

@shwetathareja
Copy link
Member

shwetathareja commented Oct 28, 2024

I am thinking if we can write a BWC integ test where we can verify deserialization from older version.

there was BWC tested added in mixed mode upgrade test for context aware template fielf. We should check what was missed there. @mgodwan @soosinha

Signed-off-by: Sooraj Sinha <[email protected]>
Metadata.Custom customMetadata = getCustomMetadata();
String fileName = randomAlphaOfLength(10);
RemoteCustomMetadata customMetadataForDownload = new RemoteCustomMetadata(
fileName,
IndexGraveyard.TYPE,
CLUSTER_UUID,
compressor,
namedWriteableRegistry
namedWriteableRegistry,
version
);
when(blobStoreTransferService.downloadBlob(anyIterable(), anyString())).thenReturn(
customMetadataForDownload.customBlobStoreFormat.serialize(customMetadata, fileName, compressor).streamInput()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in this case it will serialize with which version?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serialize will always happen with current version only

@shwetathareja
Copy link
Member

@soosinha lets add mixed version mode test if it doesn't exist.

Copy link
Contributor

✅ Gradle check result for 4bd03a7: SUCCESS

@soosinha
Copy link
Member Author

Created an issue to track mixed version tests: #16497

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants