Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Index Template with flat_object type field fails during deserialization #5425

Open
Rishikesh1159 opened this issue Feb 9, 2025 · 0 comments · May be fixed by #5426
Open

[BUG] Index Template with flat_object type field fails during deserialization #5425

Rishikesh1159 opened this issue Feb 9, 2025 · 0 comments · May be fixed by #5426
Labels
bug Something isn't working untriaged

Comments

@Rishikesh1159
Copy link
Member

Rishikesh1159 commented Feb 9, 2025 •
edited
Loading

Describe the bug
If the configuration is using an index template which has a field with flat_object type, then during the deserialization phase the pipeline fails.

To Reproduce
Steps to reproduce the behavior:

  1. Create a new data-prepper (osi) pipeline with source as DynamoDb and Sink as Opensearch collection.
  2. In data-prepper (osi) configuration add a new index-template in sink with a field of type flat_object.
index: "sample-test-index"
index_type: custom
template_type: index-template
template_content: >
{
  "template" : {"mappings":{"properties":"description":{"type":"keyword"},"configuration": 
  {"type":"flat_object"},"update_time":{"type":"date","format":"strict_date_time||epoch_millis"}}}}
}
  1. Above is the sample config template with configuration field of type field_object. We don't have any index sample-test-index already created in opensearch. We will be creating this index from data-prpper.
  2. Save the config and create the data-prepper (osi) pipeline.
  3. Confirm that index is created in opensearch and validate that index mapping is identical to the index template passed in config file.
  4. now restart the data-prepper (osi) pipeline.
  5. After pipeline is restarted, you can check the data-pepper(osi) pipeline logs. You will find an exception Failed to initialize OpenSearch sink with a retryable exception. in logs with data-prepper (osi) pipeline broken.
  6. See error :
org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception. 
org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'Builder.<variant kind>'
	at org.opensearch.client.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:89) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.opensearch._types.mapping.Property.<init>(Property.java:187) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.opensearch._types.mapping.Property$Builder.build(Property.java:1410) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:60) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:369) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:355) ~[opensearch-java-2.8.1.jar:?]
	at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.8.1.jar:?]

Expected behavior
Even with configuration field of type field_object is used it should work normally as expected.

More Explanation on Flow
This "field_type" is not a problem during the initial index creation but it's a problem after data-prepper (osi) pipeline is restarted because :

First Time (Index Creation):

Initial Flow:
Data Prepper starts
↓
Reads template from its config (has flat_object)
↓
Checks if template exists in OpenSearch (doesn't exist)
↓
Creates template using REST API
→ Just sends JSON template as is
→ No deserialization needed
↓
Creates index based on template
↓
Success! Index created with mappings

On Restart :

Restart Flow:
Data Prepper starts
↓
Reads template from its config (has flat_object)
↓
Checks if template exists in OpenSearch (exists)
↓
Gets existing template from OpenSearch
→ Makes GET request to OpenSearch
→ OpenSearch returns template JSON
→ Java client tries to convert JSON response into Java objects
    → Attempts to create Property objects for each mapping field
    → Fails when trying to deserialize flat_object type
    → Throws MissingRequiredPropertyException
↓
Fails before reaching template comparison step

Solution
From above flow of execution we have seen that opensearch-java client will be used to deserialize flat_object type and it fails. The problem is data-prepper (osi) currently uses opensearch-java client version 2.8.1 which doesn't support flat_object type. From opensearch-java client version 2.8.2 and greater we started supporting flat_object type. opensearch-project/opensearch-java#735

To resolve this we have to upgrade opensearch-java client version from 2.8.1 to atleast 2.8.2 or later.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working untriaged
Projects
Data Prepper Tracking Board
Status: Unplanned
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Upgrade opensearch-java client to 2.20.0 version. Rishikesh1159/data-prepper
1 participant
@Rishikesh1159