Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix][Connector-V2] Fix kafka parse datetime value throw NPE #7157

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

LeonYoah
Copy link
Contributor

Purpose of this pull request

#7079

  • Fixed null pointer issue when kafka uses json to read datetime type encountering null values
  • Fixed partition file path issue where path concatenation key=null if partition value was null

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

…son to read datetime type encountering null values
…concatenation key=null if partition value was null
@Hisoka-X
Copy link
Member

Thanks @LeonYoah created this PR! Could you add test case?

@Hisoka-X Hisoka-X changed the title Pr/buxfix/kafka datetime npe [Fix][Connector-V2] Fix kafka parse datetime value throw NPE Jul 10, 2024
@LeonYoah
Copy link
Contributor Author

Thanks @LeonYoah created this PR! Could you add test case?
Receive!

Comment on lines +253 to +255
if ("".equals(jsonNode.asText())) {
return null;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to confirm one thing first, is the data in kafka "" or null? If it is null, we should continue. If it is "", I think it is normal to report an error. Please refer #7181

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to confirm one thing first, is the data in kafka "" or null? If it is null, we should continue. If it is "", I think it is normal to report an error. Please refer #7181

I have read your pr, your design is very good, it is no problem to throw wrong theory, but there is another scenario you need to consider, that is, a batch of data sent from upstream, for example, {"date_field":"2023-02-01"},{"date_field":""}, should we return empty strings or pass null to the downstream database instead of parsing them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to confirm one thing first, is the data in kafka "" or null? If it is null, we should continue. If it is "", I think it is normal to report an error. Please refer #7181
I task this batch of tasks can not be due to an empty string caused by the task error directly quit

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add an option to control this behavior. If a field cannot be parsed, return null or throw an error. Just like hive. Not just datetime type, this rule is suitable for any type of parsing. cc @hailin0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add an option to control this behavior. If a field cannot be parsed, return null or throw an error. Just like hive. Not just datetime type, this rule is suitable for any type of parsing. cc @hailin0

Good idea. You can control whether to throw an exception or return null with parameters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants