-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Fix][Connector-V2] Fix kafka parse datetime value throw NPE #7157
base: dev
Are you sure you want to change the base?
Conversation
…son to read datetime type encountering null values
…concatenation key=null if partition value was null
Thanks @LeonYoah created this PR! Could you add test case? |
|
if ("".equals(jsonNode.asText())) { | ||
return null; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to confirm one thing first, is the data in kafka ""
or null
? If it is null
, we should continue. If it is ""
, I think it is normal to report an error. Please refer #7181
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to confirm one thing first, is the data in kafka
""
ornull
? If it isnull
, we should continue. If it is""
, I think it is normal to report an error. Please refer #7181
I have read your pr, your design is very good, it is no problem to throw wrong theory, but there is another scenario you need to consider, that is, a batch of data sent from upstream, for example, {"date_field":"2023-02-01"},{"date_field":""}, should we return empty strings or pass null to the downstream database instead of parsing them?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to confirm one thing first, is the data in kafka
""
ornull
? If it isnull
, we should continue. If it is""
, I think it is normal to report an error. Please refer #7181
I task this batch of tasks can not be due to an empty string caused by the task error directly quit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can add an option to control this behavior. If a field cannot be parsed, return null
or throw an error. Just like hive. Not just datetime type, this rule is suitable for any type of parsing. cc @hailin0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can add an option to control this behavior. If a field cannot be parsed, return
null
or throw an error. Just like hive. Not just datetime type, this rule is suitable for any type of parsing. cc @hailin0
Good idea. You can control whether to throw an exception or return null with parameters
Purpose of this pull request
#7079
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
release-note
.