[feature](hive)support hive catalog read json table. #43469
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
hubgeter
commented
Nov 7, 2024
| @@ -332,6 +336,11 @@ class DataTypeSerDe { | |||
| Arena& mem_pool, int row_num) const; | |||
| virtual Status read_one_cell_from_json(IColumn& column, const rapidjson::Value& result) const; | |||
|
|
|||
| virtual DataTypeSerDeSPtrs get_nested_serdes() const { | |||
There was a problem hiding this comment.
maybe we can support
get_key_serde() get_value_serde() for map
get_data_serde() for array nullcolumn
get_serde(int) for struct
not get_nested_serdes() for any complex type
|
TeamCity be ut coverage result: |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
morningman
reviewed
Nov 10, 2024
| for (auto* slot_desc : slot_descs) { | ||
| if (_is_hive_table) { | ||
| //don't like _fuzzy_parse,each line read in must modify name_map once. | ||
| for (auto* v : slot_descs) { |
There was a problem hiding this comment.
I think it will seriously impact the performance.
What if we set a session variable to control it? And by default, in most cases, there is no duplicate column name in json value.
There was a problem hiding this comment.
Here we not only need to deal with the same column names, but also the order of the table column names and the json data. I think I can simplify this double for loop by using a layer of for loop plus map.
| if (slot_desc->is_nullable()) { | ||
| vectorized::IColumn* data_column_ptr = column_ptr; | ||
| DataTypeSerDeSPtr data_serde = serde; | ||
| vectorized::DataTypeSerDe::FormatOptions _options; |
There was a problem hiding this comment.
Looks like this _options can be created only once at reuse it in each call.
There was a problem hiding this comment.
yeah , i will fix it.
| *value, "Json value is null, but the column `{}` is not nullable.", | ||
| slot_desc->col_name(), valid)); | ||
| return Status::OK(); | ||
| } else if (type_desc.type == TYPE_STRUCT) { |
There was a problem hiding this comment.
Can we use switch ... case for following logic?
There was a problem hiding this comment.
i need check _is_load and type_desc , It seems that there is no way to use switch.
| } else { | ||
| return Status::InternalError<false>("It should not here."); | ||
| // Maybe we can `switch (value->GetType()) case: kNumberType`. |
There was a problem hiding this comment.
Why not using switch (value->GetType())?
There was a problem hiding this comment.
I'm not sure if using switch (value->GetType()) to get the string behavior would improve performance, but I might give it a try.
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
hubgeter
force-pushed
the
feature_hive_json_serde
branch
from
c10850b to
b478158
Compare
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
morningman
reviewed
Nov 13, 2024
| @@ -1040,7 +1178,8 @@ Status NewJsonReader::_read_one_message(std::unique_ptr<uint8_t[]>* file_buf, si | |||
| } | |||
| // ---------SIMDJSON---------- | |||
| // simdjson, replace none simdjson function if it is ready | |||
| Status NewJsonReader::_simdjson_init_reader() { | |||
| Status NewJsonReader::_simdjson_init_reader(bool is_load) { | |||
| _is_load = is_load; | |||
There was a problem hiding this comment.
already set in Status NewJsonReader::init_reader.
And not need to pass it from function parameter
| @@ -292,6 +296,18 @@ class NewJsonReader : public GenericReader { | |||
| std::unordered_map<std::string, std::string> _col_default_value_map; | |||
|
|
|||
| int32_t skip_bitmap_col_idx {-1}; | |||
|
|
|||
| bool _is_load = true; | |||
There was a problem hiding this comment.
Add comment in code to describe this field
|
run buildall |
github-actions
bot
removed
the
approved
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
hubgeter
force-pushed
the
feature_hive_json_serde
branch
from
7b9904d to
5e9e0ad
Compare
|
run buildall |
TPC-H: Total hot run time: 39819 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 196362 ms |
ClickBench: Total hot run time: 32.03 s |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
morningman
approved these changes
Nov 28, 2024
CalvinKirs
approved these changes
Nov 28, 2024
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Dec 2, 2024
Problem Summary:
Support reading json format hive table like:
```mysql
mysql> show create table basic_json_table;
CREATE TABLE `basic_json_table`(
`id` int,
`name` string,
`age` tinyint,
`salary` float,
`is_active` boolean,
`join_date` date,
`last_login` timestamp,
`height` double,
`profile` binary,
`rating` decimal(10,2))
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
```
Behavior changed:
To implement this feature, this pr modifies `new_json_reader`.
Previously, `new_json_reader` could only insert data into columnString.
In order to support inserting data into columns of other types,
`DataTypeSerDe` is introduced to insert data into columns. To maintain
compatibility with previous versions, changes to this pr are triggered
only when reading hive json tables.
Limitation of Use:
1. Currently, only query is supported, and writing is not supported.
2. Currently, only the `ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe';` scenario is supported. For
some properties specified in `with serdeproperties`, Doris does not take
effect.
3. Since Hive does not allow columns with the same name but different
case when creating a table in Json format (including inside a Struct),
we convert the field names in the Json data to lowercase when reading
the Json data file, and then match according to the lowercase field
names. For field names that are duplicated after being converted to
lowercase in the data, the value of the last field is used (consistent
with Hive behavior).
example:
```
create table json_table(
column int
)ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
a.json:
{"column":1,"COLumn",2,"COLUMN":3}
{"column":10,"COLumn",20}
{"column":100}
in Hive : load a.json to table json_table
in Doris query:
---
3
20
100
---
```
Todo(in next pr):
Merge `serde` and `json_reader` ,because they have logical conflicts.
Hive catalog support read json format table.
16 tasks
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Dec 4, 2024
Problem Summary:
Support reading json format hive table like:
```mysql
mysql> show create table basic_json_table;
CREATE TABLE `basic_json_table`(
`id` int,
`name` string,
`age` tinyint,
`salary` float,
`is_active` boolean,
`join_date` date,
`last_login` timestamp,
`height` double,
`profile` binary,
`rating` decimal(10,2))
ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe'
```
Behavior changed:
To implement this feature, this pr modifies `new_json_reader`.
Previously, `new_json_reader` could only insert data into columnString.
In order to support inserting data into columns of other types,
`DataTypeSerDe` is introduced to insert data into columns. To maintain
compatibility with previous versions, changes to this pr are triggered
only when reading hive json tables.
Limitation of Use:
1. Currently, only query is supported, and writing is not supported.
2. Currently, only the `ROW FORMAT SERDE
'org.apache.hive.hcatalog.data.JsonSerDe';` scenario is supported. For
some properties specified in `with serdeproperties`, Doris does not take
effect.
3. Since Hive does not allow columns with the same name but different
case when creating a table in Json format (including inside a Struct),
we convert the field names in the Json data to lowercase when reading
the Json data file, and then match according to the lowercase field
names. For field names that are duplicated after being converted to
lowercase in the data, the value of the last field is used (consistent
with Hive behavior).
example:
```
create table json_table(
column int
)ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';
a.json:
{"column":1,"COLumn",2,"COLUMN":3}
{"column":10,"COLumn",20}
{"column":100}
in Hive : load a.json to table json_table
in Doris query:
---
3
20
100
---
```
Todo(in next pr):
Merge `serde` and `json_reader` ,because they have logical conflicts.
Hive catalog support read json format table.
morningman
pushed a commit
that referenced
this pull request
Dec 4, 2024
16 tasks
16 tasks
eldenmoon
pushed a commit
that referenced
this pull request
Jan 2, 2025
fix coredump with new_json_reader if we read invalid data will core which source from #43469
github-actions bot
pushed a commit
that referenced
this pull request
Jan 2, 2025
fix coredump with new_json_reader if we read invalid data will core which source from #43469
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
Support reading json format hive table like:
Behavior changed:
To implement this feature, this pr modifies
new_json_reader. Previously,new_json_readercould only insert data into columnString. In order to support inserting data into columns of other types,DataTypeSerDeis introduced to insert data into columns. To maintain compatibility with previous versions, changes to this pr are triggered only when reading hive json tables.Limitation of Use:
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe';scenario is supported. For some properties specified inwith serdeproperties, Doris does not take effect.example:
Todo(in next pr):
Merge
serdeandjson_reader,because they have logical conflicts.Release note
Hive catalog support read json format table.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)