[feature](hive)Support read hive4 transaction tables. #44001
+1,453
−150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
clang-tidy review says "All clean, LGTM! 👍" |
hubgeter
force-pushed
the
feature_hive4_acid
branch
from
7669f81 to
2dec82f
Compare
hubgeter
commented
Dec 11, 2024
| //Since the hive3 library cannot read the hive4 transaction table normally, and there are many problems | ||
| // when using the Hive 4 library directly, this method is implemented. | ||
| //Ref: hive/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState | ||
| public static FileCacheValue getAcidState(RemoteFileSystem fileSystem, HivePartition partition, |
hubgeter
force-pushed
the
feature_hive4_acid
branch
from
aacbf79 to
2a523cf
Compare
|
clang-tidy review says "All clean, LGTM! 👍" |
1 similar comment
|
clang-tidy review says "All clean, LGTM! 👍" |
hubgeter
changed the title
|
run buildall |
1 similar comment
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
morningman
reviewed
Dec 17, 2024
| } else { | ||
| throw new RuntimeException(status.getErrMsg()); | ||
| } | ||
| fileCacheValues.add(AcidUtil.getAcidState( |
| } catch (Exception e) { | ||
| // Release shared load (getValidWriteIds acquire Lock). | ||
| // If no exception is throw, the lock will be released when `finalizeQuery()`. | ||
| Env.getCurrentHiveTransactionMgr().deregister(hiveTransaction.getQueryId()); |
There was a problem hiding this comment.
It is not a good design to deregister transaction here.
Looks like it is very error prone
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
morningman
previously approved these changes
Dec 20, 2024
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
run buildall |
github-actions
bot
removed
the
approved
16 tasks
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 39528 ms |
TPC-DS: Total hot run time: 196848 ms |
yiguolei
pushed a commit
that referenced
this pull request
Dec 22, 2024
### What problem does this PR solve? bp #44001 , but no hive4 acid table. Problem Summary: 1. Fixed the issue that when reading insert translaction only tables, there was no acid check, which caused multiple data reads (i.e., reading data from the previous base_n). 2. Forbidden to create, insert data, and delete aicd tables.
|
run buildall |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 32510 ms |
TPC-DS: Total hot run time: 197590 ms |
ClickBench: Total hot run time: 31.52 s |
morningman
force-pushed
the
feature_hive4_acid
branch
from
453bd6a to
dbf1765
Compare
|
run buildall |
TPC-H: Total hot run time: 32289 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 197273 ms |
ClickBench: Total hot run time: 31.11 s |
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
Support read hive4 transaction tables.
Since there are compilation problems when directly using hive4 API to get the file need to read, a function called
ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidStateis implemented.At the same time, it is forbidden to create, insert data, and delete aicd tables.
fix
'transactional_properties' = 'insert_only'table read Incorrect data read.TODO : mege HMSTransaction ,HiveTransaction class
mege HiveTransactionMgr, HiveTransactionManager class
Release note
Support read hive4 transaction tables.
Fixed the problem that the result of reading insert only transaction table in Hive catalog is incorrect.
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)