[Enhancement] implement streamload via http stream #45639
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
be/src/http/action/stream_load.cpp
Outdated
| sql_parts.emplace_back("SELECT *"); | ||
| } | ||
|
|
||
| sql_parts.emplace_back("FROM http_stream(\"format\" = \"" + format + |
There was a problem hiding this comment.
warning: escaped string literal can be written as a raw string literal [modernize-raw-string-literal]
Suggested change
| sql_parts.emplace_back("FROM http_stream(\"format\" = \"" + format + | |
| sql_parts.emplace_back(R"(FROM http_stream("format" = ")" + format + |
be/src/http/action/stream_load.cpp
Outdated
| } | ||
|
|
||
| sql_parts.emplace_back("FROM http_stream(\"format\" = \"" + format + | ||
| "\", \"column_separator\" = \"" + column_separator + "\")"); |
There was a problem hiding this comment.
warning: escaped string literal can be written as a raw string literal [modernize-raw-string-literal]
Suggested change
| "\", \"column_separator\" = \"" + column_separator + "\")"); | |
| R"(", "column_separator" = ")" + column_separator + "\")"); |
be/src/http/action/stream_load.cpp
Outdated
| std::string full_sql = "INSERT INTO " + table_name + " "; | ||
| for (size_t i = 0; i < sql_parts.size(); i++) { | ||
| full_sql += sql_parts[i]; | ||
| if (i != sql_parts.size() - 1) full_sql += ", "; |
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (i != sql_parts.size() - 1) full_sql += ", "; | |
| if (i != sql_parts.size() - 1) { full_sql += ", "; | |
| } |
|
run buildall |
1 similar comment
|
run buildall |
TPC-H: Total hot run time: 39988 ms |
TPC-DS: Total hot run time: 195790 ms |
ClickBench: Total hot run time: 32.89 s |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 39909 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 197541 ms |
ClickBench: Total hot run time: 33.09 s |
Tech-Circle-48
force-pushed
the
sss
branch
2 times, most recently
from
804640e to
a40258a
Compare
|
run buildall |
4 similar comments
|
run buildall |
|
run buildall |
|
run buildall |
|
run buildall |
TPC-H: Total hot run time: 39855 ms |
TPC-DS: Total hot run time: 189873 ms |
ClickBench: Total hot run time: 33.36 s |
|
TeamCity be ut coverage result: |
|
run buildall |
1 similar comment
|
run buildall |
|
run buildall |
1 similar comment
|
run buildall |
TPC-H: Total hot run time: 40306 ms |
|
TeamCity be ut coverage result: |
TPC-DS: Total hot run time: 197354 ms |
ClickBench: Total hot run time: 32.49 s |
|
run buildall |
TPC-H: Total hot run time: 32920 ms |
TPC-DS: Total hot run time: 197038 ms |
ClickBench: Total hot run time: 31.58 s |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 32845 ms |
TPC-DS: Total hot run time: 197464 ms |
ClickBench: Total hot run time: 30.98 s |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 32745 ms |
TPC-DS: Total hot run time: 196563 ms |
ClickBench: Total hot run time: 31.7 s |
|
TeamCity be ut coverage result: |
|
run buildall |
TPC-H: Total hot run time: 32749 ms |
TPC-DS: Total hot run time: 190795 ms |
ClickBench: Total hot run time: 32 s |
|
TeamCity be ut coverage result: |
Tech-Circle-48
force-pushed
the
sss
branch
2 times, most recently
from
ae3af22 to
25a752e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #36773
Related PR: #xxx
Problem Summary:
1. New Configuration:
• Added enable_streamload_by_httpstream configuration option.
• When enabled, Stream Load requests are processed using the HTTP Stream approach.
2. Code Modifications:
• Updated the _process_put method in stream_load.cpp:
• Added logic to construct the SQL statement from Stream Load headers.
• Assigned the generated SQL statement to the ctx (context object).
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)