[HUDI-8799] Design of RFC-84, Optimized SerDe of DataStream in Flink operators
#12697
Conversation
| ### Potential problems | ||
|
|
||
| 1. Key generators are hardly coupled with Avro `GenericRecord`. | ||
| Therefore, to support all key generators we will have to do intermediate conversion into Avro in operator, that is responsible for getting Hudi key. |
There was a problem hiding this comment.
Is there already a POC now, can we do a micro-benchmark to prove the gains.
There was a problem hiding this comment.
Yes, but I will work on it a little bit more, and will provide benchmark results with profiling on this week. Also I will provide corresponding PR to check code changes.
There was a problem hiding this comment.
@danny0405 , PR is ready: #12722.
In the description to this PR, I've mentioned costs from this extra conversion:
These costs could be accepted for now due to acceptable values: 5 798 CPU samples from 183 236 in total, which is about 3%.
Total performance improvement:
- total write time decreased from 344 s to 265 s, which is about 23%,
- data passed between Flink operators decreased from 19.4 GB to 12.9 GB, which is about 33.5%.
There was a problem hiding this comment.
@cshuo is recently working on a new RFC to add basic abstractions of schema/data type/expressions to Hudi, so that we can integrate with the engine specific "row" for both the writer and reader, the design doc would be coming out, will cc you if you have intreast in it, it's a huge task and maybe you can help with it.
There was a problem hiding this comment.
@cshuo , hi! If doesn't bother you, could you, please, check this RFC for design conflicts with your RFC, which is in progress. If there is no conflicts, I propose to move these changes further.
Otherwise, could you, please, provide some drafts, to start to work in collaboration.
There was a problem hiding this comment.
@geserdugarov Sorry for the late reply, just came back from the lunar new year holiday :) I'll take a look at your rfc and benchmark code soon.
There was a problem hiding this comment.
@geserdugarov here is RFC pr #12795, welcome to review and get involved to collaborate.
geserdugarov
force-pushed
the
master-rfc-flink-serde
branch
from
e375fd5 to
cae2226
Compare
|
@hudi-bot run azure |
geserdugarov
force-pushed
the
master-rfc-flink-serde
branch
from
cae2226 to
af1ce0f
Compare
geserdugarov
force-pushed
the
master-rfc-flink-serde
branch
from
845b519 to
5e0f4ee
Compare
github-actions
bot
added
size:L
|
@danny0405 , @cshuo , I've updated description here with added structure of I've already implemented optimization for simple bucket index and non bucket case. There is only the last one left, consistent hashing case. Opened corresponding PR: #12796. Assumption that I could do everything without custom serializer was wrong. I've faced issues with serde during conversion of |
Change Logs
Proposed optimization to reduce SerDe costs for Flink operators.
Proof of concept was presented in the corresponding claim of RFC.
For stream write into Hudi with simple bucket index, total write time decreased by 15%, which is significant for stream processing.
Impact
None for this stage.
Risk level (write none, low medium or high below)
None for this stage.
Documentation Update
No need for this stage.
Contributor's checklist