Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files #2373

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open

[WIP][CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoid sorting shuffle files #2373

wants to merge 31 commits into from

Commits on Jun 1, 2024

  1. [CELEBORN-1319] Optimize skew partition logic for Reduce Mode to avoi… 

    …d sorting shuffle files
    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    49ab9b0 View commit details
    Browse the repository at this point in the history

  2. fix unit test

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    d028cd0 View commit details
    Browse the repository at this point in the history

  3. fix ut and refactor code

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    d587e39 View commit details
    Browse the repository at this point in the history

  4. refactor code and imports

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    9da2ab8 View commit details
    Browse the repository at this point in the history

  5. fix NPE and remove unused code and fix scala 2.11 compile error

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    19ac3d0 View commit details
    Browse the repository at this point in the history

  6. format code

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    b13abd7 View commit details
    Browse the repository at this point in the history

  7. add spark 3.3 patch

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    10cde4b View commit details
    Browse the repository at this point in the history

  8. remove unused log and refactor code

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    a122e9d View commit details
    Browse the repository at this point in the history

  9. add unit tests and refactor code

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    019644c View commit details
    Browse the repository at this point in the history

  10. split the skewed partition based on the chunk range

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    b6e25b2 View commit details
    Browse the repository at this point in the history

  11. remove CelebornInputStreamSuiteJ

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    b8cc28f View commit details
    Browse the repository at this point in the history

  12. fix ut

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    26df6f7 View commit details
    Browse the repository at this point in the history

  13. refactor code for batch open stream

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    86072c5 View commit details
    Browse the repository at this point in the history

  14. Revert "refactor code for batch open stream" 

    This reverts commit 7393d45.
    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    e11d54e View commit details
    Browse the repository at this point in the history

  15. celeborn.client.dataPushFailure.tracking.enabled -> `celeborn.clien… 

    …t.adaptive.optimizeSkewedPartitionRead.enabled`
    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    ff760b3 View commit details
    Browse the repository at this point in the history

  16. refactor code for batch open stream

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    842c023 View commit details
    Browse the repository at this point in the history

  17. add ut and refactor split logic

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    5f65fbe View commit details
    Browse the repository at this point in the history

  18. refactor code according to review suggestions

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    da17e68 View commit details
    Browse the repository at this point in the history

  19. add sbt dependency

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    54fe9b7 View commit details
    Browse the repository at this point in the history

  20. update spark patch

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    20b6aa1 View commit details
    Browse the repository at this point in the history

  21. address comment

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    822e231 View commit details
    Browse the repository at this point in the history

  22. fix

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    1c9899a View commit details
    Browse the repository at this point in the history

  23. add ut

    @wangshengjie123
    wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    012c53b View commit details
    Browse the repository at this point in the history

  24. update

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    8ef1da8 View commit details
    Browse the repository at this point in the history

  25. refactor

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    b8037a0 View commit details
    Browse the repository at this point in the history

  26. add CelebornPartitionUtil.java

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    f053732 View commit details
    Browse the repository at this point in the history

  27. fix ut for jdk 8

    @cfmcgrady @wangshengjie123
    cfmcgrady authored and wangshengjie123 committed Jun 1, 2024
    Configuration menu
    Copy the full SHA
    42f10f8 View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2024

  1. rebase main and fix npe

    @wangshengjie123
    wangshengjie123 committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    b3a30be View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. fix ut

    @wangshengjie123
    wangshengjie123 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    f5715c4 View commit details
    Browse the repository at this point in the history

  2. fix npe when memory storage enabled

    @wangshengjie123
    wangshengjie123 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    eef849a View commit details
    Browse the repository at this point in the history

  3. fix code style check error

    @wangshengjie123
    wangshengjie123 committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    81947e7 View commit details
    Browse the repository at this point in the history