Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Initial DaskRunner for Beam #22421

Merged
merged 69 commits into from
Oct 25, 2022
Merged

Initial DaskRunner for Beam #22421

merged 69 commits into from
Oct 25, 2022

Commits on Oct 10, 2022

  1. WIP: Created a skeleton dask runner implementation.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    79d4603 View commit details
    Browse the repository at this point in the history

  2. WIP: Idea for a translation evaluator.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    248ec70 View commit details
    Browse the repository at this point in the history

  3. Added overrides and a visitor that translates operations.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    42452ca View commit details
    Browse the repository at this point in the history

  4. Fixed a dataclass typo.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    1da2ddd View commit details
    Browse the repository at this point in the history

  5. Expanded translations.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    14885a3 View commit details
    Browse the repository at this point in the history

  6. Core idea seems to be kinda working...

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    fca2420 View commit details
    Browse the repository at this point in the history

  7. First iteration on DaskRunnerResult (keep track of pipeline state).

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    6dd1ada View commit details
    Browse the repository at this point in the history

  8. Added minimal set of DaskRunner options.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    6675687 View commit details
    Browse the repository at this point in the history

  9. WIP: Alllmost got asserts to work! The current status is: 

    - CoGroupByKey is broken due to how tags are used with GroupByKey
- GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
- Once that is fixed, we may have test pipelines work on Dask.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    88ed36b View commit details
    Browse the repository at this point in the history

  10. With a great 1-liner from @pabloem, groupby is fixed! Now, all three … 

    …initial tests pass.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    2e3a126 View commit details
    Browse the repository at this point in the history

  11. Self-review: Cleaned up dask runner impl.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    6467b0e View commit details
    Browse the repository at this point in the history

  12. Self-review: Remove TODOs, delete commented out code, other cleanup.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    793ba86 View commit details
    Browse the repository at this point in the history

  13. First pass at linting rules.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e535792 View commit details
    Browse the repository at this point in the history

  14. WIP, include dask dependencies + test setup.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    8e32668 View commit details
    Browse the repository at this point in the history

  15. WIP: maybe better dask deps?

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    318afc2 View commit details
    Browse the repository at this point in the history

  16. Skip dask tests depending on successful import.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    b01855f View commit details
    Browse the repository at this point in the history

  17. Fixed setup.py (missing ,).

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    2c2eb8d View commit details
    Browse the repository at this point in the history

  18. Added an additional comma.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e64e9eb View commit details
    Browse the repository at this point in the history

  19. Moved skipping logic to be above dask import.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    69b118b View commit details
    Browse the repository at this point in the history

  20. Fix lint issues with dask runner tests.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    9ffc8d8 View commit details
    Browse the repository at this point in the history

  21. Adding destination for client address.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    8a2afb7 View commit details
    Browse the repository at this point in the history

  22. Changing to async produces a timeout error instead of stuck in infini… 

    …te loop.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    93f02f1 View commit details
    Browse the repository at this point in the history

  23. Close client during wait_until_finish; rm async.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    afdcf1b View commit details
    Browse the repository at this point in the history

  24. Supporting side-inputs for ParDo.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    41b5267 View commit details
    Browse the repository at this point in the history

  25. Revert "Close client during wait_until_finish; rm async." 

    This reverts commit 09365f6.
    @pabloem @alxmrs
    pabloem authored and alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e3ac3f8 View commit details
    Browse the repository at this point in the history

  26. Revert "Changing to async produces a timeout error instead of stuck i… 

    …n infinite loop."

This reverts commit 676d752.
    @pabloem @alxmrs
    pabloem authored and alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    3fddc81 View commit details
    Browse the repository at this point in the history

  27. Adding -dask tox targets onto the gradle build

    @pabloem @alxmrs
    pabloem authored and alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    9eeb9ea View commit details
    Browse the repository at this point in the history

  28. wip - added print stmt.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    b4d0999 View commit details
    Browse the repository at this point in the history

  29. wip - prove side inputs is set.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    0319ffd View commit details
    Browse the repository at this point in the history

  30. wip - prove side inputs is set in Pardo.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    0b13bb0 View commit details
    Browse the repository at this point in the history

  31. wip - rm asserts, add print

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    1e7052b View commit details
    Browse the repository at this point in the history

  32. wip - adding named inputs...

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    292e023 View commit details
    Browse the repository at this point in the history

  33. Experiments: non-named side inputs + del None in named inputs.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    31c1e2b View commit details
    Browse the repository at this point in the history

  34. None --> 'None'

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    f4ecf2f View commit details
    Browse the repository at this point in the history

  35. No default side input.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    4d24ed9 View commit details
    Browse the repository at this point in the history

  36. Pass along args + kwargs.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    ee62a4a View commit details
    Browse the repository at this point in the history

  37. Applied yapf to dask sources.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    506c719 View commit details
    Browse the repository at this point in the history

  38. Dask sources passing pylint.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    cd0ba8b View commit details
    Browse the repository at this point in the history

  39. Added dask extra to docs gen tox env.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    d0a7c63 View commit details
    Browse the repository at this point in the history

  40. Applied yapf from tox.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    775bd07 View commit details
    Browse the repository at this point in the history

  41. Include dask in mypy checks.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    efba1c9 View commit details
    Browse the repository at this point in the history

  42. Upgrading mypy support to python 3.8 since py37 support is deprecated… 

    … in dask.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    741d961 View commit details
    Browse the repository at this point in the history

  43. Manually installing an old version of dask before 3.7 support was dro… 

    …pped.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    f66458a View commit details
    Browse the repository at this point in the history

  44. fix lint: line too long.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    5dcf969 View commit details
    Browse the repository at this point in the history

  45. Fixed type errors with DaskRunnerResult. Disabled mypy type checking … 

    …in dask.
    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    ec5f613 View commit details
    Browse the repository at this point in the history

  46. Fix pytype errors (in transform_evaluator).

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    04b1f1a View commit details
    Browse the repository at this point in the history

  47. Ran isort.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    712944b View commit details
    Browse the repository at this point in the history

  48. Ran yapf again.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    567b72b View commit details
    Browse the repository at this point in the history

  49. Fix imports (one per line)

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    f53c0a4 View commit details
    Browse the repository at this point in the history

  50. isort -- alphabetical.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    fb280ad View commit details
    Browse the repository at this point in the history

  51. Added feature to CHANGES.md.

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    80ddfec View commit details
    Browse the repository at this point in the history

  52. ran yapf via tox on linux machine

    @alxmrs
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    40c4e35 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2022

  1. Merge branch 'master' into dask-runner-mvp

    @alxmrs
    alxmrs authored Oct 13, 2022
    Configuration menu
    Copy the full SHA
    a70c5f3 View commit details
    Browse the repository at this point in the history

  2. Change an import to pass CI.

    @alxmrs
    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    9fb52e5 View commit details
    Browse the repository at this point in the history

  3. Skip isort error; needed to get CI to pass.

    @alxmrs
    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    26c6016 View commit details
    Browse the repository at this point in the history

  4. Skip test logic may favor better with isort.

    @alxmrs
    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    aec19bf View commit details
    Browse the repository at this point in the history

  5. (Maybe) the last isort fix.

    @alxmrs
    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    0673235 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2022

  1. Tested pipeline options (added one fix).

    @alxmrs
    alxmrs committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    de03a32 View commit details
    Browse the repository at this point in the history

  2. Improve formatting of test.

    @alxmrs
    alxmrs committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    7e0a2c7 View commit details
    Browse the repository at this point in the history

  3. Self-review: removing side inputs. 

    In addition, adding a more helpful property to the base DaskBagOp (tranform).
    @alxmrs
    alxmrs committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    39b1e1c View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2022

  1. add dask to coverage suite in tox.

    @alxmrs
    alxmrs committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    6db49fa View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2022

  1. Merge branch 'master' into dask-runner-mvp

    @alxmrs
    alxmrs authored Oct 18, 2022
    Configuration menu
    Copy the full SHA
    036561c View commit details
    Browse the repository at this point in the history

  2. Capture value error in assert.

    @alxmrs
    alxmrs committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    191580d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:apache/beam into dask-runner-mvp

    @alxmrs
    alxmrs committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    365fc87 View commit details
    Browse the repository at this point in the history

  4. Change timeout value to 600 seconds.

    @alxmrs
    alxmrs committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    085447e View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2022

  1. ignoring broken test

    @pabloem
    pabloem authored Oct 21, 2022
    Configuration menu
    Copy the full SHA
    1a60a5e View commit details
    Browse the repository at this point in the history

  2. Update CHANGES.md

    @pabloem
    pabloem authored Oct 21, 2022
    Configuration menu
    Copy the full SHA
    c1037f8 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2022

  1. Using reflection to test the Dask client constructor.

    @alxmrs
    alxmrs committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    9e79ffd View commit details
    Browse the repository at this point in the history

  2. Better method of inspecting the constructor parameters (thanks @TomAu… 

    …gspurger!).
    @alxmrs
    alxmrs committed Oct 24, 2022
    Configuration menu
    Copy the full SHA
    f9cf45a View commit details
    Browse the repository at this point in the history