feat(dask): Dask integration (#467) #467
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #467 +/- ##
=======================================
Coverage 44.23% 44.23%
=======================================
Files 28 28
Lines 1978 1978
=======================================
Hits 875 875
Misses 1103 1103 |
Alputer
changed the title
Alputer
force-pushed
the
dask-integration
branch
from
3aa482c
to
3aa9dfd
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Aug 28, 2024
Alputer
force-pushed
the
dask-integration
branch
from
3aa9dfd
to
50952aa
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Aug 29, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Aug 29, 2024
Alputer
force-pushed
the
dask-integration
branch
from
50952aa
to
c67d08a
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 5, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 5, 2024
Alputer
force-pushed
the
dask-integration
branch
from
c67d08a
to
8caeb77
Compare
tiborsimko
reviewed
Sep 9, 2024
| @@ -4,6 +4,7 @@ The list of contributors in alphabetical order: | |||
|
|
|||
| - [Adelina Lintuluoto](https://orcid.org/0000-0002-0726-1452) | |||
| - [Agisilaos Kounelis](https://orcid.org/0000-0001-9312-3189) | |||
| - [Alp Tuna](https://orcid.org/0009-0001-1915-3993) | |||
There was a problem hiding this comment.
You can add yourself in the same commit that introduced functionality. It is not necessary to split this into two commits.
| "type": "object", | ||
| "title": "Dask cluster to be used by the workflow", | ||
| "properties": { | ||
| "image": { |
There was a problem hiding this comment.
Please introduce also help for the various properties such as image. (By means of description fields, see .)
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 9, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 9, 2024
Alputer
force-pushed
the
dask-integration
branch
from
f3206b5
to
b3e522b
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 13, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 15, 2024
Alputer
force-pushed
the
dask-integration
branch
from
b05a124
to
b8bb2a9
Compare
This was referenced Sep 20, 2024
tiborsimko
reviewed
Sep 23, 2024
| "maximum_kubernetes_jobs_timeout", | ||
| "maximum_workspace_retention_period", | ||
| "workspaces_available", | ||
| "dask_enabled" |
There was a problem hiding this comment.
Note that testing the reana-client with this reana-commons PR does not work against "older" REANA clusters that do not contain Dask support, I mean even with non-Dask workflows, for example:
$ cd reana-demo-root6-roofit
$ export REANA_SERVER_URL=https://reana.cern.ch
$ read -s REANA_ACCESS_TOKEN
$ reana-client create -w test
==> ERROR: Cannot create workflow test:
'dask_enabled' is a required propertyIt would be good to make the client workable against both "old" (Dask-unaware) and "new" (Dask-aware) clusters.
Basically, we have four cases in the test matrix to consider: old client <-> old cluster, old client <-> new cluster, new client <-> old cluster, new client <-> new cluster. The Dask support is "optional" and so ideally all these combinations should keep working.
There was a problem hiding this comment.
I removed the required part from the openapi spec and now it should be working fine with all combinations. I tested with several examples and all should be working fine.
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 25, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Sep 25, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 16, 2024
Alputer
force-pushed
the
dask-integration
branch
from
973dc63
to
923930b
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
force-pushed
the
dask-integration
branch
from
923930b
to
4d39acf
Compare
Alputer
added a commit
to Alputer/reana-commons
that referenced
this pull request
Oct 24, 2024
Alputer
force-pushed
the
dask-integration
branch
from
4d39acf
to
2ea5d7a
Compare
Alputer
force-pushed
the
dask-integration
branch
from
2ea5d7a
to
84b92f6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request contains the ongoing work of dask integration into REANA.