-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(dask): Dask integration (#467) #467
base: master
Are you sure you want to change the base?
Conversation
3aa482c
to
3aa9dfd
Compare
3aa9dfd
to
50952aa
Compare
50952aa
to
c67d08a
Compare
c67d08a
to
8caeb77
Compare
@@ -4,6 +4,7 @@ The list of contributors in alphabetical order: | |||
|
|||
- [Adelina Lintuluoto](https://orcid.org/0000-0002-0726-1452) | |||
- [Agisilaos Kounelis](https://orcid.org/0000-0001-9312-3189) | |||
- [Alp Tuna](https://orcid.org/0009-0001-1915-3993) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can add yourself in the same commit that introduced functionality. It is not necessary to split this into two commits.
"type": "object", | ||
"title": "Dask cluster to be used by the workflow", | ||
"properties": { | ||
"image": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please introduce also help for the various properties such as image
. (By means of description
fields, see .)
f3206b5
to
b3e522b
Compare
b05a124
to
b8bb2a9
Compare
"maximum_kubernetes_jobs_timeout", | ||
"maximum_workspace_retention_period", | ||
"workspaces_available", | ||
"dask_enabled" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that testing the reana-client
with this reana-commons
PR does not work against "older" REANA clusters that do not contain Dask support, I mean even with non-Dask workflows, for example:
$ cd reana-demo-root6-roofit
$ export REANA_SERVER_URL=https://reana.cern.ch
$ read -s REANA_ACCESS_TOKEN
$ reana-client create -w test
==> ERROR: Cannot create workflow test:
'dask_enabled' is a required property
It would be good to make the client workable against both "old" (Dask-unaware) and "new" (Dask-aware) clusters.
Basically, we have four cases in the test matrix to consider: old client <-> old cluster, old client <-> new cluster, new client <-> old cluster, new client <-> new cluster. The Dask support is "optional" and so ideally all these combinations should keep working.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I removed the required
part from the openapi spec and now it should be working fine with all combinations. I tested with several examples and all should be working fine.
973dc63
to
923930b
Compare
923930b
to
4d39acf
Compare
4d39acf
to
2ea5d7a
Compare
2ea5d7a
to
84b92f6
Compare
This pull request contains the ongoing work of dask integration into REANA.