Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

validate: detecting misplaced environment clause for Serial workflows #679

Closed
tiborsimko opened this issue Oct 25, 2023 · 2 comments · Fixed by reanahub/reana-commons#415, #693 or reanahub/reana-ui#366 · May be fixed by reanahub/reana-server#654
Closed

validate: detecting misplaced environment clause for Serial workflows #679

tiborsimko opened this issue Oct 25, 2023 · 2 comments · Fixed by reanahub/reana-commons#415, #693 or reanahub/reana-ui#366 · May be fixed by reanahub/reana-server#654
Assignees
@tiborsimko @giuseppe-steduto

Comments

@tiborsimko
Copy link
Member

The following Serial workflow is wrong; note the misplaced "environment" clause for the first step:

version: 0.6.0
inputs:
  files:
    - code/gendata.C
    - code/fitdata.C
  parameters:
    events: 20000
    data: results/data.root
    plot: results/plot.png
workflow:
  type: serial
  specification:
    environment:
      name: 'docker.io/reanahub/reana-env-root6:6.18.04'
    steps:
      - name: gendata
        kubernetes_memory_limit: '256Mi'
        commands:
        - mkdir -p results && root -b -q 'code/gendata.C(${events},"${data}")'
      - name: fitdata
        environment: 'docker.io/reanahub/reana-env-root6:6.18.04'
        kubernetes_memory_limit: '256Mi'
        commands:
        - root -b -q 'code/fitdata.C("${data}","${plot}")'
outputs:
  files:
    - results/plot.png

However it appears to pass the validation:

$ reana-client validate
==> Verifying REANA specification file... reana.yaml
  -> SUCCESS: Valid REANA specification file.
==> Verifying REANA specification parameters...
  -> SUCCESS: REANA specification parameters appear valid.
==> Verifying workflow parameters and commands...
  -> SUCCESS: Workflow parameters and commands appear valid.
==> Verifying dangerous workflow operations...
  -> SUCCESS: Workflow operations appear valid.

And the execution obviously fails:

$ reana-client logs -w test

==> Workflow engine logs
Workflow exited unexpectedly.
'environment'
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Oct 26, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
5750fce 
Closes reanahub/reana-client#679.
@giuseppe-steduto giuseppe-steduto mentioned this issue Oct 26, 2023
Merged
@giuseppe-steduto giuseppe-steduto self-assigned this Oct 26, 2023
@mdonadoni mdonadoni self-assigned this Oct 26, 2023
@mdonadoni mdonadoni assigned tiborsimko and unassigned mdonadoni Nov 8, 2023
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 8, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
3d5b5a6 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 13, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
7a4e54d 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 13, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
269a6c5 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 14, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
4ea3b69 
Closes reanahub/reana-client#679.
@tiborsimko
Copy link
Member Author

  1. If we go for fuller validation of workflow.specification.steps for Serial workflows, then here is I think the fairly complete list of clauses that we allow in the Serial steps specification:
  • commands: list of strings (mandatory)
  • compute_backend: string
  • environment: string (mandatory)
  • htcondor_accounting_group: string
  • htcondor_max_runtime: string
  • kerberos: boolean
  • kubernetes_job_timeout: string
  • kubernetes_memory_limit: string
  • name: string
  • rucio: boolean
  • unpacked_img: boolean
  • voms_proxy: boolean

If you encounter something else, then a WARNING should be emitted. (But no ERROR so that we can add new optional keywords to the list and so that old clients would work with newer servers.)

  1. We could also configure a list of keywords that should not be there in workflow.specification.steps for the Serial workflow engine in order to catch some common troubles:
  • inputs
  • outputs
  • resources
  1. We could also validate top-level workflow.resources clause that may be empty or may contain one of the following:
  • cvmfs: list of strings
  • kerberos: boolean

giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 21, 2023
@giuseppe-steduto
validation: make environment property mandatory in serial steps
9a314a6 
Closes reanahub/reana-client#679.
@giuseppe-steduto
Copy link
Member

Great list, thanks!

  • kubernetes_job_timeout: string

Since it's the number of seconds, this should be an integer, right? (https://docs.reana.io/advanced-usage/compute-backends/kubernetes/#custom-job-timeouts)

giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 21, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
d948545 
Change validation.utils so that the warning message for additional unexpected properties
also includes the path in reana.yaml where the unexpected property was found.

Closes reanahub#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 21, 2023
@giuseppe-steduto
validation: add workflow-engine specific clauses to specification schema
b53c3b0 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
47a1e03 
Change validation.utils so that the warning message for additional unexpected properties
also includes the path in reana.yaml where the unexpected property was found.

Closes reanahub#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
588ab5a 
Change the warning message generated when unexpected properties are
found in the REANA specification during the validation phase to include
the path to the invalidation properties.

Closes reanahub#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
a6a4974 
Change the warning message generated when unexpected properties are
found in the REANA specification during the validation phase to include
the path to the invalidating properties.

Closes reanahub#679.
@giuseppe-steduto giuseppe-steduto mentioned this issue Nov 22, 2023
Merged
giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
5f52cae 
Change the warning message generated when unexpected properties are
found in the REANA specification during the validation phase to include
the path to the invalidating properties.

Closes reanahub#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-ui that referenced this issue Nov 22, 2023
@giuseppe-steduto
launch: improve validation warning for unexpected properties
f8f7ba1 
Change the validation warning issued when unexpected properties are
found in the REANA specification to include the path to were the
invalidating keywords were found.

Closes reanahub/reana-client#679.
@giuseppe-steduto giuseppe-steduto mentioned this issue Nov 22, 2023
Merged
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: add workflow-engine specific clauses to specification schema
46c0698 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 22, 2023
@giuseppe-steduto
validation: add workflow-engine specific clauses to specification schema
4d1147f 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-commons that referenced this issue Nov 28, 2023
@giuseppe-steduto
validation: add workflow-engine specific clauses to specification schema
2e5df37 
Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-ui that referenced this issue Nov 28, 2023
@giuseppe-steduto
launch: improve validation warning for unexpected properties
153562e 
Change the validation warning issued when unexpected properties are
found in the REANA specification to include the path to were the
invalidating keywords were found.

Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-ui that referenced this issue Nov 28, 2023
@giuseppe-steduto
launch: improve validation warning for unexpected properties
47bdd5e 
Change the validation warning issued when unexpected properties are
found in the REANA specification to include the path to were the
invalidating keywords were found.

Closes reanahub/reana-client#679.
giuseppe-steduto added a commit to giuseppe-steduto/reana-client that referenced this issue Nov 30, 2023
@giuseppe-steduto
validation: improve warning message for unexpected properties
208a8e6 
Change the warning message generated when unexpected properties are
found in the REANA specification during the validation phase to include
the path to the invalidating properties.

Closes reanahub#679.
mdonadoni added a commit to mdonadoni/reana-server that referenced this issue Dec 5, 2023
@mdonadoni
launch: update openapi spec of validation warnings
b2d2625 
Closes reanahub/reana-client#679
@mdonadoni mdonadoni mentioned this issue Dec 5, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tiborsimko tiborsimko

@giuseppe-steduto giuseppe-steduto

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@tiborsimko @mdonadoni @giuseppe-steduto