Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add execution contexts #69

Draft
wants to merge 21 commits into
base: 4.0.x
Choose a base branch
from
Draft

add execution contexts #69

wants to merge 21 commits into from

Conversation

leo-schick
Copy link
Member

In WIP feature discussion in #68

@leo-schick leo-schick self-assigned this Apr 25, 2022
@leo-schick
Copy link
Member Author

leo-schick commented Apr 26, 2022

Current status

the BashExecutionContext and SshBashExecutionContext works, but logging should be optimized

Sample app

local_config.py

from mara_pipelines.contexts.bash import BashExecutionContext, SshBashExecutionContext
patch(mara_pipelines.config.execution_contexts)(lambda: {
    'bash': BashExecutionContext(),
    'srv-mara-2': SshBashExecutionContext(
        host='srv-mara-2'
    ),
})

test.py

from mara_pipelines.pipelines import Pipeline, Task
from mara_pipelines.commands.bash import RunBash

test_pipeline = Pipeline(
    id='test',
    description="",
    context='srv-mara-2')

test_pipeline.add(
    Task(id='ls',
         description="",
         commands=[
             RunBash('ls -l')
         ]))

test_pipeline.add(
    Task(id='ls2',
         description="",
         commands=[
             RunBash('ls -l .venv')
         ],
         context='bash'),
    upstreams=['ls'])

Live execution output

image

Output after execution:

image

Open tasks

  • 🐛 the log message exit execution context '{context_alias}' is not saved in the database log
  • 🚀 use better logging, don't use print, see mara_pipelines.execution.py
  • 🚀 add logging events for enter/exit of a execution context
  • 🚀 start/stop docker container when not started when calling method __enter__
  • 🚀 show somehwere in the nodes page which execution context is used

@leo-schick leo-schick force-pushed the feat-execution-context branch from 0766344 to dca0bb9 Compare May 4, 2022 19:26
leo-schick added a commit to leo-schick/mara-singer that referenced this pull request May 5, 2022
@leo-schick leo-schick force-pushed the feat-execution-context branch 2 times, most recently from 5954023 to e8e3c76 Compare May 5, 2022 14:22
@leo-schick
Copy link
Member Author

Currently known issue: when a execution context is used which is not defined in the configuration the pipeline execution does not throw any message and just hangs

@leo-schick leo-schick mentioned this pull request May 30, 2022
@leo-schick leo-schick force-pushed the feat-execution-context branch from 453d7fc to 74fa203 Compare July 11, 2022 14:55
leo-schick added a commit to leo-schick/mara-singer that referenced this pull request Sep 1, 2022
@leo-schick leo-schick force-pushed the feat-execution-context branch from 74fa203 to b926b32 Compare October 10, 2022 07:49
@leo-schick leo-schick changed the base branch from master to 4.0.x October 10, 2022 07:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants