Implement globbing for script_dirs

[Haegi]

Fixes #15
This would be a globbing approach to make the script_dirs parameter more configurable.

Example

Pipeline

from pipeline_dsl import Pipeline

with Pipeline("test", script_dirs={"yaml_files": "*.yaml"}) as pipeline:
    with pipeline.job("list-yaml-files") as job:
        @job.task()
        def list_yaml_files():
            print(pipeline.script_dir("yaml_files"))

Output

Running: list-yaml-files
['first-yaml-files.yaml', 'second-yaml-file.yaml']
None

[c0d1ngm0nk3y]

Isn't it a bit confusing that glob only work if you do not a use a list? Is there a reason?

[Haegi]

You are right. I should implement that as well. But I didn't implement it because we haven't decided if we want to go with the globbing approach.

[c0d1ngm0nk3y]

I think we can go that way. AFAIK, I was the only one promoting the other way, right :) The only think is, that this approach is not compatible, so we would have to change our pipelines. Otherwise, we might end up with binaries in the pipeline, right?

[Haegi]

If implemented globbing for script_dirs list in 36b7efc.
As there is no option for the user to specify a key, the key will be created by:

• using the dirname if no specific/direct file is specified (e.g. glob or directory)
  • Example: test for test/*.yaml or test for test/
• using the specified file as key (e.g. test.yaml)
  • Example: test.yaml for test.yaml or test/test.yaml' for test/test.yaml`

[c0d1ngm0nk3y]

globed is a list?

[Haegi]

Yes, because it can contain multiple files.

[c0d1ngm0nk3y]

@Haegi Did I get it right - if you want several files, you need something like folder/*.[sh|py], right?

[Haegi]

@Haegi Did I get it right - if you want several files, you need something like folder/*.[sh|py], right?

@c0d1ngm0nk3y Unfortunately, the python glob module doesn't support multiple file types.pathlib.Path().glob() would be an alternative but this module does only support relative paths.

[Haegi]

I've found out that pathlib.Path().glob() also doesn't work as expected. To continue I would propose to:

• either leave it like it is and search later for a solution
• implement it on our own
• or use wcmatch extended globbing capabilities (e.g. *.@(py|sh)) which would work but this would introduce a dependency. For more details see here.

@c0d1ngm0nk3y What's your opinion on how we should continue?

[c0d1ngm0nk3y]

• implement it on our own

Sounds good to me. I doubt that it will be a problem. But if, we can tackle it then…

[phil9909]

Is this PR mixing two concerns?

• Supporting globs for script_dir
• Supporting lists for script_dir

Why do it in one PR?

If we do it in one PR, we should transform the list into an object instead of having multiple isinstance(x, list) calls.

Lets sort this out in a call.