nomad variables as nextflow secrets

[jagedn]

Following our internal discussion I would like to write down the idea and concerns about it

Say we have following pipeline

process sayHello {
    container   'ubuntu:20.04'
    secret 'MY_ACCESS_KEY'
    secret 'MY_SECRET_KEY'

    input:
    val x
    output:
    stdout
    script:
    """
    echo '$x world!'
    """
}

workflow {
    Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | sayHello | view
}

In this pipeline Nextflow will use, by default , LocalSecretProvider and the nomad job submitted will have at some place MY_ACCESS_KEY=123-123 so everyone with access to the definition will be able to see the secret value

Basically, (if I understood well), we want avoid to "write" secrets into the task definition and use nomad variables, https://developer.hashicorp.com/nomad/docs/concepts/variables

The idea them is:

• write variables into the cluster using the nomad cli (nomad var put -out=table secret/creds passcode=my-long-passcode for example)
• using templates attach their values as environment variables (also as files but we'll restrict to env by the moment)

nf-nomad plugin will "detect" which secrets are required by a process and will create a template section in the job definition

cc @abhi18av

[abhi18av]

Thanks for jotting down your thoughts Jorge :)

I'll try to share my thoughts mainly as a user/maintainer of the pipelines which would run through the nf-nomad cluster, and secondly as someone who maintains a nomad cluster (would be great if @jhaezebr / @tomiles share their experience on this aspect).

Basically, (if I understood well), we want avoid to "write" secrets into the task definition and use nomad variables, https://developer.hashicorp.com/nomad/docs/concepts/variables

Agreed, I think that the Nomad Portal is a great feature for users to be able to inspect an ongoing/running job in the cluster and, I feel being able to see the env vars would be a flaw overall.

Therefore, the Nomad Variable idea might be worth exploring - but if this doesn't fly then we could simply add this as a known limitation and address this in future.

write variables into the cluster using the nomad cli (nomad var put -out=table secret/creds passcode=my-long-passcode for example)

For this part, I think we have options

A a plugin-level CLI subcommand

This option is mentioned here https://www.nextflow.io/docs/stable/cli.html#plugin to make the experience of managing nomad variables fit into the nextflow experience.

Couple of benefits of this approach are

1. we shouldn't need to modify the main nextflow codebase at all.
2. we add an entrypoint for the nf-nomad plugin to later expose other nifty abilities of the nomad command line for managing/querying the cluster.

For example

nextflow plugin nf-nomad:secrets ... 

Extend the nextflow secrets

We could patch/implement https://www.nextflow.io/docs/latest/secrets.html command to add alternative secrets store i.e. NomadSecretProvider in this case.

The cons of this approach would be that if we (or Hashicorp) change anything in our approach in the plugin, we'd need to update the main NF codebase and align to the release cycles.

@jagedn please let me know what do you think of these approaches? Happy to hear ideas/avenues I might have misunderstood or missed :)

[jagedn]

Great to confirm we're aligned

I think a SecretProvider implementation can be easier for users.

Also we can enable/disable it via nomad.secrets=enable for example

[abhi18av]

Ah, I see - so this would be within the main NF codebase or we just extend the relevant interface in the plugin?

[jagedn]

Plugin framework allows to provide more than an implementation, so our plugin can provide a SecretProvider that "overwrite" LocalProvider and use it as bridge to the cluster using the API we are using right now (I hope)

I'll work on this idea and hopefully we can have a first preview soon to evaluate it

[abhi18av]

Thanks Jorge!

For a start, ideally we can eliminate the need to share individual minio / aws specific credentials with the pipeline executors so all they'd need is a Nomad token and once they submit the jobs, the relevant creds would become available for the jobs.

As a general direction, we are strengthening our support for policy and cluster-level admin work 😎

[tomiles]

For now we have always stored them in ~/.aws/credentials and haven't had any other need for secret functionality besides that. It would be neat to have a nomad native SecretProvider. FYI for secrets in the nomad ecosystem (besides nf workloads) we use vault integration + job templating or ENV variables to access them in jobs.

[jagedn]

so this branch is a first iteration for this feature

https://github.com/nextflow-io/nf-nomad/tree/secrets-template

How to test:

• build as latest ( ./gradlew clean unzipPlugin -P version=latest -x test)
• run the server ( ./start-nomad.sh)
• using the nomad cli add 2 variables:
  -- nomad var put secrets/MY_ACCESS_KEY MY_ACCESS_KEY=1234
  -- nomad var put secrets/MY_SECRET_KEY MY_SECRET_KEY=TheSecretKey
• run the secret pipeline: nextflow run -w $(pwd)/nomad_temp/scratchdir -c secrets/nextflow.config secrets/main.nf

see the logs and check the process print the 2 values

https://github.com/nextflow-io/nf-nomad/blob/secrets-template/validation/secrets/main.nf

[jagedn]

After a little investigation (trial, error and debug) I just discovered current implementation of Nextflow doesn't load plugins during CmdSecrets and also the SecretsProvider is not able to access to the configuration so by the moment I think we can't have something as

nextflow secrets list

So I'm playing with a Cmd implementation similar to :

nextflow plugin nf-nomad:secrets list

[jagedn]

I think I made a big progress (still remains more tests) but you can taste it:

Use a release I just created in my space:

export NXF_PLUGINS_TEST_REPOSITORY=https://github.com/jagedn/nf-nomad/releases/download/9.9.9/nf-nomad-9.9.9-meta.json

Set the nomad version plugin to 9.9.9

export NOMAD_PLUGIN_VERSION=9.9.9

run the cluster (remove the nomad_temp first to have a clean installation) with sudo ./start-nomad.sh

create a namespace

./nomad namespace apply -description "QA instances" ns-qa

and now create 2 variables into this namespace using our plugin:

nextflow -c secrets/nextflow.config plugin nf-nomad:secrets set MY_SECRET_KEY TheSecretKey

nextflow -c secrets/nextflow.config plugin nf-nomad:secrets set MY_ACCESS_KEY TheAccessKey

you can see both of them in the nomad web UI (pay attention to change to the new namespace) or using the list command:

nextflow -c secrets/nextflow.config plugin nf-nomad:secrets list

Now run the pipeline

./run-pipeline.sh -c secrets/nextflow.config secrets/main.nf

To recap:

• we are allocating jobs and variables in a namespace
• we can set/get/delete variables
• the job definition creates a template for secrets defined in the process
• the SecretsProvider is a "dummy" provider (at least by the moment) so access to secrets in workflow is not implement

p.d.
I have create a release in my space because plugin command doesnt work well with the latest approach

[abhi18av]

Taking a look now Jorge - thank you!

[abhi18av]

Actually, it's working like a charm on my side

1. local-nomadlab ✅

local-nomadlab.nextflow.log

2. sun-nomadlab (fusion env) ✅

sun-nomadlab.nextflow.log

[abhi18av]

And all the resources are being generated in the correct namespace.

@jagedn , I think this is already looking good for a PR - whenever you're comfortable let's do the code review and merge this to complete the release cycle for v0.1.2 🤩

[jagedn]

I've improved a little more current implementation (branch secrets-template)

Now you don't need the extra repository jagedn and can be tested using latest approach (will include a test in the run-all pipeline sh)

In this implementation you can enable/disable secrets and also specify the path where the variables are stored in the cluster

If secrets is disabled then LocalSecretsProvider is used so an user can run the same pipeline using their own secrets (so they are visibles in the job definition) or use the nomad variables, so they are attached as templates in the job definition

[jagedn]

I'll closed this discussion as the secret implementation is merged and working

[abhi18av]

Thanks Jorge! 😊