Support for environment.yml/lockfiles

[lbdreyer]

This PR includes work previously done by @jamesp to support environment.yml and environment lockfiles instead of the bespoke env.spec and env.manifests that existed in a gitenv.

Key changes (since @jamesp changes):

• You need to supply a lockfile and environment.yml file

Previously you just needed a lockfile and a name, but this is missing some needed metadata. Particularly, the "tag" is missing. We still need the tag in order to know where to install it. Previously, this was installing it a <install_prefix>/environments/<env_name> so it could be something like /opt/conda-dist/environments/default-current. But we need to to be installed at /opt/conda-dist/environments/default/2022_01_01-1 which is symlinked from .../default/current.

• env.spec now creates the symlink, modulefiles and the environment itself (combining what the old env.spec and tagged_env.spec did)

@jamesp changes had only included one spec (env.spec.template) which did what the old tagged_env.spec did. However, then we no longer had an rpm that was creating the symlink and the modulefiles. I have added these into the env.spec.template. I did this, rather than reinstating the tagged_env.spec/env.spec combo because that was dependent on the environment git repo system. We no longer have that, so the environments we create are independent of each other. Therefore, even if we were to switch an environment to a new label (e.g. next -> current), we still have to re deploy it.

• A version number now needs to provided in environment.yml file

We need a version number for the env.specs to differentiate between the different versions of CondaDist-env-default-current.rpm. In the old conda_rpms, when using a gitenv, we would get the version number based of the number of commits in the environments git repo. As we have remove the use of a git repo and now the user supplies a environment .yml file and lock file I propose we now also require the version number in the environment.yml file, within the meta section

[lbdreyer]

Note env['meta']['released'] is now being used as the tag. That may be a problem if the tag is something like 2022_01_01-1?

[jamesp]

This would be the case if we wanted to release two versions of the same environment on the same day? Seems unlikely.

[lbdreyer]

Summary of changes discussed to be completed if we choose to continue using conda-rpms

• split out env.spec (which performs the symlink and creates the modulefiles) from tagged_env.spec (which installed the environment), effectively mimicing what we had before
• Include the created of a modulefile that access a specific tagged env (e.g. module load /default-2021_01_01)
• use the release date as the version number in the rpm name
• test the scenario where we have RPMs installed that were created using the old method, and then try to install RPMs created using the new method
• consider decommissioning conda-rpms and instead vendoring the code
• handle epoch numbers Support for packages which include an epoch in the version number #27

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ lbdreyer
❌ jamesp
_{You have signed the CLA already but the status is still pending? Let us recheck it.}