From b6d418bf573cb30680963133defff2fb1ea96fed Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 14:14:24 +0200 Subject: [PATCH 1/9] Add `dev.eessi.io.md` entry to repository list --- docs/repositories/dev.eessi.io.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 docs/repositories/dev.eessi.io.md diff --git a/docs/repositories/dev.eessi.io.md b/docs/repositories/dev.eessi.io.md new file mode 100644 index 000000000..8cf10a381 --- /dev/null +++ b/docs/repositories/dev.eessi.io.md @@ -0,0 +1,9 @@ +# `dev.eessi.io` repository + +## What is `dev.eessi.io`? + +`dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository. +This way, development version can easily be tested on systems where `dev.eessi.io` is available. + +On a system with `dev.eessi.io` mounted access is possible with `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all`. Then, all that is left is +try out the development software! From 51421c0d4d7b09e349699b89cf1fd3f61b6d59bd Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:15:20 +0200 Subject: [PATCH 2/9] Add `adding_development_software.md` `dev.eessi.io` overview and instructions --- .../adding_development_software.md | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 docs/adding_software/adding_development_software.md diff --git a/docs/adding_software/adding_development_software.md b/docs/adding_software/adding_development_software.md new file mode 100644 index 000000000..755d12232 --- /dev/null +++ b/docs/adding_software/adding_development_software.md @@ -0,0 +1,80 @@ +# Adding software to `dev.eessi.io` + +## What is `dev.eessi.io`? + +`dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository. This way, development version can easily be tested on systems where `dev.eessi.io` is available. + +## Adding software + +Using `dev.eessi.io` is similar to using EESSI's production repository `software.eessi.io`. Software builds are triggered by a bot listening to pull requests in GitHub repository (at the moment https://github.com/EESSI/dev.eessi.io). This repository is organised by project, where corresponding easystack files and easyconfig files are placed. + +``` +dev.eessi.io +├── project1 +│   ├── easyconfigs +│   └── easystacks +└── project2 + ├── easyconfigs + └── easystacks +``` + +Creating a PR that adds an entry to an easystack file under on of the projects will allow authorised users to trigger +a build with the command `bot: build` through a GitHub comment. + +## Building development versions + +### easyconfig files and `--software-commit` +The approach to build and install software is similar to that of `software.eessi.io`. +It requires an easyconfig file which for `dev.eessi.io` need not be part of https://easybuilders/easybuild-easyconfigs +as these easyconfigs can simply be placed under `project/easyconfigs`. + +To allow for development builds, we leverage the `--software-commit` functionality (requires EasyBuild v4.9.3 or higher). This lets us build a given application from +a specific commit in repository. This can also be done from a fork, by changing the `github_account` field in the easyconfig file. +We've created a template for `ESPResSo` based on the standard eaasyconfig of the most recent version. The relevant fields are: + +``` python +easyblock = 'CMakeMake' + +name = 'ESPResSo' +version = '4.2.2' +versionsuffix = '-%(software_commit)s' + +homepage = 'https://espressomd.org/wordpress' +description = """A software package for performing and analyzing scientific Molecular Dynamics simulations.""" + +github_account = 'espressomd' +source_urls = ['https://github.com/%(github_account)s/%(name)s/archive/'] + +sources = ['%(software_commit)s.tar.gz'] +``` + +One can also make new changes to the easyconfig file, for example, if the new functionality requires new build or +runtime dependencies, patches, flags, etc. It's a good idea to try installing from a specific commit locally first, +to at least see if everything is parsed correctly and confirm that the right sources are being downloaded. + +### Easystack files and triggering builds + +After the easyconfig file has been created and added to `projectX/easyconfigs`, an easystack file that picks it up +needs to be in place. This easystack file must follow a naming convention: `software-eb-X.Y.Z-dev.yml`, +where X.Y.Z correspond to the EasyBuild version used to install the software. +Following our example for `ESPREsSo`, it would look like: + +``` yml +easyconfigs: + - ESPResSo-4.2.2-foss-2023a-software-commit.eb: + options: + software-commit: 2ba17de6096933275abec0550981d9122e4e5f28 # release 4.2.2 +``` + +The `ESPResSo-4.2.2-foss-2023a-software-commit.eb` would be the easyconfig file added in the step above. +Note the option passing the `software-commit` for the development version that should be built. +For the sake of this example, the chosen commit actually corresponds to the 4.2.2 release. + +To trigger a build, all one needs to do is open a PR with the changes adding the easyconfig and easystack +files and commenting `bot: build`. This can only be done by previously authorized users. +The current build cluster builds for the `zen2` CPU microarchitecture, but this is likely to change. + +Once a build is complete and the `bot:deploy` label is added, a staging PR can be merged to deploy the +application to the `dev.eessi.io` cvmfs repository. On a system with `dev.eessi.io` mounted, then all +that is left is to `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all` and try out the software! + From 5e353f805416aefcdb70fba7733ea010d678ac1b Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:19:10 +0200 Subject: [PATCH 3/9] Add `dev.eessi.io` pages to side bar --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index ea5a7c6c7..587905e19 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -28,6 +28,7 @@ nav: - Production: repositories/software.eessi.io.md - RISC-V: repositories/riscv.eessi.io.md - Pilot: repositories/pilot.md + - Development repository: repositories/dev.eessi.io.md - Installation and configuration: - Is EESSI already installed?: getting_access/is_eessi_accessible.md - Native: getting_access/native_installation.md @@ -63,6 +64,7 @@ nav: - Contribution policy: adding_software/contribution_policy.md - Open PR: adding_software/opening_pr.md - Debugging: adding_software/debugging_failed_builds.md + - Development repository: adding_software/adding_development_software.md - For maintainers: - BTD bot: bot.md - Building software: adding_software/building_software.md From f0313447f05b4bd5b8b4eed9827d4ac383eaaacd Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:21:33 +0200 Subject: [PATCH 4/9] Update deploy.yml Build page from dev branch --- .github/workflows/deploy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 38db175d5..06fff08b2 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -2,7 +2,7 @@ name: deploy documentation (only on push to main branch) on: push: - branches: main + branches: main, dev.eessi.io # Declare default permissions as read only. permissions: read-all jobs: From 8ad2f4722159fa2c8952b306c2727da173ffbc0a Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 15:35:27 +0200 Subject: [PATCH 5/9] Update deploy.yml --- .github/workflows/deploy.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 06fff08b2..38db175d5 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -2,7 +2,7 @@ name: deploy documentation (only on push to main branch) on: push: - branches: main, dev.eessi.io + branches: main # Declare default permissions as read only. permissions: read-all jobs: From cf4048b8bf8b46ae00c368c723dd0b878bc49af2 Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Thu, 10 Oct 2024 16:00:09 +0200 Subject: [PATCH 6/9] Improve `dev.eessi.io` description Co-authored-by: Kenneth Hoste --- docs/repositories/dev.eessi.io.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/repositories/dev.eessi.io.md b/docs/repositories/dev.eessi.io.md index 8cf10a381..9b6e367df 100644 --- a/docs/repositories/dev.eessi.io.md +++ b/docs/repositories/dev.eessi.io.md @@ -3,7 +3,7 @@ ## What is `dev.eessi.io`? `dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository. -This way, development version can easily be tested on systems where `dev.eessi.io` is available. +This way, development versions of software can easily be tested on systems where the `dev.eessi.io` CernVM-FS repository is available. On a system with `dev.eessi.io` mounted access is possible with `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all`. Then, all that is left is try out the development software! From 520bda1fb24b8c8f3b5646a4dbd377d051fbf716 Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Fri, 11 Oct 2024 11:27:51 +0200 Subject: [PATCH 7/9] Apply suggestions from code review Co-authored-by: Kenneth Hoste --- docs/adding_software/adding_development_software.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adding_software/adding_development_software.md b/docs/adding_software/adding_development_software.md index 755d12232..8571f90df 100644 --- a/docs/adding_software/adding_development_software.md +++ b/docs/adding_software/adding_development_software.md @@ -18,10 +18,10 @@ dev.eessi.io └── easystacks ``` -Creating a PR that adds an entry to an easystack file under on of the projects will allow authorised users to trigger +Creating a [pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) that adds an entry to an easystack file under one of the projects will allow authorized users to trigger a build with the command `bot: build` through a GitHub comment. -## Building development versions +Deploying pre-release builds of software ### easyconfig files and `--software-commit` The approach to build and install software is similar to that of `software.eessi.io`. @@ -49,12 +49,12 @@ sources = ['%(software_commit)s.tar.gz'] ``` One can also make new changes to the easyconfig file, for example, if the new functionality requires new build or -runtime dependencies, patches, flags, etc. It's a good idea to try installing from a specific commit locally first, +runtime dependencies, patches, configuration options, etc. It's a good idea to try installing from a specific commit locally first, to at least see if everything is parsed correctly and confirm that the right sources are being downloaded. ### Easystack files and triggering builds -After the easyconfig file has been created and added to `projectX/easyconfigs`, an easystack file that picks it up +After the easyconfig file has been created and added to the `easyconfigs` subdirectory, an [easystack file](https://docs.easybuild.io/easystack-files) that picks it up needs to be in place. This easystack file must follow a naming convention: `software-eb-X.Y.Z-dev.yml`, where X.Y.Z correspond to the EasyBuild version used to install the software. Following our example for `ESPREsSo`, it would look like: From c6b3ac05588141dcc09d32d31ab03e2179bcf77e Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Fri, 11 Oct 2024 12:04:48 +0200 Subject: [PATCH 8/9] Improve repository description Co-authored-by: Kenneth Hoste --- docs/repositories/dev.eessi.io.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/repositories/dev.eessi.io.md b/docs/repositories/dev.eessi.io.md index 9b6e367df..680f5a861 100644 --- a/docs/repositories/dev.eessi.io.md +++ b/docs/repositories/dev.eessi.io.md @@ -2,7 +2,7 @@ ## What is `dev.eessi.io`? -`dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository. +`dev.eessi.io` is the development repository of EESSI. With it, developers can deploy pre-release builds of their software to EESSI. This way, development versions of software can easily be tested on systems where the `dev.eessi.io` CernVM-FS repository is available. On a system with `dev.eessi.io` mounted access is possible with `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all`. Then, all that is left is From 964ab5b91bf1fa795b09a25514e5a64652113b61 Mon Sep 17 00:00:00 2001 From: Pedro Santos Neves <10762799+Neves-P@users.noreply.github.com> Date: Fri, 11 Oct 2024 12:10:54 +0200 Subject: [PATCH 9/9] Highlight easystack naming convention Co-authored-by: Kenneth Hoste --- .../adding_development_software.md | 24 +++++++++++-------- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/docs/adding_software/adding_development_software.md b/docs/adding_software/adding_development_software.md index 8571f90df..94051c516 100644 --- a/docs/adding_software/adding_development_software.md +++ b/docs/adding_software/adding_development_software.md @@ -55,16 +55,20 @@ to at least see if everything is parsed correctly and confirm that the right sou ### Easystack files and triggering builds After the easyconfig file has been created and added to the `easyconfigs` subdirectory, an [easystack file](https://docs.easybuild.io/easystack-files) that picks it up -needs to be in place. This easystack file must follow a naming convention: `software-eb-X.Y.Z-dev.yml`, -where X.Y.Z correspond to the EasyBuild version used to install the software. -Following our example for `ESPREsSo`, it would look like: - -``` yml -easyconfigs: - - ESPResSo-4.2.2-foss-2023a-software-commit.eb: - options: - software-commit: 2ba17de6096933275abec0550981d9122e4e5f28 # release 4.2.2 -``` +needs to be in place. + +!!! note "Naming convention for easystack files" + + The easystack files must follow a naming convention: `software-eb-X.Y.Z-dev.yml`, + where X.Y.Z correspond to the EasyBuild version used to install the software. + Following our example for `ESPREsSo`, it would look like: + + ``` yml + easyconfigs: + - ESPResSo-4.2.2-foss-2023a-software-commit.eb: + options: + software-commit: 2ba17de6096933275abec0550981d9122e4e5f28 # release 4.2.2 + ``` The `ESPResSo-4.2.2-foss-2023a-software-commit.eb` would be the easyconfig file added in the step above. Note the option passing the `software-commit` for the development version that should be built.