Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add intial documentation on dev.eessi.io #321

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
80 changes: 80 additions & 0 deletions docs/adding_software/adding_development_software.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Adding software to `dev.eessi.io`

## What is `dev.eessi.io`?

`dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository. This way, development version can easily be tested on systems where `dev.eessi.io` is available.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should just point to repositories/dev.eessi.io.md


## Adding software

Using `dev.eessi.io` is similar to using EESSI's production repository `software.eessi.io`. Software builds are triggered by a bot listening to pull requests in GitHub repository (at the moment https://github.com/EESSI/dev.eessi.io). This repository is organised by project, where corresponding easystack files and easyconfig files are placed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll probably ends up with separate repos per project, so point to https://github.com/search?q=org%3AEESSI+dev.eessi.io&type=repositories ?


```
dev.eessi.io
├── project1
Comment on lines +12 to +13
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not correct I think, each repo has it's own separate file structure.

Maybe use dev.eessi.io-example here (we could even create that repo once things have stabilized a bit)

│   ├── easyconfigs
│   └── easystacks
└── project2
├── easyconfigs
└── easystacks
```

Creating a PR that adds an entry to an easystack file under on of the projects will allow authorised users to trigger
Neves-P marked this conversation as resolved.
Show resolved Hide resolved
a build with the command `bot: build` through a GitHub comment.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

refer to bot docs here?


## Building development versions
Neves-P marked this conversation as resolved.
Show resolved Hide resolved

### easyconfig files and `--software-commit`
The approach to build and install software is similar to that of `software.eessi.io`.
It requires an easyconfig file which for `dev.eessi.io` need not be part of https://easybuilders/easybuild-easyconfigs
as these easyconfigs can simply be placed under `project/easyconfigs`.

To allow for development builds, we leverage the `--software-commit` functionality (requires EasyBuild v4.9.3 or higher). This lets us build a given application from
a specific commit in repository. This can also be done from a fork, by changing the `github_account` field in the easyconfig file.
We've created a template for `ESPResSo` based on the standard eaasyconfig of the most recent version. The relevant fields are:

``` python
easyblock = 'CMakeMake'

name = 'ESPResSo'
version = '4.2.2'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use devel here?

versionsuffix = '-%(software_commit)s'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random thought: I would be brilliant if we could add support for something like %(software_commit_datestamp)s, which is automatically replaced with the datestamp of the commit (so you get module names that sort well):

name = 'ESPResSo'
version = '%(software_commit_datestamp)s-%(software_commit)s'


homepage = 'https://espressomd.org/wordpress'
description = """A software package for performing and analyzing scientific Molecular Dynamics simulations."""

github_account = 'espressomd'
source_urls = ['https://github.com/%(github_account)s/%(name)s/archive/']

sources = ['%(software_commit)s.tar.gz']
```

One can also make new changes to the easyconfig file, for example, if the new functionality requires new build or
runtime dependencies, patches, flags, etc. It's a good idea to try installing from a specific commit locally first,
Neves-P marked this conversation as resolved.
Show resolved Hide resolved
to at least see if everything is parsed correctly and confirm that the right sources are being downloaded.

### Easystack files and triggering builds

After the easyconfig file has been created and added to `projectX/easyconfigs`, an easystack file that picks it up
Neves-P marked this conversation as resolved.
Show resolved Hide resolved
needs to be in place. This easystack file must follow a naming convention: `software-eb-X.Y.Z-dev.yml`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the part about the filename a note that pops out a bit (see https://squidfunk.github.io/mkdocs-material/reference/admonitions/#usage)

where X.Y.Z correspond to the EasyBuild version used to install the software.
Following our example for `ESPREsSo`, it would look like:

``` yml
easyconfigs:
- ESPResSo-4.2.2-foss-2023a-software-commit.eb:
options:
software-commit: 2ba17de6096933275abec0550981d9122e4e5f28 # release 4.2.2
```

The `ESPResSo-4.2.2-foss-2023a-software-commit.eb` would be the easyconfig file added in the step above.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename to ESPResSo-devel-foss-2023a.eb ?

Note the option passing the `software-commit` for the development version that should be built.
For the sake of this example, the chosen commit actually corresponds to the 4.2.2 release.

To trigger a build, all one needs to do is open a PR with the changes adding the easyconfig and easystack
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, refer to bot docs

files and commenting `bot: build`. This can only be done by previously authorized users.
The current build cluster builds for the `zen2` CPU microarchitecture, but this is likely to change.

Once a build is complete and the `bot:deploy` label is added, a staging PR can be merged to deploy the
application to the `dev.eessi.io` cvmfs repository. On a system with `dev.eessi.io` mounted, then all
that is left is to `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all` and try out the software!
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

module use needs project subdir


9 changes: 9 additions & 0 deletions docs/repositories/dev.eessi.io.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# `dev.eessi.io` repository

## What is `dev.eessi.io`?

`dev.eessi.io` is the development repository of EESSI. With it, developers can build and deploy non-production ready versions of their software to a CernVM-FS repository.
Neves-P marked this conversation as resolved.
Show resolved Hide resolved
This way, development version can easily be tested on systems where `dev.eessi.io` is available.
Neves-P marked this conversation as resolved.
Show resolved Hide resolved

On a system with `dev.eessi.io` mounted access is possible with `module use /cvmfs/dev.eessi.io/versions/2023.06/modules/all`. Then, all that is left is
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will change when we use subdirectories for projects

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In fact, we probably (eventually) want a small custom init script for dev.eessi.io which:

  • prints a warning (something like "installations in dev.eessi.io are not for production use, only for testing");
  • sets up the environment for software.eessi.io;
  • sets up the environment for a specific (set of) projects in dev.eessi.io;

try out the development software!
2 changes: 2 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ nav:
- Production: repositories/software.eessi.io.md
- RISC-V: repositories/riscv.eessi.io.md
- Pilot: repositories/pilot.md
- Development repository: repositories/dev.eessi.io.md
- Installation and configuration:
- Is EESSI already installed?: getting_access/is_eessi_accessible.md
- Native: getting_access/native_installation.md
Expand Down Expand Up @@ -63,6 +64,7 @@ nav:
- Contribution policy: adding_software/contribution_policy.md
- Open PR: adding_software/opening_pr.md
- Debugging: adding_software/debugging_failed_builds.md
- Development repository: adding_software/adding_development_software.md
- For maintainers:
- BTD bot: bot.md
- Building software: adding_software/building_software.md
Expand Down
Loading