Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev 0.22 #1011

Merged
merged 116 commits into from
Jan 28, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
116 commits
Select commit Hold shift + click to select a range
58dd558
Set submodule spack to v0.22.2
dominichofer Oct 31, 2024
e95aaae
fdb: Remove package
dominichofer Oct 31, 2024
8592711
metkit: Remove package
dominichofer Oct 31, 2024
4f1ce65
py-frozendict: Remove package
dominichofer Oct 31, 2024
0116e61
xpmem: Remove package
dominichofer Oct 31, 2024
40eb0a7
py-hatchling: Update from upstream
dominichofer Oct 31, 2024
698bd7f
py-tabulate: Update from upstream
dominichofer Oct 31, 2024
561c571
icon-ham: Use spack/icon
dominichofer Oct 31, 2024
2c6231e
icon: Split package
dominichofer Oct 31, 2024
6cf047a
py-lark: Remove package
dominichofer Oct 31, 2024
118109d
Remove unnecessary tests
dominichofer Oct 31, 2024
695ac57
Migrate unit tests to pytest
dominichofer Nov 5, 2024
491591d
Rename icon to icon-c2sm
dominichofer Nov 5, 2024
2a7a337
cosmo: Remove package
dominichofer Nov 5, 2024
a1e05dd
Jenkins: Use mch/prgenv-icon:rc1
dominichofer Nov 5, 2024
9b5c5be
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 5, 2024
a2acb40
Merge branch 'main' into dev_0.22
dominichofer Nov 5, 2024
6ff2517
Add tests for 'spack find' and 'spack compilers'.
dominichofer Nov 5, 2024
de903c8
Start uenv in test stages
dominichofer Nov 5, 2024
1059f74
deactivate
dominichofer Nov 5, 2024
4a3cf6d
Simplify run_with_spack
dominichofer Nov 5, 2024
8b5222c
Keep venv
dominichofer Nov 6, 2024
77df0b9
Bring back devirtualization
dominichofer Nov 6, 2024
e073f3c
Update uenv
dominichofer Nov 6, 2024
c827041
Use .venv
dominichofer Nov 6, 2024
bce9413
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 6, 2024
5c42d6e
Shuffle Jenkinsfile
dominichofer Nov 7, 2024
e352780
revert
dominichofer Nov 7, 2024
20fe9af
uenv
dominichofer Nov 7, 2024
6fb5890
revert clean env
dominichofer Nov 7, 2024
3bd74dd
Remove fdb-fortran
dominichofer Nov 7, 2024
31fe66b
flexpart-cosmo: Add workaround
dominichofer Nov 7, 2024
1b66901
Update icon-mch tests
dominichofer Nov 7, 2024
d219b6e
Typo
dominichofer Nov 7, 2024
9353b9b
icontools: Workaround
dominichofer Nov 7, 2024
c94ed34
Fix some icon tests
dominichofer Nov 7, 2024
7362726
Fix type
dominichofer Nov 11, 2024
ea79b33
Remove icon-mch gcc test
dominichofer Nov 11, 2024
32677b1
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 11, 2024
afc297b
Merge commit 'ba82f8f97283312d769e3ca4a79a01ba2a1af5d1' into dev_0.22
dominichofer Nov 11, 2024
fe73488
Revert "cosmo: Remove package"
dominichofer Nov 11, 2024
6a6e53c
cosmo: remove package
dominichofer Nov 12, 2024
7c06aaf
cosmo: remove tests
dominichofer Nov 12, 2024
d99581c
Fix icon-mch test
dominichofer Nov 12, 2024
5a1ebe8
fix icon test
dominichofer Nov 12, 2024
94826e1
eccodes: remove version 2.19.0
dominichofer Nov 12, 2024
0ce3053
Clean up system tests
dominichofer Nov 12, 2024
c570630
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 12, 2024
59f1b02
Update uenv
dominichofer Nov 12, 2024
1ca68b1
cosmo-eccodes-definitions: Remove versions 2.19.0.x
dominichofer Nov 12, 2024
9627760
icon-mch: Add workarounds
dominichofer Nov 12, 2024
3c01785
icon-c2sm: Inherit from icon
dominichofer Nov 12, 2024
76bfb04
Improve tests
dominichofer Nov 12, 2024
e4d3aac
Disable failing package tests.
dominichofer Nov 13, 2024
7b68f2e
Move mpi workaround
dominichofer Nov 13, 2024
ca88007
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 13, 2024
1d30755
Merge remote-tracking branch 'origin/main' into dev_0.22
dominichofer Nov 25, 2024
7376c3c
icon-mch: Remove gcc test and add version tests
dominichofer Nov 25, 2024
cdc5b67
icon-c2sm: Remove copy_runscript_related_input_files because there's …
dominichofer Nov 25, 2024
c0f83dc
icon-mch: fix tags
dominichofer Nov 25, 2024
7898444
icon: Set new maintainer
dominichofer Nov 26, 2024
83da947
Rename icon-c2sm to icon
dominichofer Nov 27, 2024
c58351d
Use require
dominichofer Nov 27, 2024
65b37b9
GitHub Action: Apply Pep8-formatting
invalid-email-address Nov 27, 2024
5ff1b40
Merge branch 'main' into dev_0.22
dominichofer Nov 27, 2024
201a1c0
pytorch-fortran-proxy: requires gcc
dominichofer Dec 4, 2024
31383eb
Merge branch 'main' into dev_0.22
jonasjucker Dec 5, 2024
adf2ad9
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
e804ce0
dirty gt4py fix
huppd Dec 5, 2024
e103c77
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
huppd Dec 5, 2024
90cbaf5
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
8ac1984
rm exclaim
huppd Dec 5, 2024
e9f2c44
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
639d2b8
fix test and add icon-exlcaim
huppd Dec 5, 2024
c4eab38
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
941db8b
make out-of-source work
jonasjucker Dec 5, 2024
e0dd2af
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
jonasjucker Dec 5, 2024
82ce7e2
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
8557771
fix url
huppd Dec 5, 2024
c30fb38
rm file
huppd Dec 5, 2024
b46ca72
update test
huppd Dec 5, 2024
b94b537
cleanup
huppd Dec 5, 2024
7ea4ac7
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
ca49b5a
fix
huppd Dec 5, 2024
ef02763
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
huppd Dec 5, 2024
59ecadc
fix version
jonasjucker Dec 5, 2024
fb7ce7c
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
jonasjucker Dec 5, 2024
4ed055d
rm include
huppd Dec 5, 2024
5ad9805
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
1598542
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
huppd Dec 5, 2024
aefb2a9
fix icon-ham
jonasjucker Dec 5, 2024
81ad942
add maintainer
huppd Dec 5, 2024
a0bd445
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
huppd Dec 5, 2024
13d4c08
Update repos/c2sm/packages/icon-exclaim/package.py
jonasjucker Dec 5, 2024
57010f3
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
cb59d55
change
jonasjucker Dec 5, 2024
2aef510
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
2dabff6
disable version
huppd Dec 5, 2024
f66f2b6
GitHub Action: Apply Pep8-formatting
invalid-email-address Dec 5, 2024
ae454ec
remove loop-exchange
jonasjucker Dec 5, 2024
e4386b6
Merge branch 'dev_0.22' of github.com:C2SM/spack-c2sm into dev_0.22
jonasjucker Dec 5, 2024
42a5efe
fix icon
jonasjucker Dec 5, 2024
c40e511
dirty fix icon4py
huppd Dec 5, 2024
b139e23
readd loop-exchange
huppd Dec 6, 2024
eb6d5b3
Update package.py
jonasjucker Dec 9, 2024
8693130
fix bug again of serialbox dep, cleanup
jonasjucker Dec 9, 2024
fe686ce
reintroduce skip configure
jonasjucker Dec 9, 2024
d9e66b7
remove libtorch
jonasjucker Dec 9, 2024
52db158
upgrade docs (#1038)
jonasjucker Dec 9, 2024
ca2ba85
update docs
jonasjucker Dec 9, 2024
8150572
Update cosmo-eccodes-definitions versions to 2.36.0.3
lxavier Jan 15, 2025
98de61c
use explicit zlib-ng dependency for icon (#1040)
bcumming Jan 20, 2025
08b457a
use rc 4
jonasjucker Jan 27, 2025
979f9b7
remove obsolete packages
jonasjucker Jan 27, 2025
7869af5
adjust integration tests
jonasjucker Jan 27, 2025
615da3e
upgrade manual to build icon
jonasjucker Jan 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 12 additions & 36 deletions docs/CodeDevelopment.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,42 +3,6 @@ Develop packages

Spack offers several options for package development.
Depending on your workflow, one or the other option is preferred.
Also some packages like ICON or COSMO have their own
development workflow which is maintained by C2SM.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console

$ spack dev-build --until build <package> @<version>

This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.

Dev-build in combination with build-env
---------------------------------------

We assume that developers of a package are familiar with its build system.
Therefore, we reccomend to use Spack to set up the environment for the package.
Building and testing should be done with the package's build and test system.

.. code-block:: console

# Load Spack!
$ spack dev-build --before build <package> @develop <variant> # stops dev-build before executing the phase 'build'
$ spack build-env <package> @develop <variant> -- bash # nests a bash shell with the build env vars loaded
# Work on the package!
# Use the build system of the package! (e.g. 'make')
# Use the testing infrastructure of the package!
$ exit # to exit the nested bash

If you want multiple dev-builds at the same time, label them with separate ``@<your-label>``.
The identifier ``@develop`` is common in the Spack documentation but you can use any string.


Environments with Spack develop
-------------------------------
Expand Down Expand Up @@ -95,3 +59,15 @@ To deactivate a Spack environment, type
Most of the Spack commands are sensitive to environments, see
`Spack docs <https://spack.readthedocs.io/en/latest/environments.html#environment-sensitive-commands>`__.

Plain dev-build
---------------

This is the easiest way to build local sources.
Enter the root of your source repository and execute:

.. code-block:: console

$ spack dev-build --until build <package> @<version>

This will build the package as is. The downside of this approach is that
you need to go through all phases of a package build.
Copy link
Contributor Author

@dominichofer dominichofer Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's not true for icon. Icon sometimes skips stuff.

48 changes: 23 additions & 25 deletions docs/QuickStart.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,17 @@ To set up a Spack instance, clone the repository using a specific Spack tag (lat

$ git clone --depth 1 --recurse-submodules --shallow-submodules -b $SPACK_TAG https://github.com/C2SM/spack-c2sm.git

To load it into your command line, execute
To load it into your command line, execute one of the following commands:

.. code-block:: console

$ . spack-c2sm/setup-env.sh
$ . spack-c2sm/setup-env.sh /user-environment
$ . spack-c2sm/setup-env.sh /mch-environment/v6
$ . spack-c2sm/setup-env.sh /mch-environment/v7
$ . spack-c2sm/setup-env.sh any_other_upstream

This auto-detects your machine and configures your instance for it.
You can force a machine with an argument. The name has to match a folder in sysconfigs.

.. code-block:: console

$ . spack-c2sm/setup-env.sh balfrin

This will make upstream installation from user-environment available in spack-c2sm.

Local machines and Containers
-----------------------------
Expand Down Expand Up @@ -89,16 +87,6 @@ For ICON, they are located in ``config/cscs/spack/<version>/<machine>_<target>_<
They work with a special Spack tag, that is provided in the ICON repository at ``config/cscs/SPACK_TAG_*``.
So make sure you clone Spack with the specified tag.

.. tip::
**On Balfrin:**
In case your Spack environment requires Python, a compatability issue
with `openssl` and `git` appears.

``/usr/bin/ssh: symbol lookup error: /usr/bin/ssh: undefined symbol: EVP_KDF_CTX_free, version OPENSSL_1_1_1d``

To circumvent that simply do
``spack load git`` prior to activation of the environment.

To activate the Spack environment, type

.. code-block:: console
Expand Down Expand Up @@ -127,15 +115,25 @@ Out-of-source builds are possible as follows:

.. code-block:: console

$ mkdir cpu && cd cpu
$ cp -r ../config .
$ spack env activate -d config/cscs/spack/v0.20.1.5/daint_cpu_nvhpc
$ mkdir cpu
$ spack env activate config/cscs/spack/v0.20.1.5/daint_cpu_nvhpc
$ # tell spack to build icon in folder cpu
$ spack develop --build-directory cpu icon@develop
$ spack install

.. attention::
Out-of-source build for AutotoolsPackages is not supported by Spack.
The implementation for ICON relies on some hacks inside package.py and
only works if the build-folder is located inside the Git repo of ICON.
By executing the commands above, spack will add some lines directly into ``spack.yaml``:

.. code-block:: yaml

spack:
packages:
icon:
package_attributes:
build_directory: /scratch/mch/juckerj/icon-nwp/cpu

Any further ``spack install`` command will use the build directory specified in the ``spack.yaml`` file.
In case you want to change the build directory, edit the ``spack.yaml`` file or remove the ``build_directory`` line
and run ``spack concretize -f`` afterwards.

COSMO
-----
Expand Down
Loading