diff --git a/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/bug-report.md new file mode 100644 index 0000000..d87615d --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug-report.md @@ -0,0 +1,28 @@ +--- +name: Bug report +about: Use this template to report bugs +title: "[Bug]" +labels: 'bug' +assignees: '' + +--- + +## Current behavior (describe the bug) +>[Be sure to add a Pipeline, Label, Estimate, Assignees, and Epic](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/practices/issues.html) + +## To Reproduce + +> What computer are you running on? + +> What compilers/modules are you using? + +> Steps to reproduce the behavior + +1. +2. +3. +... + +## Expected behavior + +## Additional information (optional) diff --git a/.github/ISSUE_TEMPLATE/config.yaml b/.github/ISSUE_TEMPLATE/config.yaml new file mode 100644 index 0000000..4de2fed --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yaml @@ -0,0 +1,8 @@ +blank_issues_enabled: false +contact_links: + - name: JCSDA + url: https://jcsda.org/ + about: JCSDA web site + - name: Forums + url: https://forums.jcsda.org/ + about: JCSDA user/developer forums diff --git a/.github/ISSUE_TEMPLATE/general-issue.md b/.github/ISSUE_TEMPLATE/general-issue.md new file mode 100644 index 0000000..dc339e4 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/general-issue.md @@ -0,0 +1,26 @@ +--- +name: General issue template +about: Use this template for general issues +title: "[New issue]" +labels: '' +assignees: '' + +--- + +## Description +>Provide a detailed description of this issue. +>What problem needs to be fixed? What new capability needs to be added? +>If this is a bug, describe the current behavior (or use the bug template). +>[Be sure to add a Pipeline, Label, Estimate, Assignees, and Epic](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/practices/issues.html) + +## Requirements + +>If this is a new feature: What does the new code need to accomplish? Does it require new software dependencies (e.g. new jedi-stack components or new python modules?) +>If this is a bugfix: What is the expected behavior? + +## Acceptance Criteria (Definition of Done) +>What does it mean for this to be finished? + +## Dependencies +>What must be done before this can be done? Add issue dependencies in ZenHub as appropriate +>Does this block progress on other issues? Add this issue as a dependency to other ZenHub issues as appropriate diff --git a/README.md b/README.md index e688286..1d5eb40 100644 --- a/README.md +++ b/README.md @@ -5,210 +5,123 @@ The motivation is to define a common set of software packages and versions to be We want to provide a common set of software libraries to JEDI users and developers in order to minimize problems associated with incompatible or outdated software versions. However, we also wish to take advantage of site-specific configurations and optimizations that exploit the local hardware. For this reason, the build process is designed to use native installations of the compiler suite and the MPI libraries. The rest of the software stack is build from source for the specified compiler/mpi combination. -Building the JEDI software stack is a **Four-Step process**, as described in the following sections. - -[See here for additional tips on particular platforms](doc/Platforms.md) - -## Step 1: Set up Basic Environment +# Section 1: Current JEDI Stack Components + -This is the most context-dependent part of the build process. How you proceed depends on the system you are on. Regardless of how you proceed, this step only needs to be done once for each system. +## Required, but not sensitive to version: These can be installed via jedi-stack but are often easier to install via package manager or native modules -Note - you can skip this step and move on to Step 2 if you are building a JEDI software container because all of the required software packages are already built into the jcsda/docker_base container, which you can just pull from Docker Hub. +| software | minimum version | Notes | +| ------- | --------------- | ----- | +| compiler | --- | C++, C, and Fortran; commonly used: gnu (gcc/gfortran), clang, intel; must support C++-14 +| MPI library | --- | commonly used: Openmpi, mpich, Intel MPI +| cmake | 3.16 | | +| git-lfs | 2.11 | | -This step is most important for bare linux/unix systems, as you would get with a new cloud computing instance, a container build, or a virtual machine (e.g. [vagrant](https://www.vagrantup.com)). For such systems, Step 1 consists of running the following script (all paths are relative to the base path of the jedi-stack repository): -``` -cd buildscripts -./setup_environment.sh -``` -where `` depends on your operating system and the context of the build (for example, if you're in a container or on the cloud or on an HPC system). Examples include `docker-devel` or `ubuntu/18.04`. To see a list of supported options, run the script without any arguments: -``` -./setup_environment.sh -``` -**Warning: for some (not all) options, running this script requires root privileges.** -The purpose of this script is to install some basic software packages, the most important of which are: -* GNU compiler suite (gcc, g++, gfortran) -* [Lmod](https://lmod.readthedocs.io/en/latest/index.html) module management system -* git, [git-lfs](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/developer_tools/gitlfs.html), and [git-flow](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/developer_tools/getting-started-with-gitflow.html) -* wget and curl -* make and [CMake](https://cmake.org) -* [doxygen](http://www.doxygen.nl) and [graphviz](https://www.graphviz.org) -* Debugging tools, including [kdbg](http://www.kdbg.org/) and valgrind +## Required components of the stack on most HPC systems, laptops, and containers - installed via jedi-stack (current release versions) -Many of these are installed with package managers such as [apt-get](https://linux.die.net/man/8/apt-get) for Linux or [HomeBrew on Linux](https://docs.brew.sh/Homebrew-on-Linux) and [HomeBrew](https://brew.sh/) for Mac OSX. A few scripts are also provided in the libs directory for such packages as `Lmod` and `CMake` if you would rather install these from source. +| software | version | Notes | +| -------- | ------- | ----- | +| udunits | 2.2.26 | updated to 2.2.28 in develop | +| zlib | 1.2.11 | | +| szip | 2.1.1 | | +| lapack | 3.8.0 | Can be replaced by intel mkl | +| Boost | 1.68.0 | headers only | +| Eigen | 3.3.7 | | +| ecbuild | jcsda/3.3.2.jcsda3 | Now `ecmwf/3.6.1` in develop | +| hdf5 | 1.12.0 | | +| pnetcdf | 1.12.1 | | +| netcdf | 4.7.4, 4.5.3, 4.3.0 | versions for C, Fortran, and C++ | +| nccmp | 1.8.7.0 | Needed for running tests | +| eckit | jcsda/1.11.6.jcsda2 | Now `ecmwf/1.16.0` in develop | +| bufrlib | 11.3.2 | deprecated: replaced by NCEP-bufr | +| pybind11 | 2.5.0 | | +| gsl_lite | 0.34.0 | upgraded to 0.37.0 in develop | -If you are using your own laptop or workstation, it is likely that most of these basic packages are already installed. If any of these packages are missing from your system (such as Lmod), you can manually install them with a package manager or with the build scripts. The items near the bottom of the list are not essential - if you don't have them on your system there is no need to install them. +## New components since most recent release -If you are building on a EMC RHEL7 workstation, use [HomeBrew on Linux](https://docs.brew.sh/Homebrew-on-Linux) to manually install the basic software packages. Follow the instructions at the HomeBrew on Linux website for installing HomeBrew itself, then run the following brew commands: -~~~~~~~~ -brew install lmod -brew install git -brew install git-lfs -brew install git-flow-avh -brew install cmake -brew install doxygen -brew install graphviz -brew install sphinx-doc -~~~~~~~~ +| software | version | Notes | +| -------- | ------- | ----- | +| CGAL | 5.0.2 | Optional in principle but may become required in practice for efficiency reasons. Can be installed without gmp, mpfr dependencies | +| bufr | 11.5.0 | NCEP upstream replaces previous bufrlib | -If you are building on Mac OSX with Clang, then do not run the `setup_environments.sh` script, and instead use [HomeBrew](https://brew.sh/) to manually install the basic software packages. Follow the instructions at the HomeBrew website for installing HomeBrew itself, then run the following brew commands: -~~~~~~~~ -brew install gcc@7 # install version 7 GNU compilers, don't install version 8 GNU as it - # has known problems with JEDI -brew install lmod -brew install git -brew install git-lfs -brew install git-flow-avh -brew install wget -brew install cmake -brew install doxygen -brew install graphviz +## Required for certain components or models -# brew can be used to install any other basic tools that you want such as tkdiff, gdb, etc. -~~~~~~~~ +| software | version | Notes | +| -------- | ------- | ----- | +| PIO | 2.5.1 | Needed for MPAS | -See the [section below](#MacPython) for instructions to install python on the Mac. - -Your Mac should have come with Clang compilers for C and C++ pre-installed, so in this scheme you are adding in GFortran (GNU) for compiling Fortran code. Once you have finished with the brew install commands, make sure to set `JEDI_OPT` in your environment as described below. - -If you're on an HPC system you can largely skip this step (but you still need to set the **JEDI_OPT** environment variable, see below) because most of these packages are probably already installed and available. However, there are a few items that you may wish to add by loading the appropriate modules (if they exist) - for example: -``` -module load doxygen git-lfs -``` -Note, however, if you are using [JEDI Modules](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/jedi_environment/modules.html) you do not have to worry about this (or indeed, about this repository in general) - the JEDI team will make sure that the modules provided will include the packages you need. - -**IMPORTANT:** Another responsibility of the `setup_environment.sh` script is to define the `JEDI_OPT` environment variable and initialize the Lmod system. These actions are needed both for the build and to allow users to load the JEDI modules after you build them. `JEDI_OPT` specifies where the modules will be installed, with a default value of `JEDI_OPT=/opt/modules`. Note that this default value normally requires root permission so you would have to set the `USE_SUDO` flag (see Step 2). If you do not have root privileges (e.g. on an HPC system), you may wish to install your modules in a home or work directory, e.g. `JEDI_OPT=$HOME/opt/modules`. - -`JEDI_OPT` needs to be set and Lmod initialized in order to complete Steps 2-4. But, these actions also need to be performed in order for users to use the modules. Be sure to add the following to your shell initialization scripts (example shown is for bash): -``` -# For jedi-stack -export JEDI_OPT=/opt/modules - -# For lmod modules -. /usr/local/opt/lmod/init/profile -module use $JEDI_OPT/modulefiles/core -``` -and make sure these settings are in place before proceeding to Steps 2-4. - -## Step 2: Configure Build - -The next step is to choose what components of the stack you wish to build and to specify any other aspects of the build that you would like. This is normally done by editing one of the platform-specific `buildscripts/config/config_*.sh` files. Or, if your platform is not among the options, you can edit the `config_custom.sh` file. Then, edit the `buildscripts/config/choose_modules.sh` file to choose which modules you wish to build. Note that some are prerequisites of others. For example, you must build hdf5 before you build netcdf. - -Here we describe some of the parameter settings available in these configuration files. - -For building on Mac OSX, a configuration file (`config_mac.sh`) is provided. This configuration is set up to build using Clang 10.0.0 with gfortran and OpenMPI. You may wish to edit this file for building with a different compiler/mpi set. - -For building on an EMC RHEL7 workstation, a configuration file (`config_rhel7emc.sh`) is provided. This . - -**JEDI_COMPILER** This defines the vendor and version of the compiler you wish to use for this build. The format is the same as what you would typically use in a module load command: -``` -export COMPILER=/ -``` -For example, `COMPILER=gnu/7.3.0`. - -**JEDI_MPI** is the MPI library you wish to use for this build. The format is the same as for `COMPILER`, for example: `export MPI=openmpi/3.1.2`. - -**PREFIX** is the directory where the software packages will be installed. Normally this is set to be the same as the `JEDI_OPT` environment variable (default value `/opt/modules`), though this is not required. If `JEDI_OPT` and `PREFIX` are both the same, then the software installation trees (the top level of each being is the compiler, e.g. `gnu-7.3.0`) will branch directly off of `$JEDI_OPT` while the module files will be located in the `modulefiles subdirectory. - -**USE_SUDO** If `PREFIX` is set to a value that requires root permission to write to, such as `/opt/modules`, then this flag should be enabled. - -_**NOTE: To enable a boolean flag use a single-digit `Y` or `T`. To disable, use `N` or `F` (case insensitive)**_ - -**PKGDIR** is the directory where tarred or zipped software files will be downloaded and compiled. Unlike `PREFIX`, this is a relative path, based on the root path of the repository. Individual software packages can be downloaded manually to this directory and untarred, but this is not required. Most build scripts will look for directory `pkg/pkgName-pkgVersion` e.g. `pkg/hdf5-1_10_3`. - -**LOGDIR** is the directory where log files from the build will be written, relative to the root path of the repository. - -**OVERWRITE** If set, this flag will cause the build script to remove the current installation, if any exists, and replace it with the new version of each software package in question. If this is not set, the build will bypass software packages that are already installed. - -**NTHREADS** The number of threads to use for parallel builds - -**MAKE_CHECK** Run `make check` after build - -**MAKE_VERBOSE** Print out extra information to the log files during the build - -The remaining items enable or disable builds of each software package. The following software can optionally be built with the scripts under `buildscripts`. Unless otherwise noted, the packages are built in Step 4 using the `build_scripts.sh` script. - -* Compilers and MPI libraries - - GNU (Step 3) - - OpenMPI - - MPICH - - `jedi-` Meta-modules for all the above as well as Intel and IMPI - -* Minimal JEDI Stack - - SZip - - Zlib - - HDF5 - - NetCDF - - PNetCDF - - Udunits - - LAPACK - - Boost (Headers only) - - Eigen3 - - ncccmp - - nco - - ecbuild, eckit, fckit - - ODB - -* Supplementary Libraries - - PNG - - JPEG - - Jasper - - Armadillo - - Boost (full installation) - - FFTW - - ecCodes - - ESMF - - FMS - - ESMA-Baselibs - - nceplibs +## Required, but not necessarily in HPC modules -**IMPORTANT: Steps 2, 3, and 4 need to be repeated for each compiler/mpi combination that you wish to install.** The new packages will be installed alongside any previously-existing packages that may already exist and that are built from other compiler/mpi combinations. +| software | version | Notes | +| -------- | ------- | ----- | +| pyjedi | --- | Python tools installed in user space. Should be deprecated soon in lieu of python dependencies handled by individual repos, solo, new bufr libraries, and/or virtual environments | +| json | 3.9.1 | | +| json-schema-validator | 2.1.0 | used for testing | +| doxygen+dot | --- | used for generating documentation | +| latex | --- | used for generating documentation | -## Step 3: Set Up Compiler, MPI, and Module System +## Optional -The next step is to run this from the buildscripts directory: -``` -./setup_modules.sh [] -``` -where `` points to the configuration script that you wish to use, as described in Step 2. The name of this file is `config/config_`. For example, to use the `config/config_custom.sh` you would enter this: -``` -./setup_modules.sh custom -``` -If no arguments are specified, the default is `custom`. Note that you can skip this step as well for container builds because we currenly include only one compiler/mpi combination in each container. So, each package is only build once and there is no need for modules. +These can be build by jedi-stack but they are not required for jedi. -For building on Mac OSX, use: -``` -./setup_modules.sh mac -``` +| software | version | Notes | +| -------- | ------- | ----- | +| NCO | 4.7.9 | Upgraded to 4.9.9 in develop. Can use a native module or package installation? | +| fckit | jcsda/0.7.0.jcsda1 | Now using `ecmwf/0.9.2` and now included in default stack | +| atlas | jcsda/0.20.2.jcsda1 | Now using `ecmwf/0.24.1` and now included in default stack | +| fms | jcsda/release-stable | Currently not included in stack but moving toward including it| +| gptl | 8.0.3 | profiling tool | +| fftw | 3.3.8 | | +| Boost (full) | 1.68.0 | | +| esmf | 8_0_1 | | +| baselibs | 5.2.2 | | +| pdtoolkit | 3.25.1 | Tau dependency | +| tau2 | 3.25.1 | | +| armadillo | 1.900.1 | No longer used? | +| odc | jcsda/1.0.3 | | +| png | 1.6.35 | | +| jpeg | 9.1.0 | | +| jasper | 1.900.1 | | +| xerces | 3.1.4 | | +| nceplibs | fv3 | | +| tkdiff | 4.3.5 | | +| geos | 3.8.1 | | +| sqlite | 3.32.3 | | +| proj | 7.1.0 | | +| ecflow | 5.5.3 | Requires boost, boost-python3, openssl, and qt (install via [brew](https://brew.sh/)); see [Notes on building the Software Stack for JEDI applications on Mac OS](buildscripts/Mac/README.md)| -This script sets up the module directory tree in `$JEDI_OPT`. It also sets up the compiler and mpi modules. The compiler and mpi modules are handled separately from the rest of the build because, when possible, we wish to exploit site-specific installations that maximize performance. +# Section 2: Building the JEDI Stack -**For this reason, the compiler and mpi modules are preceded by a `jedi-` label**. For example, to load the gnu compiler module and the openmpi software library, you would enter this: -``` -module load jedi-gnu/7.3.0 -module load jedi-openmpi/3.2.1 -``` -These `jedi-` modules are really meta-modules that will both load the compiler/mpi library and modify the `MODULEPATH` so the user has access to the software packages that will be built in Step 4. On HPC systems, these meta-modules will load the native modules provided by the system administrators. For example, `module load jedi-openmpi/3.2.1` will first load the native `openmpi/3.2.1` module and then modify the `MODULEPATH` accordingly to allow users to access the JEDI libraries. If this module is not available (e.g. in a container or in the cloud), then the `openmpi/3.2.1` module will be built from source and installed into `$JEDI_OPT`. +[The procedure to build the jedi-stack is described here](doc/Build.md) -So, in short, you should never load the compiler or MPI modules directly. Instead, you should always load the `jedi-` meta-modules as demonstrated above - they will provide everything you need to load and then use the JEDI software libraries. -## Step 4: Build JEDI Stack +# Section 3: Adding a New library/package -Now all that remains is to build the stack: -``` -./build_stack.sh [] -``` -Here `` is the same as in Step 3, namely a reference to the corresponding configuration file in the `config` directory. As in Step 2, if this argument is omitted, the default is to use `config/config_custom.sh`. - -For building on Mac OSX, use: -~~~~~~~~ -./build_stack.sh mac -~~~~~~~~ - -# Adding a New library/package - -If you want to add a new library to the stack you need to follow these steps: -1. write a new build script in buildscripts/libs, using existing scripts as a template -2. define a new control flag and add it to the config files in buildscripts/config -3. Add a call to the new build script in buildscripts/build\_stack.sh -4. Create a new module template at the appropriate place in the modulefiles directory, using existing files as a template +It is desirable to limit the number of software packages included in jedi-stack for several reasons. + +Adding dependencies to JEDI may adversely affect its portability. New dependencies could pose problems for some platforms and compilers or could conflict with other stack components. As the stack grows, it takes more work to maintain on multiple systems. + +Adding optional build scripts for packages not required by JEDI also increases maintenance work to test them and keep them up to date. We do not want to support build scripts that are not commonly used. + +For these reasons, additions to the JEDI stack must first be approved by the JEDI Software Infrastructure team. + +## Vetting process + +If you want to add a new library or software package to the jedi-stack you must first do the following: + +1. Create a Zenhub issue that describes the reason for the addition. Which JEDI components require it? Why is it beneficial to add to the stack? Can it be installed or acquired a different way? (If you do not have access to the internal JCSDA ZenHub boards you can create a thread on the [JCSDA forums](https://forums.jcsda.org)). Bring it to the attention of the JEDI Software Infrastructure team (JEDI 1) by "asssigning" or tagging individual members. + +2. The issue will be discussed at one or more meetings of the JEDI 1 team, possibly in consultation with the broader JEDI team. You may be contacted for further information. + +3. The JEDI 1 Team will either approve or decline the request for an addition to the jedi-stack. You will be informed of the decision through the ZenHub issue (or forum thread) or by other means (e.g. meetings, email, etc). + +## Required Code Changes + +If your request to add a package to the JEDI stack is approved by the JEDI 1 team, then you can proceed to add it to the jedi-stack with the following code changes: + +1. write a new build script in `buildscripts/libs`, using existing scripts as a template +2. define a new control flag and add it to the `choose_modules.sh` and `config_container*` scripts in `buildscripts/config` +3. Add a call to the new build script in `buildscripts/build_stack.sh` +4. Create a new module template at the appropriate place in the `modulefiles` directory, using existing files as a template +5. Add the package to the list of jedi-stack components in the (top-level) `README.md` file diff --git a/buildscripts/Mac/README.md b/buildscripts/Mac/README.md new file mode 100644 index 0000000..2f0d306 --- /dev/null +++ b/buildscripts/Mac/README.md @@ -0,0 +1,16 @@ +# Notes on building the Software Stack for JEDI applications on Mac OS + +Installing a minimal JEDI software stack on Mac OS (*catalina 10.15.7* and newer) is described in some detail elsewhere, in particular: + +[Minimum steps for working with JEDI natively on Mac OS](https://github.com/JCSDA-internal/jedi-docs/blob/develop/howto/macos/minimum.md) + +The [brew](https://brew.sh) package manager installs packages to their own directory (`/usr/local/Cellar/)` and then symlinks their files into the system location `/usr/local`. There are exceptions: Sometimes, brew cannot symlink to header files or libraries in `/usr/local`, and for some packages that have multiple related components, they are not installed in a common root location in `/usr/local/Cellar`. + +If you are building the optional jedi-stack component [ecFlow](https://confluence.ecmwf.int/display/ECFLOW/ecflow+home), two scripts are provided to properly symlink software packages that ecFlow requires: + +| software | helper script | Notes | +| ------- | --------------- | ----- | +| openssl | linkOpenSSLmacOS.sh | backs up deprecated versions of system libraries and symlinks to brew's version | +| boost / boost-python3 | linkBoostPythonLibsmacOS.sh | co-locates boost and boost-python libraries, allowing you to set the build config variable `BOOST_ROOT` | + +Running these scripts after installing openssl and boost / boost-python3 will properly set up symlinks for jedi-stack components that depend on them to build correctly. diff --git a/buildscripts/Mac/linkBoostPythonLibsmacOS.sh b/buildscripts/Mac/linkBoostPythonLibsmacOS.sh new file mode 100755 index 0000000..57981c4 --- /dev/null +++ b/buildscripts/Mac/linkBoostPythonLibsmacOS.sh @@ -0,0 +1,24 @@ +#!/bin/bash + +# Co-locate (via symlink) brew-installed boost / boost-python libraries and headers +# brew installs these packages in different locations; a typical build from source +# code would co-locate them, so that setting BOOST_ROOT for use in building other +# software is made easier. +# Steps: +# brew install boost boost-python3 +# run this script + +# Boost libraries and headers: version and location +boostLib_version=`brew list --versions | grep "boost " | cut -d ' ' -f 2` +boostLib_cellar=`brew --cellar boost` + +# Boost python libraries: version and location +boostPython_version=`brew list --versions | grep boost-python3 | cut -d ' ' -f 2` +boostPython_cellar=`brew --cellar boost-python3` + +echo 'Linking boost-python3 libraries into boost installation location' +for lib in `ls $boostPython_cellar/$boostPython_version/lib/lib*`; do + echo "Linking $lib into $boostLib_cellar/$boostLib_version/lib" + sudo ln -s $lib $boostLib_cellar/$boostLib_version/lib +done + diff --git a/buildscripts/Mac/linkOpenSSLmacOS.sh b/buildscripts/Mac/linkOpenSSLmacOS.sh new file mode 100755 index 0000000..3e1a77e --- /dev/null +++ b/buildscripts/Mac/linkOpenSSLmacOS.sh @@ -0,0 +1,45 @@ +#!/bin/bash + +# Replace macOS-supplied openssl libraries; macOS SIP often prevents proper linking +# Steps: +# brew install openssl@1.1 +# run this script + +version=`brew list --versions | grep openssl | cut -d ' ' -f 2` +cellar=`brew --cellar openssl` + +echo 'Linking openssl@1.1 v$version libraries and pkgconfig files' + +if [ -f /usr/local/lib/libssl.dylib ]; then + echo 'backing up existing libssl.dylib' + mv /usr/local/lib/libssl.dylib /usr/local/lib/libssl_bak.dylib +fi + +if [ -f /usr/local/lib/libcrypto.dylib ]; then + echo 'backing up existing libcrypto.dylib' + mv /usr/local/lib/libcrypto.dylib /usr/local/lib/libcrypto_bak.dylib +fi + +echo 'add symlink for openssl libs' + +if [ -f $cellar/$version/lib/libssl.1.1.dylib ]; then + sudo ln -s $cellar/$version/lib/libssl.1.1.dylib /usr/local/lib/libssl.dylib +fi + +if [ -f $cellar/$version/lib/libcrypto.1.1.dylib ]; then + sudo ln -s $cellar$version/lib/libcrypto.1.1.dylib /usr/local/lib/libcrypto.dylib +fi + +echo 'add symlink for openssl pkgconfig files' + +if [ -f $cellar/$version/lib/pkgconfig/openssl.pc ]; then + sudo ln -s $cellar/$version/lib/pkgconfig/openssl.pc /usr/local/lib/pkgconfig/openssl.pc +fi + +if [ -f $cellar/$version/lib/pkgconfig/libssl.pc ]; then + sudo ln -s $cellar/$version/lib/pkgconfig/libssl.pc /usr/local/lib/pkgconfig/libssl.pc +fi + +if [ -f $cellar/$version/lib/pkgconfig/libcrypto.pc ]; then + sudo ln -s $cellar/$version/lib/pkgconfig/libcrypto.pc /usr/local/lib/pkgconfig/libcrypto.pc +fi diff --git a/buildscripts/build_stack.sh b/buildscripts/build_stack.sh index 2d445cf..56ed809 100755 --- a/buildscripts/build_stack.sh +++ b/buildscripts/build_stack.sh @@ -77,8 +77,8 @@ $MODULES && (set +x; source $MODULESHOME/init/bash; module purge; set -x) # - should add a check at some point to see if they are already there. # this can be done in each script individually # it might warrant a --force flag to force rebuild when desired -build_lib CMAKE cmake 3.17.2 -build_lib UDUNITS udunits 2.2.26 +build_lib CMAKE cmake 3.20.0 +build_lib UDUNITS udunits 2.2.28 build_lib JPEG jpeg 9.1.0 build_lib ZLIB zlib 1.2.11 build_lib PNG png 1.6.35 @@ -86,9 +86,9 @@ build_lib SZIP szip 2.1.1 build_lib LAPACK lapack 3.8.0 build_lib BOOST_HDRS boost 1.68.0 headers-only build_lib EIGEN3 eigen 3.3.7 -build_lib BUFRLIB bufrlib 11.3.2 -build_lib ECBUILD ecbuild jcsda 3.3.2.jcsda3 -build_lib CGAL cgal 5.0.2 +build_lib BUFR bufr noaa-emc 11.5.0 +build_lib ECBUILD ecbuild ecmwf 3.6.1 +build_lib CGAL cgal 5.0.4 build_lib GITLFS git-lfs 2.11.0 #---------------------- @@ -97,11 +97,11 @@ build_lib HDF5 hdf5 1.12.0 build_lib PNETCDF pnetcdf 1.12.1 build_lib NETCDF netcdf 4.7.4 4.5.3 4.3.0 build_lib NCCMP nccmp 1.8.7.0 -build_lib ECKIT eckit jcsda 1.11.6.jcsda2 -build_lib FCKIT fckit jcsda 0.7.0.jcsda1 -build_lib ATLAS atlas jcsda 0.20.2.jcsda1 +build_lib ECKIT eckit ecmwf 1.16.0 +build_lib FCKIT fckit ecmwf 0.9.2 +build_lib ATLAS atlas ecmwf 0.24.1 build_lib ODB odb 0.18.1.r2 -build_lib ODC odc jcsda 1.0.3 +build_lib ODC odc ecmwf 2021.03.0 # =============================================================================== # Optional Extensions to the JEDI Stack @@ -115,17 +115,18 @@ build_lib NCEPLIBS nceplibs fv3 build_lib TKDIFF tkdiff 4.3.5 build_lib PYJEDI pyjedi build_lib PYBIND11 pybind11 2.5.0 -build_lib GSL_LITE gsl_lite 0.34.0 +build_lib GSL_LITE gsl_lite 0.37.0 build_lib GEOS geos 3.8.1 build_lib SQLITE sqlite 3.32.3 build_lib PROJ proj 7.1.0 build_lib JSON json 3.9.1 build_lib JSON_SCHEMA_VALIDATOR json-schema-validator 2.1.0 +build_lib ECFLOW ecflow ecmwf 5.5.3 boost 1.68.0 #---------------------- # These must be rebuilt for each MPI implementation build_lib GPTL gptl 8.0.3 -build_lib NCO nco 4.7.9 +build_lib NCO nco 4.9.9 build_lib PIO pio 2.5.1 build_lib FFTW fftw 3.3.8 build_lib BOOST_FULL boost 1.68.0 diff --git a/buildscripts/config/choose_modules.sh b/buildscripts/config/choose_modules.sh index 34f6c5e..7b62d4d 100644 --- a/buildscripts/config/choose_modules.sh +++ b/buildscripts/config/choose_modules.sh @@ -4,14 +4,14 @@ # http://www.apache.org/licenses/LICENSE-2.0. # Minimal JEDI Stack -export STACK_BUILD_CMAKE=N +export STACK_BUILD_CMAKE=Y export STACK_BUILD_GITLFS=N export STACK_BUILD_SZIP=Y export STACK_BUILD_UDUNITS=Y export STACK_BUILD_ZLIB=Y -export STACK_BUILD_LAPACK=N +export STACK_BUILD_LAPACK=Y export STACK_BUILD_BOOST_HDRS=Y -export STACK_BUILD_BUFRLIB=Y +export STACK_BUILD_BUFR=Y export STACK_BUILD_EIGEN3=Y export STACK_BUILD_HDF5=Y export STACK_BUILD_PNETCDF=Y @@ -19,17 +19,17 @@ export STACK_BUILD_NETCDF=Y export STACK_BUILD_NCCMP=Y export STACK_BUILD_ECBUILD=Y export STACK_BUILD_ECKIT=Y -export STACK_BUILD_FCKIT=N -export STACK_BUILD_ATLAS=N +export STACK_BUILD_FCKIT=Y +export STACK_BUILD_ATLAS=Y export STACK_BUILD_GSL_LITE=Y +export STACK_BUILD_PYBIND11=Y # Optional Additions export STACK_BUILD_ODC=N -export STACK_BUILD_PIO=Y +export STACK_BUILD_PIO=N export STACK_BUILD_GPTL=N export STACK_BUILD_NCO=N export STACK_BUILD_PYJEDI=N -export STACK_BUILD_PYBIND11=N export STACK_BUILD_NCEPLIBS=N export STACK_BUILD_JPEG=N export STACK_BUILD_PNG=N @@ -47,5 +47,6 @@ export STACK_BUILD_GEOS=N export STACK_BUILD_SQLITE=N export STACK_BUILD_PROJ=N export STACK_BUILD_FMS=N -export STACK_BUILD_JSON=N -export STACK_BUILD_JSON_SCHEMA_VALIDATOR=N +export STACK_BUILD_JSON=Y +export STACK_BUILD_JSON_SCHEMA_VALIDATOR=Y +export STACK_BUILD_ECFLOW=N diff --git a/buildscripts/config/config_ami-gnu-openmpi-dev.sh b/buildscripts/config/config_ami-gnu-openmpi-dev.sh new file mode 100644 index 0000000..2d7c600 --- /dev/null +++ b/buildscripts/config/config_ami-gnu-openmpi-dev.sh @@ -0,0 +1,39 @@ +#!/bin/bash +# © Copyright 2020 UCAR +# This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at +# http://www.apache.org/licenses/LICENSE-2.0. + + +# Compiler/MPI combination +export JEDI_COMPILER="gnu/9.3.0" +export JEDI_MPI="openmpi/4.1.0" + +# This tells jedi-stack how you want to build the compiler and mpi modules +# valid options include: +# native-module: load a pre-existing module (common for HPC systems) +# native-pkg: use pre-installed executables located in /usr/bin or /usr/local/bin, +# as installed by package managers like apt-get or hombrewo. +# This is a common option for, e.g., gcc/g++/gfortrant +# from-source: This is to build from source +export COMPILER_BUILD="native-pkg" +export MPI_BUILD="from-source" + +# Build options +export PREFIX=/optjedi/modules +export USE_SUDO=Y +export PKGDIR=pkg +export LOGDIR=buildscripts/log +export OVERWRITE=N +export NTHREADS=4 +export MAKE_CHECK=N +export MAKE_VERBOSE=N +export MAKE_CLEAN=N +export DOWNLOAD_ONLY=F +export STACK_EXIT_ON_FAIL=T +export WGET="wget -nv" +#Global compiler flags +export FFLAGS="" +export CFLAGS="" +export CXXFLAGS="" +export LDFLAGS="" + diff --git a/buildscripts/config/config_ami-intel-impi-dev.sh b/buildscripts/config/config_ami-intel-impi-dev.sh new file mode 100644 index 0000000..004d616 --- /dev/null +++ b/buildscripts/config/config_ami-intel-impi-dev.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# © Copyright 2020 UCAR +# This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at +# http://www.apache.org/licenses/LICENSE-2.0. + + +# Compiler/MPI combination +export JEDI_COMPILER="intel/2021.2" +export JEDI_MPI="impi/2021.2" +export PATH=/usr/local/bin:$PATH +export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH +export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH +export CPATH=/usr/local/include:$CPATH + +# This tells jedi-stack how you want to build the compiler and mpi modules +# valid options include: +# native-module: load a pre-existing module (common for HPC systems) +# native-pkg: use pre-installed executables located in /usr/bin or /usr/local/bin, +# as installed by package managers like apt-get or hombrewo. +# This is a common option for, e.g., gcc/g++/gfortran +# from-source: This is to build from source +export COMPILER_BUILD="native-pkg" +export MPI_BUILD="native-pkg" + +# Build options +export PREFIX=/optjedi/modules +export USE_SUDO=Y +export PKGDIR=pkg +export LOGDIR=buildscripts/log +export OVERWRITE=N +export NTHREADS=4 +export MAKE_CHECK=N +export MAKE_VERBOSE=N +export MAKE_CLEAN=N +export DOWNLOAD_ONLY=F +export STACK_EXIT_ON_FAIL=T +export WGET="wget -nv" + +#Global compiler flags +export FFLAGS="" +export CFLAGS="" +export CXXFLAGS="" +export LDFLAGS="" + diff --git a/buildscripts/config/config_container-clang-mpich-dev.sh b/buildscripts/config/config_container-clang-mpich-dev.sh index 55fc60a..4c31f2c 100644 --- a/buildscripts/config/config_container-clang-mpich-dev.sh +++ b/buildscripts/config/config_container-clang-mpich-dev.sh @@ -27,7 +27,7 @@ export OVERWRITE=N export NTHREADS=4 export MAKE_CHECK=N export MAKE_VERBOSE=N -export MAKE_CLEAN=Y +export MAKE_CLEAN=N export DOWNLOAD_ONLY=F export STACK_EXIT_ON_FAIL=T export WGET="wget -nv" @@ -38,7 +38,7 @@ export CXXFLAGS="" export LDFLAGS="" # Minimal JEDI Stack -export STACK_BUILD_CMAKE=N +export STACK_BUILD_CMAKE=Y export STACK_BUILD_GITLFS=N export STACK_BUILD_SZIP=Y export STACK_BUILD_UDUNITS=Y @@ -46,7 +46,7 @@ export STACK_BUILD_ZLIB=Y export STACK_BUILD_LAPACK=Y export STACK_BUILD_BOOST_HDRS=Y export STACK_BUILD_EIGEN3=Y -export STACK_BUILD_BUFRLIB=Y +export STACK_BUILD_BUFR=Y export STACK_BUILD_HDF5=Y export STACK_BUILD_PNETCDF=Y export STACK_BUILD_NETCDF=Y @@ -54,16 +54,16 @@ export STACK_BUILD_NCCMP=Y export STACK_BUILD_NCO=Y export STACK_BUILD_ECBUILD=Y export STACK_BUILD_ECKIT=Y -export STACK_BUILD_FCKIT=N -export STACK_BUILD_ATLAS=N -export STACK_BUILD_ODC=Y +export STACK_BUILD_FCKIT=Y +export STACK_BUILD_ATLAS=Y +export STACK_BUILD_PYBIND11=Y +export STACK_BUILD_GSL_LITE=Y # Optional Additions +export STACK_BUILD_ODC=Y export STACK_BUILD_PIO=Y export STACK_BUILD_GPTL=N export STACK_BUILD_PYJEDI=Y -export STACK_BUILD_PYBIND11=Y -export STACK_BUILD_GSL_LITE=Y export STACK_BUILD_NCEPLIBS=N export STACK_BUILD_JPEG=N export STACK_BUILD_PNG=N @@ -76,10 +76,10 @@ export STACK_BUILD_ESMF=N export STACK_BUILD_BASELIBS=N export STACK_BUILD_PDTOOLKIT=N export STACK_BUILD_TAU2=N -export STACK_BUILD_CGAL=N +export STACK_BUILD_CGAL=Y export STACK_BUILD_GEOS=N export STACK_BUILD_SQLITE=N export STACK_BUILD_PROJ=N -export STACK_BUILD_JSON=N +export STACK_BUILD_JSON=Y export STACK_BUILD_JSON_SCHEMA_VALIDATOR=Y export STACK_BUILD_FMS=N diff --git a/buildscripts/config/config_container-gnu-openmpi-dev.sh b/buildscripts/config/config_container-gnu-openmpi-dev.sh index e01f17d..db70559 100644 --- a/buildscripts/config/config_container-gnu-openmpi-dev.sh +++ b/buildscripts/config/config_container-gnu-openmpi-dev.sh @@ -5,9 +5,8 @@ # Compiler/MPI combination -export JEDI_COMPILER="gnu/7.3.0" -export JEDI_MPI="openmpi/3.1.2" -#export JEDI_MPI="mpich/3.2.1" +export JEDI_COMPILER="gnu/9.3.0" +export JEDI_MPI="openmpi/4.1.0" # This tells jedi-stack how you want to build the compiler and mpi modules # valid options include: @@ -28,7 +27,7 @@ export OVERWRITE=N export NTHREADS=4 export MAKE_CHECK=N export MAKE_VERBOSE=N -export MAKE_CLEAN=Y +export MAKE_CLEAN=N export DOWNLOAD_ONLY=F export STACK_EXIT_ON_FAIL=T export WGET="wget -nv" @@ -39,7 +38,7 @@ export CXXFLAGS="" export LDFLAGS="" # Minimal JEDI Stack -export STACK_BUILD_CMAKE=N +export STACK_BUILD_CMAKE=Y export STACK_BUILD_GITLFS=N export STACK_BUILD_SZIP=Y export STACK_BUILD_UDUNITS=Y @@ -48,23 +47,23 @@ export STACK_BUILD_LAPACK=Y export STACK_BUILD_BOOST_HDRS=Y export STACK_BUILD_EIGEN3=Y export STACK_BUILD_HDF5=Y -export STACK_BUILD_BUFRLIB=Y +export STACK_BUILD_BUFR=Y export STACK_BUILD_PNETCDF=Y export STACK_BUILD_NETCDF=Y export STACK_BUILD_NCCMP=Y export STACK_BUILD_NCO=Y export STACK_BUILD_ECBUILD=Y export STACK_BUILD_ECKIT=Y -export STACK_BUILD_FCKIT=N -export STACK_BUILD_ATLAS=N -export STACK_BUILD_ODC=Y +export STACK_BUILD_FCKIT=Y +export STACK_BUILD_ATLAS=Y +export STACK_BUILD_PYBIND11=Y +export STACK_BUILD_GSL_LITE=Y # Optional Additions +export STACK_BUILD_ODC=Y export STACK_BUILD_PIO=Y export STACK_BUILD_GPTL=N export STACK_BUILD_PYJEDI=Y -export STACK_BUILD_PYBIND11=Y -export STACK_BUILD_GSL_LITE=Y export STACK_BUILD_NCEPLIBS=N export STACK_BUILD_JPEG=N export STACK_BUILD_PNG=N @@ -77,10 +76,10 @@ export STACK_BUILD_ESMF=N export STACK_BUILD_BASELIBS=N export STACK_BUILD_PDTOOLKIT=N export STACK_BUILD_TAU2=N -export STACK_BUILD_CGAL=N +export STACK_BUILD_CGAL=Y export STACK_BUILD_GEOS=N export STACK_BUILD_SQLITE=N export STACK_BUILD_PROJ=N -export STACK_BUILD_JSON=N +export STACK_BUILD_JSON=Y export STACK_BUILD_JSON_SCHEMA_VALIDATOR=Y export STACK_BUILD_FMS=N diff --git a/buildscripts/config/config_container-intel-impi-app.sh b/buildscripts/config/config_container-intel-impi-app.sh index 1663d01..edf3e3e 100644 --- a/buildscripts/config/config_container-intel-impi-app.sh +++ b/buildscripts/config/config_container-intel-impi-app.sh @@ -32,7 +32,7 @@ export OVERWRITE=N export NTHREADS=4 export MAKE_CHECK=N export MAKE_VERBOSE=N -export MAKE_CLEAN=Y +export MAKE_CLEAN=N export DOWNLOAD_ONLY=F export STACK_EXIT_ON_FAIL=T export WGET="wget -nv" @@ -43,7 +43,7 @@ export CXXFLAGS="" export LDFLAGS="" # Minimal JEDI Stack -export STACK_BUILD_CMAKE=N +export STACK_BUILD_CMAKE=Y export STACK_BUILD_GITLFS=N export STACK_BUILD_SZIP=Y export STACK_BUILD_UDUNITS=Y @@ -51,7 +51,7 @@ export STACK_BUILD_ZLIB=Y export STACK_BUILD_LAPACK=N export STACK_BUILD_BOOST_HDRS=Y export STACK_BUILD_EIGEN3=Y -export STACK_BUILD_BUFRLIB=N +export STACK_BUILD_BUFR=Y export STACK_BUILD_HDF5=Y export STACK_BUILD_PNETCDF=Y export STACK_BUILD_NETCDF=Y @@ -61,10 +61,12 @@ export STACK_BUILD_ECBUILD=Y export STACK_BUILD_ECKIT=Y export STACK_BUILD_FCKIT=N export STACK_BUILD_ATLAS=N -export STACK_BUILD_ODC=N +export STACK_BUILD_PYBIND11=Y +export STACK_BUILD_GSL_LITE=Y # Optional Additions -export STACK_BUILD_PIO=N +export STACK_BUILD_ODC=Y +export STACK_BUILD_PIO=Y export STACK_BUILD_PYJEDI=N export STACK_BUILD_NCEPLIBS=N export STACK_BUILD_JASPER=N @@ -76,6 +78,8 @@ export STACK_BUILD_ESMF=N export STACK_BUILD_BASELIBS=N export STACK_BUILD_PDTOOLKIT=N export STACK_BUILD_TAU2=N +export STACK_BUILD_CGAL=Y + # For help with finding packages export PATH=${PREFIX}/bin:$PATH diff --git a/buildscripts/config/config_container-intel-impi-dev.sh b/buildscripts/config/config_container-intel-impi-dev.sh index ae82df9..b16f118 100644 --- a/buildscripts/config/config_container-intel-impi-dev.sh +++ b/buildscripts/config/config_container-intel-impi-dev.sh @@ -5,9 +5,8 @@ # Compiler/MPI combination -export JEDI_COMPILER="intel/19.0.5" -export JEDI_MPI="impi/19.0.5" -source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64 +export JEDI_COMPILER="intel/2021.1" +export JEDI_MPI="impi/2021.1" export PATH=/usr/local/bin:$PATH export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH export LIBRARY_PATH=/usr/local/lib:$LIBRARY_PATH @@ -18,7 +17,7 @@ export CPATH=/usr/local/include:$CPATH # native-module: load a pre-existing module (common for HPC systems) # native-pkg: use pre-installed executables located in /usr/bin or /usr/local/bin, # as installed by package managers like apt-get or hombrewo. -# This is a common option for, e.g., gcc/g++/gfortrant +# This is a common option for, e.g., gcc/g++/gfortran # from-source: This is to build from source export COMPILER_BUILD="native-pkg" export MPI_BUILD="native-pkg" @@ -32,7 +31,7 @@ export OVERWRITE=N export NTHREADS=4 export MAKE_CHECK=N export MAKE_VERBOSE=Y -export MAKE_CLEAN=Y +export MAKE_CLEAN=N export DOWNLOAD_ONLY=F export STACK_EXIT_ON_FAIL=T export WGET="wget -nv" @@ -43,7 +42,7 @@ export CXXFLAGS="" export LDFLAGS="" # Minimal JEDI Stack -export STACK_BUILD_CMAKE=N +export STACK_BUILD_CMAKE=Y export STACK_BUILD_GITLFS=N export STACK_BUILD_SZIP=Y export STACK_BUILD_UDUNITS=Y @@ -51,7 +50,7 @@ export STACK_BUILD_ZLIB=Y export STACK_BUILD_LAPACK=N export STACK_BUILD_BOOST_HDRS=Y export STACK_BUILD_EIGEN3=Y -export STACK_BUILD_BUFRLIB=Y +export STACK_BUILD_BUFR=Y export STACK_BUILD_HDF5=Y export STACK_BUILD_PNETCDF=Y export STACK_BUILD_NETCDF=Y @@ -59,16 +58,16 @@ export STACK_BUILD_NCCMP=Y export STACK_BUILD_NCO=N export STACK_BUILD_ECBUILD=Y export STACK_BUILD_ECKIT=Y -export STACK_BUILD_FCKIT=N -export STACK_BUILD_ATLAS=N -export STACK_BUILD_ODC=Y +export STACK_BUILD_FCKIT=Y +export STACK_BUILD_ATLAS=Y +export STACK_BUILD_PYBIND11=Y +export STACK_BUILD_GSL_LITE=Y # Optional Additions +export STACK_BUILD_ODC=Y export STACK_BUILD_PIO=Y export STACK_BUILD_GPTL=N export STACK_BUILD_PYJEDI=Y -export STACK_BUILD_PYBIND11=Y -export STACK_BUILD_GSL_LITE=Y export STACK_BUILD_NCEPLIBS=N export STACK_BUILD_JPEG=N export STACK_BUILD_PNG=N @@ -81,7 +80,7 @@ export STACK_BUILD_ESMF=N export STACK_BUILD_BASELIBS=N export STACK_BUILD_PDTOOLKIT=N export STACK_BUILD_TAU2=N -export STACK_BUILD_CGAL=N -export STACK_BUILD_JSON=N +export STACK_BUILD_CGAL=Y +export STACK_BUILD_JSON=Y export STACK_BUILD_JSON_SCHEMA_VALIDATOR=Y export STACK_BUILD_FMS=N diff --git a/buildscripts/config/config_gentoo.sh b/buildscripts/config/config_gentoo.sh index 89943ca..2aede30 100644 --- a/buildscripts/config/config_gentoo.sh +++ b/buildscripts/config/config_gentoo.sh @@ -66,7 +66,6 @@ export HWLOC_ROOT=/usr ## Don't add any new dependencies to this file # Packages that produce Fortran modules -export STACK_BUILD_BUFRLIB=Y export STACK_BUILD_HDF5=Y export STACK_BUILD_PNETCDF=Y export STACK_BUILD_NETCDF=Y diff --git a/buildscripts/libs/build_atlas.sh b/buildscripts/libs/build_atlas.sh index 713bfc8..0c7dd5b 100755 --- a/buildscripts/libs/build_atlas.sh +++ b/buildscripts/libs/build_atlas.sh @@ -51,7 +51,8 @@ git checkout $version [[ -d build ]] && $SUDO rm -rf build mkdir -p build && cd build -ecbuild --prefix=$prefix --build=Release .. +# set install prefix and CMAKE_INSTALL_LIBDIR to make sure it installs as lib, not lib64 +ecbuild --build=release -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make -j${NTHREADS:-4} install diff --git a/buildscripts/libs/build_boost.sh b/buildscripts/libs/build_boost.sh index 1661596..feaee83 100755 --- a/buildscripts/libs/build_boost.sh +++ b/buildscripts/libs/build_boost.sh @@ -11,7 +11,7 @@ version=$1 cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"} software=$name\_$(echo $version | sed 's/\./_/g') -url="https://dl.bintray.com/boostorg/release/$version/source/$software.tar.gz" +url="https://boostorg.jfrog.io/artifactory/main/release/$version/source/$software.tar.gz" [[ -d $software ]] || ( $WGET $url; tar -xf $software.tar.gz ) [[ ${DOWNLOAD_ONLY} =~ [yYtT] ]] && exit 0 [[ -d $software ]] && cd $software || ( echo "$software does not exist, ABORT!"; exit 1 ) @@ -70,6 +70,7 @@ compName=$(echo $compiler | cut -d- -f1) case "$compName" in gnu ) MPICC=$(which mpicc) ; toolset=gcc ;; intel ) MPICC=$(which mpiicc) ; toolset=intel ;; + clang ) MPICC=$(which mpiicc) ; toolset=clang ;; * ) echo "Unknown compiler = $compName, ABORT!"; exit 1 ;; esac @@ -85,7 +86,7 @@ EOF rm -f $HOME/user-config.jam [[ -z $mpi ]] && rm -f ./user-config.jam || mv -f ./user-config.jam $HOME -./bootstrap.sh --with-toolset=$toolset +./bootstrap.sh --with-toolset=$toolset --with-python=`which python3` ./b2 install $debug --prefix=$BoostBuild export PATH="$BoostBuild/bin:$PATH" diff --git a/buildscripts/libs/build_bufrlib.sh b/buildscripts/libs/build_bufr.sh similarity index 54% rename from buildscripts/libs/build_bufrlib.sh rename to buildscripts/libs/build_bufr.sh index b289c6b..8380836 100755 --- a/buildscripts/libs/build_bufrlib.sh +++ b/buildscripts/libs/build_bufr.sh @@ -1,12 +1,15 @@ #!/bin/bash # © Copyright 2020 UCAR +# © Copyright 2020 NOAA/NCEP/EMC # This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at # http://www.apache.org/licenses/LICENSE-2.0. set -ex -name="bufrlib" -version=$1 +name="bufr" +# source should either be noaa-emc or jcsda +source=$1 +version=$2 # Hyphenated version used for install prefix compiler=$(echo $JEDI_COMPILER | sed 's/\//-/g') @@ -20,39 +23,47 @@ if $MODULES; then module list set -x - prefix="${PREFIX:-"/opt/modules"}/$compiler/$name/$version" + prefix="${PREFIX:-"/opt/modules"}/$compiler/$name/$source-$version" if [[ -d $prefix ]]; then [[ $OVERWRITE =~ [yYtT] ]] && ( echo "WARNING: $prefix EXISTS: OVERWRITING!";$SUDO rm -rf $prefix ) \ || ( echo "WARNING: $prefix EXISTS, SKIPPING"; exit 1 ) fi else - prefix=${BUFRLIB_ROOT:-"/usr/local"} + prefix=${BUFR_ROOT:-"/usr/local"} fi export FC=$SERIAL_FC +export F90=$SERIAL_FC export CC=$SERIAL_CC -## Implementation can be switched to NCEPLIBS-bufr when shared libraries are supported -software=bufrlib -#software=NCEPLIBS-bufr +software=NCEPLIBS-bufr # Release git tag name -tag=bufr_v$version +if [[ ${source} == "jcsda-internal" ]] +then + gitOrg="jcsda-internal" + tag=$version +else + gitOrg="${source}" + tag=bufr_v$version +fi cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"} -[[ -d $software ]] || git clone https://github.com/JCSDA/$software.git +[[ -d $software ]] || git clone https://github.com/$gitOrg/$software.git [[ ${DOWNLOAD_ONLY} =~ [yYtT] ]] && exit 0 [[ -d $software ]] && cd $software || ( echo "$software does not exist, ABORT!"; exit 1 ) git fetch git checkout --detach $tag -[[ -d build ]] && rm -rf build +#[[ -d build ]] && rm -rf build +[[ -d build ]] && $SUDO rm -rf build +mkdir -p build && cd build -cmake -H. -Bbuild -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=1 -DOPT_IPO=OFF -cd build +cmake -DENABLE_PYTHON=ON -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make install # generate modulefile from template -$MODULES && update_modules compiler $name $version \ - || echo $name $version >> ${JEDI_STACK_ROOT}/jedi-stack-contents.log +pythonVersion=$(`which python3` -c 'import sys;print(sys.version_info[0],".",sys.version_info[1],sep="")') +$MODULES && update_modules compiler $name $source-$version $pythonVersion \ + || echo $name $source-$version >> ${JEDI_STACK_ROOT}/jedi-stack-contents.log diff --git a/buildscripts/libs/build_cgal.sh b/buildscripts/libs/build_cgal.sh index b65e11e..63c5148 100755 --- a/buildscripts/libs/build_cgal.sh +++ b/buildscripts/libs/build_cgal.sh @@ -33,6 +33,7 @@ if $MODULES; then module try-load cmake module try-load boost-headers module try-load zlib + module try-load eigen module list set -x @@ -49,13 +50,23 @@ fi cd $JEDI_STACK_ROOT/${PKGDIR:-"pkg"} software="CGAL-"$version -url="https://github.com/CGAL/cgal/releases/download/releases%2FCGAL-$version/$software-library.tar.xz" +url="https://github.com/CGAL/cgal/releases/download/v$version/$software-library.tar.xz" [[ -d $software ]] || ( $WGET $url; tar -xf $software-library.tar.xz ) [[ ${DOWNLOAD_ONLY} =~ [yYtT] ]] && exit 0 [[ -d $software ]] && cd $software || ( echo "$software does not exist, ABORT!"; exit 1 ) -cmake . -DCMAKE_INSTALL_PREFIX=$prefix -VERBOSE=$MAKE_VERBOSE $SUDO make install +# Apply a patch to fix CMake intel compiler flags. +# Remove when possible or update as needed. +if [[ $version == "5.0.4" ]]; then + patch --merge -p1 < ${JEDI_STACK_ROOT}/buildscripts/libs/patches/${software}-intel-fpmodel-flag-fix.patch +else + echo "Error: Must generate new patch for unsupported CGal version: $version" + exit 1 +fi + +[[ -d _build ]] && rm -rf _build +cmake -H. -B_build -DCMAKE_INSTALL_PREFIX=$prefix -DWITH_CGAL_Qt5=0 -DCGAL_DISABLE_GMP=1 -DEIGEN3_INCLUDE_DIR=$EIGEN_ROOT/include -DCMAKE_INSTALL_LIBDIR=lib +cd _build && VERBOSE=$MAKE_VERBOSE $SUDO make install # generate modulefile from template $MODULES && update_modules core $name $version \ diff --git a/buildscripts/libs/build_cmake.sh b/buildscripts/libs/build_cmake.sh index 78e045c..006ba17 100755 --- a/buildscripts/libs/build_cmake.sh +++ b/buildscripts/libs/build_cmake.sh @@ -15,7 +15,6 @@ version=$1 if $MODULES; then module load jedi-$JEDI_COMPILER - module try-load cmake module list prefix="${PREFIX:-"/opt/modules"}/core/$name/$version" diff --git a/buildscripts/libs/build_ecbuild.sh b/buildscripts/libs/build_ecbuild.sh index a415f47..2e09d14 100755 --- a/buildscripts/libs/build_ecbuild.sh +++ b/buildscripts/libs/build_ecbuild.sh @@ -38,7 +38,8 @@ git checkout $version [[ -d build ]] && $SUDO rm -rf build mkdir -p build && cd build -cmake -DCMAKE_INSTALL_PREFIX=$prefix .. +# set install prefix and CMAKE_INSTALL_LIBDIR to make sure it installs as lib, not lib64 +cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make install diff --git a/buildscripts/libs/build_ecflow.sh b/buildscripts/libs/build_ecflow.sh new file mode 100755 index 0000000..25309c5 --- /dev/null +++ b/buildscripts/libs/build_ecflow.sh @@ -0,0 +1,126 @@ +#!/bin/bash +# © Copyright 2020 UCAR +# This software is licensed under the terms of the Apache Licence Version 2.0 which can be obtained at +# http://www.apache.org/licenses/LICENSE-2.0. + +set -x + +name="ecflow" +source=$1 +version=$2 +boost=$3 +boost_version=$4 + +cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"} + +software=$name +[[ -d $software ]] || git clone https://github.com/$source/$software.git +[[ ${DOWNLOAD_ONLY} =~ [yYtT] ]] && exit 0 +[[ -d $software ]] && cd $software || ( echo "$software does not exist, ABORT!"; exit 1 ) +git fetch --tags +git checkout $version + +compiler=$(echo $JEDI_COMPILER | sed 's/\//-/g') + +if $MODULES; then + set +x + source $MODULESHOME/init/bash + module load jedi-$JEDI_COMPILER + module try-load cmake git python qt + module list + set -x + + prefix="${PREFIX:-"/opt/modules"}/$compiler/$name/$version" + if [[ -d $prefix ]]; then + [[ $OVERWRITE =~ [yYtT] ]] && ( echo "WARNING: $prefix EXISTS: OVERWRITING!";$SUDO rm -rf $prefix ) \ + || ( echo "WARNING: $prefix EXISTS, SKIPPING"; exit 1 ) + fi +else + prefix=${ECFLOW_ROOT:-"/usr/local"} +fi + +host=$(uname -s) +if [[ "$host" == "Darwin" ]]; then + [[ -d `brew --cellar boost` ]] && boost_root=`brew --prefix boost` + [[ -z $boost_root ]] || echo "Using brew-installed boost headers and libraries" + + [[ -d `brew --cellar openssl` ]] && openssl_root=`brew --prefix openssl` + if [ -z $openssl_root ]; then + echo "OpenSSL must be installed for ecFlow, ABORT!" + exit 1 + fi + + [[ -d `brew --cellar qt` ]] && qt_root=`brew --prefix qt` + if [ ! -z $qt_root ]; then + QT="-DCMAKE_PREFIX_PATH=$qt_root" + else + echo "Qt must be installed for ecFlow UI, ABORT!" + exit 1 + fi +fi + +# boost component; build if not otherwise present +if [ -z $boost_root ]; then + echo "Building boost from source" + boost_software=$boost\_$(echo $boost_version | sed 's/\./_/g') + url="https://dl.bintray.com/boostorg/release/$boost_version/source/$boost_software.tar.gz" + [[ -d $boost_software ]] || ( $WGET $url; tar -xf $boost_software.tar.gz ) + [[ -d $boost_software ]] && cd $boost_software || ( echo "$boost_software does not exist, ABORT!"; exit 1 ) + + debug="--debug-configuration" + + BoostRoot=$(pwd) + BoostBuild=$BoostRoot/BoostBuild + build_boost=$BoostRoot/build_boost + [[ -d $BoostBuild ]] && rm -rf $BoostBuild + [[ -d $build_boost ]] && rm -rf $build_boost + cd $BoostRoot/tools/build + + compName=$(echo $compiler | cut -d- -f1) + case "$compName" in + gnu ) toolset=gcc ;; + intel ) toolset=intel ;; + clang ) toolset=clang ;; + * ) echo "Unknown compiler = $compName, ABORT!"; exit 1 ;; + esac + + ./bootstrap.sh --with-toolset=$toolset --with-python=`which python3` + ./b2 install $debug --prefix=$BoostBuild + + export PATH="$BoostBuild/bin:$PATH" + + cd $BoostRoot + b2 $debug --build-dir=$build_boost address-model=64 toolset=$toolset stage + + $SUDO mkdir -p $prefix $prefix/include + $SUDO cp -R boost $prefix/include + $SUDO mv stage/lib $prefix + + boost_root=$prefix +fi + +# ecFlow component +cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"}/$software + +ecflowVersion=$(awk '/^project/ && /ecflow/ && /VERSION/ {for (I=1;I<=NF;I++) if ($I == "VERSION") {print $(I+1)};}' CMakeLists.txt) +pythonVersion=$(`which python3` -c 'import sys;print(sys.version_info[0],".",sys.version_info[1],sep="")') + +export FC=$SERIAL_FC +export CC=$SERIAL_CC +export CXX=$SERIAL_CXX + +[[ -d build ]] && $SUDO rm -rf build +mkdir -p build && cd build + +cmake -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_BUILD_TYPE=Release \ + -DBOOST_ROOT=$boost_root -DENABLE_STATIC_BOOST_LIBS=OFF \ + -DOPENSSL_ROOT_DIR=$openssl_root $QT .. +VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} +VERBOSE=$MAKE_VERBOSE $SUDO make install + +[[ -d $prefix/lib/cmake ]] && $SUDO rm -rf $prefix/lib/cmake +[[ -d $prefix/include/boost ]] && $SUDO rm -rf $prefix/include/boost + +# generate modulefile from template +$MODULES && update_modules compiler $name $version $pythonVersion \ + || echo $name $ecflowVersion >> ${JEDI_STACK_ROOT}/jedi-stack-contents.log \ No newline at end of file diff --git a/buildscripts/libs/build_eckit.sh b/buildscripts/libs/build_eckit.sh index 42145de..315f546 100755 --- a/buildscripts/libs/build_eckit.sh +++ b/buildscripts/libs/build_eckit.sh @@ -53,7 +53,8 @@ sed -i -e 's/project( eckit CXX/project( eckit CXX Fortran/' CMakeLists.txt [[ -d build ]] && $SUDO rm -rf build mkdir -p build && cd build -ecbuild -DCMAKE_INSTALL_PREFIX=$prefix --build=Release .. +# set install prefix and CMAKE_INSTALL_LIBDIR to make sure it installs as lib, not lib64 +ecbuild --build=release -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make install diff --git a/buildscripts/libs/build_fckit.sh b/buildscripts/libs/build_fckit.sh index 957f400..f3986cc 100755 --- a/buildscripts/libs/build_fckit.sh +++ b/buildscripts/libs/build_fckit.sh @@ -49,8 +49,9 @@ git checkout $version [[ -d build ]] && $SUDO rm -rf build mkdir -p build && cd build -ecbuild -DCMAKE_INSTALL_PREFIX=$prefix --build=Release .. -VERBOSE=$MAKE_VERBOSE $SUDO make -j${NTHREADS:-4} +# set install prefix and CMAKE_INSTALL_LIBDIR to make sure it installs as lib, not lib64 +ecbuild --build=release -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_INSTALL_LIBDIR=lib .. +VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make install # generate modulefile from template diff --git a/buildscripts/libs/build_fms.sh b/buildscripts/libs/build_fms.sh index 2e31e62..5c8ab51 100755 --- a/buildscripts/libs/build_fms.sh +++ b/buildscripts/libs/build_fms.sh @@ -35,9 +35,11 @@ else fi export FC=$MPI_FC +[[ -n $FC && `$FC --version` =~ GNU\ Fortran.*\ 1[0-9]\.[0-9]+ ]] && FC_GFORTRAN_10=1 export CC=$MPI_CC export FFLAGS+=" -fPIC -w" +[[ -n $FC_GFORTRAN_10 ]] && export FFLAGS+=" -fallow-argument-mismatch" export CFLAGS+=" -fPIC -w" export FCFLAGS="$FFLAGS" diff --git a/buildscripts/libs/build_gsl_lite.sh b/buildscripts/libs/build_gsl_lite.sh index ea773c0..ebae604 100755 --- a/buildscripts/libs/build_gsl_lite.sh +++ b/buildscripts/libs/build_gsl_lite.sh @@ -39,7 +39,7 @@ git fetch git checkout $branch [[ -d build ]] && rm -rf build mkdir -p build && cd build -cmake -DCMAKE_INSTALL_PREFIX=$prefix -DGSL_LITE_OPT_BUILD_TESTS=OFF -DGSL_LITE_OPT_INSTALL_COMPAT_HEADER=ON -DCMAKE_VERBOSE_MAKEFILE=1 .. +cmake -DCMAKE_INSTALL_PREFIX=$prefix -DGSL_LITE_OPT_BUILD_TESTS=OFF -DGSL_LITE_OPT_INSTALL_COMPAT_HEADER=ON -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_VERBOSE_MAKEFILE=1 .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} #[[ $MAKE_CHECK =~ [yYtT] ]] && make test VERBOSE=$MAKE_VERBOSE $SUDO make install diff --git a/buildscripts/libs/build_json-schema-validator.sh b/buildscripts/libs/build_json-schema-validator.sh index 0147d7e..1caa082 100755 --- a/buildscripts/libs/build_json-schema-validator.sh +++ b/buildscripts/libs/build_json-schema-validator.sh @@ -13,7 +13,7 @@ if $MODULES; then source $MODULESHOME/init/bash module load jedi-$JEDI_COMPILER module try-load cmake - module try-load json + module load json module list set -x @@ -24,7 +24,8 @@ if $MODULES; then fi else - prefix=${json_ROOT:-"/usr/local"} + prefix=${JSON_ROOT:-"/usr/local"} + JSON_DIR=${JSON_DIR:-$prefix/lib/cmake/nlohmann_json} fi cd $JEDI_STACK_ROOT/${PKGDIR:-"pkg"} @@ -38,12 +39,13 @@ url="https://github.com/pboettch/json-schema-validator/archive/$tarfile" [[ -d build ]] && rm -rf build mkdir -p build && cd build -echo nlohmann_json_DIR=$nlohmann_json_DIR +[[ -n $JSON_DIR ]] || ( echo "Required json cmake configuration not found, ABORT!"; exit 1 ) cmake .. \ -DCMAKE_INSTALL_PREFIX=$prefix \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_POLICY_DEFAULT_CMP0074=NEW \ -DBUILD_SHARED_LIBS=Y \ + -Dnlohmann_json_DIR=$JSON_DIR \ -DBUILD_TESTS=$MAKE_CHECK \ -DBUILD_EXAMPLES=N [[ $MAKE_CHECK =~ [yYtT] ]] && make test diff --git a/buildscripts/libs/build_lapack.sh b/buildscripts/libs/build_lapack.sh index 57ac880..fcca1ea 100755 --- a/buildscripts/libs/build_lapack.sh +++ b/buildscripts/libs/build_lapack.sh @@ -33,8 +33,9 @@ fi export FC=$SERIAL_FC export CC=$SERIAL_CC -export FFLAGS="-fPIC ${FFLAGS}" -export CFLAGS="-fPIC ${CFLAGS}" +export FFLAGS="-fPIC" +export FCFLAGS="-fPIC" +export CFLAGS="-fPIC" cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"} @@ -47,8 +48,8 @@ url="https://github.com/Reference-LAPACK/lapack/archive/$tarball" [[ -d build ]] && rm -rf build # Add CMAKE_INSTALL_LIBDIR to make sure it will be installed under lib not lib64 -cmake -H. -Bbuild -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_Fortran_COMPILER=$SERIAL_FC +cmake -H. -Bbuild -DCMAKE_INSTALL_PREFIX=$prefix -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_LIBDIR=lib \ + -DBUILD_SHARED_LIBS=ON -DCMAKE_Fortran_COMPILER=$SERIAL_FC -DCMAKE_Fortran_FLAGS=$FCFLAGS cd build VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} [[ $MAKE_CHECK =~ [yYtT] ]] && make check diff --git a/buildscripts/libs/build_mpi.sh b/buildscripts/libs/build_mpi.sh index 532f78f..f99d642 100755 --- a/buildscripts/libs/build_mpi.sh +++ b/buildscripts/libs/build_mpi.sh @@ -33,7 +33,6 @@ export CXX=$SERIAL_CXX export FC=$SERIAL_FC export FFLAGS+=" -fPIC" -[[ -n $FC_GFORTRAN_10 ]] && export FFLAGS+=" -fallow-argument-mismatch -fallow-invalid-boz" export CFLAGS+=" -fPIC" export CXXFLAGS+=" -fPIC" export FCFLAGS=${FFLAGS} @@ -82,6 +81,14 @@ case "$name" in else extra_conf="--enable-fortran --enable-cxx" fi + # gfortran-10+ compatibility flags + if [[ -n $FC_GFORTRAN_10 ]]; then + # Use these flags to build MPICH itself, but don't add to WRAPPER_FCFLAGS + export MPICHLIB_FCFLAGS+=" -fallow-argument-mismatch -fallow-invalid-boz" + export MPICHLIB_FFLAGS=${MPICHLIB_FCFLAGS} + # Disable check for mismatched args flags in confdb/aclocal_f77.ac + export pac_cv_prog_f77_mismatched_args=yes + fi [[ -n $HWLOC_ROOT ]] && extra_conf+=" --with-hwloc-prefix=${HWLOC_ROOT}" ;; * ) diff --git a/buildscripts/libs/build_netcdf.sh b/buildscripts/libs/build_netcdf.sh index 7d3ac12..990b579 100755 --- a/buildscripts/libs/build_netcdf.sh +++ b/buildscripts/libs/build_netcdf.sh @@ -120,8 +120,6 @@ echo "########################################################################## # Load netcdf-c before building netcdf-fortran $MODULES && module load netcdf -module list - set -x cd $curr_dir diff --git a/buildscripts/libs/build_odc.sh b/buildscripts/libs/build_odc.sh index 32eb459..2bd0d49 100755 --- a/buildscripts/libs/build_odc.sh +++ b/buildscripts/libs/build_odc.sh @@ -50,7 +50,7 @@ git checkout $version [[ -d build ]] && $SUDO rm -rf build mkdir -p build && cd build -ecbuild -DCMAKE_INSTALL_PREFIX=$prefix --build=Release .. +ecbuild --build=Release -DCMAKE_INSTALL_PREFIX=$prefix .. VERBOSE=$MAKE_VERBOSE make -j${NTHREADS:-4} VERBOSE=$MAKE_VERBOSE $SUDO make -j${NTHREADS:-4} install diff --git a/buildscripts/libs/build_pyjedi.sh b/buildscripts/libs/build_pyjedi.sh index 7661a54..4c78195 100755 --- a/buildscripts/libs/build_pyjedi.sh +++ b/buildscripts/libs/build_pyjedi.sh @@ -28,14 +28,3 @@ $SUDO python3 -m pip install -U pytest $SUDO python3 -m pip install -U ford $SUDO python3 -m pip install -U xarray $SUDO python3 -m pip install -U pyodc - -##################################################################### -# ncepbufr for python -##################################################################### - -cd ${JEDI_STACK_ROOT}/${PKGDIR:-"pkg"} -git clone https://github.com/JCSDA/py-ncepbufr.git -cd py-ncepbufr - -CC=gcc python3 setup.py build -$SUDO python3 setup.py install diff --git a/buildscripts/libs/patches/CGAL-5.0.4-intel-fpmodel-flag-fix.patch b/buildscripts/libs/patches/CGAL-5.0.4-intel-fpmodel-flag-fix.patch new file mode 100644 index 0000000..9d8f425 --- /dev/null +++ b/buildscripts/libs/patches/CGAL-5.0.4-intel-fpmodel-flag-fix.patch @@ -0,0 +1,13 @@ +diff --git a/cmake/modules/CGAL_SetupCGALDependencies.cmake b/cmake/modules/CGAL_SetupCGALDependencies.cmake +index 6590e31..55c783f 100644 +--- a/cmake/modules/CGAL_SetupCGALDependencies.cmake ++++ b/cmake/modules/CGAL_SetupCGALDependencies.cmake +@@ -147,7 +147,7 @@ function(CGAL_setup_CGAL_dependencies target) + if(WIN32) + target_compile_options(${target} ${keyword} "/fp:strict") + else() +- target_compile_options(${target} ${keyword} "-fp-model" "strict") ++ target_compile_options(${target} ${keyword} "$<1:-fp-model strict>") # Genexpr enforces grouping of args + endif() + elseif(CMAKE_CXX_COMPILER_ID MATCHES "SunPro") + message( STATUS "Using SunPro compiler, using STLPort 4." ) diff --git a/buildscripts/libs/update_modules.sh b/buildscripts/libs/update_modules.sh index bc91e91..124a970 100755 --- a/buildscripts/libs/update_modules.sh +++ b/buildscripts/libs/update_modules.sh @@ -10,6 +10,12 @@ # $1 = module path: valid options are core, compiler, or mpi # $2 = package name # $3 = package version +# $4 = python version (optional) +# +# Only specify the python version if you need to set that in the destination +# lua script. A non-empty setting of $4 will cause sed to be run on the +# destination lua script to substitue "@PYTHON_VERSION@" with the python version +# number specified in $4. function update_modules { OPT="${JEDI_OPT:-$OPT}" @@ -33,6 +39,9 @@ function update_modules { cd $to_dir $SUDO mkdir -p $2; cd $2 $SUDO cp $tmpl_file $3.lua + # Argument number 4 is python version. If not empty use sed to + # substitue python version into placeholder marks in the lua script. + [[ -n "$4" ]] && $SUDO sed -i -e "s/@PYTHON_VERSION@/$4/" $3.lua # Make the latest installed version the default [[ -e default ]] && $SUDO rm -f default diff --git a/doc/Build.md b/doc/Build.md new file mode 100644 index 0000000..694af50 --- /dev/null +++ b/doc/Build.md @@ -0,0 +1,151 @@ +# Building the JEDI Stack + +Building the JEDI software stack is a **Four-Step process**, as described in the following sections. + +[See here for additional tips on particular platforms](Platforms.md) + +## Step 1: Set up Basic Environment + +This is the most context-dependent part of the build process. How you proceed depends on the system you are on. Regardless of how you proceed, this step only needs to be done once for each system. + +Note - you can skip this step and move on to Step 2 if you are building a JEDI software container because all of the required software packages are already built into the jcsda/docker_base container, which you can just pull from Docker Hub. + +This step is most important for bare linux/unix systems, as you would get with a new cloud computing instance or a virtual machine (e.g. [vagrant](https://www.vagrantup.com)). For such systems, Step 1 consists of running the following script (all paths are relative to the base path of the jedi-stack repository): +``` +cd buildscripts +./setup_environment.sh +``` +where `` depends on your operating system and the context of the build (for example, if you're in a container or on the cloud or on an HPC system). Examples include `ubuntu/18.04` or `gentoo`. To see a list of supported options, run the script without any arguments: +``` +./setup_environment.sh +``` +**Warning: for some (not all) options, running this script requires root privileges.** + +The purpose of this script is to install some basic software packages, the most important of which are: +* GNU compiler suite (gcc, g++, gfortran) +* [Lmod](https://lmod.readthedocs.io/en/latest/index.html) module management system +* git, [git-lfs](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/developer_tools/gitlfs.html), and [git-flow](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/developer_tools/getting-started-with-gitflow.html) +* wget and curl +* make and [CMake](https://cmake.org) +* [doxygen](http://www.doxygen.nl) and [graphviz](https://www.graphviz.org) +* Debugging tools, including [kdbg](http://www.kdbg.org/) and valgrind + +Many of these are installed with package managers such as [apt-get](https://linux.die.net/man/8/apt-get) for Linux or [HomeBrew on Linux](https://docs.brew.sh/Homebrew-on-Linux) and [HomeBrew](https://brew.sh/) for Mac OSX. A few scripts are also provided in the libs directory for such packages as `Lmod` and `CMake` if you would rather install these from source. + +If you are using your own laptop or workstation, it is likely that most of these basic packages are already installed. If any of these packages are missing from your system (such as Lmod), you can manually install them with a package manager or with the build scripts. The items near the bottom of the list are not essential - if you don't have them on your system there is no need to install them. + +If you are building on a EMC RHEL7 workstation, use [HomeBrew on Linux](https://docs.brew.sh/Homebrew-on-Linux) to manually install the basic software packages. Follow the instructions at the HomeBrew on Linux website for installing HomeBrew itself, then run the following brew commands: +~~~~~~~~ +brew install lmod +brew install git +brew install git-lfs +brew install git-flow-avh +brew install cmake +brew install doxygen +brew install graphviz +brew install sphinx-doc +~~~~~~~~ + +If you are building on Mac OSX with Clang, then do not run the `setup_environments.sh` script, and instead use [HomeBrew](https://brew.sh/) to manually install the basic software packages. Homebrew can be also used to install any other basic tools that you want such as tkdiff, gdb, etc. Then [follow the instructions in the jedi-docs howto document for building and running JEDI](https://github.com/JCSDA-internal/jedi-docs/blob/develop/howto/macos/minimum.md). See the [Platform document](#MacPython) for tips on installing python on the Mac. Your Mac should have come with Clang compilers for C and C++ pre-installed, so in this scheme you are adding in GFortran (GNU) for compiling Fortran code. Once you have finished with the brew install commands, make sure to set `JEDI_OPT` in your environment as described below. + +If you're on an HPC system you can largely skip this step (but you still need to set the **JEDI_OPT** environment variable, see below) because most of these packages are probably already installed and available. However, there are a few items that you may wish to add by loading the appropriate modules (if they exist) - for example: +``` +module load doxygen git-lfs +``` +Some modules such as doxygen are optional so if they are not available then no need to worry. However, if `git-lfs` is not available, you can install it using the build script that is provided in jedi-stack. + +Note, however, if you are using [JEDI Modules](https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/using/jedi_environment/modules.html) you do not have to worry about this (or indeed, about this jedi-stack repository in general) - the JEDI team will make sure that the modules provided will include the packages you need. + +**IMPORTANT:** Another responsibility of the `setup_environment.sh` script is to define the `JEDI_OPT` environment variable and initialize the Lmod system. These actions are needed both for the build and to allow users to load the JEDI modules after you build them. `JEDI_OPT` specifies where the modules will be installed, with a default value of `JEDI_OPT=/opt/modules`. Note that this default value normally requires root permission so you would have to set the `USE_SUDO` flag (see Step 2). If you do not have root privileges (e.g. on an HPC system), you may wish to install your modules in a home or work directory, e.g. `JEDI_OPT=$HOME/opt/modules`. + +`JEDI_OPT` needs to be set and Lmod initialized in order to complete Steps 2-4. But, these actions also need to be performed in order for users to use the modules. Be sure to add the following to your shell initialization scripts (example shown is for bash): +``` +# For jedi-stack +export JEDI_OPT=/opt/modules + +# For lmod modules +. /usr/local/opt/lmod/init/profile +module use $JEDI_OPT/modulefiles/core +``` +and make sure these settings are in place before proceeding to Steps 2-4. + +## Step 2: Configure Build + +The next step is to choose what components of the stack you wish to build and to specify any other aspects of the build that you would like. This is normally done by first editing one of the platform-specific `buildscripts/config/config_*.sh` files. Or, if your platform is not among the options, you can edit the `config_custom.sh` file. Then, edit the `buildscripts/config/choose_modules.sh` file to choose which modules you wish to build. Note that some are prerequisites of others. For example, you must build hdf5 before you build netcdf. + +Here we describe some of the parameter settings available in these configuration files. + +For building on Mac OSX, a configuration file (`config_mac.sh`) is provided. This configuration is set up to build using Clang 10.0.0 with gfortran and OpenMPI. You may wish to edit this file for building with a different compiler/mpi set. + +For building on an EMC RHEL7 workstation, a configuration file (`config_rhel7emc.sh`) is provided. + +**JEDI_COMPILER** This defines the vendor and version of the compiler you wish to use for this build. The format is the same as what you would typically use in a module load command: +``` +export COMPILER=/ +``` +For example, `COMPILER=gnu/7.3.0`. + +**JEDI_MPI** is the MPI library you wish to use for this build. The format is the same as for `COMPILER`, for example: `export MPI=openmpi/3.1.2`. + +**PREFIX** is the directory where the software packages will be installed. Normally this is set to be the same as the `JEDI_OPT` environment variable (default value `/opt/modules`), though this is not required. If `JEDI_OPT` and `PREFIX` are both the same, then the software installation trees (the top level of each being is the compiler, e.g. `gnu-7.3.0`) will branch directly off of `$JEDI_OPT` while the module files will be located in the `modulefiles` subdirectory. + +**USE_SUDO** If `PREFIX` is set to a value that requires root permission to write to, such as `/opt/modules`, then this flag should be enabled. + +_**NOTE: To enable a boolean flag use a single-digit `Y` or `T`. To disable, use `N` or `F` (case insensitive)**_ + +**PKGDIR** is the directory where tarred or zipped software files will be downloaded and compiled. Unlike `PREFIX`, this is a relative path, based on the root path of the repository. Individual software packages can be downloaded manually to this directory and untarred, but this is not required. Most build scripts will look for directory `pkg/pkgName-pkgVersion` e.g. `pkg/hdf5-1_10_3`. + +**LOGDIR** is the directory where log files from the build will be written, relative to the root path of the repository. + +**OVERWRITE** If set, this flag will cause the build script to remove the current installation, if any exists, and replace it with the new version of each software package in question. If this is not set, the build will bypass software packages that are already installed. + +**NTHREADS** The number of threads to use for parallel builds + +**MAKE_CHECK** Run `make check` after build + +**MAKE_VERBOSE** Print out extra information to the log files during the build + +The remaining items enable or disable builds of each software package. The following software can optionally be built with the scripts under `buildscripts`. Unless otherwise noted, the packages are built in Step 4 using the `build_scripts.sh` script. + +**IMPORTANT: Steps 2, 3, and 4 need to be repeated for each compiler/mpi combination that you wish to install.** The new packages will be installed alongside any previously-existing packages that may already exist and that are built from other compiler/mpi combinations. + +## Step 3: Set Up Compiler, MPI, and Module System + +The next step is to run this from the buildscripts directory: +``` +./setup_modules.sh [] +``` +where `` points to the configuration script that you wish to use, as described in Step 2. The name of this file is `config/config_`. For example, to use the `config/config_custom.sh` you would enter this: +``` +./setup_modules.sh custom +``` +If no arguments are specified, the default is `custom`. Note that you can skip this step as well for container builds because we currenly include only one compiler/mpi combination in each container. So, each package is only build once and there is no need for modules. + +For building on Mac OSX, use: +``` +./setup_modules.sh mac +``` + +This script sets up the module directory tree in `$JEDI_OPT`. It also sets up the compiler and mpi modules. The compiler and mpi modules are handled separately from the rest of the build because, when possible, we wish to exploit site-specific installations that maximize performance. + +**For this reason, the compiler and mpi modules are preceded by a `jedi-` label**. For example, to load the gnu compiler module and the openmpi software library, you would enter this: +``` +module load jedi-gnu/7.3.0 +module load jedi-openmpi/3.2.1 +``` +These `jedi-` modules are really meta-modules that will both load the compiler/mpi library and modify the `MODULEPATH` so the user has access to the software packages that will be built in Step 4. On HPC systems, these meta-modules will load the native modules provided by the system administrators. For example, `module load jedi-openmpi/3.2.1` will first load the native `openmpi/3.2.1` module and then modify the `MODULEPATH` accordingly to allow users to access the JEDI libraries. If this module is not available (e.g. in a container or in the cloud), then the `openmpi/3.2.1` module will be built from source and installed into `$JEDI_OPT`. + +So, in short, you should never load the compiler or MPI modules directly. Instead, you should always load the `jedi-` meta-modules as demonstrated above - they will provide everything you need to load and then use the JEDI software libraries. + +## Step 4: Build JEDI Stack + +Now all that remains is to build the stack: +``` +./build_stack.sh [] +``` +Here `` is the same as in Step 3, namely a reference to the corresponding configuration file in the `config` directory. As in Step 2, if this argument is omitted, the default is to use `config/config_custom.sh`. + +For building on Mac OSX, use: +~~~~~~~~ +./build_stack.sh mac +~~~~~~~~ diff --git a/doc/Platforms.md b/doc/Platforms.md index d0daac3..6ab8cbb 100644 --- a/doc/Platforms.md +++ b/doc/Platforms.md @@ -2,18 +2,21 @@ Here are are tips for building the jedi-stack on particular Platforms -## Setting up python for Mac OSX -It is recommended for now to skip the automatic build of the pyjedi package. This has been shut off by default in the mac configuration file. It is also recommended to use miniconda for python2 and python3. +## Mac OSX -For miniconda, get the downloads on the site: https://docs.conda.io/en/latest/miniconda.html. Select the 64-bit bash installer for both python 2.7 and 3.7. These each download a script to install miniconda on your Mac. Run each script as: +The procedure for setting up a minimial JEDI stack on Mac OSX is described in [a separate how-to document in jedi-docs](https://github.com/JCSDA-internal/jedi-docs/blob/develop/howto/macos/minimum.md). + +#### Setting up python for Mac OSX +It is recommended for now to skip the automatic build of the pyjedi package. This has been shut off by default in the mac configuration file. It is also recommended to use miniconda for python3. + +For miniconda, get the downloads on the site: https://docs.conda.io/en/latest/miniconda.html. Select the 64-bit bash installer for 3.7. These each download a script to install miniconda on your Mac. Run each script as: ~~~~~~~ -sh Miniconda2-latest-MacOSX-x86_64.sh sh Miniconda3-latest-MacOSX-x86_64.sh ~~~~~~~ -When prompted allow the install to go into your home directory, and allow the script to modify your .bash_profile file. Edit your .bash_profile file and make sure that your PATH is being set the way you want it. Keep in mind that for now the ODB API python interface only works with python 2.7 (so you should make sure that "python" will be found in your miniconda2 area). +When prompted allow the install to go into your home directory, and allow the script to modify your .bash_profile file. Edit your .bash_profile file and make sure that your PATH is being set the way you want it. -Once you have miniconda2 and 3 installed, run the conda command to install extra python packages you will need for JEDI. For both miniconda2 and 3, run: +Once you have miniconda3 installed, run the conda command to install extra python packages you will need for JEDI. Now run: ~~~~~~~ conda install setuptools conda install wheel @@ -28,10 +31,9 @@ conda install pyyaml conda install sphinx ~~~~~~~ -Then, build the ncepbufr python packages. Again for both miniconda2 and 3, run: +Then, build the ncepbufr python packages. Run: ~~~~~~~ -git clone https://github.com/JCSDA/py-ncepbufr.git # Only need to do this once. The build/install processes for both - # python2 and 3 can be run from the same clone of py-ncepbufr. +git clone https://github.com/JCSDA-internal/py-ncepbufr.git # Only need to do this once. cd py-ncepbufr python setup.py build @@ -57,9 +59,9 @@ To setup environment, choose a modules home directory. This can be anywhere, bu ~~~~~~~~~ $ export INTEL_ROOT= $ export JEDI_OPT=$HOME/opt/modules -$ buildscipts/setup_environment.sh gentoo -$ buildscipts/setup_modules.sh gentoo -$ buildscipts/build_stack.sh gentoo +$ buildscripts/setup_environment.sh gentoo +$ buildscripts/setup_modules.sh gentoo +$ buildscripts/build_stack.sh gentoo ~~~~~~~~~ The enthronement setup with `setup_environment.sh gentoo` generates a `$HOME/.jedi-stack-bashrc` with all the environment diff --git a/modulefiles/apps/jedi/clang-mpich.lua b/modulefiles/apps/jedi/clang-mpich.lua index e5c4cd8..233c3f3 100644 --- a/modulefiles/apps/jedi/clang-mpich.lua +++ b/modulefiles/apps/jedi/clang-mpich.lua @@ -9,8 +9,8 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) load("jedi-clang") -load("szip/2.1.1") -load("jedi-mpich/3.3.2") +load("szip") +load("jedi-mpich") load("hdf5") load("pnetcdf") @@ -19,13 +19,14 @@ load("netcdf") load("lapack") load("boost-headers") load("eigen") -load("bufrlib") +load("bufr") load("json") load("json-schema-validator") load("ecbuild") load("nccmp") +load("gsl_lite") setenv("CC","mpicc") setenv("FC","mpifort") diff --git a/modulefiles/apps/jedi/clang-openmpi.lua b/modulefiles/apps/jedi/clang-openmpi.lua index d9a85da..23f2229 100644 --- a/modulefiles/apps/jedi/clang-openmpi.lua +++ b/modulefiles/apps/jedi/clang-openmpi.lua @@ -9,8 +9,8 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) load("jedi-clang") -load("szip/2.1.1") -load("jedi-openmpi/4.0.3") +load("szip") +load("jedi-openmpi") load("hdf5") load("pnetcdf") @@ -19,13 +19,14 @@ load("netcdf") load("lapack") load("boost-headers") load("eigen") -load("bufrlib") +load("bufr") load("json") load("json-schema-validator") load("ecbuild") load("nccmp") +load("gsl_lite") setenv("CC","mpicc") setenv("FC","mpifort") diff --git a/modulefiles/apps/jedi/gnu-7.3.0_openmpi-3.1.2_1.0.0.lua b/modulefiles/apps/jedi/gnu-7.3.0_openmpi-3.1.2_1.0.0.lua deleted file mode 100644 index cc35e83..0000000 --- a/modulefiles/apps/jedi/gnu-7.3.0_openmpi-3.1.2_1.0.0.lua +++ /dev/null @@ -1,31 +0,0 @@ -help([[ -Load environment for running JEDI applications with GNU compilers and OpenMPI. -]]) - -local pkgName = myModuleName() -local pkgVersion = myModuleVersion() -local pkgNameVer = myModuleFullName() - -conflict(pkgName) - -load("jedi-gnu/7.3.0") -load("szip/2.1.1") -load("jedi-openmpi/3.1.2") - -load("hdf5/1.10.3") -load("pnetcdf/1.11.1") -load("netcdf/4.6.3") - -load("lapack/3.7.0") -load("boost-headers/1.68.0") -load("eigen/3.3.5") -load("json/3.9.1") -load("json-schema-validator/2.1.0") - -load("eckit/1.1.0") -load("fckit/jcsda-develop") - -whatis("Name: ".. pkgName) -whatis("Version: ".. pkgVersion) -whatis("Category: Application") -whatis("Description: JEDI Environment with OpenMPI") diff --git a/modulefiles/apps/jedi/gnu-7.3.0_mpich-3.2.1_1.0.0.lua b/modulefiles/apps/jedi/gnu-mpich.lua similarity index 56% rename from modulefiles/apps/jedi/gnu-7.3.0_mpich-3.2.1_1.0.0.lua rename to modulefiles/apps/jedi/gnu-mpich.lua index 02172cb..b7088c6 100644 --- a/modulefiles/apps/jedi/gnu-7.3.0_mpich-3.2.1_1.0.0.lua +++ b/modulefiles/apps/jedi/gnu-mpich.lua @@ -8,22 +8,23 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) -load("gnu/7.3.0") -load("szip/2.1.1") -load("mpich/3.2.1") - -load("eigen/3.3.5") -load("boost/1_68_0") -load("json/3.9.1") -load("json-schema-validator/2.1.0") - -load("ecbuild/2.9.3") - -load("eckit/1.1.0") -load("fckit/jcsda-develop") - -load("hdf5/1.10.3") -load("netcdf/4.6.3") +load("jedi-gnu") +load("szip") +load("jedi-mpich") + +load("lapack") +load("eigen") +load("boost") +load("json") +load("json-schema-validator") + +load("ecbuild") +load("eckit") + +load("hdf5") +load("netcdf") +load("nccmp") +load("gsl_lite") whatis("Name: ".. pkgName) whatis("Version: ".. pkgVersion) diff --git a/modulefiles/apps/jedi/gnu-openmpi.lua b/modulefiles/apps/jedi/gnu-openmpi.lua index 4a42610..484597f 100644 --- a/modulefiles/apps/jedi/gnu-openmpi.lua +++ b/modulefiles/apps/jedi/gnu-openmpi.lua @@ -8,28 +8,32 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) -load("jedi-gnu/7.4.0") -load("szip/2.1.1") -load("jedi-openmpi/3.1.2") - -load("hdf5/1.10.5") -load("pnetcdf/1.11.2") -load("netcdf/4.7.0") - -load("lapack/3.7.0") -load("boost-headers/1.68.0") -load("eigen/3.3.5") -load("bufrlib/11.3.2") -load("json/3.9.1") -load("json-schema-validator/2.1.0") - -load("ecbuild/jcsda-release-stable") -load("eckit/1.1.0") -load("fckit/jcsda-develop") - -setenv("CC","mpicc") -setenv("FC","mpifort") -setenv("CXX","mpicxx") +load("cmake") +load("jedi-gnu") +load("szip") +load("zlib") +load("jedi-openmpi") + +load("udunits") +load("lapack") + +load("hdf5") +load("pnetcdf") +load("netcdf") +load("nccmp") + +load("boost-headers") +load("eigen") +load("json") +load("json-schema-validator") +load("gsl_lite") +load("pybind11") +load("bufr") + +load("ecbuild") +load("eckit") +load("fckit") +load("atlas") whatis("Name: ".. pkgName) whatis("Version: ".. pkgVersion) diff --git a/modulefiles/apps/jedi/intel-impi.lua b/modulefiles/apps/jedi/intel-impi.lua index c48b6b7..9288d78 100644 --- a/modulefiles/apps/jedi/intel-impi.lua +++ b/modulefiles/apps/jedi/intel-impi.lua @@ -1,5 +1,5 @@ help([[ -Load environment for running JEDI applications with GNU compilers and OpenMPI. +Load environment for running JEDI applications with Intel OneAPI ]]) local pkgName = myModuleName() @@ -8,30 +8,33 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) -load("jedi-intel/17.0.1") -try_load("szip/2.1.1") -load("jedi-impi/17.0.1") +load("cmake") +load("jedi-intel") +load("szip") +load("zlib") +load("jedi-impi") -load("hdf5/1.10.5") -load("pnetcdf/1.11.2") -load("netcdf/4.7.0") +load("udunits") -load("lapack/3.7.0") -load("boost-headers/1.68.0") -load("eigen/3.3.5") -load("bufrlib/11.3.2") -load("json/3.9.1") -load("json-schema-validator/2.1.0") +load("hdf5") +load("pnetcdf") +load("netcdf") +load("nccmp") -load("ecbuild/jcsda-release-stable") -load("eckit/1.1.0") -load("fckit/jcsda-develop") +load("boost-headers") +load("eigen") +load("json") +load("json-schema-validator") +load("gsl_lite") +load("pybind11") +load("bufr") -setenv("CC","mpiicc") -setenv("FC","mpiifort") -setenv("CXX","mpiicpc") +load("ecbuild") +load("eckit") +load("fckit") +load("atlas") whatis("Name: ".. pkgName) whatis("Version: ".. pkgVersion) whatis("Category: Application") -whatis("Description: JEDI Environment with Intel17") +whatis("Description: JEDI Environment with Intel OneAPI") diff --git a/modulefiles/compiler/compilerName/compilerVersion/bufrlib/bufrlib.lua b/modulefiles/compiler/compilerName/compilerVersion/bufr/bufr.lua similarity index 82% rename from modulefiles/compiler/compilerName/compilerVersion/bufrlib/bufrlib.lua rename to modulefiles/compiler/compilerName/compilerVersion/bufr/bufr.lua index 6aea0c6..1d29df0 100644 --- a/modulefiles/compiler/compilerName/compilerVersion/bufrlib/bufrlib.lua +++ b/modulefiles/compiler/compilerName/compilerVersion/bufr/bufr.lua @@ -8,15 +8,19 @@ local hierA = hierarchyA(pkgNameVer,1) local compNameVer = hierA[1] local compNameVerD = compNameVer:gsub("/","-") +conflict("bufrlib") conflict(pkgName) local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" local base = pathJoin(opt,compNameVerD,pkgName,pkgVersion) +prepend_path("PATH", pathJoin(base,"bin")) +prepend_path("CPATH", pathJoin(base,"include")) prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib")) prepend_path("DYLD_LIBRARY_PATH", pathJoin(base,"lib")) prepend_path("LIBRARY_PATH", pathJoin(base,"lib")) +prepend_path("PYTHONPATH", pathJoin(base,"lib/python@PYTHON_VERSION@/site-packages")) setenv("bufr_VERSION", pkgVersion) setenv("bufr_ROOT", base) -- CMake find_package(bufr) diff --git a/modulefiles/compiler/compilerName/compilerVersion/ecflow/ecflow.lua b/modulefiles/compiler/compilerName/compilerVersion/ecflow/ecflow.lua new file mode 100644 index 0000000..6a887da --- /dev/null +++ b/modulefiles/compiler/compilerName/compilerVersion/ecflow/ecflow.lua @@ -0,0 +1,37 @@ +help([[ +]]) + +local pkgName = myModuleName() +local pkgVersion = myModuleVersion() +local pkgNameVer = myModuleFullName() + +local hierA = hierarchyA(pkgNameVer,1) +local compNameVer = hierA[1] +local compNameVerD = compNameVer:gsub("/","-") + +conflict(pkgName) + +try_load("qt") + +local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" + +local base = pathJoin(opt,compNameVerD,pkgName,pkgVersion) + +prepend_path("PATH", pathJoin(base,"bin")) +prepend_path("LD_LIBRARY_PATH", pathJoin(base,"lib")) +prepend_path("DYLD_LIBRARY_PATH", pathJoin(base,"lib")) +prepend_path("CPATH", pathJoin(base,"include")) +prepend_path("MANPATH", pathJoin(base,"share","man")) + +prepend_path("PYTHONPATH", pathJoin(base,"lib/python@PYTHON_VERSION@/site-packages/ecflow")) + +setenv("ECFLOW_ROOT", base) +setenv("ECFLOW_INCLUDES", pathJoin(base,"include")) +setenv("ECFLOW_LIBRARIES", pathJoin(base,"lib")) +setenv("ECFLOW_VERSION", pkgVersion) +setenv("ECFLOW_PYTHON", pathJoin(base,"lib/python@PYTHON_VERSION@/site-packages/ecflow")) + +whatis("Name: ".. pkgName) +whatis("Version: " .. pkgVersion) +whatis("Category: application") +whatis("Description: ecFlow Workflow Manager") diff --git a/modulefiles/core/cgal/cgal.lua b/modulefiles/core/cgal/cgal.lua index 6c0911f..6f8fcb8 100644 --- a/modulefiles/core/cgal/cgal.lua +++ b/modulefiles/core/cgal/cgal.lua @@ -7,9 +7,10 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) -prereq("boost-headers") +try_load("boost-headers") +try_load("eigen") -local opt = os.getenv("OPT") or "/opt/modules" +local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" local base = pathJoin(opt,"core",pkgName,pkgVersion) @@ -17,8 +18,6 @@ prepend_path("CPATH", pathJoin(base,"include")) prepend_path("MANPATH", pathJoin(base,"share","man")) setenv("CGAL_ROOT", base) -setenv("CGAL_PATH", base) -setenv("CGAL_DIR", base) setenv("CGAL_INCLUDE_DIRS", pathJoin(base,"include")) setenv("CGAL_LIBRARIES", pathJoin(base,"lib")) setenv("CGAL_VERSION", pkgVersion) diff --git a/modulefiles/core/eigen/eigen.lua b/modulefiles/core/eigen/eigen.lua index 715d1db..386909a 100644 --- a/modulefiles/core/eigen/eigen.lua +++ b/modulefiles/core/eigen/eigen.lua @@ -7,7 +7,7 @@ local pkgNameVer = myModuleFullName() conflict(pkgName) -try_load("boost_headers") +try_load("boost-headers") local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" diff --git a/modulefiles/core/gsl_lite/gsl_lite.lua b/modulefiles/core/gsl_lite/gsl_lite.lua index 9c395f8..8b8cfd4 100644 --- a/modulefiles/core/gsl_lite/gsl_lite.lua +++ b/modulefiles/core/gsl_lite/gsl_lite.lua @@ -11,9 +11,10 @@ local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" local base = pathJoin(opt,"core",pkgName,pkgVersion) -prepend_path("PATH", pathJoin(base,"Contents")) +prepend_path("CPATH", pathJoin(base,"include")) setenv("gsl_lite_ROOT", base) +setenv("gsl_lite_DIR", pathJoin(base,"lib","cmake","gsl-lite")) whatis("Name: ".. pkgName) whatis("Version: " .. pkgVersion) diff --git a/modulefiles/core/json/json.lua b/modulefiles/core/json/json.lua index 120b20f..9d65474 100644 --- a/modulefiles/core/json/json.lua +++ b/modulefiles/core/json/json.lua @@ -11,7 +11,8 @@ local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" local base = pathJoin(opt,"core",pkgName,pkgVersion) -setenv("nlohmann_json_ROOT", base) +setenv("JSON_ROOT", base) +setenv("JSON_DIR", pathJoin(base,"lib","cmake","nlohmann_json")) whatis("Name: ".. pkgName) whatis("Version: " .. pkgVersion) diff --git a/modulefiles/core/pybind11/pybind11.lua b/modulefiles/core/pybind11/pybind11.lua index bd408f2..a87e2f3 100644 --- a/modulefiles/core/pybind11/pybind11.lua +++ b/modulefiles/core/pybind11/pybind11.lua @@ -11,9 +11,10 @@ local opt = os.getenv("JEDI_OPT") or os.getenv("OPT") or "/opt/modules" local base = pathJoin(opt,"core",pkgName,pkgVersion) -prepend_path("PATH", pathJoin(base,"Contents")) +prepend_path("CPATH", pathJoin(base,"include")) -setenv("pybind11_DIR", base) +setenv("pybind11_ROOT", base) +setenv("pybind11_DIR", base,"share","cmake","pybind11") whatis("Name: ".. pkgName) whatis("Version: " .. pkgVersion)