Create a SEMSDevEnv.cmake file to automatically use loaded SEMS dev env

[bartlettroscoe]

Next Action Status:

Working and providing value as part of new checkin-test-sems.sh script and CI build (see #482)

CC list: @rppawlo, @bathmatt, @jgfouca, @jwillenbring, @gdsjaar, @trilinos/framework

Blocking: #410, #370

Description:

This story will be to create a SEMSDevEnv.cmake file that once included (with -DTrilinos_CONFIGURE_OPTIONS_FILES=/SEMSDevEnv.cmake), then TriBITS will automatically pick up the right compilers and TPL locations.

In addition, it would be desirable for the Trilinos configure to automatically pick up the loaded SEAMS env (like is done on the ATTB machines using the ATTB_ENV env var, see #172).

In addition to just providing the SEMSDevEnv.cmake module, this story will also scope out what might be useful for a standard Trilinos dev env. However, a new story will be created to refine what a new expanded Trilinos Primary Tested build of TPLs and Packages will look like.

List of TPLs and other requirements needed for a standard Trilinos CI build:

This following list is the current consensus for this the standard Trilinos CI build (this list is updated as consensus changes).

• CMake 3.5.2 (There are huge speed advantages in using CMake 3.5.2 vs. 2.8.11) (see)
• GCC 4.8.4 (@crtrott, see)
• OpenMPI 1.8.7 (@crtrott, see)
• Python 2.7.9 (the only version of 2.x, x > 6 provided by SEMS dev env)
• OpenMP (i.e. Pthreads) (@crtrott, see)
• BLAS and LAPACK, provided by system? (see)
• Boost v1.55.0? (see)
• Zlib 1.2.8 (the version provided by SEMS)
• HDF5 v1.8.12 (the version provided by SEMS, used by SEMS NetCDF 4.3.2)
• NetCDF 4.3.2? (the version provided by SEMS)
• ParMETIS v4.0.3 built with 32-bit index types (@kddevin, see)
• SuperLU 4.3 (used by several Trilinos packages)
• Scotch v6.0.3 built with 32-bit index types (@kddevin, see)

Tasks:

1. Get opinions on what TPLs should be included in the SEMSDevEnv.cmake module (targeting a standard Trilinos pre-push CI build) [Done]
2. Write out SEMSDevEnv.cmake to automatically set up Compilers TPLs for a given loaded SEMS env (using module load <avail>) by reading env vars starting with SEMS_. (see below) [Done]
3. Create a load_sems_dev_env.sh module that can be called as source load_sems_dev_env.sh [<compiler-and-version>] [<openmpi-and-version>], for example source load_sems_dev_env.sh gcc/4.9.2 openmpi/1.10.1 (where <compiler-and-version> and <openmpi-and-version> are given defaults if not provided). (see below) [Done]
4. Test out on full shared lib MPI and serial builds of Trilinos with all Secondary Tested packages enabled that can be given this set of TPLs. (but forgot to enable Scott and ParMETIS TPLs, see below) [Done]
5. Automatically load SEMSDevEnv.cmake if it is detected that the SEMS dev env is loaded. (see below) [Done]
6. Enable Scotch and ParMETIS TPLs in shared lib build and test with packages that support them ... Zoltan and Zoltan2 tests fail as expected (see below and Zoltan: Several tests fail with 64 bit builds of of Scotch and ParMETIS #475 and Zoltan2: Several tests fail with 64 bit builds of of Scotch and ParMETIS #476) [Done]
7. Test and get working all-static builds (i.e. build static libs and use static TPL libs) ... see commit c334dd6 [Done]
8. Investigate issues with Scotch and ParMETIS ... SEMS provides inconsistent 32-bit Scotch and 64-bit ParMETIS (see below) [Done]
9. Try to enable SuperLU 4.3 (see below) [Done]
10. Finish documentation on Trilinos GitHub Wiki (see below) [Done]
11. Have documentation and implementation reviewed (see a, b, c, d, e, and f) ... Will fix any problems if they come up. [Done]

[jwillenbring]

@dmvigi should be CC'ed on this too.

[bathmatt]

@bartlettroscoe Do you have a prelim file we can test on shiller?

[bartlettroscoe]

@bathmatt, what is a "prelim file"?

[nmhamster]

Preliminary file he can test I think.

[bartlettroscoe]

From: Bartlett, Roscoe A
Sent: Monday, April 04, 2016 7:11 PM
To: Trilinos Framework
Subject: Access to machine with SEMS Dev Env mounted?

Hello Trilinos Framework,

Is there some machine that has the SEAMS Dev Env mounted that is not being constantly hammered where I could get an account? I have some changes that I need to test that might impact a lot of Secondary Tested Trilinos packages such as:

 https://github.com/trilinos/Trilinos/pull/265
https://github.com/TriBITSPub/TriBITS/issues/56 

I can’t effectively test all of this and push unless I have most of the important TPLs built for packages like STK and SEACAS.

I would not hammer this machine. I could use very few processes. I would only use it for the final configure/build/test/push (unless something breaks before the push then I will need to fix it).

This is also a chance for me to put together a standard Trilinos configure for SEMS:

https://github.com/trilinos/Trilinos/issues/158

This could lead to a standard checkin-test-sems.sh script for a standard CI build for Trilinos.

Cheers,

-Ross

[jgfouca]

@bartlettroscoe , it's quite easy to mount the sems TPLs, so any COE RHEL machine on the SON or SRN should work.

[bartlettroscoe]

it's quite easy to mount the sems TPLs, so any COE RHEL machine on the SON or SRN should work.

Don't you need sudo to mount an NFS drive? I don't have sudo on any SNL machine. The only machines I have accounts on are the CEE server ceesrv02 and several of the ATTB machines. I don't think any of these mount the SEMS dev env partitions, do they? I don't have my own Linux machine yet (and it does not seem trivial to get one).

[nmhamster]

@bartlettroscoe we don't mount SEMS on ATTB for reasons we outlined relating to TPLs not working on some architectures and level of tuning/modification and support in the environment (where many things are non-standard until we have had time to work with vendors to find appropriate ways to integrate these into their platforms and optimize them). Have you checked SNL re-app for Linux machines, there are usually workstations there if you just need a builder box.

[bartlettroscoe]

we don't mount SEMS on ATTB for reasons we outlined relating to TPLs not working on some architectures and level of tuning/modification and support in the environment

Given that shepard is supposed to be a fairly generic machine, could the SEMS dev env partition be mounted there?

Have you checked SNL re-app for Linux machines, there are usually workstations there if you just need a builder box.

I have not. But I am little leery of spending a lot of time setting up an old slow workstation that could die at any minute.

[bathmatt]

There should be funds to buy a reasonable machine, they aren't that much.
I would hammer hanson/shiller honestly. If they are ovreloaded we need to
buy more of them. You can build on the backend nodes as well. That's what
our jenkins scripts do (As I understnad it)

On Tue, Apr 5, 2016 at 7:43 AM, Roscoe A. Bartlett <[email protected]

wrote:

we don't mount SEMS on ATTB for reasons we outlined relating to TPLs not
working on some architectures and level of tuning/modification and support
in the environment

Given that shepard is supposed to be a fairly generic machine, could the
SEMS dev env partition be mounted there?

Have you checked SNL re-app for Linux machines, there are usually
workstations there if you just need a builder box.

I have not. But I am little leery of spending a lot of time setting up an
old slow workstation that could die at any minute.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#158 (comment)

[jgfouca]

@bartlettroscoe , yes, you need root to mount, but you can just ask a sysadmin to do it.

[bartlettroscoe]

There should be funds to buy a reasonable machine, they aren't that much.

Yea, I guess I need to just go ahead and pull the trigger on getting a new COE RedHand linux machine just for myself to do this type of stuff on. I have already been given the green light to do so. I was just trying to see if it was possible to create a productive dev env at SNL without purchasing my own Linux box (or purchasing a very expensive CEE machine).

I would hammer hanson/shiller honestly.

Right. For development work related to advanced architectures, we need to use the ATTB machines. That is the purpose of #172.

But the issue is that we desperately need a standard CI env for Trilinos. We can't use the ATTB machines for that. The simplest way to build a uniform CI env for Trilinos is based of the SEMS dev env (see).

[bartlettroscoe]

It looks like I have access to the machine muir (thanks for pointing that out Brent!) that NFS mounts the SEMS dev env under under /projects/. It looks like it is already set up when you login to show modules under /projects/modulefiles/ shown with module avail. Now I just need to figure out which modules should be loaded to effectively test Trilinos. It looks like there are different builds with GCC (4.4.7, 4.7.2, 4.8.4, 4.9.2, 5.1.0) and Intel (14.0.4, 15.0.2, 16.0.1) and OpenMPI (1.6.5, 1.8.7, 1.10.1) for different versions of the TPLs for many permutations of compilers and OpenMPI for the (Trilinos) TPLs boost, hdf5, netcdf, parmetis, scotch, qd, superlu, and zlib. However, it looks like several TPLs rquired to build all of STK and SEACAS are not there. For example, if I configure Trilinos (on any machine) with:

$ cmake -DTPL_ENABLE_MPI=ON \
   -DTrilinos_ENABLE_STK=ON \
   -DTrilinos_ENABLE_SEACAS=ON \
  ../../../Trilinos

You see the enabled TPLs:

• Final set of enabled TPLs: Pthread MPI BLAS LAPACK Boost Netcdf BoostLib Matio X11 9

It looks like the X11 and Matio TPL is required by SEACAS:

-- Setting TPL_ENABLE_X11=ON because it is required by the enabled package SEACASSVDI
-- Setting TPL_ENABLE_Matio=ON because it is required by the enabled package SEACASExo2mat

Are these TPLs really important for testing Trilinos functionality downstream from SEACAS?

There are a bunch of optional TPLs that are not enabled but yet I know some people feel are important. If the remaining 88 TPLs:

• Final set of non-enabled TPLs: MKL yaml-cpp Peano CUDA CUSPARSE Thrust Cusp TBB HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd METIS ParMETIS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF Zlib HDF5 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Pnetcdf Clp GLPK qpOASES PAPI MATLABLib Eigen Lemon GLM quadmath CAMAL RTlib DLlib AmgX CGAL VTune TASMANIAN ArrayFireCPU SimMesh SimModel 88

how many should really be present and enabled for a solid pre-push test of Trilinos.

I know that SuperLUDist, SuperLU, HYPRE and PETSC are important for the IDEAS Productivity project. What about the other sparse-direct TPLs like UMFPACK and CSparse? I, for one, would really like the TPL BinUtils to be present. That is used for creating backtraces with exception handling.

Can we get a solid pre-push test of Trilinos without all of these other TPLs?

I will ask the Trilinos developers. I should also ask Trilinos customers what TPLs they enable with Trilinos.

[bartlettroscoe]

Note that BLAS and LAPACK are not listed in the SEMS modules. From looking at the automated builds on muri on CDash at:

http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2408598

It would seem these are found in the base Linux COE packages:

-- TPL_BLAS_LIBRARIES='/usr/lib64/libblas.so'
-- TPL_LAPACK_LIBRARIES='/usr/lib64/liblapack.so'

[crtrott]

Just to clarify, this doesn't mean you want to do only single compiler testing for the nightlies right?

That said for standard push right now:
gcc 4.8.4, openmpi 1.8.7

Nightlies need additionaly
gcc 4.7.2, 4.9.2, 5.1.0, intel 15, 16 and openmpi 1.10

openmpi 1.6 can be retired in my opinion, that is pretty old by now.
On the TPL front minimum:
boost, hdf5, netcdf, and zlib

The superlu version provided by SEMS is broken last I checked, and nobody cared enough to fix it since one can simply not use it in Trilinos.

[nmhamster]

@bartlettroscoe the real aim of this work shouldn't be to do this for SEMS environments per-se, it should be to ensure we have a high quality, well tested framework for execution on capacity- and capability- class production computing environments. Note that on these production machines SEMS won't be the environment because, instead, it will be supplied by vendors who have to optimize and support it (this particularly true for machines like Cray and IBM who very carefully patch and engineering an environment for their machines). The real value of SEMS is that it brings this concept down to the workstation environment for our developers and includes local support. To that end, these tests should be oriented towards configurations that will be on our production environments where we can do that/replicate the combinations to be close enough. This means running multiple of these combinations (unfortunately) to replicate the production computing systems. As a rough wag to get get started, we would need an OpenMPI 1.10 series test with GCC 4.8 and 4.9 for POWER systems (perhaps including CUDA) and then an MPICH-based environment with GCC 4.9/Intel 16.X for a machine like Trinity (note Intel relies on GCC for header files so we must be careful to select this appropriately). The SIERRA folks will also need to run on older machines with Intel 15.X and older GCC (I would suggest 4.7 for this purpose).

[bartlettroscoe]

Just to clarify, this doesn't mean you want to do only single compiler testing for the nightlies right?

No, this is just the standard pre-push CI build. This will not affect what gets tested post-push. And we want to make the pre-push CI build fast and we want to focus on what best protects other developers doing their work. So we could likely get away with just a single MPI build of Trilinos with ETI turned on, no complex or float, etc. We want this to cover a good bit but be as fast as possible.

[crtrott]

Ok, in that case as I said GCC 4.8.4 with OpenMPI 1.8.7 and I would very, very strongly advocate for enabling OpenMP because I believe by default we must exercise the threaded code path.

[bartlettroscoe]

As a rough wag to get get started, we would need an OpenMPI 1.10 series test with GCC 4.8 and 4.9 for POWER systems (perhaps including CUDA) and then an MPICH-based environment with GCC 4.9/Intel 16.X for a machine like Trinity (note Intel relies on GCC for header files so we must be careful to select this appropriately). The SIERRA folks will also need to run on older machines with Intel 15.X and older GCC (I would suggest 4.7 for this purpose).

We are not looking for a comprehensive set of builds. The post-push builds and the usage of the 'develop'/'master' branch workflow will take care of protecting these customers. What we are going for is the best single (relatively fast) pre-push CI build that we can put together.

[kddevin]

@bartlettroscoe wrote:
3) What TPLs should be enabled? (The SEMS Dev Env assumes blas and lapack are already on the system and provides boost, hdf5, netcdf, parmetis, scotch, qd, superlu, and zlib)
4) Of the TPLs that are enabled, what versions of TPLs should considered standard?

ParMETIS v4.0.3 or later, built with 32-bit index types
Scotch v6.0.3 or later, built with 32-bit index types

[bartlettroscoe]

I got a message back for SEMSHELPD-130 that SEMS is working on a sync script for the SEMS Dev Env. I am excited to try that out to see if that fixes the performance problems that I am seeing.

[tjfulle]

@bartlettroscoe, working remotely from SNL/NM, the mounted SEMS environment is unusably slow. rsyncing the mounted SEMS /projects directory was unfeasible (timeouts, crashes, etc.). I ended up writing a python script that crawls the mounted SEMS /projects directory and rsyncs smaller chunks one at a time until the entire directory is synced locally (and syncs multiple directories concurrently using Python's multiprocessing module).

[jgfouca]

@tjfulle SEMS has developed a tool that should allow you to use your TPLs locally while syncing the minimal amount of data. The tool is in the final stages of testing and should be available soon.

[tjfulle]

I look forward to using it @jgfouca !

[bartlettroscoe]

SEMS now supports a script to sync the SEMS env. I documented how I installed it (and what I had to tweak from the SEMS-provided documentation to get it to work) in:

• https://sems.sandia.gov/jira/browse/TRIL-102

In this Issue comment, I document the change in performance of the SEMS-built and now locally installed tools vs. a locally built set of executables.

My timing experiments (shown below) still shows about a factor of 2x slowdown the locally synced SEMS env tools vs. locally built tools (built from the CASL VERA dev env build scripts). This might be something that SEMS might want to look into at some point. A factor of 2x slowdown is still pretty large.

Based on this analysis, I think that I would use my locally built env for all local development and only use the SEMS env when pushing with the checkin-test-sems.sh script (see #482).

---

Detailed Notes:

A) Locally built and installed VERA Dev Env:

Using the locally built VERA Dev Env (that supplies CMake, GCC, etc.) I configured TriBITS from scratch and then built and ran tests:

$ which gcc
/home/vera_env/gcc-4.8.3/toolset/gcc-4.8.3/bin/gcc

$ which cmake
/home/vera_env/common_tools/cmake-3.6.2/bin/cmake

$ rm -r CMake*

$ time ./do-configure &> configure.out 

real    0m5.796s
user    0m3.443s
sys     0m1.167s

$ make 

[ 50%] Linking CXX executable TriBITS_GenericDriver.exe
[100%] Built target TriBITS_GenericDriver

$ time ctest -j32 &> ctest.out
real    0m16.290s
user    2m5.361s
sys     0m49.507s

$ tail ctest.out 
220/222 Test  #42: TriBITS_CTestScripts_TAAT_will_fail_return_code ...................................................................   Passed    0.21 sec
221/222 Test #186: TriBITS_CheckinTest_UnitTests .....................................................................................   Passed   15.75 sec
222/222 Test #152: TriBITS_TribitsExampleProject_WrapExternal ........................................................................   Passed   16.22 sec

100% tests passed, 0 tests failed out of 222

Label Time Summary:
TriBITS    = 416.89 sec (222 tests)

Total Test time (real) =  16.26 sec

B) Remotely built by locally synced SEMS env:

Now with the remotely built but locally synced SEMS env:

$ . ~/Trilinos.base/Trilinos/cmake/load_sems_dev_env.sh 

$ module list
Currently Loaded Modulefiles:
  1) sems-env                       4) sems-gcc/4.8.4                 7) sems-zlib/1.2.8/base          10) sems-parmetis/4.0.3/parallel
  2) sems-python/2.7.9              5) sems-openmpi/1.8.7             8) sems-hdf5/1.8.12/parallel     11) sems-scotch/6.0.3/parallel
  3) sems-cmake/3.5.2               6) sems-boost/1.55.0/base         9) sems-netcdf/4.3.2/parallel    12) sems-superlu/4.3/base

$ which gcc
/projects/sems/install/rhel6-x86_64/sems/compiler/gcc/4.8.4/base/bin/gcc

$ which cmake
/projects/sems/install/rhel6-x86_64/sems/utility/cmake/3.5.2/bin/cmake

$ time ./do-configure &> configure.out

real    0m9.461s
user    0m4.104s
sys     0m1.915s

$ make 

Scanning dependencies of target TriBITS_GenericDriver
[ 50%] Building CXX object test/core/CTestScriptsUnitTests/CMakeFiles/TriBITS_GenericDriver.dir/GenericDriver.cpp.o
[100%] Linking CXX executable TriBITS_GenericDriver.exe
[100%] Built target TriBITS_GenericDriver

$ time ctest -j32 &> ctest.out 

real    0m29.885s
user    2m23.112s
sys     5m4.017s

$ tail ctest.out 
220/222 Test #166: TriBITS_TribitsExampleProject_SimpleTpl_RPATH_CMAKE_INSTALL_RPATH .................................................   Passed    3.01 sec
221/222 Test #152: TriBITS_TribitsExampleProject_WrapExternal ........................................................................   Passed   23.10 sec
222/222 Test #186: TriBITS_CheckinTest_UnitTests .....................................................................................   Passed   29.73 sec

100% tests passed, 0 tests failed out of 222

Label Time Summary:
TriBITS    = 630.97 sec (222 tests)

Total Test time (real) =  29.76 sec

C) Analysis:

This shows a much improved speed-up of the locally synced SEMS env over the NFS-mounted env reported above. However, we are still seeing about a factor of 2x slowdown of the locally synced SEMS env vs. a locally built env.

Why is this? Is it because GCC is built to not run fast (as was suggested by someone in the past)? Is it because CMake is built with -O0 instead of -O3 (we got bit by that in CASL for over a year)? Is it because the the locally built tools take advantage of the processor arch better? Why is this?

[jgfouca]

@bartlettroscoe the cmake configure slowdown possibly has something to do with the version of cmake (3.6.2 vs 3.5.2). Every SEMS TPL/utility should have been built with -03, assuming the package doesn't ignore the CFLAGS CPPFLAGS etc env variables.

As far as the reduced performance of the built code, that is both a complete surprise and disappointing. The fact that GCC was built on a certain architecture should not prevent it from taking advantage of the current architecture, should it? One possibility is that the stdlib is not tuned to the current architecture.

One thing that would be helpful is if you can duplicate this slowdown with a different compiler, like intel.

[bartlettroscoe]

@jgfouca,

the cmake configure slowdown possibly has something to do with the version of cmake (3.6.2 vs 3.5.2).

Okay, I could test that. I could use install-cmake.py to locally build and install the exact same version of CMake as provided by SEMS and then compare. But from the contract work with Kitware over the last 1.5 years and lots of performance studies for CMake, we don't expect to see factor of nearly 2x between CMake versions after 3.3. The speedups that we have funding are more in the range of 20-50% or so.

Every SEMS TPL/utility should have been built with -03, assuming the package doesn't ignore the CFLAGS CPPFLAGS etc env variables.

That should work. That is what the install-cmake.py script does and we verified that it adds -O3 to the compile lines for the source files. It makes about a 2x difference in cmake runtimes.

As far as the reduced performance of the built code, that is both a complete surprise and disappointing.

The TriBITS test suite does not really test the performance of compiled code. It builds and runs only very tiny compiled programs. What it really tests (in a performance sense) is running cmake, gcc, g++, gfortran, etc. a bunch of times, creating a lot of little directories, etc. Someone can do real performance comparisons of the built code, but this is not that. It would likely be a good idea of someone did that at some point soon.

Thanks for all your work with the SEMS env!

[bartlettroscoe]

@bmpersc, I cleaned up the SEMSDevEnv.cmake file quite a bit. See the version of SEMSDevEnv.cmake for the commit 19afca0 which you can see the full file here. Hopefully this will address the majority of your concerns you expressed above (except for perhaps how it gets loaded).

This will get pushed soon as part of #482 with the branch better-ci-build-482.

[bartlettroscoe]

@jwillenbring and @bmpersc, do you have the time and the interest to review:

• https://github.com/trilinos/Trilinos/wiki/SEMS-Dev-Env

If not, this is committed and is being used automatically by the checkin-test-sems.sh script and the new post-push CI build.

The new SEMSDevEnv.cmake file is pretty minimal and should have addressed all of Brent's concerns. The right compilers and TPL paths are still set up automatically but no TPLs are enabled automatically. I don't see that as a bad thing.

Therefore, I would like to close this as complete if there are no objections.

[bartlettroscoe]

We are no longer going to purse extending the checkin-test-sems.sh (or checkin-test.py) script for Trilinos (see #482 (comment)) and the SEMS env is already working for this and protected by a CI sever. This is working for me and many other people so there is no further need to review this. I will fix any major problems that come up so that I and others can continue to use this.

Closing as complete.