parmetis: use github/KarypisLab/ParMETIS as source #329102

cheriimoya · 2024-07-22T08:58:38Z

Description of changes

Things done

Add a 👍 reaction to pull requests you find important.

emilazy · 2024-07-22T09:13:16Z

Perhaps we should source this from https://github.com/KarypisLab/ParMETIS instead. There is a commit corresponding to 4.0.3.

SuperSandro2000 · 2024-07-22T09:14:54Z

pkgs/by-name/pa/parmetis/package.nix

+  enableParallelBuilding = true;
+


Suggested change

enableParallelBuilding = true;

the cmake setup hook already sets this

It would be nice to include ninja in nativeBuildInputs for faster builds, though, if it works.

unsure, when building without this commit:

$ time nix-build -A parmetis nix-build -A parmetis 1.04s user 0.38s system 3% cpu 46.465 total

when building with this commit:

$ time nix-build -A parmetis nix-build -A parmetis 1.09s user 0.36s system 13% cpu 11.118 total

I think it might just be Ninja that defaults to parallel builds? I would suggest adding ninja next to cmake in nativeBuildInputs, which should be faster than even a parallel Make build.

error: builder for '/nix/store/br8kfshh9y3jgkx7gw5f2rkx3392nzwr-parmetis-4.0.3.drv' failed with exit code 1; last 10 log lines: > -- Looking for execinfo.h - found > -- Looking for getline > -- Looking for getline - found > -- checking for thread-local storage - found > -- Configuring done (3.2s) > -- Generating done (0.0s) > -- Build files have been written to: /build/parmetis-4.0.3/build/Linux-x86_64 > Running phase: buildPhase > build flags: -j12 > ninja: error: loading 'build.ninja': No such file or directory For full logs, run 'nix log /nix/store/br8kfshh9y3jgkx7gw5f2rkx3392nzwr-parmetis-4.0.3.drv'. nix-build -A parmetis 1.05s user 0.42s system 27% cpu 5.408 total

emilazy · 2024-07-22T09:35:20Z

Actually, we may want to consider switching to a fork:

cheriimoya · 2024-07-24T08:42:56Z

Okay, so I went through many hoops now and the commits I'm pushing now is what I came up with.
I tested it with a pflotran simulation and the output of the simulation didn't change after switching to this version of parmetis, which is needed for petsc, which is needed for pflotran.
Execution time changed in a negligible way.
I don't know much about C, so somebody better double check what I did 😅

Result of nixpkgs-review run on x86_64-linux 1

8 packages built:

getdp
parmetis
petsc
precice
python311Packages.pyprecice
python311Packages.pyprecice.dist
python312Packages.pyprecice
python312Packages.pyprecice.dist

emilazy · 2024-07-26T15:19:03Z

Sorry for the back and forth on this. I didn’t realize that this package is licensed under very restrictive terms. It’s not clear to me that the licence permits derivative works at all. It seems like PETSc is developed by a US government research agency, so I guess it hinges on whether “freely used” permits distributing modified versions.

Does your pflotran simulation work with the upstream GitHub commit that supposedly corresponds to 4.0.3? If so, it may be a more conservative choice to use that version instead.

cheriimoya · 2024-07-29T13:28:50Z

I'll check next monday

cheriimoya · 2024-08-06T10:00:33Z

Tested the latest commit and the simulation runs through, producing the correct results as expected. This is now using the metis package source as input, so maybe we could think about also switching the metis source.

SuperSandro2000 · 2024-08-06T11:37:29Z

pkgs/by-name/pa/parmetis/package.nix

+  '';
+
+  meta = with lib; {
+    description = "MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices";


Suggested change

description = "MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices";

description = "MPI-based parallel library that implements a variety of algorithms";

thats a bit long

I’d prefer to cut “that implements a variety of algorithms” so that some description of what the package actually does is retained…

We also have longDescription.

description = "Parallel Graph Partitioning and Fill-reducing Matrix Ordering"; longDescription = '' MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices. The algorithms implemented in ParMETIS are based on the multilevel recursive-bisection, multilevel k-way, and multi-constraint partitioning schemes '';

Thanks, I added it:)

SuperSandro2000 · 2024-08-06T11:37:38Z

pkgs/by-name/pa/parmetis/package.nix

+  };
+
+  nativeBuildInputs = [ cmake ];
+  enableParallelBuilding = true;


Suggested change

enableParallelBuilding = true;

already done by cmake

Are you sure?
If i remove the line it seems like the build takes more time:

# command used: `time nix-build . -A parmetis` # with `enableParallelBuilding = true;` nix-build . -A parmetis 1.68s user 0.34s system 8% cpu 22.698 total # vs without `enableParallelBuilding = true;` nix-build . -A parmetis 1.67s user 0.35s system 3% cpu 57.441 total

Maybe this is a result of running make in the configurePhase?

You probably want [ cmake ninja ].

I think you mentioned this before:

#329102 (comment)

Or am i using this incorrectly? I just added it to the nativeBuildInputs

Uh, sorry, I completely forgot, I was just reacting to the parallel build stuff… it should just work with CMake builds, but I guess there’s something funky going on here. I think you’d have to drop the make stuff in configurePhase and just let the CMake hook run naturally, with any required cmakeFlags corresponding to the ones set in the upstream Makefile.

I tested several different approaches: Running the tar xf during preConfigure and not touching the configurePhase, commenting out make in the configurePhase, but nothing worked. How bad would it be to just set enableParallelBuilding = true;? I can also remove the line and we can settle for only building with one thread if the option is that bad...

When setting the cmakeFlags to

cmakeFlags = [ "-DGKLIB_PATH=metis/GKlib" "-DMETIS_PATH=metis" ];

The build is running in parallel as expected, but then the build fails with

parmetis> [ 55%] Building C object libparmetis/CMakeFiles/parmetis.dir/balancemylink.c.o parmetis> [ 55%] Building C object libparmetis/CMakeFiles/parmetis.dir/akwayfm.c.o parmetis> [ 56%] Building C object libparmetis/CMakeFiles/parmetis.dir/comm.c.o parmetis> In file included from /build/source/libparmetis/balancemylink.c:14: parmetis> /build/source/libparmetis/./parmetislib.h:23:10: fatal error: ../metis/libmetis/gklib_defs.h: No such file or directory parmetis> 23 | #include "../metis/libmetis/gklib_defs.h" parmetis> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ parmetis> compilation terminated. parmetis> In file included from /build/source/libparmetis/akwayfm.c:14: parmetis> /build/source/libparmetis/./parmetislib.h:23:10: fatal error: ../metis/libmetis/gklib_defs.h: No such file or directory parmetis> 23 | #include "../metis/libmetis/gklib_defs.h" parmetis> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ parmetis> make[2]: *** [libparmetis/CMakeFiles/parmetis.dir/build.make:104: libparmetis/CMakeFiles/parmetis.dir/balancemylink.c.o] Error 1 parmetis> make[2]: *** Waiting for unfinished jobs.... parmetis> compilation terminated. parmetis> make[2]: *** [libparmetis/CMakeFiles/parmetis.dir/build.make:76: libparmetis/CMakeFiles/parmetis.dir/akwayfm.c.o] Error 1 parmetis> In file included from /build/source/libparmetis/ametis.c:15: parmetis> /build/source/libparmetis/./parmetislib.h:23:10: fatal error: ../metis/libmetis/gklib_defs.h: No such file or directory parmetis> 23 | #include "../metis/libmetis/gklib_defs.h" parmetis> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ parmetis> compilation terminated. parmetis> make[2]: *** [libparmetis/CMakeFiles/parmetis.dir/build.make:90: libparmetis/CMakeFiles/parmetis.dir/ametis.c.o] Error 1 parmetis> In file included from /build/source/libparmetis/comm.c:11: parmetis> /build/source/libparmetis/./parmetislib.h:23:10: fatal error: ../metis/libmetis/gklib_defs.h: No such file or directory parmetis> 23 | #include "../metis/libmetis/gklib_defs.h" parmetis> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ parmetis> compilation terminated. parmetis> make[2]: *** [libparmetis/CMakeFiles/parmetis.dir/build.make:118: libparmetis/CMakeFiles/parmetis.dir/comm.c.o] Error 1 parmetis> make[1]: *** [CMakeFiles/Makefile2:184: libparmetis/CMakeFiles/parmetis.dir/all] Error 2 parmetis> make: *** [Makefile:136: all] Error 2

I'd prefer to simply use the commits I provided.

It’s totally fine to set enableParallelBuilding = true;, in my opinion. But it is indeed the overridden configurePhase that is the issue here, because CMake’s includes this:

if ! [[ -v enableParallelBuilding ]]; then enableParallelBuilding=1 echo "cmake: enabled parallel building" fi

I think the issue you’re running into there might be because you’re using a relative path whereas the Makefile explicitly makes it absolute, though.

parmetis: Move to pkgs/by-name/pa/parmetis

db2396c

cheriimoya force-pushed the fix/parmetis-archive-org branch from 115ddf6 to 5b6822d Compare July 22, 2024 09:01

SuperSandro2000 changed the title ~~Fix/parmetis archive org~~ parmetis: use archive.org as source and others Jul 22, 2024

SuperSandro2000 reviewed Jul 22, 2024

View reviewed changes

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jul 22, 2024

ofborg bot requested a review from costrouc July 22, 2024 10:16

ofborg bot added 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 and removed 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Jul 22, 2024

cheriimoya force-pushed the fix/parmetis-archive-org branch from 7bd8ef5 to fbff98c Compare July 24, 2024 08:43

cheriimoya requested a review from emilazy July 24, 2024 08:44

cheriimoya changed the title ~~parmetis: use archive.org as source and others~~ parmetis: use petsc parmetis as source Jul 24, 2024

ghpzin mentioned this pull request Aug 2, 2024

Build failure: parmetis #331492

Open

cheriimoya force-pushed the fix/parmetis-archive-org branch 2 times, most recently from 0e05e78 to 8ea28bb Compare August 6, 2024 09:55

cheriimoya changed the title ~~parmetis: use petsc parmetis as source~~ parmetis: use github/KarypisLab/ParMETIS as source Aug 6, 2024

SuperSandro2000 reviewed Aug 6, 2024

View reviewed changes

cheriimoya force-pushed the fix/parmetis-archive-org branch from 8ea28bb to 8ce6a3c Compare August 6, 2024 12:32

parmetis: Use KarypisLab GitHub mirror instead of glaros source

a468bfc

cheriimoya force-pushed the fix/parmetis-archive-org branch from 8ce6a3c to a468bfc Compare August 19, 2024 14:27

emilazy approved these changes Aug 21, 2024

View reviewed changes

emilazy merged commit ac98115 into NixOS:master Aug 21, 2024
25 of 26 checks passed

cheriimoya deleted the fix/parmetis-archive-org branch August 24, 2024 07:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parmetis: use github/KarypisLab/ParMETIS as source #329102

parmetis: use github/KarypisLab/ParMETIS as source #329102

cheriimoya commented Jul 22, 2024 •

edited

Loading

emilazy commented Jul 22, 2024

SuperSandro2000 Jul 22, 2024

emilazy Jul 22, 2024

cheriimoya Jul 22, 2024

emilazy Jul 22, 2024

cheriimoya Jul 24, 2024

emilazy commented Jul 22, 2024

cheriimoya commented Jul 24, 2024

emilazy commented Jul 26, 2024

cheriimoya commented Jul 29, 2024 via email •

edited

Loading

cheriimoya commented Aug 6, 2024

SuperSandro2000 Aug 6, 2024

emilazy Aug 6, 2024

0x4A6F Aug 17, 2024

cheriimoya Aug 19, 2024

SuperSandro2000 Aug 6, 2024

cheriimoya Aug 6, 2024

cheriimoya Aug 6, 2024

emilazy Aug 6, 2024

cheriimoya Aug 6, 2024

emilazy Aug 6, 2024 •

edited

Loading

cheriimoya Aug 7, 2024

cheriimoya Aug 7, 2024

emilazy Aug 7, 2024

	description = "MPI-based parallel library that implements a variety of algorithms for partitioning unstructured graphs, meshes, and for computing fill-reducing orderings of sparse matrices";
	description = "MPI-based parallel library that implements a variety of algorithms";

parmetis: use github/KarypisLab/ParMETIS as source #329102

parmetis: use github/KarypisLab/ParMETIS as source #329102

Conversation

cheriimoya commented Jul 22, 2024 • edited Loading

Description of changes

Things done

emilazy commented Jul 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilazy commented Jul 22, 2024

cheriimoya commented Jul 24, 2024

emilazy commented Jul 26, 2024

cheriimoya commented Jul 29, 2024 via email • edited Loading

cheriimoya commented Aug 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilazy Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheriimoya commented Jul 22, 2024 •

edited

Loading

cheriimoya commented Jul 29, 2024 via email •

edited

Loading

emilazy Aug 6, 2024 •

edited

Loading