Precompile bugs? #275

mrhardman · 2024-10-08T09:34:32Z

When I try to precompile the latest master (097ee11) on a machine which is usually reliable using the manual install instructions https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, I now see this output

┌ CompositionsBase → CompositionsBaseInverseFunctionsExt
│  [pid 2781818] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   fs_event           0xcce120->0x7f36fd34cdf0
│   timer              0xe0f4f0->0x7f36fd34ce20
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
┌ LoweredCodeUtils
│  WARNING: using IR.SSAValue in module LoweredCodeUtils conflicts with an existing identifier.
│  WARNING: using IR.SlotNumber in module LoweredCodeUtils conflicts with an existing identifier.
└

Is this caused by changes related to #273?

I have checked that the branch here https://github.com/mabarnes/moment_kinetics/commits/electric_field_switch/ without the timer changes does not have the first error message (the LoweredCodeUtils error is present). In this older branch, I can still use

julia --project -O3 precompile.jl

to make moment_kinetics.so and run

mpirun -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'

successfully. However, in the latest master, with the manual install instructions I get the following error when I run the above command

--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  getting local rank failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_init failed
  --> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems.  This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):

  ompi_mpi_init: ompi_rte_init failed
  --> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[tplxint4.nat.physics.ox.ac.uk:2777267] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
ERROR: LoadError: Failed to precompile Test [8dfed614-e22c-5e08-85e1-65c5234f0b40] to "*/excalibur/moment_kinetics_test_install/.julia/compiled/v1.10/Test/jl_D9dYor".
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:2468
  [3] compilecache
    @ ./loading.jl:2340 [inlined]
  [4] (::Base.var"#968#969"{Base.PkgId})()
    @ Base ./loading.jl:1974
  [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
    @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
  [6] #mkpidlock#6
    @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
  [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
    @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
  [8] #invokelatest#2
    @ ./essentials.jl:894 [inlined]
  [9] invokelatest
    @ ./essentials.jl:889 [inlined]
 [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
    @ Base ./loading.jl:2983
 [11] maybe_cachefile_lock
    @ ./loading.jl:2980 [inlined]
 [12] _require(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:1970
 [13] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1812
 [14] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [15] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [16] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:1803
 [17] macro expansion
    @ ./loading.jl:1790 [inlined]
 [18] macro expansion
    @ ./lock.jl:267 [inlined]
 [19] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1753
 [20] #invoke_in_world#3
    @ ./essentials.jl:926 [inlined]
 [21] invoke_in_world
    @ ./essentials.jl:923 [inlined]
 [22] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1746
 [23] include(mod::Module, _path::String)
    @ Base ./Base.jl:495
 [24] include(x::String)
    @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install/moment_kinetics/test/runtests.jl:1
 [25] top-level scope
    @ ~/excalibur/moment_kinetics_test_install/moment_kinetics/test/runtests.jl:3
 [26] include(fname::String)
    @ Base.MainInclude ./client.jl:489
 [27] top-level scope
    @ none:1
in expression starting at /*/moment_kinetics/test/setup.jl:10
in expression starting at /*/moment_kinetics/test/runtests.jl:1
[tplxint4:2777215] *** Process received signal ***
[tplxint4:2777215] Signal: Segmentation fault (11)
[tplxint4:2777215] Signal code: Address not mapped (1)
[tplxint4:2777215] Failing at address: 0x30
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
[tplxint4:2777215] [ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7f324a61a090]
[tplxint4:2777215] [ 1] /usr/lib/x86_64-linux-gnu/libpmix.so.2(pmix_ptl_base_lost_connection+0x9fe)[0x7f3247ddedbe]
[tplxint4:2777215] [ 2] /usr/lib/x86_64-linux-gnu/libpmix.so.2(pmix_ptl_base_recv_handler+0x189)[0x7f3247de0069]
[tplxint4:2777215] [ 3] /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7(+0x2113f)[0x7f324a7ea13f]
[tplxint4:2777215] [ 4] /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7(event_base_loop+0x52f)[0x7f324a7ea87f]
[tplxint4:2777215] [ 5] /usr/lib/x86_64-linux-gnu/libpmix.so.2(+0x73d56)[0x7f3247da3d56]
[tplxint4:2777215] [ 6] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f324a54d609]
[tplxint4:2777215] [ 7] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f324a6f6353]
[tplxint4:2777215] *** End of error message ***
./activate_mk.sh: line 22: 2777215 Segmentation fault      (core dumped) mpirun -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'

We may need to add StatsBase to https://mabarnes.github.io/moment_kinetics/dev/manual_setup/, e.g.,

$ julia --project -O3
> using Pkg; Pkg.add("StatsBase")

Unfortunately adding StatsBase did not fix the MPI error, although it did remove the precompile warning.

@johnomotani Does this make sense, or is something else going wrong here? I am trying to make a script for @MantasAbazorius to use to setup moment kinetics on the same machine, and precompile.jl might only be making a valid moment_kinetics.so when run from command line directly (i.e., not in a bash script).

The text was updated successfully, but these errors were encountered:

mrhardman · 2024-10-08T10:31:25Z

Update: I have checked thoroughly and I can no longer successfully precompile and run mpi tests in any new install, regardless of the various choices I could make in the manual setup. Help needed!

johnomotani · 2024-10-08T12:10:29Z

I think the first warning about CompositionsBase is just a random thing - that seems to happen if something takes a 'long time' during precompilation. I've seen similar error messages that I think happened because some package was downloading something and took a while.

The LoweredCodeUtils warning I think I see similar things sometimes, and they don't seem to cause problems.

For the MPI errors - are you 100% sure that mpirun is using the same MPI as MPIPreferences.use_system_binary() found? I think use_system_binary() prints some info that might help you identify which MPI it's using. I think there are some command line arguments that will make mpirun tell you which library it's using, but I don't remember - let me know if Google doesn't tell you quickly and I can try and find them.

johnomotani · 2024-10-08T13:10:09Z

Oh, one thing that might help - if you're installing NCDatasets, don't! NCDatasets hasn't set up yet to let you link an external NetCDF library, and the Julia-provided one now links MPI (which will be the wrong MPI when you use system MPI). NCDatasets isn't required, and at least for the moment it would only work if you use the Julia-provided MPI.

johnomotani · 2024-10-08T13:10:42Z

See #254.

mrhardman · 2024-10-09T16:13:19Z

I think we need to take seriously the lines

in expression starting at /*/moment_kinetics/test/setup.jl:10
in expression starting at /*/moment_kinetics/test/runtests.jl:1

suggesting that there is an issue with the order in which the Test and MPI modules are loaded, i.e., that there is a bug with the test scripts due to the Julia test environment (not necessarily the system environment).

My new evidence is that I have tried running a simple fokker-planck simulation with run_moment_kinetics.jl and with and without the -Jmoment_kinetics.so on the same system where I have problems (using the native Julia MPI and the system MPI), and the commands

/.julia/bin/mpiexecjl --project=./ -n 2 julia --project -Jmoment_kinetics.so run_moment_kinetics.jl runs/fokker-planck-relaxation.toml

and

mpirun -n 2 julia --project -Jmoment_kinetics.so run_moment_kinetics.jl runs/fokker-planck-relaxation.toml

respectively, and I obtain the same result as the recorded test

moment_kinetics/moment_kinetics/test/fokker_planck_time_evolution_tests.jl

Lines 83 to 129 in 9284bd9

    
           const expected_zero =  
        
           expected_data( 
        
              [-3.0, -2.5, -2.0, -1.5, -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0], 
        
              [0.155051025721682, 0.644948974278318, 1.000000000000000, 1.500000000000000, 2.000000000000000, 2.500000000000000, 3.000000000000000], 
        
              # Expected phi: 
        
              [-1.254259088243025, -1.254259088243286], 
        
              # Expected n_ion: 
        
              [0.285287142537587, 0.285287142537513], 
        
              # Expected upar_ion: 
        
              [0.0, 0.0], 
        
              # Expected ppar_ion: 
        
              [0.182220654804438, 0.156448883483764], 
        
              # Expected pperp_ion 
        
              [0.143306715174515, 0.156192600834786], 
        
              # Expected qpar_ion 
        
              [0.0, 0.0], 
        
              # Expected v_t_ion 
        
              [1.046701532502699, 1.046701532502689], 
        
              # Expected dSdt 
        
              [0.000000000000000, -0.000000000865997], 
        
              # Expected f_ion: 
        
              [0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 ; 
        
               0.000706724195475 0.000477575434536 0.000266315395816 0.000076300638379 0.000013259062819 0.000001397494940 0.000000000000000 ; 
        
               0.006705212475828 0.004531109564814 0.002526730124657 0.000723920301085 0.000125798485463 0.000013259062819 0.000000000000000 ; 
        
               0.038585833650058 0.026074735222541 0.014540327934436 0.004165873701136 0.000723920300963 0.000076300638366 0.000000000000000 ; 
        
               0.134671678398729 0.091005636629981 0.050748427134097 0.014539667807029 0.002526615411287 0.000266303305116 0.000000000000000 ; 
        
               0.261709398369233 0.176852554997681 0.098620144126568 0.028255144359305 0.004910007858078 0.000517511020834 0.000000000000000 ; 
        
               0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 ; 
        
               0.261709398369233 0.176852554997682 0.098620144126568 0.028255144359305 0.004910007858078 0.000517511020834 0.000000000000000 ; 
        
               0.134671678398729 0.091005636629981 0.050748427134097 0.014539667807029 0.002526615411287 0.000266303305116 0.000000000000000 ; 
        
               0.038585833650058 0.026074735222541 0.014540327934436 0.004165873701136 0.000723920300963 0.000076300638366 0.000000000000000 ; 
        
               0.006705212475828 0.004531109564814 0.002526730124657 0.000723920301085 0.000125798485463 0.000013259062819 0.000000000000000 ; 
        
               0.000706724195475 0.000477575434536 0.000266315395816 0.000076300638379 0.000013259062819 0.000001397494940 0.000000000000000 ; 
        
               0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 ;;; 
        
               0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 ; 
        
               0.000473874379555 0.000190726186343 -0.000067413540342 -0.000219129923463 -0.000165609965457 -0.000069940808382 0.000000000000000 ; 
        
               0.007980077110978 0.005701898246888 0.003327438535529 0.000925955494251 -0.000037963012435 -0.000174809935222 0.000000000000000 ; 
        
               0.034010103035913 0.024541441396039 0.014741892515397 0.004975785949797 0.000869171347288 -0.000277723510529 0.000000000000000 ; 
        
               0.095320265225675 0.069160937049138 0.041953067849510 0.014711859721633 0.003235165444330 -0.000199356792174 0.000000000000000 ; 
        
               0.176443375955397 0.128736406466194 0.078839749105573 0.028150317244036 0.006528121645593 0.000119350436884 0.000000000000000 ; 
        
               0.215552595150568 0.157471466895046 0.096656199821596 0.034660213647059 0.008120924795998 0.000303073103709 0.000000000000000 ; 
        
               0.176443375955397 0.128736406466194 0.078839749105573 0.028150317244036 0.006528121645593 0.000119350436884 0.000000000000000 ; 
        
               0.095320265225675 0.069160937049138 0.041953067849510 0.014711859721633 0.003235165444330 -0.000199356792174 0.000000000000000 ; 
        
               0.034010103035913 0.024541441396039 0.014741892515397 0.004975785949797 0.000869171347288 -0.000277723510529 0.000000000000000 ; 
        
               0.007980077110978 0.005701898246888 0.003327438535529 0.000925955494251 -0.000037963012435 -0.000174809935222 0.000000000000000 ; 
        
               0.000473874379555 0.000190726186343 -0.000067413540342 -0.000219129923463 -0.000165609965457 -0.000069940808382 0.000000000000000 ; 
        
               0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000 0.000000000000000])

(so long as I correctly set density_amplitude = 0.0 instead of density_amplitude = 0.001. I have tried this with and without -O3. I think this indicates that MPI runs are working with -Jmoment_kinetics.so, but the test scripts are not.

@johnomotani Could you check if running tests with -Jmoment_kinetics.so breaks on one of your machines?

mrhardman · 2024-10-11T13:27:02Z

I have determined that when the moment kinetics package is such that

/.julia/bin/mpiexecjl --project=./ -n 2 julia --project -Jmoment_kinetics.so run_moment_kinetics.jl runs/fokker-planck-relaxation.toml

runs correctly but

/.julia/bin/mpiexecjl --project=./ -n 2 julia --project -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'

errors or hangs, the bug can be reproduced with the following very simple script

using Test: @testset, @test
using moment_kinetics
using MPI

showing that the MPI error may have something to do with changes to the code outside moment_kinetics: MPI and moment_kinetics are loaded successfully in run_moment_kinetics.jl. I have checked that porting the functionality from run_moment_kinetics.jl to setup.jl where modules are only loaded on processor 0 does not fix the issue.

mrhardman · 2024-10-11T14:08:56Z

I find that I can recover ability to use the -Jmoment_kinetics.so option with both tests via -e 'include("moment_kinetics/test/runtests.jl")' and run_moment_kinetics.jl so long as I roll back MPI to version=0.20.21 and HDF5_jll to version=1.12.2 with the following commands.

> using Pkg
> Pkg.add(name="package_name",version="version_number")

Note that you may have to run

> ]
resolve
instantiate

to avoid errors when compiling for the first time after choosing to use different modules.
In my (now working) test install, I have

(moment_kinetics_test_install) pkg> status
Status `~/excalibur/moment_kinetics_test_install/Project.toml`
⌃ [da04e1cc] MPI v0.20.21
  [3da0fdf6] MPIPreferences v0.1.11
⌃ [9b87118b] PackageCompiler v2.1.17
  [276daf66] SpecialFunctions v2.4.0
  [2913bbd2] StatsBase v0.34.3
  [b5ff72cc] moment_kinetics v0.1.0 `moment_kinetics`
⌃ [0234f1f7] HDF5_jll v1.12.2+2
Info Packages marked with ⌃ have new versions available and may be upgradable.

and

(moment_kinetics_test_install) pkg> status -m
Status `~/excalibur/moment_kinetics_test_install/Manifest.toml`
  [621f4979] AbstractFFTs v1.5.0
⌃ [7d9f7c33] Accessors v0.1.37
  [79e6a3ab] Adapt v4.0.4
  [66dad0bd] AliasTables v1.1.3
  [c7e460c6] ArgParse v1.2.0
  [4fba245c] ArrayInterface v7.16.0
  [13072b0f] AxisAlgorithms v1.1.0
⌃ [d360d2e6] ChainRulesCore v1.24.0
  [da1fd8a2] CodeTracking v1.3.6
  [861a8166] Combinatorics v1.0.2
  [38540f10] CommonSolve v0.2.4
  [bbf7d656] CommonSubexpressions v0.3.1
  [34da2185] Compat v4.16.0
  [a33af91c] CompositionsBase v0.1.2
  [187b0558] ConstructionBase v1.5.8
  [9a962f9c] DataAPI v1.16.0
  [864edb3b] DataStructures v0.18.20
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.15.1
⌃ [31c24e10] Distributions v0.25.111
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.0
  [442a2c76] FastGaussQuadrature v1.0.2
  [1a297f60] FillArrays v1.13.0
⌃ [6a86dc24] FiniteDiff v2.24.0
  [f6369f11] ForwardDiff v0.10.36
  [c27321d9] Glob v1.3.1
  [f67ccb44] HDF5 v0.17.2
  [34004b35] HypergeometricFunctions v0.3.24
  [18e54dd8] IntegerMathUtils v0.1.2
  [a98d9a8b] Interpolations v0.15.1
⌃ [3587e190] InverseFunctions v0.1.16
  [92d709cd] IrrationalConstants v0.2.2
  [692b3bcd] JLLWrappers v1.6.0
  [aa1ae85d] JuliaInterpreter v0.9.36
  [3db4a2ba] LegendrePolynomials v0.4.4
  [2ab3a3ac] LogExpFunctions v0.3.28
⌃ [6f1432cf] LoweredCodeUtils v3.0.2
  [2fda8390] LsqFit v0.15.0
⌃ [da04e1cc] MPI v0.20.21
  [3da0fdf6] MPIPreferences v0.1.11
  [1914dd2f] MacroTools v0.5.13
  [442fdcdd] Measures v0.3.2
  [e1d29d7a] Missings v1.2.0
  [d41bc354] NLSolversBase v7.8.3
  [77ba4419] NaNMath v1.0.2
  [6fe1bfb0] OffsetArrays v1.14.1
  [bac558e1] OrderedCollections v1.6.3
  [90014a1f] PDMats v0.11.31
⌃ [9b87118b] PackageCompiler v2.1.17
  [eebad327] PkgVersion v0.3.3
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [27ebfcd6] Primes v0.5.6
  [43287f4e] PtrArrays v1.2.1
⌃ [1fd47b50] QuadGK v2.11.0
  [be4d8f0f] Quadmath v0.5.10
  [c84ed2f1] Ratios v0.4.5
  [189a3867] Reexport v1.2.2
  [05181044] RelocatableFolders v1.0.1
  [ae029012] Requires v1.3.0
⌃ [295af30f] Revise v3.5.18
⌅ [79098fc4] Rmath v0.7.1
⌃ [f2b01f46] Roots v2.1.8
  [6c6a2e73] Scratch v1.2.1
  [efcf1570] Setfield v1.1.1
  [a2af1166] SortingAlgorithms v1.2.1
  [a0a7dd2c] SparseMatricesCSR v0.6.7
  [276daf66] SpecialFunctions v2.4.0
  [90137ffa] StaticArrays v1.9.7
  [1e83bf80] StaticArraysCore v1.4.3
  [82ae8749] StatsAPI v1.7.0
  [2913bbd2] StatsBase v0.34.3
⌃ [4c63d2b9] StatsFuns v1.3.1
  [b718987f] TextWrap v1.0.2
⌃ [a759f4b9] TimerOutputs v0.5.24
  [1986cc42] Unitful v1.21.0
  [efce3f68] WoodburyMatrices v1.0.0
  [b5ff72cc] moment_kinetics v0.1.0 `moment_kinetics`
⌃ [f5851436] FFTW_jll v3.3.10+0
⌃ [0234f1f7] HDF5_jll v1.12.2+2
⌃ [e33a78d0] Hwloc_jll v2.11.1+0
  [1d5cc7b8] IntelOpenMP_jll v2024.2.1+0
  [856f044c] MKL_jll v2024.2.0+0
⌃ [7cb0a576] MPICH_jll v4.2.2+0
⌃ [f1f71cc9] MPItrampoline_jll v5.4.0+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
⌅ [fe0851c0] OpenMPI_jll v4.1.6+0
⌅ [458c3c95] OpenSSL_jll v1.1.23+0
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
⌅ [f50d1b31] Rmath_jll v0.4.3+0
  [1317d2d5] oneTBB_jll v2021.12.0+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [05823500] OpenLibm_jll v0.8.1+2
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌃ and ⌅ have new versions available. Those with ⌃ may be upgradable, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated -m`

On commit 9284bd9, I have the following output from .julia/bin/mpiexecjl --project=./ -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'.

Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 120315       2  120317  22m20.3s
Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 121005       2  121007  22m22.9s

@johnomotani Is this finding worth reproducing or reporting? It seems like newer versions of MPI and HD5_jll might be in conflict with Test.

johnomotani · 2024-10-14T10:37:51Z

Thanks Michael! I think I may be starting to understand what's happening. In #277 I unpinned the HDF5_jll version, because the newer version can work now. However, the newer HDF5_jll does link MPI, so if you use a 'system MPI' (rather than the Julia-provided MPI) you have to also link an external HDF5. Sorry, I assumed everyone was already doing this in order to use parallel I/O, but should have flagged it and documented it better. If this is the problem, I don't know why things seemed to work when not using a system image, but library linking conflicts can just cause random errors which sometimes 'work' and sometimes crash and sometimes mess up results. You can see the commands to compile HDF5 (with the system MPI linked) at the bottom of the script `machines/generic-pc/compile_dependencies.sh`. It'd be nice if that script could run standalone, but the script needs a bit of restructuring to do that.

mrhardman · 2024-10-14T11:46:05Z

Unfortunately, I don't now think this problem is down to using the system MPI. The last set of reports were made using the julia MPI with `mpiexecjl`(i.e., not loading any system modules besides julia/1.10.2), because I found that the precompilation bug affected installs using both the system-provided MPI and the native julia MPI. I have tried to make a new install script to pin in advance to the MPI and HDF5_jll modules mentioned above, but this was not enough to avoid the errors. When I have time I will try to work out which other libraries might need to be pinned.

mrhardman · 2024-10-15T07:28:56Z

After pinning PackageCompiler, MPItrampoline and MPICH_jll to the version numbers given above, but letting the fresh install pick versions for the other modules (note that Test is not in Manifest.toml), I see the following errors

3$ .julia/bin/mpiexecjl --project=./ -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'
┌ Warning: attempting to remove probably stale pidfile
│   path = */excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/Test/JfdTE_pQyWD.ji.pidfile"
└ @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:244
┌ Warning: attempting to remove probably stale pidfile
│   path = "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/Test/JfdTE_pQyWD.ji.pidfile"
└ @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:244
┌ Warning: failed to remove pidfile on close
│   path = "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/Test/JfdTE_pQyWD.ji.pidfile"
│   removed = false
└ @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:342
┌ Warning: attempting to remove probably stale pidfile
│   path = "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/AbstractFFTsTestExt/Pr5Et_pQyWD.ji.pidfile"
└ @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:244
[cli_0]: PMIU_write error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdaf818840) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=404350479 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdaf818840) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension AbstractFFTsTestExt of AbstractFFTs, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile AbstractFFTsTestExt [f14dd36e-a047-53ac-8286-f8b5d0265aae] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/AbstractFFTsTestExt/jl_36IbHX".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_0]: PMIU_write error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffc4f06ffe0) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=135915023 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffc4f06ffe0) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension InverseFunctionsTestExt of InverseFunctions, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile InverseFunctionsTestExt [05f3ecc3-30a3-5b98-a414-5bae29b80c93] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/InverseFunctionsTestExt/jl_nyz750".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_0]: PMIU_write error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdd72111e0) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=471459343 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdd72111e0) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension AccessorsTestExt of Accessors, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile AccessorsTestExt [3f8c54d2-55ce-559e-9981-66f7a1691a45] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/AccessorsTestExt/jl_Q5TNqZ".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_0]: PMIU_write error; fd=9 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_0]: PMIU_write error; fd=9 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffd9b780f40) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_0]: PMIU_write error; fd=9 buf=:cmd=abort exitcode=404350479 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffd9b780f40) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension DistributionsTestExt of Distributions, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile DistributionsTestExt [ffbe0ea5-a612-5ff7-aaf5-cac02eef3019] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/DistributionsTestExt/jl_RIbuV0".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_1]: PMIU_write error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdee051320) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=471459343 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffdee051320) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension AbstractFFTsTestExt of AbstractFFTs, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile AbstractFFTsTestExt [f14dd36e-a047-53ac-8286-f8b5d0265aae] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/AbstractFFTsTestExt/jl_Tvfu44".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_1]: PMIU_write error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffcaf587a40) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=68806159 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7ffcaf587a40) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension InverseFunctionsTestExt of InverseFunctions, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile InverseFunctionsTestExt [05f3ecc3-30a3-5b98-a414-5bae29b80c93] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/InverseFunctionsTestExt/jl_mehCm2".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301
[cli_1]: PMIU_write error; fd=10 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=-1 message=PMI_Init failed
:
system msg for write_line failure : Broken pipe
[cli_1]: PMIU_write error; fd=10 buf=:cmd=get_appnum
:
system msg for write_line failure : Broken pipe
Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7fff4148d360) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
[cli_1]: PMIU_write error; fd=10 buf=:cmd=abort exitcode=135915023 message=Fatal error in internal_Init_thread: Other MPI error, error stack:
internal_Init_thread(48399): MPI_Init_thread(argc=(nil), argv=(nil), required=2, provided=0x7fff4148d360) failed
MPII_Init_thread(195)......:
MPIR_pmi_init(149).........:
(unknown)(): Other MPI error
:
system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension AccessorsTestExt of Accessors, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Failed to precompile AccessorsTestExt [3f8c54d2-55ce-559e-9981-66f7a1691a45] to "*/excalibur/moment_kinetics_test_install3/.julia/compiled/v1.10/AccessorsTestExt/jl_XUkpn6".
│    Stacktrace:
│      [1] error(s::String)
│        @ Base ./error.jl:35
│      [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
│        @ Base ./loading.jl:2468
│      [3] compilecache
│        @ ./loading.jl:2340 [inlined]
│      [4] (::Base.var"#968#969"{Base.PkgId})()
│        @ Base ./loading.jl:1974
│      [5] mkpidlock(f::Base.var"#968#969"{Base.PkgId}, at::String, pid::Int32; kwopts::@Kwargs{stale_age::Int64, wait::Bool})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:93
│      [6] #mkpidlock#6
│        @ /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:88 [inlined]
│      [7] trymkpidlock(::Function, ::Vararg{Any}; kwargs::@Kwargs{stale_age::Int64})
│        @ FileWatching.Pidfile /network/software/linux-x86_64/julia/1.10.2/share/julia/stdlib/v1.10/FileWatching/src/pidfile.jl:111
│      [8] #invokelatest#2
│        @ ./essentials.jl:894 [inlined]
│      [9] invokelatest
│        @ ./essentials.jl:889 [inlined]
│     [10] maybe_cachefile_lock(f::Base.var"#968#969"{Base.PkgId}, pkg::Base.PkgId, srcpath::String; stale_age::Int64)
│        @ Base ./loading.jl:2983
│     [11] maybe_cachefile_lock
│        @ ./loading.jl:2980 [inlined]
│     [12] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1970
│     [13] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:1812
│     [14] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [15] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [16] _require_prelocked
│        @ ./loading.jl:1803 [inlined]
│     [17] _require_prelocked
│        @ ./loading.jl:1802 [inlined]
│     [18] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1295
│     [19] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1330
│     [20] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1164
│     [21] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1819
│     [22] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [23] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [24] _require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:1803
│     [25] macro expansion
│        @ ./loading.jl:1790 [inlined]
│     [26] macro expansion
│        @ ./lock.jl:267 [inlined]
│     [27] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1753
│     [28] #invoke_in_world#3
│        @ ./essentials.jl:926 [inlined]
│     [29] invoke_in_world
│        @ ./essentials.jl:923 [inlined]
│     [30] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:1746
│     [31] include(mod::Module, _path::String)
│        @ Base ./Base.jl:495
│     [32] include(x::String)
│        @ Main.MomentKineticsTests ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:1
│     [33] top-level scope
│        @ ~/excalibur/moment_kinetics_test_install3/moment_kinetics/test/runtests.jl:3
│     [34] include(fname::String)
│        @ Base.MainInclude ./client.jl:489
│     [35] top-level scope
│        @ none:1
│     [36] eval
│        @ ./boot.jl:385 [inlined]
│     [37] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:291
│     [38] _start()
│        @ Base ./client.jl:552
└ @ Base loading.jl:1301

All of these errors seem to be in the form

system msg for write_line failure : Broken pipe
┌ Error: Error during loading of extension AccessorsTestExt of Accessors, use `Base.retry_load_extensions()` to retry.

To me this looks like the "test extension" of Accessors is not found.

Despite these errors, the tests ran and passed on this occasion.

Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 120315       2  120317  22m13.3s
Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 121005       2  121007  22m25.3s
[cli-0]: PMIU_write error; fd=9 buf=:cmd=finalize
:
system msg for write_line failure : Broken pipe
[cli-1]: PMIU_write error; fd=10 buf=:cmd=finalize
:
system msg for write_line failure : Broken pipe

mrhardman · 2024-10-15T13:37:05Z

On 57e0d1d I think I have found a adequate resolution (for now) to the problem discussed here. The missing information in my previous setup was the extra module requirements that have crept in. Explicitly, if one uses a setup script as below

#!/bin/bash
# simple moment_kinetics install script
# not supporting parallel HDF5
# not supporting diagnostics (yet)

git clone https://github.com/mabarnes/moment_kinetics.git moment_kinetics_test_install
cd moment_kinetics_test_install

# set up modules and environment variables
# need this everytime you use Julia
# JULIA_DEPOT_PATH must be set to be the same for each use of a specific install
module load julia/1.10.2
export JULIA_DEPOT_PATH=$(pwd)/.julia
echo $JULIA_DEPOT_PATH

# develop moment_kinetics, no plots, no symbolic function tests
touch Project.toml
julia --project -O3 -e 'using Pkg; Pkg.develop(path="./moment_kinetics")'
julia --project -O3 -e 'using Pkg; Pkg.add("MPIPreferences")'
julia --project -O3 -e 'using Pkg; Pkg.add("MPI")'
julia --project -O3 -e 'using Pkg; Pkg.add("Test")'
julia --project -O3 -e 'using Pkg; Pkg.add("SpecialFunctions")'
julia --project -O3 -e 'using Pkg; Pkg.add("PackageCompiler")'
julia --project -O3 -e 'using Pkg; Pkg.add("StatsBase")'

# setup MPI preferences and binary
julia --project -O3 -e 'using Pkg; Pkg.instantiate()'
julia --project -O3 -e 'using Pkg; Pkg.resolve()'
julia --project -O3 -e 'using MPI; MPI.install_mpiexecjl(force=true)'
julia --project -O3 -e 'using MPIPreferences; MPIPreferences.use_jll_binary()'
julia --project -O3 -e 'using Pkg; Pkg.instantiate()'
julia --project -O3 -e 'using Pkg; Pkg.resolve()'

# generate moment_kinetics.so
julia --project -O3 precompile.jl

# test with moment_kinetics.so and native Julia MPI executable
.julia/bin/mpiexecjl --project=./ -n 2 julia --project -O3 -Jmoment_kinetics.so -e 'include("moment_kinetics/test/runtests.jl")'

Then on the system on which I am working, I finally obtain passing tests with MPI using the moment_kinetics.so file, i.e., the final output is as follows.

Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 120315       2  120317  29m57.8s
Test Summary:         |   Pass  Broken   Total      Time
moment_kinetics tests | 121149       2  121151  30m00.4s

To try to help new manual installs of the future to be less painful, I have suggested changes in #285. The list of packages used in the install on my present system is given below.

Status `~/excalibur/moment_kinetics_test_install2/Project.toml`
  [da04e1cc] MPI v0.20.22
  [3da0fdf6] MPIPreferences v0.1.11
  [9b87118b] PackageCompiler v2.1.18
  [276daf66] SpecialFunctions v2.4.0
  [2913bbd2] StatsBase v0.34.3
  [b5ff72cc] moment_kinetics v0.1.0 `moment_kinetics`
  [8dfed614] Test
Status `~/excalibur/moment_kinetics_test_install2/Manifest.toml`
  [621f4979] AbstractFFTs v1.5.0
  [7d9f7c33] Accessors v0.1.38
  [79e6a3ab] Adapt v4.0.4
  [66dad0bd] AliasTables v1.1.3
  [c7e460c6] ArgParse v1.2.0
  [4fba245c] ArrayInterface v7.16.0
  [13072b0f] AxisAlgorithms v1.1.0
  [d360d2e6] ChainRulesCore v1.25.0
  [da1fd8a2] CodeTracking v1.3.6
  [861a8166] Combinatorics v1.0.2
  [38540f10] CommonSolve v0.2.4
  [bbf7d656] CommonSubexpressions v0.3.1
  [34da2185] Compat v4.16.0
  [a33af91c] CompositionsBase v0.1.2
  [187b0558] ConstructionBase v1.5.8
  [9a962f9c] DataAPI v1.16.0
  [864edb3b] DataStructures v0.18.20
  [163ba53b] DiffResults v1.1.0
  [b552c78f] DiffRules v1.15.1
  [31c24e10] Distributions v0.25.112
  [ffbed154] DocStringExtensions v0.9.3
  [e2ba6199] ExprTools v0.1.10
  [7a1cc6ca] FFTW v1.8.0
  [442a2c76] FastGaussQuadrature v1.0.2
  [1a297f60] FillArrays v1.13.0
  [6a86dc24] FiniteDiff v2.26.0
  [f6369f11] ForwardDiff v0.10.36
  [c27321d9] Glob v1.3.1
  [f67ccb44] HDF5 v0.17.2
  [34004b35] HypergeometricFunctions v0.3.24
  [18e54dd8] IntegerMathUtils v0.1.2
  [a98d9a8b] Interpolations v0.15.1
  [3587e190] InverseFunctions v0.1.17
  [92d709cd] IrrationalConstants v0.2.2
  [692b3bcd] JLLWrappers v1.6.1
  [aa1ae85d] JuliaInterpreter v0.9.36
  [3db4a2ba] LegendrePolynomials v0.4.4
  [2ab3a3ac] LogExpFunctions v0.3.28
  [6f1432cf] LoweredCodeUtils v3.0.3
  [2fda8390] LsqFit v0.15.0
  [da04e1cc] MPI v0.20.22
  [3da0fdf6] MPIPreferences v0.1.11
  [1914dd2f] MacroTools v0.5.13
  [442fdcdd] Measures v0.3.2
  [e1d29d7a] Missings v1.2.0
  [d41bc354] NLSolversBase v7.8.3
  [77ba4419] NaNMath v1.0.2
  [6fe1bfb0] OffsetArrays v1.14.1
  [bac558e1] OrderedCollections v1.6.3
  [90014a1f] PDMats v0.11.31
  [9b87118b] PackageCompiler v2.1.18
  [eebad327] PkgVersion v0.3.3
  [aea7be01] PrecompileTools v1.2.1
  [21216c6a] Preferences v1.4.3
  [27ebfcd6] Primes v0.5.6
  [43287f4e] PtrArrays v1.2.1
  [1fd47b50] QuadGK v2.11.1
  [be4d8f0f] Quadmath v0.5.10
  [c84ed2f1] Ratios v0.4.5
  [189a3867] Reexport v1.2.2
  [05181044] RelocatableFolders v1.0.1
  [ae029012] Requires v1.3.0
  [295af30f] Revise v3.6.1
  [79098fc4] Rmath v0.8.0
  [f2b01f46] Roots v2.2.1
  [6c6a2e73] Scratch v1.2.1
  [efcf1570] Setfield v1.1.1
  [a2af1166] SortingAlgorithms v1.2.1
  [a0a7dd2c] SparseMatricesCSR v0.6.7
  [276daf66] SpecialFunctions v2.4.0
  [90137ffa] StaticArrays v1.9.7
  [1e83bf80] StaticArraysCore v1.4.3
  [82ae8749] StatsAPI v1.7.0
  [2913bbd2] StatsBase v0.34.3
  [4c63d2b9] StatsFuns v1.3.2
  [b718987f] TextWrap v1.0.2
  [a759f4b9] TimerOutputs v0.5.25
  [1986cc42] Unitful v1.21.0
  [efce3f68] WoodburyMatrices v1.0.0
  [b5ff72cc] moment_kinetics v0.1.0 `moment_kinetics`
  [f5851436] FFTW_jll v3.3.10+1
⌃ [0234f1f7] HDF5_jll v1.14.2+1
  [e33a78d0] Hwloc_jll v2.11.2+0
  [1d5cc7b8] IntelOpenMP_jll v2024.2.1+0
  [1d63c593] LLVMOpenMP_jll v18.1.7+0
  [856f044c] MKL_jll v2024.2.0+0
  [7cb0a576] MPICH_jll v4.2.3+0
  [f1f71cc9] MPItrampoline_jll v5.5.1+0
  [9237b28f] MicrosoftMPI_jll v10.1.4+2
  [fe0851c0] OpenMPI_jll v5.0.5+0
  [458c3c95] OpenSSL_jll v3.0.15+1
  [efe28fd5] OpenSpecFun_jll v0.5.5+0
  [f50d1b31] Rmath_jll v0.5.1+0
  [477f73a3] libaec_jll v1.1.2+0
  [1317d2d5] oneTBB_jll v2021.12.0+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [8ba89e20] Distributed
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [9fa8497b] Future
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.4
  [76f85450] LibGit2
  [8f399da3] Libdl
  [37e2e46d] LinearAlgebra
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.10.0
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [1a1011a3] SharedArrays
  [6462fe0b] Sockets
  [2f01184e] SparseArrays v1.10.0
  [10745b16] Statistics v1.10.0
  [4607b0f0] SuiteSparse
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [e66e0078] CompilerSupportLibraries_jll v1.1.0+0
  [deac9b47] LibCURL_jll v8.4.0+0
  [e37daf67] LibGit2_jll v1.6.4+0
  [29816b5a] LibSSH2_jll v1.11.0+1
  [c8ffd9c3] MbedTLS_jll v2.28.2+1
  [14a3606d] MozillaCACerts_jll v2023.1.10
  [4536629a] OpenBLAS_jll v0.3.23+4
  [05823500] OpenLibm_jll v0.8.1+2
  [bea87d4a] SuiteSparse_jll v7.2.1+1
  [83775a58] Zlib_jll v1.2.13+1
  [8e850b90] libblastrampoline_jll v5.8.0+1
  [8e850ede] nghttp2_jll v1.52.0+1
  [3f19e933] p7zip_jll v17.4.0+2
Info Packages marked with ⌃ have new versions available and may be upgradable.

johnomotani · 2024-10-18T09:57:01Z

Compiling a system image is now segfaulting for me using Julia-1.11.1 on (more-or-less) the master branch. Need to look into this when I get a chance...

Edit: this segfault does not happen with Julia-1.10.5.

mrhardman assigned johnomotani and mrhardman Oct 8, 2024

mrhardman added the bug Something isn't working label Oct 8, 2024

mrhardman mentioned this issue Oct 9, 2024

Option to use Julia-provided MPI in machine setup #278

Merged

mrhardman mentioned this issue Oct 15, 2024

Suggestions for online documentation of manual setup #285

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Precompile bugs? #275

Precompile bugs? #275

mrhardman commented Oct 8, 2024 •

edited

Loading

mrhardman commented Oct 8, 2024 •

edited

Loading

johnomotani commented Oct 8, 2024

johnomotani commented Oct 8, 2024

johnomotani commented Oct 8, 2024

mrhardman commented Oct 9, 2024 •

edited

Loading

mrhardman commented Oct 11, 2024

mrhardman commented Oct 11, 2024

johnomotani commented Oct 14, 2024 via email

mrhardman commented Oct 14, 2024 via email •

edited

Loading

mrhardman commented Oct 15, 2024 •

edited

Loading

mrhardman commented Oct 15, 2024

johnomotani commented Oct 18, 2024 •

edited

Loading

Precompile bugs? #275

Precompile bugs? #275

Comments

mrhardman commented Oct 8, 2024 • edited Loading

mrhardman commented Oct 8, 2024 • edited Loading

johnomotani commented Oct 8, 2024

johnomotani commented Oct 8, 2024

johnomotani commented Oct 8, 2024

mrhardman commented Oct 9, 2024 • edited Loading

mrhardman commented Oct 11, 2024

mrhardman commented Oct 11, 2024

johnomotani commented Oct 14, 2024 via email

mrhardman commented Oct 14, 2024 via email • edited Loading

mrhardman commented Oct 15, 2024 • edited Loading

mrhardman commented Oct 15, 2024

johnomotani commented Oct 18, 2024 • edited Loading

mrhardman commented Oct 8, 2024 •

edited

Loading

mrhardman commented Oct 8, 2024 •

edited

Loading

mrhardman commented Oct 9, 2024 •

edited

Loading

mrhardman commented Oct 14, 2024 via email •

edited

Loading

mrhardman commented Oct 15, 2024 •

edited

Loading

johnomotani commented Oct 18, 2024 •

edited

Loading