Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Openmpi bugs #31

Closed
wants to merge 63 commits into from
Closed

Openmpi bugs #31

wants to merge 63 commits into from

Conversation

alexandrebouchard
Copy link
Member

Potentially a new bug that showed up on Saturday after all tests reintroduced. Maybe same as JuliaParallel/MPI.jl#725 but not sure yet.

alexandrebouchard and others added 30 commits March 6, 2023 09:29
@alexandrebouchard
Keeping only the suspected faulty test (by commenting out rest (!))
23b59d3
@alexandrebouchard
Removing Turing from test to isolate only one error at the time
40001d0
@alexandrebouchard
Check behaviour on different threadlevels
ff5aac9
@alexandrebouchard
Back to same threadlevel as not changing crashing behaviour
d00ff53
@alexandrebouchard
More conservative tag ub?
69bb270
@alexandrebouchard
Revert "More conservative tag ub?"
d94252f 
This reverts commit 69bb270.
@alexandrebouchard
Trying on OpenMPI instead of mpich
bcf016d
@alexandrebouchard
Going back to mpich, trying threadlevel = :multiple
379df4b
@alexandrebouchard
Back to funneled as :multiple still crashes
2cd3571
@alexandrebouchard
Further simplification
c8a0261
@alexandrebouchard
Candidate fix
6217650
@alexandrebouchard
Reintroducing other tests
1f16936
@miguelbiron
purge tests from Turing, use only DynamicPPL
0ca198f
@alexandrebouchard
Another free
ee276fc
@alexandrebouchard
Trying to run tests with more rounds
491c26b
@miguelbiron
system MPI test
fab7934
@miguelbiron
fix typos
2ed0aae
@miguelbiron
remove step that does not make sense outside MPI.jl
c271261
@miguelbiron
setup MPIPreferences in the test env
e3f569a
@miguelbiron
possible fix to Pkg missing
77824ea
@miguelbiron
possible fix to Pkg missing
c69b76a
@miguelbiron
add missing env
523f9c1
@miguelbiron
another try
949b628
@miguelbiron
test now has its own Project.toml
895a45f
@miguelbiron
force Pkg.instantiate in ChildProcess
13c9cd2
@miguelbiron
dont use --project in ChildProcess
1772d82
@miguelbiron
ChildProcess inherits the exact same active project
bdf71ce
@miguelbiron
new approach
2bcd825
@alexandrebouchard
Changing approach for Isend/isend: explicit Waitall for all requests
eadbfcd 
A crash seemed to occur in CI during garbage collection

julia:3364 terminated with signal 11 at PC=7f997aaec971 SP=7f9936ccb970.  Backtrace:
/opt/hostedtoolcache/julia/1.8.5/x64/bin/../lib/julia/libjulia-internal.so.1(ijl_gc_safepoint+0x11)[0x7f997aaec971]

This looks like a crash during garbage collection code (this would be consistent with the fact that it only shows up with allocation-heavy test, i.e. Turing).

So even though the finalizer checks first before calling free, maybe there is something flaky there.

#30 (comment)
@alexandrebouchard
Is this a Turing multi-threading issue?
57d0921
miguelbiron and others added 22 commits March 9, 2023 14:52
@miguelbiron
add --oversubscribe for OpenMPI + rethrow exception
82a4210
@miguelbiron
simplify logging
700c712
@miguelbiron
force instantiate + precompile
ebcb3a6
@alexandrebouchard
Add mpi_args to mpi_test
ad69b34
@alexandrebouchard
Temporary: trying to speed up some key tests
6bde8fe
@alexandrebouchard
Fix
7b5e7d4
@alexandrebouchard
Fix the fix + reintroducing the system-MPI tests
fc34b85
@alexandrebouchard
Add back libmpich-dev to resume investigation on ghostbug
5ba839e
@alexandrebouchard
Trying to simplify CI setup needed to reproduce ghostbug
2b37c0f
@alexandrebouchard
Fix last commit
0847901
@alexandrebouchard
toy_mvn not enough to manifest ghostbug, trying Turing
f75a280
@miguelbiron
test mpich+openmpi using brew
93033fc
@miguelbiron
fix wrong abi detection for mpich
d46d429
@miguelbiron
move xtra args to MPI struct + remove prints + failsafe for empty cur…
7a11074 
…rent project
@miguelbiron
mpiexec args for childprocess
d89911a
@miguelbiron
re-introduce all CI tests + fix bug in building mpi cmd
aa233e8
@miguelbiron
add support for using without Project.toml
42993fd
@miguelbiron
add comment explaining why we wait on Isend
42bdd75
@miguelbiron
mpiexec_args is a Cmd now
c5cb70a
@miguelbiron
re-instate all tests
9e31b49
@alexandrebouchard
Do we have the same bug with 1 thread
f93548d
@alexandrebouchard
Explicitly disable GC in all pigeons runs
495b4eb
@alexandrebouchard
Copy link
Member Author

This might be related: JuliaPlots/Plots.jl#3583

Possibly the problem is when JJWrappers prepares the mpiexec environment

https://github.com/JuliaPackaging/JLLWrappers.jl/blob/e5b7b192484c3b9fe51dbed03df269945f8c7594/src/products/executable_generators.jl#L22

@alexandrebouchard
Copy link
Member Author

Close inspection of a failing runtest.jl based on a system MPI revealed the following:

  • the following line works if placed at the very beginning of runtest but not later on in the script
run(`/usr/local/bin/mpiexec ls`)
  • printing the value of the ENV variable reveals that it gets updated somewhere in between, with added values:
IPATH_NO_BACKTRACE=1
HFI_NO_BACKTRACE=1
OMPI_MCA_ess=singleton
ORTE_SCHIZO_DETECTION=ORTE
OMPI_MCA_orte_launch=1
PMIX_NAMESPACE=1092091905
PMIX_RANK=0
PMIX_SERVER_URI3=1092091904.0;tcp4://127.0.0.1:53361
PMIX_SERVER_URI2=1092091904.0;tcp4://127.0.0.1:53361
PMIX_SERVER_URI21=1092091904.0;tcp4://127.0.0.1:53361
PMIX_SECURITY_MODE=native
PMIX_PTL_MODULE=tcp,usock
PMIX_BFROP_BUFFER_TYPE=PMIX_BFROP_BUFFER_NON_DESC
PMIX_GDS_MODULE=ds21,ds12,hash
PMIX_SERVER_TMPDIR=/tmp/ompi.korolev.501/pid.38533
PMIX_SYSTEM_TMPDIR=/tmp
PMIX_DSTORE_21_BASE_PATH=/tmp/ompi.korolev.501/pid.38533/pmix_dstor_ds21_38533
PMIX_DSTORE_ESH_BASE_PATH=/tmp/ompi.korolev.501/pid.38533/pmix_dstor_ds12_38533
PMIX_HOSTNAME=korolev.local
PMIX_VERSION=3.2.4rc1
OMPI_MCA_orte_precondition_transports=f14ed8193d52a2bd-56a807c4118762de
OMPI_MCA_pmix=^s1,s2,cray,isolated
PMIX_MCA_mca_base_component_show_load_errors=1
PMIX_MCA_ptl=tcp,usock
PMIX_MCA_psec=native
PMIX_MCA_gds=ds21,ds12,hash
OMPI_MCA_orte_ess_num_procs=1
OMPI_APP_CTX_NUM_PROCS=1

As soon as these additional environment variables are set, it seems that calling mpiexec no longer works (returns the uninformative pipeline_error).

Frustratingly, it is not clear at all what makes that modification. Probably some hack in MPIPreferences/JLLWrapper/something else?

@alexandrebouchard
Copy link
Member Author

Possibly there: https://github.com/JuliaParallel/MPI.jl/blob/6d513bb37da76182f2c31baff44f1f53a103295d/lib/MPIPreferences/src/system.jl#L10

@miguelbiron
Copy link
Collaborator

That is very interesting! How did you figure this out!? Can you highlight the extra ENV variables that appear?

@alexandrebouchard
Copy link
Member Author

The one included are just the additional ones.

@miguelbiron
Copy link
Collaborator

Yikes...

@alexandrebouchard
Copy link
Member Author

Those issues were later on fixed in #30

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@alexandrebouchard @miguelbiron