EBs: Compiled by Default, Controlled at Runtime

[ax3l]

Enable embedded boundary compilation by default.
Control via the existing runtime options.

Close #4863

Action Items

• build system & doc default updates
• introduce runtime control, update runtime logic
• run a non-EB test, ensure it is as fast and as low high-water mark in memory consumption as with EB compile-time disabled

Follow-up

We could require that AMReX dependency is always built with -DAMReX_EB=ON and with that remove all the safety macros AMREX_USE_EB that we currently have to enable that WarpX can also build if AMReX has no EB support. Since EB support in AMReX does not trigger extra dependencies and only moderately increases code complexity for compile (as does that WarpX always requires -DAMReX_PARTICLES=ON), this might be sensible.

To make the compile-time impact more mild, @WeiqunZhang proposed to add another compile-time control to AMReX that disables the compile of a few large linear EB solvers that are not used/needed for WarpX.

[dpgrote]

This is a good thing to do. Though, is a separate input flag needed? The code can check if warpx.eb_implicit_function has been specified, or eb2.geom_type. Elsewhere, enable type input parameters have been removed, relying on specification of the controlling parameters.

[ax3l]

Thanks, Dave!

I was wondering as well, but in the end we have at least three ways to turn it on:
https://warpx.readthedocs.io/en/latest/usage/parameters.html#embedded-boundary-conditions

• warpx.eb_implicit_function
• warpx.eb_potential(x,y,z,t)
• and, currently undocumented, a way to load via STL file in AMReX

We could a) fix the lack of docs and b) write a helper function that checks them or so? Open to suggestions cc @RemiLehe @roelof-groenewald

[roelof-groenewald]

The warpx.eb_potential(x,y,z,t) parameter would not be good to use since it is perfectly fine to leave that out - it is only used by the electrostatic solver and by default the embedded boundary potential is just 0.
I think it would be reasonable to then check if either warpx.eb_implicit_function or the STL file input is specified. We don't expect to have a lot more ways of loading EBs in the future, right? So it shouldn't be hard to maintain a list of or statements to determine the runtime parameter.

[WeiqunZhang]

A separate bool ParmParse parameter is not needed unless we want to the one flag to rule them all. If we want a bool variable to be used in the code for convenience, we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

There are basically two ways to enable EB in WarpXInitEB, (1) warpx.eb_implicit_function (2) AMReX's eb2.geom_type. The STL way is the second way documented here https://amrex-codes.github.io/amrex/docs_html/EB.html?highlight=stl.

[ax3l]

Thank you for all the ideas!

we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

In ImpactX and also more in WarpX, I try to avoid using global variables in the central sim class and instead use globals from the ParmParser. That way, we do not need to keep them in sync when changing options at runtime (e.g., from Python) and do not need to include a heavy central class everywhere.
But I also just introduced WarpX::m_eb_enabled... 😅

I would for now write a free standing helper function instead that we can use :)

[WeiqunZhang]

Maybe a free function in WarpX like this

bool hasEB ()
{
#if defined(AMREX_USE_EB)
    return ! amrex::EB2::IndexSpace::empty();
#else
    return false;
#endif
}

[WeiqunZhang]

Note that amrex::IndexSpace has a static member, a vector of amrex::IndexSpace. Somehow this comes into my mind by your mention of global variable. https://en.wiktionary.org/wiki/%E6%88%91%E4%B8%8D%E5%85%A5%E5%9C%B0%E7%8D%84%EF%BC%8C%E8%AA%B0%E5%85%A5%E5%9C%B0%E7%8D%84

[WeiqunZhang]

Note that amrex::EB2::IndexSpace::empty() is true before WarpX::InitEB is called and will be false if EB is initialized in that function.

[ax3l]

Awesome, that resolves the differences 🎉
No more failing azure integration tests ✨

[ax3l]

Correctness is now ensured (see above & passing CI).

For performance, I benchmarked 3D runs on my laptop and an exclusive Perlmutter CPU node.
Somehow both machines were very noisy today, likely due to random overheads from MPI startups (srun) in the PM case and due to insufficient pinning on my laptop (and other background desktop noise).

From what I can see under the provided noise, there is now no significant performance drawback beyond the noise level for runs that compile with EB but do not use it.

Data & Notebook

2024-09-06_EB-runtime.zip

Perlmutter CPU Node

Compiled and ran in local in-memory file system /dev/shm/.
Allocated via: salloc --exclusive --nodes 1 --qos interactive --time 02:00:00 --constraint cpu --account=m4272 --ntasks-per-node=2
Compiled via: cmake --fresh -S . -B /dev/shm/build_cpu_rteb -DWarpX_COMPUTE=OMP -DWarpX_FFT=ON -DWarpX_DIMS="3" -DMPIEXEC_PREFLAGS="--cpu-bind=cores"
Ran via: ctest --test-dir /dev/shm/build_cpu_rteb -E AMReX -R run

Laptop

[EZoni]

Technically this would be

Suggested change

	for _ in range(2): # skip over: dims, numprocs
	for _ in range(2): # skip over: dims, nprocs

so that the inline comment matches the correct argument name.

I can include this in one of the CTest follow-up PRs if we do not want to rerun all CI tests again on this PR.

[ax3l]

Thanks! Yes let's put in the follow-up to avoid rerunning the CI here.

[ax3l]

added in #5223

[ax3l]

Memory footprint evaluation

I ran Examples/Physics_applications/laser_acceleration/inputs_base_3d, which does not use EBs at runtime, both on development (using WarpX_EB=OFF) and on this PR (using the new default, WarpX_EB=ON).
I used 1 MPI process and one OMP process to avoid fluctuations.

Development Memory Footprint

Pinned Memory Usage:
-------------------------------------------------------------------------
Name                                    Nalloc  Nfree    AvgMem    MaxMem
-------------------------------------------------------------------------
The_Pinned_Arena::Initialize()               1      1  1948   B  8192 KiB
ParticleContainer::addParticles           1984   1984  2411   B  5144 KiB
WarpX::InitData()                           52     52     8   B   131 KiB
main()                                      75     75   799   B   800   B
PhysicalParticleContainer::AddPlasma()       2      2   159   B   160   B
-------------------------------------------------------------------------

Cpu Memory Usage:
---------------------------------------------------------------------------
Name                                      Nalloc  Nfree    AvgMem    MaxMem
---------------------------------------------------------------------------
WarpX::InitData()                             40     40    54 MiB    54 MiB
Diagnostics::FilterComputePackFlush()         50     50   295 KiB  9448 KiB
ParticleContainer::BuildRedistributeMask      16     16  5395 KiB  7892 KiB
FabArray<FAB>::SumBoundary_nowait()         1200   1200   249 KiB  4275 KiB
WarpX::SyncCurrent()                        1200   1200   665 KiB  4275 KiB
WarpX::shiftMF()                            2364   2364  1274 KiB  4275 KiB
LaserParticleContainer::Evolve()               3      3   260 KiB   264 KiB
PhysicalParticleContainer::Evolve()            3      3   260 KiB   264 KiB
Filter::ApplyStencil(MultiFab)               709    709    12 KiB   106 KiB
---------------------------------------------------------------------------

PR Memory Footprint

Pinned Memory Usage:
-------------------------------------------------------------------------
Name                                    Nalloc  Nfree    AvgMem    MaxMem
-------------------------------------------------------------------------
The_Pinned_Arena::Initialize()               1      1  1542   B  8192 KiB
ParticleContainer::addParticles           1984   1984  2337   B  5144 KiB
WarpX::InitData()                           52     52    11   B   131 KiB
main()                                      75     75   799   B   800   B
PhysicalParticleContainer::AddPlasma()       2      2   159   B   160   B
-------------------------------------------------------------------------

Cpu Memory Usage:
---------------------------------------------------------------------------
Name                                      Nalloc  Nfree    AvgMem    MaxMem
---------------------------------------------------------------------------
WarpX::InitData()                             40     40    54 MiB    54 MiB
Diagnostics::FilterComputePackFlush()         50     50   292 KiB  9448 KiB
ParticleContainer::BuildRedistributeMask      16     16  5392 KiB  7892 KiB
FabArray<FAB>::SumBoundary_nowait()         1200   1200   241 KiB  4275 KiB
WarpX::SyncCurrent()                        1200   1200   653 KiB  4275 KiB
WarpX::shiftMF()                            2364   2364  1253 KiB  4275 KiB
LaserParticleContainer::Evolve()               3      3   260 KiB   264 KiB
PhysicalParticleContainer::Evolve()            3      3   260 KiB   264 KiB
Filter::ApplyStencil(MultiFab)               709    709    12 KiB   106 KiB
---------------------------------------------------------------------------

Summary

The maximums do not change, this looks good to me and we seem to effectively prevent allocating EB-related fields if we do not use them.

[RemiLehe]

Could we rename pml_l[xyz]fab as eb_l[xyz]fab?
This is because, with the ifdef not here anymore, it may not be clear to someone reading the code that this is related to embedded boundaries.
(We could also add a comment: // Skip the field update if this gridpoint is inside the embedded boundary.)

[RemiLehe]

Suggested change

	throw std::runtime_error("ComputeDiags only works when EBs are enabled at runtime");
	throw std::runtime_error("ChargeOnEB::ComputeDiags only works when EBs are enabled at runtime");

[RemiLehe]

Same comment as further up: we could rename this as eb_lx to make it clear that it is related to the embedded boundary.

[RemiLehe]

Should we update this?

[ax3l]

Oh lol, yes we did check that :D

[RemiLehe]

Looks good, thanks for these changes!