Add PeleMP cases for soot and spray

[marchdf]

Co-authored-by: Landon Owen [email protected]

[baperry2]

There's too many of these to run all in CI or nightly tests, but I think we should add a couple to ensure good coverage so these things don't break. Right now the Soot case is a "regression test excluded from CI". I'd propose adding "Soot-ZeroD" assuming its pretty cheap so we have something for Soot in the CI. For Spray, we should probably have "Spray-EB" and something that covers splash/breakup (Spray-A-wbreakup? - although this one looks pretty heavy, so maybe it needs to just run in the nightly).

[ldowen]

There's too many of these to run all in CI or nightly tests, but I think we should add a couple to ensure good coverage so these things don't break. Right now the Soot case is a "regression test excluded from CI". I'd propose adding "Soot-ZeroD" assuming its pretty cheap so we have something for Soot in the CI. For Spray, we should probably have "Spray-EB" and something that covers splash/breakup (Spray-A-wbreakup? - although this one looks pretty heavy, so maybe it needs to just run in the nightly).

The Spray A with breakup is definitely too much to be part of the CI. It is intended more as a way to demonstrate how to use the breakup models. A single droplet breakup test would be much more straightforward and easier to run, if we wanted to do that. It would basically use an initial droplet file like we use in the EB spray. It would have the inputs of Spray-A-wbreakup except all of the jet inputs would be gone and an initial droplet file with something like

1
0. 0.3 0.3 0. 0. 6.E4 363. 9E-3 1. 0.1 0.1 0. -1. 0.

would be referenced with particles.init_file =
It would be helpful to ensure the breakup logic continues to work with CUDA and HIP, since particle creation occurs during the particle update, which can pose some complications for GPUs.

[marchdf]

good idea, let me try that

[marchdf]

@ldowen I tried to do that case as you suggest and it is aborting. What did I do wrong? The input file is spray-a-wbreakup.inp

Initializing EB2
Successfully read inputs file ...
Starting to call amrex_probinit ...
Successfully run amrex_probinit
amrex::Error::0::ParticleContainer::InitFromAsciiFile(initspraydata.dat) failed @ 2 !!!
SIGABRT

[ldowen]

@ldowen I tried to do that case as you suggest and it is aborting. What did I do wrong? The input file is spray-a-wbreakup.inp

Initializing EB2
Successfully read inputs file ...
Starting to call amrex_probinit ...
Successfully run amrex_probinit
amrex::Error::0::ParticleContainer::InitFromAsciiFile(initspraydata.dat) failed @ 2 !!!
SIGABRT

I remember there was an issue with reading from ascii files for particles where you had to have an extra empty line after the last particle. It might be that?

[marchdf]

Now that's just plain silly. Thank you, that worked.

[ldowen]

Now that's just plain silly. Thank you, that worked.

I agree. You should make sure the other initial particle files have that. I meant to submit a bug report to AMReX like 3 years ago when I first discovered it and never did. Those files are typically only useful for test cases so I didn't prioritize it.

[marchdf]

@ldowen I am having issues with the ctest stuff for 2 of the tests and I can't seem to figure it out. Works fine with GNUMake but if I use cmake with the following on OSX, I get a very nasty segfault in the destructor of the sprayparticle container... Here is how to reproduce.

export PATH="$(brew --prefix llvm)/bin":$PATH
cmake -DCMAKE_INSTALL_PREFIX:PATH=./install \
      -DCMAKE_CXX_COMPILER:STRING=clang++ \
      -DCMAKE_C_COMPILER:STRING=clang \
      -DMPIEXEC_PREFLAGS:STRING=--oversubscribe \
      -DCMAKE_BUILD_TYPE:STRING=Debug \
      -DPELEC_DIM:STRING=3 \
      -DPELEC_ENABLE_AMREX_EB:BOOL=ON \
      -DPELEC_ENABLE_MPI:BOOL=OFF \
      -DPELEC_ENABLE_TESTS:BOOL=ON \
      -DPELEC_ENABLE_FCOMPARE:BOOL=ON \
      -DPELEC_ENABLE_MASA:BOOL=OFF \
      -DMASA_DIR:STRING="${MASA_HOME}" \
      -DPELEC_ENABLE_ALL_WARNINGS:BOOL=ON \
      -DPELEC_ENABLE_CPPCHECK:BOOL=OFF \
      -DPELEC_ENABLE_CLANG_TIDY:BOOL=OFF \
      -DPELEC_ENABLE_CUDA:BOOL=OFF \
      -DAMReX_CUDA_ARCH=Volta \
      -DPYTHON_EXECUTABLE="$(which python3)" \
      -DPELEC_ENABLE_AMREX_PARTICLES:BOOL=ON \
      -DPELEC_ENABLE_SOOT:BOOL=OFF \
      -DPELE_NUM_SOOT_MOMENTS:INT=3 \
      -DPELEC_PRECISION:STRING=DOUBLE \
      ..

cmake --build . --parallel $(sysctl -n hw.ncpu) --target PeleC-Spray-EB
ctest -R spray-eb

I have therefore excluded spray-eb and spray-a-wbreakup from the Cmake CI. They are still tested in the gnumake CI.

[marchdf]

@baperry2 @jrood-nrel I think this is as ready as it will be. Can you review?

[baperry2]

Do we need this full data file? Seems large.

[marchdf]

I have no idea. @ldowen ?

[baperry2]

I deleted a bunch of zeros that I don't think do anything

[ldowen]

So sorry for the late reply, This was included to demonstrate how a roi data file can be provided. However, this is not necessary and can be removed. Instead, we can just provide a constant injection velocity and mass flow rate in the input file. Also, the roi.dat in Spray-A-wbreakup is no longer used since the jet was removed and replaced with a droplet input file.

[baperry2]

I think we should actually add back the old input file for the Spray-A-wbreakup case, and have it there as a potential demo, while also keeping the lighter weight input for testing

I'm also fine with keeping this here in order to ensure the capability to read these files continues to be tested, now that the file has been trimmed it's not so huge.