Skip to content

Commit

Permalink
3D coupled deliverable (#209)
Browse files Browse the repository at this point in the history
* Fix a CMake syntax error that meant the 'SOLVER_LIBS' list wasn't being assembled properly.

* Skeleton for the Hermes-3 LAPD solver.

* Rename solver source directory for consistency.

* First attempt at a mesh for the H3LAPD problem - tries to match H3 resolution.

* A much lower res mesh for testing the H3LAPD solver.

* Add a convenience script for converting a .geo gmsh config to a Nektar xml. Force uncompressed xml for now.

* First version of LAPD (low-res) config.

* Add advection terms.

* Add pressure gradient terms.

* Add E_perp terms.

* Reorder H3LAPDSystem variables and functions.

* Store perp velocities at class level to allow use in multiple terms.

* Add collision terms to momentum equations and polarisation drift term to vorticity equation.

* Add some comments and docstrings.

* Add explicit values to the low res config file, rather than relying on defaults.

* Added a note on initial conditions.

* More flexible PrintArrVals

* Add density source term.

* Label low, high z boundaries in config

* Initial guess at ICs.

* Comment on the density source term

* Removed two places where Bvec=(0,0,B) and |Bvec| = 1 were assumed.

* Comments and cosmetic

* Fix domain cross section size in low, full res meshes.

* Phi solve attempt 1

* Comments in config xml.

* Set BCs.

* Fix typo in config comment.

* Fix error in config - IterativeSolverTolerance can be in GLOBALSYSSOLNINFO or PARAMETERS, but not SOLVERINFO.

* Improvements to PrintArrVals debugging func.

* Apply non-dimensionalisation factors to input params, BCs, ICs, density source.

* ne, w BCs to Dirichlet, since we don't expect Neumann to be supported for discontinous fields (phi isn't discontinous here).

* Modified timestep, number of steps for consistency with H3 and to reflect scaling of the ion cyclotron frequency.

* Calculate density-dependent Coulomb logarithm and collision frequency for electron-ion interactions.

* Rename a config file variable, for consistency, and explicitly include time scaling in the density source.

* Changed some variable and function names to clarify collision frequency calculation.

* Correction to nu_ei calc (equations doc is wrong).

* More readable expression for  nu_ei_const.

* Correct electric potential scaling factor; rearrange temperature scaling factor expression.

* Rename temperature scaling param; avoids clash with time scaling factor just in case param names aren't case sensitive.

* Allow build dir to be supplied as a relative path in run_eg.sh

* Correct m_u value.

* Different scaling approach; set B0 and derive ts, rather than vice versa.

* Rename dimensionless density and velocity params to match Hermes 3 docs.

* Fix confusion between n_e and nRef in the phi solve.

* Minor; xml formatting

* Add an H3LAPD example that uses a cuboid mesh, works with PhiSolve().

* Correct function misnomer -  AddEPerpTerms => AddEParTerms.

* Correct misleading variable names.

* Zero outarray.

* Correct a number of misnamed variables.

* Implement density floor when computing parallel velocity.

* Add convenience function to zero outarray.

* Implement polarisation drift term via an advection object to properly account for DG flux.

* Fix call to AddAdvTerms for polarisation drift term and add some error checking.

* Allow Helmsolve coefficients to be set by constant factors (not 'variable coefficients') that are read as parameters from the session file. Defaults to [1,1,1].

* Make array arg of PrintArrSize const.

* Provide a virtual function to allow subclasses to define different RHSs for the phi solve.

* Density source as separate function.

* Added Ed's implementation of the 2D-in-3D Hasegawa-Wakatani equations.

* Modified geo to xml script to allow output_basename != input_basename.

* Renamed hw config xml to distinguish clearly from full LAPD config.

* Strip LAPD-specific parameters from HW example and rename others for clarity.

* Modified H3LAPDSystem::CalcEAndAdvVels to avoid referencing momentum fields if the derived system doesn't use them.

* Remove unused momentum fields from HW example.

* Remove unused class.

* Rename all .h extensions to .hpp.

* Remove unused array of forcing objects.

* Remove superfluous function in HW session file.

* Copy neutral_particles header from SimpleSOL and add particle system to H3LAPDSystem. Builds.

* First go at adapting NeutralParticleSystem; no particle init yet.

* Add particle source to density field.

* Add background density via  a session param.

* Reuse Te_eV param in particle system.

* Fix get_point_in_subdomain() for 3D (#211)

* added add more generic implementation to get a point in the domain
* added missing memory cleanup
* added missing api docstrings to get_element functions

* Add particle initialisation.

* Reduce to 100 steps for debugging

* track particle id offset between calls to add_particles

* updated NESO-Particles submodule commit to include RNG updates

* Fix bug with assumed variable order.

* Initialise ne, w, phi to zero.

* Tweak params - makes it stable (ne doesn't NaN), but ionisation near zero.

* added lower to higher order projection

* added drift velocity

* tidy up of ionisation kernel

* removed ErrorPropagate instance that is dead code

* 7 modes => 4 modes

* Set a particle_drift_velocity in the session file.

* Tweak timestep, other params to get a stable example that demonstrates coupling.

* Increase mesh resolution slightly.

* Typo.

* Less particle output.

* Set source width via a param.

* Handle peralign args to NekMesh in geo_to_xml script.

* Revert mesh extent, particle source width to more stable vals.

* Increase particle_number_density to 1e16.

* Reduce num_particles_total to 1e4

* Triple sim time limit.

* Double sim length to try and see blob movement.

* 5 modes

* Fix a (non-critical) error in the geo_to_xml script and document the periodic BC options in the usage message.

* Make NeutralParticleSystem::integrate() behave when num_particles_total=0.

* Bugfix - actually apply the kappa value set in the input file.

* num_particles_total back to 1e5, NUMMODES to 6.

* Set alpha=0.1, kappa=10.

* Restore Ed's example as 'hw_fluid-only'.

* Set alpha=0.1, kappa=10 in hw_fluid-only e.g.

* Tweak fluid-only e.g to get stable sim that produces  turbulence. kappa=3.5, 6x large init ne amplitude, half dt, double sim length.

* Make coupled sim's length, output times, and kappa value the same as the fluid-only e.g.

* Automatically pick up RelWithDebInfo version of NekMesh in geo_to_xml script.

* Various tweaks to make it easier to run fluid-only sims, including removing requirements for ne_src field, num_particles_per_cell param to be defined when not using particles.

* Modify NektarSolverTest to allow solver name to be set independently of test fixture name.

* Add test for growth rate of energy and enstrophy.

* Remove some debugging messages.

* Rename 2D HW equation system, tests and examples in anticipation of 3D HW solver.

* Refactor equation system classes.

* Fix gamma_alpha def in GrowthRatesRecorder.

* Tweak E definition in GrowthRatesRecorder.

* Modify growth rates test to give non-zero gamma_alpha (initial n != phi).

* Tighten up growth rate tolerances.

* var rename

* Move a header.

* Add a mass conservation test for the coupled 2Din3DHW solver (alpha=kappa=0).

* Rename particle system source file.

* Set a more suitable namespace for all H3LAPD solver components.

* Remove superfluous variables from GrowthRatesRecorder.

* Consistent variable naming, ordering and doxygen markup.

* Remove superfluous variable.

* Make non-zero return code a fatal error in HW tests.

* Various tweaks to geo_to_xml script. Added description and example usage at the top of the file.

* Remove old Nektar licence headers from equation system source files and add class-level doxygen markup.

* Add some links to initial markdown for lapd e.g.

* Assert that all required fields have the same number of quadrature points.

* Changes to address various review comments.

* Handle B_xy = 0 correctly.

* Update neso-particles to use NESOASSERT fix and modify some calls to allow for namespace change.

* Use NESOASSERT for NaN check.

* Small increase to mass conservation test tolerance.

---------

Co-authored-by: Will Saunders <[email protected]>
Co-authored-by: Will Saunders <[email protected]>
  • Loading branch information
3 people authored Oct 31, 2023
1 parent a53f8de commit 0d3c970
Show file tree
Hide file tree
Showing 52 changed files with 4,486 additions and 69 deletions.
3 changes: 2 additions & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ set(LIB_SRC_FILES
${SRC_DIR}/nektar_interface/basis_reference.cpp
${SRC_DIR}/nektar_interface/expansion_looping/jacobi_coeff_mod_basis.cpp
${SRC_DIR}/nektar_interface/function_evaluation.cpp
${SRC_DIR}/nektar_interface/geometry_transport/geometry_transport_2d.cpp
${SRC_DIR}/nektar_interface/geometry_transport/geometry_transport_3d.cpp
${SRC_DIR}/nektar_interface/geometry_transport/halo_extension.cpp
${SRC_DIR}/nektar_interface/particle_cell_mapping/map_particles_2d.cpp
Expand All @@ -114,7 +115,7 @@ set(LIB_SRC_FILES
${SRC_DIR}/run_info.cpp
${SRC_DIR}/simulation.cpp
${SRC_DIR}/species.cpp)

set(LIB_SRC_FILES_IGNORE ${SRC_DIR}/main.cpp)
check_file_list(${SRC_DIR} cpp "${LIB_SRC_FILES}" "${LIB_SRC_FILES_IGNORE}")

Expand Down
4 changes: 4 additions & 0 deletions examples/H3LAPD/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Ignore xml files generated from .geos
cuboid*.xml
low_res.xml
full_res.xml
37 changes: 37 additions & 0 deletions examples/H3LAPD/2Din3D-hw/cuboid_periodic_8x8x16.geo
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//=============================== Parameters ==================================
// Lengths and resolutions in each dimension
xsize = 5;
ysize = 5;
zsize = 10;
nx = 8;
ny = 8;
nz = 16;
//=============================================================================

// Create a line in the x-direction of length <xsize>, with <nx> divisions
Point(1) = {-xsize/2, -ysize/2, 0, 0.01};
Point(2) = {xsize/2, -ysize/2, 0, 0.01};
Line(1) = {1, 2};
Transfinite Line(1) = nx+1;

// Extrude split line into meshed square/rectangle
sq = Extrude {0,ysize,0} {Curve{1}; Layers{ny}; Recombine;};

// Extrude square/rectangle into a cuboid
cbd = Extrude {0,0,zsize} {Surface{sq[1]}; Layers{nz}; Recombine;};

// Define physical volume, surfaces for BCs
// Domain
Physical Volume(0) = {cbd[1]};
// Low-x side
Physical Surface(1) = {cbd[5]};
// High-x side
Physical Surface(2) = {cbd[3]};
// Low-y side
Physical Surface(3) = {cbd[2]};
// High-y side
Physical Surface(4) = {cbd[4]};
// Low-z side
Physical Surface(5) = {sq[1]};
// High-z side
Physical Surface(6) = {cbd[0]};
128 changes: 128 additions & 0 deletions examples/H3LAPD/2Din3D-hw/hw.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
<?xml version="1.0" encoding="utf-8" ?>
<NEKTAR>

<COLLECTIONS DEFAULT="MatrixFree" />

<!--
The composite index for the domain is expected to be 0 if the mesh was generated from the included .geo file
-->
<EXPANSIONS>
<E COMPOSITE="C[0]" NUMMODES="6" TYPE="MODIFIED" FIELDS="ne,w,phi,ne_src" />
</EXPANSIONS>

<CONDITIONS>
<SOLVERINFO>
<I PROPERTY="EQTYPE" VALUE="2Din3DHW" />
<I PROPERTY="AdvectionType" VALUE="WeakDG" />
<I PROPERTY="Projection" VALUE="DisContinuous" />
<I PROPERTY="TimeIntegrationMethod" VALUE="ClassicalRungeKutta4" />
<I PROPERTY="UpwindType" VALUE="Upwind" />
</SOLVERINFO>

<GLOBALSYSSOLNINFO>
<V VAR="ne,w,phi,ne_src">
<I PROPERTY="GlobalSysSoln" VALUE="IterativeStaticCond" />
<I PROPERTY="IterativeSolverTolerance" VALUE="1e-6"/>
</V>
</GLOBALSYSSOLNINFO>

<PARAMETERS>
<!-- Timestepping and output options -->
<P> TimeStep = 0.000625 </P>
<P> NumSteps = 64000 </P>
<P> TFinal = NumSteps*TimeStep </P>
<P> IO_InfoSteps = NumSteps/1600 </P>
<P> IO_CheckSteps = NumSteps/160 </P>
<!-- Magnetic field strength -->
<P> Bxy = 1.0 </P>
<!-- d22 Coeff for Helmholtz solve -->
<P> d22 = 0.0 </P>
<!-- HW params -->
<!-- Dissipates density? -->
<P> HW_alpha = 0.1 </P>
<!-- Drives turbulence?-->
<P> HW_kappa = 3.5 </P>
<!-- Scaling factor for ICs -->
<P> s = 0.5 </P>
<!-- Neutral particle system params -->
<P> num_particles_per_cell = -1 </P>
<P> num_particle_steps_per_fluid_step = 1 </P>
<P> num_particles_total = 100000 </P>
<P> particle_num_write_particle_steps = IO_CheckSteps </P>
<P> particle_number_density = 1e16 </P>
<P> particle_position_seed = 1 </P>
<P> particle_thermal_velocity = 1.0 </P>
<P> particle_drift_velocity = 2.0 </P>
<P> particle_source_width = 0.2 </P>
<!-- Temperature in eV used to compute ionisation rate -->
<P> Te_eV = 10.0 </P>
<!-- Assumed background density in SI -->
<P> n_bg_SI = 1e18 </P>
<!-- Unit conversion factors for ionisation calc -->
<P> t_to_SI = 2e-4 </P>
<P> n_to_SI = 1e17 </P>
</PARAMETERS>

<VARIABLES>
<V ID="0"> ne </V>
<V ID="1"> w </V>
<V ID="2"> phi </V>
<V ID="3"> ne_src </V>
</VARIABLES>

<BOUNDARYREGIONS>
<B ID="0"> C[1] </B> <!-- Low x -->
<B ID="1"> C[2] </B> <!-- High x -->
<B ID="2"> C[3] </B> <!-- Low y -->
<B ID="3"> C[4] </B> <!-- High y -->
<B ID="4"> C[5] </B> <!-- Low-z end -->
<B ID="5"> C[6] </B> <!-- High-z end -->
</BOUNDARYREGIONS>

<!-- Periodic conditions for all fields on all boundaries -->
<BOUNDARYCONDITIONS>
<REGION REF="0">
<P VAR="ne" VALUE="[1]" />
<P VAR="w" VALUE="[1]" />
<P VAR="phi" VALUE="[1]" />
<P VAR="ne_src" VALUE="[1]" />
</REGION>
<REGION REF="1">
<P VAR="ne" VALUE="[0]" />
<P VAR="w" VALUE="[0]" />
<P VAR="phi" VALUE="[0]" />
<P VAR="ne_src" VALUE="[0]" />
</REGION>
<REGION REF="2">
<P VAR="ne" VALUE="[3]" />
<P VAR="w" VALUE="[3]" />
<P VAR="phi" VALUE="[3]" />
<P VAR="ne_src" VALUE="[3]" />
</REGION>
<REGION REF="3">
<P VAR="ne" VALUE="[2]" />
<P VAR="w" VALUE="[2]" />
<P VAR="phi" VALUE="[2]" />
<P VAR="ne_src" VALUE="[2]" />
</REGION>
<REGION REF="4">
<P VAR="ne" VALUE="[5]" />
<P VAR="w" VALUE="[5]" />
<P VAR="phi" VALUE="[5]" />
<P VAR="ne_src" VALUE="[5]" />
</REGION>
<REGION REF="5">
<P VAR="ne" VALUE="[4]" />
<P VAR="w" VALUE="[4]" />
<P VAR="phi" VALUE="[4]" />
<P VAR="ne_src" VALUE="[4]" />
</REGION>
</BOUNDARYCONDITIONS>
<FUNCTION NAME="InitialConditions">
<E VAR="ne" DOMAIN="0" VALUE="0" />
<E VAR="w" DOMAIN="0" VALUE="0" />
<E VAR="phi" DOMAIN="0" VALUE="0" />
<E VAR="ne_src" DOMAIN="0" VALUE="0.0" />
</FUNCTION>
</CONDITIONS>
</NEKTAR>
1 change: 1 addition & 0 deletions examples/H3LAPD/2Din3D-hw/run_cmd_template.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mpirun -np <NMPI> <SOLVER_EXEC> hw.xml cuboid.xml
37 changes: 37 additions & 0 deletions examples/H3LAPD/2Din3D-hw_fluid-only/cuboid_periodic_5x5x10.geo
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//=============================== Parameters ==================================
// Lengths and resolutions in each dimension
xsize = 5;
ysize = 5;
zsize = 10;
nx = 5;
ny = 5;
nz = 10;
//=============================================================================

// Create a line in the x-direction of length <xsize>, with <nx> divisions
Point(1) = {-xsize/2, -ysize/2, 0, 0.01};
Point(2) = {xsize/2, -ysize/2, 0, 0.01};
Line(1) = {1, 2};
Transfinite Line(1) = nx+1;

// Extrude split line into meshed square/rectangle
sq = Extrude {0,ysize,0} {Curve{1}; Layers{ny}; Recombine;};

// Extrude square/rectangle into a cuboid
cbd = Extrude {0,0,zsize} {Surface{sq[1]}; Layers{nz}; Recombine;};

// Define physical volume, surfaces for BCs
// Domain
Physical Volume(0) = {cbd[1]};
// Low-x side
Physical Surface(1) = {cbd[5]};
// High-x side
Physical Surface(2) = {cbd[3]};
// Low-y side
Physical Surface(3) = {cbd[2]};
// High-y side
Physical Surface(4) = {cbd[4]};
// Low-z side
Physical Surface(5) = {sq[1]};
// High-z side
Physical Surface(6) = {cbd[0]};
103 changes: 103 additions & 0 deletions examples/H3LAPD/2Din3D-hw_fluid-only/hw.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
<?xml version="1.0" encoding="utf-8" ?>
<NEKTAR>

<COLLECTIONS DEFAULT="MatrixFree" />

<!--
The composite index for the domain is expected to be 0 if the mesh was generated from the included .geo file
-->
<EXPANSIONS>
<E COMPOSITE="C[0]" NUMMODES="7" TYPE="MODIFIED" FIELDS="ne,w,phi" />
</EXPANSIONS>

<CONDITIONS>
<SOLVERINFO>
<I PROPERTY="EQTYPE" VALUE="2Din3DHW" />
<I PROPERTY="AdvectionType" VALUE="WeakDG" />
<I PROPERTY="Projection" VALUE="DisContinuous" />
<I PROPERTY="TimeIntegrationMethod" VALUE="ClassicalRungeKutta4" />
<I PROPERTY="UpwindType" VALUE="Upwind" />
</SOLVERINFO>

<GLOBALSYSSOLNINFO>
<V VAR="ne,w,phi,ne_src">
<I PROPERTY="GlobalSysSoln" VALUE="IterativeStaticCond" />
<I PROPERTY="IterativeSolverTolerance" VALUE="1e-6"/>
</V>
</GLOBALSYSSOLNINFO>

<PARAMETERS>
<!-- Timestepping and output options -->
<P> TimeStep = 0.00125 </P>
<P> NumSteps = 32000 </P>
<P> TFinal = NumSteps*TimeStep </P>
<P> IO_InfoSteps = NumSteps/1600 </P>
<P> IO_CheckSteps = NumSteps/160 </P>
<!-- Magnetic field strength -->
<P> Bxy = 1.0 </P>
<!-- d22 Coeff for Helmholtz solve -->
<P> d22 = 0.0 </P>
<!-- HW params -->
<P> HW_alpha = 0.1 </P>
<P> HW_kappa = 3.5 </P>
<!-- Scaling factor for ICs -->
<P> s = 0.5 </P>
<!-- No particles -->
<P> num_particles_total = 0 </P>
</PARAMETERS>

<VARIABLES>
<V ID="0"> ne </V>
<V ID="1"> w </V>
<V ID="2"> phi </V>
</VARIABLES>

<BOUNDARYREGIONS>
<B ID="0"> C[1] </B> <!-- Low x -->
<B ID="1"> C[2] </B> <!-- High x -->
<B ID="2"> C[3] </B> <!-- Low y -->
<B ID="3"> C[4] </B> <!-- High y -->
<B ID="4"> C[5] </B> <!-- Low-z end -->
<B ID="5"> C[6] </B> <!-- High-z end -->
</BOUNDARYREGIONS>

<!-- Periodic conditions for all fields on all boundaries -->
<BOUNDARYCONDITIONS>
<REGION REF="0">
<P VAR="ne" VALUE="[1]" />
<P VAR="w" VALUE="[1]" />
<P VAR="phi" VALUE="[1]" />
</REGION>
<REGION REF="1">
<P VAR="ne" VALUE="[0]" />
<P VAR="w" VALUE="[0]" />
<P VAR="phi" VALUE="[0]" />
</REGION>
<REGION REF="2">
<P VAR="ne" VALUE="[3]" />
<P VAR="w" VALUE="[3]" />
<P VAR="phi" VALUE="[3]" />
</REGION>
<REGION REF="3">
<P VAR="ne" VALUE="[2]" />
<P VAR="w" VALUE="[2]" />
<P VAR="phi" VALUE="[2]" />
</REGION>
<REGION REF="4">
<P VAR="ne" VALUE="[5]" />
<P VAR="w" VALUE="[5]" />
<P VAR="phi" VALUE="[5]" />
</REGION>
<REGION REF="5">
<P VAR="ne" VALUE="[4]" />
<P VAR="w" VALUE="[4]" />
<P VAR="phi" VALUE="[4]" />
</REGION>
</BOUNDARYCONDITIONS>
<FUNCTION NAME="InitialConditions">
<E VAR="ne" DOMAIN="0" VALUE="6*exp((-x*x-y*y)/(s*s))*sin(4*PI*z/10)" />
<E VAR="w" DOMAIN="0" VALUE="(4*exp((-x*x-y*y)/(s*s))*(-s*s+x*x+y*y)/s^4)*sin(4*PI*z/10)" />
<E VAR="phi" DOMAIN="0" VALUE="exp(-(x*x+y*y)/(s*s))*sin(4*PI*z/10)" />
</FUNCTION>
</CONDITIONS>
</NEKTAR>
1 change: 1 addition & 0 deletions examples/H3LAPD/2Din3D-hw_fluid-only/run_cmd_template.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
mpirun -np <NMPI> <SOLVER_EXEC> hw.xml cuboid.xml
31 changes: 31 additions & 0 deletions examples/H3LAPD/cuboid/cuboid.geo
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
//=============================== Parameters ==================================
// Lengths and resolutions in each dimension
xsize = 5;
ysize = 5;
zsize = 10;
nx = 5;
ny = 5;
nz = 10;
//=============================================================================

// Create a line in the x-direction of length <xsize>, with <nx> divisions
Point(1) = {-xsize/2, -ysize/2, 0, 0.01};
Point(2) = {xsize/2, -ysize/2, 0, 0.01};
Line(1) = {1, 2};
Transfinite Line(1) = nx+1;

// Extrude split line into meshed square
sq = Extrude {0,ysize,0} {Curve{1}; Layers{ny}; Recombine;};

// Extrude square into a cuboid
cbd = Extrude {0,0,zsize} {Surface{sq[1]}; Layers{nz}; Recombine;};

// Define physical volume, surfaces for BCs
// Domain
Physical Volume(0) = {cbd[1]};
// Long sides, parallel to z-axis
Physical Surface(1) = {cbd[2],cbd[3],cbd[4],cbd[5]};
// Low-z square
Physical Surface(2) = {sq[1]};
// High-z square
Physical Surface(3) = {cbd[0]};
Loading

0 comments on commit 0d3c970

Please sign in to comment.