[WIP] Overlapping overset grids tutorial

doc/source/tutorials/overlapping_overset_grids.rst

@@ -1,4 +1,127 @@
 .. _overlapping_overset_grids:
 
 Overlapping Overset Grids
-=========================
+=========================
+
+This tutorial will describe how to run a case with two overlapping meshes (NekNek) from scratch.
+We illustrate this by using the :ref:`perhill` case.
+If you are not familiar with *NekRS*, we strongly recommend you begin with the periodic hill tutorial first!
+
+The three key steps to running a case with NekNek are: 
+
+  1. Setting up the mesh with appropriate boundary conditions for the overlapping interface.
+  2. Specifying parameters to control stability and accuracy for the overlapping-Schwarz iterations.
+  3. Modifying ``velocityDirichletConditions`` to read Dirichlet boundary data for the overlapping interfaces.
+
+Pre-processing
+..............
+
+The pre-processing steps for a NekNek calculation differ from a mono-domain NekRS calculation only in terms of ensuring that the appropriate boundary condition ``int`` is specified at the overlapping interface.
+This boundary condition serves as the flag to let the solver know that Dirichlet data for these boundary conditions must come from interpolation of the data in the overlapping mesh.
+
+Mesh generation
+...............
+
+In this tutorial, we generate two meshes to span the entire domain, an upper and a lower mesh. 
+The lower mesh is generated by ``genbox`` with the following input file:
+
+.. literalinclude:: overlapping_overset_grids/lower.box
+
+and the upper mesh is generated by ``genbox`` with the following input file:
+
+.. literalinclude:: overlapping_overset_grids/upper.box
+
+Note: the lower domain spans :math:`y \in [0,2.5]` and the upper domain spans :math:`y \in [2.25,3]`. We also use ``int`` for the boundary surfaces that overlap the other domain. 
+Create two directories, ``lower`` and ``upper`` representing the two meshes then generate them by running ``genbox`` for each case to obtain ``upper/upper.re2`` and ``lower/lower.re2``.
+
+.. code-block:: console
+
+  $ genbox <<< upper.box
+  $ mv box.re2 upper/upper.re2
+  $ genbox <<< lower.box
+  $ mv box.re2 lower/upper.re2
+
+:Reminder:
+  ``genbox`` produces a file named ``box.re2``, so you will need to rename the files between generating the separate parts.
+
+
+User Defined Functions file (.udf)
+..................................
+
+As before we need to create a UDF file, to get started create a file ``hillnn.udf`` in the current directory.
+
+Modify mesh and add forcing to the flow
+---------------------------------------
+
+As for the mono-domain periodic hill case, we modify the mesh defining a function ``UDF_Setup`` and adding:
+
+.. literalinclude:: overlapping_overset_grids/hillnn.udf
+   :language: c++
+   :lines: 38,43-48,58
+
+.. _fig:hillnn_mesh:
+
+.. figure:: overlapping_overset_grids/hillnn_mesh.png
+    :align: center
+    :figclass: align-center
+    :alt: neknek_mesh
+
+    Modified box mesh graded
+
+Currently, applying a constant mass flux with ``param(54)`` and ``param(55)`` is **not** supported with overlapping overset grids. 
+For this case, we drive the flow using a constant acceleration term in ``userf`` and using this within ``UDF_Setup``:
+
+
+.. literalinclude:: overlapping_overset_grids/hillnn.udf
+   :language: c++
+   :lines: 28-36,38,43-48,56,57,58
+   :emphasize-lines: 1-8,18
+
+Specifying NekNek and control parameters
+----------------------------------------
+
+The control parameters for this case are the same as that for the mono-domain periodic hill case and impact the
+stability and accuracy of the calculation. In addition we also specify the extrapolation order for the boundary
+conditions of the overlapping interface.
+
+Create a parameter file for the lower mesh ``lower/lower.par``:
+
+.. literalinclude:: overlapping_overset_grids/lower/lower.par
+   :language: ini
+
+and another for the upper mesh ``upper/upper.par``:
+
+.. literalinclude:: overlapping_overset_grids/upper/upper.par
+   :language: ini
+
+here the extrapolation is set by the variable ``boundaryEXTOrder``.
+For this tutorial we have set our polynomial order to be :math:`N=7` using the `polynomialOrder` variable.
+
+Defining the session
+--------------------
+
+The last file we require is a session file which will combine all of the components into a single
+NekRS run, create a new file ``hillnn.sess`` and add the following:
+
+.. literalinclude:: overlapping_overset_grids/hillnn.sess
+
+the lines being in the form ``<relative-path-to-component>:<number-of-mpi-ranks>;``.
+
+Compilation
+-----------
+
+You should now have the following files required to compile and run this case:
+
+  * :download:`hillnn.udf <overlapping_overset_grids/hillnn.udf>`
+  * :download:`lower/lower.par <overlapping_overset_grids/lower/lower.par>`
+  * :download:`upper/upper.par <overlapping_overset_grids/upper/upper.par>`
+
+If for some reason you encountered an insurmountable error and were unable to generate any of the required files, you may use the provided links to download them.
+After confirming that you have all eight, you are now ready to compile
+
+.. code-block:: console
+
+  $ nekbmpi hillnn.sess 2
+
+Note the number of MPI ranks should match the total of those defined in the session file, in this case ``2``.
+If all works properly, upon compilation the executable ``nek5000`` will be generated.

doc/source/tutorials/overlapping_overset_grids/hillnn.sess

@@ -0,0 +1,2 @@
+lower/lower:1;
+upper/upper:1;

doc/source/tutorials/overlapping_overset_grids/hillnn.udf

@@ -0,0 +1,60 @@
+static int P_INTERPOLATED_BC_ID;
+
+#ifdef __okl__
+void codedFixedValueVelocity(bcData *bc) {
+  if(bc->id == p_interpolated_bc_id) {
+    bc->u = bc->uinterp;
+    bc->v = bc->vinterp;
+    bc->w = bc->winterp;
+  } else {
+    bc->u = 0.0;
+    bc->v = 0.0;
+    bc->w = 0.0;
+  }
+}
+#endif
+
+void UDF_LoadKernels(deviceKernelProperties& kernelInfo)
+{
+   setupAide& options = platform->options;
+   kernelInfo.define("p_interpolated_bc_id") = P_INTERPOLATED_BC_ID;
+}
+
+void UDF_Setup0(MPI_Comm comm, setupAide &options)
+{
+  platform->par->extract("casedata", "interpolated_bc_id", P_INTERPOLATED_BC_ID);
+}
+
+void userf(double time) {
+  const dfloat ffx{0.052};
+
+  // Get x component of non-linear momentum array
+  // using offset of 0
+  auto o_FUx = nrs->o_NLT + 0 * nrs->fieldOffset;
+  platform->linAlg->fill(nrs->meshV->Nlocal, ffx, o_FUx);
+}
+
+
+void UDF_Setup() {
+  auto mesh = nrs->meshV;
+
+  const dfloat A{4.5}, B{3.5}, C{1./6};
+
+  // mesh modification
+  for(int i{0}; i < mesh -> Nlocal; ++i) {
+    const dfloat argx{B * (std::abs(mesh->x[i] - A) - B)};
+    const dfloat A1{C * (1. + std::tanh(argx))};
+    mesh->y[i] = mesh->y[i] + A1 * (3. - mesh->y[i]);
+  }
+
+  // initial conditions
+  for(int i{0}; i < mesh -> Nlocal; ++i) {
+    nrs->U[i + 0 * nrs->fieldOffset] = 1.0;
+    nrs->U[i + 1 * nrs->fieldOffset] = 0.0;
+    nrs->U[i + 2 * nrs->fieldOffset] = 0.0;
+  }
+
+  nrs->userVelocitySource = &userf;
+}
+
+void UDF_ExecuteStep(double time, int tstep) {}

doc/source/tutorials/overlapping_overset_grids/lower/lower.box

@@ -0,0 +1,14 @@
+-3                     spatial dimension (will create box.re2)
+1                      number of fields
+#
+#    comments: two dimensional periodic hill
+#
+#========================================================
+#
+Box
+-22 5 -1                                 Nelx  Nely  Nelz
+0.0 9.0 1.                                x0 x1 ratio
+0.0 0.1 0.25 0.5 1.5 2.5                  y0 y1 ratio
+0.0 1.0 1.0                               z0 z1 ratio
+P  ,P  ,W  ,int,P  ,P           BC's:  (cbx0, cbx1, cby0, cby1, cbz0, cbz1)
+

doc/source/tutorials/overlapping_overset_grids/lower/lower.par

@@ -0,0 +1,27 @@
+[GENERAL]
+polynomialOrder = 7
+stopAt = endTime
+endTime  = 10
+variableDT = yes
+dt = targetCFL = 0.4 + initial=1e-1
+timeStepper = bdf2
+checkpointControl = steps
+checkpointInterval = 20
+udf = "../hillnn.udf"
+
+[PROBLEMTYPE]
+equation = navierstokes
+
+[VELOCITY]
+residualTol = 1e-8
+density = 1
+viscosity = -100
+
+[NEKNEK]
+boundaryEXTOrder = 2
+
+[MESH]
+file = "lower.re2"
+
+[CASEDATA]
+interpolated_bc_id = 3

doc/source/tutorials/overlapping_overset_grids/upper/upper.box

@@ -0,0 +1,13 @@
+-3 spatial dimension (will create box.re2)
+1                      number of fields
+#
+#    comments: two dimensional periodic hill
+#
+#========================================================
+#
+Box                                       
+-21 3 -1              Nelx  Nely  Nelz
+0.0 9.0 1.            x0 x1 ratio
+2.25 2.75 2.9 3.0     y0 y1 ratio
+0.0 1.0 1.0           z0 z1 ratio
+P  ,P  ,int,W  ,P  ,P    BC's:  (cbx0, cbx1, cby0, cby1, cbz0,)

doc/source/tutorials/overlapping_overset_grids/upper/upper.par

@@ -0,0 +1,27 @@
+[GENERAL]
+polynomialOrder = 7
+stopAt = endTime
+endTime  = 10
+variableDT = yes
+dt = targetCFL = 0.4 + initial=1e-1
+timeStepper = bdf2
+checkpointControl = steps
+checkpointInterval = 20
+udf = "../hillnn.udf"
+
+[PROBLEMTYPE]
+equation = navierstokes
+
+[VELOCITY]
+residualTol = 1e-8
+density = 1
+viscosity = -100
+
+[NEKNEK]
+boundaryEXTOrder = 2
+
+[MESH]
+file = "upper.re2"
+
+[CASEDATA]
+interpolated_bc_id = 3