diff --git a/doc/source/user_guide/_user_guide.rst b/doc/source/user_guide/_user_guide.rst index 1b0f496ef..a62922446 100644 --- a/doc/source/user_guide/_user_guide.rst +++ b/doc/source/user_guide/_user_guide.rst @@ -8,16 +8,14 @@ user perspective (I.E. what will be applicable to most people using nekRS). If you wish to view information relevant to developing nekRS and contributing changes please look at the :ref:`developer`. -The image below shows the high level dataflow of using nekRS. - -.. image:: ../_static/img/overview.svg - .. toctree:: installing + meshing + boundary_initial_conditions + models_properties + data_structures + postprocessing case - commonly_used_variables running - detailed_usage - plugins debugging diff --git a/doc/source/user_guide/boundary_initial_conditions.rst b/doc/source/user_guide/boundary_initial_conditions.rst new file mode 100644 index 000000000..71920f2ba --- /dev/null +++ b/doc/source/user_guide/boundary_initial_conditions.rst @@ -0,0 +1,177 @@ +.. _boundary_initial_conditions: + +Boundary and Initial conditions +=============================== + +.. _setting_ICs: + +Setting Initial Conditions with ``UDF_Setup`` +--------------------------------------------- + +This section provides an example for setting initial conditions with the +``UDF_Setup`` user-defined function that was introduced on the :ref:`Input Files ` page. +The following code snippet sets initial conditions for all three components of +velocity, the pressure, and two passive scalars. You may not necessarily have all of these +variables in your model - this example is just intended to cover all possibilities. + +For this example, the initial conditions are +:math:`V_x=sin(x)cos(y)cos(z)`, :math:`V_y=-cos(x)sin(y)cos(z)`, and :math:`V_z=0` +for the three components of velocity; +:math:`P=101325` for the pressure; and :math:`\phi_0=573` and :math:`\phi_1=100+z` for the +two passive scalars indicated generically as :math:`\phi_0` and :math:`\phi_1`. + +.. note:: + + If present, the temperature variable is represented internally in nekRS as a passive + scalar, since the form of the equation is the same as those solver for other passive + scalars such as chemical concentration. + +Because these initial conditions will +be a function of space, we must first obtain the mesh information, for which we +use the ``nrs->mesh`` pointer. All solution fields are stored in nekRS in terms of the +quadrature points (also referred to as the :term:`GLL` points). So, we will apply +the initial conditions by looping over all of these quadrature points, which for +the current :term:`MPI` process is equal to ``mesh->Np``, or the number of quadrature +points per element, and ``mesh->Nelements``, the number of elements on this process. + +Next, we can get the :math:`x`, :math:`y`, and :math:`z` coordinates for the current +quadrature point with the ``x``, ``y``, and ``z`` pointers on the ``mesh`` object. +Finally, we programmatically set initial conditions for the solution fields. ``nrs->U`` +is a single array that holds all three components of velocity; the ``nrs->fieldOffset`` +variable is used to shift between components in this array. ``nrs->P`` represents the +pressure. Finally, ``nrs->S`` is a single array that holds all of the passive scalars. +Similar to the offset performed to index into the velocity array, the +``nrs->cds->fieldOffset`` variable is used to shift between components in the ``nrs->S`` +array. + +.. code-block:: cpp + + void UDF_Setup(nrs_t* nrs) + { + mesh_t* mesh = nrs->mesh; + int num_quadrature_points = mesh->Np * mesh->Nelements; + + for (int n = 0; n < num_quadrature_points; n++) { + float x = mesh->x[n]; + float y = mesh->y[n]; + float z = mesh->z[n]; + + nrs->U[n + 0 * nrs->fieldOffset] = sin(x) * cos(y) * cos(z); + nrs->U[n + 1 * nrs->fieldOffset] = -cos(x) * sin(y) * cos(z); + nrs->U[n + 2 * nrs->fieldOffset] = 0; + + nrs->P[n] = 101325.0; + + nrs->S[n + 0 * nrs->cds->fieldOffset] = 573.0; + nrs->S[n + 1 * nrs->cds->fieldOffset] = 100.0 + z; + } + } + + +Periodic Boundary Conditions +---------------------------- + +NekRS supports periodic boundary conditions. To set up a periodic case, first +you need to run ``exo2nek`` to establish the pairings between the periodic sidesets. +All this information will be prompted on the screen by ``exo2nek``; +You will provide the sideset IDs of the periodic boundaries, a search tolerance +for identifying paired sides, and a translation vector that points from one of the +paired sidesets to the other. For example, if you want to have one periodic surface +that is a :math:`z`-plane at :math:`z=-1.0` that is paired to another :math:`z`-plane +at :math:`z=1.0`, the translation vector would be :math:`(0.0, 0.0, 2.0)`. + +After generating the mesh, you then need to modify the sideset IDs inside the +``usrdat2`` function. Any boundary that is now periodic, you need to set +``boundaryID(ifc,iel)`` to 0. For all non-periodic boundaries, you need to +"renormalize" those boundaries to "begin counting" from 1. For example, consider +an original (non-periodic) mesh with sidesets 1, 2, 3, and 4. You run ``exo2nek`` +and set up sidesets 2 and 3 as periodic. Then, in the code snippet below, you +would reset sidesets 2 and 3 in ``boundaryID`` to zero. For the remaining two +boundaries (originally 1 and 4), you need to renormalized those to boundaries +1 and 2 (because NekRS wants the boundaries to be ordered sequentially beginning +from 1). + +.. code-block:: + + subroutine usrdat2 + include 'SIZE' + include 'TOTAL' + integer e,f + + n = lx1*ly1*lz1*nelv + nxz = nx1*nz1 + nface = 2*ldim + + do iel=1,nelt + do ifc=1,2*ndim + if (boundaryID(ifc,iel).eq. 1) then + boundaryID(ifc,iel) = 1 + else if (boundaryID(ifc,iel).eq. 2) then + boundaryID(ifc,iel) = 0 + else if (boundaryID(ifc,iel) .eq. 3) then + boundaryID(ifc,iel) = 0 + else if (boundaryID(ifc,iel) .eq. 4) then + boundaryID(ifc,iel) = 2 + endif + enddo + enddo + + return + end + +Then, in the other case files, you do not need any boundary conditions for the periodic +boundaries - for instance, in the ``.par`` file for this example, the boundary conditions +set in ``boundaryTypeMap`` would only display the boundary conditions for the non-periodic +boundaries (and similarly in the ``.oudf`` file). Finally, in order to enforce periodic +flow with a constant flow rate, specify the ``constFlowRate`` parameter in the ``.par`` +file, such as + +.. code-block:: + + [GENERAL] + constFlowRate = meanVelocity=1.0 + direction=Z + +Stamping Initial Conditions +--------------------------- + +For many periodic flows, you can save significant computing time by solving the flow equations +on a shorter-height mesh, and then "stamping" that solution onto a full-height mesh (where you +might then be solving for passive scalar transport). NekRS allows you to "stamp" a partial-height +solution onto a full-height mesh using the ``gfldr`` utility. To do so, you simply need to call +the ``gfldr`` function in a loop inside of ``userchk()``. Below, ``nd`` represents the number +of times you want to stamp a short-height solution to obtain the full-height case and ``delta`` +represents the height of one short-height domain. So, the example below would represent +a previous solution (``short.fld``) on a short-height domain of height 62.42, that you want to stamp five times +onto a new mesh that has a height of 312.1. + +.. code-block:: + + subroutine userchk() + include 'SIZE' + include 'TOTAL' + + + ntot = lx1*ly1*lz1*nelv + + do nd = 0,5 + + delta = 62.421731741003335 + + do i = 1,ntot + zm1(i,1,1,1) = zm1(i,1,1,1) - delta*nd + enddo + + call gfldr('short.fld') + + do i = 1,ntot + zm1(i,1,1,1) = zm1(i,1,1,1) + delta*nd + enddo + + enddo + + return + end + +Velocity Recycling Plugin +------------------------- + diff --git a/doc/source/user_guide/case.rst b/doc/source/user_guide/case.rst index 6a574ada0..bfa178859 100644 --- a/doc/source/user_guide/case.rst +++ b/doc/source/user_guide/case.rst @@ -3,25 +3,37 @@ Case files ===================== -This page describes the input file structure and syntax needed to run a nekRS simulation. -A nekRS simulation is referred to as a "case," and at a minimum requires four files to run - +A nekRS simulation is referred to as a "case," which utilises a number of files +which are described in this page. An overview of these are presented in the the +image below . + +.. _fig:case_overview: + +.. figure:: ../_static/img/overview.svg + :align: center + :figclass: align-center + :alt: An overview of nekRS case files + +There are a minimum of three files required to run a case: * Parameter file, with ``.par`` extension * Mesh file, with ``.re2`` extension * User-defined functions for the host, with ``.udf`` extension -* User-defined functions for the device, with ``.oudf`` extension + +With one optional file + +* Trigger file with ``.upd`` extension The "case name" is then the common prefix applied to these files - for instance, a complete input description with a case name of "eddy" would be given by the files -``eddy.par``, ``eddy.re2``, ``eddy.udf``, and ``eddy.oudf``. +``eddy.par``, ``eddy.re2``, ``eddy.udf``, and ``eddy.upd``. The only restrictions on the case name are: * It must be used as the prefix on all simulation files, and * Typical restrictions for naming files for your operating system -The next four sections describe the structure and syntax for each of these four files -for a general case. -Because the :term:`Nek5000` code is a predecessor to +The next four sections describe the structure and syntax for each of these files +for a general case. Because the :term:`Nek5000` code is a predecessor to nekRS, some aspects of the current nekRS input file design are selected to enable faster translation of Nek5000 input files into nekRS input files. Because these Nek5000-based approaches require proficiency in Fortran, the inclusion of several additional input diff --git a/doc/source/user_guide/commonly_used_variables.rst b/doc/source/user_guide/data_structures.rst similarity index 97% rename from doc/source/user_guide/commonly_used_variables.rst rename to doc/source/user_guide/data_structures.rst index fa500a43a..1a5e94c1a 100644 --- a/doc/source/user_guide/commonly_used_variables.rst +++ b/doc/source/user_guide/data_structures.rst @@ -1,7 +1,9 @@ -.. _commonly_used_variables: +.. _data_structures: -Commonly-Used Variables in nekRS -================================ +Data Structures +=============== + +UDF Only?? To become a proficient user of nekRS requires some knowledge of the data structures used to store the mesh, solution fields, and simulation settings. While many @@ -27,6 +29,16 @@ The same data, but accessible on the device, is ``mesh->o_x``. Not all variables are automatically available on both the host and device, but those that are available are indicated with a :math:`\checkmark` in the "Device?" table column. +Platform +-------- + +.. _fig:case_overview: + +.. figure:: ../doxygen/doxygen_html/structplatform__t__coll__graph.png + :align: center + :figclass: align-center + :alt: Diagram of platform elements + Mesh ---- diff --git a/doc/source/user_guide/detailed_usage.rst b/doc/source/user_guide/detailed_usage.rst index 20a15c769..9884f10a6 100644 --- a/doc/source/user_guide/detailed_usage.rst +++ b/doc/source/user_guide/detailed_usage.rst @@ -13,175 +13,6 @@ in the nekRS source code are referenced - a list defining these variables and st is available on the :ref:`Commonly Used Variables ` page for reference. -.. _converting_mesh: - -Converting a Mesh to .re2 Format --------------------------------- - -The most general and flexible approach for creating a mesh is to use commercial meshing software -such as Cubit or Gmsh. After creating the mesh, it must be converted to the ``.re2`` binary format. -The following two sections describe how to convert Exodus and Gmsh meshes into ``.re2`` binary format -with scripts that ship with the Nek5000 dependency. First build these scripts following -the instructions in the :ref:`Building the Nek5000 Tool Scripts ` section. - -Converting an Exodus mesh -""""""""""""""""""""""""" - -To convert from an Exodus format mesh -(for this case, named ``my_mesh.exo``) to the ``.re2`` format, use the ``exo2nek`` script: - -.. code-block:: - - user$ exo2nek - -Then, follow the on-screen prompts associated with the ``exo2nek`` script. -``exo2nek`` will convert all elements in the Exodus mesh (TET6, WEDGE6, HEX8, HEX20) to HEX20 elements and dump into ``.re2`` format. - -Converting a Gmsh mesh -"""""""""""""""""""""" - -To convert from a Gmsh format mesh (for this case, named ``my_mesh.msh``) to the -``.re2`` format, use the ``gmsh2nek`` script: - -.. code-block:: - - user$ gmsh2nek - - Enter mesh dimension: 3 - Input (.msh) file name: my_mesh - -All your mesh should be hexahedral elements. Before exporting from Gmsh, you will need to set the mesh order to 2. -The Gmsh mesh format must also be version 2, ASCII/binary format. If your Gmsh version -shows a pop-up box when exporting the mesh, do *not* click "Save all elements" -or "Save parametric elements". - -.. _cht_mesh: - -Creating a Mesh for Conjugate Heat Transfer -------------------------------------------- - -Mesh generation for conjugate heat transfer requires an additional pre-processing -step before performing other steps of the mesh generation process such as those -described in the :ref:`Converting a Mesh to .re2 Format ` section. -The nekRS approach for conjugate heat transfer is still dependent on legacy limitations -from Nek5000. Unfortunately, you cannot -simply use a standard commercial meshing tool and define fluid and solid -regions according to block IDs - you must individually create the mesh for the fluid and -the solid, and then merge them with the ``pretex`` script. - - -.. _setting_ICs: - -Setting Initial Conditions with ``UDF_Setup`` ---------------------------------------------- - -This section provides an example for setting initial conditions with the -``UDF_Setup`` user-defined function that was introduced on the :ref:`Input Files ` page. -The following code snippet sets initial conditions for all three components of -velocity, the pressure, and two passive scalars. You may not necessarily have all of these -variables in your model - this example is just intended to cover all possibilities. - -For this example, the initial conditions are -:math:`V_x=sin(x)cos(y)cos(z)`, :math:`V_y=-cos(x)sin(y)cos(z)`, and :math:`V_z=0` -for the three components of velocity; -:math:`P=101325` for the pressure; and :math:`\phi_0=573` and :math:`\phi_1=100+z` for the -two passive scalars indicated generically as :math:`\phi_0` and :math:`\phi_1`. - -.. note:: - - If present, the temperature variable is represented internally in nekRS as a passive - scalar, since the form of the equation is the same as those solver for other passive - scalars such as chemical concentration. - -Because these initial conditions will -be a function of space, we must first obtain the mesh information, for which we -use the ``nrs->mesh`` pointer. All solution fields are stored in nekRS in terms of the -quadrature points (also referred to as the :term:`GLL` points). So, we will apply -the initial conditions by looping over all of these quadrature points, which for -the current :term:`MPI` process is equal to ``mesh->Np``, or the number of quadrature -points per element, and ``mesh->Nelements``, the number of elements on this process. - -Next, we can get the :math:`x`, :math:`y`, and :math:`z` coordinates for the current -quadrature point with the ``x``, ``y``, and ``z`` pointers on the ``mesh`` object. -Finally, we programmatically set initial conditions for the solution fields. ``nrs->U`` -is a single array that holds all three components of velocity; the ``nrs->fieldOffset`` -variable is used to shift between components in this array. ``nrs->P`` represents the -pressure. Finally, ``nrs->S`` is a single array that holds all of the passive scalars. -Similar to the offset performed to index into the velocity array, the -``nrs->cds->fieldOffset`` variable is used to shift between components in the ``nrs->S`` -array. - -.. code-block:: cpp - - void UDF_Setup(nrs_t* nrs) - { - mesh_t* mesh = nrs->mesh; - int num_quadrature_points = mesh->Np * mesh->Nelements; - - for (int n = 0; n < num_quadrature_points; n++) { - float x = mesh->x[n]; - float y = mesh->y[n]; - float z = mesh->z[n]; - - nrs->U[n + 0 * nrs->fieldOffset] = sin(x) * cos(y) * cos(z); - nrs->U[n + 1 * nrs->fieldOffset] = -cos(x) * sin(y) * cos(z); - nrs->U[n + 2 * nrs->fieldOffset] = 0; - - nrs->P[n] = 101325.0; - - nrs->S[n + 0 * nrs->cds->fieldOffset] = 573.0; - nrs->S[n + 1 * nrs->cds->fieldOffset] = 100.0 + z; - } - } - -.. _grabbing_user: - -Grabbing User .par Settings ---------------------------- - -nekRS conveniently allows the user to define their own parameters in the ``.par`` file -that can then be accessed in the ``.udf`` functions. This is useful for programmatically -setting boundary conditions, forcing terms, and many other simulation settings. For instance, -suppose that the initial condition for velocity will vary from run to run and is possibly used in several -places in the ``.udf`` functions. Rather than continually edit the ``.udf`` file (which -will require repeated just-in-time compilation), these settings can be set with user-defined -parameters in the ``.par`` file. - -As an example, we will define a parameter named ``initialVelocity`` in the ``VELOCITY`` block. - -.. code-block :: xml - - [VELOCITY] - residualTol = 1e-6 - density = 1.5 - viscosity = 2.4e-4 - boundaryTypeMap = inlet, wall, wall, wall, wall, outlet - initialVelocity = 1.5 - -To access this value in the ``.udf`` functions, call the ``extract(String key, String value, T & destination)`` -function on ``nrs->par`` as follows. - -.. code-block :: cpp - - void UDF_Setup(nrs_t* nrs) - { - double initial_Vz; - nrs->par->extract("velocity", "initialvelocity", initial_Vz); - - mesh_t* mesh = nrs->mesh; - int num_quadrature_points = mesh->Np * mesh->Nelements; - - for (int n = 0; n < num_quadrature_points; n++) { - nrs->U[n + 0 * nrs->fieldOffset] = 0; - nrs->U[n + 1 * nrs->fieldOffset] = 0; - nrs->U[n + 2 * nrs->fieldOffset] = initial_Vz; - } - } - -The extracted user parameter can then be used throughout the ``.udf`` functions, as well -as propagated to the device kernels as described in Section -:ref:`Defining Variables to Access in Device Kernels `. - .. _defining_variables_for_device: Defining Variables to Access in Device Kernels @@ -793,7 +624,7 @@ from the device to the host, use the ``nek_ocopyFrom(double time, int tstep)`` r ``nrs->o_U`` is copied to ``nrs->U``, and so on. This allows you to access the solution on the host as ``nrs->U``, ``nrs->p``, ``nrs->S``, etc. -2. Copy the nekRS solution from the nekRS host arrays to the Nek5000 backend arrays. +1. Copy the nekRS solution from the nekRS host arrays to the Nek5000 backend arrays. If you only want to access the nekRS host side arays such as ``nrs->U``, you can skip the second part by directly using :term:`OCCA` memory copy functions like the following, which @@ -803,135 +634,6 @@ copies from the device array ``nrs->o_U`` to the host array ``nrs->U``. nrs->o_U.copyTo(nrs->U); -.. _writing_output: - -Writing an Output File ----------------------- - -nekRS will automatically write output files according to the ``writeControl`` criterion -set in the ``.par`` file. However, it may be desirable to have finer-grained control of -output writing, such as if you want the solution at a specific time step, but that -time step is not an integer multiple of ``writeInterval``. In this case, you can force -the output file writing to occur by calling the ``outfld(double time, double outputTime)`` -function in the ``nekrs`` namespace. This function performs the following actions: - -1. Copy the nekRS solution from the nekRS device arrays directly to the backend - Nek5000 arrays. -2. Write an output file. - -Note that this function is slightly different from the ``nek_ocopyFrom`` function described -in the :ref:`Copying Device to Host ` section. This function is -solely intended for writing output, so no effort is expended in copying the device -solution into the nekRS host arrays - that step is bypassed, and the device solution is -copied straight into the Nek5000 backend arrays. The ``nek_ocopyFrom`` routine should really -only be used if you require access to the nekRS solution arrays on the host, while the -``outfld`` routine should be used strictly for writing output files. - -By default, nekRS will only write the velocity, pressure, and temperature to an output file. -However, you may have problem-specific fields that you want to view, such as :math:`y^+`. -To write other fields to files, nekRS re-uses the -functions that are used to write the velocity, pressure, and temperature -to write other fields. Note that this imposes limitations on both the dimensionality of fields that -can be output, as well as how they are named in the output files. -For example, suppose you would like to write three fields to a file: - - * ``o_yPlus``, a device array that holds :math:`y^+` values, and - * ``o_Uavg``, a device array that holds a time-averaged velocity field, and - * ``o_rst``, a device array that holds the one component of the Reynolds stress tensor. - -To write these three fields to an output file, use the ``writeFld`` function as follows. -The ``writeFld`` function takes eight arguments, and has a signature -``void writeFld(const char* suf, dfloat t, int coords, int FP64, void* o_u, void* o_p, void* o_s, int NSf)``. -In this example, the first parameter, ``"usr"``, is a three-character -prefix that will determine how the new output file is written. While the velocity, pressure, -and temperatures are written to files named ``case0.f``, where ``case`` is the case -name and ```` is a six-digit number indicating the time step, any additional fields -we will write are written to separate files. So for this example, we will write three fields -to files named ``usrcase0.f``. The next three parameters simply indicate the time -step that is being written, whether coordinates are written, and if the results should be written -in double precision. Next, the three fields that are to be output are provided. The order is very -important - the first of these fields must be of length ``nrs->fieldOffset * nrs->NVfields`` -because it represents a component vector field (this is how velocity is written in the usual output -file). The second of these fields must be of length ``nrs->fieldOffset``, because it represents -a non-component field (this is how pressure is written in the usual output file). Finally, -the third of these fields must be of length ``nrs->cds->fieldOffset * Nscalar``, because it -represents a passive scalar field (this is how the passive scalars are written in the usual -output file). - -.. code-block:: cpp - - void UDF_ExecuteStep(nrs_t* nrs, dfloat time, int tstep) - { - // get o_yPlus, o_Uavg, and o_rst in the scope of this function - - bool coords = true; - bool FP64 = true; - int Nscalar = nrs->cds->Nscalar; - writeFld("usr", time, coords, FP64, &o_Uavg, &o_rst, &o_yPlus, Nscalar); - } - -.. warning:: - - ``writeFld`` can only write data of type ``dfloat``. So, if you want to write an - integer field to a field, you must first convert that data to ``dfloat``. - -nekRS's output system does not have any means by which to understand *what* these fields -represent. Therefore, the names of these fields in the output file will be ``velocity``, -``pressure``, and ``temperature``, even if those names have no relationship to what is -being output. Therefore, for this example, the ``usrcase0.f`` files will -contain the following: - -* ``o_Uavg`` is written to a field named ``velocity`` -* ``o_rst`` is written to a field named ``pressure`` -* ``o_yPlus`` is written to a field named ``temperature`` - -nekRS's output system requires additional maneuvering if you wish to output -more than one of each of each of these three categories of fields. For instance, suppose -you want to output three different fields, ``o_field1``, ``o_field2``, and ``o_field3``, -each of size ``nrs->fieldOffset``. Because only one input argument to ``writeFld`` can have -these dimensions, three separate output files need to be written, and in *each* of these -files, our field of interest is named ``pressure``. To fill the other two field arguments -of the ``writeFld`` function, a void pointer is passed in to indicate that neither of -the other two fields are written. - -.. code-block:: cpp - - void UDF_ExecuteStep(nrs_t* nrs, dfloat time, int tstep) - { - // get o_field1, o_field2, o_field3 in the scope of this function - - bool coords = true; - bool FP64 = true; - int Nscalar = nrs->cds->Nscalar; - occa::memory o_null; - writeFld("fl1", time, coords, FP64, &o_null, &o_field1, &o_null, Nscalar); - writeFld("fl2", time, coords, FP64, &o_null, &o_field2, &o_null, Nscalar); - writeFld("fl3", time, coords, FP64, &o_null, &o_field3, &o_null, Nscalar); - } - -This will write three output files, which contain the following. - -* ``fl1case0.f`` contains ``o_field1``, but named ``pressure`` -* ``fl2case0.f`` contains ``o_field2``, but named ``pressure`` -* ``fl3case0.f`` contains ``o_field3``, but named ``pressure`` - -Visualizing Output Files ------------------------- - -nekRS output files all have the form ``.fld``, where ```` is the case -name and ```` is a five-digit number indicating the number of the output file (each output -file represents a single time step that is output according to the settings for -``writeControl`` and ``writeInterval`` in the ``.par`` file). These output files are in a custom -binary format that requires an additional postprocessing step in order to visualize in Paraview. -In the directory where the case files are located, run the ``visnek`` script: - -.. code-block:: - - user$ visnek case - -which will create a ``case.nek5000`` file that is viewable in Paraview. See -:ref:`Building the Nek5000 Tool Scripts ` for instructions on compiling the ``visnek`` program. - Calculating the Distance to a Wall ---------------------------------- @@ -1038,110 +740,6 @@ within ``UDF_ExecuteStep`` so that the Nek5000 backend will have been called fir nrs->o_usrwrk.copyFrom(wall_distance, n_gll_points * sizeof(dfloat), write_location * nrs->fieldOffset * sizeof(dfloat)); } -Periodic Boundary Conditions ----------------------------- - -NekRS supports periodic boundary conditions. To set up a periodic case, first -you need to run ``exo2nek`` to establish the pairings between the periodic sidesets. -All this information will be prompted on the screen by ``exo2nek``; -You will provide the sideset IDs of the periodic boundaries, a search tolerance -for identifying paired sides, and a translation vector that points from one of the -paired sidesets to the other. For example, if you want to have one periodic surface -that is a :math:`z`-plane at :math:`z=-1.0` that is paired to another :math:`z`-plane -at :math:`z=1.0`, the translation vector would be :math:`(0.0, 0.0, 2.0)`. - -After generating the mesh, you then need to modify the sideset IDs inside the -``usrdat2`` function. Any boundary that is now periodic, you need to set -``boundaryID(ifc,iel)`` to 0. For all non-periodic boundaries, you need to -"renormalize" those boundaries to "begin counting" from 1. For example, consider -an original (non-periodic) mesh with sidesets 1, 2, 3, and 4. You run ``exo2nek`` -and set up sidesets 2 and 3 as periodic. Then, in the code snippet below, you -would reset sidesets 2 and 3 in ``boundaryID`` to zero. For the remaining two -boundaries (originally 1 and 4), you need to renormalized those to boundaries -1 and 2 (because NekRS wants the boundaries to be ordered sequentially beginning -from 1). - -.. code-block:: - - subroutine usrdat2 - include 'SIZE' - include 'TOTAL' - integer e,f - - n = lx1*ly1*lz1*nelv - nxz = nx1*nz1 - nface = 2*ldim - - do iel=1,nelt - do ifc=1,2*ndim - if (boundaryID(ifc,iel).eq. 1) then - boundaryID(ifc,iel) = 1 - else if (boundaryID(ifc,iel).eq. 2) then - boundaryID(ifc,iel) = 0 - else if (boundaryID(ifc,iel) .eq. 3) then - boundaryID(ifc,iel) = 0 - else if (boundaryID(ifc,iel) .eq. 4) then - boundaryID(ifc,iel) = 2 - endif - enddo - enddo - - return - end - -Then, in the other case files, you do not need any boundary conditions for the periodic -boundaries - for instance, in the ``.par`` file for this example, the boundary conditions -set in ``boundaryTypeMap`` would only display the boundary conditions for the non-periodic -boundaries (and similarly in the ``.oudf`` file). Finally, in order to enforce periodic -flow with a constant flow rate, specify the ``constFlowRate`` parameter in the ``.par`` -file, such as - -.. code-block:: - - [GENERAL] - constFlowRate = meanVelocity=1.0 + direction=Z - -Stamping Initial Conditions ---------------------------- - -For many periodic flows, you can save significant computing time by solving the flow equations -on a shorter-height mesh, and then "stamping" that solution onto a full-height mesh (where you -might then be solving for passive scalar transport). NekRS allows you to "stamp" a partial-height -solution onto a full-height mesh using the ``gfldr`` utility. To do so, you simply need to call -the ``gfldr`` function in a loop inside of ``userchk()``. Below, ``nd`` represents the number -of times you want to stamp a short-height solution to obtain the full-height case and ``delta`` -represents the height of one short-height domain. So, the example below would represent -a previous solution (``short.fld``) on a short-height domain of height 62.42, that you want to stamp five times -onto a new mesh that has a height of 312.1. - -.. code-block:: - - subroutine userchk() - include 'SIZE' - include 'TOTAL' - - - ntot = lx1*ly1*lz1*nelv - - do nd = 0,5 - - delta = 62.421731741003335 - - do i = 1,ntot - zm1(i,1,1,1) = zm1(i,1,1,1) - delta*nd - enddo - - call gfldr('short.fld') - - do i = 1,ntot - zm1(i,1,1,1) = zm1(i,1,1,1) + delta*nd - enddo - - enddo - - return - end - .. rubric:: Footnotes diff --git a/doc/source/user_guide/installing.rst b/doc/source/user_guide/installing.rst index ae9a648fd..83b2ddea0 100644 --- a/doc/source/user_guide/installing.rst +++ b/doc/source/user_guide/installing.rst @@ -1,7 +1,7 @@ .. _installing: -Installing nekRS -================ +Installing +========== This page gives a variety of information to help install nekRS. This includes how to acquire & build nekRS appropriately to your environment, as well as some diff --git a/doc/source/user_guide/meshing.rst b/doc/source/user_guide/meshing.rst new file mode 100644 index 000000000..9ad075aa0 --- /dev/null +++ b/doc/source/user_guide/meshing.rst @@ -0,0 +1,60 @@ +.. _meshing: + +Meshing +======= + +.. _converting_mesh: + +Converting a Mesh to .re2 Format +-------------------------------- + +The most general and flexible approach for creating a mesh is to use commercial meshing software +such as Cubit or Gmsh. After creating the mesh, it must be converted to the ``.re2`` binary format. +The following two sections describe how to convert Exodus and Gmsh meshes into ``.re2`` binary format +with scripts that ship with the Nek5000 dependency. First build these scripts following +the instructions in the :ref:`Building the Nek5000 Tool Scripts ` section. + +Converting an Exodus mesh +""""""""""""""""""""""""" + +To convert from an Exodus format mesh +(for this case, named ``my_mesh.exo``) to the ``.re2`` format, use the ``exo2nek`` script: + +.. code-block:: + + user$ exo2nek + +Then, follow the on-screen prompts associated with the ``exo2nek`` script. +``exo2nek`` will convert all elements in the Exodus mesh (TET6, WEDGE6, HEX8, HEX20) to HEX20 elements and dump into ``.re2`` format. + +Converting a Gmsh mesh +"""""""""""""""""""""" + +To convert from a Gmsh format mesh (for this case, named ``my_mesh.msh``) to the +``.re2`` format, use the ``gmsh2nek`` script: + +.. code-block:: + + user$ gmsh2nek + + Enter mesh dimension: 3 + Input (.msh) file name: my_mesh + +All your mesh should be hexahedral elements. Before exporting from Gmsh, you will need to set the mesh order to 2. +The Gmsh mesh format must also be version 2, ASCII/binary format. If your Gmsh version +shows a pop-up box when exporting the mesh, do *not* click "Save all elements" +or "Save parametric elements". + +.. _cht_mesh: + +Creating a Mesh for Conjugate Heat Transfer +------------------------------------------- + +Mesh generation for conjugate heat transfer requires an additional pre-processing +step before performing other steps of the mesh generation process such as those +described in the :ref:`Converting a Mesh to .re2 Format ` section. +The nekRS approach for conjugate heat transfer is still dependent on legacy limitations +from Nek5000. Unfortunately, you cannot +simply use a standard commercial meshing tool and define fluid and solid +regions according to block IDs - you must individually create the mesh for the fluid and +the solid, and then merge them with the ``pretex`` script. \ No newline at end of file diff --git a/doc/source/user_guide/plugins.rst b/doc/source/user_guide/models_properties.rst similarity index 96% rename from doc/source/user_guide/plugins.rst rename to doc/source/user_guide/models_properties.rst index 0bb6d4ea2..d2c54684c 100644 --- a/doc/source/user_guide/plugins.rst +++ b/doc/source/user_guide/models_properties.rst @@ -1,5 +1,7 @@ -Plugins -======= +.. _models_properties: + +Models and Physical properties +============================== nekRS contains several "plugins" that provide both physics models and postprocessing capabilities. nekRS's :term:`RANS` and low-Mach models, for instance, are provided as @@ -9,13 +11,16 @@ modifications to the ``.udf`` files. Before reading this page, first consult :ref:`User-Defined Host Functions (.udf) ` so that you have the necessary background on each of the ``.udf`` functions that will be discussed. -.. _rans_plugin: +Turbulence models +----------------- + +.. _rans_model: -RANS :math:`k`-:math:`\tau` Plugin ----------------------------------- +RANS :math:`k`-:math:`\tau` Model +""""""""""""""""""""""""""""""""" The :term:`RANS` :math:`k`-:math:`\tau` plugin is available in the ``src/plugins/RANSktau.hpp`` -header file. In order to add the :math:`k`-:math:`\tau` model to your case, you need +header file. In order to add the :math:`k`-:math:`\tau` model to youddisplr case, you need to include this file in your ``.udf`` file and manually add all the requisite parts of the :math:`k`-:math:`\tau` methodology. Unless otherwise noted, all code snippets in this section are placed in the ``.udf`` file. @@ -34,7 +39,8 @@ following sections then each describe a step in the :term:`RANS` model setup usi .. _kernels: Add the Physics Kernels -""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^ + The calculations performed to add contributions to the residuals occur within :term:`OCCA` kernels. In order to add the :term:`RANS` equations, the corresponding physics kernels must first be included. The :term:`RANS` kernels are added to by @@ -58,7 +64,8 @@ The ``RANSKtau::buildKernel`` function performs two main actions - .. _rans_props: Add the Closure Properties Calculation -"""""""""""""""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Next, add the function that will update the properties used in the governing equations. An example is shown in :ref:`Setting Custom Properties ` for setting custom user-defined properties for a laminar flow scenario. The necessary steps to add @@ -112,7 +119,8 @@ The ``RANSktau::updateProperties`` function performs two main actions: :math:`\mu+\mu_T/\sigma_k`, and :math:`\mu+\mu_T/\sigma_\tau`, respectively. Add the Source Terms Calculation -"""""""""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + The same passive scalar infrastructure that is used to solve the energy conservation equation is used to solve the :math:`k` and :math:`\tau` passive scalar equations. However, these equations clearly have different forms - therefore, we need to explicitly @@ -153,7 +161,8 @@ in the ``.udf`` file: } Add the Turbulent Prandtl Number -"""""""""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + For cases with passive scalar equations, you must manually add the additional component to the diffusivity, :math:`\mu_T/Pr_T`. This is done in the function pointer to be the ``udf.properties`` function pointer *after* @@ -232,7 +241,7 @@ the diffusion coefficient in the temperature passive scalar equation as k+\frac{\mu_T}{Pr_T}C_p Initialize the RANS Solve -""""""""""""""""""""""""" +^^^^^^^^^^^^^^^^^^^^^^^^^ Finally, the last step to initialize the :term:`RANS` solve is to call the ``RANSktau::setup`` function. This function has signature @@ -278,12 +287,5 @@ then the :math:`k` scalar should be positioned as the second scalar, and ``ifld scalar. Be sure to order the scalars in the input file to respect this assumption. -Low-Mach Plugin ---------------- - -Turbulence Statistics Plugin ----------------------------- - -Velocity Recycling Plugin -------------------------- - +Low-Mach Model +-------------- diff --git a/doc/source/user_guide/postprocessing.rst b/doc/source/user_guide/postprocessing.rst new file mode 100644 index 000000000..7058c0020 --- /dev/null +++ b/doc/source/user_guide/postprocessing.rst @@ -0,0 +1,138 @@ +.. _postprocessing: + +Postprocessing +============== + +.. _writing_output: + +Writing an Output File - using UDF_ExecuteStep +---------------------------------------------- + +nekRS will automatically write output files according to the ``writeControl`` criterion +set in the ``.par`` file. However, it may be desirable to have finer-grained control of +output writing, such as if you want the solution at a specific time step, but that +time step is not an integer multiple of ``writeInterval``. In this case, you can force +the output file writing to occur by calling the ``outfld(double time, double outputTime)`` +function in the ``nekrs`` namespace. This function performs the following actions: + +1. Copy the nekRS solution from the nekRS device arrays directly to the backend + Nek5000 arrays. +2. Write an output file. + +Note that this function is slightly different from the ``nek_ocopyFrom`` function described +in the :ref:`Copying Device to Host ` section. This function is +solely intended for writing output, so no effort is expended in copying the device +solution into the nekRS host arrays - that step is bypassed, and the device solution is +copied straight into the Nek5000 backend arrays. The ``nek_ocopyFrom`` routine should really +only be used if you require access to the nekRS solution arrays on the host, while the +``outfld`` routine should be used strictly for writing output files. + +By default, nekRS will only write the velocity, pressure, and temperature to an output file. +However, you may have problem-specific fields that you want to view, such as :math:`y^+`. +To write other fields to files, nekRS re-uses the +functions that are used to write the velocity, pressure, and temperature +to write other fields. Note that this imposes limitations on both the dimensionality of fields that +can be output, as well as how they are named in the output files. +For example, suppose you would like to write three fields to a file: + + * ``o_yPlus``, a device array that holds :math:`y^+` values, and + * ``o_Uavg``, a device array that holds a time-averaged velocity field, and + * ``o_rst``, a device array that holds the one component of the Reynolds stress tensor. + +To write these three fields to an output file, use the ``writeFld`` function as follows. +The ``writeFld`` function takes eight arguments, and has a signature +``void writeFld(const char* suf, dfloat t, int coords, int FP64, void* o_u, void* o_p, void* o_s, int NSf)``. +In this example, the first parameter, ``"usr"``, is a three-character +prefix that will determine how the new output file is written. While the velocity, pressure, +and temperatures are written to files named ``case0.f``, where ``case`` is the case +name and ```` is a six-digit number indicating the time step, any additional fields +we will write are written to separate files. So for this example, we will write three fields +to files named ``usrcase0.f``. The next three parameters simply indicate the time +step that is being written, whether coordinates are written, and if the results should be written +in double precision. Next, the three fields that are to be output are provided. The order is very +important - the first of these fields must be of length ``nrs->fieldOffset * nrs->NVfields`` +because it represents a component vector field (this is how velocity is written in the usual output +file). The second of these fields must be of length ``nrs->fieldOffset``, because it represents +a non-component field (this is how pressure is written in the usual output file). Finally, +the third of these fields must be of length ``nrs->cds->fieldOffset * Nscalar``, because it +represents a passive scalar field (this is how the passive scalars are written in the usual +output file). + +.. code-block:: cpp + + void UDF_ExecuteStep(nrs_t* nrs, dfloat time, int tstep) + { + // get o_yPlus, o_Uavg, and o_rst in the scope of this function + + bool coords = true; + bool FP64 = true; + int Nscalar = nrs->cds->Nscalar; + writeFld("usr", time, coords, FP64, &o_Uavg, &o_rst, &o_yPlus, Nscalar); + } + +.. warning:: + + ``writeFld`` can only write data of type ``dfloat``. So, if you want to write an + integer field to a field, you must first convert that data to ``dfloat``. + +nekRS's output system does not have any means by which to understand *what* these fields +represent. Therefore, the names of these fields in the output file will be ``velocity``, +``pressure``, and ``temperature``, even if those names have no relationship to what is +being output. Therefore, for this example, the ``usrcase0.f`` files will +contain the following: + +* ``o_Uavg`` is written to a field named ``velocity`` +* ``o_rst`` is written to a field named ``pressure`` +* ``o_yPlus`` is written to a field named ``temperature`` + +nekRS's output system requires additional maneuvering if you wish to output +more than one of each of each of these three categories of fields. For instance, suppose +you want to output three different fields, ``o_field1``, ``o_field2``, and ``o_field3``, +each of size ``nrs->fieldOffset``. Because only one input argument to ``writeFld`` can have +these dimensions, three separate output files need to be written, and in *each* of these +files, our field of interest is named ``pressure``. To fill the other two field arguments +of the ``writeFld`` function, a void pointer is passed in to indicate that neither of +the other two fields are written. + +.. code-block:: cpp + + void UDF_ExecuteStep(nrs_t* nrs, dfloat time, int tstep) + { + // get o_field1, o_field2, o_field3 in the scope of this function + + bool coords = true; + bool FP64 = true; + int Nscalar = nrs->cds->Nscalar; + occa::memory o_null; + writeFld("fl1", time, coords, FP64, &o_null, &o_field1, &o_null, Nscalar); + writeFld("fl2", time, coords, FP64, &o_null, &o_field2, &o_null, Nscalar); + writeFld("fl3", time, coords, FP64, &o_null, &o_field3, &o_null, Nscalar); + } + +This will write three output files, which contain the following. + +* ``fl1case0.f`` contains ``o_field1``, but named ``pressure`` +* ``fl2case0.f`` contains ``o_field2``, but named ``pressure`` +* ``fl3case0.f`` contains ``o_field3``, but named ``pressure`` + +.. _vis_output: + +Visualizing Output Files +------------------------ + +nekRS output files all have the form ``.fld``, where ```` is the case +name and ```` is a five-digit number indicating the number of the output file (each output +file represents a single time step that is output according to the settings for +``writeControl`` and ``writeInterval`` in the ``.par`` file). These output files are in a custom +binary format that requires an additional postprocessing step in order to visualize in Paraview. +In the directory where the case files are located, run the ``visnek`` script: + +.. code-block:: + + user$ visnek case + +which will create a ``case.nek5000`` file that is viewable in Paraview. See +:ref:`Building the Nek5000 Tool Scripts ` for instructions on compiling the ``visnek`` program. + +Turbulence Statistics +""""""""""""""""""""" \ No newline at end of file diff --git a/doc/source/user_guide/running.rst b/doc/source/user_guide/running.rst index 171367c47..e7407e6d5 100644 --- a/doc/source/user_guide/running.rst +++ b/doc/source/user_guide/running.rst @@ -1,7 +1,7 @@ .. _running: -Running nekRS -============= +Running +======= This page gives information on how to run nekRS has been installed (see :ref:`installing` ) and appropriate input files have been generated