diff --git a/Docs/source/usage/parameters.rst b/Docs/source/usage/parameters.rst index a6ba9a2773d..af559aa1fba 100644 --- a/Docs/source/usage/parameters.rst +++ b/Docs/source/usage/parameters.rst @@ -2633,10 +2633,11 @@ Diagnostics and output In-situ visualization ^^^^^^^^^^^^^^^^^^^^^ -WarpX has four types of diagnostics: -``FullDiagnostics`` consist in dumps of fields and particles at given iterations, -``BackTransformedDiagnostics`` are used when running a simulation in a boosted frame, to reconstruct output data to the lab frame, -``BoundaryScrapingDiagnostics`` are used to collect the particles that are absorbed at the boundary, throughout the simulation, and +WarpX has five types of diagnostics: +``Full`` diagnostics consist in dumps of fields and particles at given iterations, +``TimeAveraged`` diagnostics only allow field data, which they output after averaging over a period of time, +``BackTransformed`` diagnostics are used when running a simulation in a boosted frame, to reconstruct output data to the lab frame, +``BoundaryScraping`` diagnostics are used to collect the particles that are absorbed at the boundary, throughout the simulation, and ``ReducedDiags`` allow the user to compute some reduced quantity (particle temperature, max of a field) and write a small amount of data to text files. Similar to what is done for physical species, WarpX has a class Diagnostics that allows users to initialize different diagnostics, each of them with different fields, resolution and period. This currently applies to standard diagnostics, but should be extended to back-transformed diagnostics and reduced diagnostics (and others) in a near future. @@ -2882,12 +2883,58 @@ In-situ capabilities can be used by turning on Sensei or Ascent (provided they a * ``warpx.mffile_nstreams`` (`int`) optional (default `4`) Limit the number of concurrent readers per file. + +.. _running-cpp-parameters-diagnostics-timeavg: + +Time-Averaged Diagnostics +^^^^^^^^^^^^^^^^^^^^^^^^^ + +``TimeAveraged`` diagnostics are a special type of ``Full`` diagnostics that allows for the output of time-averaged field data. +This type of diagnostics can be created using ``.diag_type = TimeAveraged``. +We support only field data and related options from the list at `Full Diagnostics`_. + +.. note:: + + As with ``Full`` diagnostics, ``TimeAveraged`` diagnostics output the initial **instantaneous** conditions of the selected fields on step 0 (unless more specific output intervals exclude output for step 0). + +In addition, ``TimeAveraged`` diagnostic options include: + +* ``.time_average_mode`` (`string`, default `none`) + Describes the operating mode for time averaged field output. + + * ``none`` for no averaging (instantaneous fields) + + * ``fixed_start`` for a diagnostic that averages all fields between the current output step and a fixed point in time + + * ``dynamic_start`` for a constant averaging period and output at different points in time (non-overlapping) + + .. note:: + + To enable time-averaged field output with intervals tightly spaced enough for overlapping averaging periods, + please create additional instances of ``TimeAveraged`` diagnostics. + +* ``.average_period_steps`` (`int`) + Configures the number of time steps in an averaging period. + Set this only in the ``dynamic_start`` mode and only if ``average_period_time`` has not already been set. + Will be ignored in the ``fixed_start`` mode (with warning). + +* ``.average_period_time`` (`float`, in seconds) + Configures the time (SI units) in an averaging period. + Set this only in the ``dynamic_start`` mode and only if ``average_period_steps`` has not already been set. + Will be ignored in the ``fixed_start`` mode (with warning). + +* ``.average_start_step`` (`int`) + Configures the time step at which time-averaging begins. + Set this only in the ``fixed_start`` mode. + Will be ignored in the ``dynamic_start`` mode (with warning). + .. _running-cpp-parameters-diagnostics-btd: BackTransformed Diagnostics ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``BackTransformed`` diag type are used when running a simulation in a boosted frame, to reconstruct output data to the lab frame. This option can be set using ``.diag_type = BackTransformed``. We support the following list of options from `Full Diagnostics`_ + ``.format``, ``.openpmd_backend``, ``.dump_rz_modes``, ``.file_prefix``, ``.diag_lo``, ``.diag_hi``, ``.write_species``, ``.species``. Additional options for this diagnostic include: diff --git a/Docs/source/usage/python.rst b/Docs/source/usage/python.rst index 38b0a31d7f3..8b40684feb9 100644 --- a/Docs/source/usage/python.rst +++ b/Docs/source/usage/python.rst @@ -114,6 +114,8 @@ Diagnostics .. autoclass:: pywarpx.picmi.FieldDiagnostic +.. autoclass:: pywarpx.picmi.TimeAveragedFieldDiagnostic + .. autoclass:: pywarpx.picmi.ElectrostaticFieldDiagnostic .. autoclass:: pywarpx.picmi.Checkpoint diff --git a/Examples/Physics_applications/laser_ion/CMakeLists.txt b/Examples/Physics_applications/laser_ion/CMakeLists.txt index f05203de0e8..66d53165290 100644 --- a/Examples/Physics_applications/laser_ion/CMakeLists.txt +++ b/Examples/Physics_applications/laser_ion/CMakeLists.txt @@ -6,8 +6,8 @@ add_warpx_test( 2 # dims 2 # nprocs inputs_test_2d_laser_ion_acc # inputs - analysis_default_openpmd_regression.py # analysis - diags/diag1/ # output + analysis_test_laser_ion.py # analysis + diags/diagInst/ # output OFF # dependency ) @@ -16,7 +16,7 @@ add_warpx_test( 2 # dims 2 # nprocs inputs_test_2d_laser_ion_acc_picmi.py # inputs - analysis_default_openpmd_regression.py # analysis - diags/diag1/ # output + analysis_test_laser_ion.py # analysis + diags/diagInst/ # output OFF # dependency ) diff --git a/Examples/Physics_applications/laser_ion/README.rst b/Examples/Physics_applications/laser_ion/README.rst index e55cf6889d4..c5dc5af3a77 100644 --- a/Examples/Physics_applications/laser_ion/README.rst +++ b/Examples/Physics_applications/laser_ion/README.rst @@ -87,7 +87,7 @@ Visualize :alt: Particle densities for electrons (top), protons (middle), and electrons again in logarithmic scale (bottom). :width: 80% - Particle densities for electrons (top), protons (middle), and electrons again in logarithmic scale (bottom). + Particle densities for electrons (top), protons (middle), and electrons again in logarithmic scale (bottom). Particle density output illustrates the evolution of the target in time and space. Logarithmic scales can help to identify where the target becomes transparent for the laser pulse (bottom panel in :numref:`fig-tnsa-densities` ). diff --git a/Examples/Physics_applications/laser_ion/analysis_default_openpmd_regression.py b/Examples/Physics_applications/laser_ion/analysis_default_openpmd_regression.py deleted file mode 120000 index 73e5ec47001..00000000000 --- a/Examples/Physics_applications/laser_ion/analysis_default_openpmd_regression.py +++ /dev/null @@ -1 +0,0 @@ -../../analysis_default_openpmd_regression.py \ No newline at end of file diff --git a/Examples/Physics_applications/laser_ion/analysis_test_laser_ion.py b/Examples/Physics_applications/laser_ion/analysis_test_laser_ion.py new file mode 100755 index 00000000000..d2106d33803 --- /dev/null +++ b/Examples/Physics_applications/laser_ion/analysis_test_laser_ion.py @@ -0,0 +1,85 @@ +#!/usr/bin/env python3 + +import os +import sys + +import numpy as np +import openpmd_api as io + +sys.path.insert(1, "../../../../warpx/Regression/Checksum/") +from checksumAPI import evaluate_checksum + + +def load_field_from_iteration( + series, iteration: int, field: str, coord: str = None +) -> np.ndarray: + """Load iteration of field data from file.""" + + it = series.iterations[iteration] + field_obj = it.meshes[f"{field}"] + + if field_obj.scalar: + field_data = field_obj[io.Mesh_Record_Component.SCALAR].load_chunk() + elif coord in [item[0] for item in list(field_obj.items())]: + field_data = field_obj[coord].load_chunk() + else: + raise Exception( + f"Specified coordinate: f{coord} is not available for field: f{field}." + ) + series.flush() + + return field_data + + +def compare_time_avg_with_instantaneous_diags(dir_inst: str, dir_avg: str): + """Compare instantaneous data (multiple iterations averaged in post-processing) with in-situ averaged data.""" + + field = "E" + coord = "z" + avg_period_steps = 5 + avg_output_step = 100 + + path_tpl_inst = f"{dir_inst}/openpmd_%T.h5" + path_tpl_avg = f"{dir_avg}/openpmd_%T.h5" + + si = io.Series(path_tpl_inst, io.Access.read_only) + sa = io.Series(path_tpl_avg, io.Access.read_only) + + ii0 = si.iterations[0] + fi0 = ii0.meshes[field][coord] + shape = fi0.shape + + data_inst = np.zeros(shape) + + for i in np.arange(avg_output_step - avg_period_steps + 1, avg_output_step + 1): + data_inst += load_field_from_iteration(si, i, field, coord) + + data_inst = data_inst / avg_period_steps + + data_avg = load_field_from_iteration(sa, avg_output_step, field, coord) + + # Compare the data + if np.allclose(data_inst, data_avg, rtol=1e-12): + print("Test passed: actual data is close to expected data.") + else: + print("Test failed: actual data is not close to expected data.") + sys.exit(1) + + +if __name__ == "__main__": + # NOTE: works only in the example directory due to relative path import + # compare checksums + evaluate_checksum( + test_name=os.path.split(os.getcwd())[1], + output_file=sys.argv[1], + output_format="openpmd", + ) + + # TODO: implement intervals parser for PICMI that allows more complex output periods + test_name = os.path.split(os.getcwd())[1] + if "picmi" not in test_name: + # Functionality test for TimeAveragedDiagnostics + compare_time_avg_with_instantaneous_diags( + dir_inst=sys.argv[1], + dir_avg="diags/diagTimeAvg/", + ) diff --git a/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc b/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc index 5ad8334e9ef..d69ed6dc375 100644 --- a/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc +++ b/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc @@ -200,18 +200,32 @@ laser1.profile_focal_distance = 4.0e-6 # focal distance from the antenna [m] ################################# # Diagnostics # -diagnostics.diags_names = diag1 openPMDfw openPMDbw +diagnostics.diags_names = diagInst diagTimeAvg openPMDfw openPMDbw -diag1.intervals = 100 -diag1.diag_type = Full -diag1.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho rho_electrons rho_hydrogen +# instantaneous field and particle diagnostic +diagInst.intervals = 100,96:100 # second interval only for CI testing the time-averaged diags +diagInst.diag_type = Full +diagInst.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho rho_electrons rho_hydrogen # reduce resolution of output fields -diag1.coarsening_ratio = 4 4 +diagInst.coarsening_ratio = 4 4 # demonstration of a spatial and momentum filter -diag1.electrons.plot_filter_function(t,x,y,z,ux,uy,uz) = (uz>=0) * (x<1.0e-6) * (x>-1.0e-6) -diag1.hydrogen.plot_filter_function(t,x,y,z,ux,uy,uz) = (uz>=0) * (x<1.0e-6) * (x>-1.0e-6) -diag1.format = openpmd -diag1.openpmd_backend = h5 +diagInst.electrons.plot_filter_function(t,x,y,z,ux,uy,uz) = (uz>=0) * (x<1.0e-6) * (x>-1.0e-6) +diagInst.hydrogen.plot_filter_function(t,x,y,z,ux,uy,uz) = (uz>=0) * (x<1.0e-6) * (x>-1.0e-6) +diagInst.format = openpmd +diagInst.openpmd_backend = h5 + +# time-averaged particle and field diagnostic +diagTimeAvg.intervals = 100 +diagTimeAvg.diag_type = TimeAveraged +diagTimeAvg.time_average_mode = dynamic_start +#diagTimeAvg.average_period_time = 2.67e-15 # period of 800 nm light waves +diagTimeAvg.average_period_steps = 5 # use only either `time` or `steps` +diagTimeAvg.write_species = 0 +diagTimeAvg.fields_to_plot = Ex Ey Ez Bx By Bz jx jy jz rho rho_electrons rho_hydrogen +# reduce resolution of output fields +diagTimeAvg.coarsening_ratio = 4 4 +diagTimeAvg.format = openpmd +diagTimeAvg.openpmd_backend = h5 openPMDfw.intervals = 100 openPMDfw.diag_type = Full diff --git a/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc_picmi.py b/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc_picmi.py index 04f9111ec5f..66ba5f64091 100755 --- a/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc_picmi.py +++ b/Examples/Physics_applications/laser_ion/inputs_test_2d_laser_ion_acc_picmi.py @@ -140,7 +140,7 @@ # Diagnostics particle_diag = picmi.ParticleDiagnostic( - name="diag1", + name="diagInst", period=100, warpx_format="openpmd", warpx_openpmd_backend="h5", @@ -153,7 +153,7 @@ for ncell_comp, cr in zip([nx, nz], coarsening_ratio): ncell_field.append(int(ncell_comp / cr)) field_diag = picmi.FieldDiagnostic( - name="diag1", + name="diagInst", grid=grid, period=100, number_of_cells=ncell_field, @@ -162,6 +162,18 @@ warpx_openpmd_backend="h5", ) +field_time_avg_diag = picmi.TimeAveragedFieldDiagnostic( + name="diagTimeAvg", + grid=grid, + period=100, + number_of_cells=ncell_field, + data_list=["B", "E", "J", "rho", "rho_electrons", "rho_hydrogen"], + warpx_format="openpmd", + warpx_openpmd_backend="h5", + warpx_time_average_mode="dynamic_start", + warpx_average_period_time=2.67e-15, +) + particle_fw_diag = picmi.ParticleDiagnostic( name="openPMDfw", period=100, @@ -292,6 +304,7 @@ # Add full diagnostics sim.add_diagnostic(particle_diag) sim.add_diagnostic(field_diag) +sim.add_diagnostic(field_time_avg_diag) sim.add_diagnostic(particle_fw_diag) sim.add_diagnostic(particle_bw_diag) # Add reduced diagnostics diff --git a/Examples/Physics_applications/laser_ion/plot_2d.py b/Examples/Physics_applications/laser_ion/plot_2d.py index f8a3b05d8a3..b3aefb80606 100644 --- a/Examples/Physics_applications/laser_ion/plot_2d.py +++ b/Examples/Physics_applications/laser_ion/plot_2d.py @@ -259,7 +259,7 @@ def visualize_particle_histogram_iteration( "-d", "--diag_dir", type=str, - default="./diags/diag1", + default="./diags/diagInst", help="Directory containing density and field diagnostics", ) parser.add_argument( diff --git a/Python/pywarpx/picmi.py b/Python/pywarpx/picmi.py index 478b4d5802e..c7a27f62df0 100644 --- a/Python/pywarpx/picmi.py +++ b/Python/pywarpx/picmi.py @@ -3271,6 +3271,57 @@ def diagnostic_initialize_inputs(self): ElectrostaticFieldDiagnostic = FieldDiagnostic +class TimeAveragedFieldDiagnostic(FieldDiagnostic): + """ + See `Input Parameters `__ for more information. + + Parameters + ---------- + warpx_time_average_mode: str + Type of time averaging diagnostic + Supported values include ``"none"``, ``"fixed_start"``, and ``"dynamic_start"`` + + * ``"none"`` for no averaging (instantaneous fields) + * ``"fixed_start"`` for a diagnostic that averages all fields between the current output step and a fixed point in time + * ``"dynamic_start"`` for a constant averaging period and output at different points in time (non-overlapping) + + warpx_average_period_steps: int, optional + Configures the number of time steps in an averaging period. + Set this only in the ``"dynamic_start"`` mode and only if ``warpx_average_period_time`` has not already been set. + Will be ignored in the ``"fixed_start"`` mode (with warning). + + warpx_average_period_time: float, optional + Configures the time (SI units) in an averaging period. + Set this only in the ``"dynamic_start"`` mode and only if ``average_period_steps`` has not already been set. + Will be ignored in the ``"fixed_start"`` mode (with warning). + + warpx_average_start_steps: int, optional + Configures the time step at which time-averaging begins. + Set this only in the ``"fixed_start"`` mode. + Will be ignored in the ``"dynamic_start"`` mode (with warning). + """ + + def init(self, kw): + super().init(kw) + self.time_average_mode = kw.pop("warpx_time_average_mode", None) + self.average_period_steps = kw.pop("warpx_average_period_steps", None) + self.average_period_time = kw.pop("warpx_average_period_time", None) + self.average_start_step = kw.pop("warpx_average_start_step", None) + + def diagnostic_initialize_inputs(self): + super().diagnostic_initialize_inputs() + + self.diagnostic.set_or_replace_attr("diag_type", "TimeAveraged") + + if "write_species" not in self.diagnostic.argvattrs: + self.diagnostic.write_species = False + + self.diagnostic.time_average_mode = self.time_average_mode + self.diagnostic.average_period_steps = self.average_period_steps + self.diagnostic.average_period_time = self.average_period_time + self.diagnostic.average_start_step = self.average_start_step + + class Checkpoint(picmistandard.base._ClassWithInit, WarpXDiagnosticBase): """ Sets up checkpointing of the simulation, allowing for later restarts diff --git a/Source/Diagnostics/BTDiagnostics.H b/Source/Diagnostics/BTDiagnostics.H index ab04f30ef18..c7137f45c9d 100644 --- a/Source/Diagnostics/BTDiagnostics.H +++ b/Source/Diagnostics/BTDiagnostics.H @@ -28,7 +28,7 @@ class BTDiagnostics final : public Diagnostics { public: - BTDiagnostics (int i, const std::string& name); + BTDiagnostics (int i, const std::string& name, DiagTypes diag_type); private: /** Whether to plot raw (i.e., NOT cell-centered) fields */ diff --git a/Source/Diagnostics/BTDiagnostics.cpp b/Source/Diagnostics/BTDiagnostics.cpp index 312bbc7ec45..4939e2fb207 100644 --- a/Source/Diagnostics/BTDiagnostics.cpp +++ b/Source/Diagnostics/BTDiagnostics.cpp @@ -55,8 +55,8 @@ namespace constexpr int permission_flag_rwxrxrx = 0755; } -BTDiagnostics::BTDiagnostics (int i, const std::string& name) - : Diagnostics{i, name}, +BTDiagnostics::BTDiagnostics (int i, const std::string& name, DiagTypes diag_type) + : Diagnostics{i, name, diag_type}, m_cell_centered_data_name("BTD_cell_centered_data_" + name) { ReadParameters(); diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.H b/Source/Diagnostics/BoundaryScrapingDiagnostics.H index 3e5fc1f19eb..f78e7b4574b 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.H +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.H @@ -23,7 +23,7 @@ public: * @param i index of diagnostics in MultiDiagnostics::alldiags * @param name diagnostics name in the inputs file */ - BoundaryScrapingDiagnostics (int i, const std::string& name); + BoundaryScrapingDiagnostics (int i, const std::string& name, DiagTypes diag_type); private: /** Read relevant parameters for BoundaryScraping */ diff --git a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp index 3757082ab4d..8df58b6fb28 100644 --- a/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp +++ b/Source/Diagnostics/BoundaryScrapingDiagnostics.cpp @@ -22,8 +22,8 @@ using namespace amrex::literals; -BoundaryScrapingDiagnostics::BoundaryScrapingDiagnostics (int i, const std::string& name) - : Diagnostics{i, name} +BoundaryScrapingDiagnostics::BoundaryScrapingDiagnostics (int i, const std::string& name, DiagTypes diag_type) + : Diagnostics{i, name, diag_type} { ReadParameters(); } diff --git a/Source/Diagnostics/Diagnostics.H b/Source/Diagnostics/Diagnostics.H index 20550364fb7..d0c70e76c1f 100644 --- a/Source/Diagnostics/Diagnostics.H +++ b/Source/Diagnostics/Diagnostics.H @@ -21,6 +21,8 @@ #include #include +/** All types of diagnostics. */ +enum struct DiagTypes {Full, BackTransformed, BoundaryScraping, TimeAveraged}; /** * \brief base class for diagnostics. * Contains main routines to filter, compute and flush diagnostics. @@ -35,7 +37,7 @@ public: * @param i index of diagnostics in MultiDiagnostics::alldiags * @param name diagnostics name in the inputs file */ - Diagnostics (int i, std::string name); + Diagnostics (int i, std::string name, DiagTypes diag_type); /** Virtual Destructor to handle clean destruction of derived classes */ virtual ~Diagnostics (); @@ -45,6 +47,8 @@ public: Diagnostics(Diagnostics&& ) = default; Diagnostics& operator=(Diagnostics&& ) = default; + /** Stores the diag type */ + DiagTypes m_diag_type; /** Pack (stack) all fields in the cell-centered output MultiFab m_mf_output. * * Fields are computed (e.g., cell-centered or back-transformed) @@ -266,6 +270,12 @@ protected: * The second vector is loops over the total number of levels. */ amrex::Vector< amrex::Vector< amrex::MultiFab > > m_mf_output; + /** summation multifab, where all fields (computed, cell-centered, and stacked) + * are summed for every step in a time averaging period. + * The first vector is for total number of snapshots. (=1 for FullDiagnostics) + * The second vector is loops over the total number of levels. + */ + amrex::Vector< amrex::Vector > m_sum_mf_output; /** Geometry that defines the domain attributes corresponding to output multifab. * Specifically, the user-defined physical co-ordinates for the diagnostics diff --git a/Source/Diagnostics/Diagnostics.cpp b/Source/Diagnostics/Diagnostics.cpp index fd079479285..0f659065185 100644 --- a/Source/Diagnostics/Diagnostics.cpp +++ b/Source/Diagnostics/Diagnostics.cpp @@ -38,8 +38,8 @@ using namespace amrex::literals; -Diagnostics::Diagnostics (int i, std::string name) - : m_diag_name(std::move(name)), m_diag_index(i) +Diagnostics::Diagnostics (int i, std::string name, DiagTypes diag_type) + : m_diag_type(diag_type), m_diag_name(std::move(name)), m_diag_index(i) { } @@ -536,6 +536,14 @@ Diagnostics::InitBaseData () m_mf_output[i].resize( nmax_lev ); } + // allocate vector of buffers and vector of levels for each buffer for summation multifab for TimeAveragedDiagnostics + if (m_diag_type == DiagTypes::TimeAveraged) { + m_sum_mf_output.resize(m_num_buffers); + for (int i = 0; i < m_num_buffers; ++i) { + m_sum_mf_output[i].resize( nmax_lev ); + } + } + // allocate vector of geometry objects corresponding to each output multifab. m_geom_output.resize( m_num_buffers ); for (int i = 0; i < m_num_buffers; ++i) { @@ -575,6 +583,15 @@ Diagnostics::ComputeAndPack () // Check that the proper number of components of mf_avg were updated. AMREX_ALWAYS_ASSERT( icomp_dst == m_varnames.size() ); + if (m_diag_type == DiagTypes::TimeAveraged) { + + const amrex::Real real_a = 1.0; + // Compute m_sum_mf_output += real_a*m_mf_output + amrex::MultiFab::Saxpy( + m_sum_mf_output[i_buffer][lev], real_a, m_mf_output[i_buffer][lev], + 0, 0, m_mf_output[i_buffer][lev].nComp(), m_mf_output[i_buffer][lev].nGrowVect()); + } + // needed for contour plots of rho, i.e. ascent/sensei if (m_format == "sensei" || m_format == "ascent") { ablastr::utils::communication::FillBoundary(m_mf_output[i_buffer][lev], WarpX::do_single_precision_comms, diff --git a/Source/Diagnostics/FullDiagnostics.H b/Source/Diagnostics/FullDiagnostics.H index 1b999a9b361..61f63aa78e2 100644 --- a/Source/Diagnostics/FullDiagnostics.H +++ b/Source/Diagnostics/FullDiagnostics.H @@ -4,12 +4,22 @@ #include "Diagnostics.H" #include "Utils/Parser/IntervalsParser.H" +#include + #include class FullDiagnostics final : public Diagnostics { public: - FullDiagnostics (int i, const std::string& name); + FullDiagnostics (int i, const std::string& name, DiagTypes diag_type); + /** Type of time averaging for diagnostics (fields only) + * None corresponds to instantaneous diags + * Static corresponds to a fixed starting step for averaging, + * will average until the end, and dump out intermediate average results + * Dynamic corresponds to a moving period for averaging where the start step + * is as many steps before the output interval as the averaging period is long. + */ + enum struct TimeAverageType {None, Static, Dynamic}; private: /** Read user-requested parameters for full diagnostics */ void ReadParameters (); @@ -25,10 +35,20 @@ private: * before writing the diagnostic. */ bool m_solver_deposits_current = true; - /** Flush m_mf_output and particles to file for the i^th buffer */ + /** Whether the diagnostics are averaging data over time or not */ + TimeAverageType m_time_average_mode = TimeAverageType::None; + /** Period to average fields over: in steps */ + int m_average_period_steps = -1; + /** Period to average fields over: in seconds */ + amrex::Real m_average_period_time = -1.0; + /** Time step to start averaging */ + int m_average_start_step = -1; + /** Flush m_mf_output or m_sum_mf_output and particles to file for the i^th buffer */ void Flush (int i_buffer, bool /* force_flush */) override; /** Flush raw data */ void FlushRaw (); + /** Initialize Data required to compute TimeAveraged diagnostics */ + void DerivedInitData () override; /** whether to compute and pack cell-centered data in m_mf_output * \param[in] step current time step * \param[in] force_flush if true, return true for any step since output must be diff --git a/Source/Diagnostics/FullDiagnostics.cpp b/Source/Diagnostics/FullDiagnostics.cpp index e5eefc82de5..eeca8ffdb44 100644 --- a/Source/Diagnostics/FullDiagnostics.cpp +++ b/Source/Diagnostics/FullDiagnostics.cpp @@ -47,8 +47,8 @@ using namespace amrex::literals; using warpx::fields::FieldType; -FullDiagnostics::FullDiagnostics (int i, const std::string& name): - Diagnostics{i, name}, +FullDiagnostics::FullDiagnostics (int i, const std::string& name, DiagTypes diag_type): + Diagnostics{i, name, diag_type}, m_solver_deposits_current{ (WarpX::electromagnetic_solver_id != ElectromagneticSolverAlgo::None) || (WarpX::electrostatic_solver_id == ElectrostaticSolverAlgo::LabFrameElectroMagnetostatic)} @@ -57,6 +57,27 @@ FullDiagnostics::FullDiagnostics (int i, const std::string& name): BackwardCompatibility(); } +void +FullDiagnostics::DerivedInitData() { + if (m_diag_type == DiagTypes::TimeAveraged) { + auto & warpx = WarpX::GetInstance(); + if (m_time_average_mode == TimeAverageType::Dynamic) { + + // already checked in ReadParameters that only one of the parameters is set + // calculate the other averaging period parameter from the other one, respectively + if (m_average_period_steps > 0) { + m_average_period_time = m_average_period_steps * warpx.getdt(0); + } else if (m_average_period_time > 0) { + m_average_period_steps = static_cast (std::round(m_average_period_time / warpx.getdt(0))); + } + amrex::Print() << Utils::TextMsg::Info( + "Initializing TimeAveragedDiagnostics " + m_diag_name + + " with an averaging period of " + std::to_string(m_average_period_steps) + " steps" + ); + } + } +} + void FullDiagnostics::InitializeParticleBuffer () { @@ -101,6 +122,92 @@ FullDiagnostics::ReadParameters () const bool plot_raw_fields_guards_specified = pp_diag_name.query("plot_raw_fields_guards", m_plot_raw_fields_guards); const bool raw_specified = plot_raw_fields_specified || plot_raw_fields_guards_specified; + if (m_diag_type == DiagTypes::TimeAveraged) { + std::string m_time_average_mode_str = "none"; + /** Whether the diagnostics are averaging data over time or not + * Valid options are "fixed_start" and "dynamic_start". + */ + pp_diag_name.get("time_average_mode", m_time_average_mode_str); + + const amrex::ParmParse pp_warpx("warpx"); + std::vector dt_interval_vec = {"-1"}; + const bool timestep_may_vary = pp_warpx.queryarr("dt_update_interval", dt_interval_vec); + amrex::Print() << Utils::TextMsg::Warn("Time step varies?" + std::to_string(timestep_may_vary)); + if (timestep_may_vary) { + WARPX_ABORT_WITH_MESSAGE( + "Time-averaged diagnostics (encountered in: " + + m_diag_name + ") are currently not supported with adaptive time-stepping" + ); + } + + if (m_time_average_mode_str == "fixed_start") { + m_time_average_mode = TimeAverageType::Static; + } else if (m_time_average_mode_str == "dynamic_start") { + m_time_average_mode = TimeAverageType::Dynamic; + } else if (m_time_average_mode_str == "none") { + m_time_average_mode = TimeAverageType::None; + } else { + WARPX_ABORT_WITH_MESSAGE( + "Unknown time averaging mode. Valid entries are: none, fixed_start, dynamic_start" + ); + } + + const bool averaging_period_steps_specified = pp_diag_name.query( + "average_period_steps", m_average_period_steps + ); + const bool averaging_period_time_specified = pp_diag_name.queryWithParser( + "average_period_time", m_average_period_time + ); + + if (m_time_average_mode == TimeAverageType::Static) { + // This fails if users do not specify a start. + pp_diag_name.get("average_start_step", m_average_start_step); + if (m_average_start_step == 0) { + WARPX_ABORT_WITH_MESSAGE( + "Static-start time-averaged diagnostic " + m_diag_name + " requires a positive (non-zero) value " + "for the 'average_start_step' parameter." + ); + } + + if (averaging_period_time_specified || averaging_period_steps_specified) { + const std::string period_spec_warn_msg = "An averaging period was specified for the 'static_start' averaging mode " \ + "but will be IGNORED. Averaging will be performed between step " \ + + std::to_string(m_average_start_step) \ + + " and the specified intervals."; + ablastr::warn_manager::WMRecordWarning( + "Diagnostics", + period_spec_warn_msg, + ablastr::warn_manager::WarnPriority::medium + ); + } + + } + + if (m_time_average_mode == TimeAverageType::Dynamic) { + // one of the two averaging period options must be set but neither none nor both + if ( + (averaging_period_steps_specified && averaging_period_time_specified) + || !(averaging_period_steps_specified || averaging_period_time_specified) + ) { + WARPX_ABORT_WITH_MESSAGE("Please specify either 'average_period_steps' or 'average_period_time', not both."); + } + + int unused_start_step = -1; + const bool averaging_start_on_dynamic_period_specified = pp_diag_name.query("average_start_step", unused_start_step); + if (averaging_start_on_dynamic_period_specified) { + const std::string start_spec_warn_msg = "An averaging start step was specified for the 'dynamic_start'" \ + "time-averaged diagnostic " + m_diag_name + " but will be IGNORED. " \ + "Averaging will begin with the first averaging period."; + ablastr::warn_manager::WMRecordWarning( + "Diagnostics", + start_spec_warn_msg, + ablastr::warn_manager::WarnPriority::medium + ); + } + } + } + + #ifdef WARPX_DIM_RZ pp_diag_name.query("dump_rz_modes", m_dump_rz_modes); #else @@ -138,11 +245,44 @@ FullDiagnostics::Flush ( int i_buffer, bool /* force_flush */ ) // is supported for BackTransformed Diagnostics, in BTDiagnostics class. auto & warpx = WarpX::GetInstance(); - m_flush_format->WriteToFile( - m_varnames, m_mf_output.at(i_buffer), m_geom_output.at(i_buffer), warpx.getistep(), - warpx.gett_new(0), - m_output_species.at(i_buffer), nlev_output, m_file_prefix, - m_file_min_digits, m_plot_raw_fields, m_plot_raw_fields_guards); + // Get the time step on coarsest level. + const int step = warpx.getistep(0); + // For time-averaged diagnostics, we still write out an instantaneous diagnostic on step 0 + // to accommodate a user workflow that only uses that type of diagnostic. + // This allows for quicker turnaround in setup by avoiding having to set an additional instantaneous diagnostic. + if (m_diag_type == DiagTypes::TimeAveraged && step > 0) { + if (m_time_average_mode == TimeAverageType::Static || m_time_average_mode == TimeAverageType::Dynamic) { + // Loop over the output levels and divide by the number of steps in the averaging period + for (int lev = 0; lev < nlev_output; ++lev) { + m_sum_mf_output.at(i_buffer).at(lev).mult(1._rt/static_cast(m_average_period_steps)); + } + + m_flush_format->WriteToFile( + m_varnames, m_sum_mf_output.at(i_buffer), m_geom_output.at(i_buffer), warpx.getistep(), + warpx.gett_new(0), + m_output_species.at(i_buffer), nlev_output, m_file_prefix, + m_file_min_digits, m_plot_raw_fields, m_plot_raw_fields_guards); + + // Reset the values in the dynamic start time-averaged diagnostics after flush + if (m_time_average_mode == TimeAverageType::Dynamic) { + for (int lev = 0; lev < nlev_output; ++lev) { + m_sum_mf_output.at(i_buffer).at(lev).setVal(0.); + } + } + } + } else { + if (m_diag_type == DiagTypes::TimeAveraged && step == 0) { + // For both dynamic_start and fixed_start at step 0 we prepare an instantaneous output + amrex::Print() << Utils::TextMsg::Info("Time-averaged diagnostic " + m_diag_name + + " is preparing an instantaneous output during step " + std::to_string(step)); + } + + m_flush_format->WriteToFile( + m_varnames, m_mf_output.at(i_buffer), m_geom_output.at(i_buffer), warpx.getistep(), + warpx.gett_new(0), + m_output_species.at(i_buffer), nlev_output, m_file_prefix, + m_file_min_digits, m_plot_raw_fields, m_plot_raw_fields_guards); + } FlushRaw(); } @@ -165,9 +305,60 @@ FullDiagnostics::DoDump (int step, int /*i_buffer*/, bool force_flush) bool FullDiagnostics::DoComputeAndPack (int step, bool force_flush) { + // Start averaging at output step (from diag.intervals) - period + 1 + bool in_averaging_period = false; + if (m_diag_type == DiagTypes::TimeAveraged) { + + if (step > 0) { + + if (m_time_average_mode == TimeAverageType::Dynamic) { + m_average_start_step = m_intervals.nextContains(step) - m_average_period_steps; + // check that the periods do not overlap and that the start step is not negative + if (m_average_start_step > 0) { + // The start step cannot be on an interval step because then we would begin a new period and also output the old one + if (m_average_start_step < m_intervals.previousContains(step)) { + WARPX_ABORT_WITH_MESSAGE( + "Averaging periods may not overlap within a single diagnostic. " + "Please create a second diagnostic for overlapping time averaging periods " + "and account for the increased memory consumption." + ); + } + } else { + WARPX_ABORT_WITH_MESSAGE( + "The step to begin time averaging (" + + std::to_string(m_average_start_step) + + ") for diagnostic " + m_diag_name + " must be a positive number." + ); + } + + if (step >= m_average_start_step && step <= m_intervals.nextContains(step)) { + in_averaging_period = true; + + if (m_time_average_mode == TimeAverageType::Static) { + // Update time averaging period to current step + m_average_period_steps = step - m_average_start_step; + } + } + // Print information on when time-averaging is active + if (in_averaging_period) { + if (step == m_average_start_step) { + amrex::Print() << Utils::TextMsg::Info( + "Begin time averaging for " + m_diag_name + " and output at step " + + std::to_string(m_intervals.nextContains(step)) + ); + } else { + amrex::Print() + << Utils::TextMsg::Info( + "Time-averaging during this step for diagnostic: " + m_diag_name); + } + } + } + } + } // Data must be computed and packed for full diagnostics // whenever the data needs to be flushed. - return (force_flush || m_intervals.contains(step+1)); + return (force_flush || m_intervals.contains(step+1) || in_averaging_period); + } void @@ -600,6 +791,7 @@ FullDiagnostics::InitializeBufferData (int i_buffer, int lev, bool restart ) { diag_dom.setHi( idim, warpx.Geom(lev).ProbLo(idim) + (ba.getCellCenteredBox( static_cast(ba.size())-1 ).bigEnd(idim) + 1) * warpx.Geom(lev).CellSize(idim)); } + } WARPX_ALWAYS_ASSERT_WITH_MESSAGE( @@ -614,6 +806,13 @@ FullDiagnostics::InitializeBufferData (int i_buffer, int lev, bool restart ) { int const ncomp = static_cast(m_varnames.size()); m_mf_output[i_buffer][lev] = amrex::MultiFab(ba, dmap, ncomp, ngrow); + if (m_diag_type == DiagTypes::TimeAveraged) { + // Allocate MultiFab for cell-centered field output accumulation. The data will be averaged before flushing. + m_sum_mf_output[i_buffer][lev] = amrex::MultiFab(ba, dmap, ncomp, ngrow); + // Initialize to zero because we add data. + m_sum_mf_output[i_buffer][lev].setVal(0.); + } + if (lev == 0) { // The extent of the domain covered by the diag multifab, m_mf_output //default non-periodic geometry for diags diff --git a/Source/Diagnostics/MultiDiagnostics.H b/Source/Diagnostics/MultiDiagnostics.H index d220396ed12..a22e20b44da 100644 --- a/Source/Diagnostics/MultiDiagnostics.H +++ b/Source/Diagnostics/MultiDiagnostics.H @@ -11,8 +11,6 @@ #include #include -/** All types of diagnostics. */ -enum struct DiagTypes {Full, BackTransformed, BoundaryScraping}; /** * \brief This class contains a vector of all diagnostics in the simulation. diff --git a/Source/Diagnostics/MultiDiagnostics.cpp b/Source/Diagnostics/MultiDiagnostics.cpp index ea14919c713..2119ac276f9 100644 --- a/Source/Diagnostics/MultiDiagnostics.cpp +++ b/Source/Diagnostics/MultiDiagnostics.cpp @@ -21,12 +21,12 @@ MultiDiagnostics::MultiDiagnostics () */ alldiags.resize( ndiags ); for (int i=0; i(i, diags_names[i]); + if ( diags_types[i] == DiagTypes::Full || diags_types[i] == DiagTypes::TimeAveraged ){ + alldiags[i] = std::make_unique(i, diags_names[i], diags_types[i]); } else if ( diags_types[i] == DiagTypes::BackTransformed ){ - alldiags[i] = std::make_unique(i, diags_names[i]); + alldiags[i] = std::make_unique(i, diags_names[i], diags_types[i]); } else if ( diags_types[i] == DiagTypes::BoundaryScraping ){ - alldiags[i] = std::make_unique(i, diags_names[i]); + alldiags[i] = std::make_unique(i, diags_names[i], diags_types[i]); } else { WARPX_ABORT_WITH_MESSAGE("Unknown diagnostic type"); } @@ -68,9 +68,10 @@ MultiDiagnostics::ReadParameters () std::string diag_type_str; pp_diag_name.get("diag_type", diag_type_str); WARPX_ALWAYS_ASSERT_WITH_MESSAGE( - diag_type_str == "Full" || diag_type_str == "BackTransformed" || diag_type_str == "BoundaryScraping", - ".diag_type must be Full or BackTransformed or BoundaryScraping"); + diag_type_str == "Full" || diag_type_str == "TimeAveraged" || diag_type_str == "BackTransformed" || diag_type_str == "BoundaryScraping", + ".diag_type must be Full, TimeAveraged, BackTransformed or BoundaryScraping"); if (diag_type_str == "Full") { diags_types[i] = DiagTypes::Full; } + if (diag_type_str == "TimeAveraged") { diags_types[i] = DiagTypes::TimeAveraged; } if (diag_type_str == "BackTransformed") { diags_types[i] = DiagTypes::BackTransformed; } if (diag_type_str == "BoundaryScraping") { diags_types[i] = DiagTypes::BoundaryScraping; } } diff --git a/Source/Diagnostics/WarpXOpenPMD.cpp b/Source/Diagnostics/WarpXOpenPMD.cpp index 4e1b6238adf..e38ae8c8300 100644 --- a/Source/Diagnostics/WarpXOpenPMD.cpp +++ b/Source/Diagnostics/WarpXOpenPMD.cpp @@ -1195,6 +1195,8 @@ WarpXOpenPMDPlot::SetupFields ( openPMD::Container< openPMD::Mesh >& meshes, if (WarpX::do_dive_cleaning) { meshes.setAttribute("chargeCorrectionParameters", "period=1"); } + // TODO set meta-data information for time-averaged quantities + // but we need information of the specific diagnostic in here }