Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/development' into add_thread_s…
Browse files Browse the repository at this point in the history 
…anitizer
  • Loading branch information
@lucafedeli88
lucafedeli88 committed Jul 22, 2024
2 parents 3f3adbf + c6bb95a commit 5be2c10
Show file tree
Hide file tree
Showing 17 changed files with 256 additions and 408 deletions.
161 changes: 31 additions & 130 deletions Docs/source/install/hpc/lassen.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,27 +24,9 @@ If you are new to this system, **please see the following resources**:
Login
-----

.. tab-set::

.. tab-item:: TOSS4 (RHEL8)

Lassen is currently transitioning to RHEL8.
During this transition, first SSH into lassen and then to the updated RHEL8/TOSS4 nodes.

.. code-block:: bash
ssh lassen.llnl.gov
ssh eatoss4
Approximately October/November 2023, the new software environment on these nodes will be the new default.

.. tab-item:: TOSS3 (RHEL7)

.. code-block:: bash
ssh lassen.llnl.gov
.. code-block:: bash
Approximately October/November 2023, this partition will become TOSS4 (RHEL8) as well.
ssh lassen.llnl.gov
.. _building-lassen-preparation:
Expand All @@ -58,135 +40,54 @@ Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git /usr/workspace/${USER}/lassen/src/warpx
.. tab-set::

.. tab-item:: TOSS4 (RHEL8)

We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx.profile``.
Create it now:

.. code-block:: bash
cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example $HOME/lassen_v100_warpx.profile
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx.profile.example
:language: bash

Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx.profile``.
Enter the edit mode by typing ``i`` and edit line 2 to read:

.. code-block:: bash
export proj="nsldt"
Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).

.. important::

Now, and as the first step on future logins to lassen, activate these environment settings:

.. code-block:: bash
We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx_toss3.profile``.
Create it now:

source $HOME/lassen_v100_warpx.profile
.. tab-item:: TOSS3 (RHEL7)

We use system software modules, add environment hints and further dependencies via the file ``$HOME/lassen_v100_warpx_toss3.profile``.
Create it now:

.. code-block:: bash
.. code-block:: bash
cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example $HOME/lassen_v100_warpx_toss3.profile
cp /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example $HOME/lassen_v100_warpx_toss3.profile
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example
:language: bash
.. literalinclude:: ../../../../Tools/machines/lassen-llnl/lassen_v100_warpx_toss3.profile.example
:language: bash

Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx_toss3.profile``.
Enter the edit mode by typing ``i`` and edit line 2 to read:
Edit the 2nd line of this script, which sets the ``export proj=""`` variable.
For example, if you are member of the project ``nsldt``, then run ``vi $HOME/lassen_v100_warpx_toss3.profile``.
Enter the edit mode by typing ``i`` and edit line 2 to read:

.. code-block:: bash
.. code-block:: bash
export proj="nsldt"
export proj="nsldt"
Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).
Exit the ``vi`` editor with ``Esc`` and then type ``:wq`` (write & quit).

.. important::
.. important::

Now, and as the first step on future logins to lassen, activate these environment settings:
Now, and as the first step on future logins to lassen, activate these environment settings:

.. code-block:: bash
.. code-block:: bash
source $HOME/lassen_v100_warpx_toss3.profile
source $HOME/lassen_v100_warpx_toss3.profile
Finally, since lassen does not yet provide software modules for some of our dependencies, install them once:

.. tab-set::

.. tab-item:: TOSS4 (RHEL8)

.. code-block:: bash
bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies.sh
source /usr/workspace/${USER}/lassen/gpu/venvs/warpx-lassen/bin/activate
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies.sh
:language: bash

.. dropdown:: AI/ML Dependencies (Optional)
:animate: fade-in-slide-down

If you plan to run AI/ML workflows depending on pyTorch, run the next step as well.
This will take a while and should be skipped if not needed.

.. code-block:: bash
runNode bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_ml.sh
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_ml.sh
:language: bash

For `optimas dependencies <https://github.com/optimas-org/optimas>`__ (incl. scikit-learn), plan another hour of build time:

.. code-block:: bash
python3 -m pip install -r /usr/workspace/${USER}/lassen/src/warpx/Tools/optimas/requirements.txt
.. tab-item:: TOSS3 (RHEL7)
.. code-block:: bash
.. code-block:: bash
bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh
source /usr/workspace/${USER}/lassen-toss3/gpu/venvs/warpx-lassen-toss3/bin/activate
bash /usr/workspace/${USER}/lassen/src/warpx/Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh
source /usr/workspace/${USER}/lassen-toss3/gpu/venvs/warpx-lassen-toss3/bin/activate
.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down

.. dropdown:: Script Details
:color: light
:icon: info
:animate: fade-in-slide-down
.. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh
:language: bash

.. literalinclude:: ../../../../Tools/machines/lassen-llnl/install_v100_dependencies_toss3.sh
:language: bash

.. _building-lassen-compilation:

Expand Down
2 changes: 1 addition & 1 deletion Source/Diagnostics/SliceDiagnostic.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ CreateSlice( const MultiFab& mf, const Vector<Geometry> &dom_geom,
ablastr::utils::communication::ParallelCopy(*smf, mf, 0, 0, ncomp, nghost_vect, nghost_vect, WarpX::do_single_precision_comms);

// interpolate if required on refined slice //
if (interpolate == 1 ) {
if (interpolate) {
InterpolateSliceValues( *smf, interp_lo, slice_cc_nd_box, dom_geom,
ncomp, nghost, slice_lo, slice_hi, SliceType, real_box);
}
Expand Down
4 changes: 2 additions & 2 deletions Source/Evolve/WarpXEvolve.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,15 +142,15 @@ WarpX::Evolve (int numsteps)
OneStep_multiJ(cur_time);
}
// Electromagnetic case: no subcycling or no mesh refinement
else if (do_subcycling == 0 || finest_level == 0)
else if ( !do_subcycling || (finest_level == 0))
{
OneStep_nosub(cur_time);
// E: guard cells are up-to-date
// B: guard cells are NOT up-to-date
// F: guard cells are NOT up-to-date
}
// Electromagnetic case: subcycling with one level of mesh refinement
else if (do_subcycling == 1 && finest_level == 1)
else if (do_subcycling && (finest_level == 1))
{
OneStep_sub1(cur_time);
}
Expand Down
12 changes: 6 additions & 6 deletions Source/Initialization/WarpXInitData.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -294,19 +294,19 @@ WarpX::PrintMainPICparameters ()
" | - v_comoving = (" << m_v_comoving[0] << "," <<
m_v_comoving[1] << "," << m_v_comoving[2] << ")\n";
}
if (WarpX::update_with_rho==1) {
if (WarpX::update_with_rho) {
amrex::Print() << " | - update with rho is ON \n";
}
if (current_correction==1) {
if (current_correction) {
amrex::Print() << " | - current correction is ON \n";
}
if (WarpX::do_dive_cleaning==1) {
if (WarpX::do_dive_cleaning) {
amrex::Print() << " | - div(E) cleaning is ON \n";
}
if (WarpX::do_divb_cleaning==1) {
if (WarpX::do_divb_cleaning) {
amrex::Print() << " | - div(B) cleaning is ON \n";
}
if (do_multi_J == 1){
if (do_multi_J){
amrex::Print() << " | - multi-J deposition is ON \n";
amrex::Print() << " | - do_multi_J_n_depositions = "
<< WarpX::do_multi_J_n_depositions << "\n";
Expand All @@ -323,7 +323,7 @@ WarpX::PrintMainPICparameters ()
amrex::Print() << " | - rho_in_time = constant \n";
}
}
if (fft_do_time_averaging == 1){
if (fft_do_time_averaging){
amrex::Print()<<" | - time-averaged is ON \n";
}
#endif // WARPX_USE_FFT
Expand Down
6 changes: 3 additions & 3 deletions Source/Parallelization/GuardCellManager.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ guardCellManager::Init (
// When using subcycling, the particles on the fine level perform two pushes
// before being redistributed ; therefore, we need one extra guard cell
// (the particles may move by 2*c*dt)
int ngx_tmp = (max_level > 0 && do_subcycling == 1) ? nox+1 : nox;
int ngy_tmp = (max_level > 0 && do_subcycling == 1) ? nox+1 : nox;
int ngz_tmp = (max_level > 0 && do_subcycling == 1) ? nox+1 : nox;
int ngx_tmp = (max_level > 0 && do_subcycling) ? nox+1 : nox;
int ngy_tmp = (max_level > 0 && do_subcycling) ? nox+1 : nox;
int ngz_tmp = (max_level > 0 && do_subcycling) ? nox+1 : nox;

const bool galilean = (v_galilean[0] != 0. || v_galilean[1] != 0. || v_galilean[2] != 0.);
const bool comoving = (v_comoving[0] != 0. || v_comoving[1] != 0. || v_comoving[2] != 0.);
Expand Down
Loading

0 comments on commit 5be2c10

Please sign in to comment.