Merge pull request #584 from oahull0112/gh-pages

Update and reformat performance recommendations

docs/Documentation/Systems/Kestrel/running.md

@@ -80,40 +80,8 @@ In the shared node partition, the value for `Number of Nodes` can be a fraction
     1 hour walltime * 0.5 nodes * 1 QoS Factor * 10 Charge Factor = 5 AUs
     ```
 
-## Job Submission Recommendations
+## Performance Recommendations
 
-#### OpenMP
-
-When running codes with OpenMP enabled, we recommend manually setting one of the following environment variables:
-
-```
-
-export OMP_PROC_BIND=spread # for non-intel built codes
-
-export KMP_AFFINITY=balanced # for codes built with intel compilers
-
-```
-You may need to export these variables even if you are not running your job with threading, i.e., with `OMP_NUM_THREADS=1`
-
-#### Scaling
-
-Currently, some applications on Kestrel are not scaling with the expected performance. We are actively working with the vendor's engineers to resolve these issues. For now, for these applications, we recommend:
-
-1. Setting the following envrionment variables (for Cray MPICH):
-```
-export MPICH_SHARED_MEM_COLL_OPT=mpi_bcast,mpi_barrier 
-export MPICH_COLL_OPT_OFF=mpi_allreduce 
-```
-
-2. Submitting jobs with the fewest number of nodes possible.
-
-3. For hybrid MPI/OpenMP codes, requesting more threads per task than you tend to request on Eagle. This may yield performance improvements.
-
-4. Building and running with Cray MPICH (or Intel MPI/cray-mpich-abi), rather than OpenMPI.
-
-5. Half-packing the nodes you request, i.e., requesting 52 cores per node instead of 104 cores per node, then spreading these cores evenly across the two sockets. This can be accomplished by including the following flags in your srun command:
-   ```
-   --ntasks-per-node=52 --distribution=cyclic:cyclic --cpu_bind=cores
-   ```
+Please see [this page](../eagle_to_kestrel_transition.md#5-performance-recommendations) for our most up-to-date performance recommendations on Kestrel.
 
 

docs/Documentation/Systems/eagle_to_kestrel_transition.md

@@ -17,6 +17,7 @@ This page is meant to provide all necessary information to transition a project
     c. General environment recommendations
 
 4. Submitting your jobs on Kestrel
+5. Review performance recommendations if scalability or performance is worse than expected
 
 If you find yourself stuck on any of the above steps, please reach out to [email protected] as soon as possible.
 
@@ -118,26 +119,26 @@ You may need to export these variables even if you are not running your job with
 
 ### MPI
 
-Currently, some applications on Kestrel are not scaling with the expected performance. We are actively working with the vendor's engineers to resolve these issues. For now, for these applications, we recommend:
+Currently, some applications on Kestrel are not scaling with the expected performance. We are actively working with the vendor's engineers to resolve these issues. For now, for these applications, we have compiled a set of recommendations that may help with performance. Note that any given recommendation may or may not apply to your specific application. We strongly recommend conducting your own performance and scalability tests on your performance-critical codes.
 
-1. Setting the following envrionment variables (for Cray MPICH):
+1. Use Cray MPICH over OpenMPI or Intel MPI. If you need help rebuilding your code so that it uses Cray MPICH, please reach out to [email protected]
+
+2. For MPI collectives-heavy applications, setting the following environment variables (for Cray MPICH):
 ```
 export MPICH_SHARED_MEM_COLL_OPT=mpi_bcast,mpi_barrier 
 export MPICH_COLL_OPT_OFF=mpi_allreduce 
 ```
+These environment variables turn off some collective optimizations that we have seen can cause slowdowns. For more information on these environment variables, visit HPE's documentation site [here](https://cpe.ext.hpe.com/docs/mpt/mpich/intro_mpi_ucx.html).
 
-2. Submitting jobs with the fewest number of nodes possible.
-
-3. For hybrid MPI/OpenMP codes, requesting more threads per task than you tend to request on Eagle. This may yield performance improvements.
+4. For hybrid MPI/OpenMP codes, requesting more threads per task than you tend to request on Eagle. This may yield performance improvements.
 
-4. Building and running with Cray MPICH (or Intel MPI/cray-mpich-abi), rather than OpenMPI.
-
-5. Half-packing the nodes you request, i.e., requesting 52 cores per node instead of 104 cores per node, then spreading these cores evenly across the two sockets. This can be accomplished by including the following flags in your srun command:
-
+6. ONLY if you are running on 10 or more nodes and are experiencing scalability issues, you can try half-packing the nodes you request, i.e., requesting 52 ranks per node instead of 104 ranks per node, then spreading these ranks evenly across the two sockets. This can be accomplished by including the following in your srun command:   
 ```
 --ntasks-per-node=52 --distribution=cyclic:cyclic --cpu_bind=cores
 ```
 
+Please note that all of these recommendations are subject to change as we continue to improve the system.
+
 ## 6. Resources
 
 1. [Accessing Kestrel](./Kestrel/index.md)