Different coupling efficiency

[rockowl]

Hi, John,
I run the same ROMS-SWAN case on two HPCs, but one is nearly two times slower. If run with ROMS-only, the efficiency is similar, indicating the slow down is due to coupling. Changing the coupling frequency does not improve the efficiency much. Would you kindly give some suggestions?
Thank you,
Yan

[jcwarner-usgs]

if you look in the standard output, you can see which model gets to the coupling exchange first. this can help to allocate the number of processors. if you attach the output file, i can show you what i mean.

[rockowl]

3935.log
Hi, John, attached please find the log file. Thank you, Yan.

[jcwarner-usgs]

for example,...

+time 20220617.001900 , step 19; iteration 1; sweep 3 grid 1
22896600 2022-06-17 00:20:00.00 1.087553E-02 1.387776E+03 1.387787E+03 2.901361E+12
(655,448,25) 3.995153E-03 1.783547E-03 3.970750E-01 1.493649E+00
+time 20220617.001900 , step 19; iteration 1; sweep 4 grid 1
+time 20220617.002000 , step 20; iteration 1; sweep 1 grid 1
+time 20220617.002000 , step 20; iteration 1; sweep 2 grid 1
+time 20220617.002000 , step 20; iteration 1; sweep 3 grid 1
+time 20220617.002000 , step 20; iteration 1; sweep 4 grid 1 <- this is swan
== SWAN grid 1 sent wave data to ROMS grid 1
** ROMS grid 1 recv data from SWAN grid 1
SWANtoROMS Min/Max DISBOT (Wm-2): 0.000000E+00 8.236075E-04
SWANtoROMS Min/Max DISSURF (Wm-2): 0.000000E+00 0.000000E+00
...

you are only writing out the roms output every 20 steps, but SWAN is the slower model here.
ROMS got to the coupling interval before swan, and then had to wait.
So it is not really 'the coupling' that is slowing it down, although that takes time, here it is swan that could use more processors.
how many procs did you set to roms and how many to swan?

[rockowl]

Hi, John,
The ROMS/SWAN CPUs ratio is 216:72=3:1 on the slow one, and is 192:48=4:1 on the fast one.
Yan

[rockowl]

Hi, John,
I can get optimal numbers for the ROMS/SWAN CPUs ratio, however, that did not change the fact that the coupling case is two times slower than on the other HPC, or the ROMS-only case.

One fact that may be useful is below:
For the ROMS-only case, 3D equations right-side terms .................... 666.069 ( 2.5380 %)
For the ROMS-SWAN case, 3D equations right-side terms .................... 32860.467 (51.8139 %)
The rest items' time scales are close to each other.
Do you have any insight about it?
Thank you,
Yan

[jcwarner-usgs]

well, when you couple roms with swan, there are a lot more physics that can be active in roms, such as bbl model, wec vf forcing, stokes computations, etc. that could slow it all down.