Long time to save nc simulation

[gab1092]

Hello! I have a doub, I'm running a simulation with 1000 elements for gas and 3000 for oil, the simulation run good but when I want save the data in a netcdf file the procces run during hours and I don't know if is normal. Is normal that save the output take many hours? I used the save_sim option to save the data. I need use less elements?
Thank you for your help.

[socolofs]

I normally do not run with that many elements, and I do not use the save_sim function very often. Can you save the data if you reduce to 5 elements each? If that works, then I guess it takes a while to save the data. Generally, it should not take longer than the simulation takes to run. Scott

…

----------- Scott A. Socolofsky, J. Walter “Deak” Porter ’22 and James W. “Bud” Porter ’51 Chair, Zachry Department of Civil and Environmental Engineering Department of Oceanography Department of Ocean Engineering Texas A&M University (979) 845-4517 http://ceprofs.civil.tamu.edu/ssocolofsky ----------- On Dec 22, 2021, at 5:58 PM, Gabriela Reséndiz Colorado ***@***.******@***.***>> wrote: This Message Is From an External Sender This message came from outside your organization. Hello! I have a doub, I'm running a simulation with 1000 elements for gas and 3000 for oil, the simulation run good but when I want save the data in a netcdf file the procces run during hours and I don't know if is normal. Is normal that save the output take many hours? I used the save_sim option to save the data. I need use less elements? Thank you for your help. — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/socolofs/tamoc/issues/18__;!!KwNVnqRv!XkEcbDcbG4IrIfxDNXfooW5RQ1ibaJePLkt3RflP2zRxvL2s3UqVA9n4mg8zg6W3$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AA6XESDZBKZA6MHGI6COND3USJQ2RANCNFSM5KTTWHBQ__;!!KwNVnqRv!XkEcbDcbG4IrIfxDNXfooW5RQ1ibaJePLkt3RflP2zRxvL2s3UqVA9n4ml_MOSnx$>. Triage notifications on the go with GitHub Mobile for iOS<https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!KwNVnqRv!XkEcbDcbG4IrIfxDNXfooW5RQ1ibaJePLkt3RflP2zRxvL2s3UqVA9n4mmuAiIyP$> or Android<https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!KwNVnqRv!XkEcbDcbG4IrIfxDNXfooW5RQ1ibaJePLkt3RflP2zRxvL2s3UqVA9n4mlvo_b_9$>. You are receiving this because you are subscribed to this thread.

[ChrisBarker-NOAA]

Is there any advantage to running that many elements in TAMOC?

Anyway -- how is the output stored in netCDF?. I have had issues with really slow (and huge) netcdf files if the data aren't stored in an efficient manner.

from a quick look at save_particle_to_nc_file in dispersed_phases.py, it looks like the data are stored in large 1D arrays, which used to be painfully bad in netcdf4, but I think that's been fixed.

However, it looks like the code loops through each particle and writes the data to the netcdf file one by one -- that could be the issue here. It is far more efficient to write an array of data at once.

you wouldn't notice with, say, < 10 particles, but with thousands, that could explain it.

If there is a use case for that many particles, perhaps the code could be refactored to save the data out a full array at at time.

[gab1092]

@socolofs Thank you! I tried with fewer elements; as you said, the problem is with many elements.
@ChrisBarker-NOAA I probed with a differing number of elements; in a quick revision, I didn't find any advantage to simulate with many elements for my simulation. I will simulate with a few elements. I will try to use the output of TAMOC as the initial condition for a simulation of surface spill using GNOME. Hence, I tried with a similar number of elements in the two simulations as an initial phase.
Thank you for the help, and happy Christmas to all!

[ChrisBarker-NOAA]

Ahh -- yes, that's a trick. It may make sense to create thousand of GNOME elements from a few TAMOC elements. You only need many elements if they are going to do something different :-) In GNOME, diffusion will spread them out, and then they will be subject to different currents, winds, etc -- so you may need a lot.