Restarts from different frames get confused about _dt.bp and Dynvector data structures

[JunoRavin]

Oftentimes, a Vlasov simulation might fail due to too low of collisionality. In these cases the last output is usually garbage (including the restart frame for all Dynvector quantities) and not what you want to restart from, so you rename a frame that looks reasonable to have the suffix restart, and then restart the simulation at higher collisionality.

This worked before when all Dynvector quantities (and cflFrac) were written out as separate files, but now that they are a single file it does not work.

I am not immediately sure what the solution should be. Perhaps an option to generate completely new dynvector files and leave it to the user to merge them together in post-processing. I am currently testing if I just comment out the read of the dynVector restart file if I can at least restart and take time-steps, but a more elegant solution is likely desired.

I am still trying to figure out if the last saved time-step size is also an issue, as the *dt.bp file appears to be empty. ReadRestart is already doing a dummy forwardEuler call. Perhaps a solution is to have an option to just get the size of the stable time-step from this dummy forwardEuler call if no *dt.bp file is detected

[ammarhakim]

Is a solution to store more than 1 restart data sets? So say last 3 datasets are stored in case you want to go back to an older set. Also, we could write a Tool to create restart files from a given set of BP files, trying to fill-in values which are missing.

[manauref]

I think keeping more than 1 restart data set is wise. We did in GDB and I believe GENE does it too, in case one of your restart sets gets corrupted or lost. I also think we should try to write a tool that creates a restart dataset from any given frame. I tried to write a python script that does it, but the fact that my python could not use the ADIOS MPI method (it only seems to work with POSIX1) lead to slightly wrong restarts (at first look they seem fine, but there were small errors). So I would encourage someone else to try. Or maybe switch to ADIOS2....

…

On Sun, 4 Oct 2020 at 12:11, Ammar ***@***.***> wrote: Is a solution to store more than 1 restart data sets? So say last 3 datasets are stored in case you want to go back to an older set. Also, we could write a Tool to create restart files from a given set of BP files, trying to fill-in values which are missing. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ6DKAH7HWR5SUG4N3NNQTSJDCGDANCNFSM4SD2UYLA> .

[JunoRavin]

I would vote for a tool to create restarts from a set of frames (perhaps this tool could open any time-history data and figure out what the time of the restart is, then copy the value over to the *_restart.bp dynvector file). I don't think saving multiple restart frames is necessary because I don't know a priori which frame I want to go back to. For example, in the shock simulations that are going bad, I'm choosing to go 5 frames back to be safe and try to give the higher collisionality more time to smooth out the structures I know are causing issues.

I also want to consult with Noah if it's possible to just fetch the size of the time-step from a dummy forwardEuler call at restart as opposed to relying on the *dt.bp file.

[nmandell]

Restarting from renamed non-restart files is no longer possible in GK, because for restart ghost cells need to get written when using sheath BCs (otherwise the BCs cannot be set properly on restart). I think the best option is having a few restart frames. I’m not sure about the best way to get dt on restart. But why is it needed? It should just be calculated normally on the first timestep.

…

On Oct 4, 2020, at 2:38 PM, James Juno ***@***.***> wrote: I would vote for a tool to create restarts from a set of frames (perhaps this tool could open any time-history data and figure out what the time of the restart is, then copy the value over to the *_restart.bp dynvector file). I don't think saving multiple restart frames is necessary because I don't know a priori which frame I want to go back to. For example, in the shock simulations that are going bad, I'm choosing to go 5 frames back to be safe and try to give the higher collisionality more time to smooth out the structures I know are causing issues. I also want to consult with Noah if it's possible to just fetch the size of the time-step from a dummy forwardEuler call at restart as opposed to relying on the *dt.bp file. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHRISUUKJN6OOURG4TWZVLSJDFLZANCNFSM4SD2UYLA>.

[JunoRavin]

dt is definitely needed for Vlasov restarts, but the system is expecting to read in a file with suffix *dt.bp here in the readRestart method:

  dtTracker:read(string.format("dt.bp"))
  local _, dtLast = dtTracker:lastData()

and then this is being passed to the setDtGlobal method. But in readRestart there's already a dummy forwardEuler method being called to set up the auxFields for some BCs, so I would think we could just use the dt from this.

[JunoRavin]

Reviving this issue because it bit me in the butt again that we have not solved this. I propose that instead of writing multiple restart files, a gkyl tool is developed that reads in a specified frame on a specified number of MPI processes and does the following sequence of steps:

1. Synchronizes the ghost cells in the outer boundary
2. Reads in the time that the files were output at, then opens the integrated diagnostics and looks for the closest time
3. Copies the output into files for restarting integrated diagnostics (such as *_fieldEnergy.bp)
4. Writes out the rest of the data to files with the appropriate suffixes (with writeGhost=true)

GK folks (@nmandell @manauref) would this tool allow GK simulations to restart from an arbitrary frame, or is there additional data that is needed?

I would also like to figure out if the *dt.bp file can be deprecated since we do dummy forwardEuler calls anyway to do things like set up boundary conditions. GK folks thoughts on this would be appreciated.

[nmandell]

GK folks (@nmandell @manauref) would this tool allow GK simulations to restart from an arbitrary frame, or is there additional data that is needed?

No, GK restarts need ghost cells, which are not usually written for regular output files

I would also like to figure out if the *dt.bp file can be deprecated since we do dummy forwardEuler calls anyway to do things like set up boundary conditions. GK folks thoughts on this would be appreciated.

In GK, *dt.bp is not used for restart. on restart the timestep is computed self-consistently. there is no information needed from previous steps needed to compute the timestep (unlike in vlasov? where amax is needed from previous step?). I do however like to have the *dt.bp file to look at as a diagnostic, to see when the timestep has dropped

[JunoRavin]

Can you compute the ghost cells from the output? If I read in the whole file, can I now compute the ghost cells and then re-write out the file with writeGhost=true?

Edit: ah yes, we should not get rid of the *dt.bp file. I am wondering if the same thing can be done for *dt.bp file as for integrated diagnostics. Apologies I misunderstood that *dt.bp has the time-step for every time-step and thought it was just the last dt which was needed for something like the positivity fix originally.

[manauref]

@nmadell which ghost cells can/do we currently not recalculate upon restart, such that we need to save them? Seems like a solvable problem (incomplete understanding here speaking...). I do agree that I like the dt.bp file. We could conceivably make it something we request, instead of something written out by default.

…

On Mon, 22 Feb 2021 at 17:43, James Juno ***@***.***> wrote: Can you compute the ghost cells from the output? If I read in the whole file, can I now compute the ghost cells and then re-write out the file with writeGhost=true? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ6DKBROHSVFMFTTNPS6KDTAKCRPANCNFSM4SD2UYLA> .

[nmandell]

no, I don't think so. when the ghost cells are set by sheath BCs, they depend on the potential at each RK stage. on restart we can only get the potential at the end of the timestep, not the intermediate RK stages.

[JunoRavin]

That's interesting that you need the potential at each RK stage. At some point I would love to know more about how that works (does your thesis go into this? I realize this is a pretty nitty gritty detail on the sheath BC). I did not realize a boundary condition could depend on the individual intermediate stages in a non-trivial way.

Is this just needed for the potential? Is there any reason not to have the potential at a given I/O frame written out with writeGhost=true? I realize that's somewhat inelegant, but it doesn't seem that cumbersome to me.

I think I would like to add this tool anyway just as a way to fix Vlasov and fluid restarts. I like that *dt.bp and the integrated moments are a single file now, but being able to restart from an arbitrary frame is really useful on the Vlasov/fluids end for robustness reasons.

[manauref]

The sequence of events in a simulation with one time step and 2-stage RK stepper, followed by a restart can be sketched as follows: sim1: RK stage1: forwarEuler: 1m: calcMoments 1p: calcPhi 1f: advance distF 1bc: applyBc to 1f using 1p RK stage2: forwarEuler: 2m: calcMoments 2p: calcPhi 2f: advance distF 2bc: applyBc to 2f using 2p combine: fnew = a*1f + b * 2f

…

----- sim1 ends ----- restart: read in: fnew 2p (potential) RK stage1: . . .

----- restart ends ----- So upon restart we read fnew's ghost cells, and we don't apply BCs (until after the 1st RK stage). These BCs were originally applied to an intra-RK distribution function using it's corresponding potential. However at the beginning of a restart we don't have intra-RK information, just the final distribution function and whatever potential we chose to write out.

On Mon, 22 Feb 2021 at 19:47, James Juno ***@***.***> wrote: That's interesting that you need the potential at each RK stage. At some point I would love to know more about how that works (does your thesis go into this? I realize this is a pretty nitty gritty detail on the sheath BC). I did not realize a boundary condition could depend on the individual intermediate stages in a non-trivial way. Is this just needed for the potential? Is there any reason not to have the potential at a given I/O frame written out with writeGhost=true? I realize that's somewhat inelegant, but it doesn't seem that cumbersome to me. I think I would like to add this tool anyway just as a way to fix Vlasov and fluid restarts. I like that *dt.bp and the integrated moments are a single file now, but being able to restart from an arbitrary frame is really useful on the Vlasov/fluids end for robustness reasons. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ6DKA6HBJYIPFPJXRWITDTAKRCLANCNFSM4SD2UYLA> .