DynVector dataset shapes sometimes change upon restart

[manauref]

The dataset inside a DynVector sometimes change shape after a restart. Noah tried to prepare postgkyl to handle this (postgkyl commit 89dbc7d5936bd31a4a26e0857bd5261f63cd996f), but it doesn't seem to always work.

This leads to errors like

pgkyl -f gk24-wham1x2v_esEnergy.bp pl
...
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 1 dimension(s)

and inspecting the file with bpls we find

bpls gk24-wham1x2v_esEnergy.bp
...
  double   TimeMesh713   {2292}
  double   Data713       {2292}
  double   TimeMesh714   {2292}
  double   Data714       {2292, 1}

Also, I've also seen the insertion of empty datasets like:

bpls gk24-wham1x2v_elc_intM0.bp
...
  double   TimeMesh1019  {0}
  double   Data1019      {0, 1}
  double   TimeMesh1020  {1}
  double   Data1020      {1, 1}
  double   TimeMesh1021  {1}
  double   Data1021      {1, 1}
  double   TimeMesh1022  {0}
  double   Data1022      {0, 1}

but I'm not sure if this is related.

[nmandell]

What’s strange is that I think I was seeing the opposite issue when I made the postgkyl change. The Data* always had two dimensions (with 1 element in the second dimension), except after a restart there was a frame with only one dimension. bpls rt-gk-sheath_esEnergy.bp double TimeMesh0 {1} double Data0 {1, 1} double TimeMesh1 {110} double Data1 {110, 1} And after restart it would have given something like Data2 {100}. This is what the postgkyl work-around fixes.

…

On Feb 2, 2021, at 10:31 AM, Manaure Francisquez ***@***.***> wrote: The dataset inside a DynVector sometimes change shape after a restart. Noah tried to prepare postgkyl to handle this (postgkyl commit 89dbc7d5936bd31a4a26e0857bd5261f63cd996f), but it doesn't seem to always work. This leads to errors like pgkyl -f gk24-wham1x2v_esEnergy.bp pl ... ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 2 dimension(s) and the array at index 1 has 1 dimension(s) and inspecting the file with bpls we find bpls gk24-wham1x2v_esEnergy.bp ... double TimeMesh713 {2292} double Data713 {2292} double TimeMesh714 {2292} double Data714 {2292, 1} Also, I've also seen the insertion of empty datasets like: bpls gk24-wham1x2v_elc_intM0.bp ... double TimeMesh1019 {0} double Data1019 {0, 1} double TimeMesh1020 {1} double Data1020 {1, 1} double TimeMesh1021 {1} double Data1021 {1, 1} double TimeMesh1022 {0} double Data1022 {0, 1} but I'm not sure if this is related. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[manauref]

It seems that the empty data sets can happen even at the beginning of a run, not just upon restart. For example

bpls gk30-wham1x2v_esEnergy.bp
  double   TimeMesh0    {1}
  double   Data0        {1, 1}
  double   TimeMesh1    {0}
  double   Data1        {0, 1}

and more recently I've seen the appearance of scalars:

  double   TimeMesh224  {1}
  double   Data224      {1, 1}
  double   TimeMesh225  {0}
  double   Data225      {0}
  double   TimeMesh226  {0}
  double   Data226      {0}
  double   TimeMesh227  scalar
  double   Data227      scalar

This, by way, makes the whole file useless from a pgkyl point of view. It'd require writing a script to read the data.

[manauref]

Commit 12acfbd seems to fix the insertion of empty datasets in DynVectors. We were trying out output data more frequently than it was being calculated. So the solution is to use a max of calcIntQuantEvery and nFrame, so that we compute integrated quantities as frequently as we output data.

[Zhuo-Liu]

I also encountered this issue on Cori with gkyl I pulled on 03/11/2021. This issue seems to happen after a restart.

[manauref]

I found a way to reproduce this problem of the appearance of the scalars, looping over a file with a bash script that performs many restarts. At around the 30th restart the scalar appears.

scalarProblem.zip

I have also determined the following, although I don't see how any of these could happen:

1. If the dynVector is flushed/cleared, and we call :write before any new data is added to it, the empty datasets occur (e.g. {0} and {0, 1}).
2. I think if self._data:size() returns nil in 342-343 of DynVector.lua, I believe an empty string is returned and the result is a scalar.