VlsvReader does not open new vlsv files

[sandroos]

Due to the multipop development the way how velocity mesh data is stored in vlsv files was changed to be more general. VlsvReader classes in python tools do not open the new files any more. The velocity meshes are now stored in the same way as spatial mesh(es).

In [53]: f = pt.vlsvfile.VlsvReader(file_name='fullf.0000000.vlsv')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-53-ce2a08adf8d2> in <module>()
----> 1 f = pt.vlsvfile.VlsvReader(file_name='fullf.0000000.vlsv')

/home/sandroos/codes/pythontools/pyVlsv/vlsvreader.py in __init__(self, file_name)
     35       #self.__read_fileindex_for_cellid()
     36       # Read parameters (Note: Reading the spatial cell locations and storing them will anyway take the most time and memory):
---> 37       self.__vxblocks = (int)(self.read_parameter("vxblocks_ini"))
     38       self.__vyblocks = (int)(self.read_parameter("vyblocks_ini"))
     39       self.__vzblocks = (int)(self.read_parameter("vzblocks_ini"))

TypeError: int() argument must be a string or a number, not 'NoneType'

Parameters vxblocks_ini, vyblocks_ini, vzblocks_ini are no longer used. This information is read from the MESH_BBOX arrays that contain mesh metadata.

Here's a recap of the new format:

• vlsv files contain 0 or more velocity meshes
• each mesh is stored analogous to spatial mesh
• velocity mesh arrays have attribute mesh=<species name> or name=<species name>

<MESH_BBOX arraysize="6" datasize="8" datatype="uint" max_velocity_ref_level="0" mesh="Electron" type="amr_ucd" vectorsize="1">2980460</MESH_BBOX>
<MESH_BBOX arraysize="6" datasize="8" datatype="uint" max_velocity_ref_level="0" mesh="Proton" type="amr_ucd" vectorsize="1">2993364</MESH_BBOX>

For example, the bounding box arrays above define the velocity mesh extents (which used to be given in vxblocks_ini etc) for particle species Electron and Proton.

(This came up when I was trying to use analyzator to extract time series from n vlsv files, but ofc that didn't work out)

TODO:

• Add interface for getting the mesh metadata stored in__xmin etc
• convert all places in the code where xmin, vxmin and other mesh metadatas are read driectly from the file

[ohannuks]

Cheers for the report, Ill look into it

[ohannuks]

Is this still an issue? Im not sure if I have the time to work on it.

I'm asking because I think there was a similar problem with vlsvextract, which led some people to believe this was a result of a bug.

[sandroos]

Yes this is still an issue, those function calls above are in python code and parameters vxblocks_ini etc. do not exist in new Vlasiator files.

[galfthan]

Master still writes out vxbloxks_ini, etc. In addition it also writes out the new velocity space metadata. Thus analysator still understands the distribution function data of the vlasiator files.

The multipop branch only writes out hte new metadata form. Before merging that it would be good to fix this issue. It is important to still support also the old format though.

[sandroos]

In multipop it doesn't make sense to write vxblocks_ini etc. because there are multiple velocity meshes. Would it be possible to probe the existence of those parameters in Python code and only read them if they are defined?

[galfthan]

I guess so. If the new one is defined then that metadata, otherwise old style is read.

[rjarvinen]

Any fixes or workarounds for this issue? I get the same error when trying to use VlsvReader on vlsv files from the parallel hybrid code.

[ykempf]

No, this has not been fixed at the moment. Based on Arto's comments above one should be able to read the MESH_BBOX and pick the three correct values.

[rjarvinen]

I think none of the parameters required by VlsvReader (lines 37-57) are available in the current vlsv format. https://github.com/fmihpc/analysator/blob/master/pyVlsv/vlsvreader.py#L37

My MESH_BBOX looks like:

<MESH_BBOX arraysize="6" datasize="4" datatype="uint" mesh="SpatialGrid" vectorsize="1">28</MESH_BBOX>

[galfthan]

I'll take a look at this now

[galfthan]

Fixed the mesh metadata reading bug. It should read both old and new vlasiator files. This has not been verified thoroughly.

@rjarvinen, you can test the fix_issue22 branch.

I tested with BCS files, and one could not plot the velocity distribution function with the picker. This, I think, is a separate issue though.

Also, to support multiple populations one would need to do more work, now it assumes there is only one "avgs" population.

[ykempf]

I have the fix for that picker somewhere, I haven't had time to go through everything to see whether it is a large-scale problem or only there.

[galfthan]

I already started some work on this, did not want to commit this unfinished thing here. In any case I guess that should be done in a separate branch

[ykempf]

Hi, my fixes for the vdf picker errors are in the branch https://github.com/fmihpc/analysator/tree/picker_fix now.
As I put in the commit message, it is probably worth checking whether any other such function calls need to specify the arguments in a less ambiguous way.

[galfthan]

I fixed another place where avgs are also read in the picker_fix branch

[sandroos]

@rjarvinen You were trying to open VLSV files from Corsair using Vlasiator analysator? These have never been exactly compatible. However, the required values should be stored in the Corsair output files as well.

[galfthan]

Most of the things which are not related to velocity space might work, that could at least be a good target. There has been/is a lot of vlasiator specific things there though this far.

[galfthan]

@ykempf, can you test the issue22 branch with multipop files and report the results? If that works, then we can merge this branch. Further incompabilities with @rjarvinen parallel hyb code could be handled by a separate issue.

[galfthan]

For the picker I opened a separate bug, #25 , let's discuss those things there.

[rjarvinen]

@galfthan Great, thanks! fix_issue22 branch can read the files from Corsair. Not everything works yet as you mentioned. E.g. calculations.cut_through tries to read xmin etc. using VlsvReader.read_parameter function, which fails of course for the new style vlsv files. It seems straightforward to fix these based on your update.

@sandroos Yes, I'm reading VLSV files from Corsair. I guess there are no fundamental reasons why I shouldn't do this?

[sandroos]

@rjarvinen I'll provide a better answer to the file issue now that I have more time:

Analysator assumes that VLSV files contain information about (one) spatial mesh and (one) velocity mesh. However, Corsair files typically do not contain velocity distributions like Vlasiator files do.

https://github.com/fmihpc/analysator/blob/master/pyVlsv/vlsvreader.py#L37

Some of the parameters read above (vxblocks_ini, vx_min, etc) are specific to Vlasiator-type velocity distributions. If Analysator was improved so that it

• checks if the input files contain velocity mesh(es)
• reads velocity mesh data and related parameters only if the file contains them

then I think Analysator would work with Corsair VLSV files as well.

[galfthan]

These xmins are scattered around in many places:

miscellaneous/rankine.py: xmin = vlsvReader.read_parameter("xmin")
pyCalculations/cut3d.py: xmin = vlsvReader.read_parameter(name="xmin")
pyCalculations/cutthrough.py: xmin = vlsvReader.read_parameter(name="xmin")
pyCalculations/fieldtracer.py: xmin = f.read_parameter('xmin')
pyCalculations/themis_observation.py: vxmin = vlsvReader.read_parameter("vxmin")
pyMayaVi/mayavigrid.py: xmin = self.vlsvReader.read_parameter("xmin")
pyMayaVi/mayavigrid.py: xmin = self.vlsvReader.read_parameter("xmin")
pyMayaVi/mayavigrid.py: mins = np.array([self.vlsvReader.read_parameter("xmin"), self.vlsvReader.read_parameter("ymin"), self.vlsvReader.read_parameter("zmin")])
pyVlsv/vlsvreader.py: self.__xmin = self.read_parameter("xmin")
pyVlsv/vlsvreader.py: self.__vxmin = self.read_parameter("vxmin")
trash_can/backstream.py: vxmin = vlsvReader.read_parameter("vxmin")
trash_can/fourier2d.py: xmin = vlsvReader.read_parameter(name="xmin")
trash_can/makeamovieauto.py: xmin = vlsvReader.read_parameter(name="xmin")
trash_can/spacecraft.py: self.__xmin = self.__vlsvfile.read_parameter(name="xmin")
trash_can/vlsvplots.py: xmin = vlsvReader.read_parameter(name="xmin")

[galfthan]

We should add functions to vlsvreader for returning xmin, etc. These are already stored in internal variables __xmin .. with the compability fixes. Better to have these compability things in just one place

[galfthan]

If no avgs mesh is defined in the file, then this branch initializes the velocity mesh to have a maximum size of 0 blocks. Some of the velocity space functions may exit gracefully, at least one could add checks for velocity spaces with a size of 0 x 0 x 0.

If and when analysator properly supports multiple populations then the case with 0 should be taken care of at the same time.

[rjarvinen]

I tried replacing old style xmin etc. calls in cutthrough.py by vlsvReader._VlsvReader__xmin in fix_issue22 branch, and was able to use cut_through function successfully for new style vlsv files from Corsair.

[galfthan]

Good! we should though fix it in the main version by writing a get style function, these __ variables are supposed to be private:
http://stackoverflow.com/questions/1301346/the-meaning-of-a-single-and-a-double-underscore-before-an-object-name-in-python

[galfthan]

I added the interface, now it should be taken in use.

[rjarvinen]

I implemented the new interface in files pyCalculations/*