Documentation on binary file serialisation

[tjgalvin]

I see in the FILE_FORMATS.md file that there are descriptions of the header information of each type of binary file.

Can we expand this to clearly specify the order in which the corresponding arrays (SOMs, neurons, transforms etc) are written to this files? Although that are example codes that show examples of reading these they can be a little difficult to follow and transplant into other codes.

[BerndDoser]

The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.

Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.

Changes in the data format should only be done in a major release, once we have a stable specification.

[tjgalvin]

The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.

Yes - the hexagonal scheme is a bit of a pain with the current infrastructure. I have taken to ignoring it completely. This makes my life a little simpler. I’m happy to continue to use the current set of binary files until something is better developed, no problem at all. Its just a case that in order to correctly reshape the files into the corresponding tensors (in my case the cartesian type SOMs) I need to know in what order floats (or other data types) were serialised into the binary file. Is it simply the case that each of the arrays follow the the order of the dimensions in the header between fields like the `som_layout` and `neuron_layout`? For the SOM file type it is easy enough to just plot and see the neurons reconstructed with correct proportions and layouts, but will become a lot harder for other file types (i.e. transform file) where I can’t simply eyeball a few neurons.

Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.

FITS it the obvious one, but HDF5 would also be good. I’m very much interested in things like this: https://arxiv.org/abs/1505.06421 <https://arxiv.org/abs/1505.06421> How widespread the libraries are is another question.

…

On 3 Dec 2019, at 4:45 pm, Bernd Doser ***@***.***> wrote: The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided. Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose. Changes in the data format should only be done in a major release, once we have a stable specification. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ACQOAJSDN7SNYQ3UDXMLJTDQWYMEFA5CNFSM4JTPOKV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFYR64I#issuecomment-561061745>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQOAJUE6756272BRKQO273QWYMEFANCNFSM4JTPOKVQ>.

[RafaelMostert]

Yes - the hexagonal scheme is a bit of a pain with the current infrastructure. I have taken to ignoring it completely. This makes my life a little simpler.

I did the same almost from the start. That makes life easier during debugging, easier during further visualization steps and easier when the SOM BMU's are folded into further ML techniques (think random forests or auto-encoders).

Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.

HDF5 is vastly superior to FITS (better python/pandas integration, faster I/O, smaller files) although both are better than XML and JSON.
Having the option to also store the WCS parameters for the cutouts in the HDF5 as suggested by the paper that Tim links to would be nice.

[BerndDoser]

Thanks for your valuable notes and suggestions. I have added some more details of the memory layout to https://github.com/HITS-AIN/PINK/blob/master/FILE_FORMATS.md#cartesian-layout. Indeed the hexagonal layout is a bit fiddly. Maybe the python interface using numpy can be helpful.

The suggested paper about HDFITS is very promising. I will think about it, how we can use it not only for the input files, but also for the result files (SOM and mapping).