hdf5_fits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table border=\"0\">\n",
    "    <tr>\n",
    "        <td>\n",
    "        <img src=\"./images/FITSlogo.gif\" width=\"300px\">\n",
    "        </td>\n",
    "        <td>\n",
    "        <img src=\"./images/HDFlogo.png\" width=\"300px\">\n",
    "        </td>  \n",
    "        <td>\n",
    "        <img src=\"./images/netcdf.png\" width=\"300px\">\n",
    "        </td>           \n",
    "    </tr>\n",
    "</table>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import time\n",
    "import cv2, wget\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import h5py\n",
    "from tables import *\n",
    "from versioned_hdf5 import VersionedHDF5File\n",
    "\n",
    "import netCDF4\n",
    "\n",
    "from astropy.io import fits\n",
    "from astropy.utils.data import download_file\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in glob.glob(\"*.h5\"):\n",
    "    os.remove(file)\n",
    "for file in glob.glob(\"*.hdf5\"):\n",
    "    os.remove(file)\n",
    "for file in glob.glob(\"*.fits\"):\n",
    "    os.remove(file)   \n",
    "    \n",
    "def print_attrs(name, obj):\n",
    "    # Create indent\n",
    "    shift = name.count('/') * '    '\n",
    "    item_name = name.split(\"/\")[-1]\n",
    "    print(shift + item_name)\n",
    "    try:\n",
    "        for key, val in obj.attrs.items():\n",
    "            print(shift + '    ' + f\"{key}: {val}\")\n",
    "    except:\n",
    "        pass    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparison between FITS vs HDF5 / NetCDF\n",
    "<hr>\n",
    "<i>The purpose of this notebook is to present the comparison between the FITS (Flexible Image Transfport System) file format and the HDF5 (Hierachical Data Format) / the NetCDF (Network Commun Data Form) file formats. The comparison is made according to several criteria: criteria used for data preservation plus a criterion for data processing</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "1. [Criteria for choosing a format](#choose_format)\n",
    "2. [User community / Sociability](#community)\n",
    "3. [Documentation](#documentation)\n",
    "4. [Freedom of use](#freedom)\n",
    "5. [Independence / autonomy](#autonomy)\n",
    "6. [Robustness](#robustness)\n",
    "7. [Compactness](#compactness)\n",
    "8. [Availability tools for processing tools](#availability)\n",
    "9. [Content additional embedded](#content)\n",
    "10. [Simplicity](#simplicity)\n",
    "11. [Stability](#stability)\n",
    "12. [Openness](#openness)\n",
    "13. [Expressivity](#expressivity)\n",
    "14. [Mastery](#mastery)\n",
    "15. [Processing](#processing)\n",
    "16. [Conclusion](#conclusion)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Criteria for choosing a format\n",
    "<a id=\"choose_format\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **User community / Sociability** : This criterion provides information on the use of the format for a given community. Is the format widely used in its target community ? By archive system ? The use of a format within its community is an indication of its suitability for the specific needs of that community. A format that is also used beyond curatorial institutions provides additional assurance. This criterion is related to the availability of tools: the larger the user community, the more likely it is that it has developed or has had tools developed.\n",
    "- **Documentation** : This criterion provides information about the availability of the file format documentation. Are the format specifications published? If so, are they maintained by a recognized standards body ? What is the their cost ? If the specifications of the format are freely available, it is possible accessible, it is possible for anyone to understand the structure and, if the format is also free, to develop tools that support it.\n",
    "- **Freedom of use** : This criterion provides information about legal issues. Are there any legal barriers for the use of the format ? If a format can be totally open (documented and usable by anyone), it also happens that limitations on the use of documented formats documented formats, in particular because of patents granting industrial property rights registered for the benefit of a for the benefit of a given organization. These patents may limit or prohibit the development of of tools that support the format.\n",
    "- **Independence / autonomy** : This criterion provides information about the autonomy of the format. Does the use of the format requires other formats, encodings, software or hardware environment ? The consultation and the use of a digital file are systematically dependent on a technical environment. In addition to the dependence on a software environment which can be proprietary, discussed in the criterion \"Freedom of use\", the use of certain formats is dependent on hardware environments, software libraries, or elements libraries, or elements that are not usually embedded in the file (for example, the dependence of most dependency of most PDFs on the fonts installed on the user's computer).\n",
    "- **Robustness** : This criterion provides information about the ability of the file format to resist to changes and the different means to validate the changes. Does the format have mechanisms to detect, ignore, or even correct corrupted portions of the signal? This criterion evaluates the resistance of files in this format to corruption. Corruption can be caused by media degradation, but even more often it is caused by transfer interruption, such as network or connection failure. As a result, file formats to be exchanged over the network via streaming are often designed to be robust. Robustness includes the concepts of resilience and error resistance. It depends on the structure of the format. It can be reinforced by the presence of digital fingerprints characterizing each zone of a file, which allows a validation tool to identify precisely the corrupted zone.\n",
    "- **Compactness** : This criterion provides information about the compression. Does the format express a significant amount of information in a constrained space ? If this compactness is linked to a method of compression method, is it reversible (without loss of information) or not ? One of the major risks to the sustainability of data is the is the budgetary risk. If the budgets that the user can allocate to the purchase of storage space, the are limited, the criterion of compactness can become decisive. Compactness can be linked to the data encoding structure or, if applicable, to the structure of the data or, if applicable, the method of compression. Lossless compression, also known as \"reversible lossless compression, also known as \"reversible\" compression, generally allows a significant reduction in file size, while ensuring that the guaranteeing the possibility, thanks to the same method to decompress the file and obtain a bit-accurate copy of the exact copy of the source file, down to the bit.\n",
    "- **Availability tools for processing tools** : This criterion provides information about the availability of tools to handle the file format. Are there any tools for validation, analysis, reading ? Is the organization responsible for format maintenance officially develop any ?\n",
    "- **Content additional embedded** : This criterion provides information about the different contents that can be included in the file format. Does the format allow to embed additional flows necessary for the use, identification and management of the file (metadata, documentation, associated visuals, etc.) ?\n",
    "- **Simplicity** : This crierion provides information about the simplicity of the data structure of the file format. Does the format have a simple or complex structure ? Maintaining skills and tools in a complex format will necessarily require a higher investment than investment than on a simple format\n",
    "- **Stability / evolutivity** : This citerion provides information about the stability of the file format. Is the format undergoing a sustained evolution and versions that follow one another with succeed one another at a high frequency? \n",
    "- **Openness** : This criterion provides information about the ability to read easily the file format. Is the format easily readable and understandable or is structure opaque? In the absence of specific tools, a relatively transparent format can be more easily interpreted and understood by a human interpreted and understood by a human using generic tools generic tools such as text, XML or hexadecimal editors. hexadecimal editors.\n",
    "- **Expressivity** : This criterion provides information about the ability to store any information in the file format. Does the format allow to encode all the information that the PDC wishes to express ?\n",
    "- **Mastery** : This criterion provides information about the ability of the community to master the file format. Does the user already have the skills and tools necessary for an accurate exploitation the format ? Because the PDC is entrusting their data to ESA, it is important that the format be controlled by both parties.\n",
    "- **Processing** : This criterion provides information about the I/O performances of the file format. How fast to extract data and manipulate data from the file format ?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - User community / Sociability\n",
    "<a id=\"community\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS is used traditionally in most of astronomical projects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2 - HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/HDFcontrib.png\">\n",
    "\n",
    "Gravitational wavelengths and radio astronomy actively use HDF5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS is used by a wider astronomy community.\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| User community / Sociability | +++ | + | + |  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Documentation\n",
    "<a id=\"documentation\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 - FITS & HDF5/NetCDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/fits_netcdf_hdf5_ogc.png\" with=\"250px\">\n",
    "\n",
    "- FITS : https://fits.gsfc.nasa.gov/standard40/fits_standard40aa-le.pdf\n",
    "- HDF5 : https://support.hdfgroup.org/HDF5/doc/H5.format.html\n",
    "- HDF5/OGC : https://www.ogc.org/standards/HDF5\n",
    "- NetCDF : https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_model.html\n",
    "- NetCDF/OGC : https://www.ogc.org/standards/netcdf\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 - Models (Extensions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, it exists only two shared data models : \n",
    "- WCS (Word Coordinate Systems) from Calabretta : https://www.atnf.csiro.au/people/mcalabre/WCS/\n",
    "- Hips (IVOA) : https://ivoa.net/documents/HiPS/20170519/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.2 - HDF5/NetCDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HDF5 data structure can define any data model.But the semantic of each attribute must be defined. \n",
    "That's why HDF5 provides other specifications for specific use cases :\n",
    "\n",
    "- Specification for image : https://support.hdfgroup.org/HDF5/doc/ADGuide/ImageSpec.html\n",
    "- Specification for Table : https://support.hdfgroup.org/HDF5/doc/HL/H5TB_Spec.html\n",
    "- Specification for dimension scale : https://support.hdfgroup.org/HDF5/doc/HL/H5DS_Spec.pdf\n",
    "\n",
    "Like these models do not define a scientific data model, HDF5 reuses **:Conventions = 'xxx'** attribute from NetCDF. Examples of conventions:\n",
    "   * Climate and Forecast : https://cfconventions.org/, (Softwares using CF conventions : https://cfconventions.org/software.html)\n",
    "   * The other conventions : https://www.unidata.ucar.edu/software/netcdf/conventions.html\n",
    "   * ftp.unidata.ucar.edu/pub/netcdf/Conventions/ (registry of conventions)\n",
    "   * Author of new conventions should submit a request to support-netcdf@unidata.ucar.edu for listing on the FTP\n",
    "\n",
    "Several metadata conventions coming from NetCdf can be reused and could be interested for PLATO as technical metadata:\n",
    "- Attribute convention for data discovery : https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_1-3\n",
    "- ISO metadata : https://wiki.esipfed.org/NetCDF,_HDF,_and_ISO_Metadata\n",
    "- Attribute convention for object discovery : https://wiki.esipfed.org/Attribute_Convention_for_Data_Discovery_Object_Conventions\n",
    "\n",
    "\n",
    "<img src=\"https://www.frontiersin.org/files/Articles/437031/fmars-06-00442-HTML/image_m/fmars-06-00442-g002.jpg\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2.1 - Image\n",
    "Below is an example of how to use the image specification in HDF5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.system('curl -O https://web.stanford.edu/~kterao/Homer.png')\n",
    "data=plt.imread('Homer.png')\n",
    "print('Data shape:',data.shape)\n",
    "with h5py.File(\"image.h5\", \"w\") as f:\n",
    "    h = f.create_dataset(\"/images\", data=data)\n",
    "    h.attrs[\"CLASS\"] = np.string_(\"IMAGE\")\n",
    "    h.attrs[\"IMAGE_VERSION\"] = np.string_(\"1.2\")\n",
    "    h.attrs[\"IMAGE_SUBCLASS\"] = np.string_(\"IMAGE_TRUECOLOR\")\n",
    "    h.attrs[\"DISPLAY_ORIGIN\"] = np.string_(\"LL\")\n",
    "    h.attrs[\"IMAGE_WHITE_IS_ZERO\"] = np.uint8(0)\n",
    "    \n",
    "    # display the data structure\n",
    "    f.visititems(print_attrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, read it :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File(\"image.h5\", \"r\") as f:\n",
    "    data = f['images']\n",
    "    plt.imshow(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2.2 - Table\n",
    "Below is an example of how to use the table specification in HDF5. This table specification is implemented by pytables (https://www.pytables.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Particle(IsDescription):\n",
    "    identity = StringCol(itemsize=22, dflt=\" \", pos=0) # character String\n",
    "    idnumber = Int16Col(dflt=1, pos = 1) # short integer\n",
    "    speed = Float32Col(dflt=1, pos = 2) # single-precision\n",
    "    \n",
    "# Open a file in write mode\n",
    "with open_file(\"table.h5\", mode = \"w\") as fileh:\n",
    "\n",
    "    # Get the HDF5 root group\n",
    "    root = fileh.root\n",
    "\n",
    "    # Create the groups\n",
    "    group1 = fileh.create_group(root, \"group1\")\n",
    "    group2 = fileh.create_group(root, \"group2\")\n",
    "\n",
    "    # Now, create an array in root group\n",
    "    array1 = fileh.create_array(root, \"array1\", [\"string\", \"array\"], \"String array\")\n",
    "\n",
    "    # Create 2 new tables in group1\n",
    "    table1 = fileh.create_table(group1, \"table1\", Particle)\n",
    "    table2 = fileh.create_table(\"/group2\", \"table2\", Particle)\n",
    "\n",
    "    # Create the last table in group2\n",
    "    array2 = fileh.create_array(\"/group1\", \"array2\", [1,2,3,4])\n",
    "\n",
    "    # Now, fill the tables\n",
    "    for table in (table1, table2):\n",
    "\n",
    "        # Get the record object associated with the table:\n",
    "        row = table.row\n",
    "\n",
    "        # Fill the table with 10 records\n",
    "        for i in range(10):\n",
    "\n",
    "            # First, assign the values to the Particle record\n",
    "            row['identity'] = 'This is particle: %2d' % (i)\n",
    "            row['idnumber'] = i\n",
    "            row['speed'] = i * 2.\n",
    "\n",
    "            # This injects the Record values\n",
    "            row.append()\n",
    "\n",
    "    # Flush the table buffers\n",
    "    table.flush()\n",
    "\n",
    "# display the data structure\n",
    "with h5py.File(\"table.h5\", \"r\") as f:\n",
    "    f.visititems(print_attrs)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2.2.3 - :Conventions  : CF_conventions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Purpose:\n",
    "#  Demonstrates how to use python to add CF attributes to an h5 file. Creates \n",
    "#  a file with 3 datasets: lat, lon, temp. Lat contains the CF attributes: \n",
    "#  units, long_name, and standard_name. Lon has the same CF attributes as the \n",
    "#  latitude dataset. Temp contains the CF attributes: units, long_name, \n",
    "#  _FillValue, coordinates, valid_min, valid_max, valid_range, scale_factor, \n",
    "#  add_offset. Outputs data to cf_example.h5\n",
    " \n",
    "with h5py.File (\"cf_example.h5\", 'w') as file:\n",
    "    \n",
    "    file.attrs[\":Conventions\"] = \"CF-1.6\"\n",
    "\n",
    "    # initialize temperature array\n",
    "    temp_array = np.ones ((180, 360), 'f')\n",
    "\n",
    "    #  values between a[60][*] and a[120][*] is around 300.0\n",
    "    for x in range (0, 60):\n",
    "        for y in range (0, 360):\n",
    "            temp_array[x][y] = 280.0 \n",
    "\n",
    "    # values between a[0][*] and a[59][*], a[121][*] and a[179][*]\n",
    "    #   is around 280.0\n",
    "        for x in range (60, 121):\n",
    "            for y in range (0, 360):\n",
    "                temp_array[x][y] = 300.0 \n",
    "\n",
    "        for x in range (121, 180):\n",
    "            for y in range (0, 360):\n",
    "                temp_array[x][y] = 280.0\n",
    "\n",
    "    temp_dset = file.create_dataset ('temp', data=temp_array, chunks=(10,10), compression='gzip')\n",
    "\n",
    "    temp_dset.attrs[\"long_name\"] = \"temperature\"\n",
    "    temp_dset.attrs[\"units\"] = \"kelvin\"\n",
    "\n",
    "    vlen = h5py.special_dtype (vlen = str)\n",
    "    temp_dset.attrs.create ('coordinates', data = ['lat', 'lon'], \n",
    "                dtype=vlen) \n",
    "\n",
    "    # must explicitly declare numerical data, or else the datatype is assumed\n",
    "    #   to be F64LE (we want F32)\n",
    "    temp_dset.attrs.create ('valid_min', data=0.0, dtype ='f')\n",
    "    temp_dset.attrs.create ('valid_max', data=400.0, dtype ='f')\n",
    "    vrange =[275, 305]\n",
    "    temp_dset.attrs.create ('valid_range', data=vrange, dtype='f') \n",
    "    temp_dset.attrs.create ('_FillValue', data=-999.0, dtype ='f') \n",
    "    temp_dset.attrs.create ('scale_factor', data=1.0, dtype='f') \n",
    "    temp_dset.attrs.create ('add_offset', data = 10.0, dtype = 'f')\n",
    "\n",
    "\n",
    "    # ************  LATITUDE  ************\n",
    "    lat_array = np.ones (180, 'f') \n",
    "    for x in range (0, 180):\n",
    "        lat_array[x] = -90.0 + x\n",
    "\n",
    "    lat_dset = file.create_dataset ('lat', data = lat_array) \n",
    "    lat_dset.attrs[\"long_name\"] = \"latitude\"\n",
    "    lat_dset.attrs[\"units\"] = \"degrees_north\"\n",
    "    lat_dset.attrs[\"standard_name\"] = \"latitude\"\n",
    "\n",
    "\n",
    "    # ************  LONGITUDE  ***********\n",
    "    lon_array = np.ones (360, 'f') \n",
    "    for x in range (0, 360):\n",
    "        lon_array[x] = -180.0 + x\n",
    "\n",
    "    lon_dset = file.create_dataset ('lon', data = lon_array)\n",
    "    lon_dset.attrs[\"long_name\"] = \"longitude\"\n",
    "    lon_dset.attrs[\"units\"] = \"degrees_east\" \n",
    "    lon_dset.attrs[\"standard_name\"]= \"longitude\"\n",
    "\n",
    "    file.flush()\n",
    "    \n",
    "with h5py.File(\"cf_example.h5\", \"r\") as f:\n",
    "    f.visititems(print_attrs)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now read it :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# need to install https://github.com/matplotlib/basemap/archive/master.zip\n",
    "#from mpl_toolkits.basemap import Basemap\n",
    "#with h5py.File(\"cf_example.h5\", \"r\") as f:\n",
    "#    map = Basemap(projection='merc',llcrnrlon=-30.,llcrnrlat=20.,urcrnrlon=70.,urcrnrlat=70.,resolution='i')  \n",
    "#    map.drawcoastlines()\n",
    "#    map.drawstates()\n",
    "#    map.drawcountries()\n",
    "#    map.drawlsmask(land_color='Linen', ocean_color='#CCFFFF') # can use HTML names or codes for colors\n",
    "#    map.drawcounties()\n",
    "#    parallels = np.arange(20,70,5.)\n",
    "#    meridians = np.arange(-30,70,5.) \n",
    "#    map.drawparallels(parallels,labels=[1,0,0,0],fontsize=10)\n",
    "#    map.drawmeridians(meridians,labels=[0,0,0,1],fontsize=10) \n",
    "#    \n",
    "#    data = f['temp']\n",
    "#    lon = f['lon']\n",
    "#    lat = f['lat']  \n",
    "#    lons,lats= np.meshgrid(lon,lat)\n",
    "#    x,y = map(lons,lats)\n",
    "#    temp = map.contourf(y,x, data)\n",
    "#    cb = map.colorbar(temp,\"bottom\", size=\"5%\", pad=\"2%\")\n",
    "#    plt.title('Example')\n",
    "#    cb.set_label(f\"{f['temp'].attrs['long_name']} in {f['temp'].attrs['units']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS lacks shared models :\n",
    "- which describe scientific data models. If one exist, it is not possible to describe in FITS that we are using it\n",
    "- which describe history provenance. The HISTORY keyword provides just a textual representation of provenance which cannot be machine-read, and with a very loose meaning outside particular applications.\n",
    "- which describe the space mission.\n",
    "\n",
    "HDF5 has the ability to use data models:\n",
    "- standard ones (image, table, dimension scale)\n",
    "- specific one by referencing the conventions in the :Conventions attribute in the root group\n",
    "\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Documentation | + | ++ | ++ |  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Freedom of use\n",
    "<a id=\"freedom\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flexible Image Transport System (FITS) is an open standard defining a digital file forma\n",
    "\n",
    "Astropy is licensed under a 3-clause BSD style license"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 - HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Hierarchical Data Format version 5 (HDF5), is an open source file format that supports large, complex, heterogeneous data\n",
    "\n",
    "h5py license : BSD 3-Clause \"New\" or \"Revised\" License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both FITS and HDF5 are freedom of use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Freedom of use | + | + | + |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5 - Independence / autonomy\n",
    "<a id=\"autonomy\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS is portable. However, an extra documentation is needed to understand the keywords. The file format cannot bring a document with this semantic and has no way to reference the data model in a standard way"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 - HDF5/NetCDF4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HDF5/NetCDF are portable. In addition, by using the CONVENTIONS, the HDF5/NetCDF can reference the data model to share the semantics with users. In addition, the HDF5 could includ the PDF documentation of the semantic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Independence / autonomy | + | ++ | ++ |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6 - Robustness\n",
    "<a id=\"robustness\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 - Integrity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a HDF5  and a FITS with checksum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "  # HDF5\n",
    "with h5py.File('./mytestfile_checksum.hdf5', 'w') as f:\n",
    "    dset1 = f.create_dataset(\"Data1\", (1000,), fletcher32=True)\n",
    "    g = f.create_group('/doc')\n",
    "    dset2 = g.create_dataset(\"Data2\", (1000,), fletcher32=True)\n",
    "    g = f.create_group('/doc1')\n",
    "    dset3 = g.create_dataset(\"Data3\", (1000,))\n",
    "    g = f.create_group('/doc2')\n",
    "    dset3 = g.create_dataset(\"Data4\", (1000,), fletcher32=True)\n",
    "    f.flush()\n",
    "    \n",
    "with h5py.File(\"mytestfile_checksum.hdf5\", \"r\") as f:\n",
    "    f.visititems(print_attrs)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FITS\n",
    "data = np.zeros((1000, 1), dtype=np.float64)\n",
    "new_hdul = fits.HDUList()\n",
    "new_hdul.append(fits.ImageHDU(data=data))\n",
    "new_hdul.writeto('test.fits', checksum=True, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1.1 - Check checksum with HDF5\n",
    "\n",
    "HDF5 files do not have an error recovery mechanism and do not journal. Accidents happen. When you're storing or transmitting a multiterabyte dataset, you'd like to be sure that the bytes that come out of the file are the same ones you put in. HDF5includes a checksum filter for just this purpose. It uses a 32-bit implementation of Fletcher's checksum, hence the name FLETCHER32. A checksum is computed when each chunk is written to the file, and recorded in the chunk's metadata. When the chunk is read, the checksum is computed again and compared to the old one. If they don't match, an error is raised and the read fails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def check_fletcher(name, node):\n",
    "    if isinstance(node, h5py.Dataset) and node.fletcher32:\n",
    "       print (name,': ', end = '')\n",
    "       try:\n",
    "           test = node[:]\n",
    "           print ('test successful')\n",
    "       except:\n",
    "           print ('test failed')\n",
    "\n",
    "\n",
    "with h5py.File('./mytestfile_checksum.hdf5','r') as h5f:\n",
    "    h5f.visititems(check_fletcher)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1.2 - Check checksum with NetCdf when the file is opened\n",
    "\n",
    "How do I know a NetCDF file is not corrupted or compromised ?\n",
    "\n",
    "If the checksum is used for a NetCDF, the file's checksum will be verified when it is accessed. \n",
    "Currently, MD5 checksum is the only algorithm used by NetCDF. \n",
    "\n",
    "The NetCDF file integrity is further enhanced so that the  potential for a buffer overflow vulnerability in the code when reading specially-crafted (invalid) NetCDF files can be prevented. ). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "nc_fid = netCDF4.Dataset('./mytestfile_checksum.hdf5', 'r')\n",
    "data1 = nc_fid.variables['Data1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1.3 - Check checksum for Fits file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdun = fits.open('test.fits', checksum=True)\n",
    "hdun.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.2 - Format & metadata validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/astropy.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2.2 - HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://cerfacs.fr/coop/json-schema-for-sci-apps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Robustness | + | + | + |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 - Compactness\n",
    "<a id=\"compactness\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_file = download_file('http://data.astropy.org/tutorials/FITS-images/HorseHead.fits', cache=True )\n",
    "hdu_list = fits.open(image_file)\n",
    "hdu_list.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   ### Save as FITS and compressed FITS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove the mask, we do not need\n",
    "del hdu_list[1]\n",
    "\n",
    "# save as FITS file\n",
    "hdu_list.writeto('./horse.fits', checksum=True, overwrite=True)\n",
    "\n",
    "# save as compressed FITS file\n",
    "hdu = fits.CompImageHDU(hdu_list[0].data, hdu_list[0].header)\n",
    "hdu.writeto('./horse_compress.fits', checksum=True, overwrite=True)\n",
    "\n",
    "# Retrieves the filse size\n",
    "non_compress = os.path.getsize('./horse.fits')\n",
    "compress = os.path.getsize('./horse_compress.fits')\n",
    "print(f\"Size of the non compressed file : {non_compress}\\nSize of compressed file : {compress}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uncompress the FITS file and display it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with fits.open('./horse_compress.fits') as hdul:\n",
    "    image = hdul[1] # When compress is apply, a primary HDU is created. The data is now in the second HDU\n",
    "    plt.imshow(image.data, cmap='gray')\n",
    "    plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other compression algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Several commonly used algorithms for compressing image tiles are supported. These include Gzip, Rice, IRAF Pixel List (PLIO), and Hcompress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.2 -  HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "   ### Save as HDF5 and compressed HDF5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image = hdu_list[0]\n",
    "np_arr = image.data\n",
    "\n",
    "# Save as HDF5 file\n",
    "with h5py.File('./horse.hdf5', 'w') as f:\n",
    "    f.create_dataset(\"img\", data=np_arr, shape=np_arr.shape, fletcher32=True)\n",
    "    \n",
    "# Save as compressed HDF5 file\n",
    "with h5py.File('./horse_compress.hdf5', 'w') as f:\n",
    "    f.create_dataset(\"img\", data=np_arr, shape=np_arr.shape, fletcher32=True, compression='gzip', compression_opts=9)    \n",
    "\n",
    "# Retrieves the filse size\n",
    "non_compress = os.path.getsize('./horse.hdf5')\n",
    "compress = os.path.getsize('./horse_compress.hdf5')\n",
    "print(f\"Size of the non compressed file : {non_compress}\\nSize of compressed file : {compress}\")    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uncompress the HDF5 file and display it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('./horse_compress.hdf5', 'r') as f:\n",
    "    data = f['img']\n",
    "    plt.imshow(image.data, cmap='gray')\n",
    "    plt.colorbar()    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lossless compression filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **GZIP filter (\"gzip\")** : Available with every installation of HDF5, so it’s best where portability is required. Good compression, moderate speed. compression_opts sets the compression level and may be an integer from 0 to 9, default is 4.\n",
    "- **LZF filter (\"lzf\")** : Available with every installation of h5py (C source code also available). Low to moderate compression, very fast. No options.\n",
    "- **SZIP filter (\"szip\")** : Patent-encumbered filter used in the NASA community. Not available with all installations of HDF5 due to legal reasons. Consult the HDF5 docs for filter options.\n",
    "\n",
    "It is possible to apply custom compression filters by the plugin mechanism and to improve the compression by the options : Scale-Offset filter, Shuffle filter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF5 | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Compactness | + | ++ | ++ | one '+' for FITS/HDF5/NetCDF4 because it is possible to add compression algorihtm by a plugin system| "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8 - Availability tools for processing tools\n",
    "<a id=\"availability\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Availability tools for processing tools | + | + | + |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9 - Content additional embedded\n",
    "<a id=\"content\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A FITS file is flat container that holds the following data objects :\n",
    "- Table\n",
    "- Image\n",
    "- Data cube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create binary Table\n",
    "counts = np.array([312, 334, 308, 317])\n",
    "names = np.array(['NGC1', 'NGC2', 'NGC3', 'NGC4'])\n",
    "values = np.arange(2*2*4).reshape(4, 2, 2)\n",
    "col1 = fits.Column(name='target', format='10A', array=names)\n",
    "col2 = fits.Column(name='counts', format='J', unit='DN', array=counts)\n",
    "col3 = fits.Column(name='notes', format='A10')\n",
    "col4 = fits.Column(name='spectrum', format='10E')\n",
    "col5 = fits.Column(name='flag', format='L', array=[True, False, True, True])\n",
    "col6 = fits.Column(name='intarray', format='4I', dim='(2, 2)', array=values)\n",
    "coldefs = fits.ColDefs([col1, col2, col3, col4, col5, col6])\n",
    "hdu_table = fits.BinTableHDU.from_columns(coldefs)\n",
    "\n",
    "# Create an Image\n",
    "n = np.arange(100.0)\n",
    "hdu_image = fits.ImageHDU(n)\n",
    "\n",
    "# Create the FITS file\n",
    "new_hdul = fits.HDUList()\n",
    "new_hdul.append(fits.PrimaryHDU())\n",
    "new_hdul.append(hdu_image)\n",
    "new_hdul.append(hdu_table)\n",
    "new_hdul.writeto('fits_content.fits', overwrite=True)\n",
    "\n",
    "# open the file\n",
    "with fits.open('fits_content.fits') as hdul:\n",
    "    print(hdul.info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.2 - HDF5 / NetCDF4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An HDF5 file (an object in itself) can be thought of as a container (or group) that holds a variety of heterogeneous data objects (or datasets). The datasets can be images, tables, graphs, and even documents, such as PDF or Excel:\n",
    "\n",
    "<img src=\"images/fileobj.png\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('./corot_102708694.hdf5', 'w') as file:\n",
    "    file.attrs.create(\"corot_id\",102708694)\n",
    "\n",
    "    star_group = file.create_group('star')\n",
    "    star_preview_group = file.create_group('/star/preview')\n",
    "    star_coordinates_group = file.create_group('/star/coordinates')\n",
    "    star_magnitudes_group = file.create_group('/star/magnitudes')\n",
    "    star_spectral_class_group = file.create_group('/star/spectral_classification')\n",
    "    star_contamination_l0_group = file.create_group('/star/contamination_l0')\n",
    "    star_catalogs_references = file.create_group('/star/catalogs_references')\n",
    "\n",
    "    star_coordinates_group.attrs.create(\"ra_deg\", 100.956126)\n",
    "    star_coordinates_group.attrs.create(\"dec_deg\", -1.063014)\n",
    "    star_coordinates_group.attrs.create(\"dec_deg\", -1.063014)\n",
    "    star_coordinates_group.attrs.create(\"ra_hms\", \"06:43:49.470\")\n",
    "    star_coordinates_group.attrs.create(\"dec_hms\", \"-01:03:46.850\")  \n",
    "\n",
    "    star_magnitudes_group.attrs.create(\"B\", \"12.78\")\n",
    "    star_magnitudes_group.attrs.create(\"V\", \"11.72\")\n",
    "    star_magnitudes_group.attrs.create(\"R\", \"11.87\")\n",
    "    star_magnitudes_group.attrs.create(\"J\", \"10.301±0.024\")\n",
    "    star_magnitudes_group.attrs.create(\"H\", \"9.88±0.022\")\n",
    "    star_magnitudes_group.attrs.create(\"K\", \"9.806±0.021\")\n",
    "\n",
    "    observations_group = file.create_group('observations')\n",
    "    spectras_group = file.create_group('spectras')\n",
    "    planets_group = file.create_group('planets')\n",
    "\n",
    "    star_spectral_class_group.attrs.create(\"Spectral type (SED)\", \"G0\")\n",
    "    star_spectral_class_group.attrs.create(\"Luminosity class (SED)\", \"IV\")\n",
    "    star_spectral_class_group.attrs.create(\"E(B-V) (SED)\", 0.15)\n",
    "\n",
    "    star_catalogs_references.attrs.create(\"PPMXL\", \"http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ50b4975daec&-out.add=.&-source=I/317/sample&PPMXL===2791718448871534422\")\n",
    "    star_catalogs_references.attrs.create(\"OBS_CAT\", \"http://cesam.oamp.fr/exodat/quick-access/obscat-info?id=102708694\")\n",
    "    star_catalogs_references.attrs.create(\"USNO-A2\", \"http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ50b49b77187f&-out.add=.&-source=I/252/out&USNO-A2.0===0825-03049717\")\n",
    "    star_catalogs_references.attrs.create(\"CMC14\", \"http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ50b49bd277da&-out.add=.&-source=I/304/out&CMC14===064349.4-010346\")\n",
    "    star_catalogs_references.attrs.create(\"TYCHO2\", \"http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ50b49d742133&-out.add=.&-source=I/259/tyc2&recno=1235387\")\n",
    "    star_catalogs_references.attrs.create(\"UCAC2_CAT\", \"http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ50b49cf22978&-out.add=.&-source=I/289/out&2UCAC=31290682\")\n",
    "\n",
    "    image_filename = \"http://cesam.oamp.fr/corot-findingcharts/fieldmap.pl?102708694\"\n",
    "    filename = wget.download(image_filename)\n",
    "    np_image = cv2.imread(filename)\n",
    "    dset = star_group.create_dataset('img', data=np_image)\n",
    "\n",
    "    xmldata = \"\"\"<xml>\n",
    "    <something>\n",
    "        <else>Text</else>\n",
    "    </something>\n",
    "    </xml>\n",
    "    \"\"\"\n",
    "    str_type = h5py.special_dtype(vlen=str)\n",
    "    ds = file.create_dataset('/quality/something.xml', shape=(1,), dtype=str_type)\n",
    "    ds[:] = xmldata\n",
    "    \n",
    "with h5py.File('./corot_102708694.hdf5','r') as f:\n",
    "    f.visititems(print_attrs)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('./corot_102708694.hdf5', 'r') as f:\n",
    "    print(f['quality']['something.xml'][0].decode())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('./corot_102708694.hdf5', 'r') as f:\n",
    "    image = f['star']['img'][:]\n",
    "    plt.imshow(image.data, cmap='gray')\n",
    "    plt.colorbar() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SAS processing will generate files from the pipelines. For each file, a set of files will have to be attached to better define the scientific files:\n",
    "- pipeline configuration management: inputs, outputs, pipeline configuration parameters, pipeline version\n",
    "- data quality: currently only a tag is required but it would be surprising if other information were not added such as a preview or more complex information\n",
    "\n",
    "For all these reasons, the use of HDF5 is an advantage compared to FITS because HDF5 allows to create one and only one file, allowing easier data management and data transfer\n",
    "\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Content additional embedded | + | +++ | +++ | |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10 - Simplicity\n",
    "<a id=\"simplicity\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both FITS and HDF5/NetCDF have a simple data model. FITS is a bit simple because metadata are visible as ASCII by opening the file by an editor.\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Simplicity | ++ | + | + |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11 - Stability / evolutivity\n",
    "<a id=\"stability\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.1 - FITS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no standard way to specify the version of a given FITS file. Consider that three keywords have been deprecated (BLOCKED, CROTA2 and EPOCH) by the latest version of FITS. Per the standard, these are “obsolete structures that should not be used in new FITS files but which shall remain valid indefinitely” ! As time passes and changes of this nature accumulate, it will be progressively harder to interpret FITS data correctly\n",
    "\n",
    "There is no standard way to specify what extensions it supports of a given FITS file. You must read the file and determine dynamically which extensions are present and whether they are understood.\n",
    "\n",
    "Explicit versioning of FITS files will help but there also needs to be a way to declare that a particular data model is being used and to validate the contents against a namespaced schema.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.2 - HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.1 - Version of the library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HDF5 has been evolving for many years now. By default, the library will write objects in the most compatible fashion possible, so that older versions will still be able to read files generated by modern programs. However, there can be feature or performance advantages if you are willing to forgo a certain level of backwards compatibility. By using the “libver” option to File, you can specify the minimum and maximum sophistication of these structures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = h5py.File('./horse.hdf5', libver='earliest') # most compatible\n",
    "f = h5py.File('./horse.hdf5', libver='latest')   # most modern"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here “latest” means that HDF5 will always use the newest version of these structures without particular concern for backwards compatibility. The “earliest” option means that HDF5 will make a best effort to be backwards compatible.\n",
    "\n",
    "The default is “earliest”.\n",
    "\n",
    "Specifying version bounds has changed from HDF5 version 1.10.2. There are two new compatibility levels: v108 (for HDF5 1.8) and v110 (for HDF5 1.10). This change enables, for example, something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = h5py.File('./horse.hdf5', libver=('earliest', 'v108'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "which enforces full backward compatibility up to HDF5 1.8. Using any HDF5 feature that requires a newer format will raise an error.\n",
    "\n",
    "latest is now an alias to another bound label that represents the latest version. Because of this, the File.libver property will not use latest in its output for HDF5 1.10.2 or later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11.2.2 - Versionning a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fileobject = h5py.File('mydata.h5', 'w')\n",
    "versioned_file = VersionedHDF5File(fileobject)\n",
    "with versioned_file.stage_version('version1') as group:\n",
    "    group['mydataset'] = np.ones(10000)                                 \n",
    "v1 = versioned_file['version1']\n",
    "v1                          "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1['mydataset']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with versioned_file.stage_version('version2') as group:\n",
    "    group['mydataset'][0] = -10\n",
    "v2 = versioned_file['version2']\n",
    "v2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1['mydataset'][()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v2['mydataset'][()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with h5py.File('./mydata.h5','r') as f:\n",
    "    f.visititems(print_attrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove a version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del versioned_file['version1']\n",
    "with h5py.File('./mydata.h5','r') as f:\n",
    "    f.visititems(print_attrs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11.3 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HDF5 supports multiple versions of the file format specification, which FITS is not capable of. Furthermore, thanks to the structure of HDF5, a file can be managed in version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Stability / evolutivity | + | +++ | +++ |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12 - Openness\n",
    "<a id=\"openness\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both FITS and HDF5/NetCDF have a strucure easy to understand\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Openness | + | + | + |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 13 - Expressivity\n",
    "<a id=\"expressivity\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13-1 - FITS¶\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flexible Image Transport System (FITS)1 is a standard format for exchanging astronomical data, independent of the hardware platform and software environment.\n",
    "A file in FITS format consists of a series of Header Data Units (HDUs), each containing two components: an ASCII text header and the binary data. The header contains a series of keywords that describe the data in a particular HDU and the data component may immediately follow the header.\n",
    "For HST FITS files, the first HDU, or primary header, contains no data. The primary header may be followed by one or more HDUs called extensions. Extensions may take the form of images, binary tables, or ASCII text tables. The data type for each extension is recorded in the XTENSION header keyword\n",
    "\n",
    "<img src=\"images/fits_dm.png\">\n",
    "\n",
    "The 80-character card image drives a number of subsequent limitations which result in poor metadata description (8-character keyword, 68-character limit in keyword values, and cumbersome CONTINUE card constructs). This out-dated restriction also results in the awkward implementation of some conventions, such as ESO HIERARCH (Wicenec et al. 2009), that can not overcome the underlying limitations of representation.\n",
    "\n",
    "Additionally, the lack of namespaces results in uncertainty over metadata meaning with other FITS files. Finally, the 2880 record is a minor but annoying restriction which results in wasteful blocks of whitespace in many FITS files, hampers the use of FITS to capture very small, but richly described data, and impedes the real-time writing of FITS files.\n",
    "\n",
    "Missing values are a common feature of most datasets, and are distinct from invalid values (such as NaN or Not a Number) that may occur for example in floating point calculations. For images with integer data types, one can make use of the BLANK keyword to represent missing values, and for tables with integer and string columns, one can make use of the TNULL header keyword. However, for floating point images or table columns, there is no mechanism for specifying missing values. This has led to the common use of NaN to represent missing floating point values. However, one should carefully distinguish between true missing values (which in an image could indicate for example an area of sky that was not observed), versus an invalid value (represented by NaN) which may represent for example a saturated pixel; such a distinction is not currently possible in FITS.\n",
    "\n",
    "The allowed character set in FITS of 7-bit US-ASCII is overly restrictive in an Unicode world (not possible to define scientific or mathematical symbols - very strange for scientific format !).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.2 - HDF5 / NetCDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HDF5 Data Model, also known as the HDF5 Abstract (or Logical) Data Model consists of the building blocks for data organization and specification in HDF5.\n",
    "\n",
    "<img src=\"images/hdf5_group.png\" width=\"600px\">\n",
    "\n",
    "\n",
    "As below, the different data models for both HDF5 and NetCDF4: \n",
    "\n",
    "<table border=\"0\">\n",
    "    <tr><td><img src=\"images/hdf5_dm.jpeg\"></td><td><img src=\"images/nc4_dm.png\"><td></tr>\n",
    "    <tr><td><i>HDF5 Data model</i></td><td><i>NetCDF 4 Data model</i></td></tr>\n",
    "</table>\n",
    "\n",
    "The mapping of the concepts between HDF5/NetCDF4:\n",
    "\n",
    "| NetCDF            |   HDF5        |\n",
    "|-------------------|---------------|\n",
    "|Dataset            |HDF5 file      |\n",
    "|Dimensions         |Dataspace      |\n",
    "|Attribute          |Attribute      |\n",
    "|Variable           |Dataset        |\n",
    "|Coordinate Variable|Dimension scale|\n",
    "\n",
    "HDF5/NetCDF4 handles UTF-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.3 - Mapping FITS/HDF5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/mapping_fits_hdf5.png\">\n",
    "<img src=\"images/mapping_fits_hdf5_2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Does the format allow to encode all the information that the producer wishes to express ? | to | to | to | to |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 13.4 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FITS can support basic data models such as tables and multi-dimensional images but lacks many higher level data models which enable scientific data description. To start with, there is no standardized way of associating the basic models in a related manner. Determining that a particular image extension contains the variance or mask for another image relies on string parsing and shared convention. As data acquisition and data reduction systems have become more complex there has been a move to storing multiple image data components in extensions within a single FITS file. The FITS extension mechanism provides a scheme for having multiple images but, as noted in Greisen (2003), in essentially a flat structure without hierarchy or inheritance. If you have nine images in the file there is no way of indicating that three of them are data, three are an error and three are a quality mask. Indeed, there is no way of specifying which triplets are related. You can use the EXTNAME header to indicate relationships but this relies on convention and string parsing rather than being a standard part of the format.\n",
    "\n",
    "SAS processing will generate files from the pipelines. For each file, a set of files will have to be attached to better define the scientific files. The possibility to arrange the information in a hierarchical way is an asset because this feature avoids the creation of rules based on the name of the XENSTION FITS\n",
    "\n",
    "The 8 characters are too restrictive to properly express good semantics. HDF5/NetCDF does not have this constraint\n",
    "\n",
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "|Expressivity | + | +++ | +++ |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14 - Mastery\n",
    "<a id=\"mastery\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scientific community in Planetology seems to know better FITS than HDF5/NetCDF. However FITS and HDF5 have a good documentation and implementation in several languages, that allows to handle any format quickly. However, we have to check if the COTS in pipeline supports HDF5 format (astropy does it)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Mastery | ++ | + | + | + |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15 - Processing\n",
    "<a id=\"processing\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.1 - Chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table><tr><td><img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig001.png\"></td><td><img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig002.png\"></td></tr><tr><td>Contiguous dataset</td><td>Chunked dataset</td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table><tr><td><img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig003.png\"></td><td><img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig004.png\"></td></tr><tr><td>Reading part of a row from a contiguous dataset</td><td>Reading part of a column from a contiguous dataset</td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table><tr><td><img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig005.png\"></td></tr><tr><td>Reading part of a column from a chunked dataset</td></tr></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15.1.1 - FITS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = np.ones((10000, 10000), dtype=np.float64)\n",
    "new_hdul = fits.HDUList()\n",
    "new_hdul.append(fits.ImageHDU(data=data))\n",
    "new_hdul.writeto('./mon_fichier.fits', overwrite=True)\n",
    "\n",
    "with fits.open(\"./mon_fichier.fits\", mode='update') as new_hdul:\n",
    "    start = time.time()\n",
    "    hdu_data = new_hdul[0].data\n",
    "    hdu_data[:, 2] = 56 * hdu_data[:, 3]\n",
    "    end = time.time()\n",
    "    print(\"10 000 records have been modified in {} seconde\".format(end - start))\n",
    "\n",
    "with fits.open(\"./mon_fichier.fits\") as new_hdul:\n",
    "    hdu_data = new_hdul[0].data\n",
    "    print(hdu_data[:, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15.1.2 - HDF5 without Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a big matrix\n",
    "my_array = np.ones((10000, 10000))\n",
    "\n",
    "# Creating a HDF5\n",
    "with h5py.File('./mon_fichier.hdf5', 'w') as my_file:\n",
    "    my_dataset = my_file.create_dataset(\"fichier de demo2\", data=my_array)\n",
    "    my_file.flush()\n",
    "\n",
    "# Now, numerical data are easily accessible\n",
    "start = time.time()\n",
    "with h5py.File('./mon_fichier.hdf5', 'r+') as my_file:\n",
    "    my_file[\"fichier de demo2\"][:,2]=56*my_file[\"fichier de demo2\"][:,3]\n",
    "    my_file.flush()\n",
    "end = time.time()\n",
    "print(\"10 000 records have been modified in {} seconde\".format(end - start))\n",
    "\n",
    "with h5py.File('./mon_fichier.hdf5', 'r') as my_file:\n",
    "    my_dataset = my_file[\"fichier de demo2\"][:]\n",
    "    print(my_dataset[1:10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15.1.3 - HDF5 with automatic Chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The “auto-chunker” tries to keep chunks mostly “square” (in N dimensions) and within certain size limits. It’s also invoked when you specify the use of compression or other filters without explicitly providing a chunk shape.\n",
    "\n",
    "By the way, the reason the automatically generated chunks are “square” in N dimensions is that the auto-chunker has no idea what you’re planning to do with the dataset, and is hedging its bets. It’s ideal for people who just want to compress a dataset and don’t want to bother with the details, but less ideal for those with specific time-critical access patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a big matrix\n",
    "my_array = np.ones((10000, 10000))\n",
    "\n",
    "# Creating a HDF5\n",
    "with h5py.File('./mon_fichier.hdf5', 'w') as my_file:\n",
    "    my_dataset = my_file.create_dataset(\"fichier de demo2\", data=my_array, chunks=True)\n",
    "    print(my_dataset.chunks)\n",
    "    my_file.flush()\n",
    "\n",
    "# Now, numerical data are easily accessible\n",
    "start = time.time()\n",
    "with h5py.File('./mon_fichier.hdf5', 'r+') as my_file:\n",
    "    my_file[\"fichier de demo2\"][:,2]=56*my_file[\"fichier de demo2\"][:,3]\n",
    "    my_file.flush()\n",
    "end = time.time()\n",
    "print(\"10 000 records have been modified in {} seconde\".format(end - start))\n",
    "\n",
    "with h5py.File('./mon_fichier.hdf5', 'r') as my_file:\n",
    "    my_dataset = my_file[\"fichier de demo2\"][:]\n",
    "    print(my_dataset[1:10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 15.1.4 - HDF5 with custom Chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if os.path.exists(\"./mon_fichier.hdf5\"):\n",
    "    os.remove(\"./mon_fichier.hdf5\")\n",
    "    \n",
    "# Creating a big matrix\n",
    "my_array = np.ones((10000, 10000))\n",
    "\n",
    "# Creating a HDF5\n",
    "with h5py.File('./mon_fichier.hdf5', 'w') as my_file:\n",
    "    my_dataset = my_file.create_dataset(\"fichier de demo2\", data=my_array, chunks=(10000,1))\n",
    "    print(my_dataset.chunks)\n",
    "    my_file.flush()\n",
    "\n",
    "# Now, numerical data are easily accessible\n",
    "start = time.time()\n",
    "with h5py.File('./mon_fichier.hdf5', 'r+') as my_file:\n",
    "    my_file[\"fichier de demo2\"][:,2]=56*my_file[\"fichier de demo2\"][:,3]\n",
    "    my_file.flush()\n",
    "end = time.time()\n",
    "print(\"10 000 records have been modified in {} seconde\".format(end - start))\n",
    "\n",
    "with h5py.File('./mon_fichier.hdf5', 'r') as my_file:\n",
    "    my_dataset = my_file[\"fichier de demo2\"][:]\n",
    "    print(my_dataset[1:10, 2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.2 - Cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/Images/Fig006.png\">\n",
    "\n",
    "h5py_cache : https://github.com/moble/h5py_cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.3 - Virtual dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://support.hdfgroup.org/images/tutrvds-ex.png\">\n",
    "\n",
    "Virtual datasets allow a number of real datasets to be mapped together into a single, sliceable dataset via an interface layer. The mapping can be made ahead of time, before the parent files are written, and is transparent to the parent dataset characteristics (SWMR, chunking, compression etc…). The datasets can be meshed in arbitrary combinations, and even the data type converted.\n",
    "\n",
    "Once a virtual dataset has been created, it can be read just like any other HDF5 dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create some sample data\n",
    "data = np.arange(0, 100).reshape(1, 100) + np.arange(1, 5).reshape(4, 1)\n",
    "\n",
    "# Create source files (0.h5 to 3.h5)\n",
    "for n in range(4):\n",
    "    with h5py.File(f\"{n}.h5\", \"w\") as f:\n",
    "        d = f.create_dataset(\"data\", (100,), \"i4\", data[n])\n",
    "\n",
    "# Assemble virtual dataset\n",
    "layout = h5py.VirtualLayout(shape=(4, 100), dtype=\"i4\")\n",
    "for n in range(4):\n",
    "    filename = \"{}.h5\".format(n)\n",
    "    vsource = h5py.VirtualSource(filename, \"data\", shape=(100,))\n",
    "    layout[n] = vsource\n",
    "\n",
    "# Add virtual dataset to output file\n",
    "with h5py.File(\"VDS.h5\", \"w\", libver=\"latest\") as f:\n",
    "    f.create_virtual_dataset(\"vdata\", layout, fillvalue=-5)\n",
    "    f.create_dataset(\"data\", data=data, dtype=\"i4\")\n",
    "\n",
    "\n",
    "# read data back\n",
    "# virtual dataset is transparent for reader!\n",
    "with h5py.File(\"VDS.h5\", \"r\") as f:\n",
    "    print(\"Virtual dataset:\")\n",
    "    print(f[\"vdata\"][:, :10])\n",
    "    print(\"Normal dataset:\")\n",
    "    print(f[\"data\"][:, :10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.4 - Query capabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1234)\n",
    "pd.set_option('display.max_rows',20)\n",
    "index = pd.MultiIndex.from_product([np.arange(10000,11000),pd.date_range('19800101',periods=10000)],names=['id','date'])\n",
    "df = pd.DataFrame(dict(id2=np.random.randint(0,1000,size=len(index)),w=np.random.randn(len(index))),index=index).reset_index().set_index(['id','date'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_hdf('test_pandas.h5','df',mode='w',data_columns=['id2'],format='table') # Work with pyTables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -lhtr test_pandas.h5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Queries\n",
    "ids=[10101,10898]\n",
    "start_date='20010101'\n",
    "end_date='20010301'\n",
    "pd.read_hdf('test_pandas.h5','df',where='date>start_date & date<end_date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_hdf('test_pandas.h5','df',where='date>start_date & date<end_date & id=ids')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_hdf('test_pandas.h5','df',where='date>start_date & date<end_date & id=ids & id2>500 & id2<600')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.5 - HPC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://discourse.pangeo.io/t/best-practices-to-go-from-1000s-of-netcdf-files-to-analyses-on-a-hpc-cluster/588\n",
    "\n",
    "Demo on GAIA : https://www.youtube.com/watch?v=CQHH5_74O0E&ab_channel=ADASS2020\n",
    "\n",
    "73 GB cube analyzed in a few seconds!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6 - Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| Processing | + | +++ | +++ |  |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 16 -  Conclusion\n",
    "<a id=\"conlusion\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Criteria | FITS | HDF | NetCDF | Remarks |\n",
    "| -------- | ---- | --- |------- | ------- |\n",
    "| User community / Sociability | +++ | + | + |  |\n",
    "| Documentation | + | ++ | ++ |  |\n",
    "| Freedom of use | + | + | + |  |\n",
    "| Independence / autonomy | + | ++ | ++ |  |\n",
    "| Robustness | + | + | + |  |\n",
    "| Compactness | + | ++ | ++ | one '+' for FITS/HDF5/NetCDF4 because it is possible to add compression algorihtm by a plugin system|\n",
    "| Availability tools for processing tools | + | + | + |  |\n",
    "| Content additional embedded | + | +++ | +++ | |\n",
    "| Simplicity | ++ | + | + |  |\n",
    "| Stability / evolutivity | + | +++ | +++ |  |\n",
    "| Openness | + | + | + |  |\n",
    "|Expressivity | + | +++ | +++ |  |\n",
    "| Mastery | ++ | + | + | + |\n",
    "| Processing | + | +++ | +++ |  |\n",
    "| Total | 18 | 25 | 25 |  |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Format de la Cellule Texte Brut",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}