python-data/solutions/ex08_netcdf4_advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "22542fd5-6792-4df8-9122-fe35f3e4ddf5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Exercise 8: NetCDF4 Advanced"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8bc8ade-8ef9-4caa-b734-d0a0df52a450",
   "metadata": {},
   "source": [
    "## Aim: Introduce more advanced uses of the netCDF4 library in Python to read and create NetCDF4 Files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e92fc14e-6e03-49c3-99c1-7a7b1c2e52cf",
   "metadata": {},
   "source": [
    "Find the teaching material here: https://unidata.github.io/netcdf4-python/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2a56e34-d2ba-451d-900c-09e33c404e24",
   "metadata": {},
   "source": [
    "### Issues covered:\n",
    "- Working with time coordinates\n",
    "- Multi-file datasets\n",
    "- Compression of variables\n",
    "- Compound datatypes\n",
    "- Enum data type"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9ba87e0-e96e-490d-a0a6-1da86bae8084",
   "metadata": {},
   "source": [
    "## Time-coordinates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c4acf3e-dc07-442e-a421-63efb78e0f79",
   "metadata": {},
   "source": [
    "Most metadata standards specify that time should be measured relative to a fixed date with units such as `hours since YY-MM-DD hh:mm:ss`. We can convert values to and from calendar dates using `num2date` and `date2num` from the `cftime` library. Two other helpful functions are `datetime` and `timedelta` from the `datetime` library."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fecb97e-5775-4575-9739-9ad8cb7c3f97",
   "metadata": {},
   "source": [
    "Q1. \n",
    "- Let's generate a list of data and time values: create a list called `dates` containing date and time values, starting from January 1st 2022, and incrementing by 6 hours for a total of 5 entries. \n",
    "- Use `date2num` to convert your list of dates to numeric values using: `units=\"hours since 2022-01-01 00:00:00\"` amd `calendar=\"gregorian\"`. Store these in an array called `times`.\n",
    "- Print the numeric times values to confirm the numeric representation.\n",
    "- Use `num2date` to convert times back to datetime objects using the same units and calendar. Store these in a list called `converted_dates`\n",
    "- Print the converted dates to verify they match the original dates list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "477bd19f-2833-4bb6-a2ae-83ebb7fc4a3d",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:39.567060Z",
     "iopub.status.busy": "2024-11-08T14:54:39.566736Z",
     "iopub.status.idle": "2024-11-08T14:54:40.032446Z",
     "shell.execute_reply": "2024-11-08T14:54:40.031854Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original dates: [datetime.datetime(2022, 1, 1, 0, 0), datetime.datetime(2022, 1, 1, 6, 0), datetime.datetime(2022, 1, 1, 12, 0), datetime.datetime(2022, 1, 1, 18, 0), datetime.datetime(2022, 1, 2, 0, 0)]\n",
      "Numeric time values (in units 'hours since 2022-01-01 00:00:00'):\n",
      "[ 0  6 12 18 24]\n",
      "Dates corresponding to numeric time values:\n",
      " [cftime.DatetimeGregorian(2022, 1, 1, 0, 0, 0, 0, has_year_zero=False)\n",
      " cftime.DatetimeGregorian(2022, 1, 1, 6, 0, 0, 0, has_year_zero=False)\n",
      " cftime.DatetimeGregorian(2022, 1, 1, 12, 0, 0, 0, has_year_zero=False)\n",
      " cftime.DatetimeGregorian(2022, 1, 1, 18, 0, 0, 0, has_year_zero=False)\n",
      " cftime.DatetimeGregorian(2022, 1, 2, 0, 0, 0, 0, has_year_zero=False)]\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "from cftime import num2date, date2num\n",
    "\n",
    "# Step 1: Generate dates list\n",
    "dates = [datetime(2022, 1, 1) + n * timedelta(hours=6) for n in range(5)]\n",
    "print(\"Original dates:\", dates)\n",
    "\n",
    "# Step 2: Convert dates to numeric time values\n",
    "units = \"hours since 2022-01-01 00:00:00\"\n",
    "calendar = \"gregorian\"\n",
    "times = date2num(dates, units=units, calendar=calendar)\n",
    "\n",
    "# Step 3: Print numeric time values\n",
    "print(\"Numeric time values (in units '{}'):\\n{}\".format(units, times))\n",
    "\n",
    "# Step 4: Convert numeric time values back to calendar dates\n",
    "converted_dates = num2date(times, units=units, calendar=calendar)\n",
    "\n",
    "# Step 5: Print converted dates\n",
    "print(\"Dates corresponding to numeric time values:\\n\", converted_dates)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7271d566-91d2-4d24-9213-39d6d28d2a0d",
   "metadata": {},
   "source": [
    "## Multi-file datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95cb2707-b27e-42ac-b80f-cf255b1a3c0f",
   "metadata": {},
   "source": [
    "Q2. Let's create multiple netCDF files with a shared variable and unlimited dimension, and use `MFDataset` to read the aggregated data as if it were contained in a single file.\n",
    "- Create 5 netCDF files named `data/datafile0.nc` through to `data/datafile4.nc`. Each file should contain:\n",
    "    - A single unlimited dimension named `time`.\n",
    "    - A variable named `temperature` with 10 integer values ranging from `file_index * 10` to `(file_index+1) * 10 - 1`.\n",
    "    - Ensure each file is saved in the `NETCDF4_CLASSIC` format.\n",
    "    - **Hint: Use a loop such as `for .. in range(..):` to do this task.**\n",
    "- Using `MFDataset` read all the `temperature` data from the 5 files at once by specifying a wildcard string `datafile*.nc` - store this in a variable `f`. Assign this data to a new variable using `temperature_data = f.variables[\"temperature\"][:]`\n",
    "- Print the aggregated `temperature` values to verify that they span from 0 to 49."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3c6df18f-87e8-4051-a701-60d169656701",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.035145Z",
     "iopub.status.busy": "2024-11-08T14:54:40.034779Z",
     "iopub.status.idle": "2024-11-08T14:54:40.104560Z",
     "shell.execute_reply": "2024-11-08T14:54:40.103872Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23\n",
      " 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47\n",
      " 48 49]\n"
     ]
    }
   ],
   "source": [
    "from netCDF4 import Dataset, MFDataset\n",
    "import numpy as np\n",
    "\n",
    "# Step 1: Create multiple netCDF files with a shared unlimited dimension and variable\n",
    "for i in range(5):\n",
    "    with Dataset(f\"data/datafile{i}.nc\", \"w\", format=\"NETCDF4_CLASSIC\") as f:\n",
    "        # Create an unlimited dimension\n",
    "        f.createDimension(\"time\", None)\n",
    "        # Create a variable associated with the 'time' dimension\n",
    "        temp_var = f.createVariable(\"temperature\", \"i4\", (\"time\",))\n",
    "        # Populate 'temperature' with a unique range of values for each file\n",
    "        temp_var[:] = np.arange(i * 10, (i+1) * 10)\n",
    "\n",
    "# Step 2: Use MFDataset to read all files at once\n",
    "try:\n",
    "    #Read and aggregate all data across all files\n",
    "    f = MFDataset(\"data/datafile*.nc\")\n",
    "    temperature_data = f.variables[\"temperature\"][:]\n",
    "    # Print the aggregated data\n",
    "    print(temperature_data)\n",
    "finally:\n",
    "    # Close the MFDataset object\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28e3eb27-a38b-4de7-beec-a04f95922561",
   "metadata": {},
   "source": [
    "## Compression of variables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2727e4a-5009-4678-b227-e40d49b576e6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Q3. Let's explore various compression options available in netCDF. \n",
    "- Run the following cell to create an array of random temperature data and create a function to create NetCDF files with given compression settings. Take a look at the function and figure out what it's doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f9fea430-9b8f-4f65-ba2e-ee600cb0e0e2",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.109135Z",
     "iopub.status.busy": "2024-11-08T14:54:40.108718Z",
     "iopub.status.idle": "2024-11-08T14:54:40.118891Z",
     "shell.execute_reply": "2024-11-08T14:54:40.118326Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# Step 1: Create a random dataset \n",
    "time_dim, level_dim, lat_dim, lon_dim = 10, 5, 50, 100\n",
    "data = np.random.rand(time_dim, level_dim, lat_dim, lon_dim) * 30 + 273.15\n",
    "\n",
    "# Step 2: Create a function to create NetCDF files with the given compression settings:\n",
    "file_path = \"data/temperature_data.nc\"\n",
    "def create_netcdf(file_path, compression=None, least_significant_digit=None, significant_digits=None):\n",
    "    with Dataset(file_path, 'w', format=\"NETCDF4\") as rootgrp:\n",
    "        # Create dimensions\n",
    "        rootgrp.createDimension(\"time\", time_dim)\n",
    "        rootgrp.createDimension(\"level\", level_dim)\n",
    "        rootgrp.createDimension(\"lat\", lat_dim)\n",
    "        rootgrp.createDimension(\"lon\", lon_dim)\n",
    "        # Define variable with compression settings\n",
    "        temp = rootgrp.createVariable(\"temp\", \"f4\", (\"time\", \"level\", \"lat\", \"lon\"), compression=compression, least_significant_digit=least_significant_digit, significant_digits=significant_digits)\n",
    "        #Assign data to the variable\n",
    "        temp[:] = data\n",
    "    # Check and print file size\n",
    "    print(f\"File size with compression={compression}, \"\n",
    "          f\"least_significant_digit={least_significant_digit}, \"\n",
    "          f\"significant_digits={significant_digits}: {os.path.getsize(file_path) / 1024:.2f} kB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e50f691-f784-4e3c-bffc-9520cc08f703",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "- Use this function to test the following cases:\n",
    "    - First, create the temperature variable without compression and observe the file size. Use the file path `data/temperature_data_no_compress.nc`.\n",
    "    - Then, enable zlib compression and observe the change in file size. Use the file path `data/temperature_data_zlib.nc`.\n",
    "    - Next, add zlib and least signigicant digit quantization (`least_significant_digit=3`) and check the file size again. Use the file path `data/temperature_data_zlib_lsd.nc`.\n",
    "    - Finally, add zlib and significant digits quantization (`significant_digits=4`) and check the file size again. Use the file path `data/temperature_data_zlib_sig.nc`.\n",
    "    - Hint: call the function using: `create_netcdf(filepath, compression, least_significant_digit, significant_digit)`. Note that the default for the compression/signigificant digits arguments is None so if you don't need them you can omit them when calling the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "73365563-9cf4-4b3b-9be8-c9814332b90a",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.121272Z",
     "iopub.status.busy": "2024-11-08T14:54:40.121015Z",
     "iopub.status.idle": "2024-11-08T14:54:40.299169Z",
     "shell.execute_reply": "2024-11-08T14:54:40.298642Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File size with compression=None, least_significant_digit=None, significant_digits=None: 983.31 kB\n",
      "File size with compression=zlib, least_significant_digit=None, significant_digits=None: 639.42 kB\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File size with compression=zlib, least_significant_digit=3, significant_digits=None: 505.93 kB\n",
      "File size with compression=zlib, least_significant_digit=None, significant_digits=4: 396.40 kB\n"
     ]
    }
   ],
   "source": [
    "# Step 3: Test different compression settings\n",
    "# 3.1 No compression\n",
    "create_netcdf(\"data/temperature_data_nocompress.nc\")\n",
    "\n",
    "# 3.2 Compression with zlib only\n",
    "create_netcdf(\"data/temperature_data_zlib.nc\", compression='zlib')\n",
    "\n",
    "# 3.3 Compression with zlib and least significant digit quantization\n",
    "create_netcdf(\"data/temperature_data_zlib_lsd.nc\", compression='zlib', least_significant_digit=3)\n",
    "\n",
    "# 3.4 Compression with zlib and significant digits quantization\n",
    "create_netcdf(\"data/temperature_data_zlib_sig.nc\", compression='zlib', significant_digits=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393a6184-658e-48fb-a10b-1c8d302bade6",
   "metadata": {},
   "source": [
    "## Compound data types"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19c598d5-5d89-430e-9178-ec9cac5efe3f",
   "metadata": {},
   "source": [
    "Q4. Let's work with compound data types and structured arrays.\n",
    "- Create a netCDF file called `data/vectors.nc` in write mode with `NETCDF4` format assigned to the variable `f`.\n",
    "- Define a compound data type that represents a 3D vector. Each vector should have 3 components:\n",
    "    - `x`: a `float33` representing the x-coordinate\n",
    "    - `y`: a `float32` representing the y-coordinate\n",
    "    - `z`: a `float32` representing the z-coordinate\n",
    "    - Hint: use `np.dtype([('x', type), ('y'..), (...)])` to define x,y,z then `f.createCompoundType()` to create the compound data type.\n",
    "- Create a dimension named `num_vectors` to store an unlimited number of vectors.\n",
    "- Create a variable called `vector_data` in the file using the compound data type from step 2, with the dimension from step 3.\n",
    "- Generate a numpy structured array with 5 sample 3D vectors:\n",
    "    - Each vector should have random values for `x`, `y` and `z` components (use `np.random.rand(num_samples)`).\n",
    "    - Store these in the structured array (initialize the array with `np.empty(num_samples, dtype)` then use `data[\"x\"]` etc to assign the data.\n",
    "    - Write them to the netCDF variable.\n",
    "- Close the file and then reopen it in read mode.\n",
    "- Read the data back into a new numpy structured array and print each vector.\n",
    "    - Hint: use `f.variables['var_name']` to read in the variable data.\n",
    "    - Hint: Use `data_in = vector_var[:]` to extract the data for the variables.\n",
    "    - Hint: Use `for i, vev in enumerate(data_in):` to loop through the data so you can print each vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c92e87db-f092-4802-a346-aaeb158b6395",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.301564Z",
     "iopub.status.busy": "2024-11-08T14:54:40.301300Z",
     "iopub.status.idle": "2024-11-08T14:54:40.313735Z",
     "shell.execute_reply": "2024-11-08T14:54:40.313229Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vector 0: (x: 0.99, y: 0.74, z: 0.50)\n",
      "Vector 1: (x: 0.07, y: 0.85, z: 0.78)\n",
      "Vector 2: (x: 0.79, y: 0.02, z: 0.28)\n",
      "Vector 3: (x: 0.21, y: 0.61, z: 0.37)\n",
      "Vector 4: (x: 0.78, y: 0.71, z: 0.08)\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Create a netCDF file in write mode.\n",
    "f = Dataset(\"data/vectors.nc\", \"w\", format=\"NETCDF4\")\n",
    "\n",
    "# Step 2: Define a compound data type for a 3D vector with x,y,z as float32 fields\n",
    "vector_dtype = np.dtype([(\"x\", np.float32), (\"y\", np.float32), (\"z\", np.float32)])\n",
    "vector_t = f.createCompoundType(vector_dtype, \"vector3D\")\n",
    "\n",
    "# Step 3: Create a dimension for storing an unlimited number of vectors\n",
    "num_vectors = f.createDimension(\"num_vectors\", None)\n",
    "\n",
    "# Step 4: Create a variable using the compound data type and the dimension\n",
    "vector_var = f.createVariable(\"vector_data\", vector_t, (\"num_vectors\",))\n",
    "\n",
    "# Step 5: Generate a numpy structured array with 5 random 3D vectors\n",
    "num_samples = 5\n",
    "data = np.empty(num_samples, dtype=vector_dtype)\n",
    "data[\"x\"] = np.random.rand(num_samples)\n",
    "data[\"y\"] = np.random.rand(num_samples)\n",
    "data[\"z\"] = np.random.rand(num_samples)\n",
    "# Write the structured array to the netCDF variable\n",
    "vector_var[:] = data\n",
    "\n",
    "# Step 6: Close and reopen in read mode\n",
    "# Close the file\n",
    "f.close()\n",
    "# Reopen the file in read-mode\n",
    "f = Dataset(\"data/vectors.nc\", \"r\")\n",
    "\n",
    "# Step 7: Read the data back into a new structured array and print each vector\n",
    "vector_var = f.variables[\"vector_data\"]\n",
    "data_in = vector_var[:]\n",
    "for i, vec in enumerate(data_in):\n",
    "    print(f\"Vector {i}: (x: {vec['x']:.2f}, y: {vec['y']:.2f}, z: {vec['z']:.2f})\")\n",
    "\n",
    "# Close the file\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a537d6b-f892-444b-b63d-6bdbe4963e83",
   "metadata": {},
   "source": [
    "## Variable-length data types"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faa9d0ba-0352-45bb-98cb-0443cecbc198",
   "metadata": {},
   "source": [
    "Q5. Let's create and manipulate variable-length (vlen) arrays\n",
    "- Create a netCDF file named `data/exercise_vlen.nc` in write mode.\n",
    "- Define dimensions:\n",
    "    - Create a dimension `a` with a size of `5`.\n",
    "    - Create a dimension `b` with a size of `4`.\n",
    "- Create a variable-length data type using `f.createVLType()` named `my_vlen_int` using `np.int32` as the datatype.\n",
    "- Use the vlen type you defined to create a variable `vlen_var` with dimensions `(\"a\", \"b\")`.\n",
    "- Populate `vlen_var` with random data:\n",
    "    - Use the following to generate the random data:\n",
    "      ```\n",
    "      data = np.empty((5,4), dtype=object)\n",
    "      for i in range(5):\n",
    "        for j in range(4):\n",
    "          random_length = random.randint(2, 8)\n",
    "          data[i,j] = np.random.randint(1, 101, size=random_length, dtype=np.int32)\n",
    "      ```\n",
    "    - Assign the data to the netCDF variable\n",
    "- Create a new dimension `c` with a size of `7`.\n",
    "- Define a variable `vlen_str_var` along dimension `c`.\n",
    "- Populate this variable with random strings of lengths between 3 and 10 using uppercase and lowercase alphabetic characters using the following:\n",
    "    ```\n",
    "    chars = string.ascii_letters\n",
    "    string_data = np.empty(7, dtype=object)\n",
    "    for i in range(7):\n",
    "        random_length = random.randint(3,10)\n",
    "        string_data[i] = ''.join(random.choice(chars) for _ in range(random_length))\n",
    "    # Assign the string data to the netCDF variable\n",
    "    str_var[:] = string_data\n",
    "    ```\n",
    "- Print the contents of `vlen_var` and `vlen_str_var`. Print the structure of the netCDF4 file to show defined dimensions, variables, and data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "70ba5df2-e554-47de-b164-11d2c926f63d",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.316032Z",
     "iopub.status.busy": "2024-11-08T14:54:40.315783Z",
     "iopub.status.idle": "2024-11-08T14:54:40.347230Z",
     "shell.execute_reply": "2024-11-08T14:54:40.346679Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": [
     "clear_answer_cell"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Contents of 'vlen_var':\n",
      " [[array([93, 90, 94, 86], dtype=int32)\n",
      "  array([93, 45, 89, 80, 43], dtype=int32)\n",
      "  array([32, 42, 75, 38], dtype=int32)\n",
      "  array([49, 57, 48, 43, 79, 46, 98, 82], dtype=int32)]\n",
      " [array([38, 98,  9], dtype=int32) array([63, 80], dtype=int32)\n",
      "  array([20, 31], dtype=int32) array([98, 58, 46], dtype=int32)]\n",
      " [array([77,  2, 51, 17], dtype=int32)\n",
      "  array([62, 98, 25, 41, 62, 42, 86, 23], dtype=int32)\n",
      "  array([44, 52, 10, 56, 80, 29, 66, 17], dtype=int32)\n",
      "  array([68, 72], dtype=int32)]\n",
      " [array([42, 74, 53], dtype=int32) array([68, 15, 91], dtype=int32)\n",
      "  array([13, 42, 30, 17, 96, 24, 88, 21], dtype=int32)\n",
      "  array([13, 96, 87, 76, 37], dtype=int32)]\n",
      " [array([51, 85, 44, 37, 87,  2, 84], dtype=int32)\n",
      "  array([30, 92, 84, 11], dtype=int32)\n",
      "  array([ 83,  91,   2,  11,  76,  55, 100], dtype=int32)\n",
      "  array([98, 56, 55, 41, 89, 49, 68], dtype=int32)]]\n",
      "Contents of 'vlen_str_var':\n",
      " ['KwF' 'AdpMJjMhcu' 'pasfMCKgA' 'nJvVKGF' 'jVbc' 'hkUa' 'uPnH']\n",
      "\n",
      "NetCDF4 file structure:\n",
      " <class 'netCDF4._netCDF4.Dataset'>\n",
      "root group (NETCDF4 data model, file format HDF5):\n",
      "    dimensions(sizes): a(5), b(4), c(7)\n",
      "    variables(dimensions): int32 vlen_var(a, b), <class 'str'> vlen_str_var(c)\n",
      "    groups: \n",
      "Details of 'vlen_var':\n",
      " <class 'netCDF4._netCDF4.Variable'>\n",
      "vlen vlen_var(a, b)\n",
      "vlen data type: int32\n",
      "unlimited dimensions: \n",
      "current shape = (5, 4)\n",
      "Details of 'vlen_str_var':\n",
      " <class 'netCDF4._netCDF4.Variable'>\n",
      "vlen vlen_str_var(c)\n",
      "vlen data type: <class 'str'>\n",
      "unlimited dimensions: \n",
      "current shape = (7,)\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "# Step 1: Create a netCDF file in write mode\n",
    "f = Dataset(\"data/exercise_vlen.nc\", \"w\")\n",
    "\n",
    "# Step 2: Define dimensions a and b\n",
    "f.createDimension(\"a\", 5)\n",
    "f.createDimension(\"b\", 4)\n",
    "\n",
    "# Step 3: Create a variable-length data type for signed 32-bit integers\n",
    "vlen_int_type = f.createVLType(np.int32, \"my_vlen_int\")\n",
    "\n",
    "# Step 4: Create and populate the variable length integer array\n",
    "vlen_var = f.createVariable(\"vlen_var\", vlen_int_type, (\"a\", \"b\"))\n",
    "\n",
    "# Step 5: Populate vlen_var with random-length integer arrays\n",
    "data = np.empty((5,4), dtype=object)\n",
    "for i in range(5):\n",
    "    for j in range(4):\n",
    "        random_length = random.randint(2, 8)\n",
    "        data[i,j] = np.random.randint(1, 101, size=random_length, dtype=np.int32)\n",
    "# Assign the data to the netCDF variable\n",
    "vlen_var[:, :] = data\n",
    "\n",
    "# Step 6: Create a dimension 'c'\n",
    "f.createDimension(\"c\", 7)\n",
    "\n",
    "# Step 7: Define a variable-length string array\n",
    "str_var = f.createVariable(\"vlen_str_var\", str, (\"c\",))\n",
    "\n",
    "# Step 8: Populate vlen_str_var with random strings of lengths 3-10\n",
    "chars = string.ascii_letters\n",
    "string_data = np.empty(7, dtype=object)\n",
    "for i in range(7):\n",
    "    random_length = random.randint(3,10)\n",
    "    string_data[i] = ''.join(random.choice(chars) for _ in range(random_length))\n",
    "# Assign the string data to the netCDF variable\n",
    "str_var[:] = string_data\n",
    "\n",
    "# Step 9: Print the contents of \"vlen_var\" and \"vlen_str_var\"\n",
    "print(\"Contents of 'vlen_var':\\n\", vlen_var[:])\n",
    "print(\"Contents of 'vlen_str_var':\\n\", str_var[:])\n",
    "# Print the structure of the NetCDF4 file\n",
    "print(\"\\nNetCDF4 file structure:\\n\", f)\n",
    "print(\"Details of 'vlen_var':\\n\", f.variables[\"vlen_var\"])\n",
    "print(\"Details of 'vlen_str_var':\\n\", f.variables[\"vlen_str_var\"])\n",
    "\n",
    "# Close the file\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55abd572-29b3-47ea-9b9f-0e25364297d5",
   "metadata": {},
   "source": [
    "## Enum data type"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f972a56-42e9-499c-ab1e-940b022c616a",
   "metadata": {},
   "source": [
    "Q6. Let's create a netCDF file to store weather observation data including an enumerated type representing different types of precipiation.\n",
    "- Create a new netCDF file called `data/weather_data.nc` in write mode with the `NETCDF4` format.\n",
    "- Create a Python dictionary `precip_dict` where:\n",
    "    - `None` maps to `0`\n",
    "    - `Rain` maps to `1`\n",
    "    - `Snow` maps to `2`\n",
    "    - `Sleet` maps to `3`\n",
    "    - `Hail` maps to `4`\n",
    "    - `Unknown` maps to `255`\n",
    "- Use this dictionary to define an Enum data type using `.createEnumType()` called `precip_t` with a base type of `np.uint8`\n",
    "- Define a dimension called `time` with an unlimited length for observations over time\n",
    "- Create a 1D variable named `precipitation` of type `precip_type` that uses the `time` dimension and has `fill_value=precip_dict['Unknown']`. The fill value indicates missing data.\n",
    "- Write the following precipiatation observations to the `precipitation` variable: `precip_var[:] = [precip_dict[k] for k in ['None', 'Rain', 'Snow', 'Unknown', 'Sleet']]`.\n",
    "- Close and reopen the file in read mode, then print the contents of the `precipitation` variable, inlcuding: the data values confirming they match the written values, the enum dictionary associated with the enum data type, verifying the precipitation mapping. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f08fe449-acc3-41e3-bb36-3f6a2e63b98b",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2024-11-08T14:54:40.349575Z",
     "iopub.status.busy": "2024-11-08T14:54:40.349315Z",
     "iopub.status.idle": "2024-11-08T14:54:40.374465Z",
     "shell.execute_reply": "2024-11-08T14:54:40.373923Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enum dictionary: {'None': 0, 'Rain': 1, 'Snow': 2, 'Sleet': 3, 'Hail': 4, 'Unknown': 255}\n",
      "Precipitation data: [0 1 2 -- 3]\n"
     ]
    }
   ],
   "source": [
    "# Step 1: Create a new netCDF file\n",
    "nc = Dataset('data/weather_data.nc', 'w', format='NETCDF4')\n",
    "\n",
    "# Step 2: Define the Enum dictionary and create the Enum type\n",
    "precip_dict = {\n",
    "    'None': 0,\n",
    "    'Rain': 1,\n",
    "    'Snow': 2,\n",
    "    'Sleet': 3,\n",
    "    'Hail': 4,\n",
    "    'Unknown': 255\n",
    "}\n",
    "\n",
    "# Step 3: Create an Enum type called 'precip_t' with base type uint8\n",
    "precip_type = nc.createEnumType(np.uint8, 'precip_t', precip_dict)\n",
    "\n",
    "# Step 4: Create a time dimension\n",
    "nc.createDimension('time', None)\n",
    "\n",
    "# Step 5: Create the precipitation variable, setting the fill_value to 'Unknown'\n",
    "precip_var = nc.createVariable('precipitation', precip_type, ('time',),\n",
    "                               fill_value=precip_dict['Unknown'])\n",
    "\n",
    "# Step 6: Write data to the variable\n",
    "precip_var[:] = [precip_dict[k] for k in ['None', 'Rain', 'Snow', 'Unknown', 'Sleet']]\n",
    "\n",
    "# Step 7: Close the file\n",
    "nc.close()\n",
    "# Reopen the file, read and print the data\n",
    "nc = Dataset('data/weather_data.nc', 'r')\n",
    "precip_var = nc.variables['precipitation']\n",
    "# Print the Enum dictionary\n",
    "print(\"Enum dictionary:\", precip_var.datatype.enum_dict)\n",
    "# Print the data stored in the variable\n",
    "print(\"Precipitation data:\", precip_var[:])\n",
    "\n",
    "# Close the file\n",
    "nc.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 + Jaspy",
   "language": "python",
   "name": "jaspy"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}