scientific_libraries/Scientific_Libraries_Numpy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89266885",
   "metadata": {},
   "source": [
    "## What is the difference between regular Python and Numpy Python?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "528c6530",
   "metadata": {},
   "source": [
    "### Data Types\n",
    "Python has specific data types, but not as many as traditional C or FORTRAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ef415a7e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'int'>\n",
      "<class 'float'>\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "print(type(1))  # Python integer - this is a 64 byte integer\n",
    "print(type(1.))  # Python float - this is a 64 byte float\n",
    "print(type('Hello World'))  # Python string which is different than a character array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b527246",
   "metadata": {},
   "source": [
    "Numpy can be thought of as a space within the Python language with more data types like C or FORTRAN and a richer set of mathematical/computational enivironments built on top of the base Python environment. All the regular Python functions still exist, just more options are available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ef81ed6d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(np.array(1)): <class 'numpy.ndarray'>\n",
      "\n",
      "np.int8: int8\n",
      "np.int16: int16\n",
      "np.int32: int32\n",
      "np.int64: int64\n",
      "np.int: int64\n",
      "\n",
      "np.float16: float16\n",
      "np.float32: float32\n",
      "np.float64: float64\n",
      "np.float: float64\n",
      "\n",
      "np.bool: bool\n",
      "np.int: int64\n",
      "np.foat: float64\n",
      "np.uint: uint64\n",
      "np.complex: complex128\n",
      "Hello World: <U11\n"
     ]
    }
   ],
   "source": [
    "import numpy as np  # Convention is to import and rename. Best to stick with convention\n",
    "\n",
    "print('type(np.array(1)):', type(np.array(1)))\n",
    "print()\n",
    "\n",
    "print('np.int8:', np.array(1, dtype=np.int8).dtype)    # Numpy integer of size 8 bytes\n",
    "print('np.int16:', np.array(1, dtype=np.int16).dtype)  # Numpy integer of size 16 bytes\n",
    "print('np.int32:', np.array(1, dtype=np.int32).dtype)  # Numpy integer of size 32 bytes\n",
    "print('np.int64:', np.array(1, dtype=np.int64).dtype)  # Numpy integer of size 64 bytes\n",
    "print('np.int:', np.array(1, dtype=int).dtype)         # Default Numpy integer of size 64 bytes\n",
    "print()\n",
    "\n",
    "print('np.float16:', np.array(1, dtype=np.float16).dtype)  # Numpy float of size 16 bytes\n",
    "print('np.float32:', np.array(1, dtype=np.float32).dtype)   # Numpy float of size 32 bytes\n",
    "print('np.float64:', np.array(1, dtype=np.float64).dtype)   # Numpy float of size 64 bytes\n",
    "print('np.float:', np.array(1, dtype=float).dtype)          # Numpy float of size 64 bytes\n",
    "print()\n",
    "\n",
    "print('np.bool:', np.array(1, dtype=np.bool_).dtype)    # Numpy boolean of size 8 bytes\n",
    "print('np.int:', np.array(1, dtype=np.int_).dtype)      # Numpy integer of size 64 bytes\n",
    "print('np.foat:', np.array(1, dtype=np.float_).dtype)   # Numpy float of size 64 bytes\n",
    "print('np.uint:', np.array(1, dtype=np.uint).dtype)     # Numpy unsigned integer of size 64 bytes\n",
    "print('np.complex:', np.array(1, dtype=complex).dtype)  # Numpy complex data object of size 128 bytes\n",
    "print('Hello World:', np.array('Hello World').dtype)    # <U11 means 11 charater Unicode String."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4efbde7",
   "metadata": {},
   "source": [
    "Numpy is based on arrays of data, with all values having the same data type in the array. This greatly increases computational performance. You can convert between basic Python and Numpy data space easily and quickly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f3ccba19",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(a): <class 'list'>\n",
      "type(b): <class 'numpy.ndarray'>\n",
      "type(b[0]): <class 'numpy.int64'>\n",
      "b.dtype: int64\n"
     ]
    }
   ],
   "source": [
    "a = [1, 2, 3]  # Create a Python list\n",
    "b = np.array([1, 2, 3])  # Create a Numpy 1-D array\n",
    "print('type(a):', type(a))  # Get type of a\n",
    "print('type(b):', type(b))  # Get type of b\n",
    "print('type(b[0]):', type(b[0]))  # Get type of index 0 of b\n",
    "print('b.dtype:', b.dtype)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5cdefbd",
   "metadata": {},
   "source": [
    "Using Numpy methods on a Python object will cause an error. Also, notice how the printout of the two are similar but slightly different. The Python list has commas, while the Numpy array does not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "03a0bebd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: [1, 2, 3]\n",
      "b: [1 2 3]\n"
     ]
    }
   ],
   "source": [
    "# print('a.dtype:', a.dtype)  # a.dtype will not work. dtype is numpy only.\n",
    "print('a:', a)  \n",
    "print('b:', b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fce6227",
   "metadata": {},
   "source": [
    "### Basic Python vs Numpy performance\n",
    "Basic Python can perform computations relatively fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d832f295",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.57 ms ± 38.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "num = 100_000\n",
    "a = list(range(0, num))  # Need to formally convert to list because of timeit\n",
    "for ii in a:\n",
    "    a[ii] = a[ii] + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39532798",
   "metadata": {},
   "source": [
    "The computation can be faster for some applications using list comprehensions. This is performing the same computation just with a different syntax performing the computations in a more optimized way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7c1ef03d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.9 ms ± 80.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "num = 100_000\n",
    "a = list(range(0, num))\n",
    "a = [a[ii] + 1 for ii in a]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38baa490",
   "metadata": {},
   "source": [
    "But using Numpy is significantly faster for computations because the code is highly optimized and uses pre-compiled C libraries under the hood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ea2be3fa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18.2 µs ± 85.5 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%%timeit\n",
    "num = 100_000\n",
    "a = np.arange(num, dtype=np.int32) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eb1c17a",
   "metadata": {},
   "source": [
    "### Switching between Basic Python space and Numpy space"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d3d974",
   "metadata": {},
   "source": [
    "Create some data using Python lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "af366b33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(a): <class 'list'>\n",
      "len(a): 10000\n"
     ]
    }
   ],
   "source": [
    "num = 10000\n",
    "a = list(range(0, num))  # Defaults to integer values\n",
    "print('type(a):', type(a))\n",
    "print('len(a):', len(a))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b9e573e",
   "metadata": {},
   "source": [
    "Convert the python list into numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b5b929f6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(b): <class 'numpy.ndarray'>\n",
      "b.size: 10000\n",
      "b.shape: (10000,)\n"
     ]
    }
   ],
   "source": [
    "b = np.array(a)\n",
    "print('type(b):', type(b))\n",
    "print('b.size:', b.size)\n",
    "print('b.shape:', b.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60265ac3",
   "metadata": {},
   "source": [
    "Convert the numpy array back into a python list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "04cb37ff",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "type(c): <class 'list'>\n",
      "len(c): 10000\n"
     ]
    }
   ],
   "source": [
    "c = list(b)\n",
    "print('type(c):', type(c))\n",
    "print('len(c):', len(c))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a511ac4d",
   "metadata": {},
   "source": [
    "### Making Numpy Arrays\n",
    "Numpy also comes with a large number of methods to create data arrays so you don't have to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b8e29d3f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: [0 1 2 3 4 5 6 7 8 9]\n",
      "b: [0 1 2 3 4 5 6 7 8 9]\n",
      "c: [1 3 5 7 9]\n",
      "d: [10  9  8  7  6  5  4  3  2]\n",
      "e: [9 8 7 6 5 4 3 2 1 0]\n"
     ]
    }
   ],
   "source": [
    "a = np.array(range(10))   # Converts the iterator range function to numpy array\n",
    "b = np.arange(10)         # Creates the numpy array directly and faster\n",
    "c = np.arange(1, 10, 2)   # Creates array counting by two\n",
    "d = np.arange(10, 1, -1)  # Creates array descending by one\n",
    "e = np.flip(a)            # Reverses array a from increasing to decreasing\n",
    "print('a:', a)\n",
    "print('b:', b)\n",
    "print('c:', c)\n",
    "print('d:', d)\n",
    "print('e:', e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7efab924",
   "metadata": {},
   "source": [
    "### Indexing, Slicing, Broadcasting\n",
    "How do you get specific data from the Numpy array? Numpy uses the same slicing as Python lists - start number to (but not including) end number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "38451e15",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: [0 1 2 3 4 5 6 7 8 9]\n",
      "a[0:5]: [0 1 2 3 4]\n",
      "a[3:]: [3 4 5 6 7 8 9]\n",
      "a[:5]: [0 1 2 3 4]\n",
      "a[3:5]: [3 4]\n",
      "a[0:-1]: [0 1 2 3 4 5 6 7 8]\n",
      "a[index] [2 5 6 8]\n",
      "a[index] [0 1 4 5 8 9]\n",
      "a[0:100]: [0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "print('a:', a)\n",
    "print('a[0:5]:', a[0:5])  # selects up to but not including index 5\n",
    "print('a[3:]:', a[3:])  # selects everything from 3 to end of array\n",
    "print('a[:5]:', a[:5])  # selects up to but not including index 5\n",
    "print('a[3:5]:', a[3:5])  # selects from 3 up to but not including index 5\n",
    "print('a[0:-1]:', a[:-1])  # selects up to but not including index 9\n",
    "index = [2, 5, 6, 8]  # Create an index list of specific indices we want\n",
    "print('a[index]', a[index])  # Use the index list to return only those indices\n",
    "# Create a list of boolean values indicating which indices to keep\n",
    "index = [True, True, False, False, True, True, False, False, True, True]\n",
    "print('a[index]', a[index])  # Return only where index is True\n",
    "print('a[0:100]:', a[0:100])  # index is past end of array?!?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b775a34f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c:\n",
      " [[0 1 2 3 4]\n",
      " [5 6 7 8 9]]\n",
      "c.shape: (2, 5)\n",
      "c.size: 10\n",
      "\n",
      "a:\n",
      " [[0. 0.]\n",
      " [0. 0.]]\n",
      "\n",
      "b:\n",
      " [[1 1]\n",
      " [1 1]]\n",
      "\n",
      "c:\n",
      " [[7 7]\n",
      " [7 7]]\n",
      "\n",
      "d:\n",
      " [[1. 0. 0.]\n",
      " [0. 1. 0.]\n",
      " [0. 0. 1.]]\n",
      "\n",
      "e: [[0.97174353 0.0345394  0.86575348 0.34057533]\n",
      " [0.58298364 0.58011791 0.52108608 0.43024907]]\n"
     ]
    }
   ],
   "source": [
    "c = np.arange(10)  # Create a 1-D array\n",
    "c = c.reshape((2, 5))  # Change to a 2-D array\n",
    "print('c:\\n', c)\n",
    "print('c.shape:', c.shape)\n",
    "print('c.size:', c.size)\n",
    "print()\n",
    "\n",
    "a = np.zeros((2, 2))    # Create an array of all zeros. Notice it defaults to type float\n",
    "print('a:\\n', a)\n",
    "\n",
    "b = np.ones((2, 2), dtype=int)    # Create an array of all ones\n",
    "print('\\nb:\\n', b)\n",
    "\n",
    "c = np.full((2, 2), 7, dtype=np.int16)   # Create a constant array\n",
    "print('\\nc:\\n', c)\n",
    "\n",
    "d = np.eye(3)           # Create a 3x3 identity matrix\n",
    "print('\\nd:\\n', d)\n",
    "\n",
    "e = np.random.random((2, 4))  # Create a float array filled with random values between 0 and 1\n",
    "print('\\ne:', e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f1b514",
   "metadata": {},
   "source": [
    "One of the most important features of Numpy is Broadcasting, where a single operation is performed on all values of the Numpy array without the need for a loop. This creates more simple and readable code, and is significantly faster. The general rule is if it's possible to remove a loop by Broadcasting, do it!\n",
    "\n",
    "Create an array of all zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b89b166c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.zeros(20, dtype=int)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3016749b",
   "metadata": {},
   "source": [
    "Add a value of 1 to every value in the array. Also notice that the initial array was of type integer. But we are adding a float, so all values are first upconverted to type float and then a value of 1.0 is added to every value in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "84cd68c2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1.])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = a + 1.0  # Note how it upconverted from int to float\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31991094",
   "metadata": {},
   "source": [
    "Here we change the data type of the entire array from float to integer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "99418ef2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = a.astype(int)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dbc445e",
   "metadata": {},
   "source": [
    "Here we add a value to a subset of the array by adding 10 to values matching index from 3 to 7. The rest of the values are unchanged."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4df83133",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1,  1,  1, 11, 11, 11, 11, 11,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[3:8] = a[3:8] + 10\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f604093c",
   "metadata": {},
   "source": [
    "Here we use shorthand notation to add a constant of 1000 to every value in the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8b089c87",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1001, 1001, 1001, 1011, 1011, 1011, 1011, 1011, 1001, 1001, 1001,\n",
       "       1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a += 1000\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90d0ee4e",
   "metadata": {},
   "source": [
    "### The IEEE Not A Number value\n",
    "There is a special Numpy value called NaN which is used in calculations to represent a value that is not a number. Think of it as missing data, or a bad value that should not be propagated through the data. One of the most important things to remember about NaN is that it taints anything it touches!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b3c3cd73",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n",
      "<class 'float'>\n",
      "nan nan nan nan\n"
     ]
    }
   ],
   "source": [
    "print(np.nan)\n",
    "print(type(np.nan))   # Python type says is of type float\n",
    "print(11 + np.nan, 12 - np.nan, 13 * np.nan, 14 / np.nan) # Anything that uses NaN becomes a NaN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86274170",
   "metadata": {},
   "source": [
    "Because NaN is special, it acts funny. Looking for NaN requires some specific logic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2a04bfe6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 == 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "8665530f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.zeros(1, dtype=int) == np.zeros(1, dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4720fd60",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.nan == np.nan  # What is going on here?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b561dc13",
   "metadata": {},
   "source": [
    "Will need to use the Numpy methods of searching the arrays if you want to do comparisons for finding where NaNs are located"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ea32da25",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.isnan(np.nan)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79dca5ff",
   "metadata": {},
   "source": [
    "To use NaNs properly requires a litte bit of extra thought, but is greatly worth it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "dad28b12",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a: [0. 1. 2. 3. 4. 5. 6. 7. 8. 9.]\n",
      "a.min(): 0.0\n",
      "np.min(a): 0.0\n"
     ]
    }
   ],
   "source": [
    "a = np.arange(10, dtype=float)\n",
    "print('a:', a)\n",
    "print('a.min():', a.min())      # This is Python min method, not numpy. Maybe slower?\n",
    "print('np.min(a):', np.min(a))  # This is numpy min function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6086e591",
   "metadata": {},
   "source": [
    "Now we will assign a value in the Numpy array to NaN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "742ce176",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([nan,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0] = np.nan\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a532ce7",
   "metadata": {},
   "source": [
    "Calculate the mean. Notice how it returns NaN. The NaNs are tainting all the operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ece1a980",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nan"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9653677c",
   "metadata": {},
   "source": [
    "We need to use a different method that understands how to correctly handle NaNs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9b7bbadb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.nanmean(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dead5cbf",
   "metadata": {},
   "source": [
    "### Workign with time in Numpy\n",
    "If you deal with time series data you will need to work with time data types. Oh that's right, most atmospheric data is time series data...\n",
    "\n",
    "Python has a library called _datetime_ that is great for working with dates and times. It is timezone unaware and works on one value at a time. This is a native data type within Python and you will see it all over the place. Here are some quick examples:\n",
    "\n",
    "Get the datetime now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "76220159",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2023, 10, 27, 9, 32, 28, 377319)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datetime\n",
    "\n",
    "datetime.datetime.now()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccc74087",
   "metadata": {},
   "source": [
    "Or get the datetime in UTC timezone."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4e8c7ee7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2023, 10, 27, 15, 32, 28, 380399)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.datetime.utcnow()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cb33ae1",
   "metadata": {},
   "source": [
    "Or from a timestamp of number of seconds from Epoch, Seconds from 1970-01-01 00:00:00: UTC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "783d9f86",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2012, 1, 10, 18, 12, 44)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datetime.datetime.fromtimestamp(1326244364)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6fd2ebe0",
   "metadata": {},
   "source": [
    "But when working with a lot of time samples we need to use the Numpy time data type. It can take in a string and convert to a Numpy datetime64 value. Notice when it prints out, it will print out in ISO format."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f918a73",
   "metadata": {},
   "source": [
    "To just get the date right now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "bc52b55c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2023-10-27')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('today')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fe6434",
   "metadata": {},
   "source": [
    "To get the date and time right now. Notice the time is in UTC. Most likely this is what you want, if not understand how to get what you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "c42b1f7a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2023-10-27T15:32:28')"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('now')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "dc2b8e89",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2005-02-25T03:30:55')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2005-02-25 03:30:55')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3797edd1",
   "metadata": {},
   "source": [
    "The default is to store the precision at the level of input. But we can update the precision by setting precision when setting the value or after the value is created to the desired precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "44214c8f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2012-03')"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2012-03')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e997f38b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2012-03-01T00:00:00')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2012-03', 's')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "67b0851f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2012-03-01T00:00:00')"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2012-03').astype('datetime64[s]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b973387",
   "metadata": {},
   "source": [
    "OK, so you can set precision. Why does it matter? Because you can use the precision to indicate what step size to use. Notice the following range did not provide a starting or ending day. This helps with not needing to know length of months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f61b8513",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['2005-02-01', '2005-02-02', '2005-02-03', '2005-02-04',\n",
       "       '2005-02-05', '2005-02-06', '2005-02-07', '2005-02-08',\n",
       "       '2005-02-09', '2005-02-10', '2005-02-11', '2005-02-12',\n",
       "       '2005-02-13', '2005-02-14', '2005-02-15', '2005-02-16',\n",
       "       '2005-02-17', '2005-02-18', '2005-02-19', '2005-02-20',\n",
       "       '2005-02-21', '2005-02-22', '2005-02-23', '2005-02-24',\n",
       "       '2005-02-25', '2005-02-26', '2005-02-27', '2005-02-28'],\n",
       "      dtype='datetime64[D]')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange('2005-02', '2005-03', dtype='datetime64[D]')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af05274c",
   "metadata": {},
   "source": [
    "There is also a timedelta64 data type that is the result of differencing times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "590ddfc3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.timedelta64(366,'D')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2009-01-01') - np.datetime64('2008-01-01')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc5cbf44",
   "metadata": {},
   "source": [
    "Time Deltas are important for adding or subtracting times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "4fd1f5c5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2011-06-16T10:00')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.datetime64('2011-06-15T22:00') + np.timedelta64(12, 'h')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "155a0e48",
   "metadata": {},
   "source": [
    "### Converting between Python datetime and Numpy datetime64\n",
    "At some point you will need to convert a time value from one space to another. Don't memorize this, just remember it exists and where you can find the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "2fe68059",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-10-27 15:32:28.416906\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "numpy.datetime64('2023-10-27T15:32:28.416906')"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt = datetime.datetime.utcnow()  # Get current date and time with python datetime\n",
    "print(dt)\n",
    "# Convert from Python datetime to Numpy datetime64\n",
    "dt64 = np.datetime64(dt)\n",
    "dt64"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64cebd63",
   "metadata": {},
   "source": [
    "It is a bit more complicated to convert from Numpy datetime64 to Python datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "fba4dfb2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-10-27T15:32:28\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "datetime.datetime(2023, 10, 27, 15, 32, 28)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set precision to datetime64 seconds and convert to Numpy integer.\n",
    "# This will give the number of seconds since epoch, or timestamp.\n",
    "dt64 = np.datetime64('now')\n",
    "print(dt64)\n",
    "\n",
    "dt64 = dt64.astype('datetime64[s]')\n",
    "ts = dt64.astype(int)\n",
    "\n",
    "# Then use that integer number of seconds into from time stamp method.\n",
    "datetime.datetime.utcfromtimestamp(ts)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}