t81_558_class01_intro_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 1: Python Preliminaries**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), School of Engineering and Applied Science, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module Video Material\n",
    "\n",
    "Main video lecture ([complete YouTube Playlist](https://www.youtube.com/playlist?list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)):\n",
    "\n",
    "* [Part 1.1: Course Overview](https://youtu.be/CnCf1QGsjx8)\n",
    "* [Part 1.2: Machine Learning Background for Deep Learning, Keras and Tensorflow](https://www.youtube.com/watch?v=807L9m5dWng&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Part 1.3: Python Anaconda for Deep Learning, Keras and Tensorflow](https://www.youtube.com/watch?v=uOMhboAnVNk&index=3&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [How to Submit a Module Assignment](https://www.youtube.com/watch?v=hmCGjCVhYNc)\n",
    "\n",
    "Watch one (or more) of these depending on how you want to setup your Python TensorFlow environment:\n",
    "\n",
    "* [Installing TensorFlow, Keras, and Python in Windows](https://www.youtube.com/watch?v=z0qhKP2liHs&index=5&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Installing TensorFlow, Keras, and Python in Mac](https://www.youtube.com/watch?v=RUs_qR1hKds&index=4&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN)\n",
    "* [Google Colab](https://colab.research.google.com) - Instructions coming soon\n",
    "* [Installing/Using IBM Cognitive Class Labs with TensorFlow/Keras](https://www.youtube.com/watch?v=CyWxFqMVOvg)\n",
    "* [Docker Image](https://hub.docker.com/r/heatonresearch/jupyter-python-r/) - A docker image that I created specifically for this class.  Always tested with all class assignments and notes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helpful Functions\n",
    "\n",
    "You will see these at the top of every module.  These are simply a set of reusable functions that we will make use of.  Each of them will be explained as the semester progresses.  They are explained in greater detail as the course progresses.  Class 4 contains a complete overview of these functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "import os\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "from sklearn import preprocessing\n",
    "\n",
    "\n",
    "# Encode text values to dummy variables(i.e. [1,0,0],[0,1,0],[0,0,1] for red,green,blue)\n",
    "def encode_text_dummy(df, name):\n",
    "    dummies = pd.get_dummies(df[name])\n",
    "    for x in dummies.columns:\n",
    "        dummy_name = f\"{name}-{x}\"\n",
    "        df[dummy_name] = dummies[x]\n",
    "    df.drop(name, axis=1, inplace=True)\n",
    "\n",
    "\n",
    "# Encode text values to a single dummy variable.  The new columns (which do not replace the old) will have a 1\n",
    "# at every location where the original column (name) matches each of the target_values.  One column is added for\n",
    "# each target value.\n",
    "def encode_text_single_dummy(df, name, target_values):\n",
    "    for tv in target_values:\n",
    "        l = list(df[name].astype(str))\n",
    "        l = [1 if str(x) == str(tv) else 0 for x in l]\n",
    "        name2 = f\"{name}-{tv}\"\n",
    "        df[name2] = l\n",
    "\n",
    "\n",
    "# Encode text values to indexes(i.e. [1],[2],[3] for red,green,blue).\n",
    "def encode_text_index(df, name):\n",
    "    le = preprocessing.LabelEncoder()\n",
    "    df[name] = le.fit_transform(df[name])\n",
    "    return le.classes_\n",
    "\n",
    "\n",
    "# Encode a numeric column as zscores\n",
    "def encode_numeric_zscore(df, name, mean=None, sd=None):\n",
    "    if mean is None:\n",
    "        mean = df[name].mean()\n",
    "\n",
    "    if sd is None:\n",
    "        sd = df[name].std()\n",
    "\n",
    "    df[name] = (df[name] - mean) / sd\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the median\n",
    "def missing_median(df, name):\n",
    "    med = df[name].median()\n",
    "    df[name] = df[name].fillna(med)\n",
    "\n",
    "\n",
    "# Convert all missing values in the specified column to the default\n",
    "def missing_default(df, name, default_value):\n",
    "    df[name] = df[name].fillna(default_value)\n",
    "\n",
    "\n",
    "# Convert a Pandas dataframe to the x,y inputs that TensorFlow needs\n",
    "def to_xy(df, target):\n",
    "    result = []\n",
    "    for x in df.columns:\n",
    "        if x != target:\n",
    "            result.append(x)\n",
    "    # find out the type of the target column.  Is it really this hard? :(\n",
    "    target_type = df[target].dtypes\n",
    "    target_type = target_type[0] if hasattr(\n",
    "        target_type, '__iter__') else target_type\n",
    "    # Encode to int for classification, float otherwise. TensorFlow likes 32 bits.\n",
    "    if target_type in (np.int64, np.int32):\n",
    "        # Classification\n",
    "        dummies = pd.get_dummies(df[target])\n",
    "        return df[result].values.astype(np.float32), dummies.values.astype(np.float32)\n",
    "    # Regression\n",
    "    return df[result].values.astype(np.float32), df[[target]].values.astype(np.float32)\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\"\n",
    "\n",
    "\n",
    "# Regression chart.\n",
    "def chart_regression(pred, y, sort=True):\n",
    "    t = pd.DataFrame({'pred': pred, 'y': y.flatten()})\n",
    "    if sort:\n",
    "        t.sort_values(by=['y'], inplace=True)\n",
    "    plt.plot(t['y'].tolist(), label='expected')\n",
    "    plt.plot(t['pred'].tolist(), label='prediction')\n",
    "    plt.ylabel('output')\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "\n",
    "# Remove all rows where the specified column is +/- sd standard deviations\n",
    "def remove_outliers(df, name, sd):\n",
    "    drop_rows = df.index[(np.abs(df[name] - df[name].mean())\n",
    "                          >= (sd * df[name].std()))]\n",
    "    df.drop(drop_rows, axis=0, inplace=True)\n",
    "\n",
    "\n",
    "# Encode a column to a range between normalized_low and normalized_high.\n",
    "def encode_numeric_range(df, name, normalized_low=-1, normalized_high=1,\n",
    "                         data_low=None, data_high=None):\n",
    "    if data_low is None:\n",
    "        data_low = min(df[name])\n",
    "        data_high = max(df[name])\n",
    "\n",
    "    df[name] = ((df[name] - data_low) / (data_high - data_low)) \\\n",
    "        * (normalized_high - normalized_low) + normalized_low\n",
    "\n",
    "\n",
    "# This function submits an assignment.  You can submit an assignment as much as you like, only the final\n",
    "# submission counts.  The paramaters are as follows:\n",
    "# data - Pandas dataframe output.\n",
    "# key - Your student key that was emailed to you.\n",
    "# no - The assignment class number, should be 1 through 1.\n",
    "# source_file - The full path to your Python or IPYNB file.  This must have \"_class1\" as part of its name.  \n",
    "# .             The number must match your assignment number.  For example \"_class2\" for class assignment #2.\n",
    "def submit(data,key,no,source_file=None):\n",
    "    if source_file is None and '__file__' not in globals(): raise Exception('Must specify a filename when a Jupyter notebook.')\n",
    "    if source_file is None: source_file = __file__\n",
    "    suffix = '_class{}'.format(no)\n",
    "    if suffix not in source_file: raise Exception('{} must be part of the filename.'.format(suffix))\n",
    "    with open(source_file, \"rb\") as image_file:\n",
    "        encoded_python = base64.b64encode(image_file.read()).decode('ascii')\n",
    "    ext = os.path.splitext(source_file)[-1].lower()\n",
    "    if ext not in ['.ipynb','.py']: raise Exception(\"Source file is {} must be .py or .ipynb\".format(ext))\n",
    "    r = requests.post(\"https://api.heatonresearch.com/assignment-submit\",\n",
    "        headers={'x-api-key':key}, json={'csv':base64.b64encode(data.to_csv(index=False).encode('ascii')).decode(\"ascii\"),\n",
    "        'assignment': no, 'ext':ext, 'py':encoded_python})\n",
    "    if r.status_code == 200:\n",
    "        print(\"Success: {}\".format(r.text))\n",
    "    else: print(\"Failure: {}\".format(r.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course Description\n",
    "\n",
    "Deep learning is a group of exciting new technologies for neural networks. By using a combination of advanced training techniques neural network architectural components, it is now possible to train neural networks of much greater complexity. This course will introduce the student to deep belief neural networks, regularization units (ReLU), convolution neural networks and recurrent neural networks. High performance computing (HPC) aspects will demonstrate how deep learning can be leveraged both on graphical processing units (GPUs), as well as grids. Deep learning allows a model to learn hierarchies of information in a way that is similar to the function of the human brain. Focus will be primarily upon the application of deep learning, with some introduction to the mathematical foundations of deep learning. Students will use the Python programming language to architect a deep learning model for several of real-world data sets and interpret the results of these networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignments\n",
    "\n",
    "Your grade will be calculated according to the following assignments:\n",
    "\n",
    "Assignment          |Weight|Title\n",
    "--------------------|------|-------\n",
    "Class Participation |   10%|Class attendance and participation\n",
    "Class Assignments   |   50%|10 small programming assignments (5% each)\n",
    "Kaggle Project      |   20%|\"Kaggle In-Class\" project submitted through Kaggle\n",
    "Final Project       |   20%|A computer security project\n",
    "\n",
    "The 10 class assignments will be assigned with each of the first 10 modules.  Generally, each module assignment is due just before the following module date.  Refer to syllabus for exact due dates.  The 10 class assignments are submitted using the Python submission script.  Refer to assignment 1 for details.\n",
    "\n",
    "Module   |Title\n",
    "---------|--------------\n",
    "Module 1 |[Simple Test Submission](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb)\n",
    "Module 2 |[Data Preparation in Pandas](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class2.ipynb)\n",
    "Module 3 |[Creating Columns in Pandas](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class3.ipynb)\n",
    "Module 4 |[Regression Neural Network](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class4.ipynb)\n",
    "Module 5 |[K-Fold Cross-Validation](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class5.ipynb)\n",
    "Module 6 |[Classification Neural Network](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class6.ipynb)\n",
    "Module 7 |[Computer Vision Neural Network](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class7.ipynb)\n",
    "Module 8 |[Building a Kaggle Submission File](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class8.ipynb)\n",
    "Module 9 |[Exploring Regularization](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class9.ipynb)\n",
    "Module 10|[Time Series Neural Network](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class10.ipynb)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Your Instructor: Jeff Heaton\n",
    "\n",
    "![Jeff Heaton at WUSTL Video Studio](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/jheaton1.png \"Jeff Heaton\")\n",
    "\n",
    "I will be your instructor for this course.  A brief summary of my credentials is given here:\n",
    "\n",
    "* Master of Information Management (MIM), Washington University in St. Louis, MO\n",
    "* PhD in Computer Science, Nova Southeastern University in Ft. Lauderdale, FL\n",
    "* [Vice President and Data Scientist](http://www.rgare.com/knowledge-center/media/articles/rga-where-logic-meets-curiosity), Reinsurance Group of America (RGA)\n",
    "* Senior Member, IEEE\n",
    "* jtheaton at domain name of this university\n",
    "* Other industry certifications: FLMI, ARA, ACS\n",
    "\n",
    "Social media:\n",
    "\n",
    "* [Homepage](http://www.heatonresearch.com) - My home page.  Includes my research interests and publications.\n",
    "* [Linked In](https://www.linkedin.com/in/jeffheaton) - My Linked In profile, feel free to connect.\n",
    "* [Twitter](https://twitter.com/jeffheaton) - My Twitter feed.\n",
    "* [Google Scholar](https://scholar.google.com/citations?user=1jPGeg4AAAAJ&hl=en) - My citations on Google Scholar.\n",
    "* [Research Gate](https://www.researchgate.net/profile/Jeff_Heaton) - My profile/research at Research Gate.\n",
    "* [Others](http://www.heatonresearch.com/about/) - About me and other social media sites that I am a member of."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Course Resources\n",
    "\n",
    "* [IBM Cognitive Class Labs](https://www.datascientistworkbench.com) - Free web-based platform that includes Python, Juypter Notebooks, and TensorFlow.  No setup needed.\n",
    "* [Python Anaconda](https://www.continuum.io/downloads) - Python distribution that includes many data science packages, such as Numpy, Scipy, Scikit-Learn, Pandas, and much more.\n",
    "* [Juypter Notebooks](http://jupyter.org/) - Easy to use environment that combines Python, Graphics and Text. \n",
    "* [TensorFlow](https://www.tensorflow.org/) - Google's mathematics package for deep learning.\n",
    "* [Kaggle](https://www.kaggle.com/) - Competitive data science.  Good source of sample data.\n",
    "* [Course GitHub Repository](https://github.com/jeffheaton/t81_558_deep_learning) - All of the course notebooks will be published here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What is Deep Learning\n",
    "\n",
    "The focus of this class is deep learning, which is a very popular type of machine learning that is based upon the original neural networks popularized in the 1980's. There is very little difference between how a deep neural network is calculated compared with the original neural network.  We've always been able to create and calculate deep neural networks.  A deep neural network is nothing more than a neural network with many layers.  While we've always been able to create/calculate deep neural networks, we've lacked an effective means of training them.  Deep learning provides an efficient means to train deep neural networks.\n",
    "\n",
    "## What is Machine Learning\n",
    "\n",
    "If deep learning is a type of machine learning, this begs the question, \"What is machine learning?\"  The following diagram illustrates how machine learning differs from traditional software development.\n",
    "\n",
    "![ML vs Traditional Software Development](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_ml_vs_trad.png \"Machine Learning vs Traditional Software Development\")\n",
    "\n",
    "* **Traditional Software Development** - Programmers create programs that specify how to transform input into the desired output.\n",
    "* **Machine Learning** - Programmers create models that can learn to produce the desired output for given input. This learning fills the traditional role of the computer program. \n",
    "\n",
    "Researchers have applied machine learning to many different areas.  This class will explore three specific domains for the application of deep neural networks:\n",
    "\n",
    "![Application of Machine Learning](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_ml_types.png \"Application of Machine Learning\")\n",
    "\n",
    "* **Predictive Modeling** - Several named input values are used to predict another named value that becomes the output.  For example, using four measurements of iris flowers to predict the species.  \n",
    "* **Computer Vision** - The use of machine learning to detect patterns in visual data.  For example, is an image a cat or a dog.\n",
    "* **Time Series** - The use of machine learning to detect patterns in in time.  Common applications of time series are: financial applications, speech recognition, and even natural language processing (NLP). \n",
    "\n",
    "### Regression\n",
    "\n",
    "Regression is when a model, such as a neural network, accepts input and produces a numeric output.  Consider if you were tasked to write a program that predicted how many miles per gallon (MPG) a car could achieve.  For the inputs you would probably want such features as the weight of the car, the horsepower, how large the engine is, etc.  Your program would be a combination of math and if-statements.  \n",
    "\n",
    "Machine learning lets the computer learn the \"formula\" for calculating the MPG of a car, using data.  Consider [this](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/data/auto-mpg.csv) dataset.  We can use regression machine learning models to study this data and learn how to predict the MPG for a car. \n",
    "\n",
    "### Classification\n",
    "\n",
    "The output of a classification model is what class the input belongs to.  For example, consider using four measurements of an iris flower to determine the species that the flower is in.  This dataset could be used to perform [this](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/data/iris.csv).  \n",
    "\n",
    "\n",
    "## What are Neural Networks\n",
    "\n",
    "Neural networks are one of the earliest types of machine learning model.  Neural networks were originally introduced in the 1940's and have risen and fallen [several times from popularity](http://hushmagazine.ca/living-2/business/the-believers-the-hidden-story-behind-the-code-that-runs-our-lives). Four researchers have contributed greatly to the development of neural networks.  They have consistently pushed neural network research, both through the ups and downs: \n",
    "\n",
    "![Neural Network Luminaries](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_luminaries_ann.png \"Neural Network Luminaries\")\n",
    "\n",
    "The current luminaries of artificial neural network (ANN) research and ultimately deep learning, in order as appearing in the above picture:\n",
    "\n",
    "* [Yann LeCun](http://yann.lecun.com/), Facebook and New York University - Optical character recognition and computer vision using convolutional neural networks (CNN).  The founding father of convolutional nets.\n",
    "* [Geoffrey Hinton](http://www.cs.toronto.edu/~hinton/), Google and University of Toronto. Extensive work on neural networks. Creator of deep learning and early adapter/creator of backpropagation for neural networks.\n",
    "* [Yoshua Bengio](http://www.iro.umontreal.ca/~bengioy/yoshua_en/index.html), University of Montreal. Extensive research into deep learning, neural networks, and machine learning.  He has so far remained completely in academia.\n",
    "* [Andrew Ng](http://www.andrewng.org/), Badiu and Stanford University.  Extensive research into deep learning, neural networks, and application to robotics.\n",
    "\n",
    "## Why Deep Learning?\n",
    "\n",
    "For predictive modeling neural networks are not that different than other models, such as:\n",
    "\n",
    "* Support Vector Machines\n",
    "* Random Forests\n",
    "* Gradient Boosted Machines\n",
    "\n",
    "Like these other models, neural networks can perform both **classification** and **regression**.  When applied to relatively low-dimensional predictive modeling tasks, deep neural networks do not necessarily add significant accuracy over other model types.  Andrew Ng describes the advantage of deep neural networks over traditional model types as follows:\n",
    "\n",
    "![Why Deep Learning?](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/class_1_why_deep.png \"Why Deep Learning\")\n",
    "\n",
    "Neural networks also have two additional significant advantages over other machine learning models:\n",
    "\n",
    "* **Convolutional Neural Networks** - Can scan an image for patterns within the image.\n",
    "* **Recurrent Neural Networks** - Can find patterns across several inputs, not just within a single input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python for Deep Learning\n",
    "\n",
    "Python 3.x is the programming language that will be used for this class.  Python, as a programming language, has the widest support for deep learning.  The three most popular frameworks for deep learning in Python are:\n",
    "\n",
    "* [TensorFlow](https://www.tensorflow.org/) (Google)\n",
    "* [MXNet](https://github.com/dmlc/mxnet) (Amazon)\n",
    "* [CNTK](https://cntk.ai/) (Microsoft)\n",
    "* [Theano](http://deeplearning.net/software/theano/) (University of Montreal) - Popular but discontinued. \n",
    "\n",
    "Some references on popular programming languages for AI/Data Science:\n",
    "\n",
    "* [Popular Programming Languages for AI](https://en.wikipedia.org/wiki/List_of_programming_languages_for_artificial_intelligence)\n",
    "* [Popular Programming Languages for Data Science](http://www.kdnuggets.com/2014/08/four-main-languages-analytics-data-mining-data-science.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Software Installation\n",
    "This is a technical class.  You will need to be able to compile and execute Python code that makes use of TensorFlow for deep learning. There are two options to you for accomplish this:\n",
    "\n",
    "* Install Python, TensorFlow and some IDE (Jupyter, TensorFlow, etc.)\n",
    "* Use IBM Cognitive Class Labs online\n",
    "\n",
    "## Installing Python and TensorFlow\n",
    "\n",
    "It is possible to install and run Python/TensorFlow entirely from your own computer.  Google provides TensorFlow for Windows, Mac and Linux.  Previously, TensorFlow did not support Windows.  However, as of December 2016, TensorFlow supports Windows for both CPU and GPU operation.\n",
    "\n",
    "The first step is to install Python 3.6.  As of August 2017, this is the latest version of Python 3.  I recommend using the Miniconda (Anaconda) release of Python, as it already includes many of the data science related packages that will be needed by this class.  Anaconda directly supports: Windows, Mac and Linux.  Miniconda is the minimal set of features from the very large Anaconda Python distribution.  Download Miniconda from the following URL:\n",
    "\n",
    "* [Miniconda](https://conda.io/miniconda.html)\n",
    "\n",
    "Once Miniconda has been downloaded it is easy to install Jupyter notebooks with the following command:\n",
    "\n",
    "```\n",
    "conda install jupyter\n",
    "```\n",
    "\n",
    "Once Jupyter is installed, it is started with the following command:\n",
    "\n",
    "```\n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "The following packages are needed for this course:\n",
    "\n",
    "```\n",
    "conda install scipy\n",
    "pip install --upgrade sklearn\n",
    "pip install --upgrade pandas\n",
    "pip install --upgrade pandas-datareader\n",
    "pip install --upgrade matplotlib\n",
    "pip install --upgrade pillow\n",
    "pip install --upgrade requests\n",
    "pip install --upgrade h5py\n",
    "pip install --upgrade psutil\n",
    "pip install --upgrade tensorflow==1.12.0\n",
    "pip install --upgrade keras==2.2.4\n",
    "```\n",
    "\n",
    "Notice that I am installing as specific version of TensorFlow.  As of the current semester, this is the latest version of TensorFlow.  It is very likely that Google will upgrade this during this semester. The newer version may have some incompatibilities, so it is important that we start with this version and end with the same.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Introduction\n",
    "\n",
    "\n",
    "* [Anaconda v3.6](https://www.continuum.io/downloads) Scientific Python Distribution, including:\n",
    "    * [Scikit-Learn](http://scikit-learn.org/)\n",
    "    * [Pandas](http://pandas.pydata.org/)\n",
    "    * Others: csv, json, numpy, scipy\n",
    "* [Jupyter Notebooks](http://jupyter.readthedocs.io/en/latest/install.html)\n",
    "* [PyCharm IDE](https://www.jetbrains.com/pycharm/)\n",
    "* [Cx_Oracle](http://cx-oracle.sourceforge.net/)\n",
    "* [MatPlotLib](http://matplotlib.org/)\n",
    "\n",
    "## Jupyter Notebooks\n",
    "\n",
    "Space matters in Python, indent code to define blocks\n",
    "\n",
    "Jupyter Notebooks Allow Python and Markdown to coexist.\n",
    "\n",
    "Even $\\LaTeX$:\n",
    "\n",
    "$ f'(x) = \\lim_{h\\to0} \\frac{f(x+h) - f(x)}{h}. $\n",
    "\n",
    "## Python Versions\n",
    "\n",
    "* If you see `xrange` instead of `range`, you are dealing with Python 2\n",
    "* If you see `print x` instead of `print(x)`, you are dealing with Python 2 \n",
    "* This class uses Python 3.6!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensor Flow Version: 1.12.0\n",
      "Keras Version: 2.2.4\n",
      "\n",
      "Python 3.6.7 |Anaconda, Inc.| (default, Oct 23 2018, 14:01:38) \n",
      "[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n",
      "Pandas 0.23.4\n",
      "Scikit-Learn 0.20.1\n"
     ]
    }
   ],
   "source": [
    "# What version of Python do you have?\n",
    "import sys\n",
    "\n",
    "import keras\n",
    "import pandas as pd\n",
    "import sklearn as sk\n",
    "import tensorflow as tf\n",
    "\n",
    "print(f\"Tensor Flow Version: {tf.__version__}\")\n",
    "print(f\"Keras Version: {keras.__version__}\")\n",
    "print()\n",
    "print(f\"Python {sys.version}\")\n",
    "print(f\"Pandas {pd.__version__}\")\n",
    "print(f\"Scikit-Learn {sk.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How Powerful is your Machine?\n",
    "The following code will tell you the basic stats on your machine for deep learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your CPU is: i386\n",
      "Your CPU has 8 cores.\n",
      "Total memory: 16.0GiB\n",
      "svmem(total=17179869184, available=4795609088, percent=72.1, used=10883473408, free=609239040, active=5799849984, inactive=4164542464, wired=5083623424)\n",
      "Installed GPUs:\n"
     ]
    }
   ],
   "source": [
    "import multiprocessing\n",
    "import platform\n",
    "import psutil\n",
    "from tensorflow.python.client import device_lib\n",
    "\n",
    "def sizeof_fmt(num, suffix='B'):\n",
    "    for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:\n",
    "        if abs(num) < 1024.0:\n",
    "            return \"%3.1f%s%s\" % (num, unit, suffix)\n",
    "        num /= 1024.0\n",
    "    return \"%.1f%s%s\" % (num, 'Yi', suffix)\n",
    "\n",
    "p = platform.processor()\n",
    "mem = psutil.virtual_memory()\n",
    "\n",
    "print(f'Your CPU is: {p}')\n",
    "print(f'Your CPU has {multiprocessing.cpu_count()} cores.')\n",
    "print(f'Total memory: {sizeof_fmt(mem.total)}')\n",
    "print(mem)\n",
    "\n",
    "local_device_protos = device_lib.list_local_devices()\n",
    "gpuList = [x for x in local_device_protos if x.device_type == 'GPU']\n",
    "\n",
    "print(\"Installed GPUs:\")\n",
    "for x in gpuList:\n",
    "  print(\"{} - {}\".format(x.name,x.physical_device_desc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Software used in this class:\n",
    "    \n",
    "* **Python** - The programming language.\n",
    "* **TensorFlow** - Googles deep learning framework, must have the version specified above. \n",
    "* **Keras** - [Keras](https://github.com/fchollet/keras) is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. \n",
    "* **Pandas** - Allows for data preprocessing.  Tutorial [here](http://pandas.pydata.org/pandas-docs/version/0.18.1/tutorials.html)\n",
    "* **Scikit-Learn** - Machine learning framework for Python.  Tutorial [here](http://scikit-learn.org/stable/tutorial/basic/tutorial.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Count to 10 in Python\n",
    "\n",
    "Use a **for** loop and a **range**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n",
      "4\n",
      "5\n",
      "6\n",
      "7\n",
      "8\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "#Python cares about space!  No curly braces.\n",
    "for x in range(1, 10):  # If you ever see xrange, you are in Python 2\n",
    "    print(x)  # If you ever see print x (no parenthesis), you are in Python 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Printing Numbers and Strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adding 1, sum so far is 1\n",
      "Adding 2, sum so far is 3\n",
      "Adding 3, sum so far is 6\n",
      "Adding 4, sum so far is 10\n",
      "Adding 5, sum so far is 15\n",
      "Adding 6, sum so far is 21\n",
      "Adding 7, sum so far is 28\n",
      "Adding 8, sum so far is 36\n",
      "Adding 9, sum so far is 45\n",
      "Final sum: 45\n"
     ]
    }
   ],
   "source": [
    "acc = 0\n",
    "for x in range(1, 10):\n",
    "    acc += x\n",
    "    print(f\"Adding {x}, sum so far is {acc}\")\n",
    "\n",
    "print(f\"Final sum: {acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists & Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'b', 'c', 'd']\n"
     ]
    }
   ],
   "source": [
    "c = ['a', 'b', 'c', 'd']\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "a\n",
      "b\n",
      "c\n",
      "d\n"
     ]
    }
   ],
   "source": [
    "# Iterate over a collection.\n",
    "for s in c:\n",
    "    print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:a\n",
      "1:b\n",
      "2:c\n",
      "3:d\n"
     ]
    }
   ],
   "source": [
    "# Iterate over a collection, and know where your index.  (Python is zero-based!)\n",
    "for i,c in enumerate(c):\n",
    "    print(f\"{i}:{c}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a', 'b', 'c', 'c']\n"
     ]
    }
   ],
   "source": [
    "# Manually add items, lists allow duplicates\n",
    "c = []\n",
    "c.append('a')\n",
    "c.append('b')\n",
    "c.append('c')\n",
    "c.append('c')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'b', 'c', 'a'}\n"
     ]
    }
   ],
   "source": [
    "# Manually add items, sets do not allow duplicates\n",
    "# Sets add, lists append.  I find this annoying.\n",
    "c = set()\n",
    "c.add('a')\n",
    "c.add('b')\n",
    "c.add('c')\n",
    "c.add('c')\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['a0', 'a', 'b', 'c']\n",
      "['a0', 'a', 'c']\n",
      "['a', 'c']\n"
     ]
    }
   ],
   "source": [
    "# Insert\n",
    "c = ['a', 'b', 'c']\n",
    "c.insert(0, 'a0')\n",
    "print(c)\n",
    "# Remove\n",
    "c.remove('b')\n",
    "print(c)\n",
    "# Remove at index\n",
    "del c[0]\n",
    "print(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Maps/Dictionaries/Hash Tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'Jeff', 'address': '123 Main'}\n",
      "Jeff\n",
      "Name is defined\n",
      "age undefined\n"
     ]
    }
   ],
   "source": [
    "d = {'name': \"Jeff\", 'address':\"123 Main\"}\n",
    "print(d)\n",
    "print(d['name'])\n",
    "\n",
    "if 'name' in d:\n",
    "    print(\"Name is defined\")\n",
    "\n",
    "if 'age' in d:\n",
    "    print(\"age defined\")\n",
    "else:\n",
    "    print(\"age undefined\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Key: dict_keys(['name', 'address'])\n",
      "Values: dict_values(['Jeff', '123 Main'])\n"
     ]
    }
   ],
   "source": [
    "d = {'name': \"Jeff\", 'address':\"123 Main\"}\n",
    "# All of the keys\n",
    "print(f\"Key: {d.keys()}\")\n",
    "\n",
    "# All of the values\n",
    "print(f\"Values: {d.values()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'name': 'Jeff & Tracy Heaton', 'pets': ['Wynton', 'Cricket', 'Hickory']}, {'name': 'John Smith', 'pets': ['rover']}, {'name': 'Jane Doe'}]\n",
      "Jeff & Tracy Heaton:['Wynton', 'Cricket', 'Hickory']\n",
      "John Smith:['rover']\n",
      "Jane Doe:no pets\n"
     ]
    }
   ],
   "source": [
    "# Python list & map structures\n",
    "customers = [\n",
    "    {'name': 'Jeff & Tracy Heaton', 'pets': ['Wynton', 'Cricket', 'Hickory']},\n",
    "    {'name': 'John Smith', 'pets': ['rover']},\n",
    "    {'name': 'Jane Doe'}\n",
    "]\n",
    "\n",
    "print(customers)\n",
    "\n",
    "for customer in customers:\n",
    "    print(f\"{customer['name']}:{customer.get('pets', 'no pets')}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Module 1 Assignment\n",
    "\n",
    "You can find the first assignment here: [assignment 1](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/assignments/assignment_yourname_class1.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.6 (wustl)",
   "language": "python",
   "name": "wustl"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}