Deep_Learning_Recurrent_Neural_Networks_in_Python.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPn8NvqKCx4aKJl0AhpeOEd",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Shaghayegh-Aflatounian/Churn-Prediction/blob/main/Deep_Learning_Recurrent_Neural_Networks_in_Python.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wSfP6qFB4iy9"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.layers import\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Inpute,Dense"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **IMBD**"
      ],
      "metadata": {
        "id": "3-ja377hxV4y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "you were exposed to the various applications of sequence to sequence models. In this exercise you will see how to use a pre-trained model for sentiment analysis.The model is pre-loaded in the environment on variable model. Also, the tokenized test set variables X_test and y_test and the pre-processed original text data sentences from IMDb are also available.You will learn how to pre-process the text data and how to create and train the model using Keras later in the course."
      ],
      "metadata": {
        "id": "4vwcFot5xNYI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Inspect the first sentence on `X_test`\n",
        "print(X_test[0])\n",
        "\n",
        "# Get the predicion for all the sentences\n",
        "pred = model.predict(X_test)\n",
        "\n",
        "# Transform the predition into positive (> 0.5) or negative (<= 0.5)\n",
        "pred_sentiment = [\"positive\" if x>0.5 else \"negative\" for x in pred]\n",
        "\n",
        "# Create a data frame with sentences, predictions and true values\n",
        "result = pd.DataFrame({'sentence': sentences, 'y_pred': pred_sentiment, 'y_true': y_test})\n",
        "\n",
        "# Print the first lines of the data frame\n",
        "print(result.head())"
      ],
      "metadata": {
        "id": "lMyacykPxNu_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this exercise, you will play with text data by analyzing quotes from Sheldon Cooper in The Big Bang Theory TV show. This will give you a chance to analyze sentences to obtain insights on what it's like to deal with real-world text data."
      ],
      "metadata": {
        "id": "_1KNLE0kxy_i"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **--------------------------------------------------**"
      ],
      "metadata": {
        "id": "mCd3fWKzxVE7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "# Creates a model given an activation and learning rate\n",
        "def create_model(learning_rate, activation):\n",
        "\n",
        "  \t# Create an Adam optimizer with the given learning rate\n",
        "  \topt = Adam(lr = learning_rate)\n",
        "\n",
        "  \t# Create your binary classification model\n",
        "  \tmodel = Sequential()\n",
        "  \tmodel.add(Dense(128, input_shape = (30,), activation = activation))\n",
        "  \tmodel.add(Dense(256, activation = activation))\n",
        "  \tmodel.add(Dense(1, activation = 'sigmoid'))\n",
        "\n",
        "  \t# Compile your model with your optimizer, loss, and metrics\n",
        "  \tmodel.compile(optimizer = Adam, loss = validation, metrics = ['accuracy'])\n",
        "  \treturn model"
      ],
      "metadata": {
        "id": "u5J6uBKpajC3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import KerasClassifier from tensorflow.keras scikit learn wrappers\n",
        "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "\n",
        "# Create a KerasClassifier\n",
        "model = KerasClassifier(build_fn = create_model)\n",
        "\n",
        "# Define the parameters to try out\n",
        "params = {'activation': ['relu', 'tanh'], 'batch_size': [ 32, 128,  256],\n",
        "          'epochs': [50, 100, 200], 'learning_rate': [0.1, 0.01,0.001]}\n",
        "\n",
        "# Create a randomize search cv object passing in the parameters to try\n",
        "random_search = RandomizedSearchCV(model, param_distributions = params, cv = KFold(3))\n",
        "\n",
        "# Running random_search.fit(X,y) would start the search,but it takes too long!\n",
        "show_results()"
      ],
      "metadata": {
        "id": "Y8QiL8aeHdng"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import KerasClassifier from tensorflow.keras wrappers\n",
        "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier\n",
        "\n",
        "# Create a KerasClassifier\n",
        "model = KerasClassifier(build_fn = create_model(learning_rate = 0.001, activation = 'relu'), epochs = 50,\n",
        "             batch_size = 128, verbose = 0)\n",
        "\n",
        "# Calculate the accuracy score for each fold\n",
        "kfolds = cross_val_score(X, y, model, cv = 3)\n",
        "\n",
        "# Print the mean accuracy\n",
        "print('The mean accuracy was:', kfolds.mean())\n",
        "\n",
        "# Print the accuracy standard deviation\n",
        "print('With a standard deviation of:', kfolds.std())"
      ],
      "metadata": {
        "id": "tWK7maklOJ1G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You will learn later some tuning approaches to the sentiment classification model. Also, the process of pre-processing the text data and creating, training and testing models in Keras will be detailed later in the course. Finally, you created a decision rule to determine if the sentiment would be classified as positive or negative. In many applications, the value of 0.5 is used as decision boundary, but other values can also be used depending on what metric you want to optimize."
      ],
      "metadata": {
        "id": "g0Mv65n3PRob"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The model is pre-loaded in the environment on variable model. Also, the tokenized test set variables X_test and y_test and the pre-processed original text data sentences from IMDb are also available.You will learn how to pre-process the text data and how to create and train the model using Keras later in the course."
      ],
      "metadata": {
        "id": "O9twQxVuPWJH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Inspect the first sentence on `X_test`\n",
        "print(X_test[0])\n",
        "\n",
        "# Get the predicion for all the sentences\n",
        "pred = model.predict(X_test)\n",
        "\n",
        "# Transform the predition into positive (> 0.5) or negative (<= 0.5)\n",
        "pred_sentiment = [\"positive\" if x>0.5 else \"negative\" for x in pred]\n",
        "\n",
        "# Create a data frame with sentences, predictions and true values\n",
        "result = pd.DataFrame({'sentence': sentences, 'y_pred': pred_sentiment, 'y_true': y_test})\n",
        "\n",
        "# Print the first lines of the data frame\n",
        "print(result.head())"
      ],
      "metadata": {
        "id": "6SCwmd2OPJBz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "QNusDlDUPWus"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "we need an array to transform words into number vectors> create a list of all word"
      ],
      "metadata": {
        "id": "D711RZuV8i0O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**create a dictionary -Getting used to text data**\n"
      ],
      "metadata": {
        "id": "msJJSn_D8tRL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this exercise, you will play with text data by analyzing quotes from Sheldon Cooper in The Big Bang Theory TV show. This will give you a chance to analyze sentences to obtain insights on what it's like to deal with real-world text data"
      ],
      "metadata": {
        "id": "sxfptZcdCk7j"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You will use dictionary comprehensions to create dictionaries that map words to indexes and vice versa. The use of dictionaries instead of, for example, a pandas.DataFrame is because they are more intuitive and don't add unnecessary extra complexity."
      ],
      "metadata": {
        "id": "xN0ocPelCxt_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transform the list of sentences into a list of words\n",
        "all_words = ' '.join(sheldon_quotes).split(' ')\n",
        "\n",
        "# Get number of unique words\n",
        "unique_words = list(set(all_words))\n",
        "\n",
        "# Dictionary of indexes as keys and words as values\n",
        "index_to_word = {i: wd for i, wd in enumerate(sorted(unique_words))}\n",
        "print(index_to_word)\n",
        "\n",
        "# Dictionary of words as keys and indexes as values\n",
        "word_to_index = {wd: i for i, wd in enumerate(sorted(unique_words))}\n",
        "print(word_to_index)"
      ],
      "metadata": {
        "id": "zjijWdne8ska"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**sentence to characters for prediction**\n",
        "Previously, you learned how to create dictionaries of indexes to words and vice versa. In this exercise, you will split the text by characters and continue to prepare the data for supervised learning.\n",
        "\n",
        "Splitting the texts into characters may seem strange, but it is often done for text generation. Also, the process to prepare the data is the same, the only change is how to split the texts.\n",
        "\n",
        "You will create the training data containing a list of fixed-length texts and their labels, which are the corresponding next characters.\n",
        "\n",
        "You will continue to use the dataset containing quotes from Sheldon (The Big Bang Theory), available in the sheldon_quotes variable.\n",
        "\n",
        "The print_examples() function print the pairs so you can see how the data was transformed. Use help() for details."
      ],
      "metadata": {
        "id": "T7vkJCEYOjiH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create lists to keep the sentences and the next character\n",
        "sentences = []   # ~ Training data\n",
        "next_chars = []  # ~ Training labels\n",
        "\n",
        "# Define hyperparameters\n",
        "step = 2          # ~ Step to take when reading the texts in characters\n",
        "chars_window = 10 # ~ Number of characters to use to predict the next one\n",
        "\n",
        "# Loop over the text: length `chars_window` per time with step equal to `step`\n",
        "for i in range(0, len(sheldon_quotes) - chars_window, step):\n",
        "    sentences.append(sheldon_quotes[i:i + chars_window])\n",
        "    next_chars.append(sheldon_quotes[i + chars_window])\n",
        "\n",
        "# Print 10 pairs\n",
        "print_examples(sentences, next_chars, 10)"
      ],
      "metadata": {
        "id": "tmFPbMMfOoZv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this exercise, you will transform a new text into sequences of numerical indexes on the dictionaries created before.\n",
        "\n",
        "This is useful when you already have a trained model and want to apply it on a new dataset. The preprocessing steps done on the training data should also be applied to the new text, so the model can make predictions/classifications."
      ],
      "metadata": {
        "id": "dEcO5l478r7t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Loop through the sentences and get indexes\n",
        "new_text_split = []\n",
        "for sentence in new_text:\n",
        "    sent_split = []\n",
        "    for wd in sentence.split(' '):\n",
        "        index = word_to_index.get(wd, 0)\n",
        "        sent_split.append(index)\n",
        "    new_text_split.append(sent_split)\n",
        "\n",
        "# Print the first sentence's indexes\n",
        "print(new_text_split[0])\n",
        "\n",
        "# Print the sentence converted using the dictionary\n",
        "print(' '.join([index_to_word[index] for index in new_text_split[0]]))"
      ],
      "metadata": {
        "id": "oZb6-Q9k8s53"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can see that some of the words were not found on the dictionary and have index = 0. By using the token '<UKN/>' in the training phase, you can easily use the model on unseen data without getting errors. This is also done when limiting the size of the vocabulary, say to 5,000 most frequent words, and setting the others as '<UKN/>'."
      ],
      "metadata": {
        "id": "zcfZC4naGd2D"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "$pip install tensorflow\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import Dense\n"
      ],
      "metadata": {
        "id": "EQop19FLF5BU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Different type of adding layers Sequential vs Model**"
      ],
      "metadata": {
        "id": "cdocY_dC0VEL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Instantiate the class\n",
        "model = Sequential(name=\"sequential_model\")\n",
        "\n",
        "# One LSTM layer (defining the input shape because it is the\n",
        "# initial layer)\n",
        "model.add(LSTM(128, input_shape=(None, 10), name=\"LSTM\"))\n",
        "\n",
        "# Add a dense layer with one unit\n",
        "model.add(Dense(1, activation=\"sigmoid\", name=\"output\"))\n",
        "\n",
        "# The summary shows the layers and the number of parameters\n",
        "# that will be trained\n",
        "model.summary()"
      ],
      "metadata": {
        "id": "ZD1kD2aajh9b"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**(None, 10**), which means the input can have any number of samples (the first dimension is None) and each sample has 10 features"
      ],
      "metadata": {
        "id": "8KtPoUd_ZjDP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "## keras.models.Model\n",
        "\n",
        "# Define the input layer\n",
        "main_input = Input(shape=(None, 10), name=\"input\")\n",
        "\n",
        "# One LSTM layer (input shape is already defined)\n",
        "lstm_layer = LSTM(128, name=\"LSTM\")(main_input)\n",
        "\n",
        "# Add a dense layer with one unit\n",
        "main_output = Dense(1, activation=\"sigmoid\", name=\"output\")(lstm_layer)\n",
        "\n",
        "# Instantiate the class at the end\n",
        "model = Model(inputs=main_input, outputs=main_output, name=\"modelclass_model\")\n",
        "\n",
        "# Same amount of parameters to train as before (71,297)\n",
        "model.summary()"
      ],
      "metadata": {
        "id": "iGsll5Bqp6HT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " You can see that the keras.models.Sequential is very easy to use to add layers in sequence. On the other hand, the keras.models.Model class is very flexible and is usually the choice when scientists need deep customization in their solution. Also, you saw how one layer is connected to another layer in both cases, be by adding them in sequence using the method add, or by creating a layer and calling the desired (previous) layer like a function, in the Model class API, every layer is callable on a tensor and always return a tensor."
      ],
      "metadata": {
        "id": "Dn4H8iGLz7FR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Keras preprocessing**\n"
      ],
      "metadata": {
        "id": "SxoZ9whFdBrU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**keras.preprocessing.** You will see how to use the most important modules and functions to prepare raw data to the correct input shape. Keras provides functionalities that substitute the dictionary approach you learned before."
      ],
      "metadata": {
        "id": "tQEAoKKcdND7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You will use the module **keras.preprocessing.text.Tokenizer to create a dictionary of words using the method .fit_on_texts() and change the texts into numerical ids representing the index of each word on the dictionary using the method .texts_to_sequences().**"
      ],
      "metadata": {
        "id": "ASlloK2LdZLu"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Then, use the **function .pad_sequences() from keras.preprocessing.sequence to make all the sequences have the same size (necessary for the model) by adding zeros on the small texts and cutting the big ones.**"
      ],
      "metadata": {
        "id": "oQrGDNmxdCBy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Import relevant classes/functions\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "# Build the dictionary of indexes\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(texts)\n",
        "\n",
        "# Change texts into sequence of indexes\n",
        "texts_numeric = tokenizer.texts_to_sequences(texts)\n",
        "print(\"Number of words in the sample texts: ({0}, {1})\".format(len(texts_numeric[0]), len(texts_numeric[1])))\n",
        "\n",
        "# Pad the sequences\n",
        "texts_pad = pad_sequences(texts_numeric, 60)\n",
        "print(\"Now the texts have fixed length: 60. Let's see the first one: \\n{0}\".format(texts_pad[0]))"
      ],
      "metadata": {
        "id": "JWsmHbzYZk7B"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}