transfer-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Transfer Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we use pre-trained neural networks as a starting point for our own models. The rationale behind this tutorial is as follows: if we construct a neural network that can recognise images of cars, would it be possible for the network (with some extra training) to recognise trucks?\n",
    "\n",
    "Other people have trained enormous general-purpose CNNs for image classification. We can exploit the features that these CNNs have found for our own image classification problem. (i.e. transfer the knowledge/features which have been acquired by training networks on large datasets, and applying the model along with it's weights to a new dataset).\n",
    "\n",
    "Training networks (even more so for very deep networks) on large datasets (hundreds of thousands of examples) can take very long (days/weeks). So it doesn't always make sense to re-train from sratch.\n",
    "\n",
    "Typically, transfer learning works by loading a pre-trained network and removing the final layer (which predicts class membership), since this will be particular to the problem used to train the original network. For example, the original network might have 10 classes, and our new problem might have 15 - so this is why we would need to remove the last layer. We then add on our own final layer, which contains our own output nodes (based on the number of classes, or in certain cases just a single output). We then retrain the network (either just the final layer, or the whole network).\n",
    "\n",
    "The following image summarises the principle behind transfer learning in neural networks.\n",
    "\n",
    "![](transfer_learning_setup.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "library(keras)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say where the data is..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_directory <- \"data/invasives/sample/train/\"\n",
    "validation_directory <- \"data/invasives/sample/validation/\"\n",
    "test_directory <- \"data/invasives/sample/test/\"\n",
    "\n",
    "# once you are satisfied the code is working, run full dataset\n",
    "# train_directory <- \"data/invasives/train/\"\n",
    "# validation_directory <- \"data/invasives/validation/\"\n",
    "# test_directory <- \"data/invasives/test/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And work out how many images we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_samples <- length(list.files(paste(train_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(train_directory,\"non_invasive\",sep=\"\")))\n",
    "\n",
    "validation_samples <- length(list.files(paste(validation_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(validation_directory,\"non_invasive\",sep=\"\")))\n",
    "\n",
    "test_samples <- length(list.files(paste(test_directory,\"invasive\",sep=\"\"))) +\n",
    "    length(list.files(paste(test_directory,\"non_invasive\",sep=\"\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_samples\n",
    "validation_samples\n",
    "test_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case we will use the VGG16 pre-trained network. \n",
    "\n",
    "VGG16's architecture is made up of 5 convolutional blocks, each block is made up of several convolutional layers. Max pooling is applied between these blocks. The architecutre has 16 layers and looks as follows:\n",
    "\n",
    "![](vgg16.jpg)\n",
    "\n",
    "This network needs input images to have a dimension of 224x224x3, so we set desired image height and width accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "img_height <- 224\n",
    "img_width <- 224\n",
    "batch_size <- 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data generators\n",
    "\n",
    "Since the data is neatly organised in folders, we can make use of flow_images_from_directory to easily read in the data. We do this for our training, validation and testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_generator <- flow_images_from_directory(\n",
    "  train_directory, \n",
    "  generator = image_data_generator(),\n",
    "  target_size = c(img_height, img_width),\n",
    "  color_mode = \"rgb\",\n",
    "  class_mode = \"binary\", \n",
    "  batch_size = batch_size, \n",
    "  shuffle = TRUE,\n",
    "  seed = 123)\n",
    "\n",
    "validation_generator <- flow_images_from_directory(\n",
    "  validation_directory, \n",
    "  generator = image_data_generator(), \n",
    "  target_size = c(img_height, img_width), \n",
    "  color_mode = \"rgb\", \n",
    "  classes = NULL,\n",
    "  class_mode = \"binary\", \n",
    "  batch_size = batch_size, \n",
    "  shuffle = TRUE,\n",
    "  seed = 123)\n",
    "\n",
    "test_generator <- flow_images_from_directory(\n",
    "  test_directory, \n",
    "  generator = image_data_generator(),\n",
    "  target_size = c(img_height, img_width), \n",
    "  color_mode = \"rgb\", \n",
    "  class_mode = \"binary\", \n",
    "  batch_size = 1,\n",
    "  shuffle = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading pre-trained model and adding custom layers\n",
    "\n",
    "Here, include_top=FALSE means that we are not including the last 3 fully connected layers that are present in the orginal VGG16 architecture. weights='imagenet' means that the model will use the weights which were obtained when originally training on the ImageNet dataset (millions of) Additional references are available here: https://tensorflow.rstudio.com/keras/reference/application_vgg.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "base_model <- application_vgg16(weights = \"imagenet\", \n",
    "                                       include_top = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choices of weights are \"imagenet\" or \"None\". None means that the weights will be randomly initialised.\n",
    "\n",
    "Imagenet has roughly 14 million images categorised into roughly 17 thousand classes. So it makes sense to use models that have good performance on this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add our custom layers\n",
    "\n",
    "Here we add a global average 2d pooling layer followed by two fully connected layers. The the last layer there is a single output node. Why is there a single output node? Why are we using the sigmoid function instead of, say, ReLU?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions <- base_model$output %>% \n",
    "  layer_global_average_pooling_2d(trainable=T) %>% \n",
    "  layer_dense(units = 512, activation = \"relu\", trainable=T) %>% \n",
    "  layer_dense(units = 1, activation = \"sigmoid\", trainable=T)\n",
    "\n",
    "model <- keras_model(inputs = base_model$input, \n",
    "                     outputs = predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Print out a summary of the model.\n",
    "### Take note of the number of trainable parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here we \"Freeze\" some layers, i.e. we tell the model not to learn those weights in those layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (layer in base_model$layers)\n",
    "  layer$trainable <- FALSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Now print out the summary and have a look at the number of parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the model\n",
    "\n",
    "This is a typical implementation of stochastic gradient descent with a learning rate of 0.0001.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% compile(\n",
    "  loss = \"binary_crossentropy\",\n",
    "  optimizer = optimizer_sgd(lr = 0.0001, \n",
    "                            momentum = 0.9, \n",
    "                            decay = 1e-5),\n",
    "  metrics = \"accuracy\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the model\n",
    "\n",
    "Train the model on the training data, validate on the validation data. Run for 5 pochs. This is a typical implementation. Here we use fit_generator() because we read in our data using a generator above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% fit_generator(\n",
    "  train_generator,\n",
    "  steps_per_epoch = as.integer(train_samples / batch_size), \n",
    "  epochs = 3, \n",
    "  validation_data = validation_generator,\n",
    "  validation_steps = as.integer(validation_samples / batch_size),\n",
    "  verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% evaluate_generator(\n",
    "    test_generator,\n",
    "    steps = test_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Various other pre-trained models for Keras are avaiable here: https://keras.rstudio.com/articles/applications.html The website also shows a simple example on how to load the model. If you do not have access to a GPU then this approach is a good place to start.\n",
    "\n",
    "# Example 2\n",
    "\n",
    "Now let's implement transfer learning for CIFAR-10. More details here: https://www.cs.toronto.edu/~kriz/cifar.html What is the dataset about? What can you say about the dimensions of the data? How many output nodes do you think we need in the last layer?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Load the CIFAR-10 dataset\n",
    "cifar10 <- dataset_cifar10()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CIFAR10 dataset has 50,000 training images and 10,000 test images. Here to speed things up we just take the first 1000 images from each of the training and test datasets. You might want to increase this number depending on the memory and time available to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Feature scale RGB values in test and train inputs  \n",
    "x_train <- cifar10$train$x[1:1000,,,]/255\n",
    "x_test <- cifar10$test$x[1:1000,,,]/255\n",
    "y_train <- to_categorical(cifar10$train$y[1:1000], num_classes = 10)\n",
    "y_test <- to_categorical(cifar10$test$y[1:1000], num_classes = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now load the pretrained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "base_model <- application_vgg16(weights = \"imagenet\", \n",
    "                                       include_top = FALSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here once again we tell the model not to re-train every weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for (layer in base_model$layers)\n",
    "  layer$trainable <- FALSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay here we need to implement our new last layers slightly differently to the invasive species dataset. CIFAR-10. Firstly, we know that this is a classification problem, and that there are more than just 2 classes. So, from this, we know that we should have 10 units in the last layer. We need to use a softmax activation function. Softmax outputs the probability for each class, so this is the perfect activation function to use. Here we are adding only two fully connected layers, feel free to experiment with other layers, units or even add dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictions <- base_model$output %>% \n",
    "  layer_global_average_pooling_2d() %>% \n",
    "  layer_dense(units = 1024, activation = \"relu\") %>% \n",
    "  layer_dense(units = 10, activation = \"softmax\")\n",
    "\n",
    "model <- keras_model(inputs = base_model$input, \n",
    "                     outputs = predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the network architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "summary(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model has 15 million parameters. Yikes! We don't have to spend hours and re-train a lot of those parameters - thanks transfer learning!\n",
    "\n",
    "Now that we have defined a model, we need to define the loss function and tell the model which optimiser it will use. In the previous example we used stochastic gradient descent. Let's use a different one this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "opt<-optimizer_adam(lr= 0.001)\n",
    "\n",
    "model %>% compile(\n",
    "  loss = \"categorical_crossentropy\",\n",
    "  optimizer = opt,\n",
    "  metrics = \"accuracy\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, we aren't used a data generator since all of our data are already in the x and y variables. So we can't call the fit function like above. Of course, we should have a separate validation and test set, but for simplicity we will just use the test set here as validation data. Instead we call it this way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% fit( x_train,y_train ,batch_size=32,\n",
    "               epochs=1,validation_data = list(x_test, y_test),\n",
    "               shuffle=TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And to test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model %>% evaluate(x_test, y_test, batch_size=32, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "More models are available here: https://keras.io/applications/#vgg16"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}