CRNN Model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "code_folding": []
   },
   "source": [
    "# Convolutional Recurrent Neural Network for Text Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand intuition behind this model follow these blogs: \n",
    "1. [Creating a CRNN model to recognize text in an image (Part-1)](https://theailearner.com/2019/05/29/creating-a-crnn-model-to-recognize-text-in-an-image-part-1/)\n",
    "2. [Creating a CRNN model to recognize text in an image (Part-2)](https://theailearner.com/2019/05/29/creating-a-crnn-model-to-recognize-text-in-an-image-part-2/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An optical character recognition problem is basically a type of image-based sequence recognition problem. And for sequence recognition problem, most suited neural networks are recurrent neural networks(RNN) while for an image-based problem most suited are convolution neural networks(CNN). To cop up with the OCR problems we need to combine both of these CNN and RNN.\n",
    "\n",
    "We can break the implementation of CRNN network into following steps:\n",
    "\n",
    "1. Collecting Dataset\n",
    "2. Preprocessing Data\n",
    "3. Creating Network Architecture\n",
    "4. Defining Loss Function\n",
    "5. Training Model\n",
    "6. Decoding Outputs from Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import fnmatch\n",
    "import cv2\n",
    "import numpy as np\n",
    "import string\n",
    "import time\n",
    "\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "from keras.layers import Dense, LSTM, Reshape, BatchNormalization, Input, Conv2D, MaxPool2D, Lambda, Bidirectional\n",
    "from keras.models import Model\n",
    "from keras.activations import relu, sigmoid, softmax\n",
    "import keras.backend as K\n",
    "from keras.utils import to_categorical\n",
    "from keras.callbacks import ModelCheckpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "#ignore warnings in the output\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[name: \"/device:CPU:0\"\n",
      "device_type: \"CPU\"\n",
      "memory_limit: 268435456\n",
      "locality {\n",
      "}\n",
      "incarnation: 472655688989744329\n",
      ", name: \"/device:GPU:0\"\n",
      "device_type: \"GPU\"\n",
      "memory_limit: 3209412608\n",
      "locality {\n",
      "  bus_id: 1\n",
      "  links {\n",
      "  }\n",
      "}\n",
      "incarnation: 5197767360891115092\n",
      "physical_device_desc: \"device: 0, name: Quadro M1200, pci bus id: 0000:01:00.0, compute capability: 5.0\"\n",
      "]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "\n",
    "# Check all available devices if GPU is available\n",
    "print(device_lib.list_local_devices())\n",
    "sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we will use data provided by Visual Geometry Group. This is a huge dataset total of 10 GB images. Here I have used only 135000 images for the training set and 15000 images for validation dataset.\n",
    "\n",
    "To download the dataset either you can directly download from [this link](https://www.robots.ox.ac.uk/~vgg/data/text/#sec-synth) or use the following commands to download the data and unzip."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "wget https://www.robots.ox.ac.uk/~vgg/data/text/mjsynth.tar.gz\n",
    " \n",
    "tar -xvzf mjsynth.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are having our dataset, to make it acceptable for our model we need to use some preprocessing. We need to preprocess both the input image and output labels. To preprocess our input image we will use followings:\n",
    "\n",
    "1. Read the image and convert into a gray-scale image\n",
    "2. Make each image of size (128,32) by using padding\n",
    "3. Expand image dimension as (128,32,1) to make it compatible with the input shape of architecture\n",
    "4. Normalize the image pixel values by dividing it with 255.\n",
    "\n",
    "To preprocess the output labels use the followings:\n",
    "\n",
    "1. Read the text from the name of the image as the image name contains text written inside the image.\n",
    "2. Encode each character of a word into some numerical value by creating a function( as ‘a’:0, ‘b’:1 …….. ‘z’:26 etc ). Let    say we are having the word ‘abab’ then our encoded label would be [0,1,0,1]\n",
    "3. Compute the maximum length from words and pad every output label to make it of the same size as the maximum length. This    is done to make it compatible with the output shape of our RNN architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# char_list:   'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'\n",
    "# total number of our output classes: len(char_list)\n",
    "char_list = string.ascii_letters+string.digits\n",
    " \n",
    "def encode_to_labels(txt):\n",
    "    # encoding each output word into digits\n",
    "    dig_lst = []\n",
    "    for index, char in enumerate(txt):\n",
    "        try:\n",
    "            dig_lst.append(char_list.index(char))\n",
    "        except:\n",
    "            print(char)\n",
    "        \n",
    "    return dig_lst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'mjsynth.tar/mnt/ramdisk/max/90kDICT32px'\n",
    " \n",
    " \n",
    "# lists for training dataset\n",
    "training_img = []\n",
    "training_txt = []\n",
    "train_input_length = []\n",
    "train_label_length = []\n",
    "orig_txt = []\n",
    " \n",
    "#lists for validation dataset\n",
    "valid_img = []\n",
    "valid_txt = []\n",
    "valid_input_length = []\n",
    "valid_label_length = []\n",
    "valid_orig_txt = []\n",
    " \n",
    "max_label_len = 0\n",
    " \n",
    "i =1 \n",
    "flag = 0\n",
    " \n",
    "for root, dirnames, filenames in os.walk(path):\n",
    " \n",
    "    for f_name in fnmatch.filter(filenames, '*.jpg'):\n",
    "        # read input image and convert into gray scale image\n",
    "        img = cv2.cvtColor(cv2.imread(os.path.join(root, f_name)), cv2.COLOR_BGR2GRAY)   \n",
    " \n",
    "        # convert each image of shape (32, 128, 1)\n",
    "        w, h = img.shape\n",
    "        if h > 128 or w > 32:\n",
    "            continue\n",
    "        if w < 32:\n",
    "            add_zeros = np.ones((32-w, h))*255\n",
    "            img = np.concatenate((img, add_zeros))\n",
    " \n",
    "        if h < 128:\n",
    "            add_zeros = np.ones((32, 128-h))*255\n",
    "            img = np.concatenate((img, add_zeros), axis=1)\n",
    "        img = np.expand_dims(img , axis = 2)\n",
    "        \n",
    "        # Normalize each image\n",
    "        img = img/255.\n",
    "        \n",
    "        # get the text from the image\n",
    "        txt = f_name.split('_')[1]\n",
    "        \n",
    "        # compute maximum length of the text\n",
    "        if len(txt) > max_label_len:\n",
    "            max_label_len = len(txt)\n",
    "            \n",
    "           \n",
    "        # split the 150000 data into validation and training dataset as 10% and 90% respectively\n",
    "        if i%10 == 0:     \n",
    "            valid_orig_txt.append(txt)   \n",
    "            valid_label_length.append(len(txt))\n",
    "            valid_input_length.append(31)\n",
    "            valid_img.append(img)\n",
    "            valid_txt.append(encode_to_labels(txt))\n",
    "        else:\n",
    "            orig_txt.append(txt)   \n",
    "            train_label_length.append(len(txt))\n",
    "            train_input_length.append(31)\n",
    "            training_img.append(img)\n",
    "            training_txt.append(encode_to_labels(txt)) \n",
    "        \n",
    "        # break the loop if total data is 150000\n",
    "        if i == 150000:\n",
    "            flag = 1\n",
    "            break\n",
    "        i+=1\n",
    "    if flag == 1:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pad each output label to maximum text length\n",
    " \n",
    "train_padded_txt = pad_sequences(training_txt, maxlen=max_label_len, padding='post', value = len(char_list))\n",
    "valid_padded_txt = pad_sequences(valid_txt, maxlen=max_label_len, padding='post', value = len(char_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Network Archtecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network architecture is inspired by this [paper](https://arxiv.org/pdf/1507.05717.pdf). Let’s see the steps that we used to create the architecture:\n",
    "\n",
    "1. Input shape for our architecture having an input image of height 32 and width 128.\n",
    "2. Here we used seven convolution layers of which 6 are having kernel size (3,3) and the last one is of size (2.2). And the    number of filters is increased from 64 to 512 layer by layer.\n",
    "3. Two max-pooling layers are added with size (2,2) and then two max-pooling layers of size (2,1) are added to extract        features with a larger width to predict long texts.\n",
    "4. Also, we used batch normalization layers after fifth and sixth convolution layers which accelerates the training       process.\n",
    "5. Then we used a lambda function to squeeze the output from conv layer and make it compatible with LSTM layer.\n",
    "6. Then used two Bidirectional LSTM layers each of which has 128 units. This RNN layer gives the output of size (batch_size, 31, 63). Where 63 is the total number of output classes including blank character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input with shape of height=32 and width=128 \n",
    "inputs = Input(shape=(32,128,1))\n",
    " \n",
    "# convolution layer with kernel size (3,3)\n",
    "conv_1 = Conv2D(64, (3,3), activation = 'relu', padding='same')(inputs)\n",
    "# poolig layer with kernel size (2,2)\n",
    "pool_1 = MaxPool2D(pool_size=(2, 2), strides=2)(conv_1)\n",
    " \n",
    "conv_2 = Conv2D(128, (3,3), activation = 'relu', padding='same')(pool_1)\n",
    "pool_2 = MaxPool2D(pool_size=(2, 2), strides=2)(conv_2)\n",
    " \n",
    "conv_3 = Conv2D(256, (3,3), activation = 'relu', padding='same')(pool_2)\n",
    " \n",
    "conv_4 = Conv2D(256, (3,3), activation = 'relu', padding='same')(conv_3)\n",
    "# poolig layer with kernel size (2,1)\n",
    "pool_4 = MaxPool2D(pool_size=(2, 1))(conv_4)\n",
    " \n",
    "conv_5 = Conv2D(512, (3,3), activation = 'relu', padding='same')(pool_4)\n",
    "# Batch normalization layer\n",
    "batch_norm_5 = BatchNormalization()(conv_5)\n",
    " \n",
    "conv_6 = Conv2D(512, (3,3), activation = 'relu', padding='same')(batch_norm_5)\n",
    "batch_norm_6 = BatchNormalization()(conv_6)\n",
    "pool_6 = MaxPool2D(pool_size=(2, 1))(batch_norm_6)\n",
    " \n",
    "conv_7 = Conv2D(512, (2,2), activation = 'relu')(pool_6)\n",
    " \n",
    "squeezed = Lambda(lambda x: K.squeeze(x, 1))(conv_7)\n",
    " \n",
    "# bidirectional LSTM layers with units=128\n",
    "blstm_1 = Bidirectional(LSTM(128, return_sequences=True, dropout = 0.2))(squeezed)\n",
    "blstm_2 = Bidirectional(LSTM(128, return_sequences=True, dropout = 0.2))(blstm_1)\n",
    " \n",
    "outputs = Dense(len(char_list)+1, activation = 'softmax')(blstm_2)\n",
    "\n",
    "# model to be used at test time\n",
    "act_model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 32, 128, 1)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 32, 128, 64)       640       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 16, 64, 64)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 16, 64, 128)       73856     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 8, 32, 128)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 8, 32, 256)        295168    \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 8, 32, 256)        590080    \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 4, 32, 256)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_5 (Conv2D)            (None, 4, 32, 512)        1180160   \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 4, 32, 512)        2048      \n",
      "_________________________________________________________________\n",
      "conv2d_6 (Conv2D)            (None, 4, 32, 512)        2359808   \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 4, 32, 512)        2048      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_4 (MaxPooling2 (None, 2, 32, 512)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_7 (Conv2D)            (None, 1, 31, 512)        1049088   \n",
      "_________________________________________________________________\n",
      "lambda_1 (Lambda)            (None, 31, 512)           0         \n",
      "_________________________________________________________________\n",
      "bidirectional_1 (Bidirection (None, 31, 256)           656384    \n",
      "_________________________________________________________________\n",
      "bidirectional_2 (Bidirection (None, 31, 256)           394240    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 31, 63)            16191     \n",
      "=================================================================\n",
      "Total params: 6,619,711\n",
      "Trainable params: 6,617,663\n",
      "Non-trainable params: 2,048\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "act_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we are using the CTC loss function. CTC loss is very helpful in text recognition problems. It helps us to prevent annotating each time step and help us to get rid of the problem where a single character can span multiple time step which needs further processing if we do not use CTC. If you want to know more about CTC( Connectionist Temporal Classification ) please follow [this blog](https://theailearner.com/2019/05/29/connectionist-temporal-classificationctc/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A CTC loss function requires four arguments to compute the loss, predicted outputs, ground truth labels, input sequence length to LSTM and ground truth label length. To get this we need to create a custom loss function and then pass it to the model. To make it compatible with our model, we will create a model which takes these four inputs and outputs the loss. This model will be used for training and for testing we will use the model that we have created earlier “act_model”. Let’s see the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = Input(name='the_labels', shape=[max_label_len], dtype='float32')\n",
    "input_length = Input(name='input_length', shape=[1], dtype='int64')\n",
    "label_length = Input(name='label_length', shape=[1], dtype='int64')\n",
    " \n",
    " \n",
    "def ctc_lambda_func(args):\n",
    "    y_pred, labels, input_length, label_length = args\n",
    " \n",
    "    return K.ctc_batch_cost(labels, y_pred, input_length, label_length)\n",
    " \n",
    " \n",
    "loss_out = Lambda(ctc_lambda_func, output_shape=(1,), name='ctc')([outputs, labels, input_length, label_length])\n",
    "\n",
    "#model to be used at training time\n",
    "model = Model(inputs=[inputs, labels, input_length, label_length], outputs=loss_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### train the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the model we will use Adam optimizer. Also, we can use Keras callbacks functionality to save the weights of the best model on the basis of validation loss.\n",
    "In model.compile(), you can see that I have only taken y_pred and neglected y_true. This is because I have already taken labels as input to the model earlier.\n",
    "labels as input to the model earlier.\n",
    "\n",
    "Now train your model on 135000 training images and 15000 validation images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss={'ctc': lambda y_true, y_pred: y_pred}, optimizer = 'adam')\n",
    " \n",
    "filepath=\"best_model.hdf5\"\n",
    "checkpoint = ModelCheckpoint(filepath=filepath, monitor='val_loss', verbose=1, save_best_only=True, mode='auto')\n",
    "callbacks_list = [checkpoint]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_img = np.array(training_img)\n",
    "train_input_length = np.array(train_input_length)\n",
    "train_label_length = np.array(train_label_length)\n",
    "\n",
    "valid_img = np.array(valid_img)\n",
    "valid_input_length = np.array(valid_input_length)\n",
    "valid_label_length = np.array(valid_label_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 135000 samples, validate on 15000 samples\n",
      "Epoch 1/10\n",
      "135000/135000 [==============================] - 586s 4ms/step - loss: 27.5512 - val_loss: 26.9004\n",
      "\n",
      "Epoch 00001: val_loss improved from inf to 26.90039, saving model to best_model.hdf5\n",
      "Epoch 2/10\n",
      "135000/135000 [==============================] - 612s 5ms/step - loss: 26.1934 - val_loss: 26.9729\n",
      "\n",
      "Epoch 00002: val_loss did not improve from 26.90039\n",
      "Epoch 3/10\n",
      "135000/135000 [==============================] - 619s 5ms/step - loss: 21.8621 - val_loss: 21.6336\n",
      "\n",
      "Epoch 00003: val_loss improved from 26.90039 to 21.63363, saving model to best_model.hdf5\n",
      "Epoch 4/10\n",
      "135000/135000 [==============================] - 610s 5ms/step - loss: 8.3180 - val_loss: 5.4096\n",
      "\n",
      "Epoch 00004: val_loss improved from 21.63363 to 5.40958, saving model to best_model.hdf5\n",
      "Epoch 5/10\n",
      "135000/135000 [==============================] - 640s 5ms/step - loss: 4.3365 - val_loss: 3.9244\n",
      "\n",
      "Epoch 00005: val_loss improved from 5.40958 to 3.92440, saving model to best_model.hdf5\n",
      "Epoch 6/10\n",
      "135000/135000 [==============================] - 658s 5ms/step - loss: 3.4060 - val_loss: 5.2163\n",
      "\n",
      "Epoch 00006: val_loss did not improve from 3.92440\n",
      "Epoch 7/10\n",
      "135000/135000 [==============================] - 608s 5ms/step - loss: 3.2047 - val_loss: 3.4186\n",
      "\n",
      "Epoch 00007: val_loss improved from 3.92440 to 3.41857, saving model to best_model.hdf5\n",
      "Epoch 8/10\n",
      "135000/135000 [==============================] - 634s 5ms/step - loss: 2.6313 - val_loss: 3.1864\n",
      "\n",
      "Epoch 00008: val_loss improved from 3.41857 to 3.18637, saving model to best_model.hdf5\n",
      "Epoch 9/10\n",
      "135000/135000 [==============================] - 677s 5ms/step - loss: 2.3521 - val_loss: 3.0067\n",
      "\n",
      "Epoch 00009: val_loss improved from 3.18637 to 3.00666, saving model to best_model.hdf5\n",
      "Epoch 10/10\n",
      "135000/135000 [==============================] - 702s 5ms/step - loss: 2.1227 - val_loss: 3.0125\n",
      "\n",
      "Epoch 00010: val_loss did not improve from 3.00666\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x299611aa048>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 256\n",
    "epochs = 10\n",
    "model.fit(x=[training_img, train_padded_txt, train_input_length, train_label_length], y=np.zeros(len(training_img)), batch_size=batch_size, epochs = epochs, validation_data = ([valid_img, valid_padded_txt, valid_input_length, valid_label_length], [np.zeros(len(valid_img))]), verbose = 1, callbacks = callbacks_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model is now trained with 135000 images. Now its time to test the model. We can not use our training model because it also requires labels as input and at test time we can not have labels. So to test the model we will use ” act_model ” that we have created earlier which takes only one input: test images.\n",
    "\n",
    "As our model predicts the probability for each class at each time step, we need to use some transcription function to convert it into actual texts. Here we will use the CTC decoder to get the output text. Let’s see the code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original_text =   Expend\n",
      "predicted text = Expend\n",
      "\n",
      "original_text =   RAKE\n",
      "predicted text = RAKE\n",
      "\n",
      "original_text =   IMAM\n",
      "predicted text = MAM\n",
      "\n",
      "original_text =   kraft\n",
      "predicted text = kralt\n",
      "\n",
      "original_text =   deceleration\n",
      "predicted text = deceleration\n",
      "\n",
      "original_text =   FOXHUNTING\n",
      "predicted text = Foxhunting\n",
      "\n",
      "original_text =   Renaud\n",
      "predicted text = Renaud\n",
      "\n",
      "original_text =   Trenchant\n",
      "predicted text = Trenchant\n",
      "\n",
      "original_text =   HOD\n",
      "predicted text = HOD\n",
      "\n",
      "original_text =   sculpt\n",
      "predicted text = sculpt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# load the saved best model weights\n",
    "act_model.load_weights('best_model.hdf5')\n",
    " \n",
    "# predict outputs on validation images\n",
    "prediction = act_model.predict(valid_img[:10])\n",
    " \n",
    "# use CTC decoder\n",
    "out = K.get_value(K.ctc_decode(prediction, input_length=np.ones(prediction.shape[0])*prediction.shape[1],\n",
    "                         greedy=True)[0][0])\n",
    " \n",
    "# see the results\n",
    "i = 0\n",
    "for x in out:\n",
    "    print(\"original_text =  \", valid_orig_txt[i])\n",
    "    print(\"predicted text = \", end = '')\n",
    "    for p in x:  \n",
    "        if int(p) != -1:\n",
    "            print(char_list[int(p)], end = '')       \n",
    "    print('\\n')\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}