TF2_0_notes_training_and_evaluation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "TF2.0 notes - training and evaluation.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/michelucci/TF20-Notes/blob/master/TF2_0_notes_training_and_evaluation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNvub3HzbfPw",
        "colab_type": "text"
      },
      "source": [
        "__NOTE__: this method of choosing to use TF 2.X, works only\n",
        "in Google Colab, due to the use of the magic command ```%tensorflow_version```. Just something to keep in mind."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SkEhvJohHTYw",
        "colab_type": "code",
        "outputId": "675cfbfc-327b-489f-cea4-5c2defaa6bda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "\n",
        "tf.keras.backend.clear_session()  # For easy reset of notebook state."
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "83cOyeiSHVSg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TUBimlEUXYxr",
        "colab_type": "text"
      },
      "source": [
        "# References\n",
        "\n",
        "The notes originated from the TF 2.0 documentation (what else?). You can find it here. It has been expanded and modified by me. So errors are probably my fault...\n",
        "\n",
        "https://www.tensorflow.org/beta/guide/keras/training_and_evaluation\n",
        "\n",
        "Feel free to get in touch with me anytime at [umberto.michelucci@toelt.ai](mailto:umberto.michelucci@toelt.ai)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3IXb-tNlIQFd",
        "colab_type": "text"
      },
      "source": [
        "# Data preparation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D1MSlSRvHiC2",
        "colab_type": "code",
        "outputId": "615703c3-600e-41f0-85f3-d819319bf4a4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Load a toy dataset for the sake of this example\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "# Preprocess the data (these are Numpy arrays)\n",
        "x_train = x_train.reshape(60000, 784).astype('float32') / 255\n",
        "x_test = x_test.reshape(10000, 784).astype('float32') / 255\n",
        "\n",
        "y_train = y_train.astype('float32')\n",
        "y_test = y_test.astype('float32')\n",
        "\n",
        "# Reserve 10,000 samples for validation\n",
        "x_val = x_train[-10000:]\n",
        "y_val = y_train[-10000:]\n",
        "x_train = x_train[:-10000]\n",
        "y_train = y_train[:-10000]"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RMxvQAJBb4kl",
        "colab_type": "text"
      },
      "source": [
        "# Model specification"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6qsdN1FEcFJd",
        "colab_type": "text"
      },
      "source": [
        "Since we are working with the MNIST dataset (not again...), \n",
        "we can create a dumb feed forward network with Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v3qK5OyUb7Xb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "inputs = keras.Input(shape=(784,), name='digits')\n",
        "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
        "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
        "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
        "\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NJ57KM6ecMy0",
        "colab_type": "text"
      },
      "source": [
        "__NOTE__: we have used, in case you have not noticed,\n",
        "the functional APIs here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3zT7kH55IS2o",
        "colab_type": "text"
      },
      "source": [
        "# Model Compilation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cr_MANSqHmw1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer=keras.optimizers.RMSprop(),  # Optimizer\n",
        "              # Loss function to minimize\n",
        "              loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "              # List of metrics to monitor\n",
        "              metrics=[keras.metrics.SparseCategoricalAccuracy()])\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJjx3lF9IUnX",
        "colab_type": "text"
      },
      "source": [
        "# Model Fit"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PxQeQPyeH5VL",
        "colab_type": "code",
        "outputId": "e9ce4580-cd22-4d33-805f-760de4afcb20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        }
      },
      "source": [
        "history = model.fit(x_train, y_train,\n",
        "                    batch_size=64,\n",
        "                    epochs=3,\n",
        "                    validation_data=(x_val, y_val))"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 50000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "WARNING:tensorflow:Entity <function Function._initialize_uninitialized_variables.<locals>.initialize_variables at 0x7fcb09eb4950> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
            "WARNING: Entity <function Function._initialize_uninitialized_variables.<locals>.initialize_variables at 0x7fcb09eb4950> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: module 'gast' has no attribute 'Num'\n",
            "50000/50000 [==============================] - 4s 73us/sample - loss: 0.3348 - sparse_categorical_accuracy: 0.9053 - val_loss: 0.1968 - val_sparse_categorical_accuracy: 0.9429\n",
            "Epoch 2/3\n",
            "50000/50000 [==============================] - 3s 52us/sample - loss: 0.1601 - sparse_categorical_accuracy: 0.9527 - val_loss: 0.1432 - val_sparse_categorical_accuracy: 0.9583\n",
            "Epoch 3/3\n",
            "50000/50000 [==============================] - 3s 52us/sample - loss: 0.1173 - sparse_categorical_accuracy: 0.9652 - val_loss: 0.1275 - val_sparse_categorical_accuracy: 0.9621\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aWuibGQMH81q",
        "colab_type": "code",
        "outputId": "fda288fa-d81e-48d9-ab50-c0f0477a1802",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 76
        }
      },
      "source": [
        "print('\\nhistory dict:', history.history)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "history dict: {'loss': [0.3348192236375809, 0.16012076459884644, 0.11725219012498855], 'sparse_categorical_accuracy': [0.90534, 0.95266, 0.96522], 'val_loss': [0.19677914538383484, 0.14316599332690239, 0.12751918614506721], 'val_sparse_categorical_accuracy': [0.9429, 0.9583, 0.9621]}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pmk93XIKIB56",
        "colab_type": "code",
        "outputId": "b502a9ab-989a-4109-eb9d-ee1a75d414ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        }
      },
      "source": [
        "print('\\n# Evaluate on test data')\n",
        "results = model.evaluate(x_test, y_test, batch_size=128)\n",
        "print('test loss, test acc:', results)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "# Evaluate on test data\n",
            "\r10000/1 [================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 0s 14us/sample - loss: 0.0765 - sparse_categorical_accuracy: 0.9605\n",
            "test loss, test acc: [0.13459609513282775, 0.9605]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e2_Jk0NBIE3z",
        "colab_type": "code",
        "outputId": "b0746049-1992-4c8c-e5c6-c73d442a41d1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "print('\\n# Generate predictions for 3 samples')\n",
        "predictions = model.predict(x_test[:3])\n",
        "print('predictions shape:', predictions.shape)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "# Generate predictions for 3 samples\n",
            "predictions shape: (3, 10)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XzJjEsxnII-S",
        "colab_type": "text"
      },
      "source": [
        "# Better job with functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T25_AjDuIZzg",
        "colab_type": "text"
      },
      "source": [
        "## Let's create some functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IGZ6m5KeIb0o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_uncompiled_model():\n",
        "  inputs = keras.Input(shape=(784,), name='digits')\n",
        "  x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
        "  x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
        "  outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
        "  model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "  return model\n",
        "\n",
        "def get_compiled_model():\n",
        "  model = get_uncompiled_model()\n",
        "  model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['sparse_categorical_accuracy'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x5qi_FVzIe0x",
        "colab_type": "text"
      },
      "source": [
        "Many built-in optimizers, losses, and metrics are available\n",
        "In general, you won't have to create from scratch your own losses, metrics, or optimizers, because what you need is likely already part of the Keras API:\n",
        "\n",
        "Optimizers: - ```SGD()``` (with or without momentum) - ```RMSprop()``` - ```Adam()``` - etc.\n",
        "\n",
        "Losses: - ```MeanSquaredError()``` - ```KLDivergence()``` - ```CosineSimilarity(```) - etc.\n",
        "\n",
        "Metrics: - ```AUC()``` - ```Precision()``` - ```Recall()``` - etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8sEo1dLiIraX",
        "colab_type": "text"
      },
      "source": [
        "## Custom losses\n",
        "There are two ways to provide custom losses with Keras. The first example creates a function that accepts inputs y_true and y_pred. The following example shows a loss function that computes the average distance between the real data and the predictions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nFse2eRLIwMO",
        "colab_type": "code",
        "outputId": "9e90d99e-539c-445b-fa32-9a62a87e1d0d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 151
        }
      },
      "source": [
        "def basic_loss_function(y_true, y_pred):\n",
        "    return tf.math.reduce_mean(y_true - y_pred)\n",
        "\n",
        "model.compile(optimizer=keras.optimizers.Adam(),\n",
        "              loss=basic_loss_function)\n",
        "\n",
        "model.fit(x_train, y_train, batch_size=64, epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 50000 samples\n",
            "Epoch 1/3\n",
            "50000/50000 [==============================] - 2s 48us/sample - loss: 4.3488\n",
            "Epoch 2/3\n",
            "50000/50000 [==============================] - 2s 42us/sample - loss: 4.3488\n",
            "Epoch 3/3\n",
            "50000/50000 [==============================] - 2s 42us/sample - loss: 4.3488\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514ec3f518>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y_PJDGpnI17n",
        "colab_type": "text"
      },
      "source": [
        "## Custom metrics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ufzuKye0K9fL",
        "colab_type": "text"
      },
      "source": [
        "If you need a metric that isn't part of the API, you can easily create custom metrics by subclassing the ```Metric``` class. You will need to implement 4 methods:\n",
        "\n",
        "- ```__init__(self)```, in which you will create state variables for your metric.\n",
        "\n",
        "- ```update_state(self, y_true, y_pred, sample_weight=None)```, which uses the targets y_true and the model predictions y_pred to update the state variables.\n",
        "\n",
        "- ```result(self)```, which uses the state variables to compute the final results.\n",
        "\n",
        "- ```reset_states(self)```, which reinitializes the state of the metric.\n",
        "\n",
        "State update and results computation are kept separate (in ```update_state(```) and ```result()```, respectively) because in some cases, results computation might be very expensive, and would only be done periodically.\n",
        "\n",
        "Here's a simple example showing how to implement a ```CatgoricalTruePositives``` metric, that counts how many samples where correctly classified as belonging to a given class:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lNLkBklwLUC7",
        "colab_type": "code",
        "outputId": "1a9b9109-2baa-4b10-889e-1ff3f041dcc2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 151
        }
      },
      "source": [
        "class CatgoricalTruePositives(keras.metrics.Metric):\n",
        "\n",
        "    def __init__(self, name='categorical_true_positives', **kwargs):\n",
        "      super(CatgoricalTruePositives, self).__init__(name=name, **kwargs)\n",
        "      self.true_positives = self.add_weight(name='tp', initializer='zeros')\n",
        "\n",
        "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
        "      y_pred = tf.reshape(tf.argmax(y_pred, axis=1), shape=(-1, 1))\n",
        "      values = tf.cast(y_true, 'int32') == tf.cast(y_pred, 'int32')\n",
        "      values = tf.cast(values, 'float32')\n",
        "      if sample_weight is not None:\n",
        "        sample_weight = tf.cast(sample_weight, 'float32')\n",
        "        values = tf.multiply(values, sample_weight)\n",
        "      self.true_positives.assign_add(tf.reduce_sum(values))\n",
        "\n",
        "    def result(self):\n",
        "      return self.true_positives\n",
        "\n",
        "    def reset_states(self):\n",
        "      # The state of the metric will be reset at the start of each epoch.\n",
        "      self.true_positives.assign(0.)\n",
        "\n",
        "\n",
        "model.compile(optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "              loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "              metrics=[CatgoricalTruePositives()])\n",
        "model.fit(x_train, y_train,\n",
        "          batch_size=64,\n",
        "          epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 50000 samples\n",
            "Epoch 1/3\n",
            "50000/50000 [==============================] - 3s 57us/sample - loss: 0.0926 - categorical_true_positives: 48569.0000\n",
            "Epoch 2/3\n",
            "50000/50000 [==============================] - 2s 48us/sample - loss: 0.0757 - categorical_true_positives: 48860.0000\n",
            "Epoch 3/3\n",
            "50000/50000 [==============================] - 2s 48us/sample - loss: 0.0651 - categorical_true_positives: 48966.0000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514ec2f240>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TUIvrwIkLquv",
        "colab_type": "text"
      },
      "source": [
        "# Automatically setting up a validation holdout set"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G1Z9fsNWMK0s",
        "colab_type": "text"
      },
      "source": [
        "Automatically setting apart a validation holdout set\n",
        "In the first end-to-end example you saw, we used the ```validation_data``` argument to pass a tuple of Numpy arrays ```(x_val, y_val)``` to the model for evaluating a validation loss and validation metrics at the end of each epoch.\n",
        "\n",
        "Here's another option: the argument ```validation_split``` allows you to automatically reserve part of your training data for validation. The argument value represents the fraction of the data to be reserved for validation, so it should be set to a number higher than 0 and lower than 1. For instance, ```validation_split=0.2``` means \"use 20% of the data for validation\", and ```validation_split=0.6``` means \"use 60% of the data for validation\".\n",
        "\n",
        "The way the validation is computed is by taking the last x% samples of the arrays received by the fit call, before any shuffling.\n",
        "\n",
        "You can only use ```validation_split``` when training with Numpy data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hl_rsDv9Mbu2",
        "colab_type": "code",
        "outputId": "41c71602-ee26-4c8f-9174-2efab11b2db2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 171
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "model.fit(x_train, y_train, batch_size=64, validation_split=0.2, epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 40000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "40000/40000 [==============================] - 3s 68us/sample - loss: 0.3807 - sparse_categorical_accuracy: 0.8920 - val_loss: 0.2477 - val_sparse_categorical_accuracy: 0.9270\n",
            "Epoch 2/3\n",
            "40000/40000 [==============================] - 2s 52us/sample - loss: 0.1848 - sparse_categorical_accuracy: 0.9457 - val_loss: 0.1753 - val_sparse_categorical_accuracy: 0.9476\n",
            "Epoch 3/3\n",
            "40000/40000 [==============================] - 2s 53us/sample - loss: 0.1326 - sparse_categorical_accuracy: 0.9601 - val_loss: 0.1598 - val_sparse_categorical_accuracy: 0.9522\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514fa8feb8>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ilxCV7P-Md6e",
        "colab_type": "text"
      },
      "source": [
        "# Using ```tf.data.Datasets```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-S2hjtpMm9X",
        "colab_type": "code",
        "outputId": "f2b06bf1-3ed9-4356-c7d3-e860266e3752",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# First, let's create a training Dataset instance.\n",
        "# For the sake of our example, we'll use the same MNIST data as before.\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "# Shuffle and slice the dataset.\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Now we get a test dataset.\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
        "test_dataset = test_dataset.batch(64)\n",
        "\n",
        "# Since the dataset already takes care of batching,\n",
        "# we don't pass a `batch_size` argument.\n",
        "model.fit(train_dataset, epochs=3)\n",
        "\n",
        "# You can also evaluate or predict on a dataset.\n",
        "print('\\n# Evaluate')\n",
        "model.evaluate(test_dataset)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "782/782 [==============================] - 6s 7ms/step - loss: 0.3370 - sparse_categorical_accuracy: 0.9048\n",
            "Epoch 2/3\n",
            "782/782 [==============================] - 3s 4ms/step - loss: 0.1563 - sparse_categorical_accuracy: 0.9536\n",
            "Epoch 3/3\n",
            "782/782 [==============================] - 3s 3ms/step - loss: 0.1153 - sparse_categorical_accuracy: 0.9652\n",
            "\n",
            "# Evaluate\n",
            "157/157 [==============================] - 0s 2ms/step - loss: 0.1217 - sparse_categorical_accuracy: 0.9625\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.1216799323246844, 0.9625]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zg5VCCXNcgM",
        "colab_type": "text"
      },
      "source": [
        "If you want to run training only on a specific number of batches from this Dataset, you can pass the ```steps_per_epoch argument```, which specifies how many training steps the model should run using this Dataset before moving on to the next epoch.\n",
        "\n",
        "If you do this, the dataset is not reset at the end of each epoch, instead we just keep drawing the next batches. The dataset will eventually run out of data (unless it is an infinitely-looping dataset)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L-4pRgH3N5Dt",
        "colab_type": "code",
        "outputId": "f5f82852-30ad-4738-d691-96fcfab6f3fb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Only use the 100 batches per epoch (that's 64 * 100 samples)\n",
        "model.fit(train_dataset.take(100), epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "100/100 [==============================] - 1s 13ms/step - loss: 0.7970 - sparse_categorical_accuracy: 0.7989\n",
            "Epoch 2/3\n",
            "100/100 [==============================] - 0s 5ms/step - loss: 0.3435 - sparse_categorical_accuracy: 0.9069\n",
            "Epoch 3/3\n",
            "100/100 [==============================] - 0s 5ms/step - loss: 0.2659 - sparse_categorical_accuracy: 0.9253\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514c194630>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PmdXYL8iN8-d",
        "colab_type": "text"
      },
      "source": [
        "## Explanation of the ```buffer_size``` parameter in ```shuffle()```\n",
        "\n",
        "By contrast, the ```buffer_size``` argument to ```tf.data.Dataset.shuffle()``` affects the randomness of the transformation. We designed the ```Dataset.shuffle()``` transformation (like the ```tf.train.shuffle_batch()``` function that it replaces) to handle datasets that are too large to fit in memory. Instead of shuffling the entire dataset, it maintains a buffer of ```buffer_size``` elements, and randomly selects the next element from that buffer (replacing it with the next input element, if one is available). Changing the value of ```buffer_size``` affects how uniform the shuffling is: if ```buffer_size``` is greater than the number of elements in the dataset, you get a uniform shuffle; if it is 1 then you get no shuffling at all. For very large datasets, a typical \"good enough\" approach is to randomly shard the data into multiple files once before training, then shuffle the filenames uniformly, and then use a smaller shuffle buffer. However, the appropriate choice will depend on the exact nature of your training job.\n",
        "\n",
        "From: https://stackoverflow.com/questions/46444018/meaning-of-buffer-size-in-dataset-map-dataset-prefetch-and-dataset-shuffle"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zoc_WQapO_KN",
        "colab_type": "text"
      },
      "source": [
        "## Validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rg53AOrNPYDO",
        "colab_type": "code",
        "outputId": "da5874cb-d003-4db3-b0b1-ddc6c747ca22",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Prepare the validation dataset\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "val_dataset = val_dataset.batch(64)\n",
        "\n",
        "model.fit(train_dataset, epochs=3, validation_data=val_dataset)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "782/782 [==============================] - 6s 7ms/step - loss: 0.3295 - sparse_categorical_accuracy: 0.9075 - val_loss: 0.0000e+00 - val_sparse_categorical_accuracy: 0.0000e+00\n",
            "Epoch 2/3\n",
            "782/782 [==============================] - 3s 4ms/step - loss: 0.1603 - sparse_categorical_accuracy: 0.9523 - val_loss: 0.1390 - val_sparse_categorical_accuracy: 0.9617\n",
            "Epoch 3/3\n",
            "782/782 [==============================] - 3s 4ms/step - loss: 0.1182 - sparse_categorical_accuracy: 0.9649 - val_loss: 0.1188 - val_sparse_categorical_accuracy: 0.9647\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514c25b208>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IJjqTv8Pbw-",
        "colab_type": "text"
      },
      "source": [
        "At the end of each epoch, the model will iterate over the validation Dataset and compute the validation loss and validation metrics.\n",
        "\n",
        "If you want to run validation only on a specific number of batches from this Dataset, you can pass the ```validation_steps``` argument, which specifies how many validation steps the model should run with the validation Dataset before interrupting validation and moving on to the next epoch:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eHNlwBqTPie1",
        "colab_type": "code",
        "outputId": "10d39050-2b3c-4920-f677-c675235f6a0a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Prepare the validation dataset\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "val_dataset = val_dataset.batch(64)\n",
        "\n",
        "model.fit(train_dataset, epochs=3,\n",
        "          # Only run validation using the first 10 batches of the dataset\n",
        "          # using the `validation_steps` argument\n",
        "          validation_data=val_dataset, validation_steps=10)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "782/782 [==============================] - 6s 7ms/step - loss: 0.3389 - sparse_categorical_accuracy: 0.9050 - val_loss: 0.0000e+00 - val_sparse_categorical_accuracy: 0.0000e+00\n",
            "Epoch 2/3\n",
            "782/782 [==============================] - 3s 3ms/step - loss: 0.1581 - sparse_categorical_accuracy: 0.9533 - val_loss: 0.2267 - val_sparse_categorical_accuracy: 0.9375\n",
            "Epoch 3/3\n",
            "782/782 [==============================] - 3s 3ms/step - loss: 0.1149 - sparse_categorical_accuracy: 0.9656 - val_loss: 0.1594 - val_sparse_categorical_accuracy: 0.9531\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514bbc8f98>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WCED1n9vPoT2",
        "colab_type": "text"
      },
      "source": [
        "# Passing data to multi-input, multi-output models\n",
        "\n",
        "In the previous examples, we were considering a model with a single input (a tensor of shape ```(764,)```) and a single output (a prediction tensor of shape```(10,)```). But what about models that have multiple inputs or outputs?\n",
        "\n",
        "Consider the following model, which has an image input of shape ```(32, 32, 3)``` (that's ```(height, width, channels)```) and a timeseries input of shape ```(None, 10)``` (that's ```(timesteps, features)```). Our model will have two outputs computed from the combination of these inputs: a \"score\" (of shape ```(1,)```) and a probability distribution over 5 classes (of shape ```(10,)```)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RgddOW8oQgck",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "image_input = keras.Input(shape=(32, 32, 3), name='img_input')\n",
        "timeseries_input = keras.Input(shape=(None, 10), name='ts_input')\n",
        "\n",
        "x1 = layers.Conv2D(3, 3)(image_input)\n",
        "x1 = layers.GlobalMaxPooling2D()(x1)\n",
        "\n",
        "x2 = layers.Conv1D(3, 3)(timeseries_input)\n",
        "x2 = layers.GlobalMaxPooling1D()(x2)\n",
        "\n",
        "x = layers.concatenate([x1, x2])\n",
        "\n",
        "score_output = layers.Dense(1, name='score_output')(x)\n",
        "class_output = layers.Dense(5, activation='softmax', name='class_output')(x)\n",
        "\n",
        "model = keras.Model(inputs=[image_input, timeseries_input],\n",
        "                    outputs=[score_output, class_output])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XK4EFTvYQiyk",
        "colab_type": "code",
        "outputId": "97675f3a-4771-41aa-909f-392472b18aa6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 553
        }
      },
      "source": [
        "keras.utils.plot_model(model, 'multi_input_and_output_model.png', show_shapes=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABIsAAAIECAYAAACZua5ZAAAABmJLR0QA/wD/AP+gvaeTAAAgAElE\nQVR4nOzdeViVdd4/8PcRDhwOsigKIoqy5IKZWtoTqFnjZCrlylY2M9Too1YDplOMOiSiYC6PcrlQ\nU5EzT1aCywWaIj5mDDrjUpnLUCpipsLIIiqrcoDP7w9/3OOJRZYD53B4v66LP7zP9/5+vvfNOZ43\n33tTiYiAiIiIiIiIiIgI2NHF2CMgIiIiIiIiIiLTwckiIiIiIiIiIiJScLKIiIiIiIiIiIgUnCwi\nIiIiIiIiIiKFpbEH0NkcO3YM69evN/YwiIjIjC1cuBC+vr7GHgaRyQsMDDT2EIiIqB47duww9hA6\nPZ5Z1M6uXbuGnTt3GnsYRCZl586duH79urGH0aEcP34cx48fN/YwyATt3LkT165dM/YwiDoEfv+Q\nObh+/Tr/vmgBfv5NE9/PpoNnFhkJZ0qJ/kOlUuGtt95CUFCQsYfSYdQeDef/JfRLKpXK2EMg6lD4\n/UMdXVJSEoKDg5kJmon50zTVvp/J+HhmERERERERERERKThZRERERERERERECk4WERERERERERGR\ngpNFRERERERERESk4GQREREREREREREpOFlERGZj//79cHBwwN69e409FJM0b948qFQq5eeVV16p\n0+bQoUNYvHgxampqMH36dLi7u0Oj0cDNzQ1Tp07F2bNnm1139erVGDRoEGxsbGBra4tBgwYhMjIS\nxcXFeu2io6Ph4+MDe3t7WFtbw9vbG++88w5KS0tbtL2mXHfPnj1YvXo1qqur9dZNTk7W+x316NGj\nRWMgIiJqLuaoxhkjRxkyjz3Y54YNG+Dn59dgm6NHj2L06NHQarVwdXVFREQE7t27p7zOHNM5cLKI\niMyGiBh7CCave/fuSE1NxYULF5CQkKD32rJly7Bx40YsWbIENTU1OHLkCD7//HMUFRXh6NGjqKio\nwNNPP43c3Nxm1Txy5AjmzJmDq1evIi8vDytWrMDq1asREBCg1+7w4cN48803ceXKFRQWFiI2NhZx\ncXEIDAxs0baact0pU6ZAo9Fg/PjxuH37trJ86tSpuH79OjIyMjB58uQW1SciImoJ5qiHa+8cZcg8\nBgBZWVl4+umnsXDhQpSXl9fbJjMzExMmTMD48eNRUFCA3bt345NPPsH8+fOVNswxnYRQu0pMTBTu\ndiJ9ACQxMdHYwzCo8vJy8fX1bbP+AwICJCAgoFnrzJ07V9zc3Op9bdWqVTJgwACpqKgQERGdTicv\nvPCCXpuTJ08KAImJiWlW3enTpyv91goMDBQAkpubqyzz9/eXqqoqvXZBQUECQK5evdqsmh2hrohI\nWFiY+Pr6ik6nq9NPeHi4ODk5Nbu+OX6eiNoKPy9kDszx74u2zlEizf/8GyNHGTKPnT59WmbMmCHb\ntm2T4cOHy7Bhw+ptFxwcLB4eHlJTU6MsW7t2rahUKvnxxx/12rZFjjHH93MHlcQzi4iI2kBCQgLy\n8/ONPYwmuXTpEiIjI7F8+XJoNBoAgKWlZZ3T0D09PQEA2dnZzep/9+7dSr+13NzcAEDvUq8vv/wS\nFhYWeu1qT19u6OhXR64LAFFRUTh9+jTi4uKaXYeIiMhcMUfBoP0AwLBhw7Br1y7MmjUL1tbW9bap\nqqrCvn37MG7cOKhUKmX5pEmTICJISUnRa88cY944WUREZuHo0aNwd3eHSqXC5s2bAQDx8fGwtbWF\nVqtFSkoKJk2aBHt7e/Tp0wdffPGFsu7GjRuh0Wjg7OyMefPmwdXVFRqNBn5+fjhx4oTSLiwsDFZW\nVujVq5ey7I033oCtrS1UKhUKCwsBAAsWLMCiRYuQnZ0NlUoFb29vAMCBAwdgb2+PmJiY9tglTbZx\n40aICKZMmdJou4qKCgCAvb19q2tmZWXB0dER/fr1a7RdTk4ObGxs4OHh0eqapli3W7duGDduHOLi\n4nj6PxERGQ1zVMu1d44yZB77pcuXL6O0tBTu7u56y728vACgzr2SmGPMGyeLiMgsjBkzBv/85z/1\nlr3++ut46623UFFRATs7OyQmJiI7Oxuenp6YM2cOdDodgPvhJTQ0FOXl5QgPD8eVK1dw6tQpVFVV\n4bnnnsO1a9cA3A8DQUFBejW2bNmC5cuX6y2Li4vDiy++CC8vL4gILl26BADKTQBramraZB+01L59\n+zBw4EBotdpG2508eRLA/X3dEjqdDjk5Odi8eTMOHTqETZs2wcrKqsH25eXlOHz4MObMmdNou45e\nd8SIEcjJycGZM2daXIuIiKg1mKNarr1ylKH7qc+NGzcAAHZ2dnrLNRoNbGxskJeXV2cd5hjzZWns\nARARtQc/Pz/l1OCQkBAcOXIEV69eVY6UAPdP9R08eDAAwMfHB/Hx8Rg1ahS2bt2Kd999t9Vj8Pf3\nr/MkLmMrKyvDTz/9hBdeeKHBNnl5efj666/x9ttvw9fX96FHzhrSt29f5OXlwcnJCWvWrEFwcHCj\n7WNjY+Hq6oqVK1e2qF5HqfvII48AAM6dO4fhw4e3qiYREVFbYI6qX3vmKEP105jaJ5798vJ8AFCr\n1cpZTQ9ijjFfPLOIiDqd2rM7ao+INWTkyJHQarU4f/58ewzLKPLz8yEijR4N8/X1RXh4OKZNm4bU\n1FSo1eoW1bp27Rry8/Px+eef429/+xtGjBjR4P0Idu/ejaSkJKSlpdU5umVudWv3fX1H64iIiEwN\nc9R/tGeOMlQ/jamdEKyqqqrzWmVlJWxsbOosZ44xXzyziIioEdbW1igoKDD2MNrM3bt3AaDBGx0C\ngLOzMxISEjBkyJBW1VKr1ejZsycmTJgADw8PDBgwQHlM/YO2b9+O9evXIz09Hb17925VzY5QtzZ4\n1f4uiIiIzAVzlOFylKH6aUzt/aR+eQZXeXk57t69C1dX1zrrMMeYL04WERE1QKfT4fbt2+jTp4+x\nh9Jmar/ga+8DUJ+ePXvC0dHRoHW9vb1hYWGBzMxMveWbNm1CWloaDh8+jK5duxq0pinWBe4fqQNQ\n79E6IiKijoo56j5D5ai2yGO/5OHhATs7O/z88896y2vvG/XYY4/VWYc5xnzxMjQiogakp6dDRPDU\nU08pyywtLR962nVH4uzsDJVKhTt37jTYZu/evcqj35vr5s2bePnll+ssz8rKQnV1Nfr27QsAEBFE\nRETg3LlzSE5ObvWEjanXfVDtvndxcWlVbSIiIlPCHHVfa3JUW/TTGEtLS0yePBkZGRl6NxJPTU2F\nSqWq9z5JzDHmi5NFRET/X01NDW7duoWqqiqcPXsWCxYsgLu7O0JDQ5U23t7eKCoqQnJyMnQ6HQoK\nCuocfQGA7t27Izc3F1euXEFJSQl0Oh1SU1NN7pGvWq0Wnp6euH79er2vX7p0CS4uLvXenDkkJAQu\nLi44depUg/3b2tri4MGDOHz4MIqLi6HT6fD999/jd7/7HWxtbbFw4UIAwA8//IA1a9bgo48+glqt\nhkql0vtZt26dWdV9UO2+Hzp0aIP9EhERmTrmqLpam6MM3U9TREZGIi8vD8uWLUNZWRmOHTuGtWvX\nIjQ0FAMHDqzTnjnGfHGyiIjMwubNmzFq1CgAQEREBKZOnYr4+Hhs2LABwP3TZi9fvoyPPvoIixYt\nAgBMnDgRWVlZSh93797F0KFDYWNjg7Fjx2LAgAH4+uuv9a5Df/311/Hss8/ipZdewsCBA7FixQrl\ntFtfX1/l8bDz58+Hs7MzfHx8MHnyZBQVFbXLfmgJf39/ZGZm1vuECxFpcL3Kykrk5+cjJSWlwTYa\njQajR4/G7Nmz4ebmBjs7OwQGBqJ///44fvw4Hn300YfWMce6D/rmm2/g5uZW76ndRERE7YE5quXa\nMkcZsp/jx49jzJgx6N27N06cOIEzZ87A1dUVo0ePRkZGhtJuyJAhSEtLw8GDB+Hk5ISZM2fitdde\nw/vvv19vv8wxZkyoXSUmJgp3O5E+AJKYmGjUMcydO1e6d+9u1DE0R0BAgAQEBDRrnblz54qbm1ud\n5VlZWWJpaSmffvpps/qrrq6WsWPHSkJCQrPWay1zqltYWCgajUbWrVtX57Xw8HBxcnJqdp+m8Hki\n6ij4eSFzYAp/X3S0HCXS/M+/qeYoY+UikbbJMabwfiYREUnimUVERP9fYzcnNBcVFRVIS0tDVlaW\nckNCb29vREdHIzo6GqWlpU3qp7q6GsnJySgpKUFISEhbDtms60ZFRWH48OEICwsDcP/IYW5uLo4e\nParcTJKIiKgjYI5q/xxlrFxUiznGvHGyiIioEykqKsLEiRMxYMAAvPbaa8ryxYsXIzAwECEhIY3e\npLFWeno6du3ahdTUVGi12rYcstnWXb9+PU6fPo39+/dDrVYDAFJSUuDm5oaxY8di3759BqlDRERE\nhmFqOcpYuQhgjukMOFnUAezfvx8ODg7Yu3evsYdSL1MfX1McP34cgwcPRpcuXaBSqeDi4oKVK1ca\ne1h6du3aBU9PT+XGu7169cIrr7xi7GGZhSVLlmDr1q24c+cOPDw8sHPnTmMPqU188MEHEBHlZ9u2\nbXqvx8TEICwsDKtWrXpoX+PHj8dnn32GXr16tdVwzbpuSkoK7t27h/T0dHTr1k1ZPm3aNL3fUWFh\noUHqEZFpY5ZqH8xSbYM56j5j5Chj5SLmmM7B0tgDoIeTZtyA1RhMfXxN8dRTT+HHH3/ExIkTkZaW\nhgsXLsDR0dHYw9Izc+ZMzJw5E97e3igsLMSNGzeMPSSzERsbi9jYWGMPwyRMmDABEyZMMPYwzN7U\nqVMxdepUYw+DiEwEs1T7YJZqG8xR/9FZchRzTOfAM4s6AH9/f9y5cwcvvviisYdSL1MaX0VFBfz8\n/Iw9DIMwp20hIiIyB2313cws1TbMaVuIiNobJ4sIwP0jWjt27MCHH35o7KG0SkJCAvLz8409DIMw\np20hIiIyB53hu9mcttGctoWIqL1xssjEHT16FO7u7lCpVNi8eTMAIC4uDra2tujSpQueeOIJuLi4\nQK1Ww9bWFo8//jjGjh2Lvn37QqPRwNHREe+8845en9XV1YiNjcXAgQNhY2ODHj16wMPDA7GxsQgK\nCmr1+OLj42FrawutVouUlBRMmjQJ9vb26NOnD7744gtl3Y0bN0Kj0cDZ2Rnz5s2Dq6srNBoN/Pz8\ncOLECaVdWFgYrKys9K7FfeONN2BrawuVSqVcC7tgwQIsWrQI2dnZUKlU8Pb2BgAcOHAA9vb2iImJ\nada2meK2NNeRI0fg4+MDBwcHaDQaDB06FGlpaQCA2bNnK9fse3l54fvvvwcAvPrqq9BqtXBwcMCe\nPXsA3H/PvPvuu3B3d4eNjQ0ee+wxJCYmAgDWrFkDrVYLOzs75OfnY9GiRXBzc8OFCxdaNGYiIiJT\n1NB389///nc8+eST0Gq1sLe3x9ChQ1FcXNzkfpml2ndbmotZiog6LaF2lZiYKM3d7deuXRMAsmnT\nJmXZsmXLBICcOHFCysrKpLCwUCZOnCgAZN++fVJQUCBlZWUSFhYmAOT06dPKujExMWJhYSEpKSlS\nXl4u3333nbi4uMgzzzzTom2qb3xLly4VAPLVV1/JnTt3JD8/X8aOHSu2trZSWVmptJs7d67Y2trK\nDz/8IHfv3pXMzEwZNWqU2NnZydWrV5V2s2bNEhcXF726a9euFQBSUFCgLJs5c6Z4eXnptfvyyy/F\nzs5OoqOjH7otzz//vACQW7dumeS2iIh4eXmJg4PDQ7dFRGTHjh0SFRUlRUVFcvPmTXnqqafEyclJ\nr4aFhYXk5OTorffyyy/Lnj17lH//8Y9/FGtra9m5c6fcunVLlixZIl26dJFvvvlGbx+Fh4fLpk2b\nZMaMGfLjjz82aYwiIgAkMTGxye1JJCAgQAICAow9DDJB/DwRNV1zPy+//G4uLS0Ve3t7Wb16tVRU\nVMiNGzdkxowZet/nTcEs1X7bImJ+Waolf18Qvy9NFd/PJiOJZxZ1cD4+PtBqtXBycsJLL70EAHB3\nd0ePHj2g1WqVJzycP39eWSc5ORlPPPEEpkyZAhsbGzz++OOYOnUqMjIyUFlZadDx+fn5wd7eHj17\n9kRISAjKyspw9epVvTaWlpYYPHgwrK2t4ePjg/j4eJSUlGDr1q0GGYO/vz+Ki4sRGRnZqn5MYVua\nKyAgAMuWLUO3bt3QvXt3TJkyBTdv3kRBQQEAYP78+aiurtYbX3FxMb755htMnjwZAHD37l3Ex8dj\n+vTpmDlzJhwdHfHnP/8ZarW6zna99957ePPNN7Fr1y4MGjSo/TaUiIjICK5cuYLi4mIMGTIEGo0G\nLi4u2LVrF3r06GGwGqaQP5ilmKWIqPPh09DMiJWVFQCgqqpKWaZWqwEAOp1OWXb37l1oNBq9daur\nq6FWq2FhYdHm43twLPUZOXIktFqt3gSXqemo21L7fqiurgYA/OpXv8KAAQPwySefYMmSJVCpVNi+\nfTtCQkKU98KFCxdQXl6ORx99VOnHxsYGvXr1Muh2BQcHIzg42GD9dRYqlcrYQyAi6rQ8PT3h7OyM\nV155BeHh4QgNDUX//v3brF5HzR/16ajbYspZipmg+Zg/iRrGyaJOaPLkyVi7di1SUlIwYcIEZGZm\nIjk5GS+88EKbThY1h7W1tXLEpqMz5rbs27cPa9euRWZmJoqLi+sEMpVKhXnz5mHhwoX46quv8Otf\n/xr/+7//i88++0xpU1ZWBgD485//jD//+c9667u6uhpsrAsWLICvr6/B+jN3GzZsAAC89dZbRh4J\nmRqGXqL2Y2Njg8OHD+NPf/oTYmJiEB0djaCgIGzduhU2NjZGHRuzlGF0pCxVew8kaprg4GDmTxN0\n7NgxxMXFGXsYBE4WdUpRUVH47rvvEBoaitLSUri6uiIoKKhFNy1sCzqdDrdv30afPn2MPZRWa+9t\nycjIwHfffYe33noLV69exfTp0zFjxgx88skn6N27NzZt2lTnhuehoaFYsmQJPv74Y/Tt2xf29vbo\n16+f8nrPnj0B3J+cWLBgQZuN3dfXt9k3WO/MduzYAQDcZ1QHJ4uI2teQIUOwd+9eFBQUYP369Xjv\nvfcwZMiQVl+y1RrMUi3XkbMUM0HzBAcHM3+aKE4WmQZOFnVCmZmZyM7ORkFBASwtTe8tkJ6eDhHB\nU089pSyztLR86GnKpqi9t+W7776Dra0tAODcuXPQ6XR4/fXX4enpCaD+05O7deuG4OBgbN++HXZ2\ndpgzZ47e67VP1jt9+nSbjJmIiKijys3Nxe3bt+Hj44OePXti1apVOHjwIH744QejjotZquWYpYiI\n7uMNrjuhN998E+7u7igtLTX2UAAANTU1uHXrFqqqqnD27FksWLAA7u7uCA0NVdp4e3ujqKgIycnJ\n0Ol0KCgowM8//1ynr+7duyM3NxdXrlxBSUkJdDodUlNTW/y4V1PblobodDrk5eUhPT1dCTju7u4A\ngEOHDuHu3bvIysrSe/Tsg+bPn4979+7hyy+/xIsvvqj3mkajwauvvoovvvgC8fHxKC4uRnV1Na5f\nv45///vfzd1FREREHdYvv5t//vlnzJs3D+fPn0dlZSW+//57/Pzzz3oTG+2BWYpZiojI4Iz7NLbO\np7mPAty0aZP06tVLAIhWq5UpU6ZIXFycaLVaASD9+/eXI0eOyHvvvScODg4CQFxcXOSzzz6T7du3\ni4uLiwCQbt26yRdffCEiIocPHxYnJycBoPyo1WoZPHiw7Nq1q1nbU9/4tmzZoozvkUcekezsbPnw\nww/F3t5eAEi/fv3k4sWLInL/EalqtVrc3NzE0tJS7O3tZdq0aZKdna1X5+bNm/Lss8+KRqMRDw8P\n+cMf/iBvv/22ABBvb2/lcaqnTp2Sfv36iY2NjYwZM0Zu3Lgh+/fvFzs7O1m5cmWD23H8+HEZMmSI\ndOnSRQBIr169JCYmxqS25f333xcvLy+931t9P7t371ZqRURESPfu3cXR0VECAwNl8+bNAkC8vLz0\nHkErIjJixAhZvHhxvfvn3r17EhERIe7u7mJpaSk9e/aUmTNnSmZmpqxevVpsbGwEgPTt21c+/fTT\nprx19ICPLm22gIAACQgIMPYwyATx80TUdM39vPzyu/nEiRPi5+cn3bp1EwsLC+ndu7csXbpUqqqq\nmtwnsxSzVGuzFB813jL8vjRNfD+bjCSViIihJ6CoYUlJSQgODoYxd3t8fDyysrKUG+QCQGVlJf70\npz8hPj4et27darebMs6bNw87duzAzZs326VeW+ro2+Lv74/NmzfDw8Oj3WurVCokJibymvFmCAwM\nBPCfexcR1eLniajpzOHz0tHzx4M6+rYYK0uZwt8XHZE5fP7NEd/PJmOH6d2whtrUjRs3EBYWVuea\naSsrK7i7u0On00Gn07XrEzxqHz1qDjrStuh0OuXxr2fPnoVGozHKRBERERG1TkfKHw/TkbaFWYqI\nzBnvWdTJ2NjYQK1WIyEhAXl5edDpdMjNzcXHH3+Md999FyEhIcjNzYVKpXroT0hIiLE3h1ohIiIC\nWVlZuHjxIl599VWsWLHC2EOiNjZv3jy9z/Arr7xSp82hQ4ewePFi1NTUYPr06XB3d4dGo4Gbmxum\nTp2Ks2fPNrvu6tWrMWjQINjY2MDW1haDBg1CZGQkiouL9dpFR0fDx8cH9vb2sLa2hre3N955550W\n31/NlOvu2bMHq1evrvNHUXJyst7vqEePHi0aAxEZ1/nz55mlOgFmqc7FGDnKkHnswT43bNgAPz+/\nBtscPXoUo0ePhlarhaurKyIiInDv3j3ldeaYTsKoV8F1QqZwDWZGRob8+te/Fnt7e7GwsBAHBwfx\n8/OTLVu2iE6na7dxLF68WKysrJR7L+3YsaPdahtaR9yWpUuXSpcuXaRv376yZ88eo44FvGa82Vpy\nz6K5c+dK9+7dJTU1VS5cuCB3797Ve/3dd9+VF198UYqLi0Wn04mTk5McOXJEysrK5PLly/Lcc8+J\ng4OD5OTkNKuuv7+/rFu3TvLz86WkpESSkpJErVbLc889p9du3LhxsmXLFrl586YUFxdLYmKiqNVq\nmThxYrPqdZS6cXFxMm7cOLl165ayrKamRq5fvy4ZGRkyefJkcXJyanZ9fp6Imq6jf146Yv5oSEfc\nFlPJUqbw90VH1NzPvzFylCHzmIjIxYsXZfTo0QJAhg0bVm+bf/3rX2JjYyORkZFSWloq//znP6VH\njx7y6quv6rVrqxzD97PJSOJvoZ3xzU9Ul7HDenl5ufj6+naoGi2dLHJzc6v3tVWrVsmAAQOkoqJC\nRO6HkxdeeEGvzcmTJwWAxMTENKvu9OnTlX5rBQYGCgDJzc1Vlvn7+9e5KWxQUJAAqHMTUXOoKyIS\nFhYmvr6+9U7Uh4eHc7KIqI3x80LmwBT+vuiIWaolk0XtnaMMmcdOnz4tM2bMkG3btsnw4cMbnCwK\nDg4WDw8PqampUZatXbtWVCqV/Pjjj3pt2yLHmML7mUREJImXoRFRp5eQkID8/PwOX6OlLl26hMjI\nSCxfvhwajQYAYGlpib179+q18/T0BABkZ2c3q//du3cr/dZyc3MDAL1Lvb788ktYWFjotas9fbm8\nvLxZNTtCXQCIiorC6dOnERcX1+w6REREpqIzZ6m2zFGGzGPDhg3Drl27MGvWLFhbW9fbpqqqCvv2\n7cO4ceOgUqmU5ZMmTYKIICUlRa89c4x542QREXU4IoL169dj8ODBsLa2Rrdu3TBt2jScP39eaRMW\nFgYrKyv06tVLWfbGG2/A1tYWKpUKhYWFAIAFCxZg0aJFyM7Ohkqlgre3NzZu3AiNRgNnZ2fMmzcP\nrq6u0Gg08PPzw4kTJwxSAwAOHDgAe3t7xMTEtOn+epiNGzdCRDBlypRG21VUVAAA7O3tW10zKysL\njo6O6NevX6PtcnJyYGNjY7Abhppa3W7dumHcuHGIi4vjUz+IiKjdMEsZTnvnKEPmsV+6fPkySktL\n4e7urrfcy8sLAOrcK4k5xrxxsoiIOpyoqCgsXrwYS5cuRX5+PjIyMnDt2jWMHTsWeXl5AO5/cf/y\nUahbtmzB8uXL9ZbFxcXhxRdfhJeXF0QEly5dQlhYGEJDQ1FeXo7w8HBcuXIFp06dQlVVFZ577jlc\nu3at1TWA/zzxpaamxnA7pwX27duHgQMHQqvVNtru5MmTAIAxY8a0qI5Op0NOTg42b96MQ4cOYdOm\nTbCysmqwfXl5OQ4fPow5c+Y02q6j1x0xYgRycnJw5syZFtciIiJqDmYpw2mvHGXofupz48YNAICd\nnZ3eco1GAxsbG+W98SDmGPPFySIi6lAqKiqwfv16zJgxA6+88gocHBwwdOhQfPDBBygsLMSHH35o\nsFqWlpbKETcfHx/Ex8ejpKQEW7duNUj//v7+KC4uRmRkpEH6a4mysjL89NNPyhGj+uTl5WH79u0I\nDw+Hr6/vQ4+cNaRv377o06cPoqKisGbNGgQHBzfaPjY2Fq6urli5cmWL6nWUuo888ggA4Ny5c62q\nR0RE1BTMUobTnjnKUP00pvaJZ7+8PB8A1Gq1clbTg5hjzBcni4ioQ8nMzERpaSlGjhypt3zUqFGw\nsrLSO7XZ0EaOHAmtVqt3inZHl5+fDxFp9GiYr68vwsPDMW3aNKSmpkKtVreo1rVr15Cfn4/PP/8c\nf/vb3zBixIgG7z2we/duJCUlIS0trc7RLXOrW7vv6ztaR0REZGjMUobTnjnKUP00pvaeS1VVVXVe\nq6yshI2NTZ3lzDHmy9LYAyAiao7bt28DALp27VrnNUdHR5SUlLRpfWtraxQUFLRpjfZ09+5dAGjw\nRocA4OzsjISEBAwZMqRVtdRqNXr27IkJEybAw8MDAwYMQGxsbJ2bIm7fvh3r169Heno6evfu3aqa\nHaFubfCq/V0QERG1JWYpw2nPHGWofhpTe++o4uJiveXl5eW4e/cuXF1d66zDHGO+OFlERB2Ko6Mj\nANQbZG7fvo0+ffq0WW2dTtfmNdpb7Rd87TX/9enZs6ey3w3F29sbFhYWyG05CKEAACAASURBVMzM\n1Fu+adMmpKWl4fDhw/WGWHOrC9w/Ugeg3qN1REREhsYsZTjtmaPaIo/9koeHB+zs7PDzzz/rLa+9\nR9Rjjz1WZx3mGPPFy9CIqEN59NFH0bVrV3z77bd6y0+cOIHKyko88cQTyjJLS0vodDqD1U5PT4eI\n4KmnnmqzGu3N2dkZKpUKd+7cabDN3r17lUe/N9fNmzfx8ssv11melZWF6upq9O3bF8D9p7JERETg\n3LlzSE5ObvWEjanXfVDtvndxcWlVbSIioqZgljKcts5RbdFPYywtLTF58mRkZGTo3TQ8NTUVKpWq\n3vskMceYL04WEVGHotFosGjRIuzevRvbtm1DcXExzp07h/nz58PV1RVz585V2np7e6OoqAjJycnQ\n6XQoKCioc6QEALp3747c3FxcuXIFJSUlSmCpqanBrVu3UFVVhbNnz2LBggVwd3dHaGioQWqkpqYa\n/XGvWq0Wnp6euH79er2vX7p0CS4uLvXenDkkJAQuLi44depUg/3b2tri4MGDOHz4MIqLi6HT6fD9\n99/jd7/7HWxtbbFw4UIAwA8//IA1a9bgo48+glqthkql0vtZt26dWdV9UO2+Hzp0aIP9EhERGQqz\nlOG0dY4ydD9NERkZiby8PCxbtgxlZWU4duwY1q5di9DQUAwcOLBOe+YY88XJIiLqcJYtW4bY2FhE\nR0ejR48eGDduHPr374/09HTY2toq7V5//XU8++yzeOmllzBw4ECsWLFCOUXW19dXeWzr/Pnz4ezs\nDB8fH0yePBlFRUUA7l97PXToUNjY2GDs2LEYMGAAvv76a73r0ltbwxT4+/sjMzOz3idciEiD61VW\nViI/Px8pKSkNttFoNBg9ejRmz54NNzc32NnZITAwEP3798fx48fx6KOPPrSOOdZ90DfffAM3N7d6\nT+0mIiJqC8xShtOWOcqQ/Rw/fhxjxoxB7969ceLECZw5cwaurq4YPXo0MjIylHZDhgxBWloaDh48\nCCcnJ8ycOROvvfYa3n///Xr7ZY4xY0LtKjExUbjbifQBkMTERGMPQ8/cuXOle/fuxh5GgwICAiQg\nIKBZ68ydO1fc3NzqLM/KyhJLS0v59NNPm9VfdXW1jB07VhISEpq1XmuZU93CwkLRaDSybt26Oq+F\nh4eLk5NTs/s0xc8Tkani54XMgan+fWHqWaq5n39TzVHGykUibZNjTPX93Akl8cwiIqIGNHazwo6q\noqICaWlpyMrKUm5I6O3tjejoaERHR6O0tLRJ/VRXVyM5ORklJSUICQlpyyGbdd2oqCgMHz4cYWFh\nAO4fOczNzcXRo0eVm0kSERF1VOaWpUwtRxkrF9VijjFvnCwiIupEioqKMHHiRAwYMACvvfaasnzx\n4sUIDAxESEhIozdprJWeno5du3YhNTUVWq22LYdstnXXr1+P06dPY//+/VCr1QCAlJQUuLm5YezY\nsdi3b59B6hAREZFhmFqOMlYuAphjOgNOFhER/cKSJUuwdetW3LlzBx4eHti5c6exh2QQH3zwAURE\n+dm2bZve6zExMQgLC8OqVase2tf48ePx2WefoVevXm01XLOum5KSgnv37iE9PR3dunVTlk+bNk3v\nd1RYWGiQekRERO3JHLOUKeYoY+Ui5pjOwdLYAyAiMjWxsbGIjY019jCMYsKECZgwYYKxh2H2pk6d\niqlTpxp7GERERG2is2apzpKjmGM6B55ZRERERERERERECk4WERERERERERGRgpNFRERERERERESk\n4GQREREREREREREpeINrI0lKSjL2EIhMyrFjx4w9hA7l+vXrAPh/CRFRa/H7hzq62vcwM0Hz8fNv\nevg7MR0qERFjD6IzSUpKQnBwsLGHQUREZiwxMRFBQUHGHgaRyVOpVMYeAhER1YPTFEa3g5NFRGTS\nav/g5dEyIiIiovoxLxGRge3gPYuIiIiIiIiIiEjBySIiIiIiIiIiIlJwsoiIiIiIiIiIiBScLCIi\nIiIiIiIiIgUni4iIiIiIiIiISMHJIiIiIiIiIiIiUnCyiIiIiIiIiIiIFJwsIiIiIiIiIiIiBSeL\niIiIiIiIiIhIwckiIiIiIiIiIiJScLKIiIiIiIiIiIgUnCwiIiIiIiIiIiIFJ4uIiIiIiIiIiEjB\nySIiIiIiIiIiIlJwsoiIiIiIiIiIiBScLCIiIiIiIiIiIgUni4iIiIiIiIiISMHJIiIiIiIiIiIi\nUnCyiIiIiIiIiIiIFJwsIiIiIiIiIiIiBSeLiIiIiIiIiIhIwckiIiIiIiIiIiJScLKIiIiIiIiI\niIgUnCwiIiIiIiIiIiIFJ4uIiIiIiIiIiEjBySIiIiIiIiIiIlJwsoiIiIiIiIiIiBScLCIiIiIi\nIiIiIgUni4iIiIiIiIiISMHJIiIiIiIiIiIiUnCyiIiIiIiIiIiIFJwsIiIiIiIiIiIiBSeLiIiI\niIiIiIhIoRIRMfYgiIgA4LPPPkNCQgJqamqUZT/99BMAwMPDQ1nWpUsX/P73v8esWbPafYxERERE\nxsS8RETtYAcni4jIZJw9exbDhg1rUtszZ87gsccea+MREREREZkW5iUiagecLCIi0zJo0CBcuHCh\n0Tbe3t7IyspqpxERERERmRbmJSJqYzt4zyIiMim/+c1voFarG3xdrVbj1VdfbccREREREZkW5iUi\nams8s4iITMrly5fh7e2Nxv5rysrKgre3dzuOioiIiMh0MC8RURvjmUVEZFo8PT3x+OOPQ6VS1XlN\npVJh5MiRDD5ERETUqTEvEVFb42QREZmc3/72t7CwsKiz3MLCAr/97W+NMCIiIiIi08K8RERtiZeh\nEZHJyc/Ph6urq94jYYH7j4DNzc2Fi4uLkUZGREREZBqYl4ioDfEyNCIyPc7Ozhg3bpze0TILCws8\n88wzDD5EREREYF4iorbFySIiMkm/+c1v6ty08Te/+Y2RRkNERERkepiXiKit8DI0IjJJxcXF6Nmz\nJyorKwHcfwRsfn4+HB0djTwyIiIiItPAvEREbYSXoRGRabK3t8fEiRNhaWkJS0tLTJ48mcGHiIiI\n6AHMS0TUVjhZREQm65VXXkF1dTWqq6sxa9YsYw+HiIiIyOQwLxFRW+BlaERksu7evYsePXpARFBY\nWAgbGxtjD4mIiIjIpDAvEVEb2GFp7BFQ20hKSkJwcLCxh0FkMFqt1thDIGq1xMREBAUFGXsYRGQg\nzFtkapiXyBQw75gHThaZucTERGMPgTqR4OBgLFiwAL6+vgbr8/Tp01CpVBg2bJjB+jQlGzZsAAC8\n9dZbRh4JtTX+QUlkvpi3qD3Vl7fMPS+1FvNW+2HeMR+cLDJznNGl9hQcHAxfX1+Dvu9mzJgBALC0\nNM//rnbs2AGAn9XOgOGJyHzx/3BqT/XlLXPPS63FvNV+mHfMB/83ISKTxtBDRERE1DjmJSIyND4N\njYiIiIiIiIiIFJwsIiIiIiIiIiIiBSeLiIiIiIiIiIhIwckiIiIiIiIiIiJScLKIiEzO/v374eDg\ngL179xp7KCbv0KFDWLx4MWpqajB9+nS4u7tDo9HAzc0NU6dOxdmzZ5vd5+rVqzFo0CDY2NjA1tYW\ngwYNQmRkJIqLi/XaRUdHw8fHB/b29rC2toa3tzfeeecdlJaWtmhbTLnunj17sHr1alRXV7eoBhER\nkalh3mo6Q+ctQ+a2B/vcsGED/Pz8Gmxz9OhRjB49GlqtFq6uroiIiMC9e/eU15l36EGcLCIikyMi\nxh5Ch7Bs2TJs3LgRS5YsQU1NDY4cOYLPP/8cRUVFOHr0KCoqKvD0008jNze3Wf0eOXIEc+bMwdWr\nV5GXl4cVK1Zg9erVCAgI0Gt3+PBhvPnmm7hy5QoKCwsRGxuLuLg4BAYGtmh7TLnulClToNFoMH78\neNy+fbtFdYiIiEwJ81bTtEXeMmRuA4CsrCw8/fTTWLhwIcrLy+ttk5mZiQkTJmD8+PEoKCjA7t27\n8cknn2D+/PlKG+Yd0iNklhITE4W/XmpvACQxMdHYwzCo8vJy8fX1bbP+AwICJCAgoNnrrVq1SgYM\nGCAVFRUiIqLT6eSFF17Qa3Py5EkBIDExMc3qe/r06Uq/tQIDAwWA5ObmKsv8/f2lqqpKr11QUJAA\nkKtXrzarZkeoKyISFhYmvr6+otPpml3HHD8fRJ0d8xYZgzl+n3S2vGXI3Hb69GmZMWOGbNu2TYYP\nHy7Dhg2rt11wcLB4eHhITU2Nsmzt2rWiUqnkxx9/1GvLvEMiksQzi4iIGpGQkID8/HxjD0PPpUuX\nEBkZieXLl0Oj0QAALC0t65xG7unpCQDIzs5uVv+7d+9W+q3l5uYGAHqXen355ZewsLDQa9ejRw8A\naPCoVkeuCwBRUVE4ffo04uLiml2HiIiI6tfZ8pYhc9uwYcOwa9cuzJo1C9bW1vW2qaqqwr59+zBu\n3DioVCpl+aRJkyAiSElJ0WvPvEMAL0MjIhNz9OhRuLu7Q6VSYfPmzQCA+Ph42NraQqvVIiUlBZMm\nTYK9vT369OmDL774Qll348aN0Gg0cHZ2xrx58+Dq6gqNRgM/Pz+cOHFCaRcWFgYrKyv06tVLWfbG\nG2/A1tYWKpUKhYWFAIAFCxZg0aJFyM7Ohkqlgre3NwDgwIEDsLe3R0xMTHvskjo2btwIEcGUKVMa\nbVdRUQEAsLe3b3XNrKwsODo6ol+/fo22y8nJgY2NDTw8PFpd0xTrduvWDePGjUNcXBxP3yciog6L\neevh2jtvGTK3/dLly5dRWloKd3d3veVeXl4AUOdeScw7BHCyiIhMzJgxY/DPf/5Tb9nrr7+Ot956\nCxUVFbCzs0NiYiKys7Ph6emJOXPmQKfTAbgfSkJDQ1FeXo7w8HBcuXIFp06dQlVVFZ577jlcu3YN\nwP0v/6CgIL0aW7ZswfLly/WWxcXF4cUXX4SXlxdEBJcuXQIA5aZ/NTU1bbIPHmbfvn0YOHAgtFpt\no+1OnjwJ4P4+bQmdToecnBxs3rwZhw4dwqZNm2BlZdVg+/Lychw+fBhz5sxptF1HrztixAjk5OTg\nzJkzLa5FRERkTMxbD9deecvQ/dTnxo0bAAA7Ozu95RqNBjY2NsjLy6uzDvMOWRp7AEREzeHn56ec\nChwSEoIjR47g6tWrypER4P6pvYMHDwYA+Pj4ID4+HqNGjcLWrVvx7rvvtnoM/v7+dZ7Q1V7Kysrw\n008/4YUXXmiwTV5eHr7++mu8/fbb8PX1fegRsYb07dsXeXl5cHJywpo1axAcHNxo+9jYWLi6umLl\nypUtqtdR6j7yyCMAgHPnzmH48OGtqklERGSKmLfaL28Zqp/G1D7x7JeX8QOAWq1Wzmp6EPMO8cwi\nIuqwas/6qD3S1ZCRI0dCq9Xi/Pnz7TGsNpWfnw8RafQol6+vL8LDwzFt2jSkpqZCrVa3qNa1a9eQ\nn5+Pzz//HH/7298wYsSIBu8nsHv3biQlJSEtLa3OUStzq1u77+s7CkdERGRumLfqZ6i8Zah+GlM7\n8VdVVVXntcrKStjY2NRZzrxDPLOIiDoFa2trFBQUGHsYrXb37l0AaPAGhgDg7OyMhIQEDBkypFW1\n1Go1evbsiQkTJsDDwwMDBgxQHlP/oO3bt2P9+vVIT09H7969W1WzI9StDVS1vwsiIiK6j3mr+QzV\nT2Nq7xv1yzO1ysvLcffuXbi6utZZh3mHOFlERGZPp9Ph9u3b6NOnj7GH0mq1X9y11/HXp2fPnnB0\ndDRoXW9vb1hYWCAzM1Nv+aZNm5CWlobDhw+ja9euBq1pinWB+0fgANR7FI6IiKizYt5qmbbIbb/k\n4eEBOzs7/Pzzz3rLa+8P9dhjj9VZh3mHeBkaEZm99PR0iAieeuopZZmlpeVDT6c2Rc7OzlCpVLhz\n506Dbfbu3as8+r25bt68iZdffrnO8qysLFRXV6Nv374AABFBREQEzp07h+Tk5FZP2Jh63QfV7nsX\nF5dW1SYiIjInzFstY6h+GmNpaYnJkycjIyND74bhqampUKlU9d4niXmHOFlERGanpqYGt27dQlVV\nFc6ePYsFCxbA3d0doaGhShtvb28UFRUhOTkZOp0OBQUFdY62AED37t2Rm5uLK1euoKSkBDqdDqmp\nqUZ7lKtWq4WnpyeuX79e7+uXLl2Ci4tLvTdnDgkJgYuLC06dOtVg/7a2tjh48CAOHz6M4uJi6HQ6\nfP/99/jd734HW1tbLFy4EADwww8/YM2aNfjoo4+gVquhUqn0ftatW2dWdR9Uu++HDh3aYL9ERETm\njnmr5XnL0P00RWRkJPLy8rBs2TKUlZXh2LFjWLt2LUJDQzFw4MA67Zl3iJNFRGRSNm/ejFGjRgEA\nIiIiMHXqVMTHx2PDhg0A7p8me/nyZXz00UdYtGgRAGDixInIyspS+rh79y6GDh0KGxsbjB07FgMG\nDMDXX3+td93566+/jmeffRYvvfQSBg4ciBUrViin2fr6+iqPfZ0/fz6cnZ3h4+ODyZMno6ioqF32\nQ2P8/f2RmZlZ75MrRKTB9SorK5Gfn4+UlJQG22g0GowePRqzZ8+Gm5sb7OzsEBgYiP79++P48eN4\n9NFHH1rHHOs+6JtvvoGbm1u9p2wTERF1BMxbD9eWecuQ/Rw/fhxjxoxB7969ceLECZw5cwaurq4Y\nPXo0MjIylHZDhgxBWloaDh48CCcnJ8ycOROvvfYa3n///Xr7Zd4hCJmlxMRE4a+X2hsASUxMNOoY\n5s6dK927dzfqGJojICBAAgICmrVOVlaWWFpayqefftqs9aqrq2Xs2LGSkJDQrPVay5zqFhYWikaj\nkXXr1jV7XVP4fBCRYTFvkTGYwvcJ81bDDJU/jJWfRJh3SEREknhmERGZncZuRmgOvL29ER0djejo\naJSWljZpnerqaiQnJ6OkpAQhISFtPELzrRsVFYXhw4cjLCzMYH0SERF1RMxbdRkqfxgrP9Vi3iGA\nl6FRBxAdHQ0fHx/Y29vD2toa3t7eeOeddx76n/bs2bNhZ2cHlUqF06dPt7j+hQsX8Ic//AFDhgyB\nnZ0dLC0t4eDggAEDBsDf3x/Hjh1rcd+G0pR9tGvXLnh6eta5x4uVlRWcnZ3xzDPPYO3atbh165YR\nt4SaavHixQgMDERISEijN1+slZ6ejl27diE1NRVarbYdRmh+ddevX4/Tp09j//79UKvVBumTiMjU\n1NTUYMOGDfDz82tS+86Ut2o1to+Yt8yLsfKWsfITwLxDDzD2uU3UNszptOhx48bJli1b5ObNm1Jc\nXCyJiYmiVqtl4sSJD133iy++EADy/ffft6j2xx9/LGq1Wp5++mk5cOCA3Lp1S+7evSvZ2dmyfft2\n8fPzk7/85S8t6tuQmrOPvLy8xMHBQUREampq5NatW/L1119LaGioqFQqcXV1lW+++aZF44CRTztd\nvHixWFlZCQDp37+/7Nixw2hjaaqWnBb9oLS0NImIiDDgiKg+ycnJEhsbK1VVVS3uw9ifDyIyPHPK\nWyIiFy9elNGjRwsAGTZsWJPX6yx5S6Tp+4h5y7QwbzUN8w49IMnSOFNURE3XtWtXzJ07FxYWFgCA\noKAg7Nq1C0lJSbh27Vq9j7Y2hOPHj2Pu3LkYN24c0tLSYGn5n4+Lp6cnPD094ejoqHejP2Np6T5S\nqVRwdHTEM888g2eeeQb+/v4IDg6Gv78/Ll68CAcHh/bcjFaLjY1FbGyssYfRriZMmIAJEyYYexhm\nb+rUqZg6daqxh0FE1GbOnDmD6OhozJ8/H2VlZc16oEBrdKS81dJ9xLzV8XWWvMW8Qw/iZWhk8r78\n8ktlEqRWjx49AADl5eWNrqtSqVpcd+XKlaiursaqVav0gsuDnn/+ebz55pstrmEordlHDwoICEBo\naCjy8/PxwQcfGHSMREREZLqGDRuGXbt2YdasWXpPs2qKzpK3WrOPHsS8RUQdASeLSM+nn36KkSNH\nQqPRwNbWFv3798eKFSsA3H+04/r16zF48GBYW1ujW7dumDZtGs6fP6+sHx8fD1tbW2i1WqSkpGDS\npEmwt7dHnz598MUXXyjtBg8eDJVKhS5duuCJJ55QJjTeeecdODg4QKPR4K9//WuD48zJyYGNjQ08\nPDyUZSKCtWvXYuDAgbC2toaDgwPefvvtOuseOHAA9vb2iImJabD/yspKfPXVV3BycsKTTz7Z5P1n\n6vuoKUJDQwEAqampzVqPiIiImqaj5K36MG8ZBvMWEZk8410CR22pJdfQb9iwQQDIqlWr5ObNm1JU\nVCR/+ctfZNasWSIi8u6774qVlZV8+umncvv2bTl79qw8/vjj0qNHD7lx44bSz9KlSwWAfPXVV3Ln\nzh3Jz8+XsWPHiq2trVRWVoqISFVVlfTv31/c3d3rXBP71ltvyYYNGxocZ1lZmdjZ2UlYWJje8qVL\nl4pKpZL/+Z//kVu3bkl5ebls2bKlzjX0X375pdjZ2Ul0dHSDNS5evCgA5Kmnnmr6DuwA+0hE/xr6\n+hQXFwsA6du3b7O2XYTXKLdEa6+hp46Dnw8i82Oueeu//uu/GrwfD/PWfY3tIxHmLVPDvNV++P40\nG0mcLDJTzQ0vlZWV4ujoKM8++6ze8qqqKomLi5Py8nLp2rWrhISE6L1+8uRJAaAXBGq/mCsqKpRl\ntSHi0qVLyrLasJSUlKQsKysrE3d3d7lz506DY126dKkMGDBAiouLlWXl5eWi1Wrlueee02vb0hsu\nfvvttwJAfv3rXzd5HVPfR7UeFl5ERFQqlTg6Ojbapj78cmg+hpfOg58PIvNjrnmroYkQ5q3/aO1k\nkQjzVnti3mo/fH+aDd7gmu47e/Ysbt++jeeff15vuYWFBcLDw/Htt9+itLQUI0eO1Ht91KhRsLKy\nwokTJxrt38rKCgCg0+mUZbNnz0ZUVBTi4uIQGBgIANi2bRumTZsGe3v7evvZvXs3kpKScPDgQdjZ\n2SnLL126hPLycowfP77pG92Irl27Amje/X4yMzNNeh81Ve0NGxvq/2FM6dG2HcH169cBAElJSUYe\nCRERtbWOkrcawrxlOMxb7Yt5i6j5OFlEAIDi4mIAgKOjY72v3759G8B/vtQf5OjoiJKSkmbX7Nq1\nK/77v/8ba9euxcmTJ/Hkk0/i/fffx86dO+ttv337dqxfvx7p6eno3bu33mu1XwA9e/Zs9jjq079/\nf2g0Gly8eLHJ65j6Pmqq2m0eNGhQi9aPi4tDXFxci9btzIKDg409BCIiamMdIW81hnnLcJi3jIN5\ni6jpeINrAgBlYqGwsLDe12tDTX1fwLdv30afPn1aVDcsLAxqtRobNmxARkYG+vbtCy8vrzrtNm3a\nhG3btuHw4cP1ToJoNBoAwL1791o0jl+ytrbG888/j8LCQvzjH/9osF1RURFmz54NwPT3UVMdOHAA\nADBp0qQWrZ+YmAgR4U8TfwICAhAQEGD0cfCn7X+IiEw9bz0M85bhMG+17w/zVvv9kPngZBEBuH9k\np3v37jh48GC9rz/66KPo2rUrvv32W73lJ06cQGVlJZ544okW1e3Tpw+CgoKwc+dOREZGYsGCBXqv\niwgiIiJw7tw5JCcn13sUqXZ8Xbp0wd///vcWjaM+UVFRsLa2xsKFC1FRUVFvm3/961/KY15NfR81\nxY0bN7Bhwwb06dMHr732Wov7ISIiorpMNW81VWfOW4bEvEVEHQEniwjA/SM7S5YsQUZGBsLCwpCT\nk4OamhqUlJTghx9+gEajwaJFi7B7925s27YNxcXFOHfuHObPnw9XV1fMnTu3xbUXLVqEqqoq3Lp1\nC7/61a/0Xvvhhx+wZs0afPTRR1Cr1VCpVHo/69atA3D/dOiZM2di586dSEhIQHFxMc6ePYsPP/yw\nTr3U1NSHPsoVAIYPH47PPvsM//rXvzB27Fjs378fd+7cgU6nw08//YSPPvoIv//976FWqwHA5PfR\ng0QEpaWlqKmpgYigoKAAiYmJGD16NCwsLJCcnNxm1+gTERF1Vqaat5qqM+etlmDeIqIOTcgsteRR\nriIimzdvlqFDh4pGoxGNRiMjRoyQLVu2iIhITU2NrF27Vh555BFRq9XSrVs3mT59uly4cEFZf8uW\nLaLVagWAPPLII5KdnS0ffvih2NvbCwDp16+fXLx4sU7dZ599Vj7++OM6y8+dOycAGvxZu3at0rak\npERmz54tTk5O0rVrVxkzZoy8++67AkD69OkjZ86cERGR/fv3i52dnaxcubJJ++Tq1avyxz/+UYYO\nHSpdu3YVCwsLcXR0lBEjRsjvf/97+cc//qG0NeV9tGfPHnnsscdEq9WKlZWVdOnSRQAoT+J48skn\nJTo6Wm7evNmk/VIf8OkHzcanc3Qe/HwQmR9zyVsiIseOHZPRo0eLq6urkiF69eolfn5+8ve//11p\n11nzVlP3EfOWaWLeaj98f5qNJJUILyw0R0lJSQgODuZ1o9SuVCoVEhMTERQUZOyhdBi1T13ZsWOH\nkUdCbY2fDyLzw7xFxsDvk+Zj3mo/fH+ajR28DI2IiIiIiIiIiBScLCIiIiIiIiIiIgUni4iIOrBD\nhw5h8eLFqKmpwfTp0+Hu7g6NRgM3NzdMnToVZ8+ebXafq1evxqBBg2BjYwNbW1sMGjQIkZGRKC4u\nrtP26NGjGD16NLRaLVxdXREREdHiRyqbct09e/Zg9erVqK6ublENIiIiMi+GzmDtneWYbehhOFlE\nRNRBLVu2DBs3bsSSJUtQU1ODI0eO4PPPP0dRURGOHj2KiooKPP3008jNzW1Wv0eOHMGcOXNw9epV\n5OXlYcWKFVi9ejUCAgL02mVmZmLChAkYP348CgoKsHv3bnzyySeYP39+i7bHlOtOmTIFGo0G48eP\nx+3bt1tUh4iIiMxDW2Sw9s5yzDb0MJwsIiKzUVFRAT8/vw5foynee+89bN++HUlJSbCzswMA+Pr6\nYsyYMdBqtfDw8EBMTAzu3LmDv/71r83q28rKCm+88QZ69uyJrl2707Z7igAAIABJREFUIjAwENOm\nTcP//d//4d///rfSbsWKFejVqxeWL18OW1tb+Pr6IiIiAn/9619x/vz5Zm+TqdcNDw/HsGHDMHny\nZFRVVTW7DhERkTnoTHmrPm2Zwdo7yzHbUGM4WUREZiMhIQH5+fkdvsbDXLp0CZGRkVi+fDk0Gg0A\nwNLSEnv37tVr5+npCQDIzs5uVv+7d+9W+q3l5uYGACgtLQUAVFVVYd++fRg3bhxUKpXSbtKkSRAR\npKSkNG+jTLxuraioKJw+fRpxcXHNrkNERGQOOkveqk9bZrD2znK1mG2oIZwsIiKjERGsX78egwcP\nhrW1Nbp164Zp06bpnR0SFhYGKysr9OrVS1n2xhtvwNbWFiqVCoWFhQCABQsWYNGiRcjOzoZKpYK3\ntzc2btwIjUYDZ2dnzJs3D66urtBoNPDz88OJEycMUgMADhw4AHt7e8TExLTp/qq1ceNGiAimTJnS\naLuKigoAgL29fatrZmVlwdHREf369QMAXL58GaWlpXB3d9dr5+XlBQAtur7elOvW6tatG8aNG4e4\nuDg+KpuIiDoE5i3Dae8M1pZZrhazDTWEk0VEZDRRUVFYvHgxli5divz8fGRkZODatWsYO3Ys8vLy\nANz/Ug4KCtJbb8uWLVi+fLnesri4OLz44ovw8vKCiODSpUsICwtDaGgoysvLER4ejitXruDUqVOo\nqqrCc889h2vXrrW6BgDlxoA1NTWG2zmN2LdvHwYOHAitVttou5MnTwIAxowZ06I6Op0OOTk52Lx5\nMw4dOoRNmzbBysoKAHDjxg0AUE6/rqXRaGBjY6P8/syl7oNGjBiBnJwcnDlzpsW1iIiI2gvzluG0\nVwYzVD/MNtQanCwiIqOoqKjA+vXrMWPGDLzyyitwcHDA0KFD8cEHH6CwsBAffvihwWpZWloqR9N8\nfHwQHx+PkpISbN261SD9+/v7o7i4GJGRkQbprzFlZWX46aeflDNp6pOXl4ft27cjPDwcvr6+Dz36\n1ZC+ffuiT58+iIqKwpo1axAcHKy8VvvkMQsLizrrqdVq5UiYudR90COPPAIAOHfuXItrERERtQfm\nLcNpzwzWHlnuQcw2VB9OFhGRUWRmZqK0tBQjR47UWz5q1ChYWVnpnbZsaCNHjoRWq23RzZCNLT8/\nHyLy/9i78/ioqvv/4+8hC5OdLQk0IUBAQBSFIi1h+QpFeQjIZgkEEEypstkCSltL+0MQSyxqkS+L\nWiyyRBEI0CBqlCJQccEFRGjcAAFZKgkESIBAJpnz+4PHzJeYhUwyk5kkr+fjweOhd86953NvzmQ+\n+cw955b7jVZCQoKmTZumoUOHKiMjQwEBAZXq6/jx48rKytKaNWu0atUqde7c2bl+gGMefGmLIRYU\nFCgoKKhSffpqv9dzXPuq3MUEAEB1IN9yn+rMwaojl7seuQ1KQ7EIgFc4HtEZGhpa4rUGDRooLy/P\no/3Xr19f2dnZHu3DE65cuSLpWvxliYqK0vbt27V48WJFRERUuq+AgABFRkaqX79+Wrt2rTIzM5WS\nkiJJzvUGcnNzi+1z+fJlXblyRc2aNatV/V7PUZBy/CwAAPBV5FvuU505WHXkctcjt0FpKBYB8IoG\nDRpIUqlJyvnz5xUbG+uxvm02m8f78BTHh7lj3n5pIiMjndfXXdq0aSM/Pz9lZmZKklq1aqWwsDAd\nO3asWDvHugK33XZbrer3egUFBZJUpbuYAACoDuRb7lOdOVh15HLXI7dBaSgWAfCKW2+9VaGhofrs\ns8+Kbf/4449VUFCgLl26OLf5+/vLZrO5re+dO3fKGKNu3bp5rA9PiYqKksVi0YULF8pss2XLFufj\nUV119uxZjR49usT2gwcPqqioSM2bN5d07XoNGDBA7733XrGFJjMyMmSxWFyeW+/r/V7Pce2jo6Nd\n6gsAgOpGvuU+ns7B3HUcchu4C8UiAF5htVo1Y8YMbdq0Sa+88opyc3N14MABTZ48Wc2aNdPEiROd\nbdu0aaOcnBylp6fLZrMpOzu7xJ0lktSoUSOdOnVKR48eVV5enjMZsdvtOnfunAoLC7V//35Nnz5d\ncXFxSk5OdksfGRkZ1fYo1+DgYMXHx+vEiROlvn7o0CFFR0eXuoBhUlKSoqOjtXfv3jKPHxISoq1b\nt2r79u3Kzc2VzWbT559/rgceeEAhISF69NFHnW1nzZql06dPa/bs2bp06ZI++ugjPfPMM0pOTla7\ndu1qXb8OjmvfsWPHMo8LAIAvIN9yH0/nYO46DrkN3IViEQCvmT17tlJSUjR37lw1adJEd955p1q2\nbKmdO3cqJCTE2W7KlCnq06ePRo0apXbt2unJJ5903iabkJDgfCTr5MmTFRUVpQ4dOmjAgAHKycmR\ndG3+dceOHRUUFKRevXqpbdu22rFjR7E551XtozoNHDhQmZmZpT75yxhT5n4FBQXKysrS5s2by2xj\ntVrVo0cPPfjgg4qJiVFYWJgSExPVsmVL7d69W7feequz7S233KJ33nlHW7duVePGjfXLX/5S48eP\n1wsvvFAr+3X49NNPFRMT47YpbwAAeBL5lvt4Mgdz13HIbeA2BrXSunXrDD9eVDdJZt26dd4Oo5iJ\nEyeaRo0aeTuMMg0fPtwMHz7cpX0OHjxo/P39TWpqqkv7FRUVmV69epnly5e7tF9V1aZ+z5w5Y6xW\nq3n22Wdd3tcX3x8AqoZ8C97gi58ntTHfKo23czBfy21K44vjE5WynjuLANR65S1EWBO1adNGc+fO\n1dy5c3Xx4sUK7VNUVKT09HTl5eUpKSnJwxHW3n7nzJmjTp06aerUqW47JgAAtUFty7dK480cjNwG\n1Y1iEQDUQDNnzlRiYqKSkpLKXWjRYefOndq4caMyMjIUHBxcDRHWvn4XLFigffv26a233lJAQIBb\njgkAAGoWb+Vg5DaobhSLANRaf/rTn7RixQpduHBBrVq10oYNG7wdklvNmzdPU6dO1VNPPXXDtn37\n9tWrr76qpk2bVkNkta/fzZs36+rVq9q5c6caNmzolmMCAFAb1PZ8qzTeyMHIbVDd/L0dAAB4SkpK\nilJSUrwdhkf169dP/fr183YYtd6QIUM0ZMgQb4cBAIDPqQv5Vmlqeg5GboMb4c4iAAAAAAAAOFEs\nAgAAAAAAgBPFIgAAAAAAADhRLAIAAAAAAIATC1zXcomJid4OAXXMc889p7S0NG+HUWPs3r1bEu9V\nAKjJ+B2O6ka+5RryLcB1FmOM8XYQcL+PPvpICxYs8HYYQJV9/vnnkqTOnTt7ORKg6h599FElJCR4\nOwwAbkK+BV9BvgRfQr5TK6RRLALg00aMGCFJWr9+vZcjAQAA8E3kSwDcLI01iwAAAAAAAOBEsQgA\nAAAAAABOFIsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAAAE4UiwAAAAAAAOBEsQgA\nAAAAAABOFIsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAAAE4UiwAAAAAAAOBEsQgA\nAAAAAABOFIsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAAAE4UiwAAAAAAAOBEsQgA\nAAAAAABOFIsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAAAE4UiwAAAAAAAOBEsQgA\nAAAAAABOFIsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAAAE4UiwAAAAAAAOBEsQgA\nAAAAAABO/t4OAAAcLl++rKtXrxbbVlBQIEk6d+5cse3169dXcHBwtcUGAADgC8iXAFQHikUAfMbK\nlSv18MMPl/pao0aNiv3/0qVLNWXKlOoICwAAwGeQLwGoDhZjjPF2EAAgSdnZ2WrWrJmKiorKbefn\n56f//ve/ioyMrKbIAAAAfAP5EoBqkMaaRQB8RmRkpPr27Ss/P78y2/j5+emuu+4i8QEAAHUS+RKA\n6kCxCIBPuf/++1XeDY/GGN1///3VGBEAAIBvIV8C4GlMQwPgU/Ly8hQZGVli4UaHwMBAZWdnKzw8\nvJojAwAA8A3kSwA8jGloAHxLWFiYBg0apICAgBKv+fv7a8iQISQ+AACgTiNfAuBpFIsA+JwxY8ao\nsLCwxPaioiKNGTPGCxEBAAD4FvIlAJ7ENDQAPqegoEBNmjRRXl5ese2hoaE6c+aM6tev76XIAAAA\nfAP5EgAPYhoaAN8TGBioxMREBQYGOrcFBARo5MiRJD4AAAAiXwLgWRSLAPik0aNHq6CgwPn/NptN\no0eP9mJEAAAAvoV8CYCnMA0NgE+y2+1q2rSpsrOzJUlNmjTRDz/8ID8/Py9HBgAA4BvIlwB4CNPQ\nAPimevXqafTo0QoMDFRAQIDGjBlD4gMAAHAd8iUAnkKxCIDPGjVqlAoKCrilGgAAoAzkSwA8wf/H\nG06cOKEPP/zQG7EAQDHGGDVu3FiSdOTIER09etS7AQGApO7duys2NtYjx16/fr1Hjgug9iJfAlBV\nI0aMKLGtxJpF69ev18iRI6stKAAAgJpk3bp1pSZV7mCxWDxyXAAAgLKUspR1Wok7i8ppDADV7ssv\nv5QkdejQwcuR1B0Wi8WjfwzXRomJiZKktLQ0L0cCT6uOYg7vPwCuIl+qHcgnXOe42YX6ReWUd7NQ\nmcUiAPAFJD0AAADlI18C4G4scA0AAAAAAAAnikUAAAAAAABwolgEAAAAAAAAJ4pFAAAAAAAAcKJY\nBAAAAAAAACeKRQAAj3jrrbcUERGhLVu2eDsUn7dt2zbNnDlTdrtdw4YNU1xcnKxWq2JiYjRkyBDt\n37/f5WPOnz9f7du3V1BQkEJCQtS+fXvNmjVLubm5Jdq+//776tGjh4KDg9WsWTM99thjunr1aqXO\nxZf7ff311zV//nwVFRVVqg8AAGoCcrCKc3cOVt25nCdzG4pFAACPMMZ4O4QaYfbs2Vq0aJH+9Kc/\nyW63a9euXVqzZo1ycnL0/vvvKz8/X//zP/+jU6dOuXTcXbt26aGHHtL333+v06dP68knn9T8+fM1\nfPjwYu0yMzPVr18/9e3bV9nZ2dq0aZNefvllTZ48uVLn48v9Dh48WFarVX379tX58+cr1Q8AAL6O\nHKxiPJGDVXcu59HcxvzIunXrTCmbAQB1hCSzbt06b4fhVpcvXzYJCQkeO/7w4cPN8OHDXd7vqaee\nMm3btjX5+fnGGGNsNpu59957i7X55JNPjCQzb948l449bNgw53EdEhMTjSRz6tQp57aRI0eaVq1a\nGbvd7tz2zDPPGIvFYr766itXT8nn+zXGmKlTp5qEhARjs9lc7sfT74/a+P4DAFRMZfMJX+bpHKyy\n9QtP5WDeyOWMqXxuU871W8+dRQCAWm/58uXKysrydhjFHDp0SLNmzdITTzwhq9UqSfL39y9xy3h8\nfLwk6fDhwy4df9OmTc7jOsTExEiSLl68KEkqLCzUm2++qTvvvFMWi8XZrn///jLGaPPmza6dlI/3\n6zBnzhzt27dPCxcudLkfAABQcXUtB6vuXM7BE7kNxSIAgNu9//77iouLk8Vi0ZIlSyRJzz//vEJC\nQhQcHKzNmzerf//+Cg8PV2xsrF577TXnvosWLZLValVUVJQmTZqkZs2ayWq1qnv37vr444+d7aZO\nnarAwEA1bdrUue3hhx9WSEiILBaLzpw5I0maPn26ZsyYocOHD8tisahNmzaSpLffflvh4eGaN29e\ndVySEhYtWiRjjAYPHlxuu/z8fElSeHh4lfs8ePCgGjRooBYtWkiSvvvuO128eFFxcXHF2rVu3VqS\nKjW/3pf7dWjYsKHuvPNOLVy4kFv1AQC1CjnYjVV3DubJXM7BE7kNxSIAgNv17NlTH374YbFtU6ZM\n0SOPPKL8/HyFhYVp3bp1Onz4sOLj4/XQQw/JZrNJupaAJCcn6/Lly5o2bZqOHj2qvXv3qrCwUHff\nfbeOHz8u6doH/YgRI4r1sXTpUj3xxBPFti1cuFCDBg1S69atZYzRoUOHJMm5EKDdbvfINbiRN998\nU+3atVNwcHC57T755BNJ165pZdhsNp08eVJLlizRtm3btHjxYgUGBkqSfvjhB0lSWFhYsX2sVquC\ngoJ0+vTpSvXpq/1er3Pnzjp58qS++OKLSvcFAICvIQe7serKwdx1HG/lNv5uOQoAAC7o3r2787ba\npKQk7dq1S99//73zzhLp2m28N998sySpQ4cOev7559W1a1etWLFCjz/+eJVjGDhwYKlP6KoOly5d\n0pEjR3TvvfeW2eb06dPasWOHfv/73yshIeGG336VpXnz5jp9+rQaN26sp59+WiNHjnS+5njymJ+f\nX4n9AgICnN+E1ZZ+r3fTTTdJkg4cOKBOnTpVuj8AAGoScrDqy8GqI5e7nrtzG+4sAgB4leObEce3\nWmW54447FBwcrK+//ro6wvKorKwsGWPK/UYrISFB06ZN09ChQ5WRkaGAgIBK9XX8+HFlZWVpzZo1\nWrVqlTp37uxcO8CRLBYWFpbYr6CgQEFBQZXq01f7vZ7j2lflLiYAAGoycrDSuSsHq45c7nruzm0o\nFgEAaoz69esrOzvb22FU2ZUrVyRdO5+yREVFafv27Vq8eLEiIiIq3VdAQIAiIyPVr18/rV27VpmZ\nmUpJSZEk51oDP/527/Lly7py5YqaNWtWq/q9nqMg5fhZAACAspGDua46crnruTu3oVgEAKgRbDab\nzp8/r9jYWG+HUmWOD3PHnP3SREZGqkGDBm7tt02bNvLz81NmZqYkqVWrVgoLC9OxY8eKtXOsKXDb\nbbfVqn6vV1BQIElVuosJAIC6gByscqojl7ueu3MbikUAgBph586dMsaoW7duzm3+/v43vHXaF0VF\nRclisejChQtlttmyZYvz8aiuOnv2rEaPHl1i+8GDB1VUVKTmzZtLunb9BgwYoPfee6/YIpMZGRmy\nWCwuz6339X6v57j20dHRLvUFAEBdQw5WOdWRy13P3bkNxSIAgE+y2+06d+6cCgsLtX//fk2fPl1x\ncXFKTk52tmnTpo1ycnKUnp4um82m7OzsEnerSFKjRo106tQpHT16VHl5ebLZbMrIyPDaY1uDg4MV\nHx+vEydOlPr6oUOHFB0dXeoChklJSYqOjtbevXvLPH5ISIi2bt2q7du3Kzc3VzabTZ9//rkeeOAB\nhYSE6NFHH3W2nTVrlk6fPq3Zs2fr0qVL+uijj/TMM88oOTlZ7dq1q3X9OjiufceOHcs8LgAAdRE5\nWOVzMHcdxxdyG4pFAAC3W7Jkibp27SpJeuyxxzRkyBA9//zzeu655yRdm2b03Xff6aWXXtKMGTMk\nSffcc48OHjzoPMaVK1fUsWNHBQUFqVevXmrbtq127NhRbI75lClT1KdPH40aNUrt2rXTk08+6bz1\nNiEhwfmI18mTJysqKkodOnTQgAEDlJOTUy3XoTwDBw5UZmZmqU/+MsaUuV9BQYGysrK0efPmMttY\nrVb16NFDDz74oGJiYhQWFqbExES1bNlSu3fv1q233upse8stt+idd97R1q1b1bhxY/3yl7/U+PHj\n9cILL9TKfh0+/fRTxcTEuG3KGwAAvoAc7MY8mYO56zg+kduYH1m3bp0pZTMAoI6QZNatW+fVGCZO\nnGgaNWrk1RhcMXz4cDN8+HCX9jl48KDx9/c3qampLu1XVFRkevXqZZYvX+7SflVVm/o9c+aMsVqt\n5tlnn3V5X0+/P3zh/QcA8I7K5BPuVtNysMrUL7ydg/lSblPO9VvPnUUAAJ9U3sKDtUGbNm00d+5c\nzZ07VxcvXqzQPkVFRUpPT1deXp6SkpI8HGHt7XfOnDnq1KmTpk6d6rZjAgBQW5CDleSunKQm5TbV\nVix69tlnnYtJvfjiiy7t27VrV/n5+alTp05eiwE3Vtb1feuttxQREaEtW7Z4rO+5c+eqQ4cOCg8P\nV/369dWmTRv94Q9/uOGb/8EHH1RYWJgsFov27dtX6f6//fZb/fa3v9Utt9yi8PBwBQYGKjIyUu3b\nt9d9992nf/7zn862vvpe2Lhxo+Lj42WxWGSxWDRr1qxyj7FgwQJZLBbVq1dP7du313vvvVelmK73\n41gsFosCAgIUExOjMWPG6KuvvnJbXz/m6+O4tGtjsVgUGBioqKgo9e7dW88884zOnTvnsTjhPjNn\nzlRiYqKSkpLKXWjRYefOndq4caMyMjIUHBxcDRHWvn4XLFigffv26a233lJAQIBbjunrfPVzB+7j\nzc8uB7vdrueee07du3evUHtyMHKwH/P1cUwOVrt4KwerUbmNC7chVdnBgweNJPPCCy+4vG/fvn3N\n7bff7tUYcGOlXd833njDhIeHm9dff91j/d55551m6dKl5uzZsyY3N9esW7fOBAQEmHvuueeG+772\n2mtGkvn8888r1feKFStMYGCg6dmzp3n77bfNuXPnzJUrV8zhw4fNli1bzMCBA83EiROL7ePL74XW\nrVsbSaZp06amoKCg1H0LCwtNixYtjCTTt2/fKsdSltatW5uIiAhjjDEXL140r7/+uomLizOhoaHm\n66+/9li/NWEcX39t7Ha7OXfunNmxY4dJTk42FovFNGvWzHz66aeVikNengYzc+ZMExgYaCSZli1b\nmrS0NK/FUlFVvW38nXfeMY899pgbI0Jp0tPTTUpKiiksLKz0MTz9/vDU8X35cwfu4a3PLmOM+fbb\nb02PHj2MJJfGCjkYOdiP1YRx7MkczNvT0GpiDlbV+kVNz8GqmtuUNw3N331lJ8+zWCzeDgGVMHDg\nwApVa6siNDRUEydOlJ+fnyRpxIgR2rhxo9avX6/jx4+X+mhBd9i9e7cefPBB9erVS//617/k7/9/\nb6n4+HjFx8erQ4cOevrpp93ar6ffC126dNGePXuUnp6uxMTEEq9v3LhRMTExpT7xwFNCQkI0aNAg\nFRUVadiwYVq8eLGWLFlSbf378ji2WCxq0KCBevfurd69e2vgwIEaOXKkBg4cqG+//VYREREejdvd\nUlJSlJKS4u0wqlW/fv3Ur18/b4dR6w0ZMkRDhgzxdhg1EjlYzVQdn11ffPGF5s6dq8mTJ+vSpUvl\nLurqTuRg5GDuVNlxTA5W89X0HMyTuY3HpqEZY5SWlqZly5a57Zh15XZxlK+0sfXGG284/8B2aNKk\niSTp8uXL5R6vKh/68+bNU1FRkZ566qliScr14uPj3X7LvaffC1OmTJGkEk8lcliwYIHz6QnV7Wc/\n+5kk6T//+Y9X+ncXd4/j6w0fPlzJycnKyspiugdQB5GDwVNKG1u33367Nm7cqDFjxhR7UlJFkIOV\nRA7mee4ex9cjB0Nt4pZiUVFRkVJSUtSuXTsFBQWpSZMmatWqlVJSUjRixIhy9zXGaMGCBbr55ptV\nv359NWzYUEOHDtXXX39dou2hQ4fUvn17hYSEOB/j9/777xdrs2vXLnXo0EERERGyWq3q2LGj3nnn\nnSqf48KFCxUSEqJ69eqpS5cuio6OVkBAgEJCQvTTn/5UvXr1UvPmzWW1WtWgQQP94Q9/qHBcK1eu\nVGhoqCwWixo2bKj09HR99tlnatGihfz8/DR69GiXYl20aJGsVquioqI0adIkNWvWTFarVd27d9fH\nH39crG1Fr78rP6frvf/++4qLi5PFYnF+A/H8888rJCREwcHB2rx5s/r376/w8HDFxsbqtddeK7Z/\nVcbWyZMnFRQUpFatWhU7j2eeeUbt2rVT/fr1FRERod///vcl9n377bcVHh6uefPmlXn8goICbdu2\nTY0aNVK3bt3KjaUifOm98Itf/EI333yzduzYoW+++abYax988IEuX75cZgXe02O9sLBQkop9kNe1\ncVwRycnJkqSMjAyX9gNQs5CDkYOVxZufXaUhByMHqw3juCLIwVBruDBnrUzz5s0zfn5+ZvPmzeby\n5ctmz549Jjo62vTu3btYu9LmoD7++OMmMDDQpKammvPnz5v9+/ebn/70p6ZJkybmhx9+cLbr27ev\niY+PN0eOHDE2m8385z//MT//+c+N1Wo13377rbNdWlqamTNnjsnJyTFnz5413bp1M40bNy43hoqa\nPXu2kWQ+/vhjc+nSJXPmzBlzzz33GEnmzTffNNnZ2ebSpUtm6tSpRpLZt29fheP68ssvTXBwsHng\ngQec22bOnGn+8Y9/uBynMdceeRgSEmK+/PJLc+XKFZOZmWm6du1qwsLCzPfff+9sV9HrX9F2pV3f\n48ePG0lm8eLFzm1//vOfjSTz7rvvmgsXLpisrCzTq1cvExISUmyOdkXH1o9dunTJhIWFmalTpxbb\n/uc//9lYLBbzt7/9zZw7d85cvnzZLF26tMR8+TfeeMOEhYWZuXPnltnHt99+aySZbt26lRtLaXz5\nvdC6dWtz5MgR87//+79Gkpk+fXqx14cNG2ZWrFhh8vLySp0v786xfv2ccIfU1FQjyfz+9793+drV\nlnFc1rW5Xm5urpFkmjdvXm4fpRGP7naZt9cYQPXx9PvD1eOTg5GDOfjCZ9fPf/7zMtd6IQcjB6sN\n47isa3O9quRg5BOu8+Say3VBeWsWuaVY1LVrV/Ozn/2s2LYJEyaYevXqmatXrzq3/fjNf/nyZRMa\nGmqSkpKK7fvJJ58YScU+JEpbUG7//v1Gkvnd735XZmwpKSlGksnKyio1Blc4EpW8vDzntlWrVhlJ\n5sCBAyXiX7t2bYXjMsaYv//970aSeeWVV8yaNWvMo48+6nKMDhMnTizxS+zTTz81kswTTzxhjKn4\n9Xfl5+TqL/j8/HznNkfCcOjQIee2io6tH/vzn/9s2rZta3Jzc53bLl++bIKDg83dd99drG1lF1f8\n7LPPjCRz1113ubSfMb79XnAkKufPnzchISGmYcOG5vLly8YYYw4fPmxiY2PN1atXy0xUbtSvMRUf\n6z9eXHHDhg0mOjraREVFmRMnTrh07WrLOC7t2pTFYrGYBg0HfBaTAAAgAElEQVQalNumNBSLXEdy\nV3f4WrGIHIwczMEXPrvK+iObHIwcrDaM49KuTVkqm4ORT7iOYlHVeHyB6ytXrshqtRbbVlRUpICA\ngBLrb1wvMzNTFy9e1B133FFse9euXRUYGFjidt0f69ixoyIiIrR///4y2zjmFRcVFd3oNColMDBQ\n0v/dlnl9nzabzaW4JkyYoH/961+aNGmS7rrrLm3YsMGtsd5xxx0KDg523u5Z0etf1Z9TRTmu5fXX\nrTJja9OmTVq/fr22bt2qsLAw5/ZDhw7p8uXL6tu3r1viDQ0NlSRdunSp1NfXr1+vxx57TEePHpUk\ntW/fXv/+978VFRVVoq0vvhciIiI0evRovfTSS1q7dq1+9atf6bnnntOUKVMUGBiogoKCCh2nqmP9\nwoULslgs8vPzU9OmTTVgwADNnj1bMTExkureOK4ox+KM4eHhLu8rSc8995zS0tIqtW9dtHv3bkkq\ndTFSwJPIwcjB3MFdn11lIQcjB6sN47iiqpqD7d69m3zCBSdOnJBEDlZZjutXGresWTRgwADt2bNH\nmzdvVn5+vj777DOlp6fr3nvvLfdNeP78eUn/9wv/eg0aNFBeXt4N+w4ICCj2C+HNN99U7969FRkZ\nqfr165eYt+4tFY1r3rx5unjxorKysjwSR/369ZWdnS2p4tffHT+nynJ1bK1du1Z//etftXPnTrVs\n2bLYa443QmRkpFtia9GiherXr69Dhw6V+vqIESN05MgRtWjRQtHR0frqq69KTVIk330vOBZZfPHF\nF3X+/HmlpaVp0qRJ5e7j7rEeEREhY4wKCwt14sQJvfzyy2rRooXz9bo2jivq22+/lXQtQQZQe5GD\n3Rg5WOVUdmyVhhyMHKwi7TzBneO4osjBUFu45c6iOXPmaM+ePUpOTtbFixfVrFkzjRgxotyF6aRr\nvxwklfoL4vz584qNjS13/8LCQuXk5CguLk6S9P3332vYsGG677779PLLL+snP/mJFi9e7PVkpaJx\n2Ww2TZs2zfmUg7/85S+aPXu22+Kw2WzFrmtFr39Vf05V4crYWrx4sd555x1t37691A8jx7cKV69e\ndUtsVqtVd911l958803t3r27Sgss+up7oVOnTurWrZt2796tiRMnKjExUQ0bNiyzvTfGel0bxxX1\n9ttvS5L69+9fqf0feeQRjy38WBs5vs3ibqzaz9ceIU8OVj5ysMqr7NgqDTkYOVhF2nmCO8dxRVU1\nB+vWrRv5hAvWr1+vkSNHcs0qyXH9SuOWYlFmZqYOHz6s7OzsMh9dWZpbb71VoaGh+uyzz4pt//jj\nj1VQUKAuXbqUu/+OHTtkt9v105/+VJJ04MAB2Ww2TZkyRfHx8ZJ8I6mraFy//e1v9dBDD+m+++7T\nyZMn9eSTT6pfv35KSEhwSxw7d+6UMcb5gVrR61/Vn1NVVGRsGWP0xz/+UefOnVN6enqZ7W699VbV\nq1dP//73vzV58mS3xPfEE09o69at+v3vf6/t27dX+nGqvvxemDJlinbv3q0NGzbo4MGD5bb1xliv\na+O4In744Qc999xzio2N1fjx4yt9HAC+jxysfORglVfZsVUacjBysNowjiuCHAy1iVumof3mN79R\nXFycLl686NJ+VqtVM2bM0KZNm/TKK68oNzdXBw4c0OTJk9WsWTNNnDixWPuCggJduHBBhYWF2rt3\nr6ZOnaoWLVo4H0/oqOhv27ZNV65c0cGDB902B7YqKhLX0qVLFRMTo/vuu0+SlJKSog4dOmjMmDHK\nzc2tVL92u13nzp1TYWGh9u/fr+nTpysuLs55vSp6/V39OblTRcbWl19+qaefflovvfSSAgICZLFY\niv179tlnJV279fmXv/ylNmzYoOXLlys3N1f79+/XsmXLShwzIyPjho9tlaQuXbooNTVVe/bsUe/e\nvfX222/rv//9rwoLC3Xs2DGlpqYqJyfnhufpy++FESNGqEmTJho2bJgz+SiLN8Z6XRvH1zPG6OLF\ni7Lb7TLGKDs7W+vWrVOPHj3k5+en9PT0Ss+XB1AzkIOVjxys8io7tkpDDkYOVhvG8fXIwVAnuLAa\ndpm2b99uGjdubCQ5/wUEBJibb77ZbNy40RhjzN/+9jcTHR1tJJmQkBBz3333GWOMsdvt5plnnjE3\n3XSTCQgIMA0bNjTDhg0z33zzTbE+VqxYYfr06WOioqKMv7+/ady4sRk1apQ5duxYsXaPPfaYadSo\nkWnQoIFJTEw0S5YsMZJM69atzfTp00uNoSIWLlxogoODjSTTsmVLs2vXLvPXv/7VREREGEkmOjra\nvPrqq2bt2rXOPho2bGhee+21G8bVqVMnY7FYTKNGjcyHH35ojDHmkUceMfXq1TOSTEREhPnss89c\n+plMnDjRBAQEmJiYGOPv72/Cw8PN0KFDzeHDh4u1q+j1r0i70n7GixcvNk2bNjWSTHBwsBk8eLBZ\nunSp81redNNN5vDhw2bZsmUmPDzcSDItWrRwPn60ImPrwIEDxV7/8b9nnnnGGWNeXp558MEHTePG\njU1oaKjp2bOnefzxx40kExsba7744gtjjDFvvfWWCQsLM3/5y18qdL2PHDlipk+fbm655RYTEhJi\nrFaradWqlenVq5f54x//aN57771yr5MrPwtPvRc2bdpkWrdubSSZJk2amN/85jfOY/3hD39wjk1j\njPl//+//OX+u9erVMx06dDC7du26Yb8VHesffPCBadu2rfNn2KxZM5OYmFjm9a9L4/j11183t912\nmwkODjaBgYHOa+d46sbPfvYzM3fuXHP27Nnyhmy5xNPQXMbTS+oOT78/XD0+ORg5mDHe++wyxpiP\nPvrI9OjRwzRr1szZrmnTpqZ79+7m3//+t7MdORg5WE0fx9WRg5FPuI6noVVNeU9Dc0uxaOnSpWb6\n9OnFtl29etU88sgjpn79+s5HPqL6TJw40TRq1MjbYVQZYwu1QU0bxxSLXEdyV3f4WrGopv1+qQvI\nwQDfUdPGMfmE6ygWVU15xaIqT9z84YcfNHXqVO3bt6/Y9sDAQMXFxclms8lmsykoKKiqXcFFnnpU\nbXVhbKE2YBwD8BR+v/gucjDA+xjHQNVUec2ioKAgBQQEaPny5Tp9+rRsNptOnTqlf/zjH3r88ceV\nlJTks/M1v/766xLrgpT2Lykpyduh1qhY3aUmjy3AgXEMX7Zt2zbNnDlTdrtdw4YNU1xcnKxWq2Ji\nYjRkyBDt37/f5WPOnz9f7du3V1BQkEJCQtS+fXvNmjWr2HoYr7/+uubPn1/j/6D2tpr8+6Um5TU1\nKVZ3qcljC3BgHMNd6my+5MJtSGV67733zF133WXCw8ONn5+fiYiIMN27dzdLly41NputkjdEobJm\nzpxpAgMDnXP709LSvB1SpTG2UBvUtHEspqG5rCbeNv7444+bQYMGmdzcXGOz2Uzjxo3Nrl27zKVL\nl8x3331n7r77bhMREWFOnjzp0nEHDhxonn32WZOVlWXy8vLM+vXrTUBAgLn77ruLtVu4cKG58847\nzblz59x5Wh7n6feHq8evab9fajtyMMC31LRxXBPzCW/z9DS02p4veXzNIgBA7eHtYtHly5dNQkJC\njeqjpiV3Tz31lGnbtq3Jz883xhhjs9nMvffeW6zNJ598YiSZefPmuXTsYcOGOY/rkJiYaCSZU6dO\nFds+depUk5CQ4JMJe1l8rVgEAKg9fCGfqGl5mCfrF3UhXyqvWFTlaWgAALjT8uXLlZWVVeP78FWH\nDh3SrFmz9MQTT8hqtUqS/P39tWXLlmLtHI9pPnz4sEvH37Rpk/O4DjExMZJU4tHFc+bM0b59+7Rw\n4UKX+gAAAJ5BHnYN+ZIb1iwCANRtxhgtWLBAN998s+rXr6+GDRtq6NCh+vrrr51tpk6dqsDAQDVt\n2tS57eGHH1ZISIgsFovOnDkjSZo+fbpmzJihw4cPy2KxqE2bNlq0aJGsVquioqI0adIkNWvWTFar\nVd27d9fHH3/slj4k6e2331Z4eLjmzZvn0evlbYsWLZIxRoMHDy63XX5+viS5ZT2HgwcPqkGDBmrR\nokWx7Q0bNtSdd96phQsXyhhT5X4AAKhryMM8g3yJYhEAoIrmzJmjmTNn6s9//rOysrL03nvv6fjx\n4+rVq5dOnz4t6doH7ogRI4rtt3TpUj3xxBPFti1cuFCDBg1S69atZYzRoUOHNHXqVCUnJ+vy5cua\nNm2ajh49qr1796qwsFB33323jh8/XuU+pP97epHdbnffxfFBb775ptq1a6fg4OBy233yySeSpJ49\ne1aqH5vNppMnT2rJkiXatm2bFi9erMDAwBLtOnfurJMnT+qLL76oVD8AANRl5GGeQb5EsQgAUAX5\n+flasGCB7rvvPt1///2KiIhQx44d9eKLL+rMmTNatmyZ2/ry9/d3fmvWoUMHPf/888rLy9OKFSvc\ncvyBAwcqNzdXs2bNcsvxfNGlS5d05MgRtW7dusw2p0+f1tq1azVt2jQlJCTc8Bu1sjRv3lyxsbGa\nM2eOnn76aY0cObLUdjfddJMk6cCBA5XqBwCAuoo8zDPIl66hWAQAqLTMzExdvHhRd9xxR7HtXbt2\nVWBgYLHbk93tjjvuUHBwcLHbrFG+rKwsGWPK/ZYsISFB06ZN09ChQ5WRkaGAgIBK9XX8+HFlZWVp\nzZo1WrVqlTp37lzq+gSOWBzffgIAgIohD/MM8qVrKBYBACrt/PnzkqTQ0NASrzVo0EB5eXke7b9+\n/frKzs72aB+1yZUrVyRdu25liYqK0vbt27V48WJFRERUuq+AgABFRkaqX79+Wrt2rTIzM5WSklKi\nXVBQULHYAABAxZCHeQb50jUUiwAAldagQQNJKjUZOX/+vGJjYz3Wt81m83gftY0j0XCsC1CayMhI\n58/VXdq0aSM/Pz9lZmaWeK2goKBYbAAAoGLIwzyDfOkaikUAgEq79dZbFRoaqs8++6zY9o8//lgF\nBQXq0qWLc5u/v79sNpvb+t65c6eMMerWrZvH+qhtoqKiZLFYdOHChTLbbNmyxfnoVledPXtWo0eP\nLrH94MGDKioqUvPmzUu85oglOjq6Un0CAFBXkYd5BvnSNRSLAACVZrVaNWPGDG3atEmvvPKKcnNz\ndeDAAU2ePFnNmjXTxIkTnW3btGmjnJwcpaeny2azKTs7W8eOHStxzEaNGunUqVM6evSo8vLynEmH\n3W7XuXPnVFhYqP3792v69OmKi4tTcnKyW/rIyMjwqUe2ekJwcLDi4+N14sSJUl8/dOiQoqOjS11c\nMSkpSdHR0dq7d2+Zxw8JCdHWrVu1fft25ebmymaz6fPPP9cDDzygkJAQPfrooyX2ccTSsWPHSp4V\nAAB1E3mYZ5AvXUOxCABQJbNnz1ZKSormzp2rJk2a6M4771TLli21c+dOhYSEONtNmTJFffr00ahR\no9SuXTs9+eSTzltpExISnI9enTx5sqKiotShQwcNGDBAOTk5kq7N0e7YsaOCgoLUq1cvtW3bVjt2\n7Cg2n7yqfdQFAwcOVGZmpvLz80u8Zowpc7+CggJlZWVp8+bNZbaxWq3q0aOHHnzwQcXExCgsLEyJ\niYlq2bKldu/erVtvvbXEPp9++qliYmJ02223Ve6EAACow8jDPIN8SbKYH53p+vXrNXLkyHIvAACg\n9rJYLFq3bp1GjBjh7VCcJk2apLS0NJ09e9bboZQqMTFRkpSWlublSG7s0KFDuvnmm7VixQrdf//9\nFd7Pbrerd+/eSk5O1vjx490Sy9mzZxUbG6u//OUvmjFjhluO6Wmefn/44vsPAFA9fDWf8OU8zFP1\ni7qSL5Vz/dK4swgAUCOUt8ggKq5NmzaaO3eu5s6dq4sXL1Zon6KiIqWnpysvL09JSUlui2XOnDnq\n1KmTpk6d6rZjAgAA96treRj5EtPQAACoc2bOnKnExEQlJSWVu3ijw86dO7Vx40ZlZGQoODjYLTEs\nWLBA+/bt01tvvaWAgAC3HBMAAMBd6nq+RLEIAODT/vSnP2nFihW6cOGCWrVqpQ0bNng7pFph3rx5\nmjp1qp566qkbtu3bt69effVVNW3a1C19b968WVevXtXOnTvVsGFDtxwTAAC4X13Pw+pyvuRf7T0C\nAOCClJQUpaSkeDuMWqlfv37q169ftfc7ZMgQDRkypNr7BQAAriEPq7v5EncWAQAAAAAAwIliEQAA\nAAAAAJwoFgEAAAAAAMCJYhEAAAAAAACcKBYBAAAAAADAqcynoVksluqMAwDgQ0aOHKmRI0d6O4wa\nh89OuAPvPwCo28gnXMc1c78SxaLu3btr3bp13ogFAOAh77zzjt555x2dPHlSsbGx6t27t3r16qUG\nDRp4OzSgxunevbvHjk0Ohtru22+/1c6dO/XRRx/p6tWr6tSpk4YPH674+HhvhwYAuI7FGGO8HQQA\noHrs2bNHq1ev1po1a5STk6Nf/OIXGjt2rBITExUUFOTt8AAAtdCpU6eUlpam5cuX68CBA+rQoYPG\njRun5ORkRUdHezs8AEBJaRSLAKAOunr1ql5//XWtXr1ab7/9tkJDQzVixAiNHTtWPXv29HZ4AIAa\njs8ZAKjRKBYBQF136tQppaamavny5Tp48CDf+AIAKs1xB+urr76q8+fPq0+fPtzBCgA1D8UiAMD/\ncST5r7zyii5cuKA+ffpowoQJGjp0qAICArwdHgDABzmmmb388svav3+/br75Zj3wwAN64IEH1LRp\nU2+HBwBwHcUiAEBJV65c0ZYtW7R69WplZGQoIiJCw4cP16RJk9S5c2dvhwcA8DKmmQFArUaxCABQ\nvhMnTujVV1/VSy+9pMOHDzunqY0fP16RkZHeDg8AUI3KelDC8OHDFRwc7O3wAADuQbEIAFBxe/bs\n0bJly7RmzRrZbDbdfffdGjduHNPUAKAW++9//6v169drxYoV+uKLL9S+fXuNHDlS48ePV1xcnLfD\nAwC4H8UiAIDr8vPz9cYbb2jZsmV699131bRpUyUmJmr8+PG6/fbbvR0eAKCKrl69qq1btyo1NVX/\n/Oc/mWYGAHULxSIAQNUcP35ca9as0d///ncdOXJEXbp00dixY3X//fercePG3g4PAOACppkBAESx\nCADgLna7XR9++KFSU1P16quvqrCwUIMHD9bYsWPVv39/+fv7eztEAEApyppm9qtf/UotWrTwdngA\ngOpHsQgA4H65ublKT09Xamqq3n33Xf3kJz/R/fffr1//+te66aabvB0eANR5108zS09PV3BwsEaO\nHKmxY8eqR48eslgs3g4RAOA9FIsAAJ71zTff6LXXXtPKlSt17NgxdenSRRMmTNCoUaMUFhbm7fAA\noE7JzMxUamqqli9fzjQzAEBZKBYBAKqH3W7X9u3btXr1am3cuFF2u12DBg3ShAkT1LdvX77FBgAP\ncUwzW7lypfbt26d27dopKSmJaWYAgLJQLAIAVL/z589r/fr1Wr16tT744APFxsZqzJgxeuihh9S6\ndWtvhwcANV5p08wGDx6scePGUaAHANwIxSIAgHd99dVXWrVqlVauXKns7GwlJCRo3LhxGj16tEJD\nQ70dHgDUKI5pZi+//LLOnj3LNDMAQGVQLAIA+IaioiLt2LFDy5Yt0+bNm+Xn56d7772XaWoAcAM5\nOTnasGGDXnzxRX3++efOaWbJyclq2bKlt8MDANQ8FIsAAL7n3LlzSktL09///nft3btXcXFxGjVq\nlCZOnKhWrVp5OzwA8LqCggK98847TDMDAHgCxSIAgG9zTKlYsWKFzpw545ymNmbMGIWEhHg7PACo\nVj+eZsbvRACAB1AsAgDUDHyLDqCu+vE0s7Zt22rUqFFMMwMAeArFIgBAzfPDDz9o3bp1zsdAt2/f\nXiNHjuQx0ABqjR8XyIOCgjRkyBAK5ACA6kCxCABQszmmZCxfvlw5OTnOJ/8kJiYqKCjI2+EBgEuY\negsA8AEUiwAAtcPVq1e1detWpaam6p///KdCQ0M1YsQIjR07Vj179vR2eABQJsc0Mxb1BwD4CIpF\nAIDa59SpU0pLS9Py5ct14MABdejQQePGjVNycrKio6O9HR4AqKioSDt27NCyZcuYZgYA8DUUiwAA\ntduePXu0evVqvfLKK7pw4YL69OmjCRMmaMiQIQoMDPR2eADqGKaZAQBqAIpFAIC64cqVK9qyZYtW\nr16tjIwMhYeHKzExURMnTtRPf/pTb4cHoBY7d+6c0tLSmGYGAKgpKBYBAOqekydP6pVXXtE//vEP\nHTp0yDlNbfz48YqMjPR2eABqgeunmW3evFlWq5VpZgCAmoJiEQCgbtuzZ4+WLVum1157TVevXlW/\nfv00btw4DR06VAEBAd4OD0AN45hmtnLlSmVnZzunmY0ePVqhoaHeDg8AgIqgWAQAgCTl5+frjTfe\n0LJly/Tuu++qadOmSkxM1Pjx43X77bd7OzwAPswxzWz16tX64IMP1Lx5c40ePVoTJkxQfHy8t8MD\nAMBVFIsAAPix48ePa82aNVq2bJm+++47denSRWPHjtWYMWPUpEkTb4cHwAf8eJqZn5+f7r33Xk2Y\nMIFpZgCAmo5iEQAAZbHb7frwww+VmpqqV199VYWFhRo8eLDGjh2r/v37y9/f39shAqhmX375pVav\nXs00MwBAbUaxCACAisjNzVV6erpSU1P17rvvqlmzZho7dqx+/etf66abbvJ2eAA8qKxpZg899JBa\nt27t7fAAAHA3ikUAALjqm2++0WuvvaZVq1bp6NGj6tKliyZMmKBRo0YpLCzM2+EBcAOmmQEA6jCK\nRQAAVJbdbtf27du1evVqbdy4UXa7XYMGDdLYsWM1YMAA+fn5eTtEAC766quvtGrVKq1cuVKnT592\nFoOZZgYAqEMoFgEA4A4XLlzQunXrnNNUYmNjNWbMGKapADXA+fPntX79eqaZAQBwDcUiAADczXFn\nwqpVq/TDDz9wZwLggxzTzFavXq0NGzZIEtPMAAC4hmIRAACewpongO8pq5jLmmMAADhRLAIAoDr8\n+GlKcXFxGjVqlCZOnKhWrVp5OzygVvvxNDPHNNEHH3xQbdq08XZ4AAD4GopFAABUt8zMTKWmpmrF\nihU6c+aMEhISNG7cOI0ZM0YhISEuHSs3N1fh4eEeihTwDRcvXnR5CmdZC9BzZx8AADdEsQgAAG+5\nfppaenq6goODNXjwYI0bN67Cf8zOmDFDZ8+e1bJlyxQYGFgNUQPVp6ioSDNmzFB+fr7+/ve/V2if\nr7/+WmvXrtWKFSv0/fffM80MAADXUSwCAMAX5OTkaMOGDXrhhRe0b98+tWvXTklJSfrVr36lFi1a\nlLqPzWZT06ZNlZOTo4SEBL3++utq0qRJNUcOeEZubq4SExO1detWhYaGKisrS0FBQaW2ZZoZAABu\nRbEIAABf45imtnz5cuXk5OgXv/iFxo4dq+HDhys4ONjZbvPmzRo6dKgkKSAgQE2bNlVGRoZuueUW\nb4UOuMV3332n/v3768iRI7LZbKpXr55effVVJSUlOduUNc1s7NixGjBggPz8/Lx4BgAA1GgUiwAA\n8FVXr17V1q1blZqaqn/+858KDQ3ViBEjNHbsWPXs2VODBw9WRkaGCgsLJUn+/v4KDAzU2rVrNWjQ\nIC9HD1TOBx98oEGDBikvL885tv38/NS7d29t27bNOc1s5cqVOnbsGNPMAABwP4pFAADUBCdPnlRq\naqpWrlypb775Rh06dNA333yjoqKiYu0sFossFotSUlL02GOPeSlaoHJefvllTZw4UcaYUsd2p06d\n9PnnnysuLk4PPPCAxo0bxzQzAADcj2IRAAA1zYcffqjf/e53+uSTT0r8Qe1gsVj0q1/9Si+++KIC\nAgKqOULANcYYzZkzR3Pnzi2zTUBAgG677TbNnz9fffr0Ub169aoxQgAA6hSKRQAA1ES33HKLvvrq\nK5X3Me7v76+uXbtq8+bNioyMrMbogIq7ePGiRo8erTfffFN2u73ctq1atdLhw4d57D0AAJ6Vxlcy\nAADUMHv37tWXX35ZbqFIkgoLC/XZZ5+pa9eu+vrrr6spOqDiTpw4oYSEBGVkZNywUCRJR44c0e7d\nu6shMgAA6jaKRQAA1DCrVq2q8NQym82mkydP6mc/+5neffddD0cGVNwHH3yg2267Td98841zIesb\n8ff314oVKzwcGQAAYBoaANRBCxYs0EcffeTtMFAJdrtdb7zxhgoKCoptv35aTmlTdOx2uywWizp3\n7qz4+HiPxwmU59ixY9qzZ4/sdnuZaw85UtQfp6r+/v4aNGiQ/Pz8PB4n3O/RRx9VQkKCt8MAAJQv\nzd/bEQAAqt9HH32k3bt3q1u3bt4OBS6y2+3On5vdbnfekVFUVORc7LqwsFDGGBljZLPZJF37g7uw\nsFDnz5/X2bNn1bhxY++cQC114sQJ7d69W8OHD/d2KD7PMQZbtmyp7777Tk2bNlVYWJj8/PycBSA/\nPz9nEcnf37/Ef9vtdopFNdCGDRuUmJhIsQgAagCKRQBQR3Xr1k1paWneDgOoFdavX6+RI0fynnKR\nxWLRc889pxEjRng7FFQDFiYHgJqDNYsAAAAAAADgRLEIAAAAAAAAThSLAAAAAAAA4ESxCAAAAAAA\nAE4UiwAAAAAAAOBEsQgAAMBHvPXWW4qIiNCWLVu8HYpPmjRpkiwWi/Pf/fffX6LNtm3bNHPmTNnt\ndg0bNkxxcXGyWq2KiYnRkCFDtH//fpf7nT9/vtq3b6+goCCFhISoffv2mjVrlnJzc51tXn/9dc2f\nP19FRUVVOkdfPY/09PRi175JkyZVPkcAgO+iWAQAAOAjjDHeDsHnNWrUSBkZGfrmm2+0fPnyYq/N\nnj1bixYt0p/+9CfZ7Xbt2rVLa9asUU5Ojt5//33l5+frf/7nf3Tq1CmX+ty1a5ceeughff/99zp9\n+rSefPJJzZ8/X8OHD3e2GTx4sKxWq/r27avz589X6eu5ygUAACAASURBVBx98TyGDBmiEydO6L33\n3tOAAQOqdH4AAN9HsQgAAMBHDBw4UBcuXNCgQYO8HYry8/PVvXt3b4dRQlBQkO655x61bdtW9evX\nd27/61//qrVr12r9+vUKCwuTJCUkJKhnz54KDg5Wq1atNG/ePF24cEErV650qc/AwEA9/PDDioyM\nVGhoqBITEzV06FD961//0n//+19nu2nTpun222/XgAEDVFhYWKnz89XzsFgsiomJUa9evXTTTTdV\n6twAADUHxSIAAACUsHz5cmVlZXk7jAo5dOiQZs2apSeeeEJWq1WS5O/vX2I6X3x8vCTp8OHDLh1/\n06ZNzuM6xMTESJIuXrxYbPucOXO0b98+LVy40KU+pNpzHgCAmo9iEQAAgA94//33FRcXJ4vFoiVL\nlkiSnn/+eYWEhCg4OFibN29W//79FR4ertjYWL322mvOfRctWiSr1aqoqChNmjRJzZo1k9VqVffu\n3fXxxx87202dOlWBgYFq2rSpc9vDDz+skJAQWSwWnTlzRpI0ffp0zZgxQ4cPH5bFYlGbNm0kSW+/\n/bbCw8M1b9686rgkFbZo0SIZYzR48OBy2+Xn50uSwsPDq9znwYMH1aBBA7Vo0aLY9oYNG+rOO+/U\nwoULXZ5WWFvOAwBQ81EsAgAA8AE9e/bUhx9+WGzblClT9Mgjjyg/P19hYWFat26dDh8+rPj4eD30\n0EOy2WySrhWBkpOTdfnyZU2bNk1Hjx7V3r17VVhYqLvvvlvHjx+XdK0YMWLEiGJ9LF26VE888USx\nbQsXLtSgQYPUunVrGWN06NAhSXIuemy32z1yDSrrzTffVLt27RQcHFxuu08++UTStWtdGTabTSdP\nntSSJUu0bds2LV68WIGBgSXade7cWSdPntQXX3zh0vFry3kAAGo+ikUAAAA1QPfu3RUeHq7IyEgl\nJSXp0qVL+v7774u18ff3180336z69eurQ4cOev7555WXl6cVK1a4JYaBAwcqNzdXs2bNcsvx3OHS\npUs6cuSIWrduXWab06dPa+3atZo2bZoSEhJueOdOWZo3b67Y2FjNmTNHTz/9tEaOHFlqO8eaPgcO\nHKjwsWvLeQAAageKRQAAADWM4y4Qx51FZbnjjjsUHBysr7/+ujrC8oqsrCwZY8q9GychIUHTpk3T\n0KFDlZGRoYCAgEr1dfz4cWVlZWnNmjVatWqVOnfuXOq6To5YTp8+XeFj15bzAADUDhSLAAAAarH6\n9esrOzvb22F4zJUrVySp2JPRfiwqKkrbt2/X4sWLFRERUem+AgICFBkZqX79+mnt2rXKzMxUSkpK\niXZBQUHFYquI2nIeAIDagWIRAABALWWz2XT+/HnFxsZ6OxSPcRQ0HOsplSYyMlINGjRwa79t2rSR\nn5+fMjMzS7xWUFBQLLaKqC3nAQCoHSgWAQAA1FI7d+6UMUbdunVzbvP397/h9LWaJCoqShaLRRcu\nXCizzZYtW5yPiHfV2bNnNXr06BLbDx48qKKiIjVv3rzEa45YoqOjK9xPbTkPAEDtQLEIAACglrDb\n7Tp37pwKCwu1f/9+TZ8+XXFxcUpOTna2adOmjXJycpSeni6bzabs7GwdO3asxLEaNWqkU6dO6ejR\no8rLy5PNZlNGRobCw8M1b968ajyr8gUHBys+Pl4nTpwo9fVDhw4pOjq61EWck5KSFB0drb1795Z5\n/JCQEG3dulXbt29Xbm6ubDabPv/8cz3wwAMKCQnRo48+WmIfRywdO3ascD814TwAAHUHxSIAAAAf\nsGTJEnXt2lWS9Nhjj2nIkCF6/vnn9dxzz0mSbrvtNn333Xd66aWXNGPGDEnSPffco4MHDzqPceXK\nFXXs2FFBQUHq1auX2rZtqx07dhRbB2fKlCnq06ePRo0apXbt2unJJ590TjNKSEjQ8ePHJUmTJ09W\nVFSUOnTooAEDBignJ6darkNlDBw4UJmZmcrPzy/xmjGmzP0KCgqUlZWlzZs3l9nGarWqR48eevDB\nBxUTE6OwsDAlJiaqZcuW2r17t2699dYS+3z66aeKiYnRbbfdVuF+asJ5AADqDn9vBwAAAADpN7/5\njX7zm9+U2D5lypRi/x8fH6+HHnqo1GOEhYWVeWeKQ6NGjbR9+/YS259++uli/9+5c2cdPXq02Lb+\n/7+9ew+Lqs7/AP4eHWAY7l5QhPACWggYGiaQpj6UpqbihUu/fFpqf4mXFtxoF6VUJPGytsq6Qumu\nS49mCagLmWL9tFi1FDVFFPMCijdWUZCbjDAw398fLbMRCIwMc4B5v56Hf875nu/nc86Z+cB8OHPO\npEkoLy9vdn4p/O53v0NiYiJ2796NOXPmNFg3ePDgxz7Na9euXRg3bhz69+/f7PwtNXl+qbi4GIcO\nHcLKlSshk8l0itPR94OIiIwHrywiIiIi6iKauzlyV6FSqfD111/jypUr2hswu7q6IjY2FrGxsais\nrGzVPHV1dUhLS0NFRQVCQkL0ll9MTAy8vLwQHh6uc5yOvB9CCBQWFuLo0aPIy8vTWxwiIuqY2Cwi\nIiIiok6jpKQEr7zyCoYMGYK33npLu3zJkiUIDAxESEhIszeJrpeZmYndu3cjIyMDSqVSL7mtX78e\n2dnZ2L9/P0xMTJ4oTkfdj/T0dDg6OmLMmDHYt2+fXuIQEVHHxWYRERFRJ3Tp0iX87ne/g7u7O6ys\nrCCXy2FjY4MhQ4ZgypQpOHbsmNQpkgFFR0cjKSkJZWVlGDhwIHbt2iV1Su3ik08+gRBC+/PZZ581\nWB8XF4fw8HCsXr26xbn8/f2xY8cO9O3bVy+5paeno7q6GpmZmbCzs2tTnI64HwEBAQ2O/f379/US\nj4iIOiaZaO5ueURE1CUFBgYCAFJTUyXOhJ7E1q1bMX/+fPj6+iI6OhqjRo2Cubk5bt++jZMnT2Lj\nxo34zW9+g7lz50qdqtFISUlBcHBwszchpsZkMhmSk5MRFBQkdSpkADzfRESdRiqvLCIioi5JpVLB\nz8+vy8U+fvw4wsLCMGbMGBw6dAgTJ06Era0tzMzMMGjQIAQHB2PZsmXae7l0RF313BARERF1FXwa\nGhERdUlbt25FUVFRl4u9cuVK1NXVYfXq1ZDLm/41PnHiREycOLFd4utDVz03RERERF0FrywiIqJW\n2759O7y9vaFQKGBhYYEBAwbgww8/BPDzk3LWr18PNzc3mJmZwc7ODgEBAbh48aJ2+8TERFhYWECp\nVCI9PR2TJk2CtbU1nJyc8MUXX+gU78iRIxg6dChsbGygUCjg6emJr7/+GgCwaNEiREZGIj8/HzKZ\nDK6urgB+fmrQsmXL4OzsDHNzcwwbNgzJyck656bv2ABw4MABWFtbIy4u7rHHv6amBocOHULPnj3x\n/PPPt/q88dy07dwQERERGR1BRERGZ/bs2WL27Nk6bbNhwwYBQKxevVoUFxeLkpISsXnzZvH6668L\nIYRYtmyZMDU1Fdu3bxelpaUiJydHjBgxQvTq1UvcuXNHO8/7778vAIhDhw6JsrIyUVRUJMaMGSMs\nLCxETU1Nq+OlpqaKmJgYUVJSIoqLi4WPj4/o2bOndvtZs2YJFxeXBvvw3nvvCTMzM7Fr1y7x4MED\nER0dLbp16yZOnjypU27tEfurr74SVlZWIjY29rHn4PLlywKA8PHxad1J+w+em7bFbo3k5GTBP6t0\nB0AkJydLnQYZCM83EVGnkcK/aoiIjJCuzaKamhpha2srxo8f32B5bW2tiI+PF1VVVcLS0lKEhIQ0\nWH/ixAkBoEEDpP5Dv0ql0i5LSEgQAEReXl6r4jVl1apVAoAoKioSQjRuCqhUKqFUKhvkWFVVJczM\nzMSCBQtanVt7xW6NU6dOCQDipZdeavU2PDeGOTdsFj0ZNg+MC883EVGnkcJ7FhERUYtycnJQWlra\n6D443bt3R0REBE6dOoXKykp4e3s3WD9y5EiYmpoiKyur2flNTU0BAGq1ulXxmmJiYgLg568UNeXS\npUuoqqqCh4eHdpm5uTn69u3b4OtYLeVmyNi/ZmlpCQCoqqpq9Ta5ubk8N+0Q+3FkMpnO2xi74OBg\nBAcHS50GERER/QKbRURE1KLy8nIAgK2tbZPrS0tLAfy3mfFLtra2qKio0Gs8ANi3bx/WrVuH3Nxc\nlJeXN9swAICHDx8CAD744AN88MEHDdY5ODjolJ9UsQcMGACFQoHLly+3ehueG8PFBsB7HekoODgY\nixYtgq+vr9SpkAGwKUhE1HmwWURERC3q168fAOD+/ftNrq9vHDTVeCgtLYWTk5Ne4924cQMzZszA\nzJkz8Y9//AP9+vXDX//6V/zxj3987Jy9e/cGAGzYsAGLFi3SKZ+OEtvMzAwTJ05Eeno6vv/+e7zw\nwgtNjispKcEf//hH/P3vf+e5MVDsekFBQW2ew5gEBwfD19eXx81IsFlERNR58GloRETUogEDBqBH\njx745ptvmlzv4eEBS0tLnDp1qsHyrKws1NTU4LnnntNrvHPnzkGtVmPBggUYNGgQFApFi1//eeqp\np6BQKJCdna1TLh0pNgDExMTAzMwM7777LlQqVZNjzp8/D7n85/8H8dwY7twQERERdRVsFhERUYvM\nzMwQHR2Nw4cPIzw8HLdv34ZGo0FFRQUuXLgAhUKByMhI7NmzB5999hnKy8tx7tw5zJ8/Hw4ODggL\nC9NrPGdnZwDAwYMH8ejRI1y5cqXRvXd69OiBwsJCFBQUoKKiAt27d8ebb76JL774AomJiSgvL0dd\nXR1u3bqFf//7363Orb1iZ2RkwNraGnFxcc3G9/Lywo4dO3D+/HmMGTMG+/fvR1lZGdRqNa5du4a/\n/e1v+O1vf6u9Vw/PjWFiExEREXUpUt9im4iIDE/Xp6HV27Rpk/D09BQKhUIoFAoxfPhwkZCQIIQQ\nQqPRiHXr1onBgwcLExMTYWdnJ2bMmCEuXbqk3T4hIUEolUoBQAwePFjk5+eLLVu2CGtrawFA9O/f\nX1y+fLlV8aKiokSPHj2Era2tCAwMFJs2bRIAhIuLi7hx44Y4ffq06N+/vzA3NxejR48Wd+7cEdXV\n1SIqKko4OzsLuVwuevfuLWbNmiVyc3N1yk3fsYUQYv/+/cLKykqsXLmyVefixo0b4r333hOenp7C\n0tJSdO/eXdja2orhw4eL3/72t+L777/XjuW5adu5aQ0+De3JgE/HMio830REnUaKTAghJOhRERGR\nhAIDAwEAqampEmdC1DWkpKQgODgY/LNKNzKZDMnJybxnkZHg+SYi6jRS+TU0IiIiIiIiIiLSYrOI\niIiIiEgHBw8exJIlS6DRaDBjxgw4OztDoVDA0dER06dPR05OzhPNq1arsWrVKri6usLU1BS2trbw\n8PBAQUEBAODLL7/E2rVrUVdXp8e9ISIiaozNIiIiIiKiVlq+fDk2btyI6OhoaDQaHDlyBJ9//jlK\nSkpw9OhRqFQqvPjiiygsLNR57uDgYGzbtg07duxAVVUVfvrpJ7i4uKCyshIAMG3aNCgUCvj7+6O0\ntFTfu0ZERKTFZhERERFRF6BSqeDn59fpY3Rka9aswc6dO5GSkgIrKysAgK+vL0aPHg2lUomBAwci\nLi4OZWVl+PTTT3Wae+fOnUhLS0NqaipGjRoFuVwOBwcHpKenw8PDQzsuIiICzz77LCZPnoza2lp9\n7h4REZEWm0VEREREXcDWrVtRVFTU6WN0VHl5eVi6dClWrFgBhUIBAJDL5di7d2+DcYMGDQIA5Ofn\n6zT/xx9/jBEjRsDT07PFsTExMcjOzkZ8fLxOMYiIiFqLzSIiIiIiCQghsH79eri5ucHMzAx2dnYI\nCAjAxYsXtWPCw8NhamqKvn37apctXLgQFhYWkMlkuH//PgBg0aJFiIyMRH5+PmQyGVxdXbFx40Yo\nFArY29tj3rx5cHBwgEKhgJ+fH7KysvQSAwAOHDgAa2trxMXFtevxktrGjRshhMC0adOaHadSqQAA\n1tbWrZ67pqYGx48fh5eXV6vG29nZYezYsYiPj+cT+IiIqF2wWUREREQkgZiYGCxZsgTvv/8+ioqK\ncPjwYdy8eRNjxozB3bt3AfzcoPj1Y8YTEhKwYsWKBsvi4+MxdepUuLi4QAiBvLw8hIeHIzQ0FFVV\nVYiIiEBBQQFOnz6N2tpavPzyy7h582abYwDQ3mxZo9Ho7+B0QPv27cPTTz8NpVLZ7LgTJ04AAEaP\nHt3quQsLC1FTU4Mff/wR48eP1zb23NzckJCQ0GRDaPjw4bh9+zbOnj2r244QERG1AptFRERERAam\nUqmwfv16zJw5E3PmzIGNjQ08PT3xySef4P79+9iyZYveYsnlcu3VS0OHDkViYiIqKiqQlJSkl/mn\nTJmC8vJyLF26VC/zdUQPHz7EtWvX4OLi8tgxd+/exc6dOxEREQFfX98Wr0D6pfobWPfu3RtxcXHI\nzc3F3bt3ERAQgHfeeQeff/55o20GDx4MADh37pyOe0NERNQyNouIiIiIDCw3NxeVlZXw9vZusHzk\nyJEwNTVt8DUxffP29oZSqWzwdTdqXlFREYQQzV5V5Ovri4iICAQEBCAjIwMmJiatnt/MzAwA4O7u\nDj8/P/To0QM2NjZYsWIFbGxsmmwe1udSfxUaERGRPsmlToCIiIjI2NQ/9tzS0rLROltbW1RUVLRr\nfDMzM9y7d69dY3Qljx49AvDfpk5T7O3tsXXrVri7u+s8v4ODAwBo7w9Vz9TUFP3792/yZtnm5uYN\nciMiItInXllEREREZGC2trYA0GRTqLS0FE5OTu0WW61Wt3uMrqa+MVN/f6am9O7dW3tedWVpaYnB\ngwfjwoULjdbV1tbCxsam0fKampoGuREREekTm0VEREREBubh4QFLS0ucOnWqwfKsrCzU1NTgueee\n0y6Ty+VQq9V6i52ZmQkhBHx8fNotRldjb28PmUyGsrKyx47Zu3cvHB0dnzhGcHAwzpw5g6tXr2qX\nVVVV4fr16/D09Gw0vj6XPn36PHFMIiKix2GziIiIiMjAFAoFIiMjsWfPHnz22WcoLy/HuXPnMH/+\nfDg4OCAsLEw71tXVFSUlJUhLS4Narca9e/dw/fr1RnP26NEDhYWFKCgoQEVFhbb5o9Fo8ODBA9TW\n1iInJweLFi2Cs7MzQkND9RIjIyMD1tbWiIuL0/+B6iCUSiUGDRqEW7duNbk+Ly8Pffr0QXBwcKN1\nISEh6NOnD06fPt1sjHfffRf9+/dHaGgobty4geLiYkRFRUGlUmHx4sWNxtfn0lQjiYiIqK3YLCIi\nIiKSwPLly7Fq1SrExsaiV69eGDt2LAYMGIDMzExYWFhoxy1YsADjx4/Ha6+9hqeffhoffvih9qtH\nvr6+uHnzJgBg/vz5sLe3x9ChQzF58mSUlJQA+PmeNp6enjA3N8eYMWMwZMgQfPfddw3uv9PWGMZg\nypQpyM3NhUqlarSuqUfb16upqUFRURHS09Obnd/Ozg5HjhyBk5MTvLy84OjoiBMnTmDfvn3w8vJq\nNP7kyZNwdHTEsGHDdN8ZIiKiFshEc7/diIioSwoMDAQApKamSpwJUdeQkpKC4ODgZpsGUpg3bx5S\nU1NRXFwsdSpNkslkSE5ORlBQkNSptCgvLw9ubm5ISkrCnDlzWr2dRqPBuHHjEBoairfeeksvuRQX\nF8PJyQkrV65EZGSkXuY0hM50vomIjFwqrywiIiIi6sKauykztZ6rqytiY2MRGxuLysrKVm1TV1eH\ntLQ0VFRUICQkRG+5xMTEwMvLC+Hh4Xqbk4iI6JfYLCIiIiIiaoUlS5YgMDAQISEhzd7sul5mZiZ2\n796NjIwMKJVKveSwfv16ZGdnY//+/TAxMdHLnERERL/GZhERERFRFxQdHY2kpCSUlZVh4MCB2LVr\nl9QpdQlxcXEIDw/H6tWrWxzr7++PHTt2oG/fvnqJnZ6ejurqamRmZsLOzk4vcxIRETVFLnUCRERE\nRKR/q1atwqpVq6ROo0uaMGECJkyYYPC406dPx/Tp0w0el4iIjA+vLCIiIiIiIiIiIi02i4iIiIiI\niIiISIvNIiIiIiIiIiIi0mKziIiIiIiIiIiItHiDayIiI3Xr1i2kpKRInQYZISEEZDKZ1Gno1bFj\nxwCgU7ynOtrxrz92RERE1HHIhBBC6iSIiMiwAgMD+RhtIiIyuOTkZAQFBUmdBhERNS+VzSIiIiJq\nd2fPnsVvf/tb5ObmYunSpfjjH/8IuZwXOBtaYWEh5s+fj7179+Ltt9/Gn/70J9jY2EidFhEREXUs\nqbxnEREREbUbtVqNtWvX4vnnn4eZmRlOnz6N6OhoNook0q9fP6SnpyM9PR1fffUV3NzcsGfPHqnT\nIiIiog6GzSIiIiJqF8eOHYOXlxdiY2MRGxuLw4cPw83NTeq0CMDUqVORm5uLqVOnYtasWQgKCkJR\nUZHUaREREVEHwWYRERER6VVVVRUWL16MMWPGwNnZGRcuXEBUVBS6d+8udWr0C7a2tti8eTMyMjJw\n4sQJPPPMM9iyZYvUaREREVEHwGYRERER6c3XX3+NoUOHYvPmzUhMTMT+/fvRv39/qdOiZrzyyiu4\ncOEC5s6di/nz52Py5Mm4ceOG1GkRERGRhNgsIiIiojYrLS1FWFgYJk2aBE9PT5w/fx5z587tUI9o\np8dTKpVYs2YNjhw5goKCAri5uWHt2rXQaDRSp0ZEREQSYLOIiIiI2mTv3r3w8PDAl19+idTUVOzd\nuxeOjo5Sp0VPwM/PD6dPn8Yf/vAHLFu2DC+++CIuXrwodVpERERkYGwWERER0RO5c+cOAgMDMW3a\nNPj5+eH8+fOYNWuW1GlRGykUCsTExODkyZOorq7G8OHDERMTg5qaGqlTIyIiIgNhs4iIiIh0lpqa\nCg8PD/z444/4v//7P6SkpKBnz55Sp0V6NGzYMBw7dgxr1qzBRx99hJEjR+LUqVNSp0VEREQGwGYR\nERERtVpBQQEmTJiAkJAQzJo1Czk5OXjppZekTovaiVwuR0REBM6ePYuePXvC19cXixcvxqNHj6RO\njYiIiNoRm0VERETUIo1Ggy1btsDT0xOFhYX44YcfsHnzZlhaWkqdGhmAi4sLDh06hISEBCQmJsLT\n0xPfffed1GkRERFRO2GziIiIiJqVm5uLF154Ae+88w4WLlyIH3/8EaNGjZI6LTIwmUyGuXPn4uLF\ni/Dw8IC/vz/CwsJQUVEhdWpERESkZ2wWERERUZPUajXWrl2L5557DtXV1cjKysKaNWtgZmYmdWok\noX79+uGf//wnkpOTsWfPHjzzzDNIT0+XOi0iIiLSIzaLiIiIqJHs7Gz4+PhgxYoVWLFiBU6ePInh\nw4dLnRZ1IIGBgcjNzYW/vz8CAgIQFBSE+/fvS50WERER6QGbRURERKSlUqmwePFieHt7w8LCAtnZ\n2YiKikL37t2lTo06IHt7e2zbtg1fffUVjh8/Dg8PD2zbtk3qtIiIiKiN2CwiIiIiAMCRI0cwfPhw\nfPLJJ0hMTMS//vUvDBkyROq0qBOYMmUKzp07h+DgYLz55puYOnUqbt26JXVaRERE9ITYLCIiIjJy\nZWVliIiIwLhx4+Dq6orz589j7ty5kMlkUqdGnYiNjQ3+8pe/IDMzE1euXIGHhwe2bNkCIYTUqRER\nEZGO2CwiIiIyYvv27YOnpyd27tyJpKQkfPXVV3BycpI6LerExowZgzNnzmDevHlYsGABxo4di8uX\nL0udFhEREemAzSIiIiIjVFRUhDfeeAOvvvoqfHx8kJubizfeeEPqtKiLMDc3x5o1a3Dq1Ck8fPgQ\nXl5eWLt2Lerq6qROjYiIiFqBzSIiIiIjk5qaCg8PDxw6dAhpaWlISUlBr169pE6LuiAvLy8cP34c\ny5cvx/LlyzFy5EicOXNG6rSIiIioBWwWERERGYnCwkLMmDEDwcHBmDFjBi5evIjp06dLnRZ1cSYm\nJoiKisKPP/4IMzMzjBo1CosXL0Z1dbXUqREREdFjsFlERETUxQkhsGXLFri5ueHcuXM4dOgQNm/e\nDCsrK6lTIyPi7u6O77//Hps2bUJCQgK8vb2RlZUldVpERETUBDaLiIiIurD8/Hz4+/tj4cKFmD9/\nPs6fP4/x48dLnRYZqW7dumHu3LnIycmBg4MD/Pz8EBYWhsrKSqlTIyIiol9gs4iIiKgLqq2txV/+\n8hc8++yzKC4uxrFjx7BmzRooFAqpUyPCwIED8c0332Dnzp3YvXs3hg0bhoMHD0qdFhEREf0Hm0VE\nRERdTE5ODvz8/LB48WK89957OHnyJLy9vaVOi6iRwMBAnD9/Hs899xwmTJiAN954AyUlJVKnRURE\nZPTYLCIiIuoiHj16hJiYGIwcORKmpqY4c+YMYmJiYGpqKnVqRI/Vt29fpKamIj09Hd9++y3c3d2x\ne/duqdMiIiIyamwWERERdQE//PADRowYgXXr1iE2NhaHDx/GM888I3VaRK02depUnD9/HtOmTUNg\nYCCmTp2KwsJCqdMiIiIySmwWERERdWJVVVVYvHgxxowZg/79++Onn35CVFQUunXjr3jqfGxtbbF5\n82ZkZGTg3LlzcHd3x5YtW6ROi4iIyOjwL0kiIqJO6sCBAxg6dCi2bNmCjz/+GBkZGXB2dpY6LaI2\nmzhxIi5cuICwsDAsWLAAkyZNwvXr16VOi4iIyGiwWURERNTJlJaWIiwsDJMnT8bzzz+PixcvYu7c\nuVKnRaRXSqUSa9asweHDh3H9+nUMHToUa9euhUajkTo1IiKiLk8mhBBSJ0FERESts3fvXsybNw8a\njQYJCQmYOXOm1CkRtTu1Wo3169dj2bJl8Pb2xt///ne4ublJnRYREVFXlcori4iIiDqBO3fuYPbs\n2Zg+fTr8/f2Rm5vLRhEZDRMTE0RFReHUqVNQq9UYMWIEYmJiUFNTI3VqREREXRKbRURERB2YEALb\ntm2Du7s7Tp8+jW+++Qbbtm1Djx49pE6NyOA89DlBbQAAIABJREFUPT3xww8/YM2aNfjoo48wcuRI\nnDp1Suq0iIiIuhw2i4iIiDqoa9euYcKECXjzzTcxe/Zs5OTk4KWXXpI6LSJJyeVyRERE4OzZs+jV\nqxd8fX0RERGBhw8fSp0aERFRl8FmERERUQej0WiwZcsWDBs2DHfu3MGxY8ewefNmWFpaSp0aUYfh\n4uKCgwcPIiEhAZ9++imeffZZfPfdd1KnRURE1CWwWURERNSBnD9/Hn5+fnjnnXewcOFCnDp1Cs8/\n/7zUaRF1SDKZDHPnzsXFixcxbNgw+Pv7IywsDOXl5VKnRkRE1KmxWURERGQA2dnZ+MMf/vDY9Wq1\nGmvXroW3tzdkMhnOnDmDNWvWwMzMzIBZEnVODg4O2LNnD5KTk/HPf/4Tbm5uSEtLe+z4qqoqzJw5\nEyqVyoBZEhERdR5sFhEREbWzkpISvPrqq/jzn/+MI0eONFp/5swZjBo1CitWrMCKFStw9OhRuLu7\nS5ApUecWGBiIS5cu4dVXX8WMGTMQFBSE+/fvNxq3ePFi/POf/8TChQslyJKIiKjjY7OIiIioHWk0\nGrz22msoKipCt27dEBoaikePHgEAVCoVFi9ejJEjR8LKygrZ2dmIiopC9+7dJc6aqPOys7PD5s2b\nsW/fPhw/fhzu7u7Ytm2bdv2xY8ewadMmAEBSUhL+8Y9/SJUqERFRh8VmERERUTuKjY3FwYMHoVar\nUVdXhxs3biA2NhZHjhyBl5cXPvnkEyQmJiIzMxNDhgyROl2iLmPy5Mk4f/48QkJC8Oabb+LVV19F\nfn4+QkND0a3bf/8EnjdvHn788UcJMyUiIup4ZEIIIXUSREREXdGhQ4cwYcIEaDSaBsu7desGIQQC\nAgKQkJAABwcHiTIkMg6ZmZl4++23UVRUhIcPH6Kurk67Ti6Xw97eHjk5OejZs6eEWRIREXUYqWwW\nERERtYObN29i2LBhKC8vb9QsksvlcHR0xJUrV2BiYiJRhkTG5eTJk/Dx8Wn0fgQAExMT+Pv7Y9++\nfQ2uOiIiIjJSqfxtSEREpGfV1dWYNm0aHj582OQH09raWty6dQvr1q2TIDsi41NXV4e5c+c+9n5g\narUa33zzDVavXm3gzIiIiDomNouIiIj0LDw8HOfPn4darX7smLq6OsTExODChQsGzIzIOP35z39G\nTk5Os+9JjUaDpUuX4uuvvzZgZkRERB0Tv4ZGRESkR9u3b8cbb7zRqrEymQyjRo3C999/z6++ELWT\nvLw8uLu7o6ampsWx3bp1g42NDc6ePYunnnrKANkRERF1SPwaGhERkb7k5OTg7bffbnaMXC6HTCaD\nTCaDq6srvL29cffuXQNlSGR8rl+/joCAAPTq1QsA0L1798feK0yj0aCyshIzZsxoVXOJiIioq+KV\nRURERHpQUVGB4cOH4/r166itrdUuNzMzQ3V1Nbp16wYXFxeMHz8eL730EsaPH6/98EpEhnH16lUc\nPXoUR48exbfffov8/HzIZDKYmJg0aA7J5XKEhYVh06ZNEmZLREQkGT4NjYiIqK2EEJg+fTr27t2L\nbt26QaPRQC6Xw8vLCy+//DJefPFFvPDCC7CyspI6VSL6hRs3buBf//oXDh8+jG+//RZXr17VfiVU\no9Fgx44d+J//+R+JsyQiIjI4NouI6L9SUlIQHBwsdRpERAY3e/ZspKamttv8Mpms3eYmIupI+PGS\nqEtIlUudARF1PMnJyVKnQNQmGzZsAAD8/ve/b/dYGo0GV69exYABAyCXd95fq8eOHUN8fLxRvv/r\nXy/tbdGiRfD19TVILNKPiooKPHjwAM7OzlKnIgljrgttERwcbJTv9/rXCxF1DZ33r1oiajdBQUFS\np0DUJvVXiPC1rJv4+HijPGbteUXRL/n6+hrl8aXOzVjrQlsEBwcb7fudzSKiroNPQyMiIiIiIiIi\nIi02i4iIiIiIiIiISIvNIiIiIiIiIiIi0mKziIiIiIiIiIiItNgsIiIiIiIiIiIiLTaLiIiIHmP/\n/v2wsbHB3r17pU6lwzt48CCWLFkCjUaDGTNmwNnZGQqFAo6Ojpg+fTpycnKeaF61Wo1Vq1bB1dUV\npqamsLW1hYeHBwoKCgAAX375JdauXYu6ujo97g0R6RNraeuxlhJRR8FmERER0WMIIaROoVNYvnw5\nNm7ciOjoaGg0Ghw5cgSff/45SkpKcPToUahUKrz44osoLCzUee7g4GBs27YNO3bsQFVVFX766Se4\nuLigsrISADBt2jQoFAr4+/ujtLRU37tGRHrAWto6rKVE1JGwWURERPQYU6ZMQVlZGaZOnSp1KlCp\nVPDz85M6jUbWrFmDnTt3IiUlBVZWVgAAX19fjB49GkqlEgMHDkRcXBzKysrw6aef6jT3zp07kZaW\nhtTUVIwaNQpyuRwODg5IT0+Hh4eHdlxERASeffZZTJ48GbW1tfrcPSLSA9bSlrGWElFHw2YRERFR\nJ7B161YUFRVJnUYDeXl5WLp0KVasWAGFQgEAkMvljb5qMmjQIABAfn6+TvN//PHHGDFiBDw9PVsc\nGxMTg+zsbMTHx+sUg4iMC2tp81hLiagem0VERERNOHr0KJydnSGTybBp0yYAQGJiIiwsLKBUKpGe\nno5JkybB2toaTk5O+OKLL7Tbbty4EQqFAvb29pg3bx4cHBygUCjg5+eHrKws7bjw8HCYmpqib9++\n2mULFy6EhYUFZDIZ7t+/DwBYtGgRIiMjkZ+fD5lMBldXVwDAgQMHYG1tjbi4OEMckkY2btwIIQSm\nTZvW7DiVSgUAsLa2bvXcNTU1OH78OLy8vFo13s7ODmPHjkV8fDy/8kLUgbCWtoy1lIg6IjaLiIiI\nmjB69Gj88MMPDZYtWLAAv//976FSqWBlZYXk5GTk5+dj0KBBePvtt6FWqwH8/MElNDQUVVVViIiI\nQEFBAU6fPo3a2lq8/PLLuHnzJoCfPyAEBQU1iJGQkIAVK1Y0WBYfH4+pU6fCxcUFQgjk5eUBgPZG\npBqNpl2OQUv27duHp59+GkqlstlxJ06cAPDzMW2twsJC1NTU4Mcff8T48eO1HxLd3NyQkJDQ5IeY\n4cOH4/bt2zh79qxuO0JE7Ya1tGWspUTUEbFZRERE9AT8/PxgbW2N3r17IyQkBA8fPsSNGzcajJHL\n5XBzc4OZmRmGDh2KxMREVFRUICkpSS85TJkyBeXl5Vi6dKle5tPFw4cPce3aNbi4uDx2zN27d7Fz\n505ERETA19e3xf+a/1L9TVd79+6NuLg45Obm4u7duwgICMA777yDzz//vNE2gwcPBgCcO3dOx70h\nIqmwlrKWElHHxGYRERFRG5mamgKA9r/hj+Pt7Q2lUomLFy8aIq12VVRUBCFEs/8J9/X1RUREBAIC\nApCRkQETE5NWz29mZgYAcHd3h5+fH3r06AEbGxusWLECNjY22LJlS6Nt6nO5e/eujntDRB0Ba2nT\nWEuJSApyqRMgIiIyJmZmZrh3757UabTZo0ePAPz3g0hT7O3tsXXrVri7u+s8v4ODAwBo7zVSz9TU\nFP3792/yBq/m5uYNciOirou1tHVYS4noSfHKIiIiIgNRq9UoLS2Fk5OT1Km0Wf2Hifp7fTSld+/e\nsLW1faL5LS0tMXjwYFy4cKHRutraWtjY2DRaXlNT0yA3IuqaWEtbj7WUiJ4Um0VEREQGkpmZCSEE\nfHx8tMvkcnmLX7noiOzt7SGTyVBWVvbYMXv37oWjo+MTxwgODsaZM2dw9epV7bKqqipcv369yUdA\n1+fSp0+fJ45JRB0fa6luWEuJ6EmwWURERNRONBoNHjx4gNraWuTk5GDRokVwdnZGaGiodoyrqytK\nSkqQlpYGtVqNe/fu4fr1643m6tGjBwoLC1FQUICKigqo1WpkZGRI9rhnpVKJQYMG4datW02uz8vL\nQ58+fRAcHNxoXUhICPr06YPTp083G+Pdd99F//79ERoaihs3bqC4uBhRUVFQqVRYvHhxo/H1uTT1\n4YeIOi/WUtZSIjI8NouIiIiasGnTJowcORIAEBUVhenTpyMxMREbNmwAAAwbNgxXr17F3/72N0RG\nRgIAXnnlFVy5ckU7x6NHj+Dp6Qlzc3OMGTMGQ4YMwXfffdfg3hQLFizA+PHj8dprr+Hpp5/Ghx9+\nqL3039fXV/to6Pnz58Pe3h5Dhw7F5MmTUVJSYpDj0JwpU6YgNzcXKpWq0bqmHsdcr6amBkVFRUhP\nT292fjs7Oxw5cgROTk7w8vKCo6MjTpw4gX379sHLy6vR+JMnT8LR0RHDhg3TfWeIqF2wlraMtZSI\nOiKZaK4CEZFRSUlJQXBwcLN/mBB1BoGBgQCA1NRUyXKYN28eUlNTUVxcLFkOuniS939eXh7c3NyQ\nlJSEOXPmtHo7jUaDcePGITQ0FG+99daTpNtIcXExnJycsHLlSu0HztYyxOtFJpMhOTkZQUFB7RaD\nSN86wt8Fna2WArq/37tKLe0Irxci0ptUXllERETUTpq7YWlX4OrqitjYWMTGxqKysrJV29TV1SEt\nLQ0VFRUICQnRWy4xMTHw8vJCeHi43uYkoo6BtbQx1lIiam9sFhERGZHdu3dj0KBBkMlkDX5MTU1h\nb2+PcePGYd26dXjw4IHUqVInsWTJEgQGBiIkJKTZG7TWy8zMxO7du5GRkQGlUqmXHNavX4/s7Gzs\n378fJiYmepmzI/voo4+0N8X95JNPpE7HKLGWkr6xlhJRR8NmERGREZk1axauXr0KFxcX2NjYQAgB\njUaDoqIipKSkYODAgYiKioK7uztOnToldbqdVnR0NJKSklBWVoaBAwdi165dUqfUruLi4hAeHo7V\nq1e3ONbf3x87duxA37599RI7PT0d1dXVyMzMhJ2dnV7m7Ojee+89/PDDD1KnYdRYSw2DtfTxWEuJ\nqL2xWUREpGcqlQp+fn6dJoZMJoOtrS3GjRuHpKQkpKSk4O7du5gyZUqr/rtJja1atQrV1dUQQuDa\ntWuYPXu21Cm1uwkTJmDNmjUGjzt9+nQsWbIE3bt3N3hsal+spcRaajispUT0a2wWERHp2datW1FU\nVNRpY8yePRuhoaEoKiriV1yISDKspURERNJhs4iI2uRf//oXnn/+eSiVSlhbW8PT0xPl5eXa9du3\nb4e3tzcUCgUsLCwwYMAAfPjhhwB+fhzs+vXr4ebmBjMzM9jZ2SEgIAAXL17Ubv+nP/0JSqUSVlZW\nKCoqQmRkJBwdHXHp0iXU1dVh2bJlcHZ2hrm5OYYNG4bk5GSd96E1eYSHh8PU1LTB5d4LFy6EhYUF\nZDIZ7t+/DwBYtGgRIiMjkZ+fD5lMBldXV2zcuBEKhQL29vaYN28eHBwcoFAo4Ofnh6ysLL3EAIAD\nBw7A2toacXFxOh+DXwsNDQUAZGRkaJc1d7wTExNhYWEBpVKJ9PR0TJo0CdbW1nBycsIXX3zRYO7m\nXjP6OqdEXUFz9bMpR44cwdChQ2FjYwOFQgFPT098/fXX2vXNvfdaquWtwVraGGspERF1WoKI6D+S\nk5OFLmWhsrJSWFtbi7Vr1wqVSiXu3LkjZs6cKe7duyeEEGLDhg0CgFi9erUoLi4WJSUlYvPmzeL1\n118XQgixbNkyYWpqKrZv3y5KS0tFTk6OGDFihOjVq5e4c+eONs77778vAIiIiAjx17/+VcycOVP8\n9NNP4r333hNmZmZi165d4sGDByI6Olp069ZNnDx5Uqf9bm0er7/+uujTp0+DbdetWycAaPdZCCFm\nzZolXFxcGowLCwsTFhYW4sKFC+LRo0ciNzdXjBw5UlhZWYkbN27oJcZXX30lrKysRGxsbIv77OLi\nImxsbB67vry8XAAQTz31lHZZS8e7/jwdOnRIlJWViaKiIjFmzBhhYWEhampqhBAtv2b0dU5nz54t\nZs+erdM2xk7X939XYojXCwCRnJzc6vEt1c8rV64IAOLjjz/WbpOamipiYmJESUmJKC4uFj4+PqJn\nz55CiObfey29L1uLtbSxzl5LjbkutIWu7/eugq8Xoi4lhe9mItLS9Zf8+fPnBQDx1VdfNVpXU1Mj\nbG1txfjx4xssr62tFfHx8aKqqkpYWlqKkJCQButPnDghADT4I73+D2eVSqVdplKphFKpbLB9VVWV\nMDMzEwsWLGj1PuiSR1s/4Pz6A8XJkycFALFixQq9xNBFSx9whBBCJpMJW1tbIUTrjndT5ykhIUEA\nEHl5eUKI5l8z+jqnQrBZ9CSM+Y/8jtYsaql+CtF0s+jXVq1aJQCIoqKiZt97za1rLdbSx+vMtdSY\n60JbsFlERF1AirxdLlciIqMwaNAg2NvbY86cOYiIiEBoaCgGDBgAAMjJyUFpaSkmTpzYYJvu3bsj\nIiICp06dQmVlJby9vRusHzlyJExNTRt8paAply5dQlVVFTw8PLTLzM3N0bdv3wZfeWhJbm5um/Jo\nC29vbyiVSp3yNZSHDx9CCAFra2sAT368TU1NAQBqtRpA868ZfZ3Terdu3UJKSorO2xmrY8eOAYBR\nHrNbt27ByclJ6jS0WqqfrVX/6Ou6urpm33vNrWst1tKmdYVaChhnXWir+ppqTIxxn4m6MjaLiOiJ\nmZub49tvv8XixYsRFxeH2NhYBAUFISkpSXvfBFtb2ya3LS0tBQBYWlo2Wmdra4uKiopmYz98+BAA\n8MEHH+CDDz5osM7BwaHV+9DWPNrKzMwM9+7da9cYT+Ly5csAgGeeeQaA/o53c68ZfcWod/z4cQQH\nB+u8nbEz1mPWkZ6y1FL9fJx9+/Zh3bp1yM3NRXl5ubaxADT/3mtpXWuwljatK9RSwHjrQlvEx8cj\nPj5e6jSIiJ4Yb3BNRG3i7u6OvXv3orCwEFFRUUhOTsZHH32Efv36AYD2RqK/Vv8hqKkPEKWlpS3+\nl793794AgA0bNkAI0eBHl/9stTWPtlCr1e0e40kdOHAAADBp0iQA+jvewONfM/qMAfz84f/X8/Dn\n8T/1N7+VOg8pfjpSowhAi/WzKTdu3MCMGTPQt29fZGVloaysDGvXrm0w5nHvvZbWtQZradO6Qi0F\njLMutOUHAJKTkyXPw9A/vIk6UdfCZhERPbHCwkJcuHABwM9/AK9evRojRozAhQsXMGDAAPTo0QPf\nfPNNk9t6eHjA0tISp06darA8KysLNTU1eO6555qN/dRTT0GhUCA7O7tN+6BLHnK5vMF/6tsqMzMT\nQgj4+Pi0W4wncefOHWzYsAFOTk546623AOjveDf3mtFXDKLOrqX62ZRz585BrVZjwYIFGDRoEBQK\nBWQymXZ9c++95ta1FmtpY6ylRETUmbFZRERPrLCwEPPmzcPFixdRU1ODM2fO4Pr16/Dx8YGZmRmi\no6Nx+PBhhIeH4/bt29BoNKioqMCFCxegUCgQGRmJPXv24LPPPkN5eTnOnTuH+fPnw8HBAWFhYc3G\nVigUePPNN/HFF18gMTER5eXlqKurw61bt/Dvf/+71fugSx6urq4oKSlBWloa1Go17t27h+vXrzea\ns0ePHigsLERBQQEqKiq0H1g0Gg0ePHiA2tpa5OTkYNGiRXB2dtY+WrmtMTIyMnR63LMQApWVldBo\nNBBC4N69e0hOTsYLL7yA7t27Iy0tTXufDX0d7+ZeM/qKQdTZtVQ/m+Ls7AwAOHjwIB49eoQrV640\nuE9Qc++95ta1FmspaykREXUxgojoP3R9ikVBQYHw8/MTdnZ2onv37qJfv37i/fffF7W1tdoxmzZt\nEp6enkKhUAiFQiGGDx8uEhIShBBCaDQasW7dOjF48GBhYmIi7OzsxIwZM8SlS5e0269du1aYm5tr\nHz28fft27brq6moRFRUlnJ2dhVwuF7179xazZs0Subm5Ou13a/IQQoji4mIxfvx4oVAoxMCBA8Xv\nfvc78Yc//EEAEK6urtrHNp8+fVr0799fmJubi9GjR4s7d+6IsLAwYWJiIhwdHYVcLhfW1tYiICBA\n5Ofn6y3G/v37hZWVlVi5cuVj9/XLL78Uw4YNE0qlUpiamopu3boJANqn9Tz//PMiNjZWFBcXN9q2\nueOdkJAglEqlACAGDx4s8vPzxZYtW4S1tbUAIPr37y8uX77c4mtGX+eUT0PTnTE/xaajPQ2t3uPq\n55///GfRp08fAUBYWFiImTNnCiGEiIqKEj169BC2trYiMDBQbNq0SQAQLi4u4siRI49977WmlrcG\na2nXq6XGXBfa4kne710BXy9EXUqKTIj/fLGWiIxeSkoKgoODwbKgf/PmzUNqaiqKi4ulTsUoBAYG\nAgBSU1MlzqTzMOb3vyFeLzKZDMnJyQgKCmq3GMaAtdSwjLkutIWxvt/5eiHqUlL5NTQiIgOpq6uT\nOgUiok6PtZSIiKj9sVlERF3OxYsXIZPJWvwJCQmROlUiog6LtZSIiMh4sVlERF3OM88806pHvO7c\nudMg+URHRyMpKQllZWUYOHAgdu3aZZC4RIZ08OBBLFmyBBqNBjNmzICzszMUCgUcHR0xffp05OTk\nPPHcGo0GGzZsgJ+fX6N1X375JdauXcurTdoBaymR4bVHLV25cmWTjV4PDw/tGNZSIvo1NouIiNrZ\nqlWrUF1dDSEErl27htmzZ0udEpFeLV++HBs3bkR0dDQ0Gg2OHDmCzz//HCUlJTh69ChUKhVefPFF\nFBYW6jz3lStX8OKLL+Ldd99FVVVVo/XTpk2DQqGAv78/SktL9bE71EGxllJX1561tCWspUT0a2wW\nERERtQOVStXklTCdLUZL1qxZg507dyIlJQVWVlYAAF9fX4wePRpKpRIDBw5EXFwcysrK8Omnn+o0\n99mzZ7F48WLMnz8fXl5ejx0XERGBZ599FpMnT0ZtbW1bdoeIOhjW0rbXUgDYvn17o6sCz58/32AM\naykR/RKbRURERO1g69atKCoq6vQxmpOXl4elS5dixYoVUCgUAAC5XI69e/c2GDdo0CAAQH5+vk7z\nP/vss9i9ezdef/11mJmZNTs2JiYG2dnZiI+P1ykGEXVsrKX/9aS1VBespURUj80iIiIiAEIIrF+/\nHm5ubjAzM4OdnR0CAgJw8eJF7Zjw8HCYmpqib9++2mULFy6EhYUFZDIZ7t+/DwBYtGgRIiMjkZ+f\nD5lMBldXV2zcuBEKhQL29vaYN28eHBwcoFAo4Ofnh6ysLL3EAIADBw7A2toacXFx7Xq8AGDjxo0Q\nQmDatGnNjlOpVAAAa2vrdsvFzs4OY8eORXx8PB/bTCQh1lLdsZYSUUfEZhERERF+/m/qkiVL8P77\n76OoqAiHDx/GzZs3MWbMGNy9exfAz3/QBwUFNdguISEBK1asaLAsPj4eU6dOhYuLC4QQyMvLQ3h4\nOEJDQ1FVVYWIiAgUFBTg9OnTqK2txcsvv4ybN2+2OQbw38eKazQa/R2cx9i3bx+efvppKJXKZsed\nOHECADB69Oh2zWf48OG4ffs2zp49265xiOjxWEt1Z4haumTJEtjZ2cHU1BQDBw5EQEAATp482eRY\n1lIiAtgsIiIigkqlwvr16zFz5kzMmTMHNjY28PT0xCeffIL79+9jy5Yteosll8u1/3EfOnQoEhMT\nUVFRgaSkJL3MP2XKFJSXl2Pp0qV6me9xHj58iGvXrsHFxeWxY+7evYudO3ciIiICvr6+Lf7XvK0G\nDx4MADh37ly7xiGiprGW6s4QtfQ3v/kNvvzyS9y8eROVlZX44osvcOPGDYwdOxa5ubmNxrOWEhHA\nZhERERFyc3NRWVkJb2/vBstHjhwJU1PTBl9t0Ddvb28olcoGX9HoDIqKiiCEaPY/4b6+voiIiEBA\nQAAyMjJgYmLSrjnV51J/9QIRGRZrqe4MUUufeuopDB8+HJaWljA1NYWPjw+SkpKgUqmQkJDQaDxr\nKREBgFzqBIiIiKRW/5hgS0vLRutsbW1RUVHRrvHNzMxw7969do2hb48ePQKAZm88bW9vj61bt8Ld\n3d0gOZmbmzfIjYgMi7VUd1LVUk9PT3Tv3h2XL19utI61lIgAXllEREQEW1tbAGjyg0xpaSmcnJza\nLbZarW73GO2h/sNE/X09mtK7d2/tsTWEmpoaAP/NjYgMi7VUd1LVUo1GA41G02STirWUiAA2i4iI\niODh4QFLS0ucOnWqwfKsrCzU1NTgueee0y6Ty+VQq9V6i52ZmQkhBHx8fNotRnuwt7eHTCZDWVnZ\nY8fs3bsXjo6OBsupPpc+ffoYLCYR/Rdrqe4MUUsnTpzYaNnJkychhICvr2+jdaylRASwWURERASF\nQoHIyEjs2bMHn332GcrLy3Hu3DnMnz8fDg4OCAsL0451dXVFSUkJ0tLSoFarce/ePVy/fr3RnD16\n9EBhYSEKCgpQUVGh/cCi0Wjw4MED1NbWIicnB4sWLYKzszNCQ0P1EiMjI8Mgj3tWKpUYNGgQbt26\n1eT6vLw89OnTB8HBwY3WhYSEoE+fPjh9+rRec6rPxdPTU6/zElHrsJbqzhC19Pbt29i5cydKS0uh\nVqtx7Ngx/O///i+cnZ0xf/78RuNZS4kIYLOIiIgIALB8+XKsWrUKsbGx6NWrF8aOHYsBAwYgMzMT\nFhYW2nELFizA+PHj8dprr+Hpp5/Ghx9+qL1U39fXV/vY5vnz58Pe3h5Dhw7F5MmTUVJSAuDne0B4\nenrC3NwcY8aMwZAhQ/Ddd981+CpAW2MYypQpU5CbmwuVStVonRDisdvV1NSgqKgI6enpzc5//Phx\njB49Gv369UNWVhbOnj0LBwcHvPDCCzh8+HCj8SdPnoSjoyOGDRum+84QkV6wluquvWvpK6+8gg8+\n+ABOTk5QKpUICgrCCy+8gOPHj6Nnz56NxrOWEhEAyERzFYiIjEpKSgqCg4Ob/cOEqDMIDAwEAKSm\npkqcSUPz5s1DamoqiouLpU6lkSd5/+fl5cHNzQ1JSUmYM2dOq7fTaDQYN24cQkND8dZbbz1Juo0U\nFxfDyckJK1euRGRkpE7bGuL1IpPJkJx+yjDTAAACBElEQVScjKCgoHaLQaRvHfXvgo5cSwHd3+9d\npZZ21NcLET2RVF5ZREREZEDN3cS0s3F1dUVsbCxiY2NRWVnZqm3q6uqQlpaGiooKhISE6C2XmJgY\neHl5ITw8XG9zElHHxVrKWkpE7YvNIiIiInpiS5YsQWBgIEJCQpq9QWu9zMxM7N69GxkZGVAqlXrJ\nYf369cjOzsb+/fthYmKilzmJiAyJtZSIOho2i4iIiAwgOjoaSUlJKCsrw8CBA7Fr1y6pU9KbuLg4\nhIeHY/Xq1S2O9ff3x44dO9C3b1+9xE5PT0d1dTUyMzNhZ2enlzmJqONiLf0ZaykRtTe51AkQEREZ\ng1WrVmHVqlVSp9FuJkyYgAkTJhg87vTp0zF9+nSDxyUiabCWtg/WUiL6NV5ZREREREREREREWmwW\nERERERERERGRFptFRERERERERESkxWYRERERERERERFp8QbXRNRIYGCg1CkQtcnx48cB8LWsi1u3\nbgEwzmN2/Phx+Pj4tHucDRs2IDU1td3jEOmLMdeFtjLG93v964WIugaZEEJInQQRdQzHjh3D+vXr\npU6DiMjgfH198e6777bb/PywTUTGwtiaZERdVCqbRUREREREREREVC+V9ywiIiIiIiIiIiItNouI\niIiIiIiIiEiLzSIiIiIiIiIiItJis4iIiIiIiIiIiLT+H5GA37ptJAoKAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nYPdWikUQtBM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[keras.losses.MeanSquaredError(),\n",
        "          keras.losses.CategoricalCrossentropy()])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H1Wy0bb4RZ4E",
        "colab_type": "text"
      },
      "source": [
        "If we only passed a single loss function to the model, the same loss function would be applied to every output, which is not appropriate here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wcqCh4VURdEL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss={'score_output': keras.losses.MeanSquaredError(),\n",
        "          'class_output': keras.losses.CategoricalCrossentropy()},\n",
        "    metrics={'score_output': [keras.metrics.MeanAbsolutePercentageError(),\n",
        "                              keras.metrics.MeanAbsoluteError()],\n",
        "             'class_output': [keras.metrics.CategoricalAccuracy()]})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-KFF01QERkZ8",
        "colab_type": "text"
      },
      "source": [
        "You could also chose not to compute a loss for certain outputs, if these outputs meant for prediction but not for training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5Sfv_CX1RoT7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# List loss version\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[None, keras.losses.CategoricalCrossentropy()])\n",
        "\n",
        "# Or dict loss version\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss={'class_output': keras.losses.CategoricalCrossentropy()})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "luiBkX1ARtmE",
        "colab_type": "code",
        "outputId": "5027bba6-24af-4797-ca08-981c57bed732",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[keras.losses.MeanSquaredError(),\n",
        "          keras.losses.CategoricalCrossentropy()])\n",
        "\n",
        "# Generate dummy Numpy data\n",
        "img_data = np.random.random_sample(size=(100, 32, 32, 3))\n",
        "ts_data = np.random.random_sample(size=(100, 20, 10))\n",
        "score_targets = np.random.random_sample(size=(100, 1))\n",
        "class_targets = np.random.random_sample(size=(100, 5))\n",
        "\n",
        "# Fit on lists\n",
        "model.fit([img_data, ts_data], [score_targets, class_targets],\n",
        "          batch_size=32,\n",
        "          epochs=3)\n",
        "\n",
        "# Alernatively, fit on dicts\n",
        "model.fit({'img_input': img_data, 'ts_input': ts_data},\n",
        "          {'score_output': score_targets, 'class_output': class_targets},\n",
        "          batch_size=32,\n",
        "          epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 100 samples\n",
            "Epoch 1/3\n",
            "100/100 [==============================] - 1s 9ms/sample - loss: 5.4007 - score_output_loss: 0.7716 - class_output_loss: 4.3417\n",
            "Epoch 2/3\n",
            "100/100 [==============================] - 0s 573us/sample - loss: 5.0339 - score_output_loss: 0.4582 - class_output_loss: 4.4898\n",
            "Epoch 3/3\n",
            "100/100 [==============================] - 0s 522us/sample - loss: 4.8682 - score_output_loss: 0.3535 - class_output_loss: 4.3451\n",
            "Train on 100 samples\n",
            "Epoch 1/3\n",
            "100/100 [==============================] - 0s 655us/sample - loss: 4.7594 - score_output_loss: 0.2209 - class_output_loss: 4.6201\n",
            "Epoch 2/3\n",
            "100/100 [==============================] - 0s 518us/sample - loss: 4.7109 - score_output_loss: 0.1794 - class_output_loss: 4.4980\n",
            "Epoch 3/3\n",
            "100/100 [==============================] - 0s 527us/sample - loss: 4.6708 - score_output_loss: 0.1501 - class_output_loss: 4.6839\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f5143fed978>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5rDdJGxcRuA8",
        "colab_type": "code",
        "outputId": "28cb2b77-83da-4909-c578-11db4c95bb5c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 154
        }
      },
      "source": [
        "train_dataset = tf.data.Dataset.from_tensor_slices(\n",
        "    ({'img_input': img_data, 'ts_input': ts_data},\n",
        "     {'score_output': score_targets, 'class_output': class_targets}))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "model.fit(train_dataset, epochs=3)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "2/2 [==============================] - 1s 326ms/step - loss: 4.6314 - score_output_loss: 0.1546 - class_output_loss: 4.4768\n",
            "Epoch 2/3\n",
            "2/2 [==============================] - 0s 32ms/step - loss: 4.6047 - score_output_loss: 0.1375 - class_output_loss: 4.5333\n",
            "Epoch 3/3\n",
            "2/2 [==============================] - 0s 31ms/step - loss: 4.6573 - score_output_loss: 0.1184 - class_output_loss: 4.5021\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f5144a94da0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sPdaHO8tR59U",
        "colab_type": "text"
      },
      "source": [
        "# Callbacks classes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QO-zpw9aSQFs",
        "colab_type": "text"
      },
      "source": [
        "Examples of why one would need to use callbacks classes\n",
        "\n",
        "- Doing validation at different points during training (beyond the built-in per-epoch validation)\n",
        "- Checkpointing the model at regular intervals or when it exceeds a certain accuracy threshold\n",
        "- Changing the learning rate of the model when training seems to be plateauing\n",
        "- Doing fine-tuning of the top layers when training seems to be plateauing\n",
        "- Sending email or instant message notifications when training ends or where a certain performance threshold is exceeded\n",
        "- Etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AKW94B9iTObJ",
        "colab_type": "text"
      },
      "source": [
        "## Early stopping callback examples"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LoXeODOrS3UD",
        "colab_type": "code",
        "outputId": "bb5e5250-e29b-47c0-f084-d2c8ca7bbd79",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "callbacks = [\n",
        "    keras.callbacks.EarlyStopping(\n",
        "        # Stop training when `val_loss` is no longer improving\n",
        "        monitor='val_loss',\n",
        "        # \"no longer improving\" being defined as \"no better than 1e-2 less\"\n",
        "        min_delta=1e-2,\n",
        "        # \"no longer improving\" being further defined as \"for at least 2 epochs\"\n",
        "        patience=2,\n",
        "        verbose=1)\n",
        "]\n",
        "model.fit(x_train, y_train,\n",
        "          epochs=20,\n",
        "          batch_size=64,\n",
        "          callbacks=callbacks,\n",
        "          validation_split=0.2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 40000 samples, validate on 10000 samples\n",
            "Epoch 1/20\n",
            "40000/40000 [==============================] - 3s 65us/sample - loss: 0.3676 - sparse_categorical_accuracy: 0.8978 - val_loss: 0.2194 - val_sparse_categorical_accuracy: 0.9346\n",
            "Epoch 2/20\n",
            "40000/40000 [==============================] - 2s 58us/sample - loss: 0.1722 - sparse_categorical_accuracy: 0.9485 - val_loss: 0.1766 - val_sparse_categorical_accuracy: 0.9462\n",
            "Epoch 3/20\n",
            "40000/40000 [==============================] - 2s 59us/sample - loss: 0.1232 - sparse_categorical_accuracy: 0.9620 - val_loss: 0.1644 - val_sparse_categorical_accuracy: 0.9507\n",
            "Epoch 4/20\n",
            "40000/40000 [==============================] - 2s 60us/sample - loss: 0.0971 - sparse_categorical_accuracy: 0.9710 - val_loss: 0.1390 - val_sparse_categorical_accuracy: 0.9591\n",
            "Epoch 5/20\n",
            "40000/40000 [==============================] - 2s 53us/sample - loss: 0.0799 - sparse_categorical_accuracy: 0.9756 - val_loss: 0.1316 - val_sparse_categorical_accuracy: 0.9631\n",
            "Epoch 6/20\n",
            "40000/40000 [==============================] - 2s 54us/sample - loss: 0.0663 - sparse_categorical_accuracy: 0.9797 - val_loss: 0.1298 - val_sparse_categorical_accuracy: 0.9628\n",
            "Epoch 00006: early stopping\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f5144cde160>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3g8axh4RTGLK",
        "colab_type": "text"
      },
      "source": [
        "## Types of callbacks\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KndEc-ffTv1D",
        "colab_type": "text"
      },
      "source": [
        "- ```ModelCheckpoint```: Periodically save the model.\n",
        "- ```EarlyStopping```: Stop training when training is no longer improving the validation metrics.\n",
        "- ```TensorBoard```: periodically write model logs that can be visualized in TensorBoard (more details in the section \"Visualization\").\n",
        "- ```CSVLogger```: streams loss and metrics data to a CSV file.\n",
        "- etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "amauut46T4jr",
        "colab_type": "text"
      },
      "source": [
        "## Custom callbacks\n",
        "\n",
        "You can create a custom callback by extending the base class ```keras.callbacks.Callback```. A callback has access to its associated model through the class property ```self.model```."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WeB4uVk7UC5u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class LossHistory(keras.callbacks.Callback):\n",
        "\n",
        "    def on_train_begin(self, logs):\n",
        "        self.losses = []\n",
        "\n",
        "    def on_batch_end(self, batch, logs):\n",
        "        self.losses.append(logs.get('loss'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c5uslzyrUMBL",
        "colab_type": "text"
      },
      "source": [
        "## Checkpointing callbacks"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "px_WxPZPURYL",
        "colab_type": "code",
        "outputId": "cf331f33-0b6f-4fe5-c875-37031ef8e20e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        }
      },
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "callbacks = [\n",
        "    keras.callbacks.ModelCheckpoint(\n",
        "        filepath='mymodel_{epoch}.h5',\n",
        "        # Path where to save the model\n",
        "        # The two parameters below mean that we will overwrite\n",
        "        # the current checkpoint if and only if\n",
        "        # the `val_loss` score has improved.\n",
        "        save_best_only=True,\n",
        "        monitor='val_loss',\n",
        "        verbose=1)\n",
        "]\n",
        "model.fit(x_train, y_train,\n",
        "          epochs=3,\n",
        "          batch_size=64,\n",
        "          callbacks=callbacks,\n",
        "          validation_split=0.2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 40000 samples, validate on 10000 samples\n",
            "Epoch 1/3\n",
            "39872/40000 [============================>.] - ETA: 0s - loss: 0.3746 - sparse_categorical_accuracy: 0.8938\n",
            "Epoch 00001: val_loss improved from inf to 0.24213, saving model to mymodel_1.h5\n",
            "40000/40000 [==============================] - 3s 68us/sample - loss: 0.3744 - sparse_categorical_accuracy: 0.8939 - val_loss: 0.2421 - val_sparse_categorical_accuracy: 0.9273\n",
            "Epoch 2/3\n",
            "39168/40000 [============================>.] - ETA: 0s - loss: 0.1758 - sparse_categorical_accuracy: 0.9473\n",
            "Epoch 00002: val_loss improved from 0.24213 to 0.18390, saving model to mymodel_2.h5\n",
            "40000/40000 [==============================] - 2s 57us/sample - loss: 0.1764 - sparse_categorical_accuracy: 0.9471 - val_loss: 0.1839 - val_sparse_categorical_accuracy: 0.9420\n",
            "Epoch 3/3\n",
            "39872/40000 [============================>.] - ETA: 0s - loss: 0.1284 - sparse_categorical_accuracy: 0.9622\n",
            "Epoch 00003: val_loss improved from 0.18390 to 0.15560, saving model to mymodel_3.h5\n",
            "40000/40000 [==============================] - 2s 54us/sample - loss: 0.1283 - sparse_categorical_accuracy: 0.9622 - val_loss: 0.1556 - val_sparse_categorical_accuracy: 0.9513\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f514368d9b0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4i2A_Eb1UVP2",
        "colab_type": "code",
        "outputId": "d3cc9cb2-7ac5-4a8f-ea4f-23c6e52be733",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 50
        }
      },
      "source": [
        "!ls"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "multi_input_and_output_model.png  mymodel_2.h5\tsample_data\n",
            "mymodel_1.h5\t\t\t  mymodel_3.h5\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQG7tJZeUWKL",
        "colab_type": "text"
      },
      "source": [
        "# Learning rate decay methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2S12CcmLUseb",
        "colab_type": "text"
      },
      "source": [
        "A common pattern when training deep learning models is to gradually reduce the learning as training progresses. This is generally known as \"learning rate decay\".\n",
        "\n",
        "The learning decay schedule could be static (fixed in advance, as a function of the current epoch or the current batch index), or dynamic (responding to the current behavior of the model, in particular the validation loss).\n",
        "\n",
        "Passing a schedule to an optimizer\n",
        "You can easily use a static learning rate decay schedule by passing a schedule object as the learning_rate argument in your optimizer:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g7mGY39xUw8L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "initial_learning_rate = 0.1\n",
        "lr_schedule = keras.optimizers.schedules.ExponentialDecay(\n",
        "    initial_learning_rate,\n",
        "    decay_steps=100000,\n",
        "    decay_rate=0.96,\n",
        "    staircase=True)\n",
        "\n",
        "optimizer = keras.optimizers.RMSprop(learning_rate=lr_schedule)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RgU9alVEUyzr",
        "colab_type": "text"
      },
      "source": [
        "## Main types of schedules\n",
        "\n",
        "Several built-in schedules are available: ```ExponentialDecay```, ```PiecewiseConstantDecay```, ```PolynomialDecay```, and ```InverseTimeDecay```."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFTCAaWNU32D",
        "colab_type": "text"
      },
      "source": [
        "## Using callbacks and learning rate decay\n",
        "\n",
        "Using callbacks to implement a dynamic learning rate schedule\n",
        "A dynamic learning rate schedule (for instance, decreasing the learning rate when the validation loss is no longer improving) cannot be achieved with these schedule objects since the optimizer does not have access to validation metrics.\n",
        "\n",
        "However, callbacks do have access to all metrics, including validation metrics! You can thus achieve this pattern by using a callback that modifies the current learning rate on the optimizer. In fact, this is even built-in as the ReduceLROnPlateau callback."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f_-hw8GaVFKP",
        "colab_type": "text"
      },
      "source": [
        "# My own Training (GradientTape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osG0gZ7yVSf6",
        "colab_type": "text"
      },
      "source": [
        "Here is an example on how to build your own training loop completely from scratch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zOovRG7FVn1T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get the model.\n",
        "inputs = keras.Input(shape=(784,), name='digits')\n",
        "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
        "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
        "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "# Instantiate an optimizer.\n",
        "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
        "# Instantiate a loss function.\n",
        "loss_fn = keras.losses.SparseCategoricalCrossentropy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "67JPZzl7VqVk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Prepare the training dataset.\n",
        "batch_size = 64\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lql-ItnuVs7b",
        "colab_type": "text"
      },
      "source": [
        "Now we iterate over epochs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XtlRiEKpVwE6",
        "colab_type": "code",
        "outputId": "fcc59f9d-4e4e-4c6f-849b-aece609eadaf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 577
        }
      },
      "source": [
        "# Iterate over epochs.\n",
        "for epoch in range(3):\n",
        "  print('Start of epoch %d' % (epoch,))\n",
        "\n",
        "  # Iterate over the batches of the dataset.\n",
        "  for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
        "\n",
        "    # Open a GradientTape to record the operations run\n",
        "    # during the forward pass, which enables autodifferentiation.\n",
        "    with tf.GradientTape() as tape:\n",
        "\n",
        "      # Run the forward pass of the layer.\n",
        "      # The operations that the layer applies\n",
        "      # to its inputs are going to be recorded\n",
        "      # on the GradientTape.\n",
        "      logits = model(x_batch_train)  # Logits for this minibatch\n",
        "\n",
        "      # Compute the loss value for this minibatch.\n",
        "      loss_value = loss_fn(y_batch_train, logits)\n",
        "\n",
        "    # Use the gradient tape to automatically retrieve\n",
        "    # the gradients of the trainable variables with respect to the loss.\n",
        "    grads = tape.gradient(loss_value, model.trainable_weights)\n",
        "\n",
        "    # Run one step of gradient descent by updating\n",
        "    # the value of the variables to minimize the loss.\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
        "\n",
        "    # Log every 200 batches.\n",
        "    if step % 200 == 0:\n",
        "        print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))\n",
        "        print('Seen so far: %s samples' % ((step + 1) * 64))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Start of epoch 0\n",
            "WARNING:tensorflow:From /tensorflow-2.0.0-rc0/python3.6/tensorflow_core/python/ops/math_grad.py:1394: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Training loss (for one batch) at step 0: 2.291839599609375\n",
            "Seen so far: 64 samples\n",
            "Training loss (for one batch) at step 200: 2.242462158203125\n",
            "Seen so far: 12864 samples\n",
            "Training loss (for one batch) at step 400: 2.1596622467041016\n",
            "Seen so far: 25664 samples\n",
            "Training loss (for one batch) at step 600: 2.1509897708892822\n",
            "Seen so far: 38464 samples\n",
            "Start of epoch 1\n",
            "Training loss (for one batch) at step 0: 1.9326403141021729\n",
            "Seen so far: 64 samples\n",
            "Training loss (for one batch) at step 200: 1.9350440502166748\n",
            "Seen so far: 12864 samples\n",
            "Training loss (for one batch) at step 400: 1.8397014141082764\n",
            "Seen so far: 25664 samples\n",
            "Training loss (for one batch) at step 600: 1.7012557983398438\n",
            "Seen so far: 38464 samples\n",
            "Start of epoch 2\n",
            "Training loss (for one batch) at step 0: 1.6696522235870361\n",
            "Seen so far: 64 samples\n",
            "Training loss (for one batch) at step 200: 1.647040605545044\n",
            "Seen so far: 12864 samples\n",
            "Training loss (for one batch) at step 400: 1.5034797191619873\n",
            "Seen so far: 25664 samples\n",
            "Training loss (for one batch) at step 600: 1.260719895362854\n",
            "Seen so far: 38464 samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mcdEmkJ2WQVr",
        "colab_type": "text"
      },
      "source": [
        "## Using ```metrics``` in self-developed loops"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qsg9DPndW1li",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get model\n",
        "inputs = keras.Input(shape=(784,), name='digits')\n",
        "x = layers.Dense(64, activation='relu', name='dense_1')(inputs)\n",
        "x = layers.Dense(64, activation='relu', name='dense_2')(x)\n",
        "outputs = layers.Dense(10, activation='softmax', name='predictions')(x)\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "# Instantiate an optimizer to train the model.\n",
        "optimizer = keras.optimizers.SGD(learning_rate=1e-3)\n",
        "# Instantiate a loss function.\n",
        "loss_fn = keras.losses.SparseCategoricalCrossentropy()\n",
        "\n",
        "# Prepare the metrics.\n",
        "train_acc_metric = keras.metrics.SparseCategoricalAccuracy()\n",
        "val_acc_metric = keras.metrics.SparseCategoricalAccuracy()\n",
        "\n",
        "# Prepare the training dataset.\n",
        "batch_size = 64\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n",
        "\n",
        "# Prepare the validation dataset.\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "val_dataset = val_dataset.batch(64)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G12oPLh6W5sj",
        "colab_type": "code",
        "outputId": "d630cf63-a18b-4bf5-9d72-4335461e3d97",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 179
        }
      },
      "source": [
        "\n",
        "# Iterate over epochs.\n",
        "for epoch in range(3):\n",
        "  print('Start of epoch %d' % (epoch,))\n",
        "\n",
        "  # Iterate over the batches of the dataset.\n",
        "  for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):\n",
        "    with tf.GradientTape() as tape:\n",
        "      logits = model(x_batch_train)\n",
        "      loss_value = loss_fn(y_batch_train, logits)\n",
        "    grads = tape.gradient(loss_value, model.trainable_weights)\n",
        "    optimizer.apply_gradients(zip(grads, model.trainable_weights))\n",
        "\n",
        "    # Update training metric.\n",
        "    train_acc_metric(y_batch_train, logits)\n",
        "\n",
        "    # Log every 200 batches.\n",
        "    #if step % 200 == 0:\n",
        "    #    print('Training loss (for one batch) at step %s: %s' % (step, float(loss_value)))\n",
        "    #    print('Seen so far: %s samples' % ((step + 1) * 64))\n",
        "\n",
        "  # Display metrics at the end of each epoch.\n",
        "  train_acc = train_acc_metric.result()\n",
        "  print('Training acc over epoch: %s' % (float(train_acc),))\n",
        "  # Reset training metrics at the end of each epoch\n",
        "  train_acc_metric.reset_states()\n",
        "\n",
        "  # Run a validation loop at the end of each epoch.\n",
        "  for x_batch_val, y_batch_val in val_dataset:\n",
        "    val_logits = model(x_batch_val)\n",
        "    # Update val metrics\n",
        "    val_acc_metric(y_batch_val, val_logits)\n",
        "  val_acc = val_acc_metric.result()\n",
        "  val_acc_metric.reset_states()\n",
        "  print('Validation acc: %s' % (float(val_acc),))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Start of epoch 0\n",
            "Training acc over epoch: 0.7538999915122986\n",
            "Validation acc: 0.805400013923645\n",
            "Start of epoch 1\n",
            "Training acc over epoch: 0.7954400181770325\n",
            "Validation acc: 0.8337000012397766\n",
            "Start of epoch 2\n",
            "Training acc over epoch: 0.8209599852561951\n",
            "Validation acc: 0.8507999777793884\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hCONEvqhW6Wa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}