smart_warehouse2x3.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "smart_warehouse.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNGDG18Y/YnHL5xcOyNS8Mk",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/StefanFischer/SAKI-Project4/blob/main/smart_warehouse2x3.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zaXA3dWwjHkM"
      },
      "source": [
        "# Smart Warehouse Implementation\n",
        "\n",
        "The purpose of this exercise is to develop an algorithm that optimizes the route that a robot takes for pick-up and storage of items in a warehouse.\n",
        "\n",
        "There are the following constraints :\n",
        "*   Size of warehouse is 2x2 or 2x3\n",
        "*   There is separate start/stop position outside the storage space\n",
        "*   The first position the robot can move is (0,0)\n",
        "*   Robots can move to adjacent fields (but not diagonally)\n",
        "*   There are three types of items, identified by color (white, blue, red)\n",
        "\n",
        "The problem can be formulated as a Markov Decision Process consisting of:\n",
        "*   Transition Probability Matrix (TPM)\n",
        "*   Reward Matrix\n",
        "*   States\n",
        "*   Actions\n",
        "\n",
        "The goal is to create a policy, which returns for each state the action which is optimal to take."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YHxy2zZ--_zQ",
        "outputId": "a4230287-44d7-44e8-81e6-9d5b00cd6915"
      },
      "source": [
        "pip install pymdptoolbox"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: pymdptoolbox in /usr/local/lib/python3.7/dist-packages (4.0b3)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from pymdptoolbox) (1.4.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pymdptoolbox) (1.19.5)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ixQEZTzINLdu"
      },
      "source": [
        "import numpy as np\n",
        "import mdptoolbox\n",
        "import matplotlib.pyplot as plt\n",
        "import csv\n",
        "from scipy.sparse import csr_matrix"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ew2iboHckhil"
      },
      "source": [
        "**Configuration of the MDP problem**\n",
        "\n",
        "The states consist of the storage state of each shelf, which can be filled with one of the three items or be empty.\n",
        "Furthermore, there is the input given what item should be stored or restored.\n",
        "\n",
        "Besides this also the rewards/penalties (negative rewards) are defined here:\n",
        "*   distance penalties: the distance the robot has to cover to store or restore an item\n",
        "*   unable to store/restore item: penelize robot if he can not store/restore a item\n",
        "*   refusing item penalty: if there is a place left in storage but robot rejects item, penalize it\n",
        "\n",
        "Here, the discount is also configured, which defines if the agent should take future return into account(value: ]0,1[ )\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EYrlzT_Y-Pgk"
      },
      "source": [
        "\n",
        "warehouse_size = 6\n",
        "storage_state = ['0','r','b','w'] #0 = empty, r = red, b = blue, w = white\n",
        "requirements = ['sr', 'sb', 'sw', 'rr', 'rb', 'rw'] #sb/sr/sw = store_blue/red/white, rb/rr/rw = restore blue/red/white\n",
        "\n",
        "reward_con_in = -50\n",
        "reward_con_out = -50\n",
        "discount = 0.999\n",
        "penalty_refusing = -10000\n",
        "prints = False\n",
        "\n",
        "\n",
        "reward_pro = list()\n",
        "if warehouse_size == 4:\n",
        "    reward_pro = [-1, -2, -2, -3]\n",
        "elif warehouse_size == 6:\n",
        "    reward_pro = [-1, -2, -2, -3, -3, -4]\n",
        "elif warehouse_size == 9:\n",
        "    reward_pro = [-1, -2, -2, -2, -3, -3, -3, -4, -4, -5]\n",
        "\n",
        "\n",
        "path_to_trainings_file = \"./Exercise 4 - Reinforcement Learning Data - warehousetraining.txt\"\n"
      ],
      "execution_count": 61,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VJkgo7u4mgfj"
      },
      "source": [
        "**Create all possible states**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XarKwSdW-T3K"
      },
      "source": [
        "\n",
        "def get_states(warehouse_size, storage_state, requirements):\n",
        "    states = []\n",
        "\n",
        "    if warehouse_size == 4:\n",
        "        for x1 in storage_state:\n",
        "            for x2 in storage_state:\n",
        "                for x3 in storage_state:\n",
        "                    for x4 in storage_state:\n",
        "                        for r in requirements:\n",
        "                            st = (x1, x2, x3, x4, r)\n",
        "                            states.append(st)\n",
        "\n",
        "    if warehouse_size == 6:\n",
        "        for x1 in storage_state:\n",
        "            for x2 in storage_state:\n",
        "                for x3 in storage_state:\n",
        "                    for x4 in storage_state:\n",
        "                        for x5 in storage_state:\n",
        "                            for x6 in storage_state:\n",
        "                                for r in requirements:\n",
        "                                    st = (x1, x2, x3, x4, x5, x6, r)\n",
        "                                    states.append(st)\n",
        "\n",
        "    if warehouse_size == 9:\n",
        "        for x1 in storage_state:\n",
        "            for x2 in storage_state:\n",
        "                for x3 in storage_state:\n",
        "                    for x4 in storage_state:\n",
        "                        for x5 in storage_state:\n",
        "                            for x6 in storage_state:\n",
        "                                for x7 in storage_state:\n",
        "                                    for x8 in storage_state:\n",
        "                                        for x9 in storage_state:\n",
        "                                            for r in requirements:\n",
        "                                                st = (x1, x2, x3, x4, x5, x6, x7, x8, x9, r)\n",
        "                                                states.append(st)\n",
        "    return states\n",
        "\n",
        "states = get_states(warehouse_size, storage_state, requirements)\n",
        "\n"
      ],
      "execution_count": 62,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8wmt779qmkxl"
      },
      "source": [
        "**Compute the probability of each item getting stored/restored**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iJLR77rY-ZyH",
        "outputId": "5d865c18-0085-4775-a969-507be0511604"
      },
      "source": [
        "\n",
        "def compute_item_probability(path_to_file):\n",
        "    file = open(path_to_file, 'r')\n",
        "    count = 0\n",
        "    red_count = 0\n",
        "    blue_count = 0\n",
        "    white_count = 0\n",
        "\n",
        "    # Assumption: all stored items are also restored later in time\n",
        "    for line in file:\n",
        "        count += 1\n",
        "        if \"restore\" in line:\n",
        "            if \"red\" in line:\n",
        "                red_count += 1\n",
        "            if \"white\" in line:\n",
        "                white_count += 1\n",
        "            if \"blue\" in line:\n",
        "                blue_count += 1\n",
        "\n",
        "    # Closing files\n",
        "    file.close()\n",
        "\n",
        "    return red_count / count, blue_count / count, white_count / count\n",
        "\n",
        "probRed, probBlue, probWhite = compute_item_probability(path_to_trainings_file)\n",
        "print(probRed, probBlue, probWhite)\n"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.24686157912124215 0.12528906508093823 0.1278493557978196\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rn5RT3GNmqVd"
      },
      "source": [
        "**Init the penalty matrix for rejecting items while storage is free**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DfsWfqvrLq24"
      },
      "source": [
        "# penalties for the agent if he refuses to store a item\n",
        "penalties_mat = np.zeros((len(states), warehouse_size))"
      ],
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sEVreRWUm10t"
      },
      "source": [
        "**Computation of the TPM and filling the penalty matrix**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fGNm8Pm7-cL3"
      },
      "source": [
        "\n",
        "def get_tpm(warehouse_size, storage_state_size, requirements_size, probRed, probBlue, probWhite, penalties_mat):\n",
        "    trans_prob = []\n",
        "\n",
        "    probRed = round(probRed, 4)\n",
        "    probWhite = round(probWhite, 4)\n",
        "    probBlue = round(probBlue, 4)\n",
        "\n",
        "    probs = {'sr': probRed, 'sw': probWhite, 'sb': probBlue,\n",
        "             'rr': probRed, 'rw': probWhite, 'rb': probBlue}\n",
        "\n",
        "    # actions to take: as action the specific shelf is used for storing or restoring\n",
        "    # iterate over all different shelfes in warehouse\n",
        "    for idx_tr in range(warehouse_size):\n",
        "        trans = np.zeros((storage_state_size ** warehouse_size * requirements_size, storage_state_size ** warehouse_size * requirements_size), dtype=np.float16)\n",
        "        for idx_st, state in enumerate(states):\n",
        "            for idx_ne, next_state in enumerate(states):\n",
        "                # store\n",
        "                if state[-1][0] == 's':\n",
        "                    # store possible\n",
        "                    if state[idx_tr] == '0':\n",
        "                        trans_state = list(state)\n",
        "                        # apply action\n",
        "                        trans_state[idx_tr] = state[-1][1]\n",
        "                        if tuple(trans_state[:-1]) == next_state[:-1]:\n",
        "                            trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                        continue\n",
        "                    # store impossible\n",
        "                    if state[:-1] == next_state[:-1]:\n",
        "                        trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "\n",
        "                        if '0' in state[:-1]:\n",
        "                            penalties_mat[idx_st, idx_tr] = penalty_refusing\n",
        "                        continue\n",
        "                # restore\n",
        "                if state[-1][0] == 'r':\n",
        "                    if state[-1][1] == 'r':\n",
        "                        # restore possible\n",
        "                        if state[idx_tr] == 'r':\n",
        "                            trans_state = list(state)\n",
        "                            # apply action\n",
        "                            trans_state[idx_tr] = '0'\n",
        "                            if tuple(trans_state[:-1]) == next_state[:-1]:\n",
        "                                trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "                        # restore impossible: stay in state\n",
        "                        if state[:-1] == next_state[:-1]:\n",
        "                            trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "                    if state[-1][1] == 'w':\n",
        "                        # restore possible\n",
        "                        if state[idx_tr] == 'w':\n",
        "                            trans_state = list(state)\n",
        "                            # apply action\n",
        "                            trans_state[idx_tr] = '0'\n",
        "                            if tuple(trans_state[:-1]) == next_state[:-1]:\n",
        "                                trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "                        # restore impossible: stay in state\n",
        "                        if state[:-1] == next_state[:-1]:\n",
        "                            trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "                    if state[-1][1] == 'b':\n",
        "                        # restore possible\n",
        "                        if state[idx_tr] == 'b':\n",
        "                            trans_state = list(state)\n",
        "                            # apply action\n",
        "                            trans_state[idx_tr] = '0'\n",
        "                            if tuple(trans_state[:-1]) == next_state[:-1]:\n",
        "                                trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "                        # restore impossible: stay in state\n",
        "                        if state[:-1] == next_state[:-1]:\n",
        "                            trans[idx_st][idx_ne] = probs[next_state[-1]]\n",
        "                            continue\n",
        "        dense_matrix = csr_matrix(trans)\n",
        "        trans_prob.append(dense_matrix)\n",
        "\n",
        "    return trans_prob, penalties_mat\n",
        "\n",
        "trans_prob, penalties_mat = get_tpm(warehouse_size, len(storage_state), len(requirements), probRed, probBlue, probWhite, penalties_mat)#"
      ],
      "execution_count": 65,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bx3sWqhCnAMM"
      },
      "source": [
        "**Filling the reward matrix and add penalty matrix on top of it**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yiww935w-gv_",
        "outputId": "01d5eb3f-b9d2-42e0-b584-03c69916746d"
      },
      "source": [
        "\n",
        "def store_possible(state, action):\n",
        "    return True if state[action] == '0' else False\n",
        "\n",
        "\n",
        "def restore_possible(state, action):\n",
        "    return True if state[action] == state[-1][1] else False\n",
        "\n",
        "\n",
        "def get_rewards(warehouse_size, requirements_size, storage_state_size, reward_pro, reward_con_in, reward_con_out):\n",
        "    reward_mat = np.zeros((storage_state_size ** warehouse_size * requirements_size, warehouse_size))\n",
        "\n",
        "    for idx_st, state in enumerate(states):\n",
        "        # store\n",
        "        if state[-1][0] == 's':\n",
        "            for action in range(warehouse_size):\n",
        "                reward_mat[idx_st][action] = reward_pro[action] if store_possible(state, action) else reward_con_in\n",
        "        # restore\n",
        "        else:\n",
        "            for action in range(warehouse_size):\n",
        "                reward_mat[idx_st][action] = reward_pro[action] if restore_possible(state, action) else reward_con_out\n",
        "    return reward_mat\n",
        "\n",
        "\n",
        "reward_mat = get_rewards(warehouse_size, len(requirements), len(storage_state), reward_pro, reward_con_in, reward_con_out)\n",
        "print(reward_mat)\n",
        "\n",
        "# add penalty for refusing to store\n",
        "reward_mat = reward_mat + penalties_mat"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ -1.  -2.  -2.  -3.  -3.  -4.]\n",
            " [ -1.  -2.  -2.  -3.  -3.  -4.]\n",
            " [ -1.  -2.  -2.  -3.  -3.  -4.]\n",
            " ...\n",
            " [-50. -50. -50. -50. -50. -50.]\n",
            " [-50. -50. -50. -50. -50. -50.]\n",
            " [ -1.  -2.  -2.  -3.  -3.  -4.]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cAGFmwC9nL4D"
      },
      "source": [
        "**Solving the MDP**\n",
        "\n",
        "using the py-mdptoolbox solve the MDP with value-iteration"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XZ7GBePQ-jSf"
      },
      "source": [
        "# add penalty for refusing to store\n",
        "reward_mat = reward_mat + penalties_mat\n",
        "\n",
        "# Define the MDP with toolbox\n",
        "mdpresultValue = mdptoolbox.mdp.ValueIteration(trans_prob,reward_mat,discount=discount)\n",
        "\n",
        "# Run the MDP\n",
        "mdpresultValue.run()\n"
      ],
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "deOKsuX9oH3r"
      },
      "source": [
        "**Define Environment**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ueqlSzkcoTq0"
      },
      "source": [
        "class Environment:\n",
        "    def __init__(self, warehouse_size):\n",
        "        self.warehouse_size = warehouse_size\n",
        "        self.fields = []\n",
        "        for i in range(warehouse_size):\n",
        "            self.fields.append(0)"
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EtCOWdEupgWE"
      },
      "source": [
        "**Define greedy robot**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hOWyjHUeoXyV"
      },
      "source": [
        "class Greedy:\n",
        "    def __init__(self, warehouse_size, reward_mat):\n",
        "        self.env = Environment(warehouse_size)\n",
        "        self.reward = 0\n",
        "        self.refused = 0\n",
        "        self.colours = {'blue': 1, 'white': 2, 'red': 3}\n",
        "        self.rewards = reward_pro[:warehouse_size]\n",
        "\n",
        "    def reformat_env_printing(self, fields):\n",
        "        printout = []\n",
        "        for f in fields:\n",
        "          if f == 0:\n",
        "            printout.append(0)\n",
        "          elif f == 1:\n",
        "            printout.append('b')\n",
        "          elif f == 2:\n",
        "            printout.append('w')\n",
        "          elif f == 3:\n",
        "            printout.append('r')\n",
        "        return printout\n",
        "\n",
        "    def next_action(self, action, colour):\n",
        "        if action is 'store':\n",
        "            position = np.argmin(self.env.fields)\n",
        "\n",
        "            # is action valid?\n",
        "            if self.env.fields[position] != 0:\n",
        "                self.reward += reward_con_in\n",
        "                self.refused += 1\n",
        "                if prints: print('Greedy: Could not {} in {}'.format(action + ' ' + colour, self.reformat_env_printing(self.env.fields)))\n",
        "                # keep state as no action was done\n",
        "            else:\n",
        "                self.reward += self.rewards[position]\n",
        "                self.env.fields[position] = self.colours[colour]\n",
        "                if prints: print('Greedy: {} in {}'.format(action + ' ' + colour, self.reformat_env_printing(self.env.fields)))\n",
        "\n",
        "        if action is 'restore':\n",
        "            if self.colours[colour] not in self.env.fields:\n",
        "                self.reward += reward_con_out\n",
        "                self.refused += 1\n",
        "                # keep state as no action was done\n",
        "                if prints: print('Greedy: Could not {} in {}'.format(action + ' ' + colour, self.reformat_env_printing(self.env.fields)))\n",
        "                return\n",
        "\n",
        "            position = self.env.fields.index(self.colours[colour])\n",
        "            # is action valid?\n",
        "            if self.env.fields[position] != self.colours[colour]:\n",
        "                self.reward += reward_con_out\n",
        "                self.refused += 1\n",
        "                if prints: print('Greedy: Could not {} in {}'.format(action + ' ' + colour, self.reformat_env_printing(self.env.fields)))\n",
        "                # keep state as no action was done\n",
        "            else:\n",
        "                self.reward += self.rewards[position]\n",
        "                self.env.fields[position] = 0\n",
        "                if prints: print('Greedy: {} in {}'.format(action + ' ' + colour, self.reformat_env_printing(self.env.fields)))"
      ],
      "execution_count": 69,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W2a4pZzbpld1"
      },
      "source": [
        "**Define smart robot**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y6RgKJzP-k-A"
      },
      "source": [
        "class Smart:\n",
        "    def __init__(self, warehouse_size, policy, states, reward_mat):\n",
        "        self.policy = policy\n",
        "        self.states = states\n",
        "        self.env = Environment(warehouse_size)\n",
        "        self.reward = 0\n",
        "        self.refused = 0\n",
        "        self.reward_mat = reward_mat\n",
        "\n",
        "    def next_action(self, action, colour):\n",
        "        act = action[0] + colour[0]\n",
        "        current_state = tuple([str(field) for field in self.env.fields]) + (act,)\n",
        "        idx_st = self.states.index(current_state)\n",
        "        pol = self.policy[idx_st]\n",
        "        # store\n",
        "        if act[0] == 's':\n",
        "            if self.env.fields[pol] != 0:\n",
        "                if prints: print('Smart: Could not {} in {}'.format(action + ' ' + colour, self.env.fields))\n",
        "                self.refused += 1\n",
        "                self.reward += self.reward_mat[idx_st, pol]\n",
        "                return\n",
        "\n",
        "            self.env.fields[pol] = act[1]\n",
        "            self.reward += self.reward_mat[idx_st, pol]\n",
        "            if prints: print('Smart: {} in {}'.format(action + ' ' + colour, self.env.fields))\n",
        "        # restore\n",
        "        if act[0] == 'r':\n",
        "            if self.env.fields[pol] != act[1]:\n",
        "                if prints: print('Smart: Could not {} in {}'.format(action + ' ' + colour, self.env.fields))\n",
        "                self.refused += 1\n",
        "                self.reward += self.reward_mat[idx_st, pol]\n",
        "                return\n",
        "\n",
        "            self.env.fields[pol] = 0\n",
        "            self.reward += self.reward_mat[idx_st, pol]\n",
        "            if prints: print('Smart: {} in {}'.format(action + ' ' + colour, self.env.fields))\n",
        "\n"
      ],
      "execution_count": 70,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7WPaFSVMpoHd"
      },
      "source": [
        "**Define Simulation**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6HgG4jMpSIl"
      },
      "source": [
        "class Simulation:\n",
        "    def __init__(self, warehouse_size, policy, states, reward_mat, testfile):\n",
        "        self.greedy = Greedy(warehouse_size, reward_mat)\n",
        "        self.smart = Smart(warehouse_size, policy, states, reward_mat)\n",
        "        a = {'store\\tred\\n': ('store', 'red'),\n",
        "             'store\\twhite\\n': ('store', 'white'),\n",
        "             'store\\tblue\\n': ('store', 'blue'),\n",
        "             'restore\\tred\\n': ('restore', 'red'),\n",
        "             'restore\\twhite\\n': ('restore', 'white'),\n",
        "             'restore\\tblue\\n': ('restore', 'blue')}\n",
        "        self.order = []\n",
        "        with open(testfile, 'r') as f:\n",
        "            for line in f:\n",
        "                if line in a:\n",
        "                    self.order.append(a[line])\n",
        "        self.index = 0\n",
        "\n",
        "    def run(self):\n",
        "        moves = len(self.order)\n",
        "\n",
        "        for i in range(0, moves, 1):\n",
        "            self.update()\n",
        "            if prints: print(\"Update\")\n",
        "\n",
        "    def update(self):\n",
        "        action, colour = self.order[self.index]\n",
        "        self.index += 1\n",
        "        self.greedy.next_action(action, colour)\n",
        "        self.smart.next_action(action, colour)\n",
        "\n",
        "    def print_reward(self):\n",
        "        print('Greedy Robot: ' + str(self.greedy.reward))\n",
        "        print('Greedy Refused: ' + str(self.greedy.refused))\n",
        "        print('Smart Robot: ' + str(self.smart.reward))\n",
        "        print('Smart Refused: ' + str(self.smart.refused))\n",
        "\n",
        "        performance_diff = self.greedy.reward - self.smart.reward\n",
        "        performance_diff_per_step = performance_diff / self.index\n",
        "        print('Full Performance difference: ' + str(performance_diff))\n",
        "        print('Performance difference per step: ' + str(round(performance_diff_per_step, 5)))\n"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iyVJcluCptcc"
      },
      "source": [
        "**Run simulation and evaluate Result**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Sy-HdDkn-rb_",
        "outputId": "8d27dd9c-9139-4e43-c4af-6e9147b97abd"
      },
      "source": [
        "sim = Simulation(warehouse_size, mdpresultValue.policy, states, reward_mat, 'Exercise 4 - Reinforcement Learning Data - warehousetraining.txt')\n",
        "sim.run()\n",
        "\n",
        "print(\"Evaluation - WarehouseTraining\")\n",
        "sim.print_reward()"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Evaluation - WarehouseTraining\n",
            "Greedy Robot: -25686\n",
            "Greedy Refused: 0\n",
            "Smart Robot: -26020.0\n",
            "Smart Refused: 0\n",
            "Full Performance difference: 334.0\n",
            "Performance difference per step: 0.02759\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-e41MvMJJt6b",
        "outputId": "de23b163-ff9c-42f8-bd6c-265c11c783cb"
      },
      "source": [
        "sim = Simulation(warehouse_size, mdpresultValue.policy, states, reward_mat, 'Exercise 4 - Reinforcement Learning Data - warehouseorder.txt')\n",
        "sim.run()\n",
        "\n",
        "\n",
        "print(\"Evaluation - WarehouseOrder\")\n",
        "sim.print_reward()"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Evaluation - WarehouseOrder\n",
            "Greedy Robot: -124\n",
            "Greedy Refused: 0\n",
            "Smart Robot: -130.0\n",
            "Smart Refused: 0\n",
            "Full Performance difference: 6.0\n",
            "Performance difference per step: 0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "00OhEw-uqwmf"
      },
      "source": [
        "with open('Value-Iteration-Policy', 'w') as f:\n",
        "    write = csv.writer(f)\n",
        "    write.writerow(mdpresultValue.policy)"
      ],
      "execution_count": 74,
      "outputs": []
    }
  ]
}