Hello

DP/Gamblers Problem.ipynb

@@ -0,0 +1,158 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "### This is Example 4.3. Gambler’s Problem from Sutton's book.\n",
+    "\n",
+    "A gambler has the opportunity to make bets on the outcomes of a sequence of coin flips. \n",
+    "If the coin comes up heads, he wins as many dollars as he has staked on that flip; \n",
+    "if it is tails, he loses his stake. The game ends when the gambler wins by reaching his goal of $100, \n",
+    "or loses by running out of money. \n",
+    "\n",
+    "On each flip, the gambler must decide what portion of his capital to stake, in integer numbers of dollars. \n",
+    "This problem can be formulated as an undiscounted, episodic, finite MDP. \n",
+    "\n",
+    "The state is the gambler’s capital, s ∈ {1, 2, . . . , 99}.\n",
+    "The actions are stakes, a ∈ {0, 1, . . . , min(s, 100 − s)}. \n",
+    "The reward is zero on all transitions except those on which the gambler reaches his goal, when it is +1.\n",
+    "\n",
+    "The state-value function then gives the probability of winning from each state. A policy is a mapping from levels of capital to stakes. The optimal policy maximizes the probability of reaching the goal. Let p_h denote the probability of the coin coming up heads. If p_h is known, then the entire problem is known and it can be solved, for instance, by value iteration.\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import sys\n",
+    "import matplotlib.pyplot as plt\n",
+    "if \"../\" not in sys.path:\n",
+    "  sys.path.append(\"../\") "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "\n",
+    "### Exercise 4.9 (programming)\n",
+    "\n",
+    "Implement value iteration for the gambler’s problem and solve it for p_h = 0.25 and p_h = 0.55.\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "def value_iteration_for_gamblers(p_h, theta=0.0001, discount_factor=1.0):\n",
+    "    \"\"\"\n",
+    "    Args:\n",
+    "        p_h: Probability of the coin coming up heads\n",
+    "    \"\"\"\n",
+    "    \n",
+    "    def one_step_lookahead(s, V, rewards):\n",
+    "        \"\"\"\n",
+    "        Helper function to calculate the value for all action in a given state.\n",
+    "        \n",
+    "        Args:\n",
+    "            s: The gambler’s capital. Integer.\n",
+    "            V: The vector that contains values at each state. \n",
+    "            rewards: The reward vector.\n",
+    "                        \n",
+    "        Returns:\n",
+    "            A vector containing the expected value of each action. \n",
+    "            Its length equals to the number of actions.\n",
+    "        \"\"\"\n",
+    "        \n",
+    "        # Implement!\n",
+    "        \n",
+    "        return A\n",
+    "    \n",
+    "    # Implement!\n",
+    "    \n",
+    "    return policy, V"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "policy, v = value_iteration_for_gamblers(0.25)\n",
+    "\n",
+    "print(\"Optimized Policy:\")\n",
+    "print(policy)\n",
+    "print(\"\")\n",
+    "\n",
+    "print(\"Optimized Value Function:\")\n",
+    "print(v)\n",
+    "print(\"\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Plotting Final Policy (action stake) vs State (Capital)\n",
+    "\n",
+    "# Implement!"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "# Plotting Capital vs Final Policy\n",
+    "\n",
+    "# Implement!\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

DP/Policy Evaluation Solution.ipynb

@@ -2,12 +2,11 @@
  "cells": [
   {
    "cell_type": "code",
-   "execution_count": 53,
-   "metadata": {
-    "collapsed": false
-   },
+   "execution_count": 1,
+   "metadata": {},
    "outputs": [],
    "source": [
+    "from IPython.core.debugger import set_trace\n",
     "import numpy as np\n",
     "import pprint\n",
     "import sys\n",
@@ -18,10 +17,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 54,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 2,
+   "metadata": {},
    "outputs": [],
    "source": [
     "pp = pprint.PrettyPrinter(indent=2)\n",
@@ -30,10 +27,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 55,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": 3,
+   "metadata": {},
    "outputs": [],
    "source": [
     "def policy_eval(policy, env, discount_factor=1.0, theta=0.00001):\n",
@@ -43,9 +38,11 @@
     "    Args:\n",
     "        policy: [S, A] shaped matrix representing the policy.\n",
     "        env: OpenAI env. env.P represents the transition probabilities of the environment.\n",
-    "            env.P[s][a] is a (prob, next_state, reward, done) tuple.\n",
+    "            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n",
+    "            env.nS is a number of states in the environment. \n",
+    "            env.nA is a number of actions in the environment.\n",
     "        theta: We stop evaluation once our value function change is less than theta for all states.\n",
-    "        discount_factor: lambda discount factor.\n",
+    "        discount_factor: Gamma discount factor.\n",
     "    \n",
     "    Returns:\n",
     "        Vector of length env.nS representing the value function.\n",
@@ -61,7 +58,7 @@
     "            for a, action_prob in enumerate(policy[s]):\n",
     "                # For each action, look at the possible next states...\n",
     "                for  prob, next_state, reward, done in env.P[s][a]:\n",
-    "                    # Calculate the expected value\n",
+    "                    # Calculate the expected value. Ref: Sutton book eq. 4.6.\n",
     "                    v += action_prob * prob * (reward + discount_factor * V[next_state])\n",
     "            # How much our value function changed (across any states)\n",
     "            delta = max(delta, np.abs(v - V[s]))\n",
@@ -74,10 +71,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 56,
-   "metadata": {
-    "collapsed": false
-   },
+   "execution_count": 4,
+   "metadata": {},
    "outputs": [],
    "source": [
     "random_policy = np.ones([env.nS, env.nA]) / env.nA\n",
@@ -86,10 +81,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 57,
-   "metadata": {
-    "collapsed": false
-   },
+   "execution_count": 5,
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -98,7 +91,8 @@
       "Value Function:\n",
       "[  0.         -13.99993529 -19.99990698 -21.99989761 -13.99993529\n",
       " -17.9999206  -19.99991379 -19.99991477 -19.99990698 -19.99991379\n",
-      " -17.99992725 -13.99994569 -21.99989761 -19.99991477 -13.99994569   0.        ]\n",
+      " -17.99992725 -13.99994569 -21.99989761 -19.99991477 -13.99994569\n",
+      "   0.        ]\n",
       "\n",
       "Reshaped Grid Value Function:\n",
       "[[  0.         -13.99993529 -19.99990698 -21.99989761]\n",
@@ -121,10 +115,8 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 51,
-   "metadata": {
-    "collapsed": false
-   },
+   "execution_count": 6,
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Test: Make sure the evaluated policy is what we expected\n",
@@ -135,9 +127,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "collapsed": true
-   },
+   "metadata": {},
    "outputs": [],
    "source": []
   }
@@ -158,9 +148,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.1"
+   "version": "3.6.4"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
 }

DP/Policy Evaluation.ipynb

@@ -4,7 +4,7 @@
    "cell_type": "code",
    "execution_count": 23,
    "metadata": {
-    "collapsed": false
+    "collapsed": true
    },
    "outputs": [],
    "source": [
@@ -30,7 +30,7 @@
    "cell_type": "code",
    "execution_count": 25,
    "metadata": {
-    "collapsed": false
+    "collapsed": true
    },
    "outputs": [],
    "source": [
@@ -41,9 +41,11 @@
     "    Args:\n",
     "        policy: [S, A] shaped matrix representing the policy.\n",
     "        env: OpenAI env. env.P represents the transition probabilities of the environment.\n",
-    "            env.P[s][a] is a (prob, next_state, reward, done) tuple.\n",
+    "            env.P[s][a] is a list of transition tuples (prob, next_state, reward, done).\n",
+    "            env.nS is a number of states in the environment. \n",
+    "            env.nA is a number of actions in the environment.\n",
     "        theta: We stop evaluation once our value function change is less than theta for all states.\n",
-    "        discount_factor: gamma discount factor.\n",
+    "        discount_factor: Gamma discount factor.\n",
     "    \n",
     "    Returns:\n",
     "        Vector of length env.nS representing the value function.\n",
@@ -60,7 +62,7 @@
    "cell_type": "code",
    "execution_count": 26,
    "metadata": {
-    "collapsed": false
+    "collapsed": true
    },
    "outputs": [],
    "source": [
@@ -71,9 +73,7 @@
   {
    "cell_type": "code",
    "execution_count": 22,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "ename": "AssertionError",
@@ -121,9 +121,9 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.5.1"
+   "version": "3.5.2"
   }
  },
  "nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
 }