Project 2: Continuous Control

Introduction

This project implemented the Deep Deterministic Policy Gradient (DDPG) algorithm for training an agent to make the arms catching the balls correctly.This project is also a requirement from Udacity Reinforcement Learning program.

In this environment, a double-jointed arm can move to target locations. A reward of +0.1 is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.

The observation space consists of 33 variables corresponding to position, rotation, velocity, and angular velocities of the arm. Each action is a vector with four numbers, corresponding to torque applicable to two joints. Every entry in the action vector should be a number between -1 and 1.

Solving the Environment

Option 2: Solve the Second Version

The barrier for solving the second version of the environment is slightly different, to take into account the presence of many agents. In particular, your agents must get an average score of +30 (over 100 consecutive episodes, and over all agents). Specifically,

• After each episode, we add up the rewards that each agent received (without discounting), to get a score for each agent. This yields 20 (potentially different) scores. We then take the average of these 20 scores.
• This yields an average score for each episode (where the average is over all 20 agents).

The environment is considered solved, when the average (over 100 episodes) of those average scores is at least +30.

Getting Started

1. Download the environment from one of the links below. You need only select the environment that matches your operating system:

   • Version 1: One (1) Agent

     • Linux: click here
     • Mac OSX: click here
     • Windows (32-bit): click here
     • Windows (64-bit): click here

   • Version 2: Twenty (20) Agents

     • Linux: click here
     • Mac OSX: click here
     • Windows (32-bit): click here
     • Windows (64-bit): click here

   (For Windows users) Check out this link if you need help with determining if your computer is running a 32-bit version or 64-bit version of the Windows operating system.

   (For AWS) If you'd like to train the agent on AWS (and have not enabled a virtual screen), then please use this link (version 1) or this link (version 2) to obtain the "headless" version of the environment. You will not be able to watch the agent without enabling a virtual screen, but you will be able to train the agent. (To watch the agent, you should follow the instructions to enable a virtual screen, and then download the environment for the Linux operating system above.)

2. Place the file in the DRLND GitHub repository, in the p2_continuous-control/ folder, and unzip (or decompress) the file.

3. Follow the steps in the original DRLND repository to download the required dependencies and set up an Anaconda environment with Python = 3.6. CAUTION: The python version must be 3.6 to avoid any confliction with the Unity Agent version of 0.4.0.

Instructions

Follow the instructions in Continuous_Control.ipynb to get started with training your own agent. You can also run a pre-trained agent to evaluate its performance.

Some important files:

• Continuous_Control.ipynb --> The training and testing process.
• Plotting.ipynb --> For plotting
• DDPG_Agent.py --> The DDPG agent that handles the learning process.
• DDPG_model.py --> The Deep Q-learning Network architecture.
• checkpoint_actor --> The pre-trained parameters of the DDPG actor
• checkpoint_critic --> The pre-trained parameters of the DDPG critic
• Test_model --> The backup folder
• REPORT.md --> The report for this project.

Training result

With this hyperparameters, the agent converges very fast and get the instantaneous score over 30 within just 20 episode and the average scores over 30.0 in just 65 episodes. Furthermore, the running time of training process is just 3 hours.

Testing result

The scores are very noisy but always greater than 30.0. Hence the average score is around 37.13. When watching the agent plays in the GUI, we see that all the arms catching the green balls very well.

References

• [1] Lillicrap et. al.. Continuous control through deep reinforcement learning
• [2] Silver et. al.. Deterministic Policy Gradients Algorithms
• [3] DDPG implementation from Udacity