[OSPP-Week2] Check the kuhn_nfsp experiment and read papers

[peterchen96]

@findmyway

Hi, I am a student of the OSPP project Implement Multi-Agent Reinforcement Learning Algorithms in Julia. This summer, my main mission is to implement some multi-agent RL algorithms(such as NFSP and PSRO) in RLZoo and add relative experiments in RLExperiment. Following is my current progress and this week's plan.

Current Progress

I have implemented the Neural Fictitious Self-play(NFSP) algorithm roughly and test it on the Kuhn Poker game(KuhnPokerEnv). See my project for more details.

This Week's Plan

• Use nash_conv to evaluate the experiment:

  I should add some prob function in QBasedpolicy for some specific stages. For now, nash_conv is not available for QBasedpolicy.

• Organize current codes and add them to the corresponding directories in RL.jl.

[findmyway]

Besides, as we mentioned before, the julia version of KuhnPokerEnv may need to implement the clone method to calculate nash_conv

[peterchen96]

There is a copy method for AbstractEnv. Maybe add Base.:(==) and Base.hash methods for AbstractEnv is enough, which just like in OpenSpielEnv.

[peterchen96]

For now, nash_conv can be used in the kuhn_nfsp experiment, although its result looks bad.

So in this week, I would do the following things:

• Check the process of kuhn_nfsp to correct the result of the experiment. For more details about the experiment see https://github.com/peterchen96/nfsp_demo.

• Submit nfsp algorithm and kuhn_nfsp experiment to ReinforcementLearning.jl step by step.

• Read papers about overview for cooperative Multi-agent Reinforcement Learning, such as https://arxiv.org/abs/1908.03963.

[findmyway]

You'd better create a draft PR so that I can offer suggestions from my side early, otherwise, things may out of control very quickly.

[findmyway]

And I can also help to diagnose the possible problems.

[peterchen96]

Ok, I'll create a PR later today.

[peterchen96]

Tasking List for adding Neural Fictitious Self-play(nfsp) algorithm and kuhn_nfsp experiment

NFSPAgent's structure:

• rl_agent:

  Agent(
      policy = QBasedPolicy(
          learner = DQNLearner,
          explorer = EpsilonGreedyExplorer,
      ),
      trajectory = CircularArraySARTTrajectory
  )

  where rl_agent(use DQN as an example) works to search for the best response from the self-play process.

• sl_agent:

  Agent(
      policy = BehaviorCloningPolicy,
      trajectory = ReservoirTrajectory,
  )

  where sl_agent learns the best response for the state from rl_agent's policy.

Task list

• add AverageLearner in RLZoo use BehaviorCloingPolicy as sl_agent for now
• add nfsp and its relative operations
• add kuhn_nfsp experiment

New progress is in #402.