Deep Reinforcement Learning Course

Welcome to our four-week course on Deep Reinforcement Learning (Deep RL), starting September 8, 2023.

Our objective with this course is to deepen our collective knowledge about RL and RL software development skills. The ultimate goal is to empower each of us to effectively contribute to RL projects, while fostering a learning culture within STR.

We will be using world-class open-source materials from institutions such as Stanford University, UC Berkeley, Anyscale Ray RLlib, OpenAI, DeepMind, and more. We structure the class in a way that encourages members to explore applications of RL across our diverse range of divisions, sparking new ideas and innovations.

Course Schedule

Here's a snapshot of what you can expect:

• Week 1 (Sept. 8): Intro to RL Fundamentals + Environment Design for Single Agent Scenarios
• Week 2 (Sept. 15): Intro to Policy Gradient and Actor-Critic Methods
• Week 3 (Sept. 22): Intro to Multi-agent Learning Theory & Training Strategies
• Week 4 (Sept. 29): Student Agents Competition Showdown + Advanced Applicable RL Techniques Overview: Model-Based RL, Hierarchical RL, Imitation Learning, Meta-learning

Objective

The primary aim of this course is to equip STR employees with fundamental RL understanding and RL software development skills to effectively contribute to RL programs at STR.

Key Outcomes

• Nurture in-house RL talent as contributors and consultants to support RL programs at STR.
• Create in-house RL learning materials for company-wide use on the LinkedIn Learning Platform.
• Leverage top-notch open-sourced materials across academic and industry (UC Berkeley, Stanford, Anyscale Ray RLlib, OpenAI, and more).
• Inspire innovations and creative applications of RL to new and ongoing programs across all divisions of the company.
• Gain hands-on experience with industry-standard Open-Source Collaborative MLOps workflow.
• Implement and develop RL agents with RLlib, an industry-Grade Reinforcement Learning open-source library.
• Use free GPU resources from Google CoLab to train RL agents and submit assignments for auto-grading via GitHub Classroom and CI/CD Actions.
• Collaboratively generate live leaderboard analysis dashboards with industry-standard ML tracking and report tools MLflow.

Format

This course provides a four-week learning experience facilitated by two instructors for eight students.

• Lecture Format: 90-minute lectures are held every other Friday, starting from 09/08/2023. Each lecture consists of a 50-minute talk followed by 30-minute in-class hands-on exercises. Selected student homework showcases will be presented at the beginning of each class.
• Homework Format: All students will solve the same task with their agents to try to beat the baseline performance threshold. Students are using the same RL framework and environments with clear directions on where to modify the code for their assignments. Extra credit assignments will encourage students to be creative and apply their domain knowledge and expertise to push beyond course materials. Instructors will run students’ agents in the evaluation framework and provide performance results on the leaderboard for in-class discussions.

References:

MultiGrid

Our course materials were built on top of the public repository "MultiGrid" by Ini Oguntola. MultiGrid provides a collection of fast multi-agent discrete gridworld environments for reinforcement learning. It is designed to be fast and easily customizable. For more information, you can visit the original MultiGrid repository.

@software{multigrid,
  author = {Oguntola, Ini},
  title = {Fast Multi-Agent Gridworld Environments for Gymnasium},
  url = {https://github.com/ini/multigrid},
  year = {2023},
}

CleanRL

Our course materials incorporate insights from "CleanRL", a project led by Shengyi(Costa) Huang alongside collaborators. CleanRL is renowned for its high-quality single-file implementations of deep reinforcement learning algorithms. The principal objective of CleanRL is to offer simplified yet efficient codebases for understanding and integrating deep RL algorithms. For a deeper dive into the intricacies, methodologies, and implementations, you can review the original CleanRL repository or refer to their publication in the Journal of Machine Learning Research.

@article{huang2022cleanrl,
  author  = {Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},
  title   = {CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {274},
  pages   = {1--18},
  url     = {http://jmlr.org/papers/v23/21-1342.html}
}