-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathAAAI24-lab.html
346 lines (306 loc) · 10.7 KB
/
AAAI24-lab.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
<!DOCTYPE html>
<!--[if IE 8 ]>
<html class="no-js oldie ie8" lang="en"> <![endif]-->
<!--[if IE 9 ]>
<html class="no-js oldie ie9" lang="en"> <![endif]-->
<!--[if (gte IE 9)|!(IE)]><!-->
<html> <!--<![endif]-->
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
#go-top {
position: fixed;
bottom: 0;
right: 0;
z-index: 600;
display: none;
}
.container {
margin-left: auto;
margin-right: auto;
display: flex;
align-items: center;
justify-content: center;
width: 50%;
}
table {
font-family: arial, sans-serif;
border-collapse: collapse;
width: 80%;
margin-left: auto;
margin-right: auto;
}
td, th {
border: 1px solid #dddddd;
text-align: left;
padding: 8px;
}
tr:nth-child(even) {
background-color: #dddddd;
}
body, html {
height: 100%;
margin: 0;
font-family: Arial, Helvetica, sans-serif;
scroll-behavior: smooth;
}
.hero-image {
background-image: linear-gradient(rgba(0, 0, 0, 0.5), rgba(0, 0, 0, 0.5)), url("images/banner2.png");
height: 50%;
background-position: center;
background-repeat: no-repeat;
background-size: cover;
position: relative;
}
.hero-text {
text-align: center;
position: absolute;
top: 50%;
left: 50%;
transform: translate(-50%, -50%);
color: white;
}
.section-head-text {
color: rgba(56, 66, 78, 0.5);
text-align: center;
font-size: 1.6rem;
line-height: 1.875;
margin-bottom: 0.3rem;
letter-spacing: .1rem;
}
.section-text {
margin-left: auto;
margin-right: auto;
width:65%;
margin-bottom: 0.1rem;
word-wrap: break-all;
line-height: 1.3;
}
footer {
left: 0;
bottom: 0;
width: 100%;
background-color: black;
color: grey;
text-align: center;
height: 45px;
}
.center {
height: 45px;
position: relative;
}
.center p {
margin: 0;
position: absolute;
top: 50%;
left: 50%;
-ms-transform: translate(-50%, -50%);
transform: translate(-50%, -50%);
}
#myBtn {
display: none;
position: fixed;
bottom: 10px;
right: 10px;
z-index: 99;
font-size: 18px;
border: none;
outline: none;
background-color: rgb(0, 80, 128, 0.3);
color: white;
cursor: pointer;
padding: 15px;
border-radius: 4px;
}
#myBtn:hover {
background-color: rgb(0, 80, 128, 1.0);
}
</style>
</head>
<body id="top">
<div class="hero-image">
<div class="hero-text">
<h1 style="font-size:50px">LH2: Introduction to MDP Modeling and Interaction via RDDL and pyRDDLGym</h1>
<h2>AAAI-2024 Vancouver</h2>
</div>
</div>
<section>
<div class="section-head-text">
<h2>Overview</h2>
</div>
<div class="section-text">
<p>
RDDL (pronounced as “riddle”) stands for the Relational Dynamic Influence Diagram Language.
It is the domain modeling language utilized in the International Conference on Automated Planning and Scheduling (ICAPS) in the years 2011, 2014, 2018,
and most recently in 2023 for the Probabilistic Planning and Reinforcement Learning track of the International Planning Competitions.
RDDL was designed to efficiently represent real-world stochastic planning problems, specifically Markov Decision Processes (MDPs), with a focus on factored MDPs characterized by highly structured transition and reward functions.
This tutorial aims to provide basic understanding of RDDL, including recent language enhancements and features, through a practical example.
We will introduce a problem based on a real-world scenario and incrementally build up its representation in RDDL, starting from the raw mathematical equations up to a full RDDL description.
Furthermore, we will introduce “pyRDDLGym,” a new Python framework for the generation of Gym environments from RDDL descriptions.
This facilitates interaction with Reinforcement Learning (RL) agents via the standard Gym interface, as well as enables planning agents to work with the model.
In a series of exercises, we will explore the capabilities of pyRDDLGym, which includes generation of the Dynamic Bayesian Networks (DBN) and eXtended Algebraic Decision Diagrams (XADD)-based conditional probability functions,
as well as offering both generic and custom visualization options. We will also generate a functional environment for the example problem.
To close the loop from a mathematical representation to a fully operational policy, we will utilize the built-in model-based anytime backpropagation planner,
known as “JaxPlan.” This will enable us to obtain solutions for the example problem, effectively closing the gap between theoretical description and a practical working policy.
</p>
</div>
</section>
<section>
<div class="section-head-text">
<h2>Outline</h2>
</div>
<div class="section-text">
<ul>
<li><b>Introduction to stochastic planning problem classes and languages</b></li>
<ul>
<li>General overview</li>
<li>MDP and stochastic problems with introduction of a running example</li>
<li>POMDP extensions</li>
</ul>
<li><b>RDDL overview</b></li>
<ul>
<li>Language overview capabilities</li>
<li>Domain and problem components</li>
<li>Modeling of the running example in RDDL (grounded)</li>
<li>Scaling up by lifting the domain specification</li>
</ul>
<li><b>Introduction to pyRDDLGym</b></li>
<ul>
<li>Vision</li>
<li>Repository structure</li>
<li>Executing the running example with the built-in visualizer and a simple policy</li>
<li>Custom visualization</li>
<li>Auxiliary (diagnostic) tools</li>
</ul>
<li><b>Hands-on and interactive exercises</b></li>
<ul>
<li>Mathematical modeling</li>
<li>RDDL representation</li>
<li>Interaction with a simple agent</li>
<li>Extending the model</li>
<li>Scalling up from one to many in a heart beat</li>
<li>Using the model based solver JaxPlan</li>
<li>Applying RL tools in pyRDDLGym</li>
</ul>
</ul>
</div>
<p></p>
</section>
<section>
<div class="section-head-text">
<h2>Schedule</h2>
</div>
<div class="section-text">
<table>
<tr>
<th>Time</th>
<th>Topic</th>
</tr>
<tr>
<td>8:30-9:30am</th>
<td>Introduction to stochastic planning problem classes and languages</th>
</tr>
<tr>
<td>9:30-10:00am</td>
<td>RDDL overview</td>
</tr>
<tr>
<td>10:00-10:30am</td>
<td>Introduction to pyRDDLGym</td>
</tr>
<tr>
<td>10:30am-11:00am</td>
<td>Break</td>
</tr>
<tr>
<td>11:00am-12:30pm</td>
<td>Hands-on and interactive exercises</td>
</tr>
</table>
</div>
</section>
<section>
<div class="section-head-text">
<h2>Audience</h2>
</div>
<div class="section-text">
<ul>
<li><b>Target audience:</b> Machine learning and applied math researchers and practitioners.</li>
<li><b>Prerequisite knowledge:</b> The audience is assumed to have basic knowledge of Markov Decision Processes (MDPs)
(e.g., through Reinforcement Learning or Planning), a basic knowledge of Python, and preferably (though not required) a familiarity with OpenAI Gym environment.</li>
</ul>
</div>
</section>
<section>
<div class="section-head-text">
<h2>Resources</h2>
</div>
<div class="section-text">
<ul>
<li>Slides</li>
<ul>
<li><a href="https://pyrddlgym-project.github.io/data/Tutorial/AAAI2024/RDDL_Lab_AAAI_2024_Part_1.pdf">RDDL slides</a></li>
<li><a href="https://pyrddlgym-project.github.io/data/Tutorial/AAAI2024/RDDL_Lab_AAAI_2024_Part_2.pdf">pyRDDLGym slides</a></li>
</ul>
<li>Notebooks</li>
<ul>
<li><a href="https://colab.research.google.com/drive/19q8YSJcHzzCoFp3OiQBeYg2qsztbVkja?usp=sharing">Lab hands-on notebook</a></li>
<li><a href="https://colab.research.google.com/drive/1uC6IhYaMa6pqyogYKAKzE7L5yLqdHdDx?usp=sharing">pyRDDLGym playground</a></li>
</ul>
</ul>
</div>
</section>
<section>
<div class="section-head-text">
<h2>Organizers</h2>
</div>
<div class="container">
<div class="section-text">
<center>
<img src="https://aaai.org/wp-content/uploads/2023/11/Scott_Sanner_White_Background.png" width="161" height="201">
<p><a href="https://www.mie.utoronto.ca/faculty_staff/sanner/"><b>Scott Sanner</b></a></p>
<p>Scott Sanner is an Associate Professor at the University of Toronto specializing in AI topics such as sequential decision-making, recommender systems, and machine/deep learning applications.
He is an Associate Editor for three journals and a recipient of four paper awards and a Google Faculty Research Award</p>
</center>
</div>
<div class="section-text">
<center>
<img src="https://aaai.org/wp-content/uploads/2023/11/Ayal-white-background.png" width="173" height="201">
<p><a href="https://sites.google.com/view/ataitler/home"><b>Ayal Taitler</b></a></p>
<p>Ayal Taitler is a Postdoctoral Fellow at the University of Toronto, working with Prof. Scott Sanner. His research interests lie at the intersection of reinforcement learning,
automated planning, and control theory, and its application to robotics and intelligent transportation. With over 10 years of experience in software engineering and AI research.</p>
</center>
</div>
</div>
</section>
<footer>
<!--
<p></p>
<span>© Copyright <a href="https://pyrddlgym-project.github.io">pyRDDLGym</a> 2023.</span>
<p></p> -->
<div class="center">
<p>© Copyright <a href="https://pyrddlgym-project.github.io">pyRDDLGym</a> 2023. | Design by <a href="https://sites.google.com/view/ataitler/home">Ayal Taitler</a></p>
</div>
<!--<p style="transform: translate(-50%, -50%)">© Copyright <a href="https://pyrddlgym-project.github.io">pyRDDLGym</a> 2023.</p>-->
<button onclick="topFunction()" id="myBtn" title="Go to top"><img src="images/arrow-up.png" alt="Top" width="20" height="20"/></button>
</footer>
<script>
// Get the button
let mybutton = document.getElementById("myBtn");
// When the user scrolls down 20px from the top of the document, show the button
window.onscroll = function() {scrollFunction()};
function scrollFunction() {
if (document.body.scrollTop > 20 || document.documentElement.scrollTop > 20) {
mybutton.style.display = "block";
} else {
mybutton.style.display = "none";
}
}
// When the user clicks on the button, scroll to the top of the document
function topFunction() {
document.body.scrollTop = 0;
document.documentElement.scrollTop = 0;
}
</script>
</body>
</html>