General questions about the project

[Teleriann]

Hello, Iam not quite skilled in AI and programming, also in English so pardon my mistakes. Iam making bachelor thesis about TMInterface and about TMRL as projects for reinforcement learning. If you do not mind, I have few questions that I would like to ask.

1. ,
   

When trying out my reward function what does all of the "things" mean? like r is probably reward, what is d, t and o?

2. I have no idea why, but while training my CPU is running at 100% and my gpu is running on 30% average, what could be the issue?

I have 5800X3D CPU with 4070 GPU.

3. Is there some tutorial that fully explains how to run gymnasium environment like some video or something? Like I said Iam not really skilled in this type of things so I would like to learn it if I can from some source.

4. Is there some documentation that explains whole config file? Like I know what some things mean like window height and lenght for example but I have no idea of start_obs_capture or other things like that. It is probably something about time delay before starting to capture observations or so, but like I said, I do not really know.

5. Last question for now about wandb, I have registered, created my project, put API key to configuration etc... when I start the training I get linked to "fullrun" but even after waiting for 30minutes nothing shows up, what could be wrong?

Sorry for being annoying or asking stupid questions, I would like to learn as much as possible about this project to provide right information in my thesis, thank you guys.

[koi0823]

i have question too how can i set the time and second
what i facing problem is

before this i set my timestep timeout factor to 80 what chatgpt generate for me

i just stuck every 46 sec and 50 sec

[yannbouteiller]

You can find documentation about config.json here

[yannbouteiller]

Hello, Iam not quite skilled in AI and programming, also in English so pardon my mistakes. Iam making bachelor thesis about TMInterface and about TMRL as projects for reinforcement learning. If you do not mind, I have few questions that I would like to ask.

Hi!

Several people have been asking about config.json, so I finally wrote documentation explaining all its relevant entries here ☺️ (other entries are outdated, we should remove them at some point)

When trying out my reward function what does all of the "things" mean? like r is probably reward, what is d, t and o?

r is the reward, d is the termination signal, t is the truncation signal, and o represents the observation fed to the model

2. I have no idea why, but while training my CPU is running at 100% and my gpu is running on 30% average, what could be the issue?

CPU-bound operations, this is to be expected especially if the Trainer and the Worker are on the same machine

3. Is there some tutorial that fully explains how to run gymnasium environment like some video or something? Like I said Iam not really skilled in this type of things so I would like to learn it if I can from some source.

We have a tutorial for Real-Time Gym if you want to build your own gymnasium environments in real-time settings, but this is a bit advanced and probably not what you are looking for. There are probably videos that explain how to use Gymnasium environments, I don't know any but I would be surprised if this did not exist. Anyway, Gymnasium is straightforward to use if you have minimal Python knowledge, you can find an example of usage in the tmrl readme here (you can read the code from env = get_environment(): env is a Gymnasium environment).

5. Last question for now about wandb, I have registered, created my project, put API key to configuration etc... when I start the training I get linked to "fullrun" but even after waiting for 30minutes nothing shows up, what could be wrong?

I think the default duration of an "epoch" is kinda long. Logs are sent to wandb at the end of each "epoch".

Sorry for being annoying or asking stupid questions, I would like to learn as much as possible about this project to provide right information in my thesis, thank you guys.

No problem, and there is no stupid question :)

[Teleriann]

I would like to ask one more question, in wandb graphs what does "round_test" and "episode_lenght_test" mean? All of my graphs look okay and I understand what they mean, but this two graphs have just value 0 all the time and there is nothing.

[yannbouteiller]

round_time is supposed to be the time in seconds that you PC takes to complete a training "round", and episode_length_test is supposed to be number of time-steps your policy completes in test episodes. It is a bit weird that they are 0.

[Teleriann]

I do not understaind it either...

[Teleriann]

My question is, what should i writer to my bachelors? Should I just write what it means with note that it is not working and needs to being fixed or is there some problem why only mine is not working?

[yannbouteiller]

My question is, what should i writer to my bachelors? Should I just write what it means with note that it is not working and needs to being fixed or is there some problem why only mine is not working?

You should just ignore these metrics, they are for debugging purpose and are not meaningful. The idle_time here is normal, it just shows that your trainer is never idle.

The metrics that are meaningful are only the reward and losses.

(And losses are super hard to interpret in deep RL, in fact the only metric that is really meaningful is the reward)