Question about the Lidar

[NDR008]

Hi everyone, I am working on an ambitious project (it is in particularly ambitious because I am not a hard core CS person, and am trying to do this project for my MSc in CS final project).

But it is similar to this project in that my project proposal looks at establishing tools/framework for interfacing a PlayStation 1 emulator with Python to then develop an agent that is developed using SAC to play Gran Turismo 1. So in part similar to what Sony did with Sophy (but less ambitious on how well I hope it will work).

Early on I was sure I can simply use openCV to process the screen and perform some edge detection to do something similar to your Lidar lines. However I've struggled with this approach. I then took a detour at using Ghidra to reverse engineer the game.
Made some useful progress in finding internal variables like vehicle speed - which will make things simpler and more similar to your solution for TrackMania 2020 (i.e. I do not need to use a NN to read the speed and so on).

But I could not make head-way on the lidar. Where can I look at how you've implement the spatial recognition / lidar?

Oh by the way - I read somewhere that you could only get 20FPS of screen capture?
Here I was experimenting with different ways: https://youtu.be/0t97yrOAKXs?t=1398
I was able to find a library called d3d that gave me VERY fast screen capture.
(Ironically for my project I do not plan to use it because I think I will use the emulator to share over the display-data directly to the python environment).

[yannbouteiller]

Hi!

The way we compute the LIDAR in the TM LIDAR environment is very specific to the plain road in TrackMania and probably won't work in Gran Turismo. We just look for black pixels on the trajectory of each ray, you can find the corresponding code here

This was just to speed up training in simple "toy" environments: nowadays we use full unprocessed images with CNNs.

The 20FPS value is not a bottleneck with screenshot capture. It is only an arbitrary choice for the control frequency that you can change in the config.json file if you wish. We chose 20FPS by default because (1) it leaves enough time between frames to infer the first dynamic moments of the car from the short history of 4 frames that we use by default, (2) it leaves enough time to compute the output of rather big deep learning models, and (3) it leaves enough time to capture screenshots.

[yannbouteiller]

Unless you edited the code, the config.json used by tmrl is in the TmrlData folder of your home directory

[yannbouteiller]

(And BTW if you cloned and modded the repo, you want to install the library by doing pip install -e . where your setup.py file is located. If you do pip install tmrl instead, this will install the old 0.3 version that is on PyPI)

[NDR008]

Need to figure out where/what is tmrl pointing to then, because it is th TMRLDATA folder that I modified for the setup.py... hmmmm
but things can go messy with mixes of pip / conda / manual-setups....

[NDR008]

I did not know this:
pip install -e .

I cloned the git, and then tried to work my way through setup.py.... HAH! 👍
This is cool, IF I can get this to work on GT1, I wonder if I can say I did enough for my dissertation.
Anyway, thanks.
By the way - is it ok if I keep using this discussion tab or would you rather have my ping you elsewhere? Especially when I step into the GT stuff?

[yannbouteiller]

You can keep using this discussion, np, it is here for that :)

[yannbouteiller]

(BTW don't use the image/CNN agent yet because I am training a new one currently, using a different preprocessor. The up-to-date code for that is on the yb/dev branch, I'll push to master shortly)

[NDR008]

ah, figured, evn tho I had manually modified things... actually I still have a TmrlData folder under user
I wonder were / what I need to modify to mak use of my other folder.

[NDR008]

Anyway, went for the default TmrlData folder now and woohoo it works! 👍
Thanks!!!

[NDR008]

Are these warnings:

C:\Users\USER\anaconda3\envs\tmrl\lib\site-packages\rtgym-0.7-py3.10.egg\rtgym\envs\real_time_env.py:365: UserWarning: Time-step timed out. Elapsed since last time-step: 1.5455756187438965
C:\Users\USER\anaconda3\envs\tmrl\lib\site-packages\rtgym-0.7-py3.10.egg\rtgym\envs\real_time_env.py:365: UserWarning: Time-step timed out. Elapsed since last time-step: 0.33658623695373535
C:\Users\USER\anaconda3\envs\tmrl\lib\site-packages\rtgym-0.7-py3.10.egg\rtgym\envs\real_time_env.py:365: UserWarning: Time-step timed out. Elapsed since last time-step: 0.3390779495239258

Part of the magic that rtgym is working on, and some sort of "too unsynced" situation?

[NDR008]

https://youtu.be/oOU20m6e62U
By the way, is there a reason that the lap time is so non-repeatable?
Is it because of the time-stamp / real-time problem variance or something on my end or something totally different?

[NDR008]

Sony did not pause, but they had insane computation power, and all in-game data pumped out by the game in an idealised manner.

For speed retrieval basic vehicle state parameters - that is mostly easy / done - I have figured out the memory addresses of several parameters such as gear, speed, turbo-boost, etc.
https://github.com/NDR008/TensorFlowPSX/blob/8abda279a98a4cd4ccff995cc54865304205c9e4/ReduxLua/gt.lua#L182
I can even remove the whole HUD by flipping 1 flag of the in-game emulated memory, to give any CNN a pure "virtual world" image.

When you say go for CNN - still something like the LIDAR would still be necessary right? I do not have the "black pixels" to help me out.

Regarding reward - that dilemma has been on my mind, and I saw that you wrote about it too, that speed would be a naïve reward.
There is a variable I found which seems to be related to "track segment" but... the variable I found suggests segments are pretty big (like start-finish line to first corner is one segment) - so the agent would not be able to determine if it did well or not.
The map was also on my mind, but I was not sure how to encode it as I would have to employ a CNN to visually understand it OR spend some more time trying to reverse engineer the map function in Ghidra (if I can find the related function - C and reverse engineering is also part of the "new" challenge stuff).

If I choose to pause or skip frames, I could investigate the ability of an AI agent at different frequency of observations. It is one of the things I am critical about on Sony publication.

[NDR008]

On the bugfix, I should have persevered. I had tried cpu().a.numpy() - misunderstood the syntax of that call.

[yannbouteiller]

A CNN enables you to use raw images, no need for a LIDAR. The LIDAR is just faster to train because it uses a simple MLP, but with the right hyperparameters CNNs are still a realistic solution (with the new hyperparameters that I pushed in the default config.json of Release 0.4.0, tmrl seems to train CNNs quite quickly in TrackMania, I am impressed :P)

For your reward function, the way I look at it, reverse-engineering of this map seems the simplest solution. Of course you could use the speed which would be much easier, but then your AI would only learn to go fast.

[NDR008]

Or to do burnouts and get the speedo-high without going anywhere.
Interesting, the full-on CNN vs LIDAR, if workable would be... really nice, as I could relate it to the real auto-industry dilemma - the Tesla non Lidar vs the everybody else with RADAR + LIDAR + Cameras...

Thanks I guess I will try spending some time on encoding the map, as for the CNN, I will probably pester you a bit on that later on.
Did you use a custom built CNN or did you use any pre-trained models like ResNet?

[yannbouteiller]

I use the custom and very simple CNN that you can find here

We have experimented with a few more complex models (pre-trained or not) such as ResNet, EffNet, MobileNet etc, but without much success so far. Truth is, deep RL does not like fancy tricks that supervised classification loves such as batch normalization. Plus, real-time keeps you from using anything too computational if you cannot run it fast enough.

[NDR008]

By the way - is the CNN version (non-LIDAR) working now?

[yannbouteiller]

It is working, and starting today it is the default on PyPI. I have just released 0.4.0

I am still training the CNN baseline and I will silently update the release over time, but the current baseline is sensible (at least when using the canadian skin, I have not tried with other skins)

[yannbouteiller]

Also, interestingly, it kinda generalizes to the end of the tmrl-train track although it has never seen it during training, which I expected would be hard for a CNN.

[NDR008]

Not sure what I am doing wrong...
I reinstalled tmrl, redownloaded the resource file from github, overwrote everything that was inside the trmldata folder, and I know have an agent that drives continuously left?

(Swapping back to lidar works)

[NDR008]

Nevermind, realised this is trained with a visible car
Is there a reason to making this on your end?

[yannbouteiller]

Yes, the CNN policy is trained on the default camera using the canadian flag skin.

The reason for doing this is that it makes the environment more Markov than giving only the front camera. In particular the agent can know whether it is drifting by looking at the marks behind the car, which is not possible with the front camera only (front camera only would likely require an RNN for achieving this)

[NDR008]

Since this discussion was about the Lidar, I think it would be better to close this discussion and open a new, but my next question is about training.
I want to understand a bit more about tmrl, because figuring out how to adapt for my needs.

The instructions say:
Record a reward for this track:
Execute:
Press e to start recording
Complete the track

I am not understanding what this means/does. Is the reward not a reward function that you coded?
Is it something that you learn by playing the game yourself?

Also I keep getting this warning from teh trainer.
INFO:root:--- NOW RUNNING SAC on TrackMania ---
wandb: Currently logged in as: tmrl. Use wandb login --relogin to force relogin
wandb: ERROR Error while calling W&B API: run tmrl/tmrl/SAC_4_imgs_pretrained was previously created and deleted; try a new run name (<Response [409]>)
wandb: ERROR Error while calling W&B API: run tmrl/tmrl/SAC_4_imgs_pretrained was previously created and deleted; try a new run name (<Response [409]>)

[yannbouteiller]

Hi! So, for the warning, you want to change the name of your run in config.json (or use your own wandb credentials - you get the warning because you try to log learning curves on wandb using a run name that has been used already)

About the reward function, the code that computes the reward in tmrl uses a trajectory that you need to record by following the instructions that you cite. The way this trajectory is used is described here