Oddities with this system.

[SoftwareGuy]

Greetings,

I thought I'd provide some information on how it's going so far after I had to piece together a lot of stuff from various upstream issue tickets and whatnot to even get the thing running.

1. On first run, I see the AI is going through the first 300 species and it's getting basic scores of 39, 40 and sometimes 300 for those "oh wow it did something" moments. I assume this is normal, because SethBling's MarI/O does this for quite some time before it starts mutating and actually playing the game.
2. I've been keeping an eye on my machine's load average and the machine is a Core i5, 16GB of System RAM. It's averaging around 11.50 - 14. I assume this is normal as I see a lot of python3 processes running their tasks.
3. The ROM supplied apparently does not automatically start up at World 1-1. Instead, it seems to be randomly starting at first levels in different worlds. For example, on the title screen when it's at World 5-3, I see a bullet bill fly across the screen. Is this normal, or is this ROM a "bad" dump? Other Worlds include World 3-3, 6-3, sometimes 4-3 and 1-3. I assume it will then go back to World 1-1 and play from there?
4. Is saving and loading pools implemented? This is a important question because if not, then well... the training will be forfeited if it can't save/load.
5. The instructions need to be improved. Please add sections about installing the specific version of gym that has 'scoreboard' contained within (refer to the ppaequiette upstream issue tracker about that)
6. When does the tk window update? Does it update every generation or whenever it feels like it?

I will keep adding my findings in this thread as I evaluate it, however for a linux-powered variant of a learning AI, you're doing great work. Keep it up.

[kolt-mcb]

Hi SoftwareGuy,

Let me say how much I appreciate your comments, my Instructions are lacking/ not existent.

1. I hope you have used the latest release, I should have a new release in a week or so.

2. I am also hoping when you installed gyms mario env you that you installed the fork on my account, I dont do anything with the score board and the official release makes you beat by so much before progressing and I have no interest in that. depending on what code your running, if you just cloned from master it might do allot of thins, ive tried playing random levels, just select platform levels, so i dont think you have an issue with your rom dump.

3. Saving is totally functional! its super great! (not really I should rewrite all of it) but it totally works, big files can take a while, I have had 500+ mb files.

4. tk/matplotlib graph update on each generation (this should also be rewritten to use new data structure. )

final notes, I run this on a 32 core server with 16gb of ram, I want every ounce of speed I can get so I usually put /tmp on a ram disk so the emulator stores key presses from lua console as fast as possible, if you have an ssd this is probably not necessary, I say this because I have never had an issue with ram.(there are totally memory leaks in either my code/emulator/graphics things??? because after several days of training you computer will need to be rebooted no question)

[SoftwareGuy]

Thanks for the reply! Let me shoot back some replies 😄

I believe I did use your latest release. I had to do some modifications to the script though (I think). I think /tmp is mounted as tmpfs in memory, but I'm not sure about that.

I think the console output needs a bit of a revamp too. While saving and loading seems to be okay, there's no output in the console that would indicate that said activity has occurred (unless you get the "child specie X genome Y was born" texts). Within safe_file and load_file I added a simple print('Save/load completed!') call. Then when I save and load, I get textual feedback the script actually loaded everything correctly.

It might also be worth catching any exceptions that might occur during operation. An operation did throw in the towel once or twice and the script literally caught fire and I had to run good 'ol pkill python3.

As for memory leaks, I'm not 100% sure but I noticed that if you have two FCEUX instances running at the same time sometimes they fight each other for frame rate and you'll get things like 1 FPS while python scripts in the background spawn and do things. I am not completely sure why there are multiple forked (?) processes. Running with one instance and 200 population seems to be fine.

[SoftwareGuy]

Alright, so I took a look and I was using upstream's environment, instead of yours. Whoops.

What an easier way of installing the mario environment would be to just symlink your ppaequiette_super_mario directory to /path/to/python/lib/site-packages/gym/envs/gym_super_mario. I did this as I have your version of the ppaequiette_super_mario environment under $HOME/MoreIO/koltafrickenfer-gym-super-mario. Seems to be fine and avoids the usage of gym_pull which caused a Segmentation Fault after it seemed to install it. Now I'm getting the same FCEUX window layout as per your screenshot, however the window looks very distorted like it's been stretched a little too far - what's up with that?

[kolt-mcb]

I adjusted the resolution to fit 12 instances of the game on my screen at once, you can adjust theses settings in nes_env.py line 51 and it should look like this self.cmd_args = ['--xscale 1.5', '--yscale 1.45', '-f 0','--opengl 1.5','--sound 0','--nogui']. if you need other settings try man fceux from command line.

when I had used gym_pull at the time it was not deprecated. A symlink is fine, I prefer gym_pull but install how ever you like.

If you are running more than one instance of the game and fighting for resources than you need more resources.

[SoftwareGuy]

Brilliant - I changed the scale back to 1 respectively. The opengl parameter also doesn't take a decimal, it's either 0 or 1. When I set it to 1, I noticed my GPU took over the grunt work of rendering the FCEUX layout and I was getting the top end of 50 and 60.1 FPS. I also enabled the new PPU engine and a few other switches that I'd need to see the modified nes_env.py that I use.

I noticed that when Mario dies, the "Distance" as it's called seems to increase a lot before it actually stops. Is this going to be refined? I don't think it's valid to have the distance increase a lot after Mario falls into a pit, if you know what I mean, unless the game actually keeps him moving after locking the camera where he died.

If SethBling's code does it though, then i guess you're emulating his code behavior and my argument is void. You can easily replicate this behavior by getting 'walkers' that walk right constantly and then fall into a pit. If the game doesn't reset it, the distance usually increases from 100 to 300 (not sure if it's random or if it's determined by something) and then stops.

Another feature request I'd like to submit is a Generation Count on the TK UI, so I can leave this running for a day and come back to check on what generation it's up to. Maybe also auto-save on generation, so that we don't suffer from data loss.

At the moment it seems my species are learning how to walk, no jumpers or the ones that just keep pressing A yet.

[koltmcbride]

there is code in the emulator script that identifies when mario falls beneath the screen, but I think I broke it, yeah I know bad me... I do need a high score as well as generation counter, maybe I can get this done this weekend and get a new release out.

[SoftwareGuy]

Do you mind if I cloned and started submitting pull requests? I could contribute things that I find might be beneficial for people who want to kick start the AI, and also tidy up the CLI output a bit.

[kolt-mcb]

That would make me so happy I’m not sure you would believe it, I started a new job and haven’t had time/ been motivated to clean up the interface.

[SoftwareGuy]

Alright. It'll give me something to do while I'm idle between my real life job commitments.
I'll start by cleaning up the CLI, and then work with Tk.

[SoftwareGuy]

Got the latest commit, and loaded my pool data into the new version. The graph was instantly updated to what the network was before I shut down the box. Started the run again and while it immediately reset at World 1-1 (probably because it had no training data?), the new run is "intelligent". This is a big improvement over Sethbling's code, because his save/load system breaks unless you use a certain version of Bizhawk.

[SoftwareGuy]

Hmm, okay after a generation is complete and it goes onto the next one, a window titled <tk 2> appears and shows what seems to be button press graph. However, soon after it does that, this error popped up in the console:

Exception in thread Thread-116:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/[username]/MoreIO/core2/lib/neat.py", line 273, in nextGeneration
    children.append(specie.breedChildren())
  File "/home/[username]/MoreIO/core2/lib/neat.py", line 753, in breedChildren
    child = self.crossover(genome1,genome2)
  File "/home/[username]/MoreIO/core2/lib/neat.py", line 778, in crossover
    g2Rate= g2.mutationRates[mutation]
KeyError: 'ConectionCostRate'

Looks like it's just a typo in the name?

[kolt-mcb]

for some reason one parent has a mutationRate called 'connectionCostRate' and the other does not. this is something I was testing and is now disabled by default, I am not sure if the build you initially had even had this in it. I suspect that you are using a old save file where this was not implemented at all if you really want I can write some code to validate a connectionCostRate so old save files work but I would rather you just do another run... you should have been able to run many hundreds of generations at this point? I would have thought you should have reached some cap on your runs score at this point, where the entire population has reached some local maximum and no improvement happens at that point, is that not the case? have you tried any of the openai enviroments in main.py?

[SoftwareGuy]

I was using a old pool data file from the version that I had modded before your latest commits. That's probably why it was erroring.

I am currently running a "fresh batch". As there's no generation counter I have no idea how many where made. I didn't continuously have this running either all the time - sometimes I'd need to reboot the machine it was running on, so I'd save, reboot, and then rerun the python script.

I'll keep an eye on and see if this new batch throws the towel when it tries to make a new generation.

[kolt-mcb]

the x axis of the graph roughly displays the number of generations, for example
is roughly 250 generations, but your right I need a counter that displays the exact generation count.

[kolt-mcb]

I also need a graph of the score, average score per percentile.. I've been lazy :(

[SoftwareGuy]

Ah, I think mine was close to 100. I got to a stage that they wouldn't progress on 5-3, the Marios would just bonk themselves under a platform and die.

Still wondering why the game is resetting all the time on World 1-1... I might look into the environment file, because I do want it to try the first level - at the moment it really seems fixated in World X-3 where X is usually either 3, 5 or 6.

Would be really nice if we could implement a "linear" mode, where we specify the levels to learn in order. So the AI has to work out how to beat 1-1 before going onto 1-2 and so on. This was also a problem on SethBlings because it would time out before the next level loaded.

I don't know if there's any RAM addresses in the emulator that we could watch that are non-zero when a level is completed, though. Probably is, I'd need to look at the disassembly or learn some NES-fu with a memory monitor in an emulator.

[SoftwareGuy]

Although some levels are possibly going to be impossible to beat as I've seen upstream comments say - some of the castles would have to be omitted because it would get stuck at the puzzles that reset part of the level if you do it wrong.

[kolt-mcb]

I have your build set to play some specifics! crap I didnt meant to do that!

[kolt-mcb]

would like me to instruct you how to select levels or would like a release that plays all 32 levels (is slower)?

[SoftwareGuy]

Probably the former, if that's alright with you?

[kolt-mcb]

a note on playing linearly, you can not play this way, a better method is to play all levels on one run and take an average score of this. playing linearly means that NNs can have greedy strategies that only do well on the initial level. In short the most optimal way to rate a NN would be to place mario in every possible situation of the game, this would be the most accurate possible score but would take an unreasonable amount of time to progress.

In the future I would like to be able to select what levels are being played from the gui, since no one else was using this code to my knowledge I didn't really have a good reason to implement this over editing the JobTrainer function on line 108, where range(32) is all levels and for example [0,5,10,16] would play levels 1,6,11 and 17.

also a new a release has been posted.

[SoftwareGuy]

Got the latest release. Noticed that some of the children that "do nothing" and stay put now score 256 instead of just 40?

I also left it running overnight and no errors occurred when generations were completed, and a new one was spawned. It is definitely slower for each run, but the odd child will adventure into the unknown and usually get some good score before getting goomba'd.

If I can migrate this setup to a low-power system (ARM? Atom Quad-core?) I might set up a stream showcasing this AI system. However, one question - I'm using Ubuntu MATE and I'm not sure what you use for your linux (looks Ubuntu), but do you know any way to make the windows not overlap each other? Right now I have to manually drag the FCEUX windows away from each other, because they'll spawn one, then a second later, the second one spawns on top of the other.

[kolt-mcb]

I do know a way! but im a bit surprised that this doesnt all ready work for you..

1. I use an install of ubuntu desktop with mate installed, this by default will not over lap episodes. see here, https://clips.twitch.tv/DistinctApatheticVampireCharlietheUnicorn

2. maybe it was ubuntu server with mate installed I might have to check but I know it has unity installed as well.

3. you can use compiz to configure window animations and such.

4. I will have a new build, I have fixed an issue where members of the population might breed with a member of the population that was killed. I need to test if this actually improves performance.

[SoftwareGuy]

Compiz apparently doesn't like my nVidia GPU. It makes the screen super blurry and that makes my eyes hurt. Apparently I cannot load anymore - I saved the run data as "pooldata2.pool" and click "Start Run" with the default 2 jobs and 300 population. Nothing happens. Waited a while. Still nothing. No FCEUX instances, but I do get the plot in the window appear when loaded.

However, I do get the "[file] loaded..." text that appears when the file loads. It seems to also spawn the children correctly.

[kolt-mcb]

no you shouldnt have any issues with your save file

[kolt-mcb]

what gpu are you using?

[SoftwareGuy]

GPU is a nVidia GeForce GTX 750 Ti. I just pulled the latest commit to my machine, ran python3 mario.py, click start run, nothing happened... I see some python3 mario.py processes in top, but nothing else.

[kolt-mcb]

I tested an old save file and had no issues, does it run when not using a save file?
I have used this on amd graphics cards, a gtx 1080 and a gtx 660 ti

[SoftwareGuy]

Turns out that Ubuntu must have fucked around with something. I rebooted the machine and booted up MoreIO and everything started correctly this time around. As a bonus, the first species wandered right, got up to the first item box on World 1-1 and stopped.

[kolt-mcb]

I hope you are enjoying the program :)

[SoftwareGuy]

Hmm, okay so on my test box I got a error after letting it run for about 3 or 4 days straight that said "MemoryError". Seems to be that the AI's memory grew to the extent that my 32-bit ARM development board couldn't address more virtual memory (does that even make sense?) even though I had about 600MB free. FCEUX instances were "frozen" due to the script probably breaking the pipe that feeds them instructions.

I'll look into implementing a auto-save feature, like what SethBling's code originally did. Each generation it saves the pool as "autosave_genX.pool" so that if something like this occurs, it's easier to recover than lose 4 days of AI training.... last save I did was on the 18th January...