Save file format questions

[euan-forrester]

Hello, I'm the author of savefileconverter.com: https://github.com/euan-forrester/save-file-converter and I'm looking into adding support for MiSTer Saturn files to https://savefileconverter.com/#/mister

From what I've seen, it looks like MiSTer Saturn save files are the same as mednafen's (i.e. just what's written by the Saturn BIOS) but are "byte expanded" from 8 -> 16 bits by inserting bytes containing 0x00. It looks like only the Saturn's internal memory is emulated at the moment, and users currently can't save to cartridge memory?

Is this format set in stone now, or are there plans to change it in the future? Did I miss any details about how it works? Are there plans in the future to emulate cartridge save memory as well?

Thanks for all of your work on this core, and for any help you're able to give me adding support for it to savefileconverter!

[srg320]

From what I've seen, it looks like MiSTer Saturn save files are the same as mednafen's (i.e. just what's written by the Saturn BIOS) but are "byte expanded" from 8 -> 16 bits by inserting bytes containing 0x00.

Yes. Because 8-bit SRAM is connected to the low byte of the 16-bit bus.

It looks like only the Saturn's internal memory is emulated at the moment, and users currently can't save to cartridge memory?

Right.

Is this format set in stone now, or are there plans to change it in the future?

I don't plan to change it.

Are there plans in the future to emulate cartridge save memory as well?

Yes.

[mhmetalheart]

Hello,
I also do some exchange job for Saturn saves, hope I can give you some suggestions.

1. You can just use PS KAI to import it into Saturn RAM or the cartridge. You need short the file name. and change to *.bin. For example: Idol Janshi Suchie-Pai II (Japan) (Disc 1).sav. You need change to: Idol.bin (less then 8 letter). Now your PS KAI can read it.
2. if you export from Saturn by PSKAI you got an BUP file, you need use your tool: SaturnSaveConverter. You will got RAW, then, use: SS_Backup_RAM_Parser, inject the save back to the MiSTer. If you export by sdloader, you should got full RAM backup or Cartridge. You need export the single save you need by SS_Backup_RAM_Parser, then, use your tool like above.
3. Also I can provide some information about the saroo. That's the most hardest: if you try to converter it to Saroo, that was painful. You need use your tool: SaturnSaveConverter. convert to SSF, hook, (single save file). Then, Import it to Saroo. Also need convert to RAW like above from console to MiSTer).
   Hope you can update your tool to help us, make the save exchange much more easier.

[euan-forrester]

@mhmetalheart : Thank you for that overview! It matches up with what I've seen in my research, and I'm planning on supporting all of those formats (full memory image, .BUP, MiSTer, and Saroo) on the savefileconverter site and you'll be able to convert between them.

You added some great details that I didn't know because I don't have access to all of the hardware to test with. May I ping you in the future for some testing?

You can find my email or discord here: https://savefileconverter.com/#/about

[euan-forrester]

@srg320 Thank you for your quick reply, and everything sounds great!

I just have a couple more questions about when you support cartridge saves in the future:

• Is it likely that they'll look similar to the current internal memory saves? Like, "byte expanded" but otherwise identical to what the Saturn BIOS writes out?
• Will they be in a separate file, or instead will they be optionally concatenated with the internal memory saves into one file -- similar to how the Sega CD core handles it?

Thank you again for your reply: it was very helpful!

[mhmetalheart]

@euan-forrester just only 'separate file'. That will use the SD card to make the 'internal' to be an infinite save device which named by the game name. Easy to find out. That save file can contain the bios, PSKAI, and savedata. And needn't to do another painful thing: to make a custom internal or cartridge which have HUUUUUUGE space like saroo did (if still like the original one, that space is not enough to contain a lot of games, so, need 'change' another internal or cartridge, painful). Easy, and very elegant.

@srg320 Actually, I have a save issue, because of no 'save cartridge', so, one game cannot play: [Sega Saturn de Hakken!! Tamagotchi Park]. On console that stupid game need check the cartridge which contain a save file in it. Otherwise, cannot start the game. So, how to play that game on MiSTer?

[srg320]

I just have a couple more questions about when you support cartridge saves in the future:

• Is it likely that they'll look similar to the current internal memory saves? Like, "byte expanded" but otherwise identical to what the Saturn BIOS writes out?

Most likely, but I'm not sure yet.

• Will they be in a separate file, or instead will they be optionally concatenated with the internal memory saves into one file -- similar to how the Sega CD core handles it?

Common file. First 64Kb of internal memory.

@srg320 Actually, I have a save issue, because of no 'save cartridge', so, one game cannot play: [Sega Saturn de Hakken!! Tamagotchi Park]. On console that stupid game need check the cartridge which contain a save file in it. Otherwise, cannot start the game. So, how to play that game on MiSTer?

I've seen this issue #166. You'll have to wait.

[euan-forrester]

@srg320 That all sounds perfect and makes sense. Thank you very much for all of your answers, and I'm looking forward to getting Saturn support added to the site!

[euan-forrester]

@srg320 Hey, one thing I noticed in the example files I was provided is that one of them had some semi-random garbage data in the "expanded" bytes for the first 0x40 bytes of the file.

Is this an artifact of an older version of the core or is it something I should expect going forward? If so, is there a maximum offset that this garbage can occur, or should I expect it throughout the file?

I've attached the file if you're curious to take a look: Sega Rally Championship Plus (Japan).sav.zip

Thank you as always for all of your help!

[srg320]

The high byte is ignored in the game code. The value of the byte does not matter. In the original it is an open bus, read as 0xFF.

[euan-forrester]

Gotcha -- thanks for that!

[Zet-sensei]

@mhmetalheart Backup cart support is added, will need also a new Main MiSTer file.
0c0e010

[euan-forrester]

Oh nice -- excellent! Thank you!

I had a look through the commit but wasn't able to understand it: can someone please attach a sample file save file that includes backup cart data that I can test with?

[Zet-sensei]

Please see that savefile with a complete Rayman playthrough on the internal RAM + a save after the first level on the Backup Cart.

Rayman (USA) (R2).zip

Once you could understand the format, I will need some help for converting a Tamagotchi Save Cart to a MiSTer compatible file.
We have a .BUP file of an original cart done and that save file is necessary for launching that specific game.
TAMAGOTCHI_.zip

[mhmetalheart]

Just ignore my last comment.
JOB DONE! I understand what you did!
See attach. Should be what you need.
Sega Saturn de Hakken!! Tamagotchi Park (Japan).zip

[mhmetalheart]

Just use the: SS_Backup_RAM_Parser. Load that Rayman.sav. Then, insert the BUP (or whatever, for example, I use my own bin file). Then, save, and rename to TAmagotchi.sav, put it back to MiSTer save folder. Load game, will start.
BUT! after that, you can start the game, but in game, press start, select the right icon (quit game), won't ask you 'save data' (no save data screen). So... Still failed... What should I do now? Rename to *.bin. And 'load cartridges' in OSD?

[mhmetalheart]

I change the *.sav to *.bin. Loaded the Rayman.bin. can found that backup cart. But I didn't see my inserted TAMAGOTCHI file. Let me keep try. Please wait.

[mhmetalheart]

@euan-forrester Sir, I'm totally failed. Now, I can do nothing.
As I can see, lf load by 64 cluster, I can see 2 rayman file. But load 256, just can see one. And after I delete one in 256. I can see nothing in the 'backup cart' on MiSTer. So, that means: if save in 256=backup cart. if save in 64=ram.
Use SS_Backup_RAM_Parser. both 0.95 and 0.99 (mostly 0.95, I always have crash on 0.99, but don't know why).
So, I insert the tamagotchi.bin (must be the SSF HOOK, otherwise, the comment was totally unreadable. in my experience on other saves, if unusual, then, cannot read) Both in 64 & 256 cluster mode. After that, I put the bin back to MiSTer, load the backup cart. Check the memory, I can see nothing, but the space isn't 80XX which is 6901, that means the save file is THERE! But unreadable.
Then, I do some test, I insert some saves into it (both in SSF HOOK). in 64 cluster. Then, read the file on MiSTer. It make me totally confused. some of them hidden, can see nothing. Some of them in backup cart, some of them in ram.
I can do nothing now. Need your help.

[euan-forrester]

@Zet-sensei Thank you for providing that file!

@mhmetalheart Once I've added support for these formats to the savefileconverter site it will cover the use cases you outlined and I'll be able to support you on the savefileconverter discord.

@srg320 I took a look at the file, and I see that at offset 0x10000 is the backup cart data as expected.

What I found unexpected though was that block 0 of the backup cart data wasn't completely filled with BackUpRam Format. In mednafen, all 0x200 bytes of block 0 are filled with that string (which lets the reader infer the block size), but in this file only the first 0x40 bytes of block 0 have the string and the remaining 0x1C0 bytes are just 0x00.

So before I add support for this to the savefileconverter site I just wanted to check if the format is finalized or if you plan any further changes.

Thank you as always!

[mhmetalheart]

@Zet-sensei @srg320
I thought whole night. I understand, you put the internal and external memory together as one save file. But as a user, I don't think that's a good idea. That means all the users now their save file will changed. But just only a little games need the external cart to save, not all of them. And that file was a little big, the core need a bit longer to save it. Now all them games will be 1MB save file in the SD card. And most users will be confused 'why I cannot put all my saves into one backup cart?'.
So, can you find another way, let the external memory just a single 'cart' which can loaded by the 'load cartridges' only? That means make the core have 2 save files, 1 with the game as the *.sav. Another one is the backup cart. If need load it by ourselves. And single cart is very easy to convert to emulator or real console cart. : )

[srg320]

What should I do now? Rename to *.bin. And 'load cartridges' in OSD?

The Load Cartridge menu is for ROM cartridge only.

@srg320 I took a look at the file, and I see that at offset 0x10000 is the backup cart data as expected.

What I found unexpected though was that block 0 of the backup cart data wasn't completely filled with BackUpRam Format. In mednafen, all 0x200 bytes of block 0 are filled with that string (which lets the reader infer the block size), but in this file only the first 0x40 bytes of block 0 have the string and the remaining 0x1C0 bytes are just 0x00.

I don't know what reason mednafen fills 0x200 bytes. But I do know that the bios fills 0x40 bytes when erased.

I thought whole night. I understand, you put the internal and external memory together as one save file. But as a user, I don't think that's a good idea. That means all the users now their save file will changed. But just only a little games need the external cart to save, not all of them. And that file was a little big, the core need a bit longer to save it. Now all them games will be 1MB save file in the SD card. And most users will be confused 'why I cannot put all my saves into one backup cart?'.

The core loads/saves the backup cartridge memory only when the option Cartridge->Backup is selected. Just select the option for those specific games and don't select for the rest.

So, can you find another way, let the external memory just a single 'cart' which can loaded by the 'load cartridges' only? That means make the core have 2 save files, 1 with the game as the *.sav. Another one is the backup cart. If need load it by ourselves. And single cart is very easy to convert to emulator or real console cart. : )

First you have to reach a consensus, and then I'll figure out how to do it.

[mhmetalheart]

@euan-forrester I have a question like I said above. Yes, we need think about which way is the best way. As a common user, I thought the external backup cart as a single save file is a good idea, not combine with the internal ram. In this way, you can make the convert tool much more easier, right? And I tried a lot of way, I haven't any idea that how to import the tamagotchi save file into the external. Please help.

[euan-forrester]

@srg320

I don't know what reason mednafen fills 0x200 bytes. But I do know that the bios fills 0x40 bytes when erased.

Huh, well fair enough then! I assumed it was so that you can infer the block size of the file. I took a quick look through their source code and found that you're right and they do indeed set that string manually rather than through the BIOS.

First you have to reach a consensus, and then I'll figure out how to do it.

If you're asking my opinion, I vote to be consistent with the Sega CD core and have the cart memory optionally appended to the internal memory, only when it contains data.

I think that having a common cart memory was fine back in the day when we all had only a few games. But nowadays, especially on devices like the mister, users can try a lot of games quickly and could end up quickly filling a common cart memory.

Then we've put the burden of managing that memory on the user:

• They would have to delete saves they no longer need, to make room for new ones
• Importing saves from other devices would be a pain because they'd need to be appended to this file while preserving the other saves there, using an external program such as the savefileconverter site. If I've learned anything from running this site, it's that many users really like frequently porting their save files back and forth from the mister :).

I'd be wary of assuming that save file I/O was a bottleneck without profiling, but if it is there's a few things I can think of to improve it:

• First, writing out a non "byte expanded" file would halve the amount of I/O to do
• Second, we could it compresses the cart memory similar to what mednafen does. It uses gzip, which may be too heavyweight for the mister. But I'll bet the mister could go a long way just using Run-Length Encoding: a non "byte expanded" cart save file will be almost all 0x00.

[euan-forrester]

@mhmetalheart

I haven't any idea that how to import the tamagotchi save file into the external. Please help.

This will be supported on the savefileconverter.com site once I release the various Saturn tools I'm working on for the site. Until then I can't help you, but once I release these tools you'll be able to do this and I can support you on the savefileconverter discord.

[mhmetalheart]

Sega Saturn de Hakken!! Tamagotchi Park (Japan).zip
Hello. I successfully made a sav file which contain the tamagotchi save. Please check.
(I ask a Chinese hacker YZB, for help, and change the binary file. So, it won't require the save data in the external cart. Then, I start the game, save the game, not got that save file)
Hope that is useful for both of you.

And I still think make the external cart as a single cart *.bin file is much better than together with game sav files. But I'm just a gamer, so, you can do what you like. I will always agree on it.