[Feature] Implement proper support for MAME CHDs

[lemonadecbc]

MAME CHDs are expected to be stored in this directory structure:

<machine name>/<rom>.chd

Currently there's no way to tell oxyromon to sort files like this, but chd-files are stored in root and/or 1G1R-folder. The idea is to allow using oxyromon sort-roms with parameter that would allow use of machine-tag from dat-file and store disk-tagged CHD's when sorting the rom files and exporting them.

[alucryd]

Going to tackle this next, need to get my hands on a MAME game with a HD or LD file.

[alucryd]

@lemonadecbc Almost done with this one, although I just realized that the MAME CHD DATs store the SHA1 of the compressed data, not the uncompressed data. This is the complete opposite of everything else out there which is quite unfortunate as I'll need to handle this special case. Also they don't provide size information at all, which is another bummer...

Edit: Correction, it's not even the SHA1 of the compressed CHD, it's the SHA1 of the uncompressed data + CHD header + CHD metadata, which is different from the compressed, and uncompressed data. It's going to be fun :)

[alucryd]

Importing MAME CHDs should now work with d1858b7

They come with limited support for now, commands like check-roms won't work as I don't have any data hash to work with in the DAT. I could extract the CHD and enrich the database with the computed hashes, but this will be extremely time consuming, and there's no guarantee the extracted data won't be corrupted given the huge sizes.

Another option would be to use a dir2dat DAT instead (like the ones from Pleasuredome), but then I would need to make an exception and compute the CHD hash without extracting it first.

In any case I would advise using separate DATs for ROMs and CHDs for now as I'm not sure how rebuild-roms will behave with CHDs (well I'm pretty sure the answer would be badly, ie they would be zipped with the regular roms).

[alucryd]

Just released 0.20.0, leaving this open for now as there are still a couple things I need to take care of.

[lemonadecbc]

I was going to try this new feature (thanks for awesome job!), but not sure which and how the CHD dat should be imported as all the MAME v0.272 DAT's from ProgrettoSnaps seem to give error:

"Error: SimpleError { err: "Failed to deserialize DAT file, duplicate field rom" }"

For example MAME_CHD_272.dat is in Logigx format, but is giving this error. Is there something wrong with this MAME version or is this a bug introduced with 0.20.0 upgrade?

[alucryd]

Thanks for the heads up, I see the Progetto DATs mix rom and disk tags, which is causing the issue. All my testing was done using pleasuredome's DATs, where the CHD DAT only has disk tags: https://pleasuredome.github.io/pleasuredome/mame/

I'll adapt the logic to try and support both.

[alucryd]

Fixed in 04501c8.

Separated rom and disk parsing, as a quick hack I still merged them afterwards. Still a first step towards a working rebuild-roms subcommand for games with disks, treating disks differently will allow me to be skip them when zipping arcade roms.

[lemonadecbc]

Sorry if I'm rushing here, but did you try to fix already parsing Progetto DATs? I see the DATs are now being processed further, but there's another error this time:

Rom "os12.rom" not found in game "torchh" parent/bios, please fix your DAT file
⠉ 327/1226 505.9174/s 00:00:00 (00:00:01) [#####################-----------------------------------------------------------]thread 'main' panicked at src/import_dats.rs:552:42:
called Option::unwrap() on a None value
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

CHD dat for Progetto also seems to get a lot of "please fix your DAT file" warnings before this final fatal error.

Pleasuredome DATs seem to work well, thanks for hint.

[lemonadecbc]

By the way, I also noticed that Pleasuredome's CHD DAT for 0.272 (merged) does crash when trying to run sort-roms -w:

Wanted:
⠁ thread 'main' panicked at src/sort_roms.rs:403:38:
called Option::unwrap() on a None value
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

That line of code seems to reference to rom.crc, but the DAT file only has sha1 and no CRC. If I manually edit DAT to contain crc="00000000" for all disks, error goes away.

[alucryd]

Did not try with -w, thanks for the heads up, that's an easy fix I'll add in a couple hours.

As for progetto I'll have a look. These warnings happen when there are references to non-existent files in parent roms, these need to be fixed by the DAT provider. I'll see where the error comes from though, I probably expect CRC to be non-empty like with the sort-roms failure.

[alucryd]

The develop branch can now import the progetto DATs, and -w works again with sort-roms. I'll resurrect my mister in the near future and start exploring arcade again. Will fix and improve arcade in oxyromon as I go along. I'll probably make a patch release tomorrow with the recent fixes.

[alucryd]

Just released 0.20.1. I switched to Progetto's DATs as they seem to be the most popular. I'll probably work on a self-healing feature for rebuild-roms next as I noticed there are a lot of games with identical files (despite being named differently) in MAME, oxyromon could automatically copy missing files from other games to complete partial games. Then I also need to make 1G1R work for arcade games in sort-roms, even though there are no region information in these DATs, I could still put parents and bioses in the 1G1R folder.