Cross-Platform Filepaths

[mharradon]

Hello,

Thanks for this tool! It still chugs along extracting Blizzard files :)

It appears when run on a Unix platform that files get extracted into single files concatenating subdirectory names with Windows separators e.g. a single file called A\B\file.xml. I'm not sure if that's due to Windows convention inside the particular files I have or if that's a universal property of MPQs.

It seems desirable to have a normal directory tree get extracted from that (and then to support recompressing from that normal directory tree).

Does the problem as I describe make sense? Would PRs for this functionality be of interest?

Thank you!

[eagleflo]

Hey there! You're welcome, this was a fun little project to work on when StarCraft II was just released.

All of the files I used to test mpyq with when developing it had a flat namespace, so subdirectories never really came into the picture. The filenames within an archive are indeed hardcoded into the (listfile) of the archive. (Out of curiosity, which MPQ archive are you currently working with?)

At this point changing this behavior would be a big breaking change. It would have to be released as a new major version.

If you're interested in developing a PR for this, I'd start with looking at:

mpyq/mpyq.py

Lines 262 to 271 in 6bfba18

	def extract_to_disk(self):
	"""Extract all files and write them to disk."""
	archive_name, extension = os.path.splitext(os.path.basename(self.file.name))
	if not os.path.isdir(os.path.join(os.getcwd(), archive_name)):
	os.mkdir(archive_name)
	os.chdir(archive_name)
	for filename, data in self.extract().items():
	f = open(filename, 'wb')
	f.write(data or b'')
	f.close()

It could be as simple as calling os.path.split on filename to get the directory and then calling os.makedirs to create the necessary subdirectories before writing the file itself. Unfortunately the exist_ok parameter only appeared in Python 3.2, so there may be some extra effort involved.

[mharradon]

Thanks for the reply!

Yes, I think it's as easy as the implementation you describe. I'm using this on SC2Map files created by DeepMind for use as minigames in reinforcement learning tasks - maybe it's unique to those files, but I would have expected many SC2Map files have a similar issue.

For instance, for the "CollectMineralShards" minigames, after extraction on my Mac:

> ls
Base.SC2Data\GameData\ValidatorData.xml
CellAttribute_Vur
ComponentList.SC2Components
DocumentHeader
DocumentInfo
DocumentInfo.version
GameData.version
GameText.version
MapInfo
MapInfo.version
MapScript.galaxy
Minimap.tga
Objects
Objects.version
Preload.xml
PreloadAssetDB.txt
Regions
Regions.version
Triggers
Triggers.version
enUS.SC2Data\LocalizedData\GameHotkeys.txt
enUS.SC2Data\LocalizedData\GameStrings.txt
enUS.SC2Data\LocalizedData\TriggerStrings.txt
t3CellFlags
t3FluffDoodad
t3HardTile
t3HeightMap
t3SyncCliffLevel
t3SyncHeightMap
t3SyncTextureInfo
t3Terrain.version
t3Terrain.xml
t3TextureMasks
t3VertCol
t3Water

Those backslashed paths are actually single files - remarkably, when repacking directories like this (with mpqtool written in Rust) the maps work on SC2 both when these backslashed files are used and when I manually "repair" the directory structure. The only issue is that on Windows these are invalid filenames.

It also appears on Windows that an exception is thrown when these maps are unpacked at file creation as one would expect.