Add export-map script

[timothymtorres]

This adds a Lua tool that exports all the tile data in a fortress map to a JSON file. It's based on export-map.lua by @mikerenfro that I rewrote. I used his repo as my starting point while I was looking for DFHack scripts to export tile data.

With this script you can select tile flags and attributes to be copied and extracted:

• tiletype
• shape
• special
• variant
• hidden
• light
• subterranean
• outside
• aquifer
• material
• evilness
• liquid
• flow

Depending on the options selected, the material tables will be included as KEYS to reference the number IDs. Also big shoutout to @quietust for helping me find the aquifer data.

[myk002]

please also add a line to the New Tools section of changelog.txt: https://github.com/DFHack/scripts/blob/master/changelog.txt#L29

[myk002]

What are the client requirements that led to this output format? The choices made here seem strange to me, and I could use some context.

[timothymtorres]

So right now I am taking the JSON data and importing it via another language called DM that is built around a game called Space Station 13. The conversion code is here:

https://github.com/timothymtorres/tgstation/blob/b19d164b370a7408b2ae0ec10c88553ba34a33a5/code/modules/admin/verbs/map_export.dm#L316-L578

The end goal here is to import DF maps directly without having to write my own terrain generation from scratch. I'm aiming to mod SS13 heavily enough so it can become a multiplayer version of DF.

This is what it looks like when you convert a DF map to SS13:

[myk002]

I meant what led to this particular output format? I've never seen this style before. For example:

``ARGUMENT_OPTION_ORDER``
    The order of the selected options for how data is arranged at a map
    position.

    Example 1:
        ``{"material": 1, "shape": 2, "hidden": 3}``

        ``map[z][y][x] = {material_data, shape_data, hidden_data}``

    Example 2:
        ``{"variant": 3, "light": 1, "outside": 2, "aquifer": 4}``

        ``map[z][y][x] = {light_data, outside_data, variant_data, aquifer_data}``

Why not have string keys instead of a dynamic-but-consistent ordering of elements? Doesn't that require undue complexity in the client? I'd expect something like:

map[z][y][x] = {light=light_data, outside=outside_data, variant=variant_data, aquifer=aquifer_data}

and not have an "argument option order" at all.

[timothymtorres]

I've never seen this style before.

I sort of wrote it on the fly, but I'm sure other libraries have used a similar technique.

Initially I considered using keys but the problem is time. I/O operations like writing to and reading from JSON are expensive in any language, especially with large amounts of data. Using dynamic positional arguments is a compression method for our data so we don't have to write as many bytes of data. I could have also used a library like messagepack to really squeeze the data together even further, but then it would become even more unreadable. It's also why I use numbers for most of the data instead of strings but provide a matching key table to convert the numbers to strings.

Anyway math time!

Using the example: map[z][y][x] = {light_data, outside_data, variant_data, aquifer_data}

• JSON with positional args - {true, true, 5, 2} ~7 bytes of data
• JSON with string keys - {light=true, outside=true, variant=5, aquifer=2} ~38 bytes of data

Assuming true/false is 1 byte.

In our example string keys results in 5 times slower performance. Considering how big maps are, this is a big deal. This only considers exporting the data. Importing runs into the same performance cost, so really it's x2. If you export 7 bytes of data, you need to import 7 bytes of data as well.

tldr; - It's a compression method so data processing is significantly faster.

[timothymtorres]

pre-commit.ci autofix

[myk002]

I don't think this is true. It's Lua lists that start at [1]. JSON arrays are zero-indexed. I don't think it is required to push the complexity of shifting the coordinates to the client.

[timothymtorres]

Interesting.

The JSON library I was using in DM to import the data counts the arrays starting at 1. They probably did that for compatibility reasons since DM arrays start at [1]. Thought it was universal.

Let me tweak the wording to reference that.

[timothymtorres]

Applied via 9338977