Support "graphics extensions" in mods

[Pidgeot]

Prompted by discussion in the DFgraphics thread, starting here, with my responses summarised here and followup here (links for background, and include design rationale).

The motivating issue is that creators want to support graphics for mod-specific creatures and items; this means supplying additional creature graphics and (hopefully) TwbT overrides. These components should be supplied by mods, while graphics packs remain vanilla-only.

I think the most elegant way to list the changes here is in the order they should be implemented - see the links above for a structured approach.

1. Adjust graphics install. A graphics pack should be treated as the last mod to be merged into a set of mods (including the empty set), to ease handling and ensure the graphics pack wins any clobbering (eg of image files).

2. Mod merging already handles the case of provided graphics raws, though at present this will usually lead to problems (see 1). Document this and suggest how to avoid problems - ie add, not override.

3. (or later, subject to upstream) After/if TwbT support multiple overrides files and subtype names. PyLNP must allow mods to add image files to <df>/data/art/, and overrides files to <df>/data/init/. Use the usual log-on-clobber approach for merging mods.

At this stage mods have first-class support only for a single graphics pack or style. The final step is to allow mods to support several "graphics extensions"...

• Each mod may include one or more graphics extensions under LNP/Mods/<my_mod>/graphics-extensions/. After my_mod is merged, skipping the graphics-extensions/ tree, PyLNP decides which if any extension to install (see below).
• This extension is then merged exactly as if it was a top-level mod (raw-wise; manifest and log handling is different, and no recursive extensions).
• Advice for creators: avoid non-graphics-related content in extensions without good reason. Consider providing the no-graphics version as base and using extensions for all graphics packs, including your 'default' pack.

...and PyLNP to select which to use at install time. This last point is a standing design question. Some options:

• In any case, we're finding a match between the extension, and the graphics pack.
• For the extension this means the dirname, unless we decide to add a new mapping in manifest.json (which I'd rather not do).
• For the graphics packs, we should use the name as displayed; if no matches are found try using the dirnames before giving up.
• How to match? I favor case-insensitive substring testing (extension.lower() in gfx_pack.lower()), but could easily be convinced to do something else.
• There will certainly be cases eventually were more than one graphics pack matches. Should we: (a) just use the first - in which case, how are they sorted? or (b) decline to use any? Logging is a must, either way.

I'll start work on the first section above.

[Issue created by PeridexisErrant: 2016-10-09]
[Last updated on bitbucket: 2016-10-10]

[Comment created by PeridexisErrant: 2016-10-10]

it literally doesn't work, as in "I get a stack trace when I try to install a graphics pack"

Yep, that second one looks very familiar. Like I said, it doesn't work! (but I think is moving in the right direction).

[Comment created by jecowa: 2016-10-10]
It doesn't let me install any graphics pack that modifies a vanilla file in the "raw" folder or one of its subdirectories. This may be related to the problem that PyLNP is having with merging raws. As a temporary solution, could you make it replace the raws instead of merging them?

This is the most common error I get when switching graphics:

#!errorlog

ERROR: Note: Failed to read JSON from ./../../LNP/Graphics/ TwbT_GemSet_24x/manifest.json, ignoring data - details follow
Traceback (most recent call last):
  File "./core/json_config.py", line 41, in __init__
    self.data = json.load(open(filename), **enc_dict)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 4 column 5 (char 108)

One time, I got this error:

#!errorlog

ERROR: Merging  TwbT_GemSet_24x: : file "graphics/graphics_example.txt": : Something went wrong while installing graphics
Traceback (most recent call last):
  File "./core/graphics.py", line 99, in install_graphics
    if not update_graphics_raws(paths.get('df', 'raw'), pack):
  File "./core/graphics.py", line 275, in update_graphics_raws
    if mods.update_raw_dir(raw_dir, gfx=(pack, built_graphics)):
  File "./core/mods.py", line 436, in update_raw_dir
    if -1 in merge_all_mods(mods_list, gfx[0]):
  File "./core/mods.py", line 174, in merge_all_mods
    status = merge_a_mod(paths.get('graphics', gfx), is_graphics=True)
  File "./core/mods.py", line 230, in merge_a_mod
    paths.get('baselines', 'temp', 'raw'))
  File "./core/mods.py", line 283, in merge_folder
    status = max(status, merge_file(mod_f, van_f, gen_f))
  File "./core/mods.py", line 321, in merge_file
    status, gen_lines = merge_line_list(mod_lines, van_lines, gen_lines)
  File "./core/mods.py", line 362, in merge_line_list
    outfile.extend(block)

[Comment created by PeridexisErrant: 2016-10-10]
https://bitbucket.org/PeridexisErrant/python-lnp/branch/gfx-exts-test#diff

Here's a prototype implementation, which is mostly feature-complete except for the following problems:

• that diff is against some minor refactorings in my default branch (cosmetic only though)
• support for TwbT changes is speculative and may not work, as well as being incomplete
• the simplify function in baselines shouldn't delete overrides, art in mods, or graphics extensions in mods anymore
• we'll need to rethink the mods UI to make graphics an explicit part of it at the end, thanks to the new order. This will also involve some docs changes re: use ascii, and core logic to support the UI change
• it literally doesn't work, as in "I get a stack trace when I try to install a graphics pack"

@Pidgeot I think it's ready for more active collaboration, if you've got the time to work out why it's broken :/

[Comment created by PeridexisErrant: 2016-10-10]
Thanks @jecowa - you're correct on all counts, and I agree that going with the longest matching substring (plus logging warnings and setting the 'merged with some issues'/yellow status) should work well.

[Comment created by jecowa: 2016-10-10]

How to match? I favor case-insensitive substring testing (extension.lower() in gfx_pack.lower()), but could easily be convinced to do something else.

Case-insensitive substring testing sounds good. I don't know what the "extension.lower()" and "gfx_pack.lower()" functions are, though. I guess it searches the currently-installed graphics pack's title if available, otherwise it uses the directory name.

There will certainly be cases eventually were more than one graphics pack matches. Should we: (a) just use the first - in which case, how are they sorted? or (b) decline to use any? Logging is a must, either way.

Do you mean cases where more than one graphics-extension matches the currently-installed graphics pack? –Like if a mod had extensions for both "Spacefox" and "Mayday" and the user made his own graphics pack titled "Spacefox Mayday"? In addition to logging, maybe ask the user which graphics-extension he would prefer (if it's not too much trouble).

Maybe for matches, the graphics-extension packs should be sorted by string length, that way longer more specific graphics packs are searched for first. So a mod author could have a graphics-extension module for a "Spacefox Mayday" in addition to having graphics-extensions for "Spacefox" and "Mayday".

So, in a case where more than one graphics-extension matches the currently-installed graphics pack, maybe use the graphics-extension with the longest string?