Some types of custom Salt modules do not work

[lkubb]

Qubes OS release

4.2 rc4

Brief summary

Custom Salt modules of types that should run on the management DispVM (the "master"), not the target qube, are not present in the environment (I discovered this through outputters, the rest is assumed, but very likely).

Examples for module types that do not/should not work: master tops, outputters, SSH wrapper modules

Steps to reproduce

• Write a slick custom module of one of the unsupported types, e.g. an output module that converts all data into 1337:

# Save this as /srv/salt/_output/leet.py
in_str = "AaEeIiOoSsBbLlTt"
out_str = "4433110055881177"


def output(data, **kwargs):
    data = str(data)
    return data.translate({ord(x): y for x, y in zip(in_str, out_str)})

• qubesctl --dom0-only saltutil.sync_all
• Verify it works locally:

$ qubesctl --dom0-only test.echo 'Hi there, my elite friend!' --out leet
{'10c41': 'H1 7h3r3, my 31173 fr13nd!'}

• Then try running it on a qube.

Expected behavior

$ qubesctl --skip-dom0 --show-output --targets sys-net test.echo 'Hi there, my sad friend!' --out leet
sys-net:
  {'5y5-n37': 'H1 7h3r3, my 54d fr13nd!'}

Actual behavior

$ qubesctl --skip-dom0 --show-output --targets sys-net test.echo 'Hi there, my sad friend!' --out leet
sys-net:
      Hi there, my sad friend!
  [ERROR   ] Invalid outputter leet specified, fall back to nested

Additional context

The expected behavior above is a 1:1 copy & paste with my fix, but it requires a bit more thought imho.

Usually, salt-ssh is run from a Salt master with a permanent cache dir. In Qubes OS, a new DispVM is created for each invocation. Custom modules need to be synced from the fileserver to the cache dir to be available to Salt. For salt-ssh targets, this is done automatically for specific module types (execution modules, states, grains, renderers, returners - the rest is irrelevant). The "master" on the DVM on the other hand would require a salt-run saltutil.sync_all for any to be synced (salt-run is not present by default, but can be created easily).

Possible fixes:

1. Include salt-run in the management package and make qubes.SaltLinuxVM run salt-run saltutil.sync_all before running salt-ssh
2. Send the synced cache folder together with the other Salt configuration (which I prefer). For this to work as expected, one would have to ensure all module types are synced (by default, e.g. tops and wrappers are not with salt-call, only salt-run.
3. Do what salt-run saltutil.sync_all does in some other way.

My current fix puts the /var/cache/salt/minion/extmods dir from dom0 into the package that's sent to the management DVM (makes more semantic sense to be in sync with dom0, avoids repeated syncing). Since wrapper modules are interesting from a Qubes perspective (e.g. re #2162) (tops as well btw), I think it would make additional sense to

a) provide salt-run by default on dom0 specifically for creating the master extmods cache OR
b) override the stock saltutil.sync_all execution module function to in addition sync those module types to the dom0 minion cache dir,

after which they can be sent together with the other configuration and written to /var/cache/salt/master/extmods inside the mgmt DVM (the actual fix for this bug).

[lkubb]

Some more thoughts on the issue of getting all module types into a dom0 extmod dir (worst to best imho):

1. Overriding salt.modules.saltutil would imply copying the whole module since extmods cannot override an existing function of an inbuilt module via __virtualname__. Doing this as a separate qubesutil.sync_all (or saltutil.sync_all_master) is possible, but confusing for both novice and experienced Salt users.
2. ManageVMRunner could sync them silently using the minion opts, but I don't think repeating this on every invocation of qubesctl is a good idea because of unnecessary latency. They should seldomly change.
3. It could just be documented that you need a way to run the saltutil runner module to sync all module types, either by salt-call saltutil.runner saltutil.sync_all, creating it manually or installing the salt-master package. The minion cache dir could serve as a fallback (which would only help with output modules, but still better than nothing).*
4. qubesctl could get a flag --sync-extmods, which results in a call to salt.runners.saltutil.sync_all using the minion opts. That would avoid a) having to worry about salt-run and two extmod dirs b) unnecessary syncing c) confusing people by conflating qubes-mgmt-salt specifics with Salt internals.

For every option except (3), prepare_salt_config_for_vm would then just include the extension_modules dir (/var/cache/salt/minion/extmods usually) in the package.

* This is the only case where we end up with two possible extmod dirs which need to be accounted for.

I will implement the fix via qubesctl --sync-extmods and submit it as a proposal.