Handle plugins sequentially in apply

[Backfighter]

Currently an execution of holo apply works like this:

• Run scan for all plugins.
• Run apply for all entities from the scan phase.

I would like it to be:
For each plugin

• Run scan for plugin
• Run apply for plugin
  So the scan of the current plugin runs after the apply of the plugin before it.

But why?

So why am I requesting this?
Well is has to do with #10. We want a flexible system to make configs more generic and sharable.

I had an idea on how this can be achieved. And my solution is what I call the meta-plugin.
The meta plugin basically generates files for the other plugins using templates.
For this to work its apply needs to be run before the other plugins scan for their entities.
That's the reason for the change requested above.

The meta-plugin

So now that the why has been established I want to further detail the concept of the meta-plugin
and how I invision it to work.

The files for the meta-plugin will be located under /usr/share/holo/meta and be structured as follows:

{hologram name/group}/path/realtive/to/holo_recource_dir/file.tmpl

On apply the meta plugin loades variables from it's configurations file and complies stored templates (similar to replicator).
Variables can be defined per hologram or group (holograms can symlink files to the groups).
Environmentvariables will also be available inside the templates.
The compiled templates are then deployed into the holo_resource_dir. Using the example above the compiled template would be deployed at ${HOLO_RESOURCE_DIR}/path/realtive/to/holo_recource_dir/file.

Templated paths

Additionally one might want to define a dynamic deploy path for a template file (e.g. in oder to deploy a specific file into a users home directory).
This can be achived by placing a path specifier file besides the template file:

{hologram name/group}/path/realtive/to/holo_recource_dir/file.tmpl.path

This files contains a template string that is parsed into the files path realtive to were it resides.
It's content could for example look like this: {{var.user}}}/file meaning the compiled template would be deployed at ${HOLO_RESOURCE_DIR}/path/realtive/to/holo_recource_dir/stefan/file if the user variable is set to stefan.

Remarks

I am willing to develop the meta-plugin myself. But before I start working on it I wanted approval from you @majewsky. Do you argee with the idea? Are you willing to make the according changes to holo?

I would greatly appriciate feedback from you on this idea.

[majewsky]

Sorry for not getting back to you earlier, I really let this fall through the cracks.

On handling plugins sequentially in apply

I'm rather against this. The main reason is that I want holo scan to accurately describe the entities that holo apply will work on. If holo apply does scan-apply-scan-apply-..., the set of entities could change midway through (e.g. by manipulating a holo-users-groups resource file during holo-files apply) which violates the Principle of Least Surprise for me. If we want holo scan to be consistent with holo apply, we would have to run scan-apply-scan-apply-...-scan (i.e. the entire holo apply just without the apply for the last phase) which is obviously nonsense (and won't even work in general because holo scan is supposed to work without superuser privileges).

On the meta-plugin proposal

Given the above, doing the "meta-plugin" as an actual separate plugin is pretty much DoA, but I'm interested in having something like this as part of Holo itself.

I'm not sure if I understand your proposal in its entirety. A concrete end-to-end example would be illuminating.

A particular concern for me is that if we do anything like this, it should also cover the use-case described in #7, and I'm not seeing that in your proposal. #7 is IMO the biggest gap in Holo's resource-file model right now. It really limits the expressiveness of holograms. By comparison, #10 is not a particularly big concern for me anymore since Replicator turned out much more solid than the quick hack I had in mind. In fact, I ought to add a manpage to it, slap the "1.0" sticker on it and just close #10.

A counter-proposal

Given my current track record in terms of answering time, it probably makes sense to pack as much as possible in this response, so I'll include a counter-proposal. I let the idea of rendering resource files at scan-time simmer in my head, and was reminded of systemd generators. In case you're not familiar, systemd includes a bunch of helper programs called "generators" that produce unit files at startup time. For example, systemd-fstab-generator converts each entry of /etc/fstab into a mount unit. Generators in systemd are just programs that put a bunch of unit files in /run/systemd (which is otherwise analogous to /usr/lib/systemd).

In my proposal, a generator is any executable file at ${HOLO_RESOURCE_DIR}/generators/. I don't prescribe how the generator works in detail. If templating is desired, Replicator or any other templating tool can be used. Each generator is run once at scan-time and is given a environment variable, e.g. ${OUT} containing the path to a tempdir where it can render generated resource files. (I'm consciously saying "tempdir" here instead of something like /run/holo since holo scan may run with non-superuser privileges and not have write access to /run).

From that point, everything progresses like before. The other plugins pick up the generated resource files like they do static resource files. Since plugins expect only a single ${HOLO_RESOURCE_DIR}, we would have to symlink the static resource files into the tempdir and use the tempdir instead of /usr/share/holo, but that's only a minor inconvenience. (Praise my glorious foresight for not hardcoding any paths into the plugin interface. 😉)

Examples

The use-case from #7: We want to add initrd intel-ucode.img to all systemd-boot entries in /boot/loader/entries/*.conf.

$ cat /usr/share/holo/generators/add-intel-ucode.sh
#!/bin/bash
set -euo pipefail

for FILE in /boot/loader/entries/*.conf; do
  cat > "${OUT}/files/${FILE}.holoscript" <<-EOF
    #!/bin/sh
    sed '/^initrd / i initrd /intel-ucode.img'
EOF
  chmod +x "${OUT}/files/${FILE}.holoscript"
done

This would run replicator on all holo-users-groups resource files containing template strings.

$ cat /usr/share/holo/generators/templated-users-groups.sh
#!/bin/bash
set -euo pipefail

cd "${HOLO_RESOURCE_DIR}/users-groups"
for FILE in $(grep -l '{{.*}}' *.toml); do
  replicator < "${FILE}" > "${OUT}/users-groups/${FILE}"
done

Note that this example assumes that generated resource files take precedence over static ones in case of a name conflict, which seems like the most plausible of the possible behaviors.

You may have noticed that this example breaks my previously mentioned rule that generators ought not require superuser privileges (replicator usually needs to be root to read secrets from /etc/replicator.d). I considered if this is a problem with my proposal, but it's more of a "problem" with Replicator just having one store for all variables (unless you start fiddling around with ${REPLICATOR_INPUTS} which may turn into a mess rather quickly). If this really turns out to be a problem in practice, I'd rather solve the problem on the right layer, i.e. in Replicator.

[Backfighter]

Thank you very much for your response. I really like your counter proposal. I seems to solve the problem just right and also doesn't have the problem with dynamic paths that my initial idea had. I think it's the right way to go forward since it also allows you to use whatever template engine you prefer.

One thing I think would be important: ${HOLO_RESOURCE_DIR}/generators/ should be scanned recursively. This allows to structure your generators in any way you want so that you can find the right generators more easily (e.g. you could sort generators by deploy paths).

[majewsky]

Yeah, makes sense. Any executable file somewhere in there should be considered.

[Backfighter]

I'm currently working on a pull request for this feature and encountered an issues along the way:

Problem with the current proposal

Wouldn't the symlinking to $HOLO_RESOURCE_DIR require root privileges as well?
Maybe introducing multiple entries to $HOLO_RESOURCE_DIR is the better solution because it will also avoid confilcts with existing static files.
Additionally this would make live way easier for people that don't use packages and just put everything in $HOLO_RESOURCE_DIR into a git repository.

Changing to multiple entries in $HOLO_RESOURCE_DIR would require a new api version and plugins using the old api version would receive the normal resource dir while plugin with the newer api version would receive mutiple entries in $HOLO_RESOURCE_DIR.

Temp dir

To avoid the need for root privileges in scan we could use the existing $HOLO_CACHE_DIR. Alternatively ~/.cache/holo for normal users and /var/cache/holo for root user could be used.

Generator behavior

I would allow generators to decide themselves wether they want to overwrite files or not.
They can for example use timestamps to avoid unnecessary regenerations.
I would run generators in alphabetical order (standard file sorting).
Later generators would be allowed to overwrite stuff from previous generators.

[majewsky]

Wouldn't the symlinking to $HOLO_RESOURCE_DIR require root privileges as well?

When I say "symlink into $HOLO_RESOURCE_DIR", I don't mean "symlink into /usr/share/holo". The virtue of $HOLO_RESOURCE_DIR being a variable is that we can just point it elsewhere without breaking existing plugin implementations. So we can build another tempdir (so no root privileges are reuqired) that contains all resources (both generated resources as well as symlinks to the static resources in /usr/share/holo) and give that to plugins as $HOLO_RESOURCE_DIR.

[Backfighter]

Oh I see thank you very much for the clarification that makes way more sense.

In terms of the temp dir would you prefer $HOLO_CACHE_DIR or ~/.cache/holo and /var/cache/holo?
The latter have the adventage of being preserved during reboots and avoiding conflicts with other plugins in /tmp. While the first has the adventage that no modification to util/holo-test is needed (altough it's a minor one).

[Backfighter]

@majewsky have you seen my PR #52? What do you think?

[majewsky]

I'm aware of it. Haven't managed to review it during the day, and after work my budget for code review is quite exhausted. I'll try to get to it over the weekend.

[Backfighter]

No problem. Take your time. Just wanted to make sure you noticed it and it didn't fall between the cracks ;)