Do we need digests for Lima templates

[jandubois]

In #3009 (comment) @AkihiroSuda wrote:

[This has to support specifying sha256 digest]

base:
  location: "https://example.com/a.yaml"
  digest: "sha256:deadbeef"

As a side topic, limactl start https://... should also have a flag like --template-digest=sha256:deadbeef

I find this cumbersome to use. I think if you want a locator for an immutable template, you should use a content-addressable store to fetch it from instead of requiring an additional digest.

Trust assumptions:

• DNS will provide the correct IP address for github.com
• TLS will verify that https://github.com content originates from GitHub
• Git commit ids cannot be forged

Then you can use a raw commit id to point to a specific version of the template, e.g. to run the Lima 1.0.2 release version of template://fedora:

$ limactl start --tty=false https://raw.githubusercontent.com/lima-vm/lima/7cea6f851656ee9aac606d02ea055d49d1c41cf2/templates/fedora.yaml

We could add another scheme to download templates from OCI registries, but I would not spend any effort on it unless somebody really needs it.

These URLs will work fine with turning relative locators into absolute ones.

So I don't think we need any explicit support for checksumming template files.

---

As for using checksums for individual basedOn or file references, I think this is totally impractical. If this is important, then you would run limactl template assemble (or whatever we end up calling it) to embed everything into a single file, and then store a checksum for that.

[AkihiroSuda]

Then you can use a raw commit id to point to a specific version of the template,

Not every template is located on GitHub, so the digest should still make a sense for non-GitHub URLs

[afbjorklund]

You could use ipfs://bafybeib2irmeecuu74t3znqkvitir7yqt3tfv5l4527eisiklgslarzo6q/fedora.yaml

But then you would still have to fall back on a gateway, so same problem as with OCI references*.
http://127.0.0.1:8080/ipfs/bafybeib2irmeecuu74t3znqkvitir7yqt3tfv5l4527eisiklgslarzo6q/fedora.yaml

* suddenly we can't curl anymore, so have to wrap json parsers and whatnot (or run an oras gateway)?

[afbjorklund]

This seems to be an interesting take on the "curl pipe sh" debate, currently stuck in "curl + review + sh" limbo

https://www.tumblr.com/curlpipesh

https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script

I suppose one could use IPFS, but that is as cumbersome as ORAS. Some people use anchors as a workaround.

[afbjorklund]

Then again it already has too many (unused) flags.

• [feature] Checksum when downloading file curl/curl#1399

[AkihiroSuda]

Sent a PR to GNU coreutils instead of curl:

curl https://example.com/install.sh | sha256sum --check-stream deadbeef | sh

https://lists.gnu.org/archive/html/coreutils/2024-12/msg00005.html

[afbjorklund]

This feature was actually available in the BSD sha256, with the -c flag (no long opts).

$ sha256 -c dd0cdb334df053a5016a8665d9f52279a6550400cc7e466b6a6a9dabc6b8cd70 fedora.yaml
SHA256 (fedora.yaml) = dd0cdb334df053a5016a8665d9f52279a6550400cc7e466b6a6a9dabc6b8cd70

https://github.com/afbjorklund/bsdsum

[afbjorklund]

Currently I was using separate files for verification, which worked for everything but BuildKit (SBOM, sigh)

https://github.com/afbjorklund/lima-the-hard-way/blob/main/sha256sums.txt (as in sha256sum -c)

     -c string
             If the program was called with a name that does not end in sum,
             compare the digest of the file against this string.  If combined
             with the -q option, the calculated digest is printed in addition
             to the exit status being set.  (Note that this option is not yet
             useful if multiple files are specified.)

     -c file
             If the program was called with a name that does end in sum, the
             file passed as argument must contain digest lines generated by
             the same digest algorithm with or without the -r option (i.e., in
             either classical BSD format or in GNU coreutils format).  A line
             with the file name followed by a colon “:” and either OK or
             FAILED is written for each well-formed line in the digest file.
             If applicable, the number of failed comparisons and the number of
             lines that were skipped since they were not well-formed are
             printed at the end.  The -q option can be used to quiesce the
             output unless there are mismatched entries in the digest.

[afbjorklund]

Also implemented different output formats, which was another one of those rejected features...

Known encodings: none, base16, base32, base32hex, base58btc

[jandubois]

Then again it already has too many (unused) flags.

That is my concern: it is just another option that clutters up the --help output that nobody will ever use.

And it is not that the issue cannot be solved easily with a few lines of shell script, so I don't feel the need to provide a shortcut for a thing that nobody asked for.

[jandubois]

I think I figured it out: the checksums should be an optional suffix on the template locator:

$ limactl create https://example.com/templates/our.yaml@deadbeef

This means we don't need extra parameters in the UI, and no extra fields in the implementation; the checksums ride for free in the locator string.

And while I still don't think anybody will use this, it automatically provides for this:

basedOn: template://fedora@feedc0de
provision:
- file: script.sh@decafbad

If a locator ending in a @digest is passed to limatmpl.Read() then it will return an error if the fetched content doesn't match the checksum.

Things to bike-shed:

• do we need to specify the checksum algorithm, even though we will only support sha256, e.g. script.sh@sha256:decafbad?
• do we allow abbreviated checksums, or do we require the full length?

My bike-shed is blue with pink polka dots:

• we don't include the redundant sha256: part
• we allow checksums to be abbreviated down to 7 hex characters, but no less

Having a compact format makes things easier to read/write without line wrapping, and 7 characters are plenty protection against accidental collision. People worrying about malicious changes can use the full checksum.

@AkihiroSuda WDYT?

[jandubois]

I've implemented this in c288fcb. It does allow you to optionally specify an algorithm, but otherwise matches what I wrote above.

[AkihiroSuda]

URL can't be extended in this way.

"@" has been a portion of the URL (Appears in Twitter, Mastodon, and others)

[afbjorklund]

It made sense for things like OCI images (@sha256:123abc...), but as mentioned I don't think it is "allowed" for URI?

Other systems do other hacks like making it into anchors (#) or parameters (&) or whatever. Still confuses some tools.

But I didn't think the digest was needed for the "includes" either.

• Allow providing scripts as separate files #2442

[jandubois]

Other systems do other hacks like making it into anchors (#) or parameters (&) or whatever. Still confuses some tools.

The discussion has mostly moved to #3021 now, but the concern about "some tools" is irrelevant as limactl is the only tool that understands template locators, especially template://.

The way to download a template from any kind of template locator is (well, will be) limactl tmpl copy, which automatically supports digest suffixes once limatmpl.Read() implements them:

$ limactl tmpl copy my.yaml@sha256:60a8737145 -
basedOn:
- fedora.yaml
…
$ limactl tmpl copy my.yaml@sha256:60a8737145ff -
FATA[0000] locator "my.yaml@sha256:60a8737145ff" digest doesn't match content digest "60a87371451e"

[afbjorklund]

Sounds like yet another gateway is needed :-)

[AkihiroSuda]

Git commit ids cannot be forged

It can be sort of "forged", in the GitHub URL:

$ curl https://raw.githubusercontent.com/lima-vm/lima/79529f879b566f02eacd705e1f8746ea7cb37a5b/DEMO-3019
Demo file for <https://github.com/lima-vm/lima/discussions/3019>.

This file will be visible via <https://raw.githubusercontent.com/lima-vm/lima/.../DEMO-3019>.

However, this file does not really belong to the <https://github.com/lima-vm/lima> repo.

So, {"location": "https://raw.githubusercontent.com/ORG/REPO/TAG/FILE": "digest": "DIGEST"} is more trustworthy than https://raw.githubusercontent.com/ORG/REPO/COMMIT_HASH/FILE

[jandubois]

It can be sort of "forged", in the GitHub URL:

This did not forge the commit id. You would have to create a commit with the SHA1 of 7cea6f8 to replace the content from my link. And given that a commit with that hash already exists in the repo, I'm not sure if you could actually push it, and even if you can, if GitHub would then prefer it over the existing one.

As for "does not really belong to", that is not exactly correct, as all the forks of a repo are really only branches in the same repo on the backend. So every commit exists in the original repo and all the forks simultaneously.

That is also the reason you can't make a fork of a public repo private: if a private commit id were to leak, the data would become accessible through the public main repo.

You can of course always create a new private repo, and then push the branches from the public repo to it. That way it is not a fork, but an independent copy (and you can't do pull requests between them).

[jandubois]

I realized that the issue is more complex than just calculating digests of the template files:

provision:
- file: script.sh

Even if I have a digest for this template, it doesn't tell me if the script.sh has changed since the digest was first calculated. So hypothetically the template could include a digest for script.sh, but it doesn't, and I can't modify it.

So an obvious improvement would be to calculate the digest not on the raw template, but once it has been assembled (all dependencies have been embedded).

But that digest is still a bit brittle: if Lima is updated, and either yq or yamlfmt has some subtle change in what it outputs, the digest would now change, even though the underlying files remain the same.

Another issue would be if the template in turn depends on one of our distro templates, e.g. template://ubuntu. Every time we update the images in that template the digest of the assembled user template changes as well.

And finally, if the template creation takes --set or --param options, the digest would only match if you run an identical command.

basedOn: "template://{{.Param.distro}}"
param:
  distro: ubuntu

Now I want to run

limactl create --param distro=fedora https://templates.r.us/dynamic.yaml --digest=???

It may end up that we'll have to say: "If you want to use template digests, you have to use single-file templates, or you have to store the assembled template somewhere you control, and reference from there".

So you could do something like

limactl tmpl copy --embed --param distro=fedora https://templates.r.us/dynamic.yaml my-fedora.yaml
sha256sum my-fedora.yaml

and then share the assembled template from your own web property. That way it will have a static digest.

Anyways, this turns into a rabbit hole, and I think will be much easier to discuss once we have played with the mechanism a bit, and written a couple of multi-file templates, and seen what works well, and what doesn't.

So let's revisit this in a couple of month...

[jandubois]

Since keeping digest of many small files updated is such a chore¹, I've been wondering about alternatives. Some way how you can sign both YAML and script files, and template assembly would then be able to verify that all pieces either have a matching digest, or are signed by a trusted party (like the Lima developers).

I think I can figure out the signing part of the YAML and shell scripts (unless you tell me that the YAML files need to comply with the YAML spec to the letter and we can't create our own directive because they are all reserved).

But the problem is the the signing tool, and the list of trusted parties. This sounds like GPG is the only option, and with that we have already lost 95% of the potential users. 😞

So, I currently don't see a way to make it work, but if you have some ideas...

Footnotes

1. Look at the pain we went through updating distro images. Every time we update, users will have to update digests for using these as base templates. ↩

[afbjorklund]

The signing part and bottleneck was the biggest problem with Zero Install, as long as upstream did not get involved / on board.

The actual signature was handled quite neatly by a data document though, and I thought I would bring that from XML to YAML?

https://docs.0install.net/details/#security-note

And yes, it did use embedded GPG signatures...

---

Everyone else started talking about cosign instead, but it was a pain to use - I thought?

https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-artifacts/

The end result is that no-one/nothing is checking signatures, and the signing is left in limbo.

kubernetes/release#913 - eventually "solved" by a third party, by using OBS