replace packages.txt with calculated graph

[deitch]

This replaces packages.txt entirely with wolfictl text to generate the graph.

This is a draft for a few reasons:

• I want several pairs of eyes on this, to make sure we get it right
• the graph generation currently runs about 4.5-5 mins; I want to profile it to see where it is taking so much time and fix it up. UPDATE: this is due ti a change in the sorting algorithm. I opened an issue on it, will follow up with it. See here - UPDATE: with a few small caching improvements, now down to 3.5 mins. Not enough, but getting there
• there remains an area in CI where we use packages.txt, but it is the last step in a job, so I don't understand how it is used, and thus how to replace it

To reproduce the work this will do:

1. Clone this repo
2. Make sure you have a recent version of wolfictl or use the container image for wolfictl or sdk
3. Run wolfictl text --dir . --type name and wait a few minutes for the output

[deitch]

the graph generation currently runs about 4.5-5 mins; I want to profile it to see where it is taking so much time and fix it up

I ran a profile, the overwhelming majority of the time is in the graph sort. Attached. I will remove the stable sort and see if that is the issue, either way, I will report it upwards.

[deitch]

I think wolfictl lint will fail until digestabot brings it up to date. Let it sit for a day

[deitch]

wolfictl lint still checks the Makefile. I could have removed that check from wolfictl lint, but then the current lint wouldn't work, and other places that use it would have had issues. Instead, I updated the call to --skip-rule no-makefile-entry-for-package

[deitch]

I am leaving packages.txt in this PR, even unused. Every time someone changes a package, I need to rebase it, which is becoming painful. Let's just leave it. Once this is in, I can remove it in a future PR.

[kaniini]

Build order is almost right, but not quite yet. build-base, which explicitly depends on glibc-dev, comes before glibc. But gcc is now coming before build-base, so we are much closer.

[deitch]

There was an issue. It was not allowing self-fulfilling dependencies, and checking in an incorrect way. There is an open PR that fixes that issue.

However, it opens several new issues with the actual wolfi-dev/os dependencies. Will post here.

[deitch]

time="2023-06-18T12:15:38+03:00" level=error msg="unresolvable cycle: pax-utils:1.3.7-r0@local -> meson:1.1.1-r0@local, caused by: meson:1.1.1-r0@local -> scanelf:1.3.7-r0@local -> pax-utils:1.3.7-r0@local"
time="2023-06-18T12:15:38+03:00" level=error msg="unresolvable cycle: pcre2:10.42-r2@local -> binutils:2.40-r3@local, caused by: binutils:2.40-r3@local -> wolfi-baselayout:20230201-r3@local -> glibc-locale-posix:2.37-r7@local -> glibc:2.37-r7@local -> grep:3.11-r0@local -> pcre2-dev:10.42-r2@local -> pcre2:10.42-r2@local"
time="2023-06-18T12:15:39+03:00" level=error msg="unresolvable cycle: rustls-ffi:0.10.0-r0@local -> git:2.41.0-r0@local, caused by: git:2.41.0-r0@local -> curl-dev:8.1.2-r0@local -> curl:8.1.2-r0@local -> rustls-ffi:0.10.0-r0@local"

There are 4 dependencies here, but they fall into 2 categories.

wolfi-baselayout

1. wolfi-baselayout depends on glibc-local-posix
2. which is a subpackage of glibc
3. glibc, in turn, depends runtime on wolfi-baselayout.

This is an unresolvable loop.

pax-utils

pax-utils is due to pipelines

1. pax-utils depends in its pipelines on meson
2. which in its pipelines uses strip
3. whose needs include scanelf
4. which in turn is a subpackage of pax-utils

And we have a cycle.

pcre2

1. pcre2:10.42-r2@local depends on strip
2. which depends on binutils
3. which depends on wolfi-baselayout
4. which depends on glibc-locale-posix
5. which is a subpackage of glibc
6. which depends on grep
7. which depends on pcre2-dev
8. which is provided by pcre

rustls-ffi

1. rustls-ffi:0.10.0-r0@local depends on the pipeline git-checkout
2. which depends on git
3. which depends on curl-dev
4. which is a subpackage of curl
5. which depends on rustls-ffi

And we have a cycle

[kaniini]

I think we can drop the glibc dependency on wolfi-baselayout.

Meson, being python, can just drop strip.

Not sure what to do about rustls-ffi.

[deitch]

I can open PRs for the first two. As you said, not sure what to do about the rust one.

What about pcre?

[deitch]

I think we can drop the glibc dependency on wolfi-baselayout.

See #2937

Meson, being python, can just drop strip.

See #2938

It only partially solves the job. It frees up one loop, but reveals another.

1. pax-utils depends buildtime on meson
2. which depends buildtime on busybox
3. which depends on strip
4. which depends on scanelf
5. which is a subpackage of pax-utils

[deitch]

Do not merge this in. It is good to approve, I am going to wait until I see that the latest version of wolfi-dev/sdk is released on the nightly cycle and then I will merge it in myself.

[jdolitsky]

Do not merge this in. It is good to approve, I am going to wait until I see that the latest version of wolfi-dev/sdk is released on the nightly cycle and then I will merge it in myself.

FWIW we can manually trigger this build this whenever needed

[deitch]

Yeah, but I missed something on the Makefile anyways. Working that now

[dlorenc]

Didn't the symlinked directories break arm builds last time? Did that get fixed?

[deitch]

Didn't the symlinked directories break arm builds last time? Did that get fixed?

I am pretty sure they did, but I cannot remember what it was that fixed it. As soon as I have the Makefile demangled, I will track it down.

[deitch]

Some updates here:

1. I fixed the Makefile issue. Actually, at @kaniini 's prodding, I removed the complex .packagerules generation, and replaced it with some simple targets. However, to do so, I had to change make packages/openssl with make package/openssl or makefile got confused. I tracked down where we use this in GHA workflows, but another pair of eyes (or two) on this would be helpful.
2. That makefile bit requires an extra call, which slows it down a bit. I don't think it is slower than currently (likely similar), but another pair of eyes, again, would be helpful.
3. The issue with symlinks was kontext, the revert and comment are here.

Those need review, then we wait for wolfictl to be updated in wolfi/sdk by tomorrow.

[kaniini]

Symlinks should be good in kontext now

[deitch]

1. Updated to ghcr.io/wolfi-dev/sdk:latest@sha256:fd8c71214f6455c75ec44ae99eb9f7ffc85f260ccce69d4367eb2e5d568facd9. Checked the version from there to be:

GitVersion:    6a0feae
GitCommit:     6a0feae8fcfd8771f03575bdcc8158a5cb7d01a6

which is what is in latest build on this branch, so it is correct.
2. @kaniini confirmed above that kontext is fine with symlinks
3. Just need agreement on the Makefile changes and this can go in.

I will remove the unused packages.txt in a later PR.

[joshrwolf]

this shouldn't block, but I get these errors when trying to reproduced this:

➜ wolfictl text --dir . --type name
Error: unable to build graph:
envoy-1.26.2-r0: unable to resolve dependency gcc=12.2.0-r11: could not find package gcc=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libgcc=12.2.0-r11: could not find package libgcc=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libstdc++-dev=12.2.0-r11: could not find package libstdc++-dev=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libstdc++=12.2.0-r11: could not find package libstdc++=12.2.0-r11 in indexes
FATA[0000] error during command execution: unable to build graph:
envoy-1.26.2-r0: unable to resolve dependency gcc=12.2.0-r11: could not find package gcc=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libgcc=12.2.0-r11: could not find package libgcc=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libstdc++-dev=12.2.0-r11: could not find package libstdc++-dev=12.2.0-r11 in indexes
envoy-1.26.2-r0: unable to resolve dependency libstdc++=12.2.0-r11: could not find package libstdc++=12.2.0-r11 in indexes

digging in, I updated envoy to use:

# OLD
      - gcc=12.2.0-r11
      - libstdc++=12.2.0-r11
      - libstdc++-dev=12.2.0-r11
      - libgcc=12.2.0-r11
# NEW
      - gcc-12
      - libstdc++-12
      - libstdc++-12-dev
      - libgcc # This should be `libgcc-12`

and that got past that error, but flagged a new one:

❯ wolfictl text --dir . --type name
Error: unable to build graph:
rabbitmq-server-3.12.0-r1: unable to resolve dependency elixir<1.15: could not find package elixir<1.15 in indexes
FATA[0000] error during command execution: unable to build graph:
rabbitmq-server-3.12.0-r1: unable to resolve dependency elixir<1.15: could not find package elixir<1.15 in indexes

which I only got to work by setting to the latest elixir, which in theory breaks the rabbitmq-server package.

am I doing something wrong? or is wolfictl not parsing these edge cases correctly?

[deitch]

This appears to be correct. gcc-12.yaml has version 12.3.0-r0:

package:
  name: gcc-12
  version: 12.3.0
  epoch: 0

But envoy.yaml requires exactly 12.2.0-r11:

      - gcc=12.2.0-r11
      - libstdc++=12.2.0-r11
      - libstdc++-dev=12.2.0-r11
      - libgcc=12.2.0-r11

There is a similar issue with rabbitmq depending on elixir<1.15, when elixir.yaml is 1.15.2-r0.

[deitch]

You could make the argument that, "we had those packages at one time, and we loaded them up into our registry, so maybe they already are there." That is a good argument, but it loosens the constraints. The graph checks if we can build it with the most constrained situation possible, which means without the cache at all (from scratch). These changes prevent it.

[kaniini]

These changes prevent it.

Yes, we should not allow build-depending on something that happens to be in the cache. If we want to pin to an older dependency, then we should do the work to properly set that dependency up to be slotted.