CI speed optimization

[matthiasbeyer]

This PR experiments with CI speedup optimizations.

I didn't realize that CI in this project did not even use caching... 👀

---

Closes #482
... probably? 👀

[matthiasbeyer]

Some more changes, lets see how fast we're now. 😆

[matthiasbeyer]

Ok, I got 30 secs faster CI now... that's not nearly enough IMO. Could even be just random noise.

[matthiasbeyer]

I wonder why the tests for 1.70.0 is so much slower than the ones for 1.73.0 ... nextest did not actually make anything faster. Probably because building is slow, not running. But why is the job for 1.70.0 running 2:20min, and the 1.73.0 one only 58s? I don't know.

So overall we're now slower than before. 😢

Let's see what happens if we execute tests right away and not wait for the check phase.

[matthiasbeyer]

4 min -> 3 min now.

Not sure whether this is actually worth it... but I guess I am ready for review here. Just a quick squash of the fixup commit.

[polarathene]

I wonder why the tests for 1.70.0 is so much slower than the ones for 1.73.0

1.73.0 being the latest stable right? I already mentioned that Github likely caches this themselves as they're known to do with other high traffic sources for CI. Far cheaper for them to pull from the real location once every N (eg: minute), than every CI job running within N duration on their infrastructure.

Older toolchains are less likely to have that optimization? 🤷‍♂️ (just an assumption, that there is something different in the requests that differentiates this)

I believe it's a common practice, even for ISPs to cache popular content from vendors like Youtube / Netflix (pretty sure I had read of partnerships to do such).

---

nextest did not actually make anything faster.

I also mentioned that observation, tests did not seem to take much of the time so I didn't see it providing much benefit.

So overall we're now slower than before

Keep in mind that when using actions/cache action, it will have access to main branch cache (once CI runs on that), and then PRs can use that. However PRs maintain their own scoped cache IIRC, and that is not accessible to another PR, nor main I think 🤷‍♂️

Should be in the docs, I could be recalling that wrong. Caching makes a big difference for docker-mailserver builds.

[polarathene]

As mentioned:

Additionally keep in mind what is being cached, and the cache key generation:

This also confirms that they're using the official github cache action, just extending it to better suite cargo:

---

We can see that the action is correctly saving and restoring cache quickly, but pay attention to this:

Your current PR is using the action, then setting a toolchain, which is against the actions advice.

[polarathene]

Just for clarification, you define multiple separate jobs to run, these are all via separate runners just like matrix AFAIK?

Each separate runner then is building the project again. You could instead try minimize the jobs?

---

Here is an example of managing a check status, which I think can be used to display multiple from the same job if that suits you:

• Create new pending state
• Update with success/failure status

This makes it visible:

Bit more verbose workflow, but could help.

---

Alternatively, as suggested before, have one job start per toolchain, then subsequent jobs express a dependency to the initial job and they could bring in that cache.

If you have multiple jobs run in parallel, then initially none have that cache and all need to do the same work.

Once you have got some cache, depending on how the cache key is defined, you can find jobs falling back to earlier cache they can use.

---

As per prior comment with images from the rust-cache docs, you may need to wait until the Cargo.lock is committed for the most benefit in cache usage.