chore: codspeed benchmark

[ematipico]

Changes

This PR creates a new workflow to use continuous benchmarking using CodSpeed

Everything was set up months ago, but I couldn't follow up.

I resurrected the code, tested it locally, and now opened it.

I created a new workflow because I don't want to disrupt too much the existing workflow we use for inline benchmarking via magic command.
Once we are in a place where CodSpeed is usable, I believe we can start using that one and ditch the ones that we use via command (only the workflow, of course).

Testing

I tested the scripts locally, and they work.

CI should pass. Then, upon merging the CI should trigger benchmarking and upload the results in the dashboard. The next, rebase PR, should show a bot message.

Docs

N/A

[changeset-bot]

⚠️ No Changeset found

Latest commit: bd7579a

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[bluwy]

I'd prefer if we can avoid duplicating the code for the benchmarks, but happy to try something out first.

Also a bit concerned with this running on each push to the pull request, maybe if can run only if there's a label. But also not a blocker

[ematipico]

The variable wasn't used

[ematipico]

Also a bit concerned with this running on each push to the pull request, maybe if can run only if there's a label. But also not a blocker

That would kill the purpose of this tool. Maybe we could evaluate a reduced workload for these benchmarks

[ematipico]

This is weird, it works locally, but it fails in CI: https://github.com/withastro/astro/actions/runs/11683003640/job/32531309684?pr=12347

[bluwy]

I tested and it also works locally. I can't tell what's going on here, there's no output or errors from the preview command at all in CI. It's either the command not running at all, or it's hanging.

I'm also reading the tinybench code and it seems to be running the benchmark repeatedly instead of once, so locally it took very long to finish. I assume this is only temporary, but could a better way be using our own benchmark code and then reporting the results to codspeed ourselves manually?

I wonder if this CI issue we're seeing is related to tinybench specifically since we know it should be working when we run !bench.

[ematipico]

I'm also reading the tinybench code and it seems to be running the benchmark repeatedly instead of once, so locally it took very long to finish. I assume this is only temporary, but could a better way be using our own benchmark code and then reporting the results to codspeed ourselves manually?

This is actually expected by a benching library. Other libraries, in other ecosystems too, behave the same way. They run multiple times with the same function/task so they can create a sample out of it.

I'll try to use vitest at this point.

[bluwy]

I guess that makes sense for our benchmarks like cli-startup and memory, but for server-stress and render, we already do the repeated runs ourselves so the result reported there should be stable to be used. Using tinybench for them is like running a benchmark system within another benchmark system 😄

What we also want is to measure are things like "average render time for MDX pages" or "requests/sec for Astro pages". If we wrap with tinybench, we're instead measuring "how long it takes to benchmark the 'average render time for MDX pages'" (for example).

I'm ok with testing cli-startup or memory to setup the initial codspeed workflows, since those don't have repeated runs. And then maybe later we can circle back on server-stress and render and refactor in a way that we can reuse the reported results, or rely on tinybench for the repeated runs? (And we can figure out the preview issue later).

[ematipico]

I changed the approach now and removed the need for a preview server to test the rendering. I created an adapter that is very similar to our test adapter, we create a Request and call the app.render.

This makes sense for rendering because it removes the noise of a Node.js HTTP server. With this approach, we use the minimal NodeApp and call .render straight away.

[codspeed-hq]

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 1 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• Rendering (4.7 s)

[ematipico]

I am going to merge this for now. We will have a discussion internally about what to benchmark and how, so things might change