Build containers are not representative for the release processs

[mpkorstanje]

👓 What did you see?

This release failed because the combination of ubuntu-latest, the otp verison and elixir version stopped working. We didn't see this earlier because the Build containers use a different process.

https://github.com/cucumber/messages/actions/runs/3719910333/jobs/6309058962

[ciaranmcnulty]

Hm I've not looked at the release side and don't feel I understand it that well. How can we combine it with the build steps?

[mpkorstanje]

It depends on the language used. For some it's just a matter of tagging the repo, for some it's the build step + an additional the release, for some it is a completely different and independent step.

In this particular case I don't know the specifics. However it appears that the same toolset is used by both test and release. However because they're installed with different versions in different places the tests aren't representative anymore.

In this case GitHub upgraded Ubuntu, but it could any dependency.

[ciaranmcnulty]

I wonder whether the tests need to run as part of release, or whether we can chain the existing test workflow 'before' the release workflow?

[mpkorstanje]

Ideally not. The closer we are to the "native" process the better. Each bit of customization makes the system as a whole more complicated.

I think thats currently also my biggest concern about the build kit. We appear to be on the path of essentially rebuilding the complexity of the Makefile.

[mpkorstanje]

Looking back I think this got merged prematurely, I would be okay with reverting until we can put more time into this and see where exactly we will end.