test: allow testing frontends from different version #5594
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
area/dockerfile
area/hack
There was a problem hiding this comment.
Copilot reviewed 1 out of 6 changed files in this pull request and generated no comments.
Files not reviewed (5)
- docker-bake.hcl: Language not supported
- frontend/dockerfile/cmd/dockerfile-frontend/hack/release: Language not supported
- hack/images: Language not supported
- hack/test: Language not supported
- hack/util: Language not supported
tonistiigi
force-pushed
the
dockerfile-test-versioning
branch
from
ee0c989 to
1f94cab
Compare
tonistiigi
commented
Dec 14, 2024
| if [ -n "$TEST_SUITE_CONTEXT" ]; then | ||
| export TEST_BINARIES_CONTEXT=$currentcontext | ||
| # FIXME: something breaks with the syntax when using the context | ||
| export BUILDKIT_SYNTAX="docker/dockerfile:1.10.0" |
There was a problem hiding this comment.
this is fixed by #5596
This allows running Dockerfile tests so that the Dockerfile version and the BuildKit version are from different commits so that we can test that old Dockerfile releases remain compatible with the latest BuildKit. The tests are based on the commit of the Dockerfile frontend as we can't expect that new test would work on old frontends. In future we might consider doing it the other way as well but then we need a way to mark tests that can be ignored if they are not expected to pass because of a new feature dependency. Signed-off-by: Tonis Tiigi <[email protected]>
tonistiigi
force-pushed
the
dockerfile-test-versioning
branch
from
1f94cab to
69034c3
Compare
I will look at this, we might need something similar to |
I think we should first get the version where we test the specific version of buildkit against release versions of Dockerfile running continuously. Then we can look at what the test results look like for different combinations. It isn't as easy as |
|
Initial summary for test matrix at https://github.com/moby/buildkit/actions/runs/12385930256 (against current master)
I think the failures are worth debugging but probably not extremely critical. We do need some solution for exceptions though and can't expect everything to be green. Maybe instead of managing patches it just makes sense to create extra branches for old releases where the test fixes can be applied and test these branches instead. Eg. there we can pin None of this should be a blocker to getting this PR in. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This allows running Dockerfile tests so that the Dockerfile version and the BuildKit version are from different commits so that we can test that old Dockerfile releases remain compatible with the latest BuildKit.
The tests are based on the commit of the Dockerfile frontend as we can't expect that new test would work on old frontends. In future we might consider doing it the other way as well but then we need a way to mark tests that can be ignored if they are not expected to pass because of a new feature dependency.
Example:
This doesn't update Github workflows to trigger tests like this yet. We wouldn't do this for PRs but in follow up we should set up a matrix that can be run manually or via cron. cc @crazy-max . Atm. I have not run through the full test suites yet for previous release versions, so not sure if we will find some issues.