Use dot to separate distro major and minor version and replace distro registry with factory #3887
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bcl
reviewed
Jan 10, 2024
bcl
reviewed
Jan 10, 2024
thozza
force-pushed
the
dot-notation
branch
19 times, most recently
from
59a6034
to
0bb8d0c
Compare
|
The |
Not by my changes, but by the latest version of |
|
@thozza Thanks for the info. I'll push the fix into this PR. |
@henrywang thanks, but I think that it would be better if you could just point me to the right place to make the change. This PR is still in heavy development and I'm regularly force pushing it. And I don't want to drop your changes by accident... 😇 |
|
@thozza Sure. The change is simple. diff --git i/test/cases/ostree-ng.sh w/test/cases/ostree-ng.sh
index 4a87bc137..27fc3d436 100755
--- i/test/cases/ostree-ng.sh
+++ w/test/cases/ostree-ng.sh
@@ -125,7 +125,7 @@ case "${ID}-${VERSION_ID}" in
CONTAINER_TYPE=iot-container
INSTALLER_TYPE=iot-installer
OSTREE_REF="fedora/${VERSION_ID}/${ARCH}/iot"
- OSTREE_OSNAME=fedora
+ OSTREE_OSNAME=fedora-iot
OS_VARIANT="fedora-unknown"
EMBEDED_CONTAINER="false"
DIRS_FILES_CUSTOMIZATION="true" |
Update the osbuild/images to the version which introduces "dot notation" for distro release versions. - Replace all uses of distroregistry by distrofactory. - Delete local version of reporegistry and use the one from the osbuild/images. - Weldr: unify `createWeldrAPI()` and `createWeldrAPI2()` into a single `createTestWeldrAPI()` function`. - store/fixture: rework fixtures to allow overriding the host distro name and host architecture name. A cleanup function to restore the host distro and arch names is always part of the fixture struct. - Delete `distro_mock` package, since it is no longer used. - Bump the required version of osbuild to 98, because the OSCAP customization is using the 'compress_results' stage option, which is not available in older versions of osbuild. Signed-off-by: Tomáš Hozza <[email protected]>
Previously, the old dnfjson cache dirs for unsupported distributions were deleted in the osbuild-composer binary on startup. This is no longer possible, since the supported distros are determined by loading available repositories. Loading repositories happens in the Weldr API constructor. Move the cleanup code there. Signed-off-by: Tomáš Hozza <[email protected]>
Drop `common.GetHostDistroName()` implementation and use `distro.GetHostDistroName()` from the osbuild/images instead. Signed-off-by: Tomáš Hozza <[email protected]>
Drop `common.CurrentArch()` implementation and use `arch.Current().String()` from the osbuild/images instead. Signed-off-by: Tomáš Hozza <[email protected]>
Add new configuration option `distro_aliases`, which is a map of strings, allowing to specify distro name alias for supported distributions. Define aliases for RHEL major versions without the minor version specified. For now, the distro aliases map is not used by any API implementation and it is ignored. Signed-off-by: Tomáš Hozza <[email protected]>
Weldr API used to determine the distro name from the image type, when it was getting the repositories to use for depsolving and for the actual depsolving (solver uses the distro name to namespace cache). This used to be OK, but with the introduction of distro name aliases, the distro name used to get the distro object may not be the same as the name returned by the actual distro object. To preserve the current behavior, the same name used to get the distro object should be used to also get the repositories for depsolving and to namespace depsolving cache. Signed-off-by: Tomáš Hozza <[email protected]>
Register the distro name aliases from the configuration on the distro factory. Signed-off-by: Tomáš Hozza <[email protected]>
Any changes done to the SPEC file were not reflected in the `rpmbuild/SPECS` directory across runs of `make rpm`. One had to delete the SPEC file manually to be updated and used for the RPM build. Mark the target as PHONY to ensure that the SPEC is always updated. Signed-off-by: Tomáš Hozza <[email protected]>
While I was the one to push for TEST_MODULE_HOTFIXES to be passed as an argument to `api.sh`, when it was implemented, this is turning our to be impractical. I would like to extend `api.sh` to test distro names with and without the dot to separate major and minor version and also to test the ability to specify distro name explicitly (e.g. a distro alias). Passing all of these new options and their combinations would make the CLI go wild. Signed-off-by: Tomáš Hozza <[email protected]>
Where applicable, modify all repo config filenames to use a dot to separate the release major and minor version. Modify test cases to not remove dot from the distro version any more. Existing tests will be extended (or new tests added) to explicitly test backward compatibility and ensure that using old distro names without a dot still works. Signed-off-by: Tomáš Hozza <[email protected]>
Otherwise, the image tests will fail on globing for testing manifests. Signed-off-by: Tomáš Hozza <[email protected]>
Using the DISTRO_CODE with dot to separate major and minor version is breaking cloud-image-val testing. Specifically, the tool is using regular expression to search for an uploaded image, but the expression is not expecting any dot in the version. This will be fixed in [1], but merging it will take some time due to CIV CI being currently broken. Due to this, workaround the problem for now by making sure that the TEST_ID, which is used to construct the IMAGE_NAME, does not contain any dot in the DISTRO_CODE. [1] osbuild/cloud-image-val#290 Signed-off-by: Tomáš Hozza <[email protected]>
Update RHEL 8.9 and 9.3 repo definitions to use the CDN repos, since these are already GA. Add SAP repositories to all RHEL repo configs, to be able to build the SAP image for testing purposes. Fix minor issues found in repos (e.g. 8.8 RT repo pointing to 8.7, etc.). Signed-off-by: Tomáš Hozza <[email protected]>
Extend the `api.sh` to allow testing compose requests with distro name which does not use dot-notation (specifically when the dot is removed from the distro name as it used to be in the past). In addition to that, allow also testing the distro alias using distro name without the minor version in compose requests. Enable these two new test variants in the CI. Signed-off-by: Tomáš Hozza <[email protected]>
Add test case for verifying that distros and repo configurations work with and without the dot-notation in the on-prem scenario. Run the test case on the latest RHEL 8 and 9 versions in development and verify the default distro aliases behavior on them. Signed-off-by: Tomáš Hozza <[email protected]>
The test would loop for the job timeout limit waiting for composer container, in case the composer API does not come up for whatever reason. Modify the test case to wait only for 12x 10s before failing. In case the composer API does not come up, print the logs from the composer container and exit with non-zero code. Signed-off-by: Tomáš Hozza <[email protected]>
If a depsolve job fails, the error details were not passed as details to the manifest job error details. This may help with debugging failures. Signed-off-by: Tomáš Hozza <[email protected]>
Add any errors to job error details when an osbuild build fails. Otherwise these won't show up in the worker log, which makes debugging issues harder. Signed-off-by: Tomáš Hozza <[email protected]>
Specify the resource pool when importing the VMDK to the VCenter. This should prevent the following error: govc: default resource pool resolves to multiple instances, please specify Signed-off-by: Tomáš Hozza <[email protected]>
This is necessary to get the proper osbuild SHA to install the latest required osbuild version into the image. Signed-off-by: Tomáš Hozza <[email protected]>
achilleas-k
approved these changes
Jan 26, 2024
thozza
added a commit
to thozza/cloud-image-val
that referenced
this pull request
Feb 7, 2024
An osbuild-composer PR#3887 [1] introduces "dot-notation" for distro versions, meaning that a dot will be used to separate major and minor version in distro names. For backward compatibility, distro names without a dot will be still accepted for distros which used to accept it. The CIV code parsing a VHD name was not expecting the version to contain any dot. This results in failures such as [2]. Modify the regular expression to gracefully handle potential dot in the version string. [1] osbuild/osbuild-composer#3887 [2] https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/5943494707#L3695 Signed-off-by: Tomáš Hozza <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to: https://issues.redhat.com/browse/COMPOSER-1962
Main changes:
cmd/osbuild-playgroundcmd/osbuild-package-setscmd/osbuild-pipelinecmd/osbuild-composer-image-definitionsgen-manifestsis the tool to use and that will be removed from this repo eventually as well.osbuild/imagesv0.33.0to osbuild-composer. This is unfortunately one giant commit, because there is no other good way to ensure that composer compiles and unit tests pass after the update of theosbuild/imagespackage. It includes:createWeldrAPI()andcreateWeldrAPI2()into a singlecreateTestWeldrAPI()function`.distro_mockpackage, since it is no longer used.osbuild/images. Specifically, remove:common.GetHostDistroName()common.CurrentArch()rhel-8actually translates torhel-8.10, etc. The default configuration has these aliases defined for the next RHEL GA versions (the on-prem scenario). The service can override these to point to the current GA in its deployment.api.shtest case to enable testing of dot-notation, backward compatibility (not using dots) and distro aliases. The distro alias test always uses SAP image and inspects/etc/dnf/vars/releaseverin it. This image type is the only which contains is and the content of the file is always the release version of the distro definition used to generate the image manifest (in the form of "X.Y"). Otherwise, there would not be any other way to verify which distro definition was used to generate the manifest used to build the image. The content of/etc/os-releasealways comes from the provided repositories and it is not anyhow related to the actual distro definition release version used to generate the manifest.osbuild build failed. This is not the case any more and we get a lot more details about the actual problem.This pull request includes: