Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do OS package upgrades before doing migration #49

Merged

Conversation

ipetrov117
Copy link
Contributor

As per the official SLE Micro upgrade docs the users need to ensure that packages are updated before doing a version migration.

Sadly doing just transactional-update patch over an older image is not enough for a successful migration.

In order to not force users to have to do an additional manual command, this PR introduces an enhancement over the existing OS migration logic, where before migrating we do an automatic update of all OS packages, ensuring that the operation will succeed.

The PR introduces the following code changes:

  1. The indentations of the os-upgrade.sh template have been changed in order to make the heredoc more readable
  2. Logic has been introduced to the os-upgrade.sh script that automatically adds ExecStartPre= configurations to the systemd.service when OS migration is done (both OS package upgrade and RPM import are done in ExecStartPre= statements).
  3. A fix/workaround to a race condition that was found during the testing of this implementation.

Upgrade screenshots:
Before:
Screenshot 2024-08-16 at 18 27 25

After:
Screenshot 2024-08-16 at 18 54 32

Compressed logs of the OS upgrade pod (different logical sections have been marked for easier reading):

Defaulted container "upgrade" out of: upgrade, cordon (init)
Creating os-migration.service...
Starting os-migration.service...
---------------------------------------
<"transactional_update cleanup up" logic>
---------------------------------------
2024-08-16 13:26:14 Options: callext 4 zypper -R {} up -y --auto-agree-with-product-licenses 
2024-08-16 13:26:15 Executing `zypper -R /tmp/transactional-update-eVur6U up -y --auto-agree-with-product-licenses`:
Refreshing service 'SUSE_Linux_Enterprise_Micro_5.5_x86_64'.
Loading repository data...
Reading installed packages...

The following 301 packages are going to be upgraded:
  NetworkManager SLE-Micro-release aaa_base audit biosdevname btrfsprogs btrfsprogs-udev-rules ca-certificates ca-certificates-mozilla catatonit chrony chrony-pool-suse cni cni-plugins cockpit cockpit-bridge cockpit-networkmanager cockpit-selinux cockpit-storaged cockpit-system cockpit-ws conmon container-selinux coreutils cpio crypto-policies curl device-mapper dmidecode dracut dracut-fips dracut-kiwi-lib dracut-kiwi-oem-dump dracut-kiwi-oem-repart dracut-transactional-update e2fsprogs efibootmgr fuse-overlayfs glib2-tools glibc glibc-locale-base gnutls gpg2 grub2 grub2-i386-pc grub2-snapper-plugin grub2-x86_64-efi ignition ignition-dracut-grub2 ipset iputils kdump kernel-default kernel-firmware-all kernel-firmware-amdgpu kernel-firmware-ath10k kernel-firmware-ath11k kernel-firmware-atheros kernel-firmware-bluetooth kernel-firmware-bnx2 kernel-firmware-brcm kernel-firmware-chelsio kernel-firmware-dpaa2 kernel-firmware-i915 kernel-firmware-intel kernel-firmware-iwlwifi kernel-firmware-liquidio kernel-firmware-marvell kernel-firmware-media kernel-firmware-mediatek kernel-firmware-mellanox kernel-firmware-mwifiex kernel-firmware-network kernel-firmware-nfp kernel-firmware-nvidia kernel-firmware-platform kernel-firmware-prestera kernel-firmware-qcom kernel-firmware-qlogic kernel-firmware-radeon kernel-firmware-realtek kernel-firmware-serial kernel-firmware-sound kernel-firmware-ti kernel-firmware-ueagle kernel-firmware-usb-network kpartx krb5 less libapparmor1 libassuan0 libaudit1 libauparse0 libblkid1 libbpf1 libbtrfs0 libcom_err2 libcontainers-common libcontainers-sles-mounts libcrypt1 libcurl4 libdevmapper-event1_03 libdevmapper1_03 libduktape206 libeconf0 libexpat1 libext2fs2 libfdisk1 libfreebl3 libgcc_s1 libgio-2_0-0 libglib-2_0-0 libgmodule-2_0-0 libgnutls30 libgnutls30-hmac libgobject-2_0-0 libguestfs0 libhidapi-hidraw0 libipset13 libjansson4 libjitterentropy3 liblldp_clif1 liblvm2cmd2_03 libmount1 libmpath0 libncurses6 libndp0 libnftables1 libnghttp2-14 libnm0 libopeniscsiusr0 libopenssl1_1 libopenssl1_1-hmac libp11-kit0 libpci3 libpolkit-agent-1-0 libpolkit-gobject-1-0 libpython3_6m1_0 librados2 librbd1 libsmartcols1 libsoftokn3 libsolv-tools libsqlite3-0 libssh-config libssh2-1 libssh4 libstdc++6 libsystemd0 libtirpc-netconfig libtirpc3 libtss2-esys0 libtss2-fapi1 libtss2-mu0 libtss2-rc0 libtss2-sys1 libtss2-tcti-device0 libtss2-tctildr0 libtukit4 libudev1 libuuid1 libvirt-client libvirt-daemon libvirt-daemon-driver-interface libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-iscsi libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi libvirt-daemon-qemu libvirt-libs libxkbcommon0 libxml2-2 libxml2-tools libz1 libzck1 libzypp login_defs lvm2 mozilla-nss mozilla-nss-certs ncurses-utils netcfg nfs-client nfs-kernel-server nftables open-iscsi open-lldp openslp openssh openssh-clients openssh-common openssh-fips openssh-server openssl-1_1 p11-kit p11-kit-tools pam pam-config perl-Bootloader perl-base podman polkit procps psmisc python3 python3-Jinja2 python3-M2Crypto python3-PyYAML python3-audit python3-base python3-chardet python3-cryptography python3-idna python3-jmespath python3-libxml2 python3-nftables python3-ply python3-psutil python3-requests python3-rpm python3-salt python3-setuptools python3-simplejson python3-urllib3 qemu qemu-accel-tcg-x86 qemu-ipxe qemu-seabios qemu-sgabios qemu-tools qemu-vgabios qemu-x86 rpm runc salt salt-minion salt-transactional-update sed selinux-policy selinux-policy-targeted shadow shim sudo supportutils suse-build-key suse-module-tools suseconnect-ng sysconfig sysconfig-netconfig system-group-audit system-group-hardware system-group-kvm system-group-libvirt system-group-wheel system-user-nobody system-user-qemu system-user-tftp system-user-tss systemd systemd-container systemd-default-settings systemd-default-settings-branding-SLE-Micro systemd-presets-common-SUSE systemd-rpm-macros systemd-sysvinit tar terminfo terminfo-base tftp timezone tpm2-0-tss tpm2.0-tools transactional-update transactional-update-zypp-config trousers tukit tukitd typelib-1_0-NM-1_0 ucode-amd ucode-intel udev util-linux util-linux-systemd vim-data-common vim-small wpa_supplicant xen-libs xfsprogs yast2-logs zypper zypper-needs-restarting

...
<package_update_installation_code>
...

Reading installed packages...

Preparing to purge obsolete kernels...
Configuration: latest,latest-1,running
Running kernel release: 5.14.21-150500.55.19-default
Running kernel arch: x86_64

Resolving package dependencies...
Nothing to do.
2024-08-16 13:31:29 Application returned with exit status 0.
2024-08-16 13:31:29 Transaction completed.
2024-08-16 13:31 Trying to rebuild kdump initrd
2024-08-16 13:31:30 tukit 4.1.8 started
2024-08-16 13:31:30 Options: close 4 
2024-08-16 13:31:31 New default snapshot is #4 (/.snapshots/4/snapshot).
2024-08-16 13:31:31 Transaction completed.
---------------------------------------
</"transactional_update cleanup up" logic>
---------------------------------------
---------------------------------------
<"transactional_update --continue run rpm --import" logic>
---------------------------------------
2024-08-16 13:31 New default snapshot is #4 (/.snapshots/4/snapshot).
2024-08-16 13:31 transactional-update finished
2024-08-16 13:31 Checking for newer version.
2024-08-16 13:31 New version found - updating...
2024-08-16 13:31 transactional-update 4.1.8 started
...
2024-08-16 13:31:36 Options: call 5 rpm --import /usr/lib/rpm/gnupg/keys/gpg-pubkey-09d9ea69-645b99ce.asc 
2024-08-16 13:31:36 Executing `rpm --import /usr/lib/rpm/gnupg/keys/gpg-pubkey-09d9ea69-645b99ce.asc`:
2024-08-16 13:31:36 Application returned with exit status 0.
2024-08-16 13:31:36 Transaction completed.
2024-08-16 13:31:36 tukit 4.1.8 started
2024-08-16 13:31:36 Options: close 5 
2024-08-16 13:31:37 New default snapshot is #5 (/.snapshots/5/snapshot).
2024-08-16 13:31:37 Transaction completed.
2024-08-16 13:31 
2024-08-16 13:31 Please reboot your machine to activate the changes and avoid data loss.
2024-08-16 13:31 New default snapshot is #5 (/.snapshots/5/snapshot).
2024-08-16 13:31 transactional-update finished
---------------------------------------
</"transactional_update --continue run rpm --import" logic>
---------------------------------------
---------------------------------------
<"transactional_update --continue run zypper migration --non-interactive --product SL-Micro/6.0/x86_64 --root" logic>
---------------------------------------
2024-08-16 13:31:40 tukit 4.1.8 started
...
2024-08-16 13:31:41 Options: call 6 zypper migration --non-interactive --product SL-Micro/6.0/x86_64 --root / 
2024-08-16 13:31:42 Executing `zypper migration --non-interactive --product SL-Micro/6.0/x86_64 --root /`:

Executing '/usr/bin/zypper --non-interactive patch-check --updatestack-only'

Refreshing service 'SUSE_Linux_Enterprise_Micro_5.5_x86_64'.
Loading repository data...
Reading installed packages...

0 patches needed (0 security patches)


Executing '/usr/bin/zypper --non-interactive ref'

Repository 'SLE-Micro-5.5-Pool' is up to date.
Repository 'SLE-Micro-5.5-Updates' is up to date.
All repositories have been refreshed.

Executing '/usr/bin/zypper --disable-repositories --xmlout --non-interactive products -i'


Executing '/usr/bin/zypper --xmlout --non-interactive services -d'

Upgrading product SUSE Linux Micro 6.0 x86_64

Executing '/usr/bin/zypper --non-interactive removeservice SUSE_Linux_Enterprise_Micro_5.5_x86_64'


Executing '/usr/bin/zypper --xmlout --no-refresh --non-interactive search -s --match-exact -t product SL-Micro'


Executing '/usr/bin/zypper --xmlout --non-interactive repos -d'


Executing '/usr/bin/zypper --non-interactive removeservice SUSE_Linux_Micro_6.0_x86_64'


Executing '/usr/bin/zypper --non-interactive addservice -t ris https://scc.suse.com/access/services/2573?credentials=SUSE_Linux_Micro_6.0_x86_64 SUSE_Linux_Micro_6.0_x86_64'


Executing '/usr/bin/zypper --non-interactive modifyservice -r SUSE_Linux_Micro_6.0_x86_64'


Executing '/usr/bin/zypper --non-interactive refs SUSE_Linux_Micro_6.0_x86_64'


Executing '/usr/bin/zypper --releasever 6.0 ref -f'

Warning: Enforced setting: $releasever=6.0
Forcing raw metadata refresh
Retrieving repository 'SL-Micro-6.0-Pool' metadata [......................done]
Forcing building of repository cache
Building repository 'SL-Micro-6.0-Pool' cache [....done]
All repositories have been refreshed.

Executing '/usr/bin/zypper --non-interactive --releasever 6.0 --no-refresh dist-upgrade --no-allow-vendor-change'

Warning: Enforced setting: $releasever=6.0
Loading repository data...
Reading installed packages...
Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...

The following 407 packages are going to be upgraded:
  ModemManager NetworkManager aaa_base acl audit bash bash-completion bash-sh btrfsmaintenance btrfsprogs btrfsprogs-udev-rules checkpolicy chrony chrony-pool-suse cni-plugins cockpit cockpit-bridge cockpit-networkmanager cockpit-podman cockpit-selinux cockpit-storaged cockpit-system cockpit-tukit cockpit-ws combustion container-selinux coreutils cpio cracklib cracklib-dict-small crypto-policies cryptsetup curl dbus-1 diffutils dnsmasq dosfstools dracut dracut-fips dracut-kiwi-lib dracut-kiwi-oem-dump dracut-kiwi-oem-repart dracut-transactional-update e2fsprogs ebtables efibootmgr elfutils file file-magic filesystem findutils firewalld fuse-overlayfs gawk gettext-runtime girepository-1_0 git git-core glib-networking glib2-tools glibc glibc-locale-base gnutls gpg2 gptfdisk grep grub2 grub2-i386-pc grub2-snapper-plugin grub2-x86_64-efi gsettings-desktop-schemas guestfs-tools gzip health-checker health-checker-plugins-MicroOS hostname hwinfo info iproute2 iptables irqbalance jeos-firstboot kbd kbd-legacy kdump kernel-default kernel-firmware-all kernel-firmware-amdgpu kernel-firmware-ath10k kernel-firmware-ath11k kernel-firmware-atheros kernel-firmware-bluetooth kernel-firmware-bnx2 kernel-firmware-brcm kernel-firmware-chelsio kernel-firmware-dpaa2 kernel-firmware-i915 kernel-firmware-intel kernel-firmware-iwlwifi kernel-firmware-liquidio kernel-firmware-marvell kernel-firmware-media kernel-firmware-mediatek kernel-firmware-mellanox kernel-firmware-mwifiex kernel-firmware-network kernel-firmware-nfp kernel-firmware-nvidia kernel-firmware-platform kernel-firmware-prestera kernel-firmware-qcom kernel-firmware-qlogic kernel-firmware-radeon kernel-firmware-realtek kernel-firmware-serial kernel-firmware-sound kernel-firmware-ti kernel-firmware-ueagle kernel-firmware-usb-network kexec-tools kmod kpartx less libacl1 libaio1 libargon2-1 libasm1 libassuan0 libattr1 libaudit1 libaugeas0 libauparse0 libbd_btrfs2 libbd_crypto2 libbd_fs2 libbd_loop2 libbd_lvm2 libbd_mdraid2 libbd_part2 libbd_swap2 libbd_utils2 libblkid1 libblockdev libblockdev2 libbpf1 libbrotlicommon1 libbrotlidec1 libbsd0 libbtrfs0 libbytesize1 libcap-ng0 libcap2 libcom_err2 libconfig11 libcontainers-common libcontainers-default-policy libcontainers-sles-mounts libcrack2 libcrypt1 libcryptsetup12 libcurl4 libdbus-1-3 libdw1 libebtc0 libeconf0 libedit0 libefa1 libefivar1 libelf1 libexpat1 libext2fs2 libfdisk1 libfdt1 libfreetype6 libfuse2 libfuse3-3 libgcrypt20 libgio-2_0-0 libgirepository-1_0-1 libglib-2_0-0 libgmodule-2_0-0 libgmp10 libgnutls30 libgobject-2_0-0 libgpg-error0 libgpgme11 libgudev-1_0-0 libguestfs0 libhidapi-hidraw0 libhogweed6 libibverbs libibverbs1 libidn2-0 libinih0 libip4tc2 libip6tc2 libjpeg8 libkcapi-tools libkmod2 libksba8 liblldp_clif1 liblz4-1 liblzma5 libmagic1 libmbim libmbim-glib4 libmlx4-1 libmlx5-1 libmm-glib0 libmnl0 libmount1 libmpath0 libmpfr6 libncurses6 libndctl6 libndp0 libnetfilter_conntrack3 libnettle8 libnfnetlink0 libnftables1 libnftnl11 libnghttp2-14 libnl-config libnl3-200 libnm0 libnpth0 libnss_usrfiles2 libnuma1 libp11-kit0 libpcap1 libpci3 libpciaccess0 libpcre2-8-0 libpixman-1-0 libpng16-16 libpopt0 libproxy1 libpsl5 libpwquality-tools libpwquality1 libqmi-glib5 libqmi-tools librados2 librbd1 librdmacm1 libseccomp2 libselinux1 libsemanage-conf libsemanage2 libsepol2 libsigc-2_0-0 libsmartcols1 libsnappy1 libsqlite3-0 libssh-config libssh4 libsystemd0 libtasn1-6 libtextstyle0 libtpms0 libtss2-esys0 libtss2-fapi1 libtss2-mu0 libtss2-rc0 libtss2-sys1 libtss2-tcti-device0 libtss2-tcti-tabrmd0 libtss2-tctildr0 libtukit4 libudev1 libudisks2-0 libudisks2-0_btrfs libudisks2-0_lvm2 liburing2 libusb-1_0-0 libuuid1 libvdeplug3 libverto1 libvirt-client libvirt-daemon libvirt-daemon-driver-network libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter libvirt-daemon-driver-qemu libvirt-daemon-driver-secret libvirt-daemon-driver-storage libvirt-daemon-driver-storage-core libvirt-daemon-driver-storage-disk libvirt-daemon-driver-storage-iscsi libvirt-daemon-driver-storage-iscsi-direct libvirt-daemon-driver-storage-logical libvirt-daemon-driver-storage-mpath libvirt-daemon-driver-storage-rbd libvirt-daemon-driver-storage-scsi libvirt-daemon-qemu libvirt-libs libwrap0 libx86emu3 libxkbcommon0 libxml2-2 libxml2-tools libxtables12 libyaml-0-2 libzck1 libzio1 libzmq5 libzstd1 login_defs logrotate lsof make makedumpfile mdevctl microos-tools mokutil ncurses-utils net-tools nfs-client nfs-kernel-server nftables open-lldp openssh openssh-clients openssh-common openssh-fips openssh-server openssl p11-kit p11-kit-tools pam pam-config pam_u2f parted perl perl-Bootloader perl-Error perl-Git perl-base pigz pinentry policycoreutils policycoreutils-python-utils psmisc pv python3-audit python3-firewall python3-nftables python3-policycoreutils python3-selinux qemu qemu-accel-tcg-x86 qemu-ipxe qemu-ovmf-x86_64 qemu-seabios qemu-tools qemu-vgabios qemu-x86 rdma-core read-only-root-fs rebootmgr rollback-helper rpcbind rpm rpm-config-SUSE rsync sed selinux-policy selinux-policy-targeted selinux-tools setroubleshoot-server shadow slirp4netns snapper sudo supportutils suse-module-tools swtpm system-group-audit systemd systemd-container systemd-rpm-macros tar terminfo-base thin-provisioning-tools toolbox tpm2-0-tss tpm2-tss-engine tpm2.0-abrmd tpm2.0-tools transactional-update transactional-update-zypp-config tukit tukitd typelib-1_0-NM-1_0 ucode-amd udev udisks2 update-alternatives util-linux util-linux-systemd which xfsprogs xkeyboard-config xtables-plugins xz yast2-logs zstd
...
2024-08-16 13:41 Please reboot your machine to activate the changes and avoid data loss.
2024-08-16 13:41 New default snapshot is #6 (/.snapshots/6/snapshot).
2024-08-16 13:41 transactional-update finished
---------------------------------------
</"transactional_update --continue run zypper migration --non-interactive --product SL-Micro/6.0/x86_64 --root" logic>
---------------------------------------

@ipetrov117
Copy link
Contributor Author

Let me know what you think of the indentation changes, if you feel like that they do not offer more value I can revert them back to the previous state.

atanasdinov
atanasdinov previously approved these changes Aug 16, 2024
Copy link
Contributor

@atanasdinov atanasdinov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

internal/upgrade/templates/os-upgrade.sh.tpl Outdated Show resolved Hide resolved
internal/upgrade/templates/os-upgrade.sh.tpl Outdated Show resolved Hide resolved
@ipetrov117 ipetrov117 merged commit f841784 into suse-edge:main Aug 19, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants