Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mount /usr: Implement try-sleep loop (SAN + FC) #1218

Merged
merged 1 commit into from
May 3, 2024

Conversation

pirat89
Copy link
Member

@pirat89 pirat89 commented May 3, 2024

This problem is typical for SAN + FC when the storage needs sometimes more time for the initialisation. Implemented try-sleep loop. Retry the activation of the storage + /usr mounting in 15s. The loop can be repeated 10 times, so total time is 150s right now for the activation.

Note that this is not proper solution for the storage initialisation, however we have discovered some obstacles in the bootup process to be able to do it correctly as we would like to. Regarding limited time, we are going to deliver this solution, that should improve the experience and should be safe to not cause regressions for already working functionality. We expect to provide better solution for newer upgrades paths in future (IPU 8->9 and newer)

jira: https://issues.redhat.com/browse/RHEL-3344

Note the related PR where we will try to address this problem properly will be most likely #1202

Copy link

github-actions bot commented May 3, 2024

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

  • review please @oamg/developers to notify leapp developers of the review request
  • /packit copr-build to submit a public copr build using packit

Packit will automatically schedule regression tests for this PR's build and latest upstream leapp build. If you need a different version of leapp, e.g. from PR#42, use /packit test oamg/leapp#42
Note that first time contributors cannot run tests automatically - they will be started by a reviewer.

It is possible to schedule specific on-demand tests as well. Currently 2 test sets are supported, beaker-minimal and kernel-rt, both can be used to be run on all upgrade paths or just a couple of specific ones.
To launch on-demand tests with packit:

  • /packit test --labels kernel-rt to schedule kernel-rt tests set for all upgrade paths
  • /packit test --labels beaker-minimal-8.10to9.4,kernel-rt-8.10to9.4 to schedule kernel-rt and beaker-minimal test sets for 8.10->9.4 upgrade path

See other labels for particular jobs defined in the .packit.yaml file.

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please contact leapp-infra.

@pirat89 pirat89 changed the title mount /usr: Implement try-sleep loop - add time for storage initialis… mount /usr: Implement try-sleep loop (SAN + FC) May 3, 2024
@pirat89 pirat89 force-pushed the lvm-san-try-sleep branch 2 times, most recently from 3242d97 to c0c7e0c Compare May 3, 2024 12:55
@pirat89 pirat89 added the bug Something isn't working label May 3, 2024
@pirat89 pirat89 added this to the 8.10/9.5 milestone May 3, 2024
…ation

This problem is typical for SAN + FC when the storage needs sometimes
more time for the initialisation. Implemented try-sleep loop.
Retry the activation of the storage + /usr mounting in 15s.
The loop can be repeated 10 times, so total time is 150s right now
for the activation.

Note that this is not proper solution for the storage initialisation,
however we have discovered some obstacles in the bootup process to
be able to do it correctly as we would like to. Regarding limited
time, we are going to deliver this solution, that should improve
the experience and should be safe to not cause regressions for already
working functionality. We expect to provide better solution for
newer upgrades paths in future (IPU 8->9 and newer).

jira: https://issues.redhat.com/browse/RHEL-3344
@pirat89
Copy link
Member Author

pirat89 commented May 3, 2024

Manual testing on my LVM VMs:

  • Seems it passed correctly on my several setups. Successfully upgraded system without seeing any errors.
  • Note that I haven't tested the scenario when the storage needs more time for the initialisation.

Lsblk from tested machines (used multiple VGs):

[root@localhost ~]# lsblk
NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda            252:0    0   30G  0 disk 
├─vda1         252:1    0  200M  0 part /boot/efi
├─vda2         252:2    0    1G  0 part /boot
├─vda3         252:3    0    7G  0 part 
│ └─rhel00-var 253:3    0    7G  0 lvm  /var
├─vda4         252:4    0    4G  0 part 
│ ├─rhel-root  253:0    0    2G  0 lvm  /
│ └─rhel-swap  253:1    0    2G  0 lvm  [SWAP]
└─vda5         252:5    0    6G  0 part 
  └─rhel01-usr 253:2    0    6G  0 lvm  /usr
[root@localhost ~]# lsblk
NAME           MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda            252:0    0  30G  0 disk 
├─vda1         252:1    0   1G  0 part /boot
├─vda2         252:2    0   8G  0 part 
│ └─rhel00-var 253:3    0   7G  0 lvm  /var
├─vda3         252:3    0   7G  0 part 
│ └─rhel01-usr 253:2    0   6G  0 lvm  /usr
├─vda4         252:4    0   1K  0 part 
└─vda5         252:5    0   6G  0 part 
  ├─rhel-root  253:0    0   3G  0 lvm  /
  └─rhel-swap  253:1    0   2G  0 lvm  [SWAP]
vdb            252:16   0  20G  0 disk 
[root@localhost ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda           252:0    0   20G  0 disk 
├─vda1        252:1    0  500M  0 part /boot/efi
├─vda2        252:2    0  500M  0 part /boot
└─vda3        252:3    0   19G  0 part 
  ├─rhel-root 253:0    0    4G  0 lvm  /
  ├─rhel-swap 253:1    0    2G  0 lvm  [SWAP]
  ├─rhel-usr  253:2    0    8G  0 lvm  /usr
  └─rhel-var  253:3    0    5G  0 lvm  /var

Copy link
Member

@MichalHe MichalHe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks OK, I've tried upgrading with the patch myself using a simple setup. As the results on more complex LVM setups conducted by @pirat89 we positive, I am merging this.

@MichalHe MichalHe merged commit 64e2c58 into oamg:master May 3, 2024
15 checks passed
@pirat89 pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label May 3, 2024
@pirat89 pirat89 deleted the lvm-san-try-sleep branch May 3, 2024 15:07
@rmetrich
Copy link
Contributor

rmetrich commented May 3, 2024

Sorry I don't like that proposal much, because it happens the sleep xxx helps in many other cases than having a dedicated /usr, in fact customer cases showed that it helps as soon as more than a few LVM LVs are present.
Hence I would suggest that even if there is no /usr to wait for, sleep at least a round (15 seconds).

@pirat89
Copy link
Member Author

pirat89 commented May 4, 2024

@rmetrich thanks for the feedback. Shouldn't be that covered by the LVM activation that's checked and happens before the mount_usr is called? I hoped that it could catch it. Here is #1219 PR applying sleep always.

pirat89 added a commit to pirat89/leapp-repository that referenced this pull request May 4, 2024
Based on feedback from @rmetrich¹ the sleep period should be applied
always as the problem with initialisation is happening also on systems
with higher than few number of LVs where /usr is not on dedicated
volume.

1: oamg#1218 (comment)
pirat89 added a commit to pirat89/leapp-repository that referenced this pull request May 4, 2024
Based on feedback from @rmetrich¹ the sleep period should be applied
always as the problem with initialisation is happening also on systems
with higher than few number of LVs where /usr is not on dedicated
volume.

1: oamg#1218 (comment)
pirat89 added a commit that referenced this pull request May 6, 2024
Based on feedback from @rmetrich¹ the sleep period should be applied
always as the problem with initialisation is happening also on systems
with higher than few number of LVs where /usr is not on dedicated
volume.

1: #1218 (comment)
yuravk pushed a commit to yuravk/leapp-repository that referenced this pull request Aug 16, 2024
Based on feedback from @rmetrich¹ the sleep period should be applied
always as the problem with initialisation is happening also on systems
with higher than few number of LVs where /usr is not on dedicated
volume.

1: oamg#1218 (comment)
(cherry picked from commit da5ce33)
pirat89 added a commit to pirat89/leapp-repository that referenced this pull request Aug 16, 2024
## Packaging
- .. names of packages, dependencies, changes in provided capabilities....

## Upgrade handling
### Fixes
- Add missing RHUI GCP config info for RHEL for SAP (oamg#1253)
- Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210)
- Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212)
- Fix evaluation of PES data (oamg#1194)
- Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169)
- Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240)
- Fix the parsing of the lscpu output (oamg#1184, oamg#1208)
- Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178)
- Fix upgrade on aarch64 via RHUI on AWS (oamg#1240)
- Handle a false positive GPG check error when TargetUserSpaceInfo is missing  (oamg#1269)
- Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205)
- Update the default kernel cmdline (oamg#1193, oamg#1216)
- Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211)
- Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219)
- [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243)
- [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260)
- [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270)

### Enhancements
- [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246)
- Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199)
- Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197)
- Improve set_systemd_services_states logging (oamg#1213)
- Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152)
- Introduce experimental upgrades in 'live' mode for the testing (oamg#1248)
- Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241)
- Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274)
- Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242)
- Sort lists in dnf-plugin-data for easier overview (oamg#1231)
- [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198)
- [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206)
- [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195)
- [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249)
- [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181)
- [IPU 9 -> 10] Introduced number of changes to enable experimental IPU 9 -> 10 (oamg#1169)
- [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268)
- [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273)

## Additional changes interesting for devels
- drop unused `packager` field from gpg-signatures.json (oamg#1233)
- [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible
- [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
@pirat89 pirat89 mentioned this pull request Aug 16, 2024
pirat89 added a commit that referenced this pull request Aug 16, 2024
## Packaging
- Start building for EL 9 in the upstream repository on COPR (#1169)

## Upgrade handling
### Fixes
- Add missing RHUI GCP config info for RHEL for SAP (#1253)
- Fix creation of the post upgrade report about changes in states of systemd services (#1210)
- Fix detection of valid sshd config with internal-sftp subsystem in Leapp (#1212)
- Fix evaluation of PES data (#1194)
- Fix failing "update-ca-trust" command caused by missing util-linux package (#1169)
- Fix handling of versions in RHUI configuration for ELS and SAP upgrades (#1240)
- Fix the parsing of the lscpu output (#1184, #1208)
- Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (#1178)
- Fix upgrade on aarch64 via RHUI on AWS (#1240)
- Handle a false positive GPG check error when TargetUserSpaceInfo is missing  (#1269)
- Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (#1205)
- Update the default kernel cmdline (#1193, #1216)
- Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (#1211)
- Wait for the storage initialization when /usr is on separate file system - covering SAN (#1218, #1219)
- [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (#1243)
- [IPU 7 -> 8] Fix detection of bootable device on RAID (#1260)
- [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (#1270)

### Enhancements
- [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (#1245, #1246)
- Update leapp data files (#1280)
- Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (#1199)
- Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (#1197)
- Improve set_systemd_services_states logging (#1213)
- Include leapp command execution and defined leapp envars inside leapp.db - (#1152)
- Introduce experimental upgrades in 'live' mode for the testing (#1248)
- Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (#1241)
- Several minor improvements in messages printed in console output (#1173, #1214, #1274)
- Several minor improvements in report and error messages (#1207, #1217, #1234, #1235, #1242)
- Sort lists in dnf-plugin-data for easier overview (#1231)
- [IPU 7 -> 8] Allow upgrade of content from ELS repositories (#1198)
- [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (#1206)
- [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (#1195)
- [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (#1249)
- [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (#1181)
- [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (#1169)
- [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (#1268)
- [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (#1169, #1273)

## Additional changes interesting for devels
- drop unused `packager` field from gpg-signatures.json (#1233)
- [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible
- [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo
yuravk pushed a commit to yuravk/leapp-repository that referenced this pull request Aug 20, 2024
## Packaging
- Start building for EL 9 in the upstream repository on COPR (oamg#1169)

## Upgrade handling
### Fixes
- Add missing RHUI GCP config info for RHEL for SAP (oamg#1253)
- Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210)
- Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212)
- Fix evaluation of PES data (oamg#1194)
- Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169)
- Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240)
- Fix the parsing of the lscpu output (oamg#1184, oamg#1208)
- Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178)
- Fix upgrade on aarch64 via RHUI on AWS (oamg#1240)
- Handle a false positive GPG check error when TargetUserSpaceInfo is missing  (oamg#1269)
- Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205)
- Update the default kernel cmdline (oamg#1193, oamg#1216)
- Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211)
- Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219)
- [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243)
- [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260)
- [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270)

### Enhancements
- [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246)
- Update leapp data files (oamg#1280)
- Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199)
- Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197)
- Improve set_systemd_services_states logging (oamg#1213)
- Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152)
- Introduce experimental upgrades in 'live' mode for the testing (oamg#1248)
- Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241)
- Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274)
- Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242)
- Sort lists in dnf-plugin-data for easier overview (oamg#1231)
- [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198)
- [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206)
- [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195)
- [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249)
- [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181)
- [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169)
- [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268)
- [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273)

## Additional changes interesting for devels
- drop unused `packager` field from gpg-signatures.json (oamg#1233)
- [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible
- [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo

(cherry picked from commit 03c257b)
yuravk pushed a commit to yuravk/leapp-repository that referenced this pull request Aug 20, 2024
Based on feedback from @rmetrich¹ the sleep period should be applied
always as the problem with initialisation is happening also on systems
with higher than few number of LVs where /usr is not on dedicated
volume.

1: oamg#1218 (comment)
(cherry picked from commit da5ce33)
yuravk pushed a commit to yuravk/leapp-repository that referenced this pull request Aug 20, 2024
## Packaging
- Start building for EL 9 in the upstream repository on COPR (oamg#1169)

## Upgrade handling
### Fixes
- Add missing RHUI GCP config info for RHEL for SAP (oamg#1253)
- Fix creation of the post upgrade report about changes in states of systemd services (oamg#1210)
- Fix detection of valid sshd config with internal-sftp subsystem in Leapp (oamg#1212)
- Fix evaluation of PES data (oamg#1194)
- Fix failing "update-ca-trust" command caused by missing util-linux package (oamg#1169)
- Fix handling of versions in RHUI configuration for ELS and SAP upgrades (oamg#1240)
- Fix the parsing of the lscpu output (oamg#1184, oamg#1208)
- Fix the upgrade of systems using RHUI on AWS after changes in RHUI client package (oamg#1178)
- Fix upgrade on aarch64 via RHUI on AWS (oamg#1240)
- Handle a false positive GPG check error when TargetUserSpaceInfo is missing  (oamg#1269)
- Target by default always "GA" channel repositories unless a different channel is specified for the leapp execution (oamg#1205)
- Update the default kernel cmdline (oamg#1193, oamg#1216)
- Update the device driver deprecation data, fixing invalid fields for some AMD CPUs (oamg#1211)
- Wait for the storage initialization when /usr is on separate file system - covering SAN (oamg#1218, oamg#1219)
- [IPU 7 -> 8] Drop enforced tomcat removal for satellite when upgrading to RHEL 8.10 (oamg#1243)
- [IPU 7 -> 8] Fix detection of bootable device on RAID (oamg#1260)
- [IPU 8 -> 9] Inhibit the upgrade to RHEL 9.5 on ARM architecture due to incompatibility of the RHEL 8 bootloader and RHEL 9.5 kernel (oamg#1270)

### Enhancements
- [IPU 8 -> 9] Introduce upgrade path 8.10 -> 9.5 (oamg#1245, oamg#1246)
- Update leapp data files (oamg#1280)
- Apply solutions for leftover rpms for all major upgrade paths - including experimental actors (oamg#1199)
- Do not terminate the upgrade dracut module execution anymore if /sysroot/root/tmp_leapp_py3/.leapp_upgrade_failed exists (oamg#1197)
- Improve set_systemd_services_states logging (oamg#1213)
- Include leapp command execution and defined leapp envars inside leapp.db - (oamg#1152)
- Introduce experimental upgrades in 'live' mode for the testing (oamg#1248)
- Load obsoleted GPG keys from gpg-signatures.json file instead of hardcoding them (oamg#1241)
- Several minor improvements in messages printed in console output (oamg#1173, oamg#1214, oamg#1274)
- Several minor improvements in report and error messages (oamg#1207, oamg#1217, oamg#1234, oamg#1235, oamg#1242)
- Sort lists in dnf-plugin-data for easier overview (oamg#1231)
- [IPU 7 -> 8] Allow upgrade of content from ELS repositories (oamg#1198)
- [IPU 7 -> 8] Inhibit the upgrade when Legacy GRUB is detected (oamg#1206)
- [IPU 7 -> 8] Inhibit the upgrade when embedding area is small to prevent failed bootloader update (oamg#1195)
- [IPU 8 -> 9] Enable EL 8 > 9 upgrades on Alibaba cloud (oamg#1249)
- [IPU 8 -> 9] Enable EL 8 to 9 upgrade of Satellite/Foreman server (oamg#1181)
- [IPU 9 -> 10] Introduced number of changes to enable IPU 9 -> 10 for testing (oamg#1169)
- [IPU 9 -> 10] Prevent upgrading if NetworkManager is configured with dhcp=dhclient (oamg#1268)
- [IPU 9 -> 10] Update URLs in reports to reflect the next planned major upgrade path (oamg#1169, oamg#1273)

## Additional changes interesting for devels
- drop unused `packager` field from gpg-signatures.json (oamg#1233)
- [IPU 9 -> 10] make system_upgrade/common leapp repo Python 3.12 compatible
- [IPU 9 -> 10] introduced system_upgrade/el9toel10 leapp repo

(cherry picked from commit 03c257b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants