CASM-4348: split ConfigMap feature with migration of single ConfigMap to multiple ConfigMaps as pre-upgrade process #304
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lathanm
changed the title
lathanm
requested review from
mharding-hpe,
arnab-muk-hpe,
paulose-kuriakose,
anoop1402 and
krabhi-hpe
mharding-hpe
force-pushed
the
feature/split-cm-migration
branch
4 times, most recently
from
6fa90bd to
c06b9e8
Compare
mharding-hpe
requested changes
Nov 29, 2023
mharding-hpe
force-pushed
the
feature/split-cm-migration
branch
from
143230e to
fb8ae16
Compare
mharding-hpe
force-pushed
the
feature/split-cm-migration
branch
from
368eb40 to
774006d
Compare
haasken-hpe
reviewed
Jan 12, 2024
mharding-hpe
dismissed stale reviews from haasken-hpe and themself
via
8609521
|
@lathanm Please review the feedback from @haasken-hpe and make changes as appropriate. Then I will re-review. Thanks! |
mharding-hpe
changed the title
… to multiple ConfigMaps as pre-upgrade process Also includes: * CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the cray-product-catalog Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's 'component_versions' data in its own ConfigMap. Modify the catalog_update.py script to update data across these ConfigMaps. * CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps * CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular product version. * Added new argument max_attempts to modify_config_map function in catalog_delete.py, because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. * CASM-4504: Added label 'type=cray-product-catalog' to all cray-product-catalog related ConfigMaps * Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps * Added migration job to the configmap-hook 7e617ee CASM-4427: Combine ConfigMap data in the query output and CASM-3981: Schema update 4c46725 Replaced the data with items while mocking the data 2ccae06 modified the mock data 93a4ac5 modified the mock data fd4d6c7 Reverted prev change ebd0b6c Added one more mock data c8d9b42 Resolved error 684d867 Resolved error cbf8e04 Resolved failure bdb2bab Changed message in test_query.py as per query.py 52cc275 Resolved failure 062ee3e Removed mocking mock_loadConfigMapData output for 2 funcs ba9d0c3 Removed mocking mock_loadConfigMapData output for 2 funcs bc71071 Added mock_invalid_data 7192d1a syntax error b46bb82 message formating corrected b72c1a1 Aded one more invalid data b40b341 created mock obj with invalid yaml ca51fa5 created mock obj with invalid yaml 300f387 created mock obj with duplicate value df6487b Commented test case test_create_product_catalog_invalid_product_data ad09463 pycodestyle correction 5175e2f Fixed invalid yaml testcase and uncommented it c99ee1e Fixed pytest_mod issues 352223c Fixed typo error dd7f529 Addressed review comments c9ebd76 Fixed pymod test errors 46049f8 Merging PR 269 215cd0c Fixed pycodestyle 65b492c CASM-4368: Delete cray-product-catalog details from main and sub cm for a particular product version c1a3c9a fixed pycodestyle error 224530f CASM-4350: Changes in Update code path to accommodate Split config maps. 4bbde55 Modified message in CHANGELOG.md 465e84e Addressed review comments fee6117 Added try catch block for call create_namespaced_config_map 8e39b49 (origin/feature/catalog-delete) Addressed review comments a1fc575 (origin/casm-4427) Addressed all except review comments related to test case f9f3158 (origin/catalog-delete-ut) adding unit test cases for delete logic+making the code more testable 4029e41 Addressing the pr comments by mharding,21-07-2023 b81be8b Addressing the pr comments by abhishek,24-07-2023 542ea63 - Added positive and negative testcases for docker, s3, helm, manifests - Modified query to look for configmaps starting with the ConfigMap name passed as argument. If nothing is passed then ConfigMap name will be set to cray-product-catalog 2c0eb39 Fixed pymodd test errors 9fa89c0 Removed trailing space 09ef4cd Added condition to come out of infiinte loop while updating the ConfigMap data dc97b11 Reverting ConfigMap doesn't exist logic as per Ryan's comment c5e584d Resolved merge conflict by incorporating both suggestions. 1fd17f3 Merge pull request #273 from Cray-HPE/catalog-delete-ut 05c6214 Added test cases for catalog_update.py Modified test cases of test_data_catalog_helper.py to use unittest 153c6cb Fixed pymodtest errors 9285943 Fixed pycodestyle error 1c19cf2 Fixed pycodestyle error b28eda2 Merge branch 'develop' into feature/catalog-delete ee34158 Update cray_product_catalog/catalog_delete.py efac9bd Addressed review comments 7283217 Resolved merge conflict by incorporating both suggestions. 3edbaa6 Minor change in Changelog 5606651 Addressed some of Ryaan's review comments a5043ea Addressed Ryaan's review comments: - Added log message if attempts exceeds max_attempts(retries) - Added test case to test max retries - Added logic to exit if create_ config_map fails a266279 Fixed pycode style error c0ca0b7 Resolved merge conflict by incorporating both suggestions. cd4b443 Merge pull request #271 from Cray-HPE/feature/catalog-delete 8935515 Resolved merge conflict by incorporating both suggestions. 8c5908b Merge pull request #261 from Cray-HPE/casm-4427 6572b3d Resolved merge conflict by incorporating both suggestions. 2e61364 Merge pull request #272 from Cray-HPE/feature/casm-4350 3f0d940 CASM-4504: Add label type=cray-product-catalog for all cray-product-catalog related ConfigMaps a4aa7d2 - Addressed review comment - Fixed unit test failure due to addition of label_serlector argument to list_namespaced_config_map function b28ec2f Fixed pycodestyle error 1b9d965 (origin/CASM-4504-label) Fixed license error ddafa16 (origin/feature/split-cm-migration-dockerfile) migration init 57ebc86 Addressed review comments ae26488 Removed duplicate line ec8d935 Merge pull request #278 from Cray-HPE/CASM-4504-label 38a79df init roll back 652d1df migration init cd365f4 rebase changes 2638d1f merge from epic feature branch e5adc31 rollback and exit handler aa41547 binary cleanup 8184a2d CASM-4279: Add Kubernetes Api for create/delete/read/etc bb68076 Optimisation bab6191 Initial change to handle config map data split into multiple configmaps b1475d9 Merge pull request #290 from Cray-HPE/casm-4529-migration-api 9f8cf3c Merge branch 'feature/split-cm-migration' into migration/data-manager e0478aa Merge pull request #291 from Cray-HPE/migration/data-manager ade53cf Merge branch 'feature/split-cm-migration' into feature/split-cm-migration-rollback-exit e3cc918 Merge pull request #289 from Cray-HPE/feature/split-cm-migration-rollback-exit 02a21a8 Added rollback operation and modified rename_config_map function 1c94ba7 Fixed few corner case logic 6445ff0 Added unit test cases 3ebaa20 added rename_config_map test case- not working 73645c8 Added few more test cases to configmap data handler f343b23 ut fixes f61d384 Added unit test cases related to main.py 67bc931 Fixing pycode-style test error d4e704d Addressed review comments Added revoke and grant update permission to main ConfigMap d53c411 Updated Copyright license for setup.py e171abe Fixed UT issue cdf51bb Modified code to accept env var for name and namespace 47239ec Added check against resource_version 36c63ed Fixed pycode style error 35c6496 Modified backup restore to use kubectl create command instead of kubectl apply 4867f7e Corrected license a7fb650 merged the migration job into the orginal helm chart 4c0a014 entry in changelog and updated license in values d9b8b23 changed to get the latest docker image 9b552ad modified to use the updated docker image 9056d1f (origin/add_rollback) Merge pull request #297 from Cray-HPE/add_rollback-pending 9ec7920 (origin/feature/split-cm-migration) Merge pull request #292 from Cray-HPE/add_rollback Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Abhishek Kumar <[email protected]> Co-authored-by: Mitch Harding <[email protected]> Co-authored-by: Mukherjee <[email protected]> Co-authored-by: U-ASIAPACIFIC\nanjundl <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: anoop1402 <[email protected]> Co-authored-by: lathanm <[email protected]> Co-authored-by: Ryan Haasken <[email protected]>
…l -- set it dynamically at build time; remove extra whitespace from values.yaml
Linting of code and comments
Set version as 1.10.1
Update version date in CHANGELOG
Change new version to `2.0.0` to reflect scope of changes
mharding-hpe
force-pushed
the
feature/split-cm-migration
branch
from
8609521 to
01e4bea
Compare
lathanan
force-pushed
the
feature/split-cm-migration
branch
from
0d9062f to
341eeaa
Compare
mharding-hpe
reviewed
Feb 8, 2024
mharding-hpe
reviewed
Feb 8, 2024
mharding-hpe
reviewed
Feb 8, 2024
mharding-hpe
reviewed
Feb 8, 2024
mharding-hpe
approved these changes
Feb 8, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary and Scope
CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the
cray-product-catalog Kubernetes ConfigMap is split into multiple smaller
ConfigMaps with each product's component_versions data in its own ConfigMap.
Modify the catalog_update.py script to update data across these ConfigMaps.
CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps
CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular
product version.
Added new argument max_attempts to modify_config_map function in
catalog_delete.py, because we need not retry 100
times when read ConfigMap fails for a product ConfigMap.
Issues and Related PRs
Resolves:
Testing
Tested on:
Unit tested on Build systems
Tested on Lemon drop using dummy ConfigMaps with actual scripts.
Functional tests done on mug and beau(vshasta2)