From e18715917b027ce23a2a0dba3ca8c3e7b83ab769 Mon Sep 17 00:00:00 2001 From: "Mitch Harding (the weird one)" Date: Wed, 29 Nov 2023 14:43:18 -0500 Subject: [PATCH 01/12] CASM-4348: split ConfigMap feature with migration of single ConfigMap to multiple ConfigMaps as pre-upgrade process Also includes: * CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the cray-product-catalog Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's 'component_versions' data in its own ConfigMap. Modify the catalog_update.py script to update data across these ConfigMaps. * CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps * CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular product version. * Added new argument max_attempts to modify_config_map function in catalog_delete.py, because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. * CASM-4504: Added label 'type=cray-product-catalog' to all cray-product-catalog related ConfigMaps * Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps * Added migration job to the configmap-hook 7e617eeaea3a41184453aafcc60ff456c0a5482f CASM-4427: Combine ConfigMap data in the query output and CASM-3981: Schema update 4c467259ef4549b1f36126b81047aa7612f4f0a6 Replaced the data with items while mocking the data 2ccae068338ba41d02e73d73275b87ea38ea15b6 modified the mock data 93a4ac5d41b949a4405c7567d1ed0d25a953de66 modified the mock data fd4d6c76ff72dcd4777898e35d1881696f5e0865 Reverted prev change ebd0b6cc757d67c64aee3140e6b37a23e6da4fe5 Added one more mock data c8d9b42c89e3958d2064806b3789d581415b533b Resolved error 684d867654f363de80ecaac4eb54ed13eb2d71b3 Resolved error cbf8e044d171512ab1231e246f6464b2d4a0b584 Resolved failure bdb2bab6bd440857260f925fd8852da78355a5b4 Changed message in test_query.py as per query.py 52cc275661151766691c8bddfc3c2022c4804272 Resolved failure 062ee3e98030cca98b7a4994a531f90480060900 Removed mocking mock_loadConfigMapData output for 2 funcs ba9d0c358f05a3113149a3292afe9ee8b6524c35 Removed mocking mock_loadConfigMapData output for 2 funcs bc7107176a224b651fb4c59585cb03a757e4996a Added mock_invalid_data 7192d1a6f37a1d4036817b32174c9f047e6e52fd syntax error b46bb82bb9d25f54f76407e1fec8c3bcbbec50ad message formating corrected b72c1a1ca33de19ffd735ed1b9487bb56b717d78 Aded one more invalid data b40b341fa28ccbab0563f4c5a897a6466e4a76e1 created mock obj with invalid yaml ca51fa574b8ce729561acfae052f087cd903bd33 created mock obj with invalid yaml 300f3878c32cc5f3a753c8e577fb53d878b59d91 created mock obj with duplicate value df6487bef02726db7888749b8a326c200b4d0a9e Commented test case test_create_product_catalog_invalid_product_data ad09463e4e9a0584a3f8758ab7312f690af9259c pycodestyle correction 5175e2fd5a6560b412abe5d75b0bd9dd67213cc0 Fixed invalid yaml testcase and uncommented it c99ee1e9c593ebc4549a58d2badb51eae9ad2fc7 Fixed pytest_mod issues 352223c10ebac8b96a73442e992629a37fa80b33 Fixed typo error dd7f529f0bd3bab6cc1aa9261f8f901bbeb0b97d Addressed review comments c9ebd7606c11419efeefe5fd0d82d539968c21a2 Fixed pymod test errors 46049f88b5ceb6fcbbf897bf08384de34c94af7b Merging PR 269 215cd0cddc9241de4114b382906ec485f904274f Fixed pycodestyle 65b492c882ac5105bd592aa7fc7a7e9f92808747 CASM-4368: Delete cray-product-catalog details from main and sub cm for a particular product version c1a3c9a9621537c36b364bf9de14497d1a4aba02 fixed pycodestyle error 224530f6a62d9b3a0948e08fc2d9c9bc4992fab6 CASM-4350: Changes in Update code path to accommodate Split config maps. 4bbde5576b98debd15aa743e8e2e4b0baeae4163 Modified message in CHANGELOG.md 465e84e9e3f6e17495c45dd764ce79cdc75ab741 Addressed review comments fee61172b4db72fa7a0685cf17bc12ea6ec0be4f Added try catch block for call create_namespaced_config_map 8e39b499461cec22f3b8df96a1a0f9f9b549d910 (origin/feature/catalog-delete) Addressed review comments a1fc5758f3f0e6f7944d8cce264c84dcef9138b1 (origin/casm-4427) Addressed all except review comments related to test case f9f31582f3a13f20641b4dbd78fa9b8fc8e88e0d (origin/catalog-delete-ut) adding unit test cases for delete logic+making the code more testable 4029e41ad1d6dc24e053f4d8296a50cada8d33d1 Addressing the pr comments by mharding,21-07-2023 b81be8bf3c531fe99b2f4af04bc5437836cc585e Addressing the pr comments by abhishek,24-07-2023 542ea63c7694d7cf2f96ffe12bf958dde95b5966 - Added positive and negative testcases for docker, s3, helm, manifests - Modified query to look for configmaps starting with the ConfigMap name passed as argument. If nothing is passed then ConfigMap name will be set to cray-product-catalog 2c0eb39775948b00a81f6e52157a2dd5a7891b26 Fixed pymodd test errors 9fa89c063a03ac11a2edb35058d2b143d8273216 Removed trailing space 09ef4cda9faee20cf6e0e8826a79ea9fe073c5ae Added condition to come out of infiinte loop while updating the ConfigMap data dc97b118f7da5e6d9b5191bb8447189e821a8ff7 Reverting ConfigMap doesn't exist logic as per Ryan's comment c5e584dc7458f9a8c95632e58ba3da0b3abdbb1a Resolved merge conflict by incorporating both suggestions. 1fd17f3f60ec93c69d848e4ea8bdccf80515afb9 Merge pull request #273 from Cray-HPE/catalog-delete-ut 05c62140b97348cbd9847c994237a1c8833daefe Added test cases for catalog_update.py Modified test cases of test_data_catalog_helper.py to use unittest 153c6cb8b031da81a3ea5890fb1831e4f14b1c91 Fixed pymodtest errors 928594399a3f0572704b8ddc362c8d52f68d8b64 Fixed pycodestyle error 1c19cf25f4ae079c6c28288a490e688ca1babd6f Fixed pycodestyle error b28eda25b35695215a58513f27b804dcf3017142 Merge branch 'develop' into feature/catalog-delete ee341585a45c82f888845469b1b94705ba4519c6 Update cray_product_catalog/catalog_delete.py efac9bd4b5aa3acca22e4893b1a7e4f2e5d1c739 Addressed review comments 728321795727314a95c952c037a0f466516d6cdd Resolved merge conflict by incorporating both suggestions. 3edbaa65081757d3aeb61078db5a96a1130d0332 Minor change in Changelog 56066513bdd033024ba2d97a7747ff6d592f4dfc Addressed some of Ryaan's review comments a5043ead93e6c1670f8533c7dd69018170c6697b Addressed Ryaan's review comments: - Added log message if attempts exceeds max_attempts(retries) - Added test case to test max retries - Added logic to exit if create_ config_map fails a26627967b1189b448f22af6383896a72fac9d93 Fixed pycode style error c0ca0b788b55ef2e43801e655a0ab32bea69e972 Resolved merge conflict by incorporating both suggestions. cd4b44375b3e6f050d9d5cf4dbfcdf3ca0356301 Merge pull request #271 from Cray-HPE/feature/catalog-delete 893551566c75571a86b13238168a434ab953fd25 Resolved merge conflict by incorporating both suggestions. 8c5908bf8a2500c5a5ea7ba2c5ca29e38c92f378 Merge pull request #261 from Cray-HPE/casm-4427 6572b3d0bf392f7e6fd6df5a072391f63fb783f8 Resolved merge conflict by incorporating both suggestions. 2e61364156aa18ae0e2deb7ff876c27b24edfde9 Merge pull request #272 from Cray-HPE/feature/casm-4350 3f0d940b9bb150c3bcc541db7e0bbff5cf212c8b CASM-4504: Add label type=cray-product-catalog for all cray-product-catalog related ConfigMaps a4aa7d20ae922ee945047e0ab7b7c5b20f8f44bc - Addressed review comment - Fixed unit test failure due to addition of label_serlector argument to list_namespaced_config_map function b28ec2f8487b789ddd75cf63cda021dd5629d91e Fixed pycodestyle error 1b9d965eeffcdd28c732eedc613122b09f360c46 (origin/CASM-4504-label) Fixed license error ddafa1651b98a2153b6f8acca80eb54350219056 (origin/feature/split-cm-migration-dockerfile) migration init 57ebc86d1dc6570a259fbe772f6157f938a7cdef Addressed review comments ae26488e3c6bb37554885ccbc242b600b22a4013 Removed duplicate line ec8d9359f78741adaddd4515725aea07452cad39 Merge pull request #278 from Cray-HPE/CASM-4504-label 38a79dfab5d47f18a383d92235f5e63910df13e7 init roll back 652d1dfec93457d73fe8483d241345320987a2c6 migration init cd365f454ae8c473349b429df5a89c4f163aa368 rebase changes 2638d1f0cf2c662d3a99de904e9b846b9beb97fe merge from epic feature branch e5adc3102240adf25f57510c15c68936ac639bde rollback and exit handler aa415473e4bf19e9d56fe8baaac19a83dea74a4b binary cleanup 8184a2dc0e4c70484492dddf5b2d30af1d9c200a CASM-4279: Add Kubernetes Api for create/delete/read/etc bb6807666fae6814859d1a9d4156aab0b8448162 Optimisation bab6191522bd6b7e9d238a3942d1bb04e7977a78 Initial change to handle config map data split into multiple configmaps b1475d9fec2d4097a0f325a8eb830bebcc0ec41b Merge pull request #290 from Cray-HPE/casm-4529-migration-api 9f8cf3c1d48fd0f983e242449aad7c984de0bb85 Merge branch 'feature/split-cm-migration' into migration/data-manager e0478aaea936099093b847ea5ebf4e8d0824a6cb Merge pull request #291 from Cray-HPE/migration/data-manager ade53cfdb36d4b17e33c6c6eb6e8c37d515cd623 Merge branch 'feature/split-cm-migration' into feature/split-cm-migration-rollback-exit e3cc918483d8f8c98b91d4aef623f7168c6193d3 Merge pull request #289 from Cray-HPE/feature/split-cm-migration-rollback-exit 02a21a820110d1eabb95673caebe5ff03eb32fa7 Added rollback operation and modified rename_config_map function 1c94ba7d13614bef65a0b156a7b2f2c0289a7f17 Fixed few corner case logic 6445ff0b76451d3bc6cf408a124eec3e1d6a3f06 Added unit test cases 3ebaa20396e529f99acd6dd870d6e2ab5ee8d9bb added rename_config_map test case- not working 73645c841abbc101b39d65e2551c0588adad44f4 Added few more test cases to configmap data handler f343b2395dce9aaef27d2c5033d73615e422c0d3 ut fixes f61d3844f8b9b12505fae1ebd7d912542ced3b07 Added unit test cases related to main.py 67bc93130f7c5657460f570239dc58476471e123 Fixing pycode-style test error d4e704d3555d33e2dfa3d5a416c44c84ae88e84e Addressed review comments Added revoke and grant update permission to main ConfigMap d53c411f2b9660aa796617daf587e31b48444a8b Updated Copyright license for setup.py e171abe1371c130a9ad507993ae472056dde170e Fixed UT issue cdf51bb2064b0f0b6873cb1f015d631d15b1a39d Modified code to accept env var for name and namespace 47239ecab4471f538b913768da0d4d850dd86f15 Added check against resource_version 36c63ed9c2b478c81fe0dd47191d9a218a39ab53 Fixed pycode style error 35c6496518ee2b20391e67f2c4ea258f341ced14 Modified backup restore to use kubectl create command instead of kubectl apply 4867f7eaa030a0f85ef4c37f026b9107d6348ad6 Corrected license a7fb65009d0e88e551e224b4b6f274325c3a62c5 merged the migration job into the orginal helm chart 4c0a0147020e8fbbe6a64b1c4504d9fe93fdf2b5 entry in changelog and updated license in values d9b8b235e836ac7d99668362a6628e0bc1ea7091 changed to get the latest docker image 9b552ad3bb537cf4a532d373d6310e7b3f3fa00e modified to use the updated docker image 9056d1fb9612c10b6c031d08d7b54672a321e0d8 (origin/add_rollback) Merge pull request #297 from Cray-HPE/add_rollback-pending 9ec79203cb607b359dbb750a64921dcb0cc65724 (origin/feature/split-cm-migration) Merge pull request #292 from Cray-HPE/add_rollback Co-authored-by: Abhishek Kumar <136788642+krabhi-hpe@users.noreply.github.com> Co-authored-by: Abhishek Kumar Co-authored-by: Mitch Harding Co-authored-by: Mukherjee Co-authored-by: U-ASIAPACIFIC\nanjundl Co-authored-by: anoop1402 <131955374+anoop1402@users.noreply.github.com> Co-authored-by: anoop1402 Co-authored-by: lathanm <128785927+lathanm@users.noreply.github.com> Co-authored-by: Ryan Haasken --- .gitignore | 8 + CHANGELOG.md | 16 + .../templates/configmap-hook.yaml | 42 +- .../templates/configmap.yaml | 4 +- charts/cray-product-catalog/values.yaml | 16 +- cray_product_catalog/catalog_delete.py | 209 +++++- cray_product_catalog/catalog_update.py | 55 +- cray_product_catalog/constants.py | 8 + cray_product_catalog/migration/__init__.py | 46 ++ .../migration/config_map_data_handler.py | 182 +++++ .../migration/exit_handler.py | 97 +++ cray_product_catalog/migration/kube_apis.py | 168 +++++ cray_product_catalog/migration/main.py | 132 ++++ .../migration/rollback_handler.py | 0 cray_product_catalog/query.py | 55 +- cray_product_catalog/schema/schema.yaml | 2 +- .../util/catalog_data_helper.py | 68 ++ setup.py | 5 +- tests/migration/__init__.py | 0 tests/migration/migration_mock.py | 130 ++++ .../migration/test_config_map_data_handler.py | 677 ++++++++++++++++++ tests/migration/test_exit_handler.py | 110 +++ tests/mock_update_catalog.py | 110 +++ tests/mocks.py | 129 +++- tests/test_catalog_delete.py | 141 ++++ tests/test_query.py | 48 +- tests/test_update_catalog.py | 192 +++++ tests/util/test_data_catalog_helper.py | 124 ++++ 28 files changed, 2711 insertions(+), 63 deletions(-) create mode 100644 cray_product_catalog/migration/__init__.py create mode 100644 cray_product_catalog/migration/config_map_data_handler.py create mode 100644 cray_product_catalog/migration/exit_handler.py create mode 100644 cray_product_catalog/migration/kube_apis.py create mode 100644 cray_product_catalog/migration/main.py create mode 100644 cray_product_catalog/migration/rollback_handler.py create mode 100755 cray_product_catalog/util/catalog_data_helper.py create mode 100644 tests/migration/__init__.py create mode 100755 tests/migration/migration_mock.py create mode 100755 tests/migration/test_config_map_data_handler.py create mode 100644 tests/migration/test_exit_handler.py create mode 100755 tests/mock_update_catalog.py create mode 100644 tests/test_catalog_delete.py create mode 100755 tests/test_update_catalog.py create mode 100755 tests/util/test_data_catalog_helper.py diff --git a/.gitignore b/.gitignore index 1010a238..d6a5c321 100644 --- a/.gitignore +++ b/.gitignore @@ -5,3 +5,11 @@ dist/ # VS Code config workspace.code-workspace + + +# cache specific +*.pyc +**__pycache__** + +# removing pycharm specific +.idea \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index 824cedee..4752f468 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the + `cray-product-catalog` Kubernetes ConfigMap is split into multiple smaller + ConfigMaps with each product's `component_versions` data in its own ConfigMap. + Modify the `catalog_update.py` script to update data across these ConfigMaps. +- CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps +- CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular + product version. +- Added new argument `max_attempts` to `modify_config_map` function in + [`catalog_delete.py`](cray_product_catalog/catalog_delete.py), because we need not retry 100 + times when read ConfigMap fails for a product ConfigMap. +- CASM-4504: Added label "type=cray-product-catalog" to all cray-product-catalog related ConfigMaps +- Implemented migration of cray-product-catalog ConfigMap to multiple ConfigMaps as part of pre-upgrade steps +- Added migration job to the configmap-hook + ### Dependencies - Bump `tj-actions/changed-files` from 40 to 42 ([#307](https://github.com/Cray-HPE/cray-product-catalog/pull/307), [#309](https://github.com/Cray-HPE/cray-product-catalog/pull/309)) diff --git a/charts/cray-product-catalog/templates/configmap-hook.yaml b/charts/cray-product-catalog/templates/configmap-hook.yaml index 6dd44313..4a3ea393 100644 --- a/charts/cray-product-catalog/templates/configmap-hook.yaml +++ b/charts/cray-product-catalog/templates/configmap-hook.yaml @@ -1,7 +1,7 @@ {{/* MIT License -(C) Copyright 2021-2022 Hewlett Packard Enterprise Development LP +(C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), @@ -43,10 +43,11 @@ spec: command: ["/bin/sh"] args: - "-c" - - "kubectl get cm -n services cpc-backup -o yaml | + - "kubectl delete cm -n services cray-product-catalog && + kubectl get cm -n services cpc-backup -o yaml | yq eval '.metadata.name = \"cray-product-catalog\"' - | yq eval 'del(.metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields)' - | - kubectl apply -f - && kubectl delete cm -n services cpc-backup" + kubectl create -f - && kubectl delete cm -n services cpc-backup" --- apiVersion: batch/v1 @@ -56,6 +57,7 @@ metadata: namespace: services annotations: "helm.sh/hook": pre-upgrade,pre-rollback + "helm.sh/hook-weight": "2" spec: template: metadata: @@ -73,4 +75,36 @@ spec: - "kubectl get cm -n services cray-product-catalog -o yaml | yq eval '.metadata.name = \"cpc-backup\"' - | yq eval 'del(.metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields)' - | - kubectl apply -f -" \ No newline at end of file + kubectl create -f -" + + +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: {{ .Release.Name }}-my-job + annotations: + "helm.sh/hook": pre-upgrade + "helm.sh/hook-weight": "1" +spec: + backoffLimit: 0 + template: + spec: + containers: + - args: + - -c + - /usr/bin/catalog_migrate + command: + - /bin/sh + env: + - name: CONFIG_MAP + value: {{ .Values.migration.configMap }} + - name: CONFIG_MAP_NAMESPACE + value: {{ .Values.migration.configMapNamespace }} + image: "{{ .Values.migration.image.repository }}:{{ .Values.global.appVersion }}" + imagePullPolicy: IfNotPresent + name: migrate-catalog + restartPolicy: Never + serviceAccount: {{ .Values.migration.serviceAccount }} + serviceAccountName: {{ .Values.migration.serviceAccount }} + diff --git a/charts/cray-product-catalog/templates/configmap.yaml b/charts/cray-product-catalog/templates/configmap.yaml index 697bee06..3808c9a3 100644 --- a/charts/cray-product-catalog/templates/configmap.yaml +++ b/charts/cray-product-catalog/templates/configmap.yaml @@ -1,7 +1,7 @@ {{/* MIT License -(C) Copyright 2021-2022 Hewlett Packard Enterprise Development LP +(C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), @@ -25,4 +25,6 @@ apiVersion: v1 kind: ConfigMap metadata: name: cray-product-catalog + labels: + type: cray-product-catalog data: null diff --git a/charts/cray-product-catalog/values.yaml b/charts/cray-product-catalog/values.yaml index 8b423477..96355abe 100644 --- a/charts/cray-product-catalog/values.yaml +++ b/charts/cray-product-catalog/values.yaml @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2021-2022 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), @@ -25,3 +25,17 @@ kubectl: image: repository: artifactory.algol60.net/csm-docker/stable/docker-kubectl tag: 1.19.15 +migration: + image: + repository: artifactory.algol60.net/csm-docker/unstable/cray-product-catalog-update + tag: $DOCKER_VERSION + configMap: cray-product-catalog + configMapNamespace: services + serviceAccount: cray-product-catalog + + +global: + appVersion: 1.10.0-add-rollback-pending.106_d9b8b23 + +global: + appVersion: 1.10.0-add-rollback-pending.106_d9b8b23 diff --git a/cray_product_catalog/catalog_delete.py b/cray_product_catalog/catalog_delete.py index 43dd3457..a8b8f195 100755 --- a/cray_product_catalog/catalog_delete.py +++ b/cray_product_catalog/catalog_delete.py @@ -2,7 +2,7 @@ # # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), @@ -55,13 +55,174 @@ from cray_product_catalog.logging import configure_logging from cray_product_catalog.util import load_k8s +from cray_product_catalog.util.catalog_data_helper import format_product_cm_name +from cray_product_catalog.constants import ( + CONFIG_MAP_FIELDS, + PRODUCT_CM_FIELDS, +) urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) LOGGER = logging.getLogger(__name__) +# kubernetes API response code +ERR_NOT_FOUND = 404 +ERR_CONFLICT = 409 -def modify_config_map(name, namespace, product, product_version, key=None): +# retries +MAX_RETRIES = 100 +MAX_RETRIES_FOR_PROD_CM = 10 + + +class ModifyConfigMapUtil: + """Utility class to manage the config map modification + """ + + def __init__(self): + self.__main_cm = None + self.__product_cm = None + self.__cm_namespace = None + self.__product_name = None + self.__product_version = None + self.__max_retries_for_main_cm = None + self.__max_retries_for_prod_cm = None + self.__key = None + self.__main_cm_fields = None + self.__product_cm_fields = None + + # property definitions + @property + def main_cm(self): + return self.__main_cm + + @main_cm.setter + def main_cm(self, __main_cm): + self.__main_cm = __main_cm + + @property + def product_cm(self): + return self.__product_cm + + @product_cm.setter + def product_cm(self, __product_cm): + self.__product_cm = __product_cm + + @property + def cm_namespace(self): + return self.__cm_namespace + + @cm_namespace.setter + def cm_namespace(self, __cm_namespace): + self.__cm_namespace = __cm_namespace + + @property + def product_name(self): + return self.__product_name + + @product_name.setter + def product_name(self, __product_name): + self.__product_name = __product_name + + @property + def product_version(self): + return self.__product_version + + @product_version.setter + def product_version(self, __product_version): + self.__product_version = __product_version + + @property + def max_retries_for_main_cm(self): + return self.__max_retries_for_main_cm + + @max_retries_for_main_cm.setter + def max_retries_for_main_cm(self, __max_reties_for_main_cm): + self.__max_retries_for_main_cm = __max_reties_for_main_cm + + @property + def max_retries_for_prod_cm(self): + return self.__max_retries_for_prod_cm + + @max_retries_for_prod_cm.setter + def max_retries_for_prod_cm(self, __max_reties_for_prod_cm): + self.__max_retries_for_prod_cm = __max_reties_for_prod_cm + + @property + def key(self): + return self.__key + + @key.setter + def key(self, __key): + self.__key = __key + + @property + def main_cm_fields(self): + return self.__main_cm_fields + + @main_cm_fields.setter + def main_cm_fields(self, __main_cm_fields): + self.__main_cm_fields = __main_cm_fields + + @property + def product_cm_fields(self): + return self.__product_cm_fields + + @product_cm_fields.setter + def product_cm_fields(self, __product_cm_fields): + self.__product_cm_fields = __product_cm_fields + + # private methods + def __key_belongs_to_main_cm_fields(self): + return self.__key in self.__main_cm_fields + + def __key_belongs_to_prod_cm_fields(self): + return self.__key in self.__product_cm_fields + + def __modify_main_cm(self): + LOGGER.info("Removing from config_map=%s in namespace=%s for %s/%s (key=%s)", + self.main_cm, self.cm_namespace, self.product_name, self.product_version, self.key) + modify_config_map(self.__main_cm, self.__cm_namespace, self.__product_name, self.__product_version, + self.__key, self.__max_retries_for_main_cm, ) + + def __modify_product_cm(self): + LOGGER.info("Removing from config_map=%s in namespace=%s for %s/%s (key=%s)", + self.product_cm, self.cm_namespace, self.product_name, self.product_version, self.key) + modify_config_map(self.__product_cm, self.__cm_namespace, self.__product_name, self.__product_version, + self.__key, self.__max_retries_for_prod_cm, ) + + # public method + def modify(self): + """ + Method to initiate modification of ConfigMaps. + Before executing this method make sure to set these properties of the class: + * main_cm # name of main ConfigMap + * product_cm # name of product specific CofigMap + * cm_namespace # Namespace containing all config map + * product_name # Product name + * product_version # Product version + * max_reties_for_main_cm # Max failure retries for main ConfigMap + * max_reties_for_prod_cm # Max failure retries for product ConfigMap + * key # Key to delete, if you want to execute complete product or a particular version, ignore it + * main_cm_fields # Fields present in main ConfigMap + * product_cm_fields # Fields present in product specific ConfigMap + """ + if self.__key: + if self.__key_belongs_to_main_cm_fields(): + self.__modify_main_cm() + + elif self.__key_belongs_to_prod_cm_fields(): + self.__modify_product_cm() + + else: + LOGGER.warning("key=%s NOT present in Main/Product ConfigMap, exiting", self.key) + + return + + self.__modify_main_cm() + self.__modify_product_cm() + + +def modify_config_map(name, namespace, product, product_version, key=None, max_attempts=MAX_RETRIES): """Remove a product version from the catalog ConfigMap. If a key is specified, delete the `key` content from a specific section @@ -74,7 +235,7 @@ def modify_config_map(name, namespace, product, product_version, key=None): 4. Repeat steps 2-3 if ConfigMap does not reflect the changes requested """ k8sclient = ApiClient() - retries = 100 + retries = max_attempts retry = Retry( total=retries, read=retries, connect=retries, backoff_factor=0.3, status_forcelist=(500, 502, 503, 504) @@ -82,7 +243,6 @@ def modify_config_map(name, namespace, product, product_version, key=None): k8sclient.rest_client.pool_manager.connection_pool_kw['retries'] = retry api_instance = client.CoreV1Api(k8sclient) attempt = 0 - max_attempts = 100 while True: @@ -101,7 +261,7 @@ def modify_config_map(name, namespace, product, product_version, key=None): LOGGER.exception("Error calling read_namespaced_config_map") # ConfigMap doesn't exist yet - if err.status == 404 and attempt < max_attempts: + if err.status == ERR_NOT_FOUND and attempt < max_attempts: LOGGER.warning("ConfigMap %s/%s doesn't exist, attempting again.", namespace, name) continue raise # unrecoverable @@ -155,27 +315,48 @@ def modify_config_map(name, namespace, product, product_version, key=None): name, namespace, client.V1ConfigMap(data=config_map_data) ) LOGGER.info("ConfigMap update attempt %s successful", attempt) - except ApiException: - LOGGER.exception("Error calling patch_namespaced_config_map") + except ApiException as exc: + if exc.status == ERR_CONFLICT: + # A conflict is raised if the resourceVersion field was unexpectedly + # incremented, e.g. if another process updated the ConfigMap. This + # provides concurrency protection. + LOGGER.warning("Conflict updating ConfigMap") + else: + LOGGER.exception("Error calling patch_namespaced_config_map") def main(): """ Main function """ + + # logging configuration configure_logging() + # Parameters to identify ConfigMap and product/version to remove PRODUCT = os.environ.get("PRODUCT").strip() # required PRODUCT_VERSION = os.environ.get("PRODUCT_VERSION").strip() # required - CONFIG_MAP = os.environ.get("CONFIG_MAP", "cray-product-catalog").strip() CONFIG_MAP_NS = os.environ.get("CONFIG_MAP_NAMESPACE", "services").strip() + CONFIG_MAP = os.environ.get("CONFIG_MAP", "cray-product-catalog").strip() + PRODUCT_CONFIG_MAP = format_product_cm_name(CONFIG_MAP, PRODUCT) KEY = os.environ.get("KEY", "").strip() or None - args = (CONFIG_MAP, CONFIG_MAP_NS, PRODUCT, PRODUCT_VERSION, KEY) - LOGGER.info( - "Removing from ConfigMap=%s in namespace=%s for %s/%s (key=%s)", - *args - ) + # k8 related configurations load_k8s() - modify_config_map(*args) + + # building the utility class + modify_config_map_util = ModifyConfigMapUtil() + modify_config_map_util.main_cm = CONFIG_MAP + modify_config_map_util.product_cm = PRODUCT_CONFIG_MAP + modify_config_map_util.cm_namespace = CONFIG_MAP_NS + modify_config_map_util.product_name = PRODUCT + modify_config_map_util.product_version = PRODUCT_VERSION + modify_config_map_util.max_retries_for_main_cm = MAX_RETRIES + modify_config_map_util.max_retries_for_prod_cm = MAX_RETRIES_FOR_PROD_CM + modify_config_map_util.key = KEY + modify_config_map_util.main_cm_fields = CONFIG_MAP_FIELDS + modify_config_map_util.product_cm_fields = PRODUCT_CM_FIELDS + + # Modifying ConfigMap + modify_config_map_util.modify() if __name__ == "__main__": diff --git a/cray_product_catalog/catalog_update.py b/cray_product_catalog/catalog_update.py index 96a95d6d..be0fcbc9 100755 --- a/cray_product_catalog/catalog_update.py +++ b/cray_product_catalog/catalog_update.py @@ -54,13 +54,16 @@ from cray_product_catalog.schema.validate import validate from cray_product_catalog.util.k8s import load_k8s from cray_product_catalog.util.merge_dict import merge_dict +from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL + urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) # Parameters to identify ConfigMap and content in it to update PRODUCT = os.environ.get("PRODUCT").strip() # required PRODUCT_VERSION = os.environ.get("PRODUCT_VERSION").strip() # required -CONFIG_MAP = os.environ.get("CONFIG_MAP", "cray-product-catalog").strip() +MAIN_CONFIG_MAP = os.environ.get("CONFIG_MAP", "cray-product-catalog").strip() CONFIG_MAP_NAMESPACE = os.environ.get("CONFIG_MAP_NAMESPACE", "services").strip() # One of (YAML_CONTENT_FILE, YAML_CONTENT_STRING) required. For backwards compatibility, YAML_CONTENT # may also be given in place of YAML_CONTENT_FILE. @@ -134,6 +137,21 @@ def active_field_exists(product_data): return any("active" in product_data[version] for version in product_data) +def create_config_map(api_instance, name, namespace): + """Create new product ConfigMap.""" + try: + new_cm = V1ConfigMap() + new_cm.metadata = V1ObjectMeta(name=name, labels=PRODUCT_CATALOG_CONFIG_MAP_LABEL) + api_instance.create_namespaced_config_map( + namespace=namespace, body=new_cm + ) + LOGGER.info("Created product ConfigMap %s/%s", namespace, name) + return True + except ApiException: + LOGGER.error("Error calling create_namespaced_config_map") + return False + + def update_config_map(data: dict, name, namespace): """ Get the ConfigMap `data` to be added. @@ -155,7 +173,7 @@ def update_config_map(data: dict, name, namespace): api_instance = client.CoreV1Api(k8sclient) attempt = 0 - while True: + while attempt < retries: # Wait a while to check the ConfigMap in case multiple products are # attempting to update the same ConfigMap, or the ConfigMap doesn't @@ -172,10 +190,17 @@ def update_config_map(data: dict, name, namespace): LOGGER.exception("Error calling read_namespaced_config_map") # ConfigMap doesn't exist yet - if err.status == ERR_NOT_FOUND: + if err.status != ERR_NOT_FOUND: + raise # unrecoverable + if name == MAIN_CONFIG_MAP: + # If main ConfigMap is not found wait until it is available LOGGER.warning("ConfigMap %s/%s doesn't exist, attempting again", namespace, name) - continue - raise # unrecoverable + else: + # If product ConfigMap is not available then create + LOGGER.info("Product ConfigMap %s/%s doesn't exist, attempting to create", namespace, name) + if not create_config_map(api_instance, name, namespace): + raise # unrecoverable + continue # Determine if ConfigMap needs to be updated config_map_data = response.data or {} # if no ConfigMap data exists @@ -243,13 +268,16 @@ def update_config_map(data: dict, name, namespace): else: LOGGER.exception("Error calling replace_namespaced_config_map") + if attempt == retries: + LOGGER.error("Exceeded number of attempts; Not updating ConfigMap %s/%s.", namespace, name) + def main(): """ Main function """ configure_logging() LOGGER.info( "Updating ConfigMap=%s in namespace=%s for product/version=%s/%s", - CONFIG_MAP, CONFIG_MAP_NAMESPACE, PRODUCT, PRODUCT_VERSION + MAIN_CONFIG_MAP, CONFIG_MAP_NAMESPACE, PRODUCT, PRODUCT_VERSION ) if SET_ACTIVE_VERSION and REMOVE_ACTIVE_FIELD: @@ -283,7 +311,20 @@ def main(): if VALIDATE_SCHEMA: validate_schema(data) - update_config_map(data, CONFIG_MAP, CONFIG_MAP_NAMESPACE) + product_config_map = format_product_cm_name(MAIN_CONFIG_MAP, PRODUCT) + + LOGGER.debug("Splitting cray-product-catalog data") + main_cm_data, prod_cm_data = split_catalog_data(data) + + if prod_cm_data and product_config_map == '': + LOGGER.error("Not updating ConfigMaps because the provided product name is invalid: '%s'", PRODUCT) + raise SystemExit(1) + + update_config_map(main_cm_data, MAIN_CONFIG_MAP, CONFIG_MAP_NAMESPACE) + + # If product_config_map is not an empty string and prod_cm_data is not an empty dict + if prod_cm_data: + update_config_map(prod_cm_data, product_config_map, CONFIG_MAP_NAMESPACE) if __name__ == "__main__": diff --git a/cray_product_catalog/constants.py b/cray_product_catalog/constants.py index 41db09cd..01b5a978 100644 --- a/cray_product_catalog/constants.py +++ b/cray_product_catalog/constants.py @@ -33,3 +33,11 @@ COMPONENT_HELM = 'helm' COMPONENT_S3 = 's3' COMPONENT_MANIFESTS = 'manifests' +CONFIG_MAP_FIELDS = {'configuration', 'images', 'recipes'} +PRODUCT_CM_FIELDS = {'component_versions'} +PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY = 'type' +PRODUCT_CATALOG_CONFIG_MAP_LABEL = {PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY: PRODUCT_CATALOG_CONFIG_MAP_NAME} +PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR = "{0}={1}".format( + PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY, PRODUCT_CATALOG_CONFIG_MAP_NAME + ) +PRODUCT_CATALOG_CONFIG_MAP_REPLICA = 'cray-product-catalog-temp' diff --git a/cray_product_catalog/migration/__init__.py b/cray_product_catalog/migration/__init__.py new file mode 100644 index 00000000..7717399c --- /dev/null +++ b/cray_product_catalog/migration/__init__.py @@ -0,0 +1,46 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +""" +File defines few constants +""" + +import os +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR +from re import compile + +# ConfigMap name for temporary many config map +CONFIG_MAP_TEMP = "cray-product-catalog-temp" + +# namespace for ConfigMaps +PRODUCT_CATALOG_CONFIG_MAP_NAME = os.environ.get("CONFIG_MAP_NAME", "cray-product-catalog").strip() +PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE = os.environ.get("CONFIG_MAP_NAMESPACE", "services").strip() + +# config map names +CRAY_DATA_CATALOG_LABEL = PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR + +# product ConfigMap pattern +PRODUCT_CONFIG_MAP_PATTERN = compile('^(cray-product-catalog)-([a-z0-9.-]+)$') +RESOURCE_VERSION = 'resource_version' + +retry_count = 10 +role_name = 'cray-product-catalog' +action = 'update' diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py new file mode 100644 index 00000000..bc637d7b --- /dev/null +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller +ConfigMaps with each product's `component_versions` data in its own Product ConfigMap +""" + +import logging +import yaml + +from kubernetes.client.rest import ApiException + +from cray_product_catalog.logging import configure_logging +from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name +from cray_product_catalog.migration.kube_apis import KubernetesApi +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL +from cray_product_catalog.migration import ( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, action +) + +LOGGER = logging.getLogger(__name__) + + +class ConfigMapDataHandler: + """ Class to migrate ConfigMap data to multiple ConfigMaps """ + + def __init__(self) -> None: + self.k8s_obj = KubernetesApi() + self.config_map_data_replica = {} + configure_logging() + + def create_product_config_maps(self, product_config_map_data_list): + """Create new product ConfigMap for each product in product_config_map_data_list + + Args: + product_config_map_data_list (list): list of data to be stored in each product ConfigMap + """ + for product_data in product_config_map_data_list: + product_name = list(product_data.keys())[0] + LOGGER.debug("Creating ConfigMap for product %s", product_name) + prod_cm_name = format_product_cm_name(PRODUCT_CATALOG_CONFIG_MAP_NAME, product_name) + + if not self.k8s_obj.create_config_map(prod_cm_name, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, product_data, + PRODUCT_CATALOG_CONFIG_MAP_LABEL): + LOGGER.info("Failed to create product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + prod_cm_name) + return False + LOGGER.info("Created product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, prod_cm_name) + return True + + def create_temp_config_map(self, config_map_data): + """Create temporary main ConfigMap `cray-product-catalog-temp` + + Args: + config_map_data (dict): Data to be stored in the ConfigMap `cray-product-catalog-temp` + """ + + if self.k8s_obj.create_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + config_map_data, PRODUCT_CATALOG_CONFIG_MAP_LABEL): + LOGGER.info("Created temp ConfigMap %s/%s", + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) + return True + LOGGER.error("Creating ConfigMap %s/%s failed", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) + return False + + def migrate_config_map_data(self, config_map_data): + """Migrate cray-product-catalog ConfigMap data to multiple product ConfigMaps with + `component_versions` data for each product + + Returns: + {Dictionary, List}: Main ConfigMap Data, list of product ConfigMap data + """ + + LOGGER.info( + "Migrating data in ConfigMap=%s in namespace=%s to multiple ConfigMaps", + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + # Backed up ConfigMap Data + self.config_map_data_replica = config_map_data + # Get list of products + products_list = list(config_map_data.keys()) + product_config_map_data_list = [] + for product in products_list: + product_data = yaml.safe_load(config_map_data[product]) + # Get list of versions associated with product + product_versions_list = list(product_data.keys()) + product_versions_data = {} + main_versions_data = {} + for version_data in product_versions_list: + LOGGER.debug("Splitting cray-product-catalog data for product %s", product) + main_cm_data, prod_cm_data = split_catalog_data(product_data[version_data]) + # prod_cm_data is not an empty dictionary + if prod_cm_data: + product_config_map_data = {} + product_versions_data[version_data] = prod_cm_data + # main_cm_data is not an empty dictionary + if main_cm_data: + main_versions_data[version_data] = main_cm_data + # If `component_versions` data exists for a product, create new product config map + if product_versions_data: + product_config_map_data[product] = yaml.safe_dump(product_versions_data, default_flow_style=False) + # create_product_config_map(k8s_obj, product, product_config_map_data) + product_config_map_data_list.append(product_config_map_data) + # Data with key other than `component_versions` should be updated to config_map_data, + # so that new main ConfigMap will not have data with key `component_versions` + if main_versions_data: + config_map_data[product] = yaml.safe_dump(main_versions_data, default_flow_style=False) + else: + config_map_data[product] = '' + return config_map_data, product_config_map_data_list + + def rename_config_map(self, rename_from, rename_to, namespace, label): + """ Renaming is actually deleting one Config Map and then updating the name of other Config Map and patch it. + :param str rename_from: Name of Config Map to rename + :param str rename_to: Name of Config Map to be renamed to + :param str namespace: namespace in which Config Map has to be updated + :param dict label: label of config map to be renamed + :return: bool, If Success True else False + """ + + if not self.k8s_obj.delete_config_map(rename_to, namespace): + logging.error("Failed to delete ConfigMap %s", rename_to) + return False + attempt = 0 + del_failed = False + + while attempt < 10: + attempt += 1 + response = self.k8s_obj.read_config_map(rename_from, namespace) + if not response: + LOGGER.error("Failed to read ConfigMap %s, retrying..", rename_from) + continue + cm_data = response.data + + if self.k8s_obj.create_config_map(rename_to, namespace, cm_data, label): + if self.k8s_obj.delete_config_map(rename_from, namespace): + LOGGER.info("Renaming ConfigMap successful") + return True + else: + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) + del_failed = True + break + else: + LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) + continue + # Since only delete of backed up ConfigMap failed, retrying only delete operation + attempt = 0 + if del_failed: + while attempt < 10: + attempt += 1 + if self.k8s_obj.delete_config_map(rename_from, namespace): + break + else: + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) + continue + # Returning success as migration is successful only backed up ConfigMap is not deleted. + LOGGER.info("Renaming ConfigMap successful") + return True + return False diff --git a/cray_product_catalog/migration/exit_handler.py b/cray_product_catalog/migration/exit_handler.py new file mode 100644 index 00000000..2f997611 --- /dev/null +++ b/cray_product_catalog/migration/exit_handler.py @@ -0,0 +1,97 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +""" +File contains logic to handle exit scenarios: +a. graceful +b. non-graceful +c. Rollback case +""" + +import logging +from typing import List +from re import fullmatch + + +from cray_product_catalog.migration import CONFIG_MAP_TEMP, CRAY_DATA_CATALOG_LABEL, \ + PRODUCT_CONFIG_MAP_PATTERN +from cray_product_catalog.migration import PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE +from cray_product_catalog.migration.kube_apis import KubernetesApi + +LOGGER = logging.getLogger(__name__) + + +def _is_product_config_map(config_map_name: str) -> bool: + """Function to check product config map pattern + returns True if pattern match found + """ + if config_map_name is None or config_map_name == "": + return False + elif fullmatch(PRODUCT_CONFIG_MAP_PATTERN, config_map_name): + return True + else: + return False + + +class ExitHandler: + """Class to handle exit and rollback classes""" + + def __init__(self): + self.k8api = KubernetesApi() # kubernetes API object + + @staticmethod + def graceful_exit() -> None: + LOGGER.info("Migration not possible, no exception occurred.") + + @staticmethod + def exception_exit() -> None: + LOGGER.error("Migration not possible, exception occurred.") + + def __get_all_created_product_config_maps(self) -> List: + """Get all created product config maps""" + cm_name = filter(_is_product_config_map, + self.k8api.list_config_map_names( + label=CRAY_DATA_CATALOG_LABEL, + namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + ) + return list(cm_name) + + def rollback(self): + """Method to handle roll back + 1. Deleting temp config map + 2. Deleting all created product config map + """ + LOGGER.warning("Initiating rollback") + product_config_maps = self.__get_all_created_product_config_maps() # collecting product config map + + LOGGER.info("deleting Product ConfigMaps") # attempting to delete product config maps + non_deleted_product_config_maps = [] + for config_map in product_config_maps: + LOGGER.debug("deleting Product ConfigMap %s", config_map) + if not self.k8api.delete_config_map(name=config_map, namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE): + non_deleted_product_config_maps.append(config_map) + + if len(non_deleted_product_config_maps) > 0: # checking if any product config map is not deleted + LOGGER.error("Error in deleting ConfigMap/s %s. Delete this/these manually", + non_deleted_product_config_maps) + return + + LOGGER.info("rollback successful") diff --git a/cray_product_catalog/migration/kube_apis.py b/cray_product_catalog/migration/kube_apis.py new file mode 100644 index 00000000..1e83753e --- /dev/null +++ b/cray_product_catalog/migration/kube_apis.py @@ -0,0 +1,168 @@ +# +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Kubernets API +""" + +import logging +from kubernetes import client +from kubernetes.client.rest import ApiException +from kubernetes.client.api_client import ApiClient +from kubernetes.client.models.v1_config_map import V1ConfigMap +from kubernetes.client.models.v1_object_meta import V1ObjectMeta +from urllib3.util.retry import Retry +from urllib3.exceptions import MaxRetryError +from cray_product_catalog.logging import configure_logging +from cray_product_catalog.util.k8s import load_k8s +from . import retry_count +# retry_count=10 + +# from kubernetes import configs +# def load_k8s(): +# """ Load Kubernetes configuration """ +# try: +# config.load_incluster_config() +# except Exception: +# config.load_kube_config() + + +class KubernetesApi: + """Class for wrapping Kubernetes api""" + def __init__(self): + configure_logging() + self.logger = logging.getLogger(__name__) + load_k8s() + + retry = Retry( + total=retry_count, read=retry_count, connect=retry_count, backoff_factor=0.3, + status_forcelist=(500, 502, 503, 504) + ) + self.kclient = ApiClient() + self.kclient.rest_client.pool_manager.connection_pool_kw['retries'] = retry + self.api_instance = client.CoreV1Api(self.kclient) + + def create_config_map(self, name, namespace, data, label): + """Creates Config Map + :param dict data: Content of configmap + :param str name: config map name to be created + :param str namespace: namespace in which configmap has to be created + :param dict label: label with which configmap has to be created + :return: bool + """ + try: + cm_body = V1ConfigMap( + metadata=V1ObjectMeta( + name=name, + labels=label + ), + data=data + ) + self.api_instance.create_namespaced_config_map( + namespace=namespace, body=cm_body + ) + return True + except MaxRetryError as err: + self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + return False + except ApiException as err: + self.logger.exception('ApiException- Error:{0}'.format(err)) + return False + + def list_config_map(self, namespace, label): + """ Reads all the Config Map with certain label in particular namespace + :param str namespace: Value of namespace from where config map has to be listed + :param str label: string format of label "type=xyz" + :return: V1ConfigMapList + If there is any error, returns None + """ + if not all((label, namespace)): + self.logger.info("Either label or namespace is empty, not reading config map.") + return None + try: + return self.api_instance.list_namespaced_config_map(namespace, label_selector=label).items + except MaxRetryError as err: + self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + return None + except ApiException as err: + self.logger.exception('ApiException- Error:{0}'.format(err)) + return None + + def list_config_map_names(self, namespace, label): + """ Reads all the Config Map with certain label in particular namespace + :param str namespace: Value of namespace from where config map has to be listed + :param str label: string format of label "type=xyz" + :return: [str] + """ + cm_output = self.list_config_map(namespace, label) + + list_cm_names = [] + + if not cm_output: + return list_cm_names + + # parse the output to get only names + for cm in cm_output: + try: + list_cm_names.append(cm.metadata.name) + except Exception: + continue + + return list_cm_names + + def read_config_map(self, name, namespace): + """Reads config Map based on provided name and namespace + :param Str name: name of ConfigMap to read + :param Str namespace: namespace from which Config Map has to be read + :return: V1ConfigMap + Returns None in case of any error + """ + # Check if both values are not empty + if not all((name, namespace)): + self.logger.exception("Either name or namespace is empty, not reading config map.") + return None + try: + return self.api_instance.read_namespaced_config_map(name, namespace) + except MaxRetryError as err: + self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + return None + except ApiException as err: + self.logger.exception('ApiException- Error:{0}'.format(err)) + return None + + def delete_config_map(self, name, namespace): + """Delete the Config Map + :param Str name: name of ConfigMap to be deleted + :param Str namespace: namespace from which Config Map has to be deleted + :return: bool; If success True else False + """ + try: + self.api_instance.delete_namespaced_config_map(name, namespace) + return True + except MaxRetryError as err: + self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + return False + except ApiException as err: + self.logger.exception('ApiException- Error:{0}'.format(err)) + return False diff --git a/cray_product_catalog/migration/main.py b/cray_product_catalog/migration/main.py new file mode 100644 index 00000000..0575a982 --- /dev/null +++ b/cray_product_catalog/migration/main.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +# +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +import logging + +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL +from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler +from cray_product_catalog.migration import ( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + CONFIG_MAP_TEMP, RESOURCE_VERSION +) +from cray_product_catalog.migration.exit_handler import ExitHandler + +LOGGER = logging.getLogger(__name__) + + +def main(): + """Main function""" + LOGGER.info("Migrating %s ConfigMap data to multiple product ConfigMaps", PRODUCT_CATALOG_CONFIG_MAP_NAME) + config_map_obj = ConfigMapDataHandler() + exit_handler = ExitHandler() + attempt = 0 + max_attempts = 2 + + while (attempt < max_attempts): + attempt += 1 + migration_failed = False + init_resource_version = '' + curr_resource_version = '' + response = config_map_obj.k8s_obj.read_config_map( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + if response: + if not response.metadata.resource_version: + LOGGER.error("Error reading resourceVersion, exiting migration process...") + else: + init_resource_version = response.metadata.resource_version + if not response.data: + LOGGER.error("Error reading ConfigMap data, exiting migration process...") + else: + config_map_data = response.data + else: + LOGGER.info("Error reading ConfigMap, exiting migration process...") + raise SystemExit(1) + + try: + main_config_map_data, product_config_map_data_list = config_map_obj.migrate_config_map_data(config_map_data) + except Exception: + LOGGER.error("Failed to split ConfigMap Data, exiting migration process...") + raise SystemExit(1) + + # Create ConfigMaps for each product with `component_versions` data + if not config_map_obj.create_product_config_maps(product_config_map_data_list): + LOGGER.info("Calling rollback handler...") + exit_handler.rollback() + raise SystemExit(1) + # Create temporary main ConfigMap with all data except `component_versions` for all products + if not config_map_obj.create_temp_config_map(main_config_map_data): + LOGGER.info("Calling rollback handler...") + exit_handler.rollback() + raise SystemExit(1) + + LOGGER.info("Verifying resource_version value is same to confirm there is no change in %s", + PRODUCT_CATALOG_CONFIG_MAP_NAME) + + response = config_map_obj.k8s_obj.read_config_map( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + if response: + if not response.metadata.resource_version: + LOGGER.error("Error reading resourceVersion, exiting migration process...") + exit_handler.rollback() + raise SystemExit(1) + else: + curr_resource_version = response.metadata.resource_version + + if curr_resource_version != init_resource_version: + migration_failed = True + LOGGER.info("resource_version has changed, so cannot rename %s ConfigMap to %s ConfigMap", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + LOGGER.info("Re-trying migration process...") + exit_handler.rollback() + continue + else: + break + + if migration_failed: + LOGGER.info("ConfigMap %s is modified by other process, exiting migration process...", + PRODUCT_CATALOG_CONFIG_MAP_NAME) + raise SystemExit(1) + + LOGGER.info("Renaming %s ConfigMap name to %s ConfigMap", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + + # Creating main ConfigMap `cray-product-catalog` using the data in `cray-product-catalog-temp` + if config_map_obj.rename_config_map( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ): + LOGGER.info("Migration successful") + else: + LOGGER.info("Renaming %s to %s ConfigMap failed, calling rollback handler...", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + exit_handler.rollback() + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/cray_product_catalog/migration/rollback_handler.py b/cray_product_catalog/migration/rollback_handler.py new file mode 100644 index 00000000..e69de29b diff --git a/cray_product_catalog/query.py b/cray_product_catalog/query.py index 4988f1d3..af43daa7 100644 --- a/cray_product_catalog/query.py +++ b/cray_product_catalog/query.py @@ -43,9 +43,11 @@ COMPONENT_VERSIONS_PRODUCT_MAP_KEY, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR ) from cray_product_catalog.schema.validate import validate from cray_product_catalog.util import load_k8s +from cray_product_catalog.util.merge_dict import merge_dict LOGGER = logging.getLogger(__name__) @@ -94,28 +96,26 @@ def __init__(self, name=PRODUCT_CATALOG_CONFIG_MAP_NAME, namespace=PRODUCT_CATAL self.namespace = namespace self.k8s_client = self._get_k8s_api() try: - config_map = self.k8s_client.read_namespaced_config_map(name, namespace) + configmaps = self.k8s_client.list_namespaced_config_map( + namespace, label_selector=PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR + ).items except MaxRetryError as err: raise ProductCatalogError( - f'Unable to connect to Kubernetes to read {namespace}/{name} ConfigMap: {err}' + f'Unable to connect to Kubernetes to read {namespace} namespace: {err}' ) from err except ApiException as err: # The full string representation of ApiException is very long, so just log err.reason. raise ProductCatalogError( - f'Error reading {namespace}/{name} ConfigMap: {err.reason}' + f'Error listing ConfigMaps in {namespace} namespace: {err.reason}' ) from err - if config_map.data is None: + if len(configmaps) == 0: raise ProductCatalogError( - f'No data found in {namespace}/{name} ConfigMap.' + f'No ConfigMaps found in {namespace} namespace.' ) try: - self.products = [ - InstalledProductVersion(product_name, product_version, product_version_data) - for product_name, product_versions in config_map.data.items() - for product_version, product_version_data in safe_load(product_versions).items() - ] + self.products = load_config_map_data(self.name, configmaps) except YAMLError as err: raise ProductCatalogError( f'Failed to load ConfigMap data: {err}' @@ -125,7 +125,7 @@ def __init__(self, name=PRODUCT_CATALOG_CONFIG_MAP_NAME, namespace=PRODUCT_CATAL str(p) for p in self.products if not p.is_valid ] if invalid_products: - LOGGER.debug( + LOGGER.warning( 'The following products have product catalog data that is not valid against the expected schema: %s', ", ".join(invalid_products) ) @@ -174,6 +174,32 @@ def get_product(self, name, version=None): return matching_products[0] +def load_config_map_data(name, configmaps): + """Parse list_namespaced_config_map output and get array of InstalledProductVersion objects. + + Args: + configmaps (V1ConfigMapList): list of ConfigMap objects. + + Returns: + An array of InstalledProductVersion objects. + """ + config_map_data = {} + for cm in configmaps: + if not cm.metadata.name.startswith(name): + continue + for product_name, product_versions in cm.data.items(): + for product_version, product_version_data in safe_load(product_versions).items(): + cm_key = product_name + ':' + product_version + if cm_key in config_map_data: + config_map_data[cm_key] = merge_dict(config_map_data[cm_key], product_version_data) + else: + config_map_data[cm_key] = product_version_data + return [ + InstalledProductVersion(key.split(':',)[0], key.split(':')[1], product_version_data) + for key, product_version_data in config_map_data.items() + ] + + class InstalledProductVersion: """A representation of a version of a product that is currently installed. @@ -218,10 +244,11 @@ def docker_images(self): for component in self.component_data.get(COMPONENT_DOCKER_KEY) or []] @property - def helm_charts(self): + def helm(self): """Get Helm charts associated with this InstalledProductVersion. + Returns: - A list of tuples of (chart name, chart version) + A list of tuples of (chart_name, chart_version) """ return [(component['name'], component['version']) for component in self.component_data.get(COMPONENT_HELM) or []] @@ -229,6 +256,7 @@ def helm_charts(self): @property def s3_artifacts(self): """Get S3 artifacts associated with this InstalledProductVersion. + Returns: A list of tuples of (artifact bucket, artifact key) """ @@ -238,6 +266,7 @@ def s3_artifacts(self): @property def loftsman_manifests(self): """Get Loftsman manifests associated with this InstalledProductVersion. + Returns: A list of manifests """ diff --git a/cray_product_catalog/schema/schema.yaml b/cray_product_catalog/schema/schema.yaml index a073ac13..6b3dd5b4 100644 --- a/cray_product_catalog/schema/schema.yaml +++ b/cray_product_catalog/schema/schema.yaml @@ -176,4 +176,4 @@ definitions: items: description: Loftsman manifest artifact. type: string - minLength: 1 \ No newline at end of file + minLength: 1 diff --git a/cray_product_catalog/util/catalog_data_helper.py b/cray_product_catalog/util/catalog_data_helper.py new file mode 100755 index 00000000..672c975c --- /dev/null +++ b/cray_product_catalog/util/catalog_data_helper.py @@ -0,0 +1,68 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +""" +Contains a utility function for splitting `cray-product-catalog` data. +""" + +import re + +from cray_product_catalog.constants import ( + PRODUCT_CM_FIELDS +) + + +def split_catalog_data(data): + """Split the passed data into data needed by main and product ConfigMaps.""" + all_unique_keys = set(data.keys()) + comm_keys_bw_cms = all_unique_keys.intersection(PRODUCT_CM_FIELDS) + + # If none of the PRODUCT_CM_FIELDS are available in all_unique_keys, then + # return empty dict as second return value + if not comm_keys_bw_cms: + return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, {} + return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, \ + {key: data[key] for key in comm_keys_bw_cms} + + +def format_product_cm_name(config_map, product): + """Formatting PRODUCT_CONFIG_NAME based on the product name passed and the same is used as key + under data in the ConfigMap. + The name of a ConfigMap must be a valid DNS subdomain name. In addition, it must obey the following rules: + - contain no more than 253 characters + - contain only lowercase alphanumeric characters, hypens('-'), or periods('.') + - start with an alphanumeric character + - end with an alphanumeric character + The product name, which is a key under the data, must obey the following rule: + - contain only alphanumeric characters, hyphens ('-'), underscores ('_'), and periods ('.') + Because the product name can have uppercase characters and underscores ('_'), which are + prohibited in the ConfigMap name, we convert underscores ('_') to hyphens ('-') and uppercase + to lowercase. + """ + pat = re.compile('^([a-z0-9])*[a-z0-9.-]*([a-z0-9])$') + prod_config_map = config_map + '-' + product.replace('_', '-').lower() + + if len(prod_config_map) > 253: + return '' + if not re.fullmatch(pat, prod_config_map): + return '' + return prod_config_map diff --git a/setup.py b/setup.py index 43638ba6..70a3b169 100644 --- a/setup.py +++ b/setup.py @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2021-2022 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), @@ -60,7 +60,8 @@ entry_points={ 'console_scripts': [ 'catalog_delete=cray_product_catalog.catalog_delete:main', - 'catalog_update=cray_product_catalog.catalog_update:main' + 'catalog_update=cray_product_catalog.catalog_update:main', + 'catalog_migrate=cray_product_catalog.migration.main:main', ] } ) diff --git a/tests/migration/__init__.py b/tests/migration/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/migration/migration_mock.py b/tests/migration/migration_mock.py new file mode 100755 index 00000000..bf3d7a6e --- /dev/null +++ b/tests/migration/migration_mock.py @@ -0,0 +1,130 @@ +# +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Mock data for ConfigMapDataHandler +""" + +INITIAL_MAIN_CM_DATA = { + 'HFP-firmware': """ + 22.10.2: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3 + 23.01.1: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3""", + 'analytics': """ + 1.4.18: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 + import_branch: cray/analytics/1.4.18 + import_date: 2023-02-28 04:37:34.914586 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git + 1.4.20: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd + import_branch: cray/analytics/1.4.20 + import_date: 2023-03-23 16:55:22.295666 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git""" +} + +MAIN_CM_DATA_EXPECTED = { + 'HFP-firmware': '', + 'analytics': """1.4.18: + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 + import_branch: cray/analytics/1.4.18 + import_date: 2023-02-28 04:37:34.914586 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git +1.4.20: + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd + import_branch: cray/analytics/1.4.20 + import_date: 2023-03-23 16:55:22.295666 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git\n""" +} + +PROD_CM_DATA_LIST_EXPECTED = [ + { + 'HFP-firmware': """22.10.2: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3 +23.01.1: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3\n""" + }, + { + 'analytics': """1.4.18: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs +1.4.20: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs\n""" + } +] + + +class MockYaml: + """Mock class created to test test_create_product_catalog_invalid_product_data.""" + + def __init__(self, resource_version): + """Initialize metadata and data object of ConfigMap data.""" + self.metadata = MetaData(resource_version) + self.data = { + 'sat': '\t', + } + + +class MetaData: + """ + Class to provide dummy metadata object with name and resource_version + """ + def __init__(self, resource_version): + """Initialize ConfigMap name and resoource_version""" + self.name = 'cray-product-catalog' + self.resource_version = resource_version diff --git a/tests/migration/test_config_map_data_handler.py b/tests/migration/test_config_map_data_handler.py new file mode 100755 index 00000000..17b698e7 --- /dev/null +++ b/tests/migration/test_config_map_data_handler.py @@ -0,0 +1,677 @@ +# +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Tests for validating ConfigMapDataHandler +""" + +import unittest +from unittest.mock import patch, call, Mock +from typing import Dict, List + +from kubernetes.config import ConfigException +from kubernetes import client +from kubernetes.client.api_client import ApiClient + +from cray_product_catalog.migration.main import main +from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler +from cray_product_catalog.constants import ( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL +) +from cray_product_catalog.util.catalog_data_helper import format_product_cm_name +from cray_product_catalog.migration import CONFIG_MAP_TEMP +from tests.migration.migration_mock import ( + MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED, INITIAL_MAIN_CM_DATA, MockYaml +) + + +def mock_split_catalog_data(): + """Mocking function to return custom data""" + return MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED + + +class TestConfigMapDataHandler(unittest.TestCase): + """ Tests for validating ConfigMapDataHandler """ + + def setUp(self) -> None: + """Set up mocks.""" + self.mock_load_k8s_mig = patch('cray_product_catalog.migration.kube_apis.load_k8s').start() + self.mock_corev1api_mig = patch('cray_product_catalog.migration.kube_apis.client.CoreV1Api').start() + self.mock_ApiClient_mig = patch('cray_product_catalog.migration.kube_apis.ApiClient').start() + self.mock_client_mig = patch('cray_product_catalog.migration.kube_apis.client').start() + + self.mock_k8api_read = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.read_config_map').start() + self.mock_k8api_create = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.create_config_map').start() + self.mock_k8api_delete = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.delete_config_map').start() + + def tearDown(self) -> None: + patch.stopall() + + def test_migrate_config_map_data(self): + """ Validating the migration of data into multiple product ConfigMaps data """ + + main_cm_data: Dict + prod_cm_data_list: List + cmdh = ConfigMapDataHandler() + main_cm_data, prod_cm_data_list = cmdh.migrate_config_map_data(INITIAL_MAIN_CM_DATA) + + self.assertEqual(main_cm_data, MAIN_CM_DATA_EXPECTED) + self.assertEqual(prod_cm_data_list, PROD_CM_DATA_LIST_EXPECTED) + + def test_create_product_config_maps(self): + """ Validating product config maps are created """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice + call( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), + call( + dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), + ] + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + self.assertEqual( + captured.records[1].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") + + def test_create_second_product_config_map_failed(self): + """ Validating scenario where creation of second product ConfigMap failed """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + self.mock_k8api_create.side_effect = [True, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice + call( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call( + dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), + ] + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + self.assertEqual( + captured.records[1].getMessage(), + f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") + + def test_create_first_product_config_map_failed(self): + """ Validating scenario where creation of first product ConfigMap failed. """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + self.mock_k8api_create.side_effect = [False, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_called_once_with( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual(len(captured.records), 1) + + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + def test_create_temp_config_map(self): + """ Validating temp main config map is created """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs(level='DEBUG') as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) + + self.mock_k8api_create.assert_called_once_with( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created temp ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP}") + + def test_create_temp_config_map_failed(self): + """ Validating temp main config map creation failed """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + self.mock_k8api_create.return_value = False + + with self.assertLogs() as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) + + self.mock_k8api_create.assert_called_once_with( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Creating ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP} failed") + + def test_rename_config_map(self): + """ Validating product config maps are created """ + + with self.assertLogs(level="DEBUG") as captured: + # call method under test + self.mock_k8api_delete.return_value = True + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(rename_from=CONFIG_MAP_TEMP, + rename_to=PRODUCT_CATALOG_CONFIG_MAP_NAME, + namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + label=PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ] + ) + + self.mock_k8api_read.assert_called_once_with(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + "Renaming ConfigMap successful") + + def test_rename_config_map_failed_1(self): + """ Validating rename config map failure scenario where: + deleting cray-product-catalog ConfigMap failed. """ + + with self.assertLogs() as captured: + self.mock_k8api_delete.side_effect = [False, False] + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to delete ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}") + + def test_rename_config_map_failed_2(self): + """ Validating rename config map failure scenario where: + creating cray-product-catalog ConfigMap failed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_create.return_value = False + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + calls = [ # Delete ConfigMap called twice + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ] + self.mock_k8api_create.assert_has_calls(calls=[ + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + + ]) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to create ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}, retrying..") + + def test_rename_config_map_failed_3(self): + """ Validating rename config map failure scenario where: + first operation of deleting cray-product-catalog-temp ConfigMap failed but later passed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_k8api_delete.side_effect = [True, False, True] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ]) + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual(2, len(captured.records)) + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") + self.assertEqual( + captured.records[1].getMessage(), + f"Renaming ConfigMap successful") + + def test_rename_config_map_failed_4(self): + """ Validating rename config map failure scenario where: + everytime deleting cray-product-catalog-temp ConfigMap failed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_k8api_delete.side_effect = [True, False, False, False, False, False, False, False, + False, False, False, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ]) + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual(12, len(captured.records)) + self.assertEqual( + captured.records[-2].getMessage(), + f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") + self.assertEqual( + captured.records[-1].getMessage(), + f"Renaming ConfigMap successful") + + def test_main_for_successful_migration(self): + """Validating that migration is successful""" + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = True + + # Call method under test + main() + + self.assertEqual( + captured.records[-1].getMessage(), + f"Migration successful") + + def test_main_failed_1(self): + """Validating that migration failed as renaming failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = False + + # Call method under test + main() + + self.assertTrue( + f"Renaming cray-product-catalog-temp to cray-product-catalog ConfigMap failed, " + f"calling rollback handler..." in captured.exception + ) + + self.assertTrue( + "rollback successful" in captured.exception + ) + + def test_main_failed_2(self): + """Validating that migration failed as create temp cm failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = False + + # Call method under test + main() + + self.assertTrue( + "rollback successful" in captured.exception + ) + + self.mock_rename_cm.assert_not_called() + + def test_main_failed_3(self): + """Validating that migration failed as create product config maps failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = False + + # Call method under test + main() + + self.assertTrue( + "rollback successful" in captured.exception + ) + + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_4(self): + """Validating that migration failed as migrate_config_map failed with exception""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = Exception() + + # Call method under test + main() + + self.assertTrue( + "Failed to split ConfigMap Data, exiting migration process..." in captured.exception + ) + + self.mock_create_prod_cms.assert_not_called() + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_6(self): + """Validating that migration failed as read_config_map returned empty response / data.""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data="") + + # Call method under test + main() + + self.assertTrue( + "Error reading ConfigMap, exiting migration process..." in captured.exception + ) + + self.mock_migrate_config_map.assert_not_called() + self.mock_create_prod_cms.assert_not_called() + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_7(self): + """Validating that migration failed as initial and final resource version is different""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.side_effect = [MockYaml(1), + MockYaml(2), + MockYaml(1), + MockYaml(2)] + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + + # Call method under test + main() + + self.assertTrue( + "Re-trying migration process..." in captured.exception + ) + + self.assertTrue( + "rollback successful" in captured.exception + ) + + self.assertTrue( + f"ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME} is modified, exiting migration process..." + ) + self.mock_rename_cm.assert_not_called() + + def test_main_failed_8(self): + """Validating that migration is successful in second attempt as initial and final resource + version is different in first attempt""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertLogs(level="DEBUG") as captured: + # with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.side_effect = [MockYaml(1), + MockYaml(2), + MockYaml(1), + MockYaml(1)] + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = True + + # Call method under test + main() + # Verify the exact log message + self.assertEqual(10, len(captured.records)) + self.assertEqual( + captured.records[3].getMessage(), + "Re-trying migration process..." + ) + self.assertEqual( + captured.records[6].getMessage(), + "rollback successful" + ) + + self.assertEqual( + captured.records[-1].getMessage(), + "Migration successful" + ) diff --git a/tests/migration/test_exit_handler.py b/tests/migration/test_exit_handler.py new file mode 100644 index 00000000..44aa3458 --- /dev/null +++ b/tests/migration/test_exit_handler.py @@ -0,0 +1,110 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +""" +File contains unit test classes for validating exit handler cases. +""" + +import unittest +from unittest.mock import patch, call + +from cray_product_catalog.migration import CONFIG_MAP_TEMP +from cray_product_catalog.migration.exit_handler import _is_product_config_map, ExitHandler +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + + +class TestExitHandler(unittest.TestCase): + """unittest class for Data catalog ConfigMap deletion logic""" + + def setUp(self) -> None: + self.mock_load_k8s_mig = patch('cray_product_catalog.migration.kube_apis.load_k8s').start() + self.mock_corev1api_mig = patch('cray_product_catalog.migration.kube_apis.client.CoreV1Api').start() + self.mock_ApiClient_mig = patch('cray_product_catalog.migration.kube_apis.ApiClient').start() + + self.mock_k8api_del = patch( + 'cray_product_catalog.migration.exit_handler.KubernetesApi.delete_config_map').start() + self.mock_k8api_list = patch( + 'cray_product_catalog.migration.exit_handler.KubernetesApi.list_config_map_names').start() + + def tearDown(self) -> None: + patch.stopall() + + def test_product_config_map_pattern(self): + """Test cases for checking all valid patterns of product config map""" + base_str = "cray-product-catalog" + valid_patterns = ( + f"{base_str}-cos", + f"{base_str}-90-lojp", + f"{base_str}-cos.89-1234jk", + ) + + for valid_pattern in valid_patterns: + self.assertTrue(_is_product_config_map(valid_pattern)) + + def test_product_config_map_invalid_pattern(self): + """Test cases for checking all invalid patterns of product config map""" + base_str = "cray-product-catalog" + invalid_patterns = ( + f"{base_str}", + f"90-lojp", + f"cos2.3.45.x86" + ) + + for invalid_pattern in invalid_patterns: + self.assertFalse(_is_product_config_map(invalid_pattern)) + + def test_rollback_failure_from_product_config_map_deletion(self): + """Validating the scenario where one of the product config map is not deleted""" + + with self.assertLogs() as captured: + self.mock_k8api_del.side_effect = [True, False] # delete is called two times + + dummy_products = ["cray-product-catalog-cos", "cray-product-catalog-sma"] + self.mock_k8api_list.return_value = dummy_products + eh = ExitHandler() + eh.rollback() + # Verify the exact log message from last return + self.assertEqual(captured.records[-1].getMessage(), f"Error in deleting ConfigMap/s " + f"{[dummy_products[-1]]}. Delete this/these manually") + + def test_rollback_all_success(self): + """Validating the scenario of successful rollback""" + + with self.assertLogs() as captured: + self.mock_k8api_del.return_value = True # delete is called three times + + dummy_products = ["cray-product-catalog-cos", "cray-product-catalog-sma"] + self.mock_k8api_list.return_value = dummy_products + eh = ExitHandler() + eh.rollback() + # Verify the exact log message from last return + self.assertEqual(captured.records[-1].getMessage(), "rollback successful") + + # three calls in sequence for complete flow + self.mock_k8api_del.assert_has_calls(calls=[ + call( + name=dummy_products[0], + namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + name=dummy_products[1], + namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ] + ) diff --git a/tests/mock_update_catalog.py b/tests/mock_update_catalog.py new file mode 100755 index 00000000..ba641af5 --- /dev/null +++ b/tests/mock_update_catalog.py @@ -0,0 +1,110 @@ +# MIT License +# +# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Mock data for catalog_update unit tests +""" + +from kubernetes.client.rest import ApiException +import os +from unittest import mock +from tests.mocks import COS_VERSIONS, Name + +# Mocking environment variables before import so that: +# 1. import (create_config_map, update_config_map and main) is successful +# 2. Additionally they will be used in testcase to verify the tests. +mock.patch.dict( + os.environ, { + 'PRODUCT': 'sat', + 'PRODUCT_VERSION': '1.0.0', + 'YAML_CONTENT_STRING': 'Test data', + 'CONFIG_MAP_NAMESPACE': 'myNamespace' + } +).start() + +UPDATE_DATA = { + '2.0.0': { + 'component_versions': { + 'docker': [ + {'name': 'cray/cray-cos', 'version': '1.0.0'}, + {'name': 'cray/cos-cfs-install', 'version': '1.4.0'} + ] + } + } +} + +ERR_NOT_FOUND = 404 + + +class Response: + """ + Class to generate response for k8s api call api_instance.read_namespaced_config_map(name, namespace) + """ + def __init__(self): + self.data = COS_VERSIONS + self.metadata = Name() + + +class ApiException(ApiException): + """ + Custom Exception to define status + """ + def __init__(self): + self.status = ERR_NOT_FOUND + + +class ApiInstance(): + """ + Class to raise custom exception and ignore function calls + """ + def __init__(self, raise_exception=False): + self.raise_exception = raise_exception + self.count = 0 + + def create_namespaced_config_map(self, namespace='a', body='b'): + """ + Dummy Function to raise exception, if needed + """ + if self.raise_exception: + raise ApiException() + else: + pass + + def read_namespaced_config_map(self, name, namespace): + """ + Dummy Function to : + 1. Raise exception + 2. generate and return proper response with data and metadata + """ + # if this is called for first time return exception, so that product cm is created. + if self.count == 0: + self.count += 1 + raise ApiException() + else: + return Response() + + def patch_namespaced_config_map(self, name, namespace, body='xxx'): + """ + Dummy function to handle the call in code, does nothing + """ + pass diff --git a/tests/mocks.py b/tests/mocks.py index 85566b26..f9d34f28 100644 --- a/tests/mocks.py +++ b/tests/mocks.py @@ -26,7 +26,9 @@ """ from yaml import safe_dump +import datetime +from cray_product_catalog.query import InstalledProductVersion # Two versions of a product named SAT where: # - The two versions have have no docker images in common with one another. @@ -74,10 +76,8 @@ # Two versions of a product named COS where: # - The two versions have one docker image name and version in common -# - The first version has docker images and manifests but not helm charts, repositories, configuration, -# images, or recipes -# - The second version has docker images, helm charts, repositories, configuration, images, and recipes, -# but not manifests +# - The first version has docker and manifests but not helm charts, repositories, configuration, images, or recipes +# - The second version has docker, helm charts, repositories, configuration, images, and recipes, but not manifests COS_VERSIONS = { '2.0.0': { 'component_versions': { @@ -209,3 +209,124 @@ 'cpe': safe_dump(CPE_VERSION), 'other_product': safe_dump(OTHER_PRODUCT_VERSION) } + +# A mock version of the data returned after loading the ConfigMap data +MOCK_PRODUCTS = \ + [InstalledProductVersion('sat', version, SAT_VERSIONS.get(version)) for version in SAT_VERSIONS.keys()] + \ + [InstalledProductVersion('cos', version, COS_VERSIONS.get(version)) for version in COS_VERSIONS.keys()] + \ + [InstalledProductVersion('cpe', version, CPE_VERSION.get(version)) for version in CPE_VERSION.keys()] + \ + [InstalledProductVersion('other_product', version, OTHER_PRODUCT_VERSION.get(version)) + for version in OTHER_PRODUCT_VERSION.keys()] + + +class MockInvalidYaml: + """Mock class created to test test_create_product_catalog_invalid_product_data.""" + + def __init__(self): + """Initialize metadata and data object of ConfigMap data.""" + self.metadata = Name() + self.data = { + 'sat': '\t', + } + + +class Name: + """ + Class to provide dummy metadata object with name and resource_version + """ + def __init__(self): + """Initialize ConfigMap name and resoource_version""" + self.name = 'cray-product-catalog' + self.resource_version = 1 + + +# Helper variables for catalog_data_helper: Start +YAML_DATA = """ + active: false + component_versions: + docker: + - name: artifactory.algol60.net/uan-docker/stable/cray-uan-config + version: 1.11.1 + - name: artifactory.algol60.net/csm-docker/stable/cray-product-catalog-update + version: 1.3.2 + helm: + - name: cray-uan-install + version: 1.11.1 + repositories: + - members: + - uan-2.6.0-sle-15sp4 + name: uan-2.6-sle-15sp4 + type: group + manifests: + - config-data/argo/loftsman/uan/2.6.0-rc.1/manifests/uan.yaml + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/uan-config-management.git + commit: 6a5f52dfbfe7ea1a5f8ea5079c50995112c17025 + import_branch: cray/uan/2.6.0-rc.1-3-gcc65df9 + import_date: 2023-04-12 14:31:40.364230 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/uan-config-management.git + images: + cray-application-sles15sp4.x86_64-0.5.19: + id: 8159f93f-7e18-4875-a8a8-b0fb83c48f07""" + +YAML_DATA_MISSING_PROD_CM_DATA = """ + active: false + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/uan-config-management.git + commit: 6a5f52dfbfe7ea1a5f8ea5079c50995112c17025 + import_branch: cray/uan/2.6.0-rc.1-3-gcc65df9 + import_date: 2023-04-12 14:31:40.364230 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/uan-config-management.git + images: + cray-application-sles15sp4.x86_64-0.5.19: + id: 8159f93f-7e18-4875-a8a8-b0fb83c48f07""" + +YAML_DATA_MISSING_MAIN_DATA = """ + component_versions: + docker: + - name: artifactory.algol60.net/uan-docker/stable/cray-uan-config + version: 1.11.1 + - name: artifactory.algol60.net/csm-docker/stable/cray-product-catalog-update + version: 1.3.2 + helm: + - name: cray-uan-install + version: 1.11.1 + repositories: + - members: + - uan-2.6.0-sle-15sp4 + name: uan-2.6-sle-15sp4 + type: group + manifests: + - config-data/argo/loftsman/uan/2.6.0-rc.1/manifests/uan.yaml""" + +MAIN_CM_DATA = { + 'active': False, + 'configuration': + { + 'clone_url': 'https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/uan-config-management.git', + 'commit': '6a5f52dfbfe7ea1a5f8ea5079c50995112c17025', + 'import_branch': 'cray/uan/2.6.0-rc.1-3-gcc65df9', + 'import_date': datetime.datetime(2023, 4, 12, 14, 31, 40, 364230), + 'ssh_url': 'git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/uan-config-management.git', + 'images': {'cray-application-sles15sp4.x86_64-0.5.19': {'id': '8159f93f-7e18-4875-a8a8-b0fb83c48f07'}} + } +} + +PROD_CM_DATA = { + 'component_versions': + { + 'docker': [ + {'name': 'artifactory.algol60.net/uan-docker/stable/cray-uan-config', 'version': '1.11.1'}, + {'name': 'artifactory.algol60.net/csm-docker/stable/cray-product-catalog-update', 'version': '1.3.2'} + ], + 'helm': [ + {'name': 'cray-uan-install', 'version': '1.11.1'} + ], + 'repositories': [ + {'members': ['uan-2.6.0-sle-15sp4'], 'name': 'uan-2.6-sle-15sp4', 'type': 'group'} + ], + 'manifests': ['config-data/argo/loftsman/uan/2.6.0-rc.1/manifests/uan.yaml'] + } +} + +# Helper variables for catalog_data_helper: End diff --git a/tests/test_catalog_delete.py b/tests/test_catalog_delete.py new file mode 100644 index 00000000..64ef6695 --- /dev/null +++ b/tests/test_catalog_delete.py @@ -0,0 +1,141 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +""" +File contains unit test classes for validating ConfigMap deletion logic. +Deleting keys/Product or a specific version of a product from ConfigMap +""" +import unittest +from unittest.mock import patch, call + +from cray_product_catalog.catalog_delete import ModifyConfigMapUtil + + +class TestModifyConfigMapUtil(unittest.TestCase): + """unittest class for Data catalog ConfigMap deletion logic""" + + def setUp(self) -> None: + self.mock_modify_config_map = patch('cray_product_catalog.catalog_delete.modify_config_map').start() + + self.modify_config_map_util = ModifyConfigMapUtil() + self.modify_config_map_util.main_cm = "main_cm" + self.modify_config_map_util.product_cm = "product_cm" + self.modify_config_map_util.cm_namespace = "cm_namespace" + self.modify_config_map_util.product_name = "product_name" + self.modify_config_map_util.product_version = "product_version" + self.modify_config_map_util.max_retries_for_main_cm = 100 + self.modify_config_map_util.max_retries_for_prod_cm = 10 + self.modify_config_map_util.key = "key" + self.modify_config_map_util.main_cm_fields = ["main_a", "main_b", "main_c"] + self.modify_config_map_util.product_cm_fields = ["prod_1", "prod_2", "prod_3"] + + def tearDown(self) -> None: + patch.stopall() + + def test_object_properties(self): + """Test cases for checking objects properties are not wrongly arranged""" + mcmu = ModifyConfigMapUtil() + mcmu.main_cm = "1" + mcmu.product_cm = "2" + mcmu.cm_namespace = "3" + mcmu.product_name = "4" + mcmu.product_version = "5" + mcmu.max_retries_for_main_cm = "6" + mcmu.max_retries_for_prod_cm = "6.1" + mcmu.key = "7" + mcmu.main_cm_fields = ["8", "9", "10"] + mcmu.product_cm_fields = ["11", "12", "13"] + + self.assertEqual(mcmu.main_cm, "1") + self.assertEqual(mcmu.product_cm, "2") + self.assertEqual(mcmu.cm_namespace, "3") + self.assertEqual(mcmu.product_name, "4") + self.assertEqual(mcmu.product_version, "5") + self.assertEqual(mcmu.max_retries_for_main_cm, "6") + self.assertEqual(mcmu.max_retries_for_prod_cm, "6.1") + self.assertEqual(mcmu.key, "7") + self.assertEqual(mcmu.main_cm_fields, ["8", "9", "10"]) + self.assertEqual(mcmu.product_cm_fields, ["11", "12", "13"]) + + del mcmu + + def test_delete_from_both_config_map(self): + """Test cases to assert delete calls into both main and product ConfigMap""" + self.modify_config_map_util.key = None + self.modify_config_map_util.modify() + + self.mock_modify_config_map.assert_has_calls( + calls=[ + # main ConfigMap call + call(self.modify_config_map_util.main_cm, + self.modify_config_map_util.cm_namespace, + self.modify_config_map_util.product_name, + self.modify_config_map_util.product_version, + self.modify_config_map_util.key, + self.modify_config_map_util.max_retries_for_main_cm, + ), + # product ConfigMap call + call(self.modify_config_map_util.product_cm, + self.modify_config_map_util.cm_namespace, + self.modify_config_map_util.product_name, + self.modify_config_map_util.product_version, + self.modify_config_map_util.key, + self.modify_config_map_util.max_retries_for_prod_cm, + )] + ) + + def test_delete_from_main_config_map(self): + """Test cases to assert delete calls into main ConfigMap""" + self.modify_config_map_util.key = "main_a" + self.modify_config_map_util.modify() + + self.mock_modify_config_map.assert_called_once_with( + self.modify_config_map_util.main_cm, + self.modify_config_map_util.cm_namespace, + self.modify_config_map_util.product_name, + self.modify_config_map_util.product_version, + self.modify_config_map_util.key, + self.modify_config_map_util.max_retries_for_main_cm, + ) + + def test_delete_from_product_config_map(self): + """Test cases to assert delete calls into product ConfigMap""" + self.modify_config_map_util.key = "prod_3" + self.modify_config_map_util.modify() + + self.mock_modify_config_map.assert_called_once_with( + self.modify_config_map_util.product_cm, + self.modify_config_map_util.cm_namespace, + self.modify_config_map_util.product_name, + self.modify_config_map_util.product_version, + self.modify_config_map_util.key, + self.modify_config_map_util.max_retries_for_prod_cm, + ) + + def test_invalid_key(self): + """Test cases to assert invalid key""" + self.modify_config_map_util.key = "invalid string key" + self.modify_config_map_util.modify() + self.mock_modify_config_map.assert_not_called() + + self.modify_config_map_util.key = 909 # non string is invalid as well + self.modify_config_map_util.modify() + self.mock_modify_config_map.assert_not_called() diff --git a/tests/test_query.py b/tests/test_query.py index 3b7b109f..786fb4bc 100644 --- a/tests/test_query.py +++ b/tests/test_query.py @@ -31,16 +31,18 @@ from unittest.mock import Mock, patch from kubernetes.config import ConfigException -from yaml import safe_dump from cray_product_catalog.query import ( ProductCatalog, InstalledProductVersion, ProductCatalogError ) + +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR + from tests.mocks import ( - COS_VERSIONS, MOCK_PRODUCT_CATALOG_DATA, SAT_VERSIONS, - CPE_VERSION, MOCK_INVALID_PRODUCT_DATA + COS_VERSIONS, SAT_VERSIONS, CPE_VERSION, MOCK_PRODUCT_CATALOG_DATA, + MOCK_INVALID_PRODUCT_DATA, MOCK_PRODUCTS, MockInvalidYaml ) @@ -79,7 +81,7 @@ def setUp(self): """Set up mocks.""" self.mock_k8s_api = patch.object(ProductCatalog, '_get_k8s_api').start().return_value self.mock_product_catalog_data = copy.deepcopy(MOCK_PRODUCT_CATALOG_DATA) - self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=self.mock_product_catalog_data) + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=self.mock_product_catalog_data) def tearDown(self): """Stop patches.""" @@ -87,12 +89,16 @@ def tearDown(self): def create_and_assert_product_catalog(self): """Assert the product catalog was created as expected.""" - product_catalog = ProductCatalog('mock-name', 'mock-namespace') - self.mock_k8s_api.read_namespaced_config_map.assert_called_once_with('mock-name', 'mock-namespace') + product_catalog = ProductCatalog('cray-product-catalog', 'mock-namespace') + self.mock_k8s_api.list_namespaced_config_map.assert_called_once_with( + 'mock-namespace', label_selector=PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR + ) return product_catalog def test_create_product_catalog(self): """Test creating a simple ProductCatalog.""" + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = MOCK_PRODUCTS product_catalog = self.create_and_assert_product_catalog() expected_names_and_versions = [ (name, version) for name in ('sat', 'cos') for version in ('2.0.0', '2.0.1') @@ -104,26 +110,32 @@ def test_create_product_catalog(self): def test_create_product_catalog_invalid_product_data(self): """Test creating a ProductCatalog when the product catalog contains invalid YAML.""" - self.mock_product_catalog_data['sat'] = '\t' + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[MockInvalidYaml()]) with self.assertRaisesRegex(ProductCatalogError, 'Failed to load ConfigMap data'): self.create_and_assert_product_catalog() def test_create_product_catalog_null_data(self): """Test creating a ProductCatalog when the product catalog contains null data.""" - self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=None) + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = [] + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[]) with self.assertRaisesRegex(ProductCatalogError, - 'No data found in mock-namespace/mock-name ConfigMap.'): + 'No ConfigMaps found in mock-namespace namespace.'): self.create_and_assert_product_catalog() def check_for_invalid_product_schema(self, prod, ver): """Check for an ProductCatalog entry containing valid YAML but does not match schema. + Args: prod (string): Product Name ver (string): Product version """ - mock_data = {prod: safe_dump(MOCK_INVALID_PRODUCT_DATA.get(prod))} - self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=mock_data) - + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = [ + InstalledProductVersion(prod, ver, MOCK_INVALID_PRODUCT_DATA.get(prod).get(ver)) + ] + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[ + MOCK_INVALID_PRODUCT_DATA.get(prod).get(ver)]) with self.assertLogs(level=logging.DEBUG) as logs_cm: product_catalog = self.create_and_assert_product_catalog() @@ -156,6 +168,8 @@ def test_create_product_catalog_invalid_product_schema_for_manifests(self): def test_get_matching_product(self): """Test getting a particular product by name/version.""" + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = MOCK_PRODUCTS product_catalog = self.create_and_assert_product_catalog() expected_matching_name_and_version = ('cos', '2.0.0') actual_matching_product = product_catalog.get_product('cos', '2.0.0') @@ -167,6 +181,8 @@ def test_get_matching_product(self): def test_get_latest_matching_product(self): """Test getting the latest version of a product""" + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = MOCK_PRODUCTS product_catalog = self.create_and_assert_product_catalog() expected_matching_name_and_version = ('sat', '2.0.1') actual_matching_product = product_catalog.get_product('sat') @@ -226,7 +242,7 @@ def test_helm_charts(self): ('cray-cps', '1.8.15') ] self.assertEqual( - expected_helm_charts_versions, self.installed_product_version.helm_charts + expected_helm_charts_versions, self.installed_product_version.helm ) def test_no_helm_charts(self): @@ -234,21 +250,21 @@ def test_no_helm_charts(self): product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': {}}} ) - self.assertEqual(product_with_no_helm_charts.helm_charts, []) + self.assertEqual(product_with_no_helm_charts.helm, []) def test_no_helm_charts_null(self): """Test a product that has None under the 'helm' key returns an empty list.""" product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': None}} ) - self.assertEqual(product_with_no_helm_charts.helm_charts, []) + self.assertEqual(product_with_no_helm_charts.helm, []) def test_no_helm_charts_empty_list(self): """Test a product that has an empty list under the 'helm' key returns an empty list.""" product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': []}} ) - self.assertEqual(product_with_no_helm_charts.helm_charts, []) + self.assertEqual(product_with_no_helm_charts.helm, []) def test_s3_artifacts(self): """Test getting the s3 artifacts.""" diff --git a/tests/test_update_catalog.py b/tests/test_update_catalog.py new file mode 100755 index 00000000..94040e2e --- /dev/null +++ b/tests/test_update_catalog.py @@ -0,0 +1,192 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + + +"""Unit tests for cray_product_catalog.catalog_update module""" + +import unittest +import os +from unittest import mock +from tests.mock_update_catalog import ( + UPDATE_DATA, ApiInstance, ApiException +) +from cray_product_catalog.catalog_update import ( + create_config_map, + update_config_map, + main +) + + +class TestCatalogUpdate(unittest.TestCase): + """ + Tests for catalog update + """ + def setUp(self): + """Set up mocks.""" + self.mock_v1configmap = mock.patch('cray_product_catalog.catalog_update.V1ConfigMap').start() + self.mock_load_k8s = mock.patch('cray_product_catalog.catalog_update.load_k8s').start() + self.mock_ApiClient = mock.patch('cray_product_catalog.catalog_update.ApiClient').start() + self.mock_client = mock.patch('cray_product_catalog.catalog_update.client').start() + + def test_create_config_map_success_log(self): + """ + Test for validating create_config_map method logs + Verify expected log is generated based on the passed arguments + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + with self.assertLogs() as captured: + # call method under test + create_config_map(ApiInstance(raise_exception=False), name, namespace) + self.assertEqual(len(captured.records), 1) # check that there is only one log message + + expected_log = "Created product ConfigMap " + namespace + "/" + name + self.assertEqual(captured.records[0].getMessage(), expected_log) # Verify the exact log message + + def test_create_config_map_failure_exception(self): + """ + Verify if expected logs are generated in exception + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + + with self.assertLogs() as captured: + try: + # call method under test + create_config_map(ApiInstance(raise_exception=True), name, namespace) + except ApiException as err: + pass + self.assertEqual(len(captured.records), 1) # check that there is only one log message + self.assertEqual(captured.records[0].getMessage(), + "Error calling create_namespaced_config_map") # Verify the exact log message + + def test_update_config_map_max_retries(self): + """ + Verify update_config_map exits after max retries if Kubernetes API raise exception. + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + + self.mock_read_config_map = mock.patch( + 'cray_product_catalog.catalog_update.client.CoreV1Api.read_namespaced_config_map' + ).start().side_effect = ApiException() + + with self.assertLogs() as captured: + with mock.patch( + 'cray_product_catalog.catalog_update.random.randint', return_value=0 + ): + # call method under test + update_config_map(UPDATE_DATA, name, namespace) + # Verify the exact log message + self.assertEqual(captured.records[-1].getMessage(), + f"Exceeded number of attempts; Not updating ConfigMap {namespace}/{name}.") + + def test_update_config_map(self): + """ + Verify `create_config_map` is called if provided `name` is of product and not `CONFIG_MAP` (main cm) + """ + name = "cos" + namespace = "product" + data = UPDATE_DATA + + # mock some additional functions + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + with mock.patch( + 'cray_product_catalog.catalog_update.client.CoreV1Api', return_value=ApiInstance(raise_exception=True) + ): + with mock.patch( + 'cray_product_catalog.catalog_update.random.randint', return_value=0.5 + ): + # call method under test + update_config_map(data, name, namespace) + + # verify if create-config_map is called. Couldn't verify it with arguments as one of the arg is object. + self.mock_create_config_map.assert_called() + + def test_main_valid_product_configmap(self): + """ + Verify `update_config_map` is called with proper data if provided product information is available + """ + # Initialise some random data, need not to be exact format + prod_cm = {"Some random text for product"} + main_cm = {"Some random text for main"} + + self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + # mocking function to return custom data + def mock_split_catalog_data(): + return main_cm, prod_cm + + with mock.patch( + 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() + ): + # Call method under test + main() + # sat is from PRODUCT environment variable. + expected_product_cm = 'cray-product-catalog-sat' + # myNamespace is from CONFIG_MAP_NAMESPACE environment variable. + expected_namespace = 'myNamespace' + self.mock_update_config_map.assert_called_with(prod_cm, expected_product_cm, expected_namespace) + + def test_main_for_empty_product_configmap(self): + """ + Verify `main` throws exception when PRODUCT_CONFIG_MAP='' + """ + prod_cm = {"Some random text for product"} + main_cm = {"Some random text for main"} + + self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + # mocking function to return custom data + def mock_split_catalog_data(): + return main_cm, prod_cm + + with mock.patch( + 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() + ): + with mock.patch( + 'cray_product_catalog.catalog_update.format_product_cm_name', return_value='' + ): + with self.assertRaises(SystemExit) as context: + # call method under test + main() + # Verify the log message in exception + self.assertTrue( + "ERROR Not updating ConfigMaps because the provided product name is invalid: 'sat'" + in context.exception + ) + # Verify that update config map is not called in case of exception + self.mock_update_config_map.assert_not_called() + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/util/test_data_catalog_helper.py b/tests/util/test_data_catalog_helper.py new file mode 100755 index 00000000..cf203b29 --- /dev/null +++ b/tests/util/test_data_catalog_helper.py @@ -0,0 +1,124 @@ +# MIT License +# +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +""" +Unit tests for the cray_product_catalog.util.catalog_data_helper module +""" + +import unittest +from tests.mocks import ( + YAML_DATA, YAML_DATA_MISSING_MAIN_DATA, + YAML_DATA_MISSING_PROD_CM_DATA, + MAIN_CM_DATA, PROD_CM_DATA +) +import yaml +from typing import Dict +from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name + + +class TestCatalogDataHelper(unittest.TestCase): + """Tests for catalog_data_helper.""" + + def test_split_data_sanity(self): + """Sanity check of split of YAML into main and product-specific data | +ve test case""" + + # expected data + main_cm_data_expected = MAIN_CM_DATA + prod_cm_data_expected = PROD_CM_DATA + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_split_missing_prod_cm_data(self): + """Missing product ConfigMap data check""" + + # expected data + main_cm_data_expected = MAIN_CM_DATA + prod_cm_data_expected = {} + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_PROD_CM_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_split_missing_main_cm_data(self): + """Missing main ConfigMap data check""" + + # expected data + main_cm_data_expected = {} + prod_cm_data_expected = PROD_CM_DATA + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_MAIN_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_format_product_cm_name_sanity(self): + """Unit test case for product name formatting""" + product_name = "dummy-valid-1" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-{product_name}") + + def test_format_product_name_transform(self): + """Unit test case for valid product name transformation""" + product_name = "23dummy_valid-1.x86" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-23dummy-valid-1.x86") + + def test_format_product_name_invalid_cases(self): + """Unit test case for invalid product names""" + + # case with special characters + product_name = "po90-$_invalid" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + # large name with non-blank ConfigMap case + product_name = "ola-9" * 60 + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + # large name with blank ConfigMap case + product_name = "ola-9" * 60 + config_map = "" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + +if __name__ == '__main__': + unittest.main() From 86f583dde0f5dec5359f9157655eed5743f10ceb Mon Sep 17 00:00:00 2001 From: "Mitch Harding (the weird one)" Date: Wed, 29 Nov 2023 16:51:40 -0500 Subject: [PATCH 02/12] Do not hard code application version or stable/unstable in values.yaml -- set it dynamically at build time; remove extra whitespace from values.yaml --- charts/cray-product-catalog/values.yaml | 10 +++------- update_versions.conf | 2 ++ 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/charts/cray-product-catalog/values.yaml b/charts/cray-product-catalog/values.yaml index 96355abe..85b3f6a6 100644 --- a/charts/cray-product-catalog/values.yaml +++ b/charts/cray-product-catalog/values.yaml @@ -27,15 +27,11 @@ kubectl: tag: 1.19.15 migration: image: - repository: artifactory.algol60.net/csm-docker/unstable/cray-product-catalog-update - tag: $DOCKER_VERSION + repository: artifactory.algol60.net/csm-docker/S-T-A-B-L-E/cray-product-catalog-update + tag: 0.0.0-docker configMap: cray-product-catalog configMapNamespace: services serviceAccount: cray-product-catalog - -global: - appVersion: 1.10.0-add-rollback-pending.106_d9b8b23 - global: - appVersion: 1.10.0-add-rollback-pending.106_d9b8b23 + appVersion: 0.0.0-docker diff --git a/update_versions.conf b/update_versions.conf index 8f79c738..845c4c88 100644 --- a/update_versions.conf +++ b/update_versions.conf @@ -20,6 +20,7 @@ sourcefile: .docker_version tag: 0.0.0-docker targetfile: charts/cray-product-catalog/Chart.yaml +targetfile: charts/cray-product-catalog/values.yaml # This sourcefile does not exist as a static file in the repo. # It is created at build time. @@ -32,3 +33,4 @@ targetfile: charts/cray-product-catalog/Chart.yaml sourcefile-novalidate: .stable tag: S-T-A-B-L-E targetfile: charts/cray-product-catalog/Chart.yaml +targetfile: charts/cray-product-catalog/values.yaml From dc7dab1dcf43436c987354a43d207c375668ce1a Mon Sep 17 00:00:00 2001 From: "Mitch Harding (the weird one)" Date: Wed, 29 Nov 2023 18:07:09 -0500 Subject: [PATCH 03/12] Apply suggestions from code review Linting of code and comments --- .gitignore | 2 +- cray_product_catalog/catalog_delete.py | 14 ++-- cray_product_catalog/constants.py | 4 +- cray_product_catalog/migration/__init__.py | 8 +- .../migration/config_map_data_handler.py | 37 ++++----- .../migration/exit_handler.py | 33 ++++---- cray_product_catalog/migration/kube_apis.py | 83 ++++++++++--------- cray_product_catalog/migration/main.py | 10 +-- .../migration/test_config_map_data_handler.py | 43 +++++----- tests/migration/test_exit_handler.py | 17 ++-- tests/mock_update_catalog.py | 22 +++-- tests/mocks.py | 14 ++-- tests/test_catalog_delete.py | 4 +- tests/test_update_catalog.py | 3 +- tests/util/test_data_catalog_helper.py | 4 +- 15 files changed, 145 insertions(+), 153 deletions(-) diff --git a/.gitignore b/.gitignore index d6a5c321..dbb03c4e 100644 --- a/.gitignore +++ b/.gitignore @@ -12,4 +12,4 @@ workspace.code-workspace **__pycache__** # removing pycharm specific -.idea \ No newline at end of file +.idea diff --git a/cray_product_catalog/catalog_delete.py b/cray_product_catalog/catalog_delete.py index a8b8f195..76f5baa3 100755 --- a/cray_product_catalog/catalog_delete.py +++ b/cray_product_catalog/catalog_delete.py @@ -2,7 +2,7 @@ # # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), @@ -75,7 +75,7 @@ class ModifyConfigMapUtil: - """Utility class to manage the config map modification + """Utility class to manage the ConfigMap modification """ def __init__(self): @@ -195,16 +195,16 @@ def modify(self): """ Method to initiate modification of ConfigMaps. Before executing this method make sure to set these properties of the class: - * main_cm # name of main ConfigMap - * product_cm # name of product specific CofigMap - * cm_namespace # Namespace containing all config map + * main_cm # Name of main ConfigMap + * product_cm # name of product-specific CofigMap + * cm_namespace # Namespace containing all ConfigMaps * product_name # Product name * product_version # Product version * max_reties_for_main_cm # Max failure retries for main ConfigMap * max_reties_for_prod_cm # Max failure retries for product ConfigMap - * key # Key to delete, if you want to execute complete product or a particular version, ignore it + * key # Key to delete; if you want to execute complete product or a particular version, ignore it * main_cm_fields # Fields present in main ConfigMap - * product_cm_fields # Fields present in product specific ConfigMap + * product_cm_fields # Fields present in product-specific ConfigMap """ if self.__key: if self.__key_belongs_to_main_cm_fields(): diff --git a/cray_product_catalog/constants.py b/cray_product_catalog/constants.py index 01b5a978..1627a9fe 100644 --- a/cray_product_catalog/constants.py +++ b/cray_product_catalog/constants.py @@ -37,7 +37,5 @@ PRODUCT_CM_FIELDS = {'component_versions'} PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY = 'type' PRODUCT_CATALOG_CONFIG_MAP_LABEL = {PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY: PRODUCT_CATALOG_CONFIG_MAP_NAME} -PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR = "{0}={1}".format( - PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY, PRODUCT_CATALOG_CONFIG_MAP_NAME - ) +PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR = f"{PRODUCT_CATALOG_CONFIG_MAP_LABEL_KEY}={PRODUCT_CATALOG_CONFIG_MAP_NAME}" PRODUCT_CATALOG_CONFIG_MAP_REPLICA = 'cray-product-catalog-temp' diff --git a/cray_product_catalog/migration/__init__.py b/cray_product_catalog/migration/__init__.py index 7717399c..2a44595d 100644 --- a/cray_product_catalog/migration/__init__.py +++ b/cray_product_catalog/migration/__init__.py @@ -24,21 +24,21 @@ """ import os +import re from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR -from re import compile -# ConfigMap name for temporary many config map +# ConfigMap name for temporary main ConfigMap CONFIG_MAP_TEMP = "cray-product-catalog-temp" # namespace for ConfigMaps PRODUCT_CATALOG_CONFIG_MAP_NAME = os.environ.get("CONFIG_MAP_NAME", "cray-product-catalog").strip() PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE = os.environ.get("CONFIG_MAP_NAMESPACE", "services").strip() -# config map names +# ConfigMap names CRAY_DATA_CATALOG_LABEL = PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR # product ConfigMap pattern -PRODUCT_CONFIG_MAP_PATTERN = compile('^(cray-product-catalog)-([a-z0-9.-]+)$') +PRODUCT_CONFIG_MAP_PATTERN = re.compile('^(cray-product-catalog)-([a-z0-9.-]+)$') RESOURCE_VERSION = 'resource_version' retry_count = 10 diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py index bc637d7b..d0b0a370 100644 --- a/cray_product_catalog/migration/config_map_data_handler.py +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -1,4 +1,3 @@ -#!/usr/bin/env python3 # # MIT License # @@ -25,20 +24,19 @@ """ This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller -ConfigMaps with each product's `component_versions` data in its own Product ConfigMap +ConfigMaps with each product's `component_versions` data in its own product ConfigMap """ import logging import yaml -from kubernetes.client.rest import ApiException from cray_product_catalog.logging import configure_logging from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name from cray_product_catalog.migration.kube_apis import KubernetesApi from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL from cray_product_catalog.migration import ( - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, action + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE ) LOGGER = logging.getLogger(__name__) @@ -119,7 +117,7 @@ def migrate_config_map_data(self, config_map_data): # main_cm_data is not an empty dictionary if main_cm_data: main_versions_data[version_data] = main_cm_data - # If `component_versions` data exists for a product, create new product config map + # If `component_versions` data exists for a product, create new product ConfigMap if product_versions_data: product_config_map_data[product] = yaml.safe_dump(product_versions_data, default_flow_style=False) # create_product_config_map(k8s_obj, product, product_config_map_data) @@ -133,16 +131,17 @@ def migrate_config_map_data(self, config_map_data): return config_map_data, product_config_map_data_list def rename_config_map(self, rename_from, rename_to, namespace, label): - """ Renaming is actually deleting one Config Map and then updating the name of other Config Map and patch it. - :param str rename_from: Name of Config Map to rename - :param str rename_to: Name of Config Map to be renamed to - :param str namespace: namespace in which Config Map has to be updated - :param dict label: label of config map to be renamed + """ Renaming is actually deleting one ConfigMap and then updating the name of other ConfigMap and patching it. + :param str rename_from: Name of ConfigMap to rename + :param str rename_to: Name of ConfigMap after rename + :param str namespace: Namespace in which ConfigMap has to be updated + :param dict label: Label of ConfigMap to be renamed :return: bool, If Success True else False """ if not self.k8s_obj.delete_config_map(rename_to, namespace): - logging.error("Failed to delete ConfigMap %s", rename_to) + LOGGER.error("Failed to delete ConfigMap %s", rename_to) + return False attempt = 0 del_failed = False @@ -159,13 +158,11 @@ def rename_config_map(self, rename_from, rename_to, namespace, label): if self.k8s_obj.delete_config_map(rename_from, namespace): LOGGER.info("Renaming ConfigMap successful") return True - else: - LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) - del_failed = True - break - else: - LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) - continue + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) + del_failed = True + break + LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) + continue # Since only delete of backed up ConfigMap failed, retrying only delete operation attempt = 0 if del_failed: @@ -173,9 +170,7 @@ def rename_config_map(self, rename_from, rename_to, namespace, label): attempt += 1 if self.k8s_obj.delete_config_map(rename_from, namespace): break - else: - LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) - continue + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) # Returning success as migration is successful only backed up ConfigMap is not deleted. LOGGER.info("Renaming ConfigMap successful") return True diff --git a/cray_product_catalog/migration/exit_handler.py b/cray_product_catalog/migration/exit_handler.py index 2f997611..98e7675b 100644 --- a/cray_product_catalog/migration/exit_handler.py +++ b/cray_product_catalog/migration/exit_handler.py @@ -21,8 +21,8 @@ # OTHER DEALINGS IN THE SOFTWARE. """ File contains logic to handle exit scenarios: -a. graceful -b. non-graceful +a. Graceful +b. Non-graceful c. Rollback case """ @@ -40,22 +40,21 @@ def _is_product_config_map(config_map_name: str) -> bool: - """Function to check product config map pattern - returns True if pattern match found + """Function to check product ConfigMap pattern. + Returns True if pattern match found """ if config_map_name is None or config_map_name == "": return False - elif fullmatch(PRODUCT_CONFIG_MAP_PATTERN, config_map_name): + if fullmatch(PRODUCT_CONFIG_MAP_PATTERN, config_map_name): return True - else: - return False + return False class ExitHandler: """Class to handle exit and rollback classes""" def __init__(self): - self.k8api = KubernetesApi() # kubernetes API object + self.k8api = KubernetesApi() # Kubernetes API object @staticmethod def graceful_exit() -> None: @@ -66,7 +65,7 @@ def exception_exit() -> None: LOGGER.error("Migration not possible, exception occurred.") def __get_all_created_product_config_maps(self) -> List: - """Get all created product config maps""" + """Get all created product ConfigMaps""" cm_name = filter(_is_product_config_map, self.k8api.list_config_map_names( label=CRAY_DATA_CATALOG_LABEL, @@ -76,22 +75,22 @@ def __get_all_created_product_config_maps(self) -> List: def rollback(self): """Method to handle roll back - 1. Deleting temp config map - 2. Deleting all created product config map + 1. Deleting temporary ConfigMap + 2. Deleting all created product ConfigMaps """ LOGGER.warning("Initiating rollback") - product_config_maps = self.__get_all_created_product_config_maps() # collecting product config map + product_config_maps = self.__get_all_created_product_config_maps() # collecting product ConfigMaps - LOGGER.info("deleting Product ConfigMaps") # attempting to delete product config maps + LOGGER.info("Deleting product ConfigMaps") # attempting to delete product ConfigMaps non_deleted_product_config_maps = [] for config_map in product_config_maps: - LOGGER.debug("deleting Product ConfigMap %s", config_map) + LOGGER.debug("Deleting product ConfigMap %s", config_map) if not self.k8api.delete_config_map(name=config_map, namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE): non_deleted_product_config_maps.append(config_map) - if len(non_deleted_product_config_maps) > 0: # checking if any product config map is not deleted - LOGGER.error("Error in deleting ConfigMap/s %s. Delete this/these manually", + if non_deleted_product_config_maps: # checking if any product ConfigMap is not deleted + LOGGER.error("Error deleting ConfigMaps: %s. Delete these manually", non_deleted_product_config_maps) return - LOGGER.info("rollback successful") + LOGGER.info("Rollback successful") diff --git a/cray_product_catalog/migration/kube_apis.py b/cray_product_catalog/migration/kube_apis.py index 1e83753e..96585ecc 100644 --- a/cray_product_catalog/migration/kube_apis.py +++ b/cray_product_catalog/migration/kube_apis.py @@ -37,19 +37,10 @@ from cray_product_catalog.logging import configure_logging from cray_product_catalog.util.k8s import load_k8s from . import retry_count -# retry_count=10 - -# from kubernetes import configs -# def load_k8s(): -# """ Load Kubernetes configuration """ -# try: -# config.load_incluster_config() -# except Exception: -# config.load_kube_config() class KubernetesApi: - """Class for wrapping Kubernetes api""" + """Class for wrapping Kubernetes API""" def __init__(self): configure_logging() self.logger = logging.getLogger(__name__) @@ -64,11 +55,11 @@ def __init__(self): self.api_instance = client.CoreV1Api(self.kclient) def create_config_map(self, name, namespace, data, label): - """Creates Config Map - :param dict data: Content of configmap - :param str name: config map name to be created - :param str namespace: namespace in which configmap has to be created - :param dict label: label with which configmap has to be created + """Creates ConfigMap + :param dict data: Content of ConfigMap + :param str name: ConfigMap name to be created + :param str namespace: Namespace in which ConfigMap has to be created + :param dict label: Label with which ConfigMap has to be created :return: bool """ try: @@ -84,35 +75,45 @@ def create_config_map(self, name, namespace, data, label): ) return True except MaxRetryError as err: - self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + self.logger.exception('MaxRetryError: %s', err) return False except ApiException as err: - self.logger.exception('ApiException- Error:{0}'.format(err)) + # The full string representation of ApiException is very long, so just log err.reason. + self.logger.exception('ApiException: %s', err.reason) return False + self.logger.error('Unknown error creating ConfigMap') + return False + self.logger.error('Unknown error creating ConfigMap') + return False def list_config_map(self, namespace, label): - """ Reads all the Config Map with certain label in particular namespace - :param str namespace: Value of namespace from where config map has to be listed - :param str label: string format of label "type=xyz" + """ Reads all the ConfigMaps with certain label in particular namespace + :param str namespace: Value of namespace from where ConfigMap has to be listed + :param str label: String format of label "type=xyz" :return: V1ConfigMapList If there is any error, returns None """ if not all((label, namespace)): - self.logger.info("Either label or namespace is empty, not reading config map.") + self.logger.info("Either label or namespace is empty, not reading ConfigMap.") return None try: return self.api_instance.list_namespaced_config_map(namespace, label_selector=label).items except MaxRetryError as err: - self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + self.logger.exception('MaxRetryError: %s', err) return None except ApiException as err: - self.logger.exception('ApiException- Error:{0}'.format(err)) + # The full string representation of ApiException is very long, so just log err.reason. + self.logger.exception('ApiException: %s', err.reason) return None + self.logger.error('Unknown error listing ConfigMaps') + return None + self.logger.error('Unknown error listing ConfigMaps') + return None def list_config_map_names(self, namespace, label): - """ Reads all the Config Map with certain label in particular namespace - :param str namespace: Value of namespace from where config map has to be listed - :param str label: string format of label "type=xyz" + """ Reads all the ConfigMaps with certain label in particular namespace + :param str namespace: Value of namespace from where ConfigMap has to be listed + :param str label: String format of label "type=xyz" :return: [str] """ cm_output = self.list_config_map(namespace, label) @@ -132,37 +133,45 @@ def list_config_map_names(self, namespace, label): return list_cm_names def read_config_map(self, name, namespace): - """Reads config Map based on provided name and namespace - :param Str name: name of ConfigMap to read - :param Str namespace: namespace from which Config Map has to be read + """Reads ConfigMap based on provided name and namespace + :param Str name: Name of ConfigMap to read + :param Str namespace: Namespace from which ConfigMap has to be read :return: V1ConfigMap Returns None in case of any error """ # Check if both values are not empty if not all((name, namespace)): - self.logger.exception("Either name or namespace is empty, not reading config map.") + self.logger.exception("Either name or namespace is empty, not reading ConfigMap.") return None try: return self.api_instance.read_namespaced_config_map(name, namespace) except MaxRetryError as err: - self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + self.logger.exception('MaxRetryError: %s', err) return None except ApiException as err: - self.logger.exception('ApiException- Error:{0}'.format(err)) + # The full string representation of ApiException is very long, so just log err.reason. + self.logger.exception('ApiException: %s', err.reason) return None + self.logger.error('Unknown error reading ConfigMap') + return None + self.logger.error('Unknown error reading ConfigMap') + return None def delete_config_map(self, name, namespace): - """Delete the Config Map - :param Str name: name of ConfigMap to be deleted - :param Str namespace: namespace from which Config Map has to be deleted + """Delete the ConfigMap + :param Str name: Name of ConfigMap to be deleted + :param Str namespace: Namespace from which ConfigMap has to be deleted :return: bool; If success True else False """ try: self.api_instance.delete_namespaced_config_map(name, namespace) return True except MaxRetryError as err: - self.logger.exception('MaxRetryError - Error: {0}'.format(err)) + self.logger.exception('MaxRetryError: %s', err) return False except ApiException as err: - self.logger.exception('ApiException- Error:{0}'.format(err)) + # The full string representation of ApiException is very long, so just log err.reason. + self.logger.exception('ApiException: %s', err.reason) return False + self.logger.error('Unknown error deleting ConfigMap') + return False diff --git a/cray_product_catalog/migration/main.py b/cray_product_catalog/migration/main.py index 0575a982..76d7fc02 100644 --- a/cray_product_catalog/migration/main.py +++ b/cray_product_catalog/migration/main.py @@ -30,7 +30,7 @@ from cray_product_catalog.migration import ( PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - CONFIG_MAP_TEMP, RESOURCE_VERSION + CONFIG_MAP_TEMP ) from cray_product_catalog.migration.exit_handler import ExitHandler @@ -63,7 +63,7 @@ def main(): else: config_map_data = response.data else: - LOGGER.info("Error reading ConfigMap, exiting migration process...") + LOGGER.error("Error reading ConfigMap, exiting migration process...") raise SystemExit(1) try: @@ -94,8 +94,7 @@ def main(): LOGGER.error("Error reading resourceVersion, exiting migration process...") exit_handler.rollback() raise SystemExit(1) - else: - curr_resource_version = response.metadata.resource_version + curr_resource_version = response.metadata.resource_version if curr_resource_version != init_resource_version: migration_failed = True @@ -104,8 +103,7 @@ def main(): LOGGER.info("Re-trying migration process...") exit_handler.rollback() continue - else: - break + break if migration_failed: LOGGER.info("ConfigMap %s is modified by other process, exiting migration process...", diff --git a/tests/migration/test_config_map_data_handler.py b/tests/migration/test_config_map_data_handler.py index 17b698e7..1bebd95f 100755 --- a/tests/migration/test_config_map_data_handler.py +++ b/tests/migration/test_config_map_data_handler.py @@ -30,17 +30,12 @@ from unittest.mock import patch, call, Mock from typing import Dict, List -from kubernetes.config import ConfigException -from kubernetes import client -from kubernetes.client.api_client import ApiClient - from cray_product_catalog.migration.main import main from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler from cray_product_catalog.constants import ( PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, PRODUCT_CATALOG_CONFIG_MAP_LABEL ) -from cray_product_catalog.util.catalog_data_helper import format_product_cm_name from cray_product_catalog.migration import CONFIG_MAP_TEMP from tests.migration.migration_mock import ( MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED, INITIAL_MAIN_CM_DATA, MockYaml @@ -84,7 +79,7 @@ def test_migrate_config_map_data(self): self.assertEqual(prod_cm_data_list, PROD_CM_DATA_LIST_EXPECTED) def test_create_product_config_maps(self): - """ Validating product config maps are created """ + """ Validating product ConfigMaps are created """ # mock some additional functions self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() @@ -177,7 +172,7 @@ def test_create_first_product_config_map_failed(self): f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") def test_create_temp_config_map(self): - """ Validating temp main config map is created """ + """ Validating temp main ConfigMap is created """ # mock some additional functions self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() @@ -198,7 +193,7 @@ def test_create_temp_config_map(self): f"Created temp ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP}") def test_create_temp_config_map_failed(self): - """ Validating temp main config map creation failed """ + """ Validating temp main ConfigMap creation failed """ # mock some additional functions self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() @@ -220,7 +215,7 @@ def test_create_temp_config_map_failed(self): f"Creating ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP} failed") def test_rename_config_map(self): - """ Validating product config maps are created """ + """ Validating product ConfigMaps are created """ with self.assertLogs(level="DEBUG") as captured: # call method under test @@ -255,7 +250,7 @@ def test_rename_config_map(self): "Renaming ConfigMap successful") def test_rename_config_map_failed_1(self): - """ Validating rename config map failure scenario where: + """ Validating rename ConfigMap failure scenario where: deleting cray-product-catalog ConfigMap failed. """ with self.assertLogs() as captured: @@ -275,7 +270,7 @@ def test_rename_config_map_failed_1(self): f"Failed to delete ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}") def test_rename_config_map_failed_2(self): - """ Validating rename config map failure scenario where: + """ Validating rename ConfigMap failure scenario where: creating cray-product-catalog ConfigMap failed. """ with self.assertLogs(level="DEBUG") as captured: @@ -329,7 +324,7 @@ def test_rename_config_map_failed_2(self): f"Failed to create ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}, retrying..") def test_rename_config_map_failed_3(self): - """ Validating rename config map failure scenario where: + """ Validating rename ConfigMap failure scenario where: first operation of deleting cray-product-catalog-temp ConfigMap failed but later passed. """ with self.assertLogs(level="DEBUG") as captured: @@ -362,10 +357,10 @@ def test_rename_config_map_failed_3(self): f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") self.assertEqual( captured.records[1].getMessage(), - f"Renaming ConfigMap successful") + "Renaming ConfigMap successful") def test_rename_config_map_failed_4(self): - """ Validating rename config map failure scenario where: + """ Validating rename ConfigMap failure scenario where: everytime deleting cray-product-catalog-temp ConfigMap failed. """ with self.assertLogs(level="DEBUG") as captured: @@ -399,7 +394,7 @@ def test_rename_config_map_failed_4(self): f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") self.assertEqual( captured.records[-1].getMessage(), - f"Renaming ConfigMap successful") + "Renaming ConfigMap successful") def test_main_for_successful_migration(self): """Validating that migration is successful""" @@ -428,7 +423,7 @@ def test_main_for_successful_migration(self): self.assertEqual( captured.records[-1].getMessage(), - f"Migration successful") + "Migration successful") def test_main_failed_1(self): """Validating that migration failed as renaming failed""" @@ -457,12 +452,12 @@ def test_main_failed_1(self): main() self.assertTrue( - f"Renaming cray-product-catalog-temp to cray-product-catalog ConfigMap failed, " - f"calling rollback handler..." in captured.exception + "Renaming cray-product-catalog-temp to cray-product-catalog ConfigMap failed, " + "calling rollback handler..." in captured.exception ) self.assertTrue( - "rollback successful" in captured.exception + "Rollback successful" in captured.exception ) def test_main_failed_2(self): @@ -491,13 +486,13 @@ def test_main_failed_2(self): main() self.assertTrue( - "rollback successful" in captured.exception + "Rollback successful" in captured.exception ) self.mock_rename_cm.assert_not_called() def test_main_failed_3(self): - """Validating that migration failed as create product config maps failed""" + """Validating that migration failed as create product ConfigMaps failed""" self.mock_migrate_config_map = patch( 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' @@ -521,7 +516,7 @@ def test_main_failed_3(self): main() self.assertTrue( - "rollback successful" in captured.exception + "Rollback successful" in captured.exception ) self.mock_create_temp_cm.assert_not_called() @@ -622,7 +617,7 @@ def test_main_failed_7(self): ) self.assertTrue( - "rollback successful" in captured.exception + "Rollback successful" in captured.exception ) self.assertTrue( @@ -668,7 +663,7 @@ def test_main_failed_8(self): ) self.assertEqual( captured.records[6].getMessage(), - "rollback successful" + "Rollback successful" ) self.assertEqual( diff --git a/tests/migration/test_exit_handler.py b/tests/migration/test_exit_handler.py index 44aa3458..ee3ffe4f 100644 --- a/tests/migration/test_exit_handler.py +++ b/tests/migration/test_exit_handler.py @@ -26,7 +26,6 @@ import unittest from unittest.mock import patch, call -from cray_product_catalog.migration import CONFIG_MAP_TEMP from cray_product_catalog.migration.exit_handler import _is_product_config_map, ExitHandler from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE @@ -48,7 +47,7 @@ def tearDown(self) -> None: patch.stopall() def test_product_config_map_pattern(self): - """Test cases for checking all valid patterns of product config map""" + """Test cases for checking all valid patterns of product ConfigMap""" base_str = "cray-product-catalog" valid_patterns = ( f"{base_str}-cos", @@ -60,19 +59,19 @@ def test_product_config_map_pattern(self): self.assertTrue(_is_product_config_map(valid_pattern)) def test_product_config_map_invalid_pattern(self): - """Test cases for checking all invalid patterns of product config map""" + """Test cases for checking all invalid patterns of product ConfigMap""" base_str = "cray-product-catalog" invalid_patterns = ( f"{base_str}", - f"90-lojp", - f"cos2.3.45.x86" + "90-lojp", + "cos2.3.45.x86" ) for invalid_pattern in invalid_patterns: self.assertFalse(_is_product_config_map(invalid_pattern)) def test_rollback_failure_from_product_config_map_deletion(self): - """Validating the scenario where one of the product config map is not deleted""" + """Validating the scenario where one of the product ConfigMap is not deleted""" with self.assertLogs() as captured: self.mock_k8api_del.side_effect = [True, False] # delete is called two times @@ -82,8 +81,8 @@ def test_rollback_failure_from_product_config_map_deletion(self): eh = ExitHandler() eh.rollback() # Verify the exact log message from last return - self.assertEqual(captured.records[-1].getMessage(), f"Error in deleting ConfigMap/s " - f"{[dummy_products[-1]]}. Delete this/these manually") + self.assertEqual(captured.records[-1].getMessage(), f"Error deleting ConfigMaps: " + f"{[dummy_products[-1]]}. Delete these manually") def test_rollback_all_success(self): """Validating the scenario of successful rollback""" @@ -96,7 +95,7 @@ def test_rollback_all_success(self): eh = ExitHandler() eh.rollback() # Verify the exact log message from last return - self.assertEqual(captured.records[-1].getMessage(), "rollback successful") + self.assertEqual(captured.records[-1].getMessage(), "Rollback successful") # three calls in sequence for complete flow self.mock_k8api_del.assert_has_calls(calls=[ diff --git a/tests/mock_update_catalog.py b/tests/mock_update_catalog.py index ba641af5..97bdf1c7 100755 --- a/tests/mock_update_catalog.py +++ b/tests/mock_update_catalog.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), @@ -25,9 +25,9 @@ Mock data for catalog_update unit tests """ -from kubernetes.client.rest import ApiException import os from unittest import mock +import kubernetes.client.rest from tests.mocks import COS_VERSIONS, Name # Mocking environment variables before import so that: @@ -65,11 +65,13 @@ def __init__(self): self.metadata = Name() -class ApiException(ApiException): +class ApiException(kubernetes.client.rest.ApiException): """ Custom Exception to define status """ def __init__(self): + super().__init__() + super().__init__() self.status = ERR_NOT_FOUND @@ -83,28 +85,24 @@ def __init__(self, raise_exception=False): def create_namespaced_config_map(self, namespace='a', body='b'): """ - Dummy Function to raise exception, if needed + Dummy function to raise exception, if needed """ if self.raise_exception: raise ApiException() - else: - pass def read_namespaced_config_map(self, name, namespace): """ - Dummy Function to : + Dummy function to : 1. Raise exception - 2. generate and return proper response with data and metadata + 2. Generate and return proper response with data and metadata """ # if this is called for first time return exception, so that product cm is created. if self.count == 0: self.count += 1 raise ApiException() - else: - return Response() + return Response() def patch_namespaced_config_map(self, name, namespace, body='xxx'): """ - Dummy function to handle the call in code, does nothing + Dummy function to handle the call in code; does nothing """ - pass diff --git a/tests/mocks.py b/tests/mocks.py index f9d34f28..49283d77 100644 --- a/tests/mocks.py +++ b/tests/mocks.py @@ -25,8 +25,8 @@ Mock data for ProductCatalog and InstalledProductVersion unit tests """ -from yaml import safe_dump import datetime +from yaml import safe_dump from cray_product_catalog.query import InstalledProductVersion @@ -76,8 +76,10 @@ # Two versions of a product named COS where: # - The two versions have one docker image name and version in common -# - The first version has docker and manifests but not helm charts, repositories, configuration, images, or recipes -# - The second version has docker, helm charts, repositories, configuration, images, and recipes, but not manifests +# - The first version has docker images and manifests but not helm charts, repositories, configuration, +# images, or recipes +# - The second version has docker images, helm charts, repositories, configuration, images, and recipes, +# but not manifests COS_VERSIONS = { '2.0.0': { 'component_versions': { @@ -212,9 +214,9 @@ # A mock version of the data returned after loading the ConfigMap data MOCK_PRODUCTS = \ - [InstalledProductVersion('sat', version, SAT_VERSIONS.get(version)) for version in SAT_VERSIONS.keys()] + \ - [InstalledProductVersion('cos', version, COS_VERSIONS.get(version)) for version in COS_VERSIONS.keys()] + \ - [InstalledProductVersion('cpe', version, CPE_VERSION.get(version)) for version in CPE_VERSION.keys()] + \ + [InstalledProductVersion('sat', version, SAT_VERSIONS.get(version)) for version in SAT_VERSIONS] + \ + [InstalledProductVersion('cos', version, COS_VERSIONS.get(version)) for version in COS_VERSIONS] + \ + [InstalledProductVersion('cpe', version, CPE_VERSION.get(version)) for version in CPE_VERSION] + \ [InstalledProductVersion('other_product', version, OTHER_PRODUCT_VERSION.get(version)) for version in OTHER_PRODUCT_VERSION.keys()] diff --git a/tests/test_catalog_delete.py b/tests/test_catalog_delete.py index 64ef6695..a5ed37bf 100644 --- a/tests/test_catalog_delete.py +++ b/tests/test_catalog_delete.py @@ -21,7 +21,7 @@ # OTHER DEALINGS IN THE SOFTWARE. """ File contains unit test classes for validating ConfigMap deletion logic. -Deleting keys/Product or a specific version of a product from ConfigMap +Deleting keys/product or a specific version of a product from ConfigMap """ import unittest from unittest.mock import patch, call @@ -30,7 +30,7 @@ class TestModifyConfigMapUtil(unittest.TestCase): - """unittest class for Data catalog ConfigMap deletion logic""" + """unittest class for data catalog ConfigMap deletion logic""" def setUp(self) -> None: self.mock_modify_config_map = patch('cray_product_catalog.catalog_delete.modify_config_map').start() diff --git a/tests/test_update_catalog.py b/tests/test_update_catalog.py index 94040e2e..d0d07989 100755 --- a/tests/test_update_catalog.py +++ b/tests/test_update_catalog.py @@ -25,7 +25,6 @@ """Unit tests for cray_product_catalog.catalog_update module""" import unittest -import os from unittest import mock from tests.mock_update_catalog import ( UPDATE_DATA, ApiInstance, ApiException @@ -76,7 +75,7 @@ def test_create_config_map_failure_exception(self): try: # call method under test create_config_map(ApiInstance(raise_exception=True), name, namespace) - except ApiException as err: + except ApiException: pass self.assertEqual(len(captured.records), 1) # check that there is only one log message self.assertEqual(captured.records[0].getMessage(), diff --git a/tests/util/test_data_catalog_helper.py b/tests/util/test_data_catalog_helper.py index cf203b29..015c967c 100755 --- a/tests/util/test_data_catalog_helper.py +++ b/tests/util/test_data_catalog_helper.py @@ -25,13 +25,13 @@ """ import unittest +import yaml +from typing import Dict from tests.mocks import ( YAML_DATA, YAML_DATA_MISSING_MAIN_DATA, YAML_DATA_MISSING_PROD_CM_DATA, MAIN_CM_DATA, PROD_CM_DATA ) -import yaml -from typing import Dict from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name From d033bd8e93527571ce53491e53760f1af6396770 Mon Sep 17 00:00:00 2001 From: Mitch Harding Date: Wed, 29 Nov 2023 18:09:07 -0500 Subject: [PATCH 04/12] Update CHANGELOG.md Set version as 1.10.1 --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4752f468..b4d58603 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.10.1] - 2023-11-30 + ### Changed - CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the @@ -459,7 +461,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Change default reviewers to CMS-core-product-support -[Unreleased]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.0...HEAD +[Unreleased]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.1...HEAD + +[1.10.1]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.0...v1.10.1 [1.10.0]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.9.0...v1.10.0 From 4194f45a2a80e43011566194959ed9345d0b1c5f Mon Sep 17 00:00:00 2001 From: Latha Nanjundaiah Date: Fri, 8 Dec 2023 22:14:36 +0530 Subject: [PATCH 05/12] Addressed review comments --- cray_product_catalog/catalog_update.py | 1 + .../migration/config_map_data_handler.py | 6 ++++-- cray_product_catalog/migration/exit_handler.py | 6 +++--- cray_product_catalog/migration/main.py | 15 +++++++++++---- tests/mock_update_catalog.py | 1 - tests/test_update_catalog.py | 7 ++++--- 6 files changed, 23 insertions(+), 13 deletions(-) diff --git a/cray_product_catalog/catalog_update.py b/cray_product_catalog/catalog_update.py index be0fcbc9..5a40631e 100755 --- a/cray_product_catalog/catalog_update.py +++ b/cray_product_catalog/catalog_update.py @@ -270,6 +270,7 @@ def update_config_map(data: dict, name, namespace): if attempt == retries: LOGGER.error("Exceeded number of attempts; Not updating ConfigMap %s/%s.", namespace, name) + raise SystemExit(1) def main(): diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py index d0b0a370..7d241373 100644 --- a/cray_product_catalog/migration/config_map_data_handler.py +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -23,8 +23,10 @@ # """ -This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller -ConfigMaps with each product's `component_versions` data in its own product ConfigMap +File contains functions to +Split data in `cray-product-catalog` ConfigMap +Create temporary and product ConfigMaps +Rename ConfigMap """ import logging diff --git a/cray_product_catalog/migration/exit_handler.py b/cray_product_catalog/migration/exit_handler.py index 98e7675b..8759835d 100644 --- a/cray_product_catalog/migration/exit_handler.py +++ b/cray_product_catalog/migration/exit_handler.py @@ -31,7 +31,7 @@ from re import fullmatch -from cray_product_catalog.migration import CONFIG_MAP_TEMP, CRAY_DATA_CATALOG_LABEL, \ +from cray_product_catalog.migration import CRAY_DATA_CATALOG_LABEL, \ PRODUCT_CONFIG_MAP_PATTERN from cray_product_catalog.migration import PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE from cray_product_catalog.migration.kube_apis import KubernetesApi @@ -75,8 +75,8 @@ def __get_all_created_product_config_maps(self) -> List: def rollback(self): """Method to handle roll back - 1. Deleting temporary ConfigMap - 2. Deleting all created product ConfigMaps + Deleting temporary ConfigMap and all created product ConfigMaps + whose names are determined using the pattern PRODUCT_CONFIG_MAP_PATTERN """ LOGGER.warning("Initiating rollback") product_config_maps = self.__get_all_created_product_config_maps() # collecting product ConfigMaps diff --git a/cray_product_catalog/migration/main.py b/cray_product_catalog/migration/main.py index 76d7fc02..5d4f0698 100644 --- a/cray_product_catalog/migration/main.py +++ b/cray_product_catalog/migration/main.py @@ -23,6 +23,13 @@ # OTHER DEALINGS IN THE SOFTWARE. # +""" +This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller +ConfigMaps with each product's `component_versions` data in its own product ConfigMap. +If the split is not succesful then it rollbacks to its initial state where ConfigMap +`cray-product-catalog` will contain complete data which includes `component-versions` +""" + import logging from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL @@ -56,12 +63,12 @@ def main(): if response: if not response.metadata.resource_version: LOGGER.error("Error reading resourceVersion, exiting migration process...") - else: - init_resource_version = response.metadata.resource_version + raise SystemExit(1) + init_resource_version = response.metadata.resource_version if not response.data: LOGGER.error("Error reading ConfigMap data, exiting migration process...") - else: - config_map_data = response.data + raise SystemExit(1) + config_map_data = response.data else: LOGGER.error("Error reading ConfigMap, exiting migration process...") raise SystemExit(1) diff --git a/tests/mock_update_catalog.py b/tests/mock_update_catalog.py index 97bdf1c7..66d733cb 100755 --- a/tests/mock_update_catalog.py +++ b/tests/mock_update_catalog.py @@ -70,7 +70,6 @@ class ApiException(kubernetes.client.rest.ApiException): Custom Exception to define status """ def __init__(self): - super().__init__() super().__init__() self.status = ERR_NOT_FOUND diff --git a/tests/test_update_catalog.py b/tests/test_update_catalog.py index d0d07989..06efc265 100755 --- a/tests/test_update_catalog.py +++ b/tests/test_update_catalog.py @@ -93,15 +93,16 @@ def test_update_config_map_max_retries(self): 'cray_product_catalog.catalog_update.client.CoreV1Api.read_namespaced_config_map' ).start().side_effect = ApiException() - with self.assertLogs() as captured: + with self.assertRaises(SystemExit) as captured: with mock.patch( 'cray_product_catalog.catalog_update.random.randint', return_value=0 ): # call method under test update_config_map(UPDATE_DATA, name, namespace) # Verify the exact log message - self.assertEqual(captured.records[-1].getMessage(), - f"Exceeded number of attempts; Not updating ConfigMap {namespace}/{name}.") + self.assertTrue( + f"Exceeded number of attempts; Not updating ConfigMap {namespace}/{name}." + in captured.exception) def test_update_config_map(self): """ From 9eb20f376f280ed3fe88a987b26fc4564958050d Mon Sep 17 00:00:00 2001 From: "Mitch Harding (the weird one)" Date: Fri, 8 Dec 2023 15:20:08 -0500 Subject: [PATCH 06/12] Remove empty file: rollback_handler.py --- cray_product_catalog/migration/rollback_handler.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 cray_product_catalog/migration/rollback_handler.py diff --git a/cray_product_catalog/migration/rollback_handler.py b/cray_product_catalog/migration/rollback_handler.py deleted file mode 100644 index e69de29b..00000000 From 89d55e828c35ee015746d00b0f0a3fabde4a6cb6 Mon Sep 17 00:00:00 2001 From: "Mitch Harding (the weird one)" Date: Wed, 3 Jan 2024 14:08:34 -0500 Subject: [PATCH 07/12] Update copyright headers for new year --- .../templates/configmap-hook.yaml | 2 +- .../templates/configmap.yaml | 2 +- charts/cray-product-catalog/values.yaml | 2 +- cray_product_catalog/catalog_delete.py | 2 +- cray_product_catalog/catalog_update.py | 2 +- cray_product_catalog/constants.py | 2 +- cray_product_catalog/migration/__init__.py | 2 +- .../migration/config_map_data_handler.py | 358 ++--- .../migration/exit_handler.py | 2 +- cray_product_catalog/migration/kube_apis.py | 2 +- cray_product_catalog/migration/main.py | 274 ++-- cray_product_catalog/query.py | 2 +- cray_product_catalog/schema/schema.yaml | 2 +- .../util/catalog_data_helper.py | 136 +- setup.py | 2 +- tests/migration/migration_mock.py | 260 ++-- .../migration/test_config_map_data_handler.py | 1344 ++++++++--------- tests/migration/test_exit_handler.py | 2 +- tests/mock_update_catalog.py | 214 +-- tests/mocks.py | 2 +- tests/test_catalog_delete.py | 2 +- tests/test_query.py | 2 +- tests/test_update_catalog.py | 384 ++--- tests/util/test_data_catalog_helper.py | 248 +-- 24 files changed, 1625 insertions(+), 1625 deletions(-) diff --git a/charts/cray-product-catalog/templates/configmap-hook.yaml b/charts/cray-product-catalog/templates/configmap-hook.yaml index 4a3ea393..07023ddb 100644 --- a/charts/cray-product-catalog/templates/configmap-hook.yaml +++ b/charts/cray-product-catalog/templates/configmap-hook.yaml @@ -1,7 +1,7 @@ {{/* MIT License -(C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +(C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), diff --git a/charts/cray-product-catalog/templates/configmap.yaml b/charts/cray-product-catalog/templates/configmap.yaml index 3808c9a3..bc5f7f3c 100644 --- a/charts/cray-product-catalog/templates/configmap.yaml +++ b/charts/cray-product-catalog/templates/configmap.yaml @@ -1,7 +1,7 @@ {{/* MIT License -(C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +(C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), diff --git a/charts/cray-product-catalog/values.yaml b/charts/cray-product-catalog/values.yaml index 85b3f6a6..aca239d9 100644 --- a/charts/cray-product-catalog/values.yaml +++ b/charts/cray-product-catalog/values.yaml @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/catalog_delete.py b/cray_product_catalog/catalog_delete.py index 76f5baa3..35fc8e67 100755 --- a/cray_product_catalog/catalog_delete.py +++ b/cray_product_catalog/catalog_delete.py @@ -2,7 +2,7 @@ # # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/catalog_update.py b/cray_product_catalog/catalog_update.py index 5a40631e..4e5d012a 100755 --- a/cray_product_catalog/catalog_update.py +++ b/cray_product_catalog/catalog_update.py @@ -2,7 +2,7 @@ # # MIT License # -# (C) Copyright 2020-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2020-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/constants.py b/cray_product_catalog/constants.py index 1627a9fe..e152b3a9 100644 --- a/cray_product_catalog/constants.py +++ b/cray_product_catalog/constants.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/migration/__init__.py b/cray_product_catalog/migration/__init__.py index 2a44595d..2cfd2f84 100644 --- a/cray_product_catalog/migration/__init__.py +++ b/cray_product_catalog/migration/__init__.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py index 7d241373..0cad7497 100644 --- a/cray_product_catalog/migration/config_map_data_handler.py +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -1,179 +1,179 @@ -# -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - -""" -File contains functions to -Split data in `cray-product-catalog` ConfigMap -Create temporary and product ConfigMaps -Rename ConfigMap -""" - -import logging -import yaml - - -from cray_product_catalog.logging import configure_logging -from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name -from cray_product_catalog.migration.kube_apis import KubernetesApi -from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL -from cray_product_catalog.migration import ( - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE -) - -LOGGER = logging.getLogger(__name__) - - -class ConfigMapDataHandler: - """ Class to migrate ConfigMap data to multiple ConfigMaps """ - - def __init__(self) -> None: - self.k8s_obj = KubernetesApi() - self.config_map_data_replica = {} - configure_logging() - - def create_product_config_maps(self, product_config_map_data_list): - """Create new product ConfigMap for each product in product_config_map_data_list - - Args: - product_config_map_data_list (list): list of data to be stored in each product ConfigMap - """ - for product_data in product_config_map_data_list: - product_name = list(product_data.keys())[0] - LOGGER.debug("Creating ConfigMap for product %s", product_name) - prod_cm_name = format_product_cm_name(PRODUCT_CATALOG_CONFIG_MAP_NAME, product_name) - - if not self.k8s_obj.create_config_map(prod_cm_name, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, product_data, - PRODUCT_CATALOG_CONFIG_MAP_LABEL): - LOGGER.info("Failed to create product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - prod_cm_name) - return False - LOGGER.info("Created product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, prod_cm_name) - return True - - def create_temp_config_map(self, config_map_data): - """Create temporary main ConfigMap `cray-product-catalog-temp` - - Args: - config_map_data (dict): Data to be stored in the ConfigMap `cray-product-catalog-temp` - """ - - if self.k8s_obj.create_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - config_map_data, PRODUCT_CATALOG_CONFIG_MAP_LABEL): - LOGGER.info("Created temp ConfigMap %s/%s", - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) - return True - LOGGER.error("Creating ConfigMap %s/%s failed", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) - return False - - def migrate_config_map_data(self, config_map_data): - """Migrate cray-product-catalog ConfigMap data to multiple product ConfigMaps with - `component_versions` data for each product - - Returns: - {Dictionary, List}: Main ConfigMap Data, list of product ConfigMap data - """ - - LOGGER.info( - "Migrating data in ConfigMap=%s in namespace=%s to multiple ConfigMaps", - PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE - ) - # Backed up ConfigMap Data - self.config_map_data_replica = config_map_data - # Get list of products - products_list = list(config_map_data.keys()) - product_config_map_data_list = [] - for product in products_list: - product_data = yaml.safe_load(config_map_data[product]) - # Get list of versions associated with product - product_versions_list = list(product_data.keys()) - product_versions_data = {} - main_versions_data = {} - for version_data in product_versions_list: - LOGGER.debug("Splitting cray-product-catalog data for product %s", product) - main_cm_data, prod_cm_data = split_catalog_data(product_data[version_data]) - # prod_cm_data is not an empty dictionary - if prod_cm_data: - product_config_map_data = {} - product_versions_data[version_data] = prod_cm_data - # main_cm_data is not an empty dictionary - if main_cm_data: - main_versions_data[version_data] = main_cm_data - # If `component_versions` data exists for a product, create new product ConfigMap - if product_versions_data: - product_config_map_data[product] = yaml.safe_dump(product_versions_data, default_flow_style=False) - # create_product_config_map(k8s_obj, product, product_config_map_data) - product_config_map_data_list.append(product_config_map_data) - # Data with key other than `component_versions` should be updated to config_map_data, - # so that new main ConfigMap will not have data with key `component_versions` - if main_versions_data: - config_map_data[product] = yaml.safe_dump(main_versions_data, default_flow_style=False) - else: - config_map_data[product] = '' - return config_map_data, product_config_map_data_list - - def rename_config_map(self, rename_from, rename_to, namespace, label): - """ Renaming is actually deleting one ConfigMap and then updating the name of other ConfigMap and patching it. - :param str rename_from: Name of ConfigMap to rename - :param str rename_to: Name of ConfigMap after rename - :param str namespace: Namespace in which ConfigMap has to be updated - :param dict label: Label of ConfigMap to be renamed - :return: bool, If Success True else False - """ - - if not self.k8s_obj.delete_config_map(rename_to, namespace): - LOGGER.error("Failed to delete ConfigMap %s", rename_to) - - return False - attempt = 0 - del_failed = False - - while attempt < 10: - attempt += 1 - response = self.k8s_obj.read_config_map(rename_from, namespace) - if not response: - LOGGER.error("Failed to read ConfigMap %s, retrying..", rename_from) - continue - cm_data = response.data - - if self.k8s_obj.create_config_map(rename_to, namespace, cm_data, label): - if self.k8s_obj.delete_config_map(rename_from, namespace): - LOGGER.info("Renaming ConfigMap successful") - return True - LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) - del_failed = True - break - LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) - continue - # Since only delete of backed up ConfigMap failed, retrying only delete operation - attempt = 0 - if del_failed: - while attempt < 10: - attempt += 1 - if self.k8s_obj.delete_config_map(rename_from, namespace): - break - LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) - # Returning success as migration is successful only backed up ConfigMap is not deleted. - LOGGER.info("Renaming ConfigMap successful") - return True - return False +# +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +File contains functions to +Split data in `cray-product-catalog` ConfigMap +Create temporary and product ConfigMaps +Rename ConfigMap +""" + +import logging +import yaml + + +from cray_product_catalog.logging import configure_logging +from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name +from cray_product_catalog.migration.kube_apis import KubernetesApi +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL +from cray_product_catalog.migration import ( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE +) + +LOGGER = logging.getLogger(__name__) + + +class ConfigMapDataHandler: + """ Class to migrate ConfigMap data to multiple ConfigMaps """ + + def __init__(self) -> None: + self.k8s_obj = KubernetesApi() + self.config_map_data_replica = {} + configure_logging() + + def create_product_config_maps(self, product_config_map_data_list): + """Create new product ConfigMap for each product in product_config_map_data_list + + Args: + product_config_map_data_list (list): list of data to be stored in each product ConfigMap + """ + for product_data in product_config_map_data_list: + product_name = list(product_data.keys())[0] + LOGGER.debug("Creating ConfigMap for product %s", product_name) + prod_cm_name = format_product_cm_name(PRODUCT_CATALOG_CONFIG_MAP_NAME, product_name) + + if not self.k8s_obj.create_config_map(prod_cm_name, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, product_data, + PRODUCT_CATALOG_CONFIG_MAP_LABEL): + LOGGER.info("Failed to create product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + prod_cm_name) + return False + LOGGER.info("Created product ConfigMap %s/%s", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, prod_cm_name) + return True + + def create_temp_config_map(self, config_map_data): + """Create temporary main ConfigMap `cray-product-catalog-temp` + + Args: + config_map_data (dict): Data to be stored in the ConfigMap `cray-product-catalog-temp` + """ + + if self.k8s_obj.create_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + config_map_data, PRODUCT_CATALOG_CONFIG_MAP_LABEL): + LOGGER.info("Created temp ConfigMap %s/%s", + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) + return True + LOGGER.error("Creating ConfigMap %s/%s failed", PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, CONFIG_MAP_TEMP) + return False + + def migrate_config_map_data(self, config_map_data): + """Migrate cray-product-catalog ConfigMap data to multiple product ConfigMaps with + `component_versions` data for each product + + Returns: + {Dictionary, List}: Main ConfigMap Data, list of product ConfigMap data + """ + + LOGGER.info( + "Migrating data in ConfigMap=%s in namespace=%s to multiple ConfigMaps", + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + # Backed up ConfigMap Data + self.config_map_data_replica = config_map_data + # Get list of products + products_list = list(config_map_data.keys()) + product_config_map_data_list = [] + for product in products_list: + product_data = yaml.safe_load(config_map_data[product]) + # Get list of versions associated with product + product_versions_list = list(product_data.keys()) + product_versions_data = {} + main_versions_data = {} + for version_data in product_versions_list: + LOGGER.debug("Splitting cray-product-catalog data for product %s", product) + main_cm_data, prod_cm_data = split_catalog_data(product_data[version_data]) + # prod_cm_data is not an empty dictionary + if prod_cm_data: + product_config_map_data = {} + product_versions_data[version_data] = prod_cm_data + # main_cm_data is not an empty dictionary + if main_cm_data: + main_versions_data[version_data] = main_cm_data + # If `component_versions` data exists for a product, create new product ConfigMap + if product_versions_data: + product_config_map_data[product] = yaml.safe_dump(product_versions_data, default_flow_style=False) + # create_product_config_map(k8s_obj, product, product_config_map_data) + product_config_map_data_list.append(product_config_map_data) + # Data with key other than `component_versions` should be updated to config_map_data, + # so that new main ConfigMap will not have data with key `component_versions` + if main_versions_data: + config_map_data[product] = yaml.safe_dump(main_versions_data, default_flow_style=False) + else: + config_map_data[product] = '' + return config_map_data, product_config_map_data_list + + def rename_config_map(self, rename_from, rename_to, namespace, label): + """ Renaming is actually deleting one ConfigMap and then updating the name of other ConfigMap and patching it. + :param str rename_from: Name of ConfigMap to rename + :param str rename_to: Name of ConfigMap after rename + :param str namespace: Namespace in which ConfigMap has to be updated + :param dict label: Label of ConfigMap to be renamed + :return: bool, If Success True else False + """ + + if not self.k8s_obj.delete_config_map(rename_to, namespace): + LOGGER.error("Failed to delete ConfigMap %s", rename_to) + + return False + attempt = 0 + del_failed = False + + while attempt < 10: + attempt += 1 + response = self.k8s_obj.read_config_map(rename_from, namespace) + if not response: + LOGGER.error("Failed to read ConfigMap %s, retrying..", rename_from) + continue + cm_data = response.data + + if self.k8s_obj.create_config_map(rename_to, namespace, cm_data, label): + if self.k8s_obj.delete_config_map(rename_from, namespace): + LOGGER.info("Renaming ConfigMap successful") + return True + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) + del_failed = True + break + LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) + continue + # Since only delete of backed up ConfigMap failed, retrying only delete operation + attempt = 0 + if del_failed: + while attempt < 10: + attempt += 1 + if self.k8s_obj.delete_config_map(rename_from, namespace): + break + LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) + # Returning success as migration is successful only backed up ConfigMap is not deleted. + LOGGER.info("Renaming ConfigMap successful") + return True + return False diff --git a/cray_product_catalog/migration/exit_handler.py b/cray_product_catalog/migration/exit_handler.py index 8759835d..e04ea925 100644 --- a/cray_product_catalog/migration/exit_handler.py +++ b/cray_product_catalog/migration/exit_handler.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/migration/kube_apis.py b/cray_product_catalog/migration/kube_apis.py index 96585ecc..2c3cdcd6 100644 --- a/cray_product_catalog/migration/kube_apis.py +++ b/cray_product_catalog/migration/kube_apis.py @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/migration/main.py b/cray_product_catalog/migration/main.py index 5d4f0698..4a2111e9 100644 --- a/cray_product_catalog/migration/main.py +++ b/cray_product_catalog/migration/main.py @@ -1,137 +1,137 @@ -#!/usr/bin/env python3 -# -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - -""" -This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller -ConfigMaps with each product's `component_versions` data in its own product ConfigMap. -If the split is not succesful then it rollbacks to its initial state where ConfigMap -`cray-product-catalog` will contain complete data which includes `component-versions` -""" - -import logging - -from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL -from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler -from cray_product_catalog.migration import ( - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - CONFIG_MAP_TEMP -) -from cray_product_catalog.migration.exit_handler import ExitHandler - -LOGGER = logging.getLogger(__name__) - - -def main(): - """Main function""" - LOGGER.info("Migrating %s ConfigMap data to multiple product ConfigMaps", PRODUCT_CATALOG_CONFIG_MAP_NAME) - config_map_obj = ConfigMapDataHandler() - exit_handler = ExitHandler() - attempt = 0 - max_attempts = 2 - - while (attempt < max_attempts): - attempt += 1 - migration_failed = False - init_resource_version = '' - curr_resource_version = '' - response = config_map_obj.k8s_obj.read_config_map( - PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE - ) - if response: - if not response.metadata.resource_version: - LOGGER.error("Error reading resourceVersion, exiting migration process...") - raise SystemExit(1) - init_resource_version = response.metadata.resource_version - if not response.data: - LOGGER.error("Error reading ConfigMap data, exiting migration process...") - raise SystemExit(1) - config_map_data = response.data - else: - LOGGER.error("Error reading ConfigMap, exiting migration process...") - raise SystemExit(1) - - try: - main_config_map_data, product_config_map_data_list = config_map_obj.migrate_config_map_data(config_map_data) - except Exception: - LOGGER.error("Failed to split ConfigMap Data, exiting migration process...") - raise SystemExit(1) - - # Create ConfigMaps for each product with `component_versions` data - if not config_map_obj.create_product_config_maps(product_config_map_data_list): - LOGGER.info("Calling rollback handler...") - exit_handler.rollback() - raise SystemExit(1) - # Create temporary main ConfigMap with all data except `component_versions` for all products - if not config_map_obj.create_temp_config_map(main_config_map_data): - LOGGER.info("Calling rollback handler...") - exit_handler.rollback() - raise SystemExit(1) - - LOGGER.info("Verifying resource_version value is same to confirm there is no change in %s", - PRODUCT_CATALOG_CONFIG_MAP_NAME) - - response = config_map_obj.k8s_obj.read_config_map( - PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE - ) - if response: - if not response.metadata.resource_version: - LOGGER.error("Error reading resourceVersion, exiting migration process...") - exit_handler.rollback() - raise SystemExit(1) - curr_resource_version = response.metadata.resource_version - - if curr_resource_version != init_resource_version: - migration_failed = True - LOGGER.info("resource_version has changed, so cannot rename %s ConfigMap to %s ConfigMap", - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) - LOGGER.info("Re-trying migration process...") - exit_handler.rollback() - continue - break - - if migration_failed: - LOGGER.info("ConfigMap %s is modified by other process, exiting migration process...", - PRODUCT_CATALOG_CONFIG_MAP_NAME) - raise SystemExit(1) - - LOGGER.info("Renaming %s ConfigMap name to %s ConfigMap", - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) - - # Creating main ConfigMap `cray-product-catalog` using the data in `cray-product-catalog-temp` - if config_map_obj.rename_config_map( - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, PRODUCT_CATALOG_CONFIG_MAP_LABEL - ): - LOGGER.info("Migration successful") - else: - LOGGER.info("Renaming %s to %s ConfigMap failed, calling rollback handler...", - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) - exit_handler.rollback() - raise SystemExit(1) - - -if __name__ == "__main__": - main() +#!/usr/bin/env python3 +# +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +This script splits the data in ConfigMap `cray-product-catalog` into multiple smaller +ConfigMaps with each product's `component_versions` data in its own product ConfigMap. +If the split is not succesful then it rollbacks to its initial state where ConfigMap +`cray-product-catalog` will contain complete data which includes `component-versions` +""" + +import logging + +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL +from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler +from cray_product_catalog.migration import ( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + CONFIG_MAP_TEMP +) +from cray_product_catalog.migration.exit_handler import ExitHandler + +LOGGER = logging.getLogger(__name__) + + +def main(): + """Main function""" + LOGGER.info("Migrating %s ConfigMap data to multiple product ConfigMaps", PRODUCT_CATALOG_CONFIG_MAP_NAME) + config_map_obj = ConfigMapDataHandler() + exit_handler = ExitHandler() + attempt = 0 + max_attempts = 2 + + while (attempt < max_attempts): + attempt += 1 + migration_failed = False + init_resource_version = '' + curr_resource_version = '' + response = config_map_obj.k8s_obj.read_config_map( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + if response: + if not response.metadata.resource_version: + LOGGER.error("Error reading resourceVersion, exiting migration process...") + raise SystemExit(1) + init_resource_version = response.metadata.resource_version + if not response.data: + LOGGER.error("Error reading ConfigMap data, exiting migration process...") + raise SystemExit(1) + config_map_data = response.data + else: + LOGGER.error("Error reading ConfigMap, exiting migration process...") + raise SystemExit(1) + + try: + main_config_map_data, product_config_map_data_list = config_map_obj.migrate_config_map_data(config_map_data) + except Exception: + LOGGER.error("Failed to split ConfigMap Data, exiting migration process...") + raise SystemExit(1) + + # Create ConfigMaps for each product with `component_versions` data + if not config_map_obj.create_product_config_maps(product_config_map_data_list): + LOGGER.info("Calling rollback handler...") + exit_handler.rollback() + raise SystemExit(1) + # Create temporary main ConfigMap with all data except `component_versions` for all products + if not config_map_obj.create_temp_config_map(main_config_map_data): + LOGGER.info("Calling rollback handler...") + exit_handler.rollback() + raise SystemExit(1) + + LOGGER.info("Verifying resource_version value is same to confirm there is no change in %s", + PRODUCT_CATALOG_CONFIG_MAP_NAME) + + response = config_map_obj.k8s_obj.read_config_map( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE + ) + if response: + if not response.metadata.resource_version: + LOGGER.error("Error reading resourceVersion, exiting migration process...") + exit_handler.rollback() + raise SystemExit(1) + curr_resource_version = response.metadata.resource_version + + if curr_resource_version != init_resource_version: + migration_failed = True + LOGGER.info("resource_version has changed, so cannot rename %s ConfigMap to %s ConfigMap", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + LOGGER.info("Re-trying migration process...") + exit_handler.rollback() + continue + break + + if migration_failed: + LOGGER.info("ConfigMap %s is modified by other process, exiting migration process...", + PRODUCT_CATALOG_CONFIG_MAP_NAME) + raise SystemExit(1) + + LOGGER.info("Renaming %s ConfigMap name to %s ConfigMap", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + + # Creating main ConfigMap `cray-product-catalog` using the data in `cray-product-catalog-temp` + if config_map_obj.rename_config_map( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ): + LOGGER.info("Migration successful") + else: + LOGGER.info("Renaming %s to %s ConfigMap failed, calling rollback handler...", + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME) + exit_handler.rollback() + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/cray_product_catalog/query.py b/cray_product_catalog/query.py index af43daa7..b5e7d112 100644 --- a/cray_product_catalog/query.py +++ b/cray_product_catalog/query.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/schema/schema.yaml b/cray_product_catalog/schema/schema.yaml index 6b3dd5b4..b985a003 100644 --- a/cray_product_catalog/schema/schema.yaml +++ b/cray_product_catalog/schema/schema.yaml @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/cray_product_catalog/util/catalog_data_helper.py b/cray_product_catalog/util/catalog_data_helper.py index 672c975c..51642b3d 100755 --- a/cray_product_catalog/util/catalog_data_helper.py +++ b/cray_product_catalog/util/catalog_data_helper.py @@ -1,68 +1,68 @@ -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. - -""" -Contains a utility function for splitting `cray-product-catalog` data. -""" - -import re - -from cray_product_catalog.constants import ( - PRODUCT_CM_FIELDS -) - - -def split_catalog_data(data): - """Split the passed data into data needed by main and product ConfigMaps.""" - all_unique_keys = set(data.keys()) - comm_keys_bw_cms = all_unique_keys.intersection(PRODUCT_CM_FIELDS) - - # If none of the PRODUCT_CM_FIELDS are available in all_unique_keys, then - # return empty dict as second return value - if not comm_keys_bw_cms: - return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, {} - return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, \ - {key: data[key] for key in comm_keys_bw_cms} - - -def format_product_cm_name(config_map, product): - """Formatting PRODUCT_CONFIG_NAME based on the product name passed and the same is used as key - under data in the ConfigMap. - The name of a ConfigMap must be a valid DNS subdomain name. In addition, it must obey the following rules: - - contain no more than 253 characters - - contain only lowercase alphanumeric characters, hypens('-'), or periods('.') - - start with an alphanumeric character - - end with an alphanumeric character - The product name, which is a key under the data, must obey the following rule: - - contain only alphanumeric characters, hyphens ('-'), underscores ('_'), and periods ('.') - Because the product name can have uppercase characters and underscores ('_'), which are - prohibited in the ConfigMap name, we convert underscores ('_') to hyphens ('-') and uppercase - to lowercase. - """ - pat = re.compile('^([a-z0-9])*[a-z0-9.-]*([a-z0-9])$') - prod_config_map = config_map + '-' + product.replace('_', '-').lower() - - if len(prod_config_map) > 253: - return '' - if not re.fullmatch(pat, prod_config_map): - return '' - return prod_config_map +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +""" +Contains a utility function for splitting `cray-product-catalog` data. +""" + +import re + +from cray_product_catalog.constants import ( + PRODUCT_CM_FIELDS +) + + +def split_catalog_data(data): + """Split the passed data into data needed by main and product ConfigMaps.""" + all_unique_keys = set(data.keys()) + comm_keys_bw_cms = all_unique_keys.intersection(PRODUCT_CM_FIELDS) + + # If none of the PRODUCT_CM_FIELDS are available in all_unique_keys, then + # return empty dict as second return value + if not comm_keys_bw_cms: + return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, {} + return {key: data[key] for key in all_unique_keys - PRODUCT_CM_FIELDS}, \ + {key: data[key] for key in comm_keys_bw_cms} + + +def format_product_cm_name(config_map, product): + """Formatting PRODUCT_CONFIG_NAME based on the product name passed and the same is used as key + under data in the ConfigMap. + The name of a ConfigMap must be a valid DNS subdomain name. In addition, it must obey the following rules: + - contain no more than 253 characters + - contain only lowercase alphanumeric characters, hypens('-'), or periods('.') + - start with an alphanumeric character + - end with an alphanumeric character + The product name, which is a key under the data, must obey the following rule: + - contain only alphanumeric characters, hyphens ('-'), underscores ('_'), and periods ('.') + Because the product name can have uppercase characters and underscores ('_'), which are + prohibited in the ConfigMap name, we convert underscores ('_') to hyphens ('-') and uppercase + to lowercase. + """ + pat = re.compile('^([a-z0-9])*[a-z0-9.-]*([a-z0-9])$') + prod_config_map = config_map + '-' + product.replace('_', '-').lower() + + if len(prod_config_map) > 253: + return '' + if not re.fullmatch(pat, prod_config_map): + return '' + return prod_config_map diff --git a/setup.py b/setup.py index 70a3b169..baf0dec5 100644 --- a/setup.py +++ b/setup.py @@ -1,7 +1,7 @@ # # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/tests/migration/migration_mock.py b/tests/migration/migration_mock.py index bf3d7a6e..3fe74fc6 100755 --- a/tests/migration/migration_mock.py +++ b/tests/migration/migration_mock.py @@ -1,130 +1,130 @@ -# -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - -""" -Mock data for ConfigMapDataHandler -""" - -INITIAL_MAIN_CM_DATA = { - 'HFP-firmware': """ - 22.10.2: - component_versions: - docker: - - name: cray-product-catalog-update - version: 0.1.3 - 23.01.1: - component_versions: - docker: - - name: cray-product-catalog-update - version: 0.1.3""", - 'analytics': """ - 1.4.18: - component_versions: - s3: - - bucket: boot-images - key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs - configuration: - clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git - commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 - import_branch: cray/analytics/1.4.18 - import_date: 2023-02-28 04:37:34.914586 - ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git - 1.4.20: - component_versions: - s3: - - bucket: boot-images - key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs - configuration: - clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git - commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd - import_branch: cray/analytics/1.4.20 - import_date: 2023-03-23 16:55:22.295666 - ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git""" -} - -MAIN_CM_DATA_EXPECTED = { - 'HFP-firmware': '', - 'analytics': """1.4.18: - configuration: - clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git - commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 - import_branch: cray/analytics/1.4.18 - import_date: 2023-02-28 04:37:34.914586 - ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git -1.4.20: - configuration: - clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git - commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd - import_branch: cray/analytics/1.4.20 - import_date: 2023-03-23 16:55:22.295666 - ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git\n""" -} - -PROD_CM_DATA_LIST_EXPECTED = [ - { - 'HFP-firmware': """22.10.2: - component_versions: - docker: - - name: cray-product-catalog-update - version: 0.1.3 -23.01.1: - component_versions: - docker: - - name: cray-product-catalog-update - version: 0.1.3\n""" - }, - { - 'analytics': """1.4.18: - component_versions: - s3: - - bucket: boot-images - key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs -1.4.20: - component_versions: - s3: - - bucket: boot-images - key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs\n""" - } -] - - -class MockYaml: - """Mock class created to test test_create_product_catalog_invalid_product_data.""" - - def __init__(self, resource_version): - """Initialize metadata and data object of ConfigMap data.""" - self.metadata = MetaData(resource_version) - self.data = { - 'sat': '\t', - } - - -class MetaData: - """ - Class to provide dummy metadata object with name and resource_version - """ - def __init__(self, resource_version): - """Initialize ConfigMap name and resoource_version""" - self.name = 'cray-product-catalog' - self.resource_version = resource_version +# +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Mock data for ConfigMapDataHandler +""" + +INITIAL_MAIN_CM_DATA = { + 'HFP-firmware': """ + 22.10.2: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3 + 23.01.1: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3""", + 'analytics': """ + 1.4.18: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 + import_branch: cray/analytics/1.4.18 + import_date: 2023-02-28 04:37:34.914586 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git + 1.4.20: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd + import_branch: cray/analytics/1.4.20 + import_date: 2023-03-23 16:55:22.295666 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git""" +} + +MAIN_CM_DATA_EXPECTED = { + 'HFP-firmware': '', + 'analytics': """1.4.18: + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 4f1aee2086b58b319d4a9ee167086004fca09e47 + import_branch: cray/analytics/1.4.18 + import_date: 2023-02-28 04:37:34.914586 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git +1.4.20: + configuration: + clone_url: https://vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net/vcs/cray/analytics-config-management.git + commit: 8424f5f97f12a3403afc57ac55deca0dadc8f3dd + import_branch: cray/analytics/1.4.20 + import_date: 2023-03-23 16:55:22.295666 + ssh_url: git@vcs.cmn.lemondrop.hpc.amslabs.hpecorp.net:cray/analytics-config-management.git\n""" +} + +PROD_CM_DATA_LIST_EXPECTED = [ + { + 'HFP-firmware': """22.10.2: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3 +23.01.1: + component_versions: + docker: + - name: cray-product-catalog-update + version: 0.1.3\n""" + }, + { + 'analytics': """1.4.18: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.18.squashfs +1.4.20: + component_versions: + s3: + - bucket: boot-images + key: Analytics/Cray-Analytics.x86_64-1.4.20.squashfs\n""" + } +] + + +class MockYaml: + """Mock class created to test test_create_product_catalog_invalid_product_data.""" + + def __init__(self, resource_version): + """Initialize metadata and data object of ConfigMap data.""" + self.metadata = MetaData(resource_version) + self.data = { + 'sat': '\t', + } + + +class MetaData: + """ + Class to provide dummy metadata object with name and resource_version + """ + def __init__(self, resource_version): + """Initialize ConfigMap name and resoource_version""" + self.name = 'cray-product-catalog' + self.resource_version = resource_version diff --git a/tests/migration/test_config_map_data_handler.py b/tests/migration/test_config_map_data_handler.py index 1bebd95f..728dd373 100755 --- a/tests/migration/test_config_map_data_handler.py +++ b/tests/migration/test_config_map_data_handler.py @@ -1,672 +1,672 @@ -# -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - -""" -Tests for validating ConfigMapDataHandler -""" - -import unittest -from unittest.mock import patch, call, Mock -from typing import Dict, List - -from cray_product_catalog.migration.main import main -from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler -from cray_product_catalog.constants import ( - PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PRODUCT_CATALOG_CONFIG_MAP_LABEL -) -from cray_product_catalog.migration import CONFIG_MAP_TEMP -from tests.migration.migration_mock import ( - MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED, INITIAL_MAIN_CM_DATA, MockYaml -) - - -def mock_split_catalog_data(): - """Mocking function to return custom data""" - return MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED - - -class TestConfigMapDataHandler(unittest.TestCase): - """ Tests for validating ConfigMapDataHandler """ - - def setUp(self) -> None: - """Set up mocks.""" - self.mock_load_k8s_mig = patch('cray_product_catalog.migration.kube_apis.load_k8s').start() - self.mock_corev1api_mig = patch('cray_product_catalog.migration.kube_apis.client.CoreV1Api').start() - self.mock_ApiClient_mig = patch('cray_product_catalog.migration.kube_apis.ApiClient').start() - self.mock_client_mig = patch('cray_product_catalog.migration.kube_apis.client').start() - - self.mock_k8api_read = patch( - 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.read_config_map').start() - self.mock_k8api_create = patch( - 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.create_config_map').start() - self.mock_k8api_delete = patch( - 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.delete_config_map').start() - - def tearDown(self) -> None: - patch.stopall() - - def test_migrate_config_map_data(self): - """ Validating the migration of data into multiple product ConfigMaps data """ - - main_cm_data: Dict - prod_cm_data_list: List - cmdh = ConfigMapDataHandler() - main_cm_data, prod_cm_data_list = cmdh.migrate_config_map_data(INITIAL_MAIN_CM_DATA) - - self.assertEqual(main_cm_data, MAIN_CM_DATA_EXPECTED) - self.assertEqual(prod_cm_data_list, PROD_CM_DATA_LIST_EXPECTED) - - def test_create_product_config_maps(self): - """ Validating product ConfigMaps are created """ - - # mock some additional functions - self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() - - with self.assertLogs() as captured: - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) - - dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] - - self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice - call( - dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), - call( - dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), - ] - ) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") - - self.assertEqual( - captured.records[1].getMessage(), - f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") - - def test_create_second_product_config_map_failed(self): - """ Validating scenario where creation of second product ConfigMap failed """ - - # mock some additional functions - self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() - - with self.assertLogs() as captured: - self.mock_k8api_create.side_effect = [True, False] - - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) - - dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] - - self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice - call( - dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call( - dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), - ] - ) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") - - self.assertEqual( - captured.records[1].getMessage(), - f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") - - def test_create_first_product_config_map_failed(self): - """ Validating scenario where creation of first product ConfigMap failed. """ - - # mock some additional functions - self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() - - with self.assertLogs() as captured: - self.mock_k8api_create.side_effect = [False, False] - - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) - - dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] - - self.mock_k8api_create.assert_called_once_with( - dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL - ) - - # Verify the exact log message - self.assertEqual(len(captured.records), 1) - - self.assertEqual( - captured.records[0].getMessage(), - f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") - - def test_create_temp_config_map(self): - """ Validating temp main ConfigMap is created """ - - # mock some additional functions - self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() - - with self.assertLogs(level='DEBUG') as captured: - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) - - self.mock_k8api_create.assert_called_once_with( - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL - ) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Created temp ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP}") - - def test_create_temp_config_map_failed(self): - """ Validating temp main ConfigMap creation failed """ - - # mock some additional functions - self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() - self.mock_k8api_create.return_value = False - - with self.assertLogs() as captured: - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) - - self.mock_k8api_create.assert_called_once_with( - CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL - ) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Creating ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP} failed") - - def test_rename_config_map(self): - """ Validating product ConfigMaps are created """ - - with self.assertLogs(level="DEBUG") as captured: - # call method under test - self.mock_k8api_delete.return_value = True - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - - cmdh = ConfigMapDataHandler() - cmdh.rename_config_map(rename_from=CONFIG_MAP_TEMP, - rename_to=PRODUCT_CATALOG_CONFIG_MAP_NAME, - namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - label=PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - call(CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - ] - ) - - self.mock_k8api_read.assert_called_once_with(CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) - - self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, - PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - "Renaming ConfigMap successful") - - def test_rename_config_map_failed_1(self): - """ Validating rename ConfigMap failure scenario where: - deleting cray-product-catalog ConfigMap failed. """ - - with self.assertLogs() as captured: - self.mock_k8api_delete.side_effect = [False, False] - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.rename_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Failed to delete ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}") - - def test_rename_config_map_failed_2(self): - """ Validating rename ConfigMap failure scenario where: - creating cray-product-catalog ConfigMap failed. """ - - with self.assertLogs(level="DEBUG") as captured: - self.mock_k8api_create.return_value = False - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.rename_config_map(CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) - calls = [ # Delete ConfigMap called twice - call( - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - call( - CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - ] - self.mock_k8api_create.assert_has_calls(calls=[ - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), - - ]) - - # Verify the exact log message - self.assertEqual( - captured.records[0].getMessage(), - f"Failed to create ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}, retrying..") - - def test_rename_config_map_failed_3(self): - """ Validating rename ConfigMap failure scenario where: - first operation of deleting cray-product-catalog-temp ConfigMap failed but later passed. """ - - with self.assertLogs(level="DEBUG") as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_k8api_delete.side_effect = [True, False, True] - - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.rename_config_map(CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice - call( - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - call( - CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - ]) - self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - # Verify the exact log message - self.assertEqual(2, len(captured.records)) - self.assertEqual( - captured.records[0].getMessage(), - f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") - self.assertEqual( - captured.records[1].getMessage(), - "Renaming ConfigMap successful") - - def test_rename_config_map_failed_4(self): - """ Validating rename ConfigMap failure scenario where: - everytime deleting cray-product-catalog-temp ConfigMap failed. """ - - with self.assertLogs(level="DEBUG") as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_k8api_delete.side_effect = [True, False, False, False, False, False, False, False, - False, False, False, False] - - # call method under test - cmdh = ConfigMapDataHandler() - cmdh.rename_config_map(CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - self.mock_k8api_delete.assert_has_calls(calls=[ - call( - PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - call( - CONFIG_MAP_TEMP, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), - ]) - self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, - PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, - MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) - - # Verify the exact log message - self.assertEqual(12, len(captured.records)) - self.assertEqual( - captured.records[-2].getMessage(), - f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") - self.assertEqual( - captured.records[-1].getMessage(), - "Renaming ConfigMap successful") - - def test_main_for_successful_migration(self): - """Validating that migration is successful""" - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertLogs(level="DEBUG") as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = True - self.mock_create_temp_cm.return_value = True - self.mock_rename_cm.return_value = True - - # Call method under test - main() - - self.assertEqual( - captured.records[-1].getMessage(), - "Migration successful") - - def test_main_failed_1(self): - """Validating that migration failed as renaming failed""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = True - self.mock_create_temp_cm.return_value = True - self.mock_rename_cm.return_value = False - - # Call method under test - main() - - self.assertTrue( - "Renaming cray-product-catalog-temp to cray-product-catalog ConfigMap failed, " - "calling rollback handler..." in captured.exception - ) - - self.assertTrue( - "Rollback successful" in captured.exception - ) - - def test_main_failed_2(self): - """Validating that migration failed as create temp cm failed""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = True - self.mock_create_temp_cm.return_value = False - - # Call method under test - main() - - self.assertTrue( - "Rollback successful" in captured.exception - ) - - self.mock_rename_cm.assert_not_called() - - def test_main_failed_3(self): - """Validating that migration failed as create product ConfigMaps failed""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = False - - # Call method under test - main() - - self.assertTrue( - "Rollback successful" in captured.exception - ) - - self.mock_create_temp_cm.assert_not_called() - self.mock_rename_cm.assert_not_called() - - def test_main_failed_4(self): - """Validating that migration failed as migrate_config_map failed with exception""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) - self.mock_migrate_config_map.return_value = Exception() - - # Call method under test - main() - - self.assertTrue( - "Failed to split ConfigMap Data, exiting migration process..." in captured.exception - ) - - self.mock_create_prod_cms.assert_not_called() - self.mock_create_temp_cm.assert_not_called() - self.mock_rename_cm.assert_not_called() - - def test_main_failed_6(self): - """Validating that migration failed as read_config_map returned empty response / data.""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.return_value = Mock(data="") - - # Call method under test - main() - - self.assertTrue( - "Error reading ConfigMap, exiting migration process..." in captured.exception - ) - - self.mock_migrate_config_map.assert_not_called() - self.mock_create_prod_cms.assert_not_called() - self.mock_create_temp_cm.assert_not_called() - self.mock_rename_cm.assert_not_called() - - def test_main_failed_7(self): - """Validating that migration failed as initial and final resource version is different""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.side_effect = [MockYaml(1), - MockYaml(2), - MockYaml(1), - MockYaml(2)] - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = True - self.mock_create_temp_cm.return_value = True - - # Call method under test - main() - - self.assertTrue( - "Re-trying migration process..." in captured.exception - ) - - self.assertTrue( - "Rollback successful" in captured.exception - ) - - self.assertTrue( - f"ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME} is modified, exiting migration process..." - ) - self.mock_rename_cm.assert_not_called() - - def test_main_failed_8(self): - """Validating that migration is successful in second attempt as initial and final resource - version is different in first attempt""" - - self.mock_migrate_config_map = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' - ).start() - self.mock_create_prod_cms = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' - ).start() - self.mock_create_temp_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' - ).start() - self.mock_rename_cm = patch( - 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' - ).start() - - with self.assertLogs(level="DEBUG") as captured: - # with self.assertRaises(SystemExit) as captured: - self.mock_k8api_read.side_effect = [MockYaml(1), - MockYaml(2), - MockYaml(1), - MockYaml(1)] - self.mock_migrate_config_map.return_value = mock_split_catalog_data() - self.mock_create_prod_cms.return_value = True - self.mock_create_temp_cm.return_value = True - self.mock_rename_cm.return_value = True - - # Call method under test - main() - # Verify the exact log message - self.assertEqual(10, len(captured.records)) - self.assertEqual( - captured.records[3].getMessage(), - "Re-trying migration process..." - ) - self.assertEqual( - captured.records[6].getMessage(), - "Rollback successful" - ) - - self.assertEqual( - captured.records[-1].getMessage(), - "Migration successful" - ) +# +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Tests for validating ConfigMapDataHandler +""" + +import unittest +from unittest.mock import patch, call, Mock +from typing import Dict, List + +from cray_product_catalog.migration.main import main +from cray_product_catalog.migration.config_map_data_handler import ConfigMapDataHandler +from cray_product_catalog.constants import ( + PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL +) +from cray_product_catalog.migration import CONFIG_MAP_TEMP +from tests.migration.migration_mock import ( + MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED, INITIAL_MAIN_CM_DATA, MockYaml +) + + +def mock_split_catalog_data(): + """Mocking function to return custom data""" + return MAIN_CM_DATA_EXPECTED, PROD_CM_DATA_LIST_EXPECTED + + +class TestConfigMapDataHandler(unittest.TestCase): + """ Tests for validating ConfigMapDataHandler """ + + def setUp(self) -> None: + """Set up mocks.""" + self.mock_load_k8s_mig = patch('cray_product_catalog.migration.kube_apis.load_k8s').start() + self.mock_corev1api_mig = patch('cray_product_catalog.migration.kube_apis.client.CoreV1Api').start() + self.mock_ApiClient_mig = patch('cray_product_catalog.migration.kube_apis.ApiClient').start() + self.mock_client_mig = patch('cray_product_catalog.migration.kube_apis.client').start() + + self.mock_k8api_read = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.read_config_map').start() + self.mock_k8api_create = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.create_config_map').start() + self.mock_k8api_delete = patch( + 'cray_product_catalog.migration.config_map_data_handler.KubernetesApi.delete_config_map').start() + + def tearDown(self) -> None: + patch.stopall() + + def test_migrate_config_map_data(self): + """ Validating the migration of data into multiple product ConfigMaps data """ + + main_cm_data: Dict + prod_cm_data_list: List + cmdh = ConfigMapDataHandler() + main_cm_data, prod_cm_data_list = cmdh.migrate_config_map_data(INITIAL_MAIN_CM_DATA) + + self.assertEqual(main_cm_data, MAIN_CM_DATA_EXPECTED) + self.assertEqual(prod_cm_data_list, PROD_CM_DATA_LIST_EXPECTED) + + def test_create_product_config_maps(self): + """ Validating product ConfigMaps are created """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice + call( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), + call( + dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), call().__bool__(), + ] + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + self.assertEqual( + captured.records[1].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") + + def test_create_second_product_config_map_failed(self): + """ Validating scenario where creation of second product ConfigMap failed """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + self.mock_k8api_create.side_effect = [True, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_has_calls(calls=[ # Create ConfigMap called twice + call( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call( + dummy_prod_cm_names[1], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[1], PRODUCT_CATALOG_CONFIG_MAP_LABEL), + ] + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + self.assertEqual( + captured.records[1].getMessage(), + f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[1]}") + + def test_create_first_product_config_map_failed(self): + """ Validating scenario where creation of first product ConfigMap failed. """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs() as captured: + self.mock_k8api_create.side_effect = [False, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_product_config_maps(PROD_CM_DATA_LIST_EXPECTED) + + dummy_prod_cm_names = ['cray-product-catalog-hfp-firmware', 'cray-product-catalog-analytics'] + + self.mock_k8api_create.assert_called_once_with( + dummy_prod_cm_names[0], PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PROD_CM_DATA_LIST_EXPECTED[0], PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual(len(captured.records), 1) + + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to create product ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{dummy_prod_cm_names[0]}") + + def test_create_temp_config_map(self): + """ Validating temp main ConfigMap is created """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + + with self.assertLogs(level='DEBUG') as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) + + self.mock_k8api_create.assert_called_once_with( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Created temp ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP}") + + def test_create_temp_config_map_failed(self): + """ Validating temp main ConfigMap creation failed """ + + # mock some additional functions + self.mock_v1_object_Meta_mig = patch('cray_product_catalog.migration.kube_apis.V1ObjectMeta').start() + self.mock_k8api_create.return_value = False + + with self.assertLogs() as captured: + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.create_temp_config_map(MAIN_CM_DATA_EXPECTED) + + self.mock_k8api_create.assert_called_once_with( + CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL + ) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Creating ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE}/{CONFIG_MAP_TEMP} failed") + + def test_rename_config_map(self): + """ Validating product ConfigMaps are created """ + + with self.assertLogs(level="DEBUG") as captured: + # call method under test + self.mock_k8api_delete.return_value = True + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(rename_from=CONFIG_MAP_TEMP, + rename_to=PRODUCT_CATALOG_CONFIG_MAP_NAME, + namespace=PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + label=PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ] + ) + + self.mock_k8api_read.assert_called_once_with(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + "Renaming ConfigMap successful") + + def test_rename_config_map_failed_1(self): + """ Validating rename ConfigMap failure scenario where: + deleting cray-product-catalog ConfigMap failed. """ + + with self.assertLogs() as captured: + self.mock_k8api_delete.side_effect = [False, False] + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to delete ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}") + + def test_rename_config_map_failed_2(self): + """ Validating rename ConfigMap failure scenario where: + creating cray-product-catalog ConfigMap failed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_create.return_value = False + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE) + calls = [ # Delete ConfigMap called twice + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ] + self.mock_k8api_create.assert_has_calls(calls=[ + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + call(PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL), + + ]) + + # Verify the exact log message + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to create ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME}, retrying..") + + def test_rename_config_map_failed_3(self): + """ Validating rename ConfigMap failure scenario where: + first operation of deleting cray-product-catalog-temp ConfigMap failed but later passed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_k8api_delete.side_effect = [True, False, True] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ # Delete ConfigMap called twice + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ]) + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual(2, len(captured.records)) + self.assertEqual( + captured.records[0].getMessage(), + f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") + self.assertEqual( + captured.records[1].getMessage(), + "Renaming ConfigMap successful") + + def test_rename_config_map_failed_4(self): + """ Validating rename ConfigMap failure scenario where: + everytime deleting cray-product-catalog-temp ConfigMap failed. """ + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_k8api_delete.side_effect = [True, False, False, False, False, False, False, False, + False, False, False, False] + + # call method under test + cmdh = ConfigMapDataHandler() + cmdh.rename_config_map(CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + self.mock_k8api_delete.assert_has_calls(calls=[ + call( + PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + call( + CONFIG_MAP_TEMP, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE), + ]) + self.mock_k8api_create.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, + MAIN_CM_DATA_EXPECTED, PRODUCT_CATALOG_CONFIG_MAP_LABEL) + + # Verify the exact log message + self.assertEqual(12, len(captured.records)) + self.assertEqual( + captured.records[-2].getMessage(), + f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") + self.assertEqual( + captured.records[-1].getMessage(), + "Renaming ConfigMap successful") + + def test_main_for_successful_migration(self): + """Validating that migration is successful""" + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertLogs(level="DEBUG") as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = True + + # Call method under test + main() + + self.assertEqual( + captured.records[-1].getMessage(), + "Migration successful") + + def test_main_failed_1(self): + """Validating that migration failed as renaming failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = False + + # Call method under test + main() + + self.assertTrue( + "Renaming cray-product-catalog-temp to cray-product-catalog ConfigMap failed, " + "calling rollback handler..." in captured.exception + ) + + self.assertTrue( + "Rollback successful" in captured.exception + ) + + def test_main_failed_2(self): + """Validating that migration failed as create temp cm failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = False + + # Call method under test + main() + + self.assertTrue( + "Rollback successful" in captured.exception + ) + + self.mock_rename_cm.assert_not_called() + + def test_main_failed_3(self): + """Validating that migration failed as create product ConfigMaps failed""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = False + + # Call method under test + main() + + self.assertTrue( + "Rollback successful" in captured.exception + ) + + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_4(self): + """Validating that migration failed as migrate_config_map failed with exception""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data=MAIN_CM_DATA_EXPECTED) + self.mock_migrate_config_map.return_value = Exception() + + # Call method under test + main() + + self.assertTrue( + "Failed to split ConfigMap Data, exiting migration process..." in captured.exception + ) + + self.mock_create_prod_cms.assert_not_called() + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_6(self): + """Validating that migration failed as read_config_map returned empty response / data.""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.return_value = Mock(data="") + + # Call method under test + main() + + self.assertTrue( + "Error reading ConfigMap, exiting migration process..." in captured.exception + ) + + self.mock_migrate_config_map.assert_not_called() + self.mock_create_prod_cms.assert_not_called() + self.mock_create_temp_cm.assert_not_called() + self.mock_rename_cm.assert_not_called() + + def test_main_failed_7(self): + """Validating that migration failed as initial and final resource version is different""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.side_effect = [MockYaml(1), + MockYaml(2), + MockYaml(1), + MockYaml(2)] + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + + # Call method under test + main() + + self.assertTrue( + "Re-trying migration process..." in captured.exception + ) + + self.assertTrue( + "Rollback successful" in captured.exception + ) + + self.assertTrue( + f"ConfigMap {PRODUCT_CATALOG_CONFIG_MAP_NAME} is modified, exiting migration process..." + ) + self.mock_rename_cm.assert_not_called() + + def test_main_failed_8(self): + """Validating that migration is successful in second attempt as initial and final resource + version is different in first attempt""" + + self.mock_migrate_config_map = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.migrate_config_map_data' + ).start() + self.mock_create_prod_cms = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_product_config_maps' + ).start() + self.mock_create_temp_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.create_temp_config_map' + ).start() + self.mock_rename_cm = patch( + 'cray_product_catalog.migration.config_map_data_handler.ConfigMapDataHandler.rename_config_map' + ).start() + + with self.assertLogs(level="DEBUG") as captured: + # with self.assertRaises(SystemExit) as captured: + self.mock_k8api_read.side_effect = [MockYaml(1), + MockYaml(2), + MockYaml(1), + MockYaml(1)] + self.mock_migrate_config_map.return_value = mock_split_catalog_data() + self.mock_create_prod_cms.return_value = True + self.mock_create_temp_cm.return_value = True + self.mock_rename_cm.return_value = True + + # Call method under test + main() + # Verify the exact log message + self.assertEqual(10, len(captured.records)) + self.assertEqual( + captured.records[3].getMessage(), + "Re-trying migration process..." + ) + self.assertEqual( + captured.records[6].getMessage(), + "Rollback successful" + ) + + self.assertEqual( + captured.records[-1].getMessage(), + "Migration successful" + ) diff --git a/tests/migration/test_exit_handler.py b/tests/migration/test_exit_handler.py index ee3ffe4f..032f2fdc 100644 --- a/tests/migration/test_exit_handler.py +++ b/tests/migration/test_exit_handler.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/tests/mock_update_catalog.py b/tests/mock_update_catalog.py index 66d733cb..780a52b4 100755 --- a/tests/mock_update_catalog.py +++ b/tests/mock_update_catalog.py @@ -1,107 +1,107 @@ -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - -""" -Mock data for catalog_update unit tests -""" - -import os -from unittest import mock -import kubernetes.client.rest -from tests.mocks import COS_VERSIONS, Name - -# Mocking environment variables before import so that: -# 1. import (create_config_map, update_config_map and main) is successful -# 2. Additionally they will be used in testcase to verify the tests. -mock.patch.dict( - os.environ, { - 'PRODUCT': 'sat', - 'PRODUCT_VERSION': '1.0.0', - 'YAML_CONTENT_STRING': 'Test data', - 'CONFIG_MAP_NAMESPACE': 'myNamespace' - } -).start() - -UPDATE_DATA = { - '2.0.0': { - 'component_versions': { - 'docker': [ - {'name': 'cray/cray-cos', 'version': '1.0.0'}, - {'name': 'cray/cos-cfs-install', 'version': '1.4.0'} - ] - } - } -} - -ERR_NOT_FOUND = 404 - - -class Response: - """ - Class to generate response for k8s api call api_instance.read_namespaced_config_map(name, namespace) - """ - def __init__(self): - self.data = COS_VERSIONS - self.metadata = Name() - - -class ApiException(kubernetes.client.rest.ApiException): - """ - Custom Exception to define status - """ - def __init__(self): - super().__init__() - self.status = ERR_NOT_FOUND - - -class ApiInstance(): - """ - Class to raise custom exception and ignore function calls - """ - def __init__(self, raise_exception=False): - self.raise_exception = raise_exception - self.count = 0 - - def create_namespaced_config_map(self, namespace='a', body='b'): - """ - Dummy function to raise exception, if needed - """ - if self.raise_exception: - raise ApiException() - - def read_namespaced_config_map(self, name, namespace): - """ - Dummy function to : - 1. Raise exception - 2. Generate and return proper response with data and metadata - """ - # if this is called for first time return exception, so that product cm is created. - if self.count == 0: - self.count += 1 - raise ApiException() - return Response() - - def patch_namespaced_config_map(self, name, namespace, body='xxx'): - """ - Dummy function to handle the call in code; does nothing - """ +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + +""" +Mock data for catalog_update unit tests +""" + +import os +from unittest import mock +import kubernetes.client.rest +from tests.mocks import COS_VERSIONS, Name + +# Mocking environment variables before import so that: +# 1. import (create_config_map, update_config_map and main) is successful +# 2. Additionally they will be used in testcase to verify the tests. +mock.patch.dict( + os.environ, { + 'PRODUCT': 'sat', + 'PRODUCT_VERSION': '1.0.0', + 'YAML_CONTENT_STRING': 'Test data', + 'CONFIG_MAP_NAMESPACE': 'myNamespace' + } +).start() + +UPDATE_DATA = { + '2.0.0': { + 'component_versions': { + 'docker': [ + {'name': 'cray/cray-cos', 'version': '1.0.0'}, + {'name': 'cray/cos-cfs-install', 'version': '1.4.0'} + ] + } + } +} + +ERR_NOT_FOUND = 404 + + +class Response: + """ + Class to generate response for k8s api call api_instance.read_namespaced_config_map(name, namespace) + """ + def __init__(self): + self.data = COS_VERSIONS + self.metadata = Name() + + +class ApiException(kubernetes.client.rest.ApiException): + """ + Custom Exception to define status + """ + def __init__(self): + super().__init__() + self.status = ERR_NOT_FOUND + + +class ApiInstance(): + """ + Class to raise custom exception and ignore function calls + """ + def __init__(self, raise_exception=False): + self.raise_exception = raise_exception + self.count = 0 + + def create_namespaced_config_map(self, namespace='a', body='b'): + """ + Dummy function to raise exception, if needed + """ + if self.raise_exception: + raise ApiException() + + def read_namespaced_config_map(self, name, namespace): + """ + Dummy function to : + 1. Raise exception + 2. Generate and return proper response with data and metadata + """ + # if this is called for first time return exception, so that product cm is created. + if self.count == 0: + self.count += 1 + raise ApiException() + return Response() + + def patch_namespaced_config_map(self, name, namespace, body='xxx'): + """ + Dummy function to handle the call in code; does nothing + """ diff --git a/tests/mocks.py b/tests/mocks.py index 49283d77..df879ef6 100644 --- a/tests/mocks.py +++ b/tests/mocks.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/tests/test_catalog_delete.py b/tests/test_catalog_delete.py index a5ed37bf..62805d51 100644 --- a/tests/test_catalog_delete.py +++ b/tests/test_catalog_delete.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/tests/test_query.py b/tests/test_query.py index 786fb4bc..9f3b62e3 100644 --- a/tests/test_query.py +++ b/tests/test_query.py @@ -1,6 +1,6 @@ # MIT License # -# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP +# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP # # Permission is hereby granted, free of charge, to any person obtaining a # copy of this software and associated documentation files (the "Software"), diff --git a/tests/test_update_catalog.py b/tests/test_update_catalog.py index 06efc265..3049616a 100755 --- a/tests/test_update_catalog.py +++ b/tests/test_update_catalog.py @@ -1,192 +1,192 @@ -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. -# - - -"""Unit tests for cray_product_catalog.catalog_update module""" - -import unittest -from unittest import mock -from tests.mock_update_catalog import ( - UPDATE_DATA, ApiInstance, ApiException -) -from cray_product_catalog.catalog_update import ( - create_config_map, - update_config_map, - main -) - - -class TestCatalogUpdate(unittest.TestCase): - """ - Tests for catalog update - """ - def setUp(self): - """Set up mocks.""" - self.mock_v1configmap = mock.patch('cray_product_catalog.catalog_update.V1ConfigMap').start() - self.mock_load_k8s = mock.patch('cray_product_catalog.catalog_update.load_k8s').start() - self.mock_ApiClient = mock.patch('cray_product_catalog.catalog_update.ApiClient').start() - self.mock_client = mock.patch('cray_product_catalog.catalog_update.client').start() - - def test_create_config_map_success_log(self): - """ - Test for validating create_config_map method logs - Verify expected log is generated based on the passed arguments - """ - name = "cos" - namespace = "product" - self.mock_v1configmap.metadata = None - with self.assertLogs() as captured: - # call method under test - create_config_map(ApiInstance(raise_exception=False), name, namespace) - self.assertEqual(len(captured.records), 1) # check that there is only one log message - - expected_log = "Created product ConfigMap " + namespace + "/" + name - self.assertEqual(captured.records[0].getMessage(), expected_log) # Verify the exact log message - - def test_create_config_map_failure_exception(self): - """ - Verify if expected logs are generated in exception - """ - name = "cos" - namespace = "product" - self.mock_v1configmap.metadata = None - - with self.assertLogs() as captured: - try: - # call method under test - create_config_map(ApiInstance(raise_exception=True), name, namespace) - except ApiException: - pass - self.assertEqual(len(captured.records), 1) # check that there is only one log message - self.assertEqual(captured.records[0].getMessage(), - "Error calling create_namespaced_config_map") # Verify the exact log message - - def test_update_config_map_max_retries(self): - """ - Verify update_config_map exits after max retries if Kubernetes API raise exception. - """ - name = "cos" - namespace = "product" - self.mock_v1configmap.metadata = None - - self.mock_read_config_map = mock.patch( - 'cray_product_catalog.catalog_update.client.CoreV1Api.read_namespaced_config_map' - ).start().side_effect = ApiException() - - with self.assertRaises(SystemExit) as captured: - with mock.patch( - 'cray_product_catalog.catalog_update.random.randint', return_value=0 - ): - # call method under test - update_config_map(UPDATE_DATA, name, namespace) - # Verify the exact log message - self.assertTrue( - f"Exceeded number of attempts; Not updating ConfigMap {namespace}/{name}." - in captured.exception) - - def test_update_config_map(self): - """ - Verify `create_config_map` is called if provided `name` is of product and not `CONFIG_MAP` (main cm) - """ - name = "cos" - namespace = "product" - data = UPDATE_DATA - - # mock some additional functions - self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() - self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() - - with mock.patch( - 'cray_product_catalog.catalog_update.client.CoreV1Api', return_value=ApiInstance(raise_exception=True) - ): - with mock.patch( - 'cray_product_catalog.catalog_update.random.randint', return_value=0.5 - ): - # call method under test - update_config_map(data, name, namespace) - - # verify if create-config_map is called. Couldn't verify it with arguments as one of the arg is object. - self.mock_create_config_map.assert_called() - - def test_main_valid_product_configmap(self): - """ - Verify `update_config_map` is called with proper data if provided product information is available - """ - # Initialise some random data, need not to be exact format - prod_cm = {"Some random text for product"} - main_cm = {"Some random text for main"} - - self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() - self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() - self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() - - # mocking function to return custom data - def mock_split_catalog_data(): - return main_cm, prod_cm - - with mock.patch( - 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() - ): - # Call method under test - main() - # sat is from PRODUCT environment variable. - expected_product_cm = 'cray-product-catalog-sat' - # myNamespace is from CONFIG_MAP_NAMESPACE environment variable. - expected_namespace = 'myNamespace' - self.mock_update_config_map.assert_called_with(prod_cm, expected_product_cm, expected_namespace) - - def test_main_for_empty_product_configmap(self): - """ - Verify `main` throws exception when PRODUCT_CONFIG_MAP='' - """ - prod_cm = {"Some random text for product"} - main_cm = {"Some random text for main"} - - self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() - self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() - self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() - - # mocking function to return custom data - def mock_split_catalog_data(): - return main_cm, prod_cm - - with mock.patch( - 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() - ): - with mock.patch( - 'cray_product_catalog.catalog_update.format_product_cm_name', return_value='' - ): - with self.assertRaises(SystemExit) as context: - # call method under test - main() - # Verify the log message in exception - self.assertTrue( - "ERROR Not updating ConfigMaps because the provided product name is invalid: 'sat'" - in context.exception - ) - # Verify that update config map is not called in case of exception - self.mock_update_config_map.assert_not_called() - - -if __name__ == '__main__': - unittest.main() +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. +# + + +"""Unit tests for cray_product_catalog.catalog_update module""" + +import unittest +from unittest import mock +from tests.mock_update_catalog import ( + UPDATE_DATA, ApiInstance, ApiException +) +from cray_product_catalog.catalog_update import ( + create_config_map, + update_config_map, + main +) + + +class TestCatalogUpdate(unittest.TestCase): + """ + Tests for catalog update + """ + def setUp(self): + """Set up mocks.""" + self.mock_v1configmap = mock.patch('cray_product_catalog.catalog_update.V1ConfigMap').start() + self.mock_load_k8s = mock.patch('cray_product_catalog.catalog_update.load_k8s').start() + self.mock_ApiClient = mock.patch('cray_product_catalog.catalog_update.ApiClient').start() + self.mock_client = mock.patch('cray_product_catalog.catalog_update.client').start() + + def test_create_config_map_success_log(self): + """ + Test for validating create_config_map method logs + Verify expected log is generated based on the passed arguments + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + with self.assertLogs() as captured: + # call method under test + create_config_map(ApiInstance(raise_exception=False), name, namespace) + self.assertEqual(len(captured.records), 1) # check that there is only one log message + + expected_log = "Created product ConfigMap " + namespace + "/" + name + self.assertEqual(captured.records[0].getMessage(), expected_log) # Verify the exact log message + + def test_create_config_map_failure_exception(self): + """ + Verify if expected logs are generated in exception + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + + with self.assertLogs() as captured: + try: + # call method under test + create_config_map(ApiInstance(raise_exception=True), name, namespace) + except ApiException: + pass + self.assertEqual(len(captured.records), 1) # check that there is only one log message + self.assertEqual(captured.records[0].getMessage(), + "Error calling create_namespaced_config_map") # Verify the exact log message + + def test_update_config_map_max_retries(self): + """ + Verify update_config_map exits after max retries if Kubernetes API raise exception. + """ + name = "cos" + namespace = "product" + self.mock_v1configmap.metadata = None + + self.mock_read_config_map = mock.patch( + 'cray_product_catalog.catalog_update.client.CoreV1Api.read_namespaced_config_map' + ).start().side_effect = ApiException() + + with self.assertRaises(SystemExit) as captured: + with mock.patch( + 'cray_product_catalog.catalog_update.random.randint', return_value=0 + ): + # call method under test + update_config_map(UPDATE_DATA, name, namespace) + # Verify the exact log message + self.assertTrue( + f"Exceeded number of attempts; Not updating ConfigMap {namespace}/{name}." + in captured.exception) + + def test_update_config_map(self): + """ + Verify `create_config_map` is called if provided `name` is of product and not `CONFIG_MAP` (main cm) + """ + name = "cos" + namespace = "product" + data = UPDATE_DATA + + # mock some additional functions + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + with mock.patch( + 'cray_product_catalog.catalog_update.client.CoreV1Api', return_value=ApiInstance(raise_exception=True) + ): + with mock.patch( + 'cray_product_catalog.catalog_update.random.randint', return_value=0.5 + ): + # call method under test + update_config_map(data, name, namespace) + + # verify if create-config_map is called. Couldn't verify it with arguments as one of the arg is object. + self.mock_create_config_map.assert_called() + + def test_main_valid_product_configmap(self): + """ + Verify `update_config_map` is called with proper data if provided product information is available + """ + # Initialise some random data, need not to be exact format + prod_cm = {"Some random text for product"} + main_cm = {"Some random text for main"} + + self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + # mocking function to return custom data + def mock_split_catalog_data(): + return main_cm, prod_cm + + with mock.patch( + 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() + ): + # Call method under test + main() + # sat is from PRODUCT environment variable. + expected_product_cm = 'cray-product-catalog-sat' + # myNamespace is from CONFIG_MAP_NAMESPACE environment variable. + expected_namespace = 'myNamespace' + self.mock_update_config_map.assert_called_with(prod_cm, expected_product_cm, expected_namespace) + + def test_main_for_empty_product_configmap(self): + """ + Verify `main` throws exception when PRODUCT_CONFIG_MAP='' + """ + prod_cm = {"Some random text for product"} + main_cm = {"Some random text for main"} + + self.mock_update_config_map = mock.patch('cray_product_catalog.catalog_update.update_config_map').start() + self.mock_create_config_map = mock.patch('cray_product_catalog.catalog_update.create_config_map').start() + self.mock_v1_object_Meta = mock.patch('cray_product_catalog.catalog_update.V1ObjectMeta').start() + + # mocking function to return custom data + def mock_split_catalog_data(): + return main_cm, prod_cm + + with mock.patch( + 'cray_product_catalog.catalog_update.split_catalog_data', return_value=mock_split_catalog_data() + ): + with mock.patch( + 'cray_product_catalog.catalog_update.format_product_cm_name', return_value='' + ): + with self.assertRaises(SystemExit) as context: + # call method under test + main() + # Verify the log message in exception + self.assertTrue( + "ERROR Not updating ConfigMaps because the provided product name is invalid: 'sat'" + in context.exception + ) + # Verify that update config map is not called in case of exception + self.mock_update_config_map.assert_not_called() + + +if __name__ == '__main__': + unittest.main() diff --git a/tests/util/test_data_catalog_helper.py b/tests/util/test_data_catalog_helper.py index 015c967c..c831627e 100755 --- a/tests/util/test_data_catalog_helper.py +++ b/tests/util/test_data_catalog_helper.py @@ -1,124 +1,124 @@ -# MIT License -# -# (C) Copyright 2023 Hewlett Packard Enterprise Development LP -# -# Permission is hereby granted, free of charge, to any person obtaining a -# copy of this software and associated documentation files (the "Software"), -# to deal in the Software without restriction, including without limitation -# the rights to use, copy, modify, merge, publish, distribute, sublicense, -# and/or sell copies of the Software, and to permit persons to whom the -# Software is furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included -# in all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL -# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR -# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, -# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR -# OTHER DEALINGS IN THE SOFTWARE. - -""" -Unit tests for the cray_product_catalog.util.catalog_data_helper module -""" - -import unittest -import yaml -from typing import Dict -from tests.mocks import ( - YAML_DATA, YAML_DATA_MISSING_MAIN_DATA, - YAML_DATA_MISSING_PROD_CM_DATA, - MAIN_CM_DATA, PROD_CM_DATA -) -from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name - - -class TestCatalogDataHelper(unittest.TestCase): - """Tests for catalog_data_helper.""" - - def test_split_data_sanity(self): - """Sanity check of split of YAML into main and product-specific data | +ve test case""" - - # expected data - main_cm_data_expected = MAIN_CM_DATA - prod_cm_data_expected = PROD_CM_DATA - - # YAML raw to Python object - yaml_object: Dict = yaml.safe_load(YAML_DATA) - - main_cm_data: Dict - prod_cm_data: Dict - main_cm_data, prod_cm_data = split_catalog_data(yaml_object) - - self.assertEqual(main_cm_data, main_cm_data_expected) - self.assertEqual(prod_cm_data, prod_cm_data_expected) - - def test_split_missing_prod_cm_data(self): - """Missing product ConfigMap data check""" - - # expected data - main_cm_data_expected = MAIN_CM_DATA - prod_cm_data_expected = {} - - # YAML raw to Python object - yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_PROD_CM_DATA) - - main_cm_data: Dict - prod_cm_data: Dict - main_cm_data, prod_cm_data = split_catalog_data(yaml_object) - - self.assertEqual(main_cm_data, main_cm_data_expected) - self.assertEqual(prod_cm_data, prod_cm_data_expected) - - def test_split_missing_main_cm_data(self): - """Missing main ConfigMap data check""" - - # expected data - main_cm_data_expected = {} - prod_cm_data_expected = PROD_CM_DATA - - # YAML raw to Python object - yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_MAIN_DATA) - - main_cm_data: Dict - prod_cm_data: Dict - main_cm_data, prod_cm_data = split_catalog_data(yaml_object) - - self.assertEqual(main_cm_data, main_cm_data_expected) - self.assertEqual(prod_cm_data, prod_cm_data_expected) - - def test_format_product_cm_name_sanity(self): - """Unit test case for product name formatting""" - product_name = "dummy-valid-1" - config_map = "cm" - self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-{product_name}") - - def test_format_product_name_transform(self): - """Unit test case for valid product name transformation""" - product_name = "23dummy_valid-1.x86" - config_map = "cm" - self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-23dummy-valid-1.x86") - - def test_format_product_name_invalid_cases(self): - """Unit test case for invalid product names""" - - # case with special characters - product_name = "po90-$_invalid" - config_map = "cm" - self.assertEqual(format_product_cm_name(config_map, product_name), "") - - # large name with non-blank ConfigMap case - product_name = "ola-9" * 60 - config_map = "cm" - self.assertEqual(format_product_cm_name(config_map, product_name), "") - - # large name with blank ConfigMap case - product_name = "ola-9" * 60 - config_map = "" - self.assertEqual(format_product_cm_name(config_map, product_name), "") - - -if __name__ == '__main__': - unittest.main() +# MIT License +# +# (C) Copyright 2023-2024 Hewlett Packard Enterprise Development LP +# +# Permission is hereby granted, free of charge, to any person obtaining a +# copy of this software and associated documentation files (the "Software"), +# to deal in the Software without restriction, including without limitation +# the rights to use, copy, modify, merge, publish, distribute, sublicense, +# and/or sell copies of the Software, and to permit persons to whom the +# Software is furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included +# in all copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR +# OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +# ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +# OTHER DEALINGS IN THE SOFTWARE. + +""" +Unit tests for the cray_product_catalog.util.catalog_data_helper module +""" + +import unittest +import yaml +from typing import Dict +from tests.mocks import ( + YAML_DATA, YAML_DATA_MISSING_MAIN_DATA, + YAML_DATA_MISSING_PROD_CM_DATA, + MAIN_CM_DATA, PROD_CM_DATA +) +from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name + + +class TestCatalogDataHelper(unittest.TestCase): + """Tests for catalog_data_helper.""" + + def test_split_data_sanity(self): + """Sanity check of split of YAML into main and product-specific data | +ve test case""" + + # expected data + main_cm_data_expected = MAIN_CM_DATA + prod_cm_data_expected = PROD_CM_DATA + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_split_missing_prod_cm_data(self): + """Missing product ConfigMap data check""" + + # expected data + main_cm_data_expected = MAIN_CM_DATA + prod_cm_data_expected = {} + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_PROD_CM_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_split_missing_main_cm_data(self): + """Missing main ConfigMap data check""" + + # expected data + main_cm_data_expected = {} + prod_cm_data_expected = PROD_CM_DATA + + # YAML raw to Python object + yaml_object: Dict = yaml.safe_load(YAML_DATA_MISSING_MAIN_DATA) + + main_cm_data: Dict + prod_cm_data: Dict + main_cm_data, prod_cm_data = split_catalog_data(yaml_object) + + self.assertEqual(main_cm_data, main_cm_data_expected) + self.assertEqual(prod_cm_data, prod_cm_data_expected) + + def test_format_product_cm_name_sanity(self): + """Unit test case for product name formatting""" + product_name = "dummy-valid-1" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-{product_name}") + + def test_format_product_name_transform(self): + """Unit test case for valid product name transformation""" + product_name = "23dummy_valid-1.x86" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), f"{config_map}-23dummy-valid-1.x86") + + def test_format_product_name_invalid_cases(self): + """Unit test case for invalid product names""" + + # case with special characters + product_name = "po90-$_invalid" + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + # large name with non-blank ConfigMap case + product_name = "ola-9" * 60 + config_map = "cm" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + # large name with blank ConfigMap case + product_name = "ola-9" * 60 + config_map = "" + self.assertEqual(format_product_cm_name(config_map, product_name), "") + + +if __name__ == '__main__': + unittest.main() From 2cc07775e2f5a523b7fb9a25367e3b5688928b0e Mon Sep 17 00:00:00 2001 From: Mitch Harding Date: Wed, 3 Jan 2024 14:09:49 -0500 Subject: [PATCH 08/12] Update CHANGELOG.md Update version date in CHANGELOG --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b4d58603..8c600e9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -## [1.10.1] - 2023-11-30 +## [1.10.1] - 2024-01-03 ### Changed From 01e4beae93f74b70a692c4a80293295dadebd3a0 Mon Sep 17 00:00:00 2001 From: Mitch Harding Date: Tue, 16 Jan 2024 12:12:32 -0500 Subject: [PATCH 09/12] Apply suggestions from code review Change new version to `2.0.0` to reflect scope of changes --- CHANGELOG.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c600e9c..30a5ad74 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -## [1.10.1] - 2024-01-03 +## [2.0.0] - 2024-01-16 ### Changed @@ -461,9 +461,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Change default reviewers to CMS-core-product-support -[Unreleased]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.1...HEAD +[Unreleased]: https://github.com/Cray-HPE/cray-product-catalog/compare/v2.0.0...HEAD -[1.10.1]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.0...v1.10.1 +[2.0.0]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.10.0...v2.0.0 [1.10.0]: https://github.com/Cray-HPE/cray-product-catalog/compare/v1.9.0...v1.10.0 From 341eeaa6d080a291cd9e3d0cd5c2de812b953acd Mon Sep 17 00:00:00 2001 From: lathanm <128785927+lathanm@users.noreply.github.com> Date: Wed, 7 Feb 2024 12:51:11 +0530 Subject: [PATCH 10/12] Addressed Ryan's review comments --- CHANGELOG.md | 6 +- .../templates/configmap-hook.yaml | 7 +-- charts/cray-product-catalog/values.yaml | 2 +- cray_product_catalog/catalog_delete.py | 14 ++--- cray_product_catalog/migration/__init__.py | 8 +-- .../migration/config_map_data_handler.py | 24 ++++---- .../migration/exit_handler.py | 8 --- cray_product_catalog/migration/kube_apis.py | 18 +----- cray_product_catalog/migration/main.py | 4 +- cray_product_catalog/query.py | 59 +++++++++++++++---- .../migration/test_config_map_data_handler.py | 2 +- tests/mocks.py | 1 + tests/test_query.py | 53 +++++++++++++---- 13 files changed, 126 insertions(+), 80 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30a5ad74..52143fca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,10 +14,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - CASM-4350: To address the 1MiB size limit of Kubernetes ConfigMaps, the `cray-product-catalog` Kubernetes ConfigMap is split into multiple smaller ConfigMaps with each product's `component_versions` data in its own ConfigMap. - Modify the `catalog_update.py` script to update data across these ConfigMaps. -- CASM-4427: Implement a prototype to have granular query from main and sub ConfigMaps -- CASM-4368: Delete Cray Product Catalog details from main and sub ConfigMaps for a particular - product version. + Modified `create`, `modify`, `delete` and `query` feature to support + the split of single ConfigMap into multiple ConfigMaps. - Added new argument `max_attempts` to `modify_config_map` function in [`catalog_delete.py`](cray_product_catalog/catalog_delete.py), because we need not retry 100 times when read ConfigMap fails for a product ConfigMap. diff --git a/charts/cray-product-catalog/templates/configmap-hook.yaml b/charts/cray-product-catalog/templates/configmap-hook.yaml index 07023ddb..9835d9f2 100644 --- a/charts/cray-product-catalog/templates/configmap-hook.yaml +++ b/charts/cray-product-catalog/templates/configmap-hook.yaml @@ -82,7 +82,7 @@ spec: apiVersion: batch/v1 kind: Job metadata: - name: {{ .Release.Name }}-my-job + name: {{ .Release.Name }}-migrate annotations: "helm.sh/hook": pre-upgrade "helm.sh/hook-weight": "1" @@ -97,7 +97,7 @@ spec: command: - /bin/sh env: - - name: CONFIG_MAP + - name: CONFIG_MAP_NAME value: {{ .Values.migration.configMap }} - name: CONFIG_MAP_NAMESPACE value: {{ .Values.migration.configMapNamespace }} @@ -105,6 +105,5 @@ spec: imagePullPolicy: IfNotPresent name: migrate-catalog restartPolicy: Never - serviceAccount: {{ .Values.migration.serviceAccount }} - serviceAccountName: {{ .Values.migration.serviceAccount }} + serviceAccountName: {{ .Values.migration.serviceAccountName }} diff --git a/charts/cray-product-catalog/values.yaml b/charts/cray-product-catalog/values.yaml index aca239d9..bd636e27 100644 --- a/charts/cray-product-catalog/values.yaml +++ b/charts/cray-product-catalog/values.yaml @@ -31,7 +31,7 @@ migration: tag: 0.0.0-docker configMap: cray-product-catalog configMapNamespace: services - serviceAccount: cray-product-catalog + serviceAccountName: cray-product-catalog global: appVersion: 0.0.0-docker diff --git a/cray_product_catalog/catalog_delete.py b/cray_product_catalog/catalog_delete.py index 35fc8e67..5fb13fc7 100755 --- a/cray_product_catalog/catalog_delete.py +++ b/cray_product_catalog/catalog_delete.py @@ -136,16 +136,16 @@ def max_retries_for_main_cm(self): return self.__max_retries_for_main_cm @max_retries_for_main_cm.setter - def max_retries_for_main_cm(self, __max_reties_for_main_cm): - self.__max_retries_for_main_cm = __max_reties_for_main_cm + def max_retries_for_main_cm(self, __max_retries_for_main_cm): + self.__max_retries_for_main_cm = __max_retries_for_main_cm @property def max_retries_for_prod_cm(self): return self.__max_retries_for_prod_cm @max_retries_for_prod_cm.setter - def max_retries_for_prod_cm(self, __max_reties_for_prod_cm): - self.__max_retries_for_prod_cm = __max_reties_for_prod_cm + def max_retries_for_prod_cm(self, __max_retries_for_prod_cm): + self.__max_retries_for_prod_cm = __max_retries_for_prod_cm @property def key(self): @@ -196,12 +196,12 @@ def modify(self): Method to initiate modification of ConfigMaps. Before executing this method make sure to set these properties of the class: * main_cm # Name of main ConfigMap - * product_cm # name of product-specific CofigMap + * product_cm # name of product-specific ConfigMap * cm_namespace # Namespace containing all ConfigMaps * product_name # Product name * product_version # Product version - * max_reties_for_main_cm # Max failure retries for main ConfigMap - * max_reties_for_prod_cm # Max failure retries for product ConfigMap + * max_retries_for_main_cm # Max failure retries for main ConfigMap + * max_retries_for_prod_cm # Max failure retries for product ConfigMap * key # Key to delete; if you want to execute complete product or a particular version, ignore it * main_cm_fields # Fields present in main ConfigMap * product_cm_fields # Fields present in product-specific ConfigMap diff --git a/cray_product_catalog/migration/__init__.py b/cray_product_catalog/migration/__init__.py index 2cfd2f84..2d32c7da 100644 --- a/cray_product_catalog/migration/__init__.py +++ b/cray_product_catalog/migration/__init__.py @@ -25,10 +25,10 @@ import os import re -from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR +from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR, PRODUCT_CATALOG_CONFIG_MAP_NAME # ConfigMap name for temporary main ConfigMap -CONFIG_MAP_TEMP = "cray-product-catalog-temp" +CONFIG_MAP_TEMP = f"{PRODUCT_CATALOG_CONFIG_MAP_NAME}-temp" # namespace for ConfigMaps PRODUCT_CATALOG_CONFIG_MAP_NAME = os.environ.get("CONFIG_MAP_NAME", "cray-product-catalog").strip() @@ -41,6 +41,4 @@ PRODUCT_CONFIG_MAP_PATTERN = re.compile('^(cray-product-catalog)-([a-z0-9.-]+)$') RESOURCE_VERSION = 'resource_version' -retry_count = 10 -role_name = 'cray-product-catalog' -action = 'update' +RETRY_COUNT = 10 diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py index 0cad7497..649631ba 100644 --- a/cray_product_catalog/migration/config_map_data_handler.py +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -30,10 +30,9 @@ """ import logging -import yaml +import yaml -from cray_product_catalog.logging import configure_logging from cray_product_catalog.util.catalog_data_helper import split_catalog_data, format_product_cm_name from cray_product_catalog.migration.kube_apis import KubernetesApi from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL @@ -49,8 +48,6 @@ class ConfigMapDataHandler: def __init__(self) -> None: self.k8s_obj = KubernetesApi() - self.config_map_data_replica = {} - configure_logging() def create_product_config_maps(self, product_config_map_data_list): """Create new product ConfigMap for each product in product_config_map_data_list @@ -62,6 +59,10 @@ def create_product_config_maps(self, product_config_map_data_list): product_name = list(product_data.keys())[0] LOGGER.debug("Creating ConfigMap for product %s", product_name) prod_cm_name = format_product_cm_name(PRODUCT_CATALOG_CONFIG_MAP_NAME, product_name) + if prod_cm_name == '': + LOGGER.error("Failed to create ConfigMap %s/%s because the provided product name is invalid: '%s'", + PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, prod_cm_name, product_name) + return False if not self.k8s_obj.create_config_map(prod_cm_name, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE, product_data, PRODUCT_CATALOG_CONFIG_MAP_LABEL): @@ -98,8 +99,6 @@ def migrate_config_map_data(self, config_map_data): "Migrating data in ConfigMap=%s in namespace=%s to multiple ConfigMaps", PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE ) - # Backed up ConfigMap Data - self.config_map_data_replica = config_map_data # Get list of products products_list = list(config_map_data.keys()) product_config_map_data_list = [] @@ -121,8 +120,9 @@ def migrate_config_map_data(self, config_map_data): main_versions_data[version_data] = main_cm_data # If `component_versions` data exists for a product, create new product ConfigMap if product_versions_data: - product_config_map_data[product] = yaml.safe_dump(product_versions_data, default_flow_style=False) - # create_product_config_map(k8s_obj, product, product_config_map_data) + product_config_map_data= { + product: yaml.safe_dump(product_versions_data, default_flow_style=False) + } product_config_map_data_list.append(product_config_map_data) # Data with key other than `component_versions` should be updated to config_map_data, # so that new main ConfigMap will not have data with key `component_versions` @@ -143,7 +143,6 @@ def rename_config_map(self, rename_from, rename_to, namespace, label): if not self.k8s_obj.delete_config_map(rename_to, namespace): LOGGER.error("Failed to delete ConfigMap %s", rename_to) - return False attempt = 0 del_failed = False @@ -164,16 +163,19 @@ def rename_config_map(self, rename_from, rename_to, namespace, label): del_failed = True break LOGGER.error("Failed to create ConfigMap %s, retrying..", rename_to) - continue # Since only delete of backed up ConfigMap failed, retrying only delete operation attempt = 0 if del_failed: while attempt < 10: attempt += 1 if self.k8s_obj.delete_config_map(rename_from, namespace): + del_failed = False break LOGGER.error("Failed to delete ConfigMap %s, retrying..", rename_from) # Returning success as migration is successful only backed up ConfigMap is not deleted. - LOGGER.info("Renaming ConfigMap successful") + if del_failed: + LOGGER.info("Failed to delete ConfigMap %s, but migration is successful", rename_from) + else: + LOGGER.info("Renaming ConfigMap successful") return True return False diff --git a/cray_product_catalog/migration/exit_handler.py b/cray_product_catalog/migration/exit_handler.py index e04ea925..a218c463 100644 --- a/cray_product_catalog/migration/exit_handler.py +++ b/cray_product_catalog/migration/exit_handler.py @@ -56,14 +56,6 @@ class ExitHandler: def __init__(self): self.k8api = KubernetesApi() # Kubernetes API object - @staticmethod - def graceful_exit() -> None: - LOGGER.info("Migration not possible, no exception occurred.") - - @staticmethod - def exception_exit() -> None: - LOGGER.error("Migration not possible, exception occurred.") - def __get_all_created_product_config_maps(self) -> List: """Get all created product ConfigMaps""" cm_name = filter(_is_product_config_map, diff --git a/cray_product_catalog/migration/kube_apis.py b/cray_product_catalog/migration/kube_apis.py index 2c3cdcd6..6b9ed001 100644 --- a/cray_product_catalog/migration/kube_apis.py +++ b/cray_product_catalog/migration/kube_apis.py @@ -36,7 +36,7 @@ from urllib3.exceptions import MaxRetryError from cray_product_catalog.logging import configure_logging from cray_product_catalog.util.k8s import load_k8s -from . import retry_count +from . import RETRY_COUNT class KubernetesApi: @@ -47,7 +47,7 @@ def __init__(self): load_k8s() retry = Retry( - total=retry_count, read=retry_count, connect=retry_count, backoff_factor=0.3, + total=RETRY_COUNT, read=RETRY_COUNT, connect=RETRY_COUNT, backoff_factor=0.3, status_forcelist=(500, 502, 503, 504) ) self.kclient = ApiClient() @@ -81,10 +81,6 @@ def create_config_map(self, name, namespace, data, label): # The full string representation of ApiException is very long, so just log err.reason. self.logger.exception('ApiException: %s', err.reason) return False - self.logger.error('Unknown error creating ConfigMap') - return False - self.logger.error('Unknown error creating ConfigMap') - return False def list_config_map(self, namespace, label): """ Reads all the ConfigMaps with certain label in particular namespace @@ -105,10 +101,6 @@ def list_config_map(self, namespace, label): # The full string representation of ApiException is very long, so just log err.reason. self.logger.exception('ApiException: %s', err.reason) return None - self.logger.error('Unknown error listing ConfigMaps') - return None - self.logger.error('Unknown error listing ConfigMaps') - return None def list_config_map_names(self, namespace, label): """ Reads all the ConfigMaps with certain label in particular namespace @@ -152,10 +144,6 @@ def read_config_map(self, name, namespace): # The full string representation of ApiException is very long, so just log err.reason. self.logger.exception('ApiException: %s', err.reason) return None - self.logger.error('Unknown error reading ConfigMap') - return None - self.logger.error('Unknown error reading ConfigMap') - return None def delete_config_map(self, name, namespace): """Delete the ConfigMap @@ -173,5 +161,3 @@ def delete_config_map(self, name, namespace): # The full string representation of ApiException is very long, so just log err.reason. self.logger.exception('ApiException: %s', err.reason) return False - self.logger.error('Unknown error deleting ConfigMap') - return False diff --git a/cray_product_catalog/migration/main.py b/cray_product_catalog/migration/main.py index 4a2111e9..0a564ddd 100644 --- a/cray_product_catalog/migration/main.py +++ b/cray_product_catalog/migration/main.py @@ -51,11 +51,11 @@ def main(): exit_handler = ExitHandler() attempt = 0 max_attempts = 2 + migration_failed = False - while (attempt < max_attempts): + while attempt < max_attempts: attempt += 1 migration_failed = False - init_resource_version = '' curr_resource_version = '' response = config_map_obj.k8s_obj.read_config_map( PRODUCT_CATALOG_CONFIG_MAP_NAME, PRODUCT_CATALOG_CONFIG_MAP_NAMESPACE diff --git a/cray_product_catalog/query.py b/cray_product_catalog/query.py index b5e7d112..931c1851 100644 --- a/cray_product_catalog/query.py +++ b/cray_product_catalog/query.py @@ -110,16 +110,39 @@ def __init__(self, name=PRODUCT_CATALOG_CONFIG_MAP_NAME, namespace=PRODUCT_CATAL ) from err if len(configmaps) == 0: - raise ProductCatalogError( - f'No ConfigMaps found in {namespace} namespace.' + LOGGER.info( + 'No ConfigMaps found in namespace %s with label %s."', + namespace, PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR ) - - try: - self.products = load_config_map_data(self.name, configmaps) - except YAMLError as err: - raise ProductCatalogError( - f'Failed to load ConfigMap data: {err}' - ) from err + LOGGER.info('Getting data from ConfigMap %s/%s without label %s', + namespace, name, PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR) + try: + config_map = self.k8s_client.read_namespaced_config_map(name, namespace) + except MaxRetryError as err: + raise ProductCatalogError( + f'Unable to connect to Kubernetes to read {namespace}/{name} ConfigMap: {err}' + ) from err + except ApiException as err: + raise ProductCatalogError( + f'Error reading {namespace}/{name} ConfigMap: {err.reason}' + ) from err + if config_map.data is None: + raise ProductCatalogError( + f'No data found in {namespace}/{name} ConfigMap.' + ) + try: + self.products = load_cm_data(config_map) + except YAMLError as err: + raise ProductCatalogError( + f'Failed to load ConfigMap data: {err}' + ) from err + else: + try: + self.products = load_config_map_data(self.name, configmaps) + except YAMLError as err: + raise ProductCatalogError( + f'Failed to load ConfigMap data: {err}' + ) from err invalid_products = [ str(p) for p in self.products if not p.is_valid @@ -174,10 +197,26 @@ def get_product(self, name, version=None): return matching_products[0] +def load_cm_data(config_map): + """Parse read_namespaced_config_map output and get array of InstalledProductVersion objects. + + Args: + config_map (V1ConfigMap): ConfigMap object + + Returns: + An array of InstalledProductVersion objects. + """ + return [ + InstalledProductVersion(product_name, product_version, product_version_data) + for product_name, product_versions in config_map.data.items() + for product_version, product_version_data in safe_load(product_versions).items() + ] + def load_config_map_data(name, configmaps): """Parse list_namespaced_config_map output and get array of InstalledProductVersion objects. Args: + name: Main ConfigMap name with which all product ConfigMaps name starts configmaps (V1ConfigMapList): list of ConfigMap objects. Returns: @@ -244,7 +283,7 @@ def docker_images(self): for component in self.component_data.get(COMPONENT_DOCKER_KEY) or []] @property - def helm(self): + def helm_charts(self): """Get Helm charts associated with this InstalledProductVersion. Returns: diff --git a/tests/migration/test_config_map_data_handler.py b/tests/migration/test_config_map_data_handler.py index 728dd373..487d9b40 100755 --- a/tests/migration/test_config_map_data_handler.py +++ b/tests/migration/test_config_map_data_handler.py @@ -394,7 +394,7 @@ def test_rename_config_map_failed_4(self): f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, retrying..") self.assertEqual( captured.records[-1].getMessage(), - "Renaming ConfigMap successful") + f"Failed to delete ConfigMap {CONFIG_MAP_TEMP}, but migration is successful") def test_main_for_successful_migration(self): """Validating that migration is successful""" diff --git a/tests/mocks.py b/tests/mocks.py index df879ef6..afc6b987 100644 --- a/tests/mocks.py +++ b/tests/mocks.py @@ -30,6 +30,7 @@ from cray_product_catalog.query import InstalledProductVersion +MOCK_NAMESPACE = 'mock-namespace' # Two versions of a product named SAT where: # - The two versions have have no docker images in common with one another. # - Both have configurations, but neither have images or recipes diff --git a/tests/test_query.py b/tests/test_query.py index 9f3b62e3..31cbb38d 100644 --- a/tests/test_query.py +++ b/tests/test_query.py @@ -38,11 +38,13 @@ ProductCatalogError ) -from cray_product_catalog.constants import PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR +from cray_product_catalog.constants import ( + PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR, PRODUCT_CATALOG_CONFIG_MAP_NAME +) from tests.mocks import ( COS_VERSIONS, SAT_VERSIONS, CPE_VERSION, MOCK_PRODUCT_CATALOG_DATA, - MOCK_INVALID_PRODUCT_DATA, MOCK_PRODUCTS, MockInvalidYaml + MOCK_INVALID_PRODUCT_DATA, MOCK_PRODUCTS, MockInvalidYaml, MOCK_NAMESPACE ) @@ -82,6 +84,7 @@ def setUp(self): self.mock_k8s_api = patch.object(ProductCatalog, '_get_k8s_api').start().return_value self.mock_product_catalog_data = copy.deepcopy(MOCK_PRODUCT_CATALOG_DATA) self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=self.mock_product_catalog_data) + self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=self.mock_product_catalog_data) def tearDown(self): """Stop patches.""" @@ -89,9 +92,9 @@ def tearDown(self): def create_and_assert_product_catalog(self): """Assert the product catalog was created as expected.""" - product_catalog = ProductCatalog('cray-product-catalog', 'mock-namespace') + product_catalog = ProductCatalog(PRODUCT_CATALOG_CONFIG_MAP_NAME, MOCK_NAMESPACE) self.mock_k8s_api.list_namespaced_config_map.assert_called_once_with( - 'mock-namespace', label_selector=PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR + MOCK_NAMESPACE, label_selector=PRODUCT_CATALOG_CONFIG_MAP_LABEL_STR ) return product_catalog @@ -115,13 +118,41 @@ def test_create_product_catalog_invalid_product_data(self): self.create_and_assert_product_catalog() def test_create_product_catalog_null_data(self): - """Test creating a ProductCatalog when the product catalog contains null data.""" + """Test creating a ProductCatalog when the product catalog ConfigMaps with label + contains null data, but the product catalog ConfigMap without label has data""" + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = [] + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[]) + self.mock_load_cm_data = patch('cray_product_catalog.query.load_cm_data').start() + self.mock_load_cm_data.return_value = MOCK_PRODUCTS + with self.assertLogs(level=logging.DEBUG) as logs: + product_catalog = self.create_and_assert_product_catalog() + self.mock_k8s_api.read_namespaced_config_map.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, MOCK_NAMESPACE) + self.assertEqual(product_catalog.products, MOCK_PRODUCTS) + + def test_create_product_catalog_invalid_product_data1(self): + """Test creating a ProductCatalog when the product catalog ConfigMaps with label + contains null data, but the product catalog ConfigMap without label has invalid YAML.""" + self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() + self.mock_load_config_map_data.return_value = [] + self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[]) + self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=MockInvalidYaml().data) + with self.assertRaisesRegex(ProductCatalogError, 'Failed to load ConfigMap data'): + self.create_and_assert_product_catalog() + + def test_create_product_catalog_null_data1(self): + """Test creating a ProductCatalog when the product catalog ConfigMaps with label + and without label contains null data.""" self.mock_load_config_map_data = patch('cray_product_catalog.query.load_config_map_data').start() self.mock_load_config_map_data.return_value = [] self.mock_k8s_api.list_namespaced_config_map.return_value = Mock(items=[]) - with self.assertRaisesRegex(ProductCatalogError, - 'No ConfigMaps found in mock-namespace namespace.'): + self.mock_k8s_api.read_namespaced_config_map.return_value = Mock(data=None) + message = 'No data found in ' + MOCK_NAMESPACE + '/' + PRODUCT_CATALOG_CONFIG_MAP_NAME + ' ConfigMap.' + with self.assertRaisesRegex(ProductCatalogError, message): self.create_and_assert_product_catalog() + self.mock_k8s_api.read_namespaced_config_map.assert_called_once_with( + PRODUCT_CATALOG_CONFIG_MAP_NAME, MOCK_NAMESPACE + ) def check_for_invalid_product_schema(self, prod, ver): """Check for an ProductCatalog entry containing valid YAML but does not match schema. @@ -242,7 +273,7 @@ def test_helm_charts(self): ('cray-cps', '1.8.15') ] self.assertEqual( - expected_helm_charts_versions, self.installed_product_version.helm + expected_helm_charts_versions, self.installed_product_version.helm_charts ) def test_no_helm_charts(self): @@ -250,21 +281,21 @@ def test_no_helm_charts(self): product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': {}}} ) - self.assertEqual(product_with_no_helm_charts.helm, []) + self.assertEqual(product_with_no_helm_charts.helm_charts, []) def test_no_helm_charts_null(self): """Test a product that has None under the 'helm' key returns an empty list.""" product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': None}} ) - self.assertEqual(product_with_no_helm_charts.helm, []) + self.assertEqual(product_with_no_helm_charts.helm_charts, []) def test_no_helm_charts_empty_list(self): """Test a product that has an empty list under the 'helm' key returns an empty list.""" product_with_no_helm_charts = InstalledProductVersion( 'sat', '0.9.9', {'component_versions': {'helm': []}} ) - self.assertEqual(product_with_no_helm_charts.helm, []) + self.assertEqual(product_with_no_helm_charts.helm_charts, []) def test_s3_artifacts(self): """Test getting the s3 artifacts.""" From a0b1e3fb49c9b770cb2daa0d5a5253b56a165094 Mon Sep 17 00:00:00 2001 From: lathanm Date: Thu, 8 Feb 2024 15:57:07 +0530 Subject: [PATCH 11/12] Resolved pumod_test errors --- cray_product_catalog/migration/config_map_data_handler.py | 2 +- cray_product_catalog/query.py | 1 + tests/test_query.py | 4 +++- 3 files changed, 5 insertions(+), 2 deletions(-) diff --git a/cray_product_catalog/migration/config_map_data_handler.py b/cray_product_catalog/migration/config_map_data_handler.py index 649631ba..1ad5a4ae 100644 --- a/cray_product_catalog/migration/config_map_data_handler.py +++ b/cray_product_catalog/migration/config_map_data_handler.py @@ -120,7 +120,7 @@ def migrate_config_map_data(self, config_map_data): main_versions_data[version_data] = main_cm_data # If `component_versions` data exists for a product, create new product ConfigMap if product_versions_data: - product_config_map_data= { + product_config_map_data = { product: yaml.safe_dump(product_versions_data, default_flow_style=False) } product_config_map_data_list.append(product_config_map_data) diff --git a/cray_product_catalog/query.py b/cray_product_catalog/query.py index 931c1851..3271e826 100644 --- a/cray_product_catalog/query.py +++ b/cray_product_catalog/query.py @@ -212,6 +212,7 @@ def load_cm_data(config_map): for product_version, product_version_data in safe_load(product_versions).items() ] + def load_config_map_data(name, configmaps): """Parse list_namespaced_config_map output and get array of InstalledProductVersion objects. diff --git a/tests/test_query.py b/tests/test_query.py index 31cbb38d..77ea7f35 100644 --- a/tests/test_query.py +++ b/tests/test_query.py @@ -127,7 +127,9 @@ def test_create_product_catalog_null_data(self): self.mock_load_cm_data.return_value = MOCK_PRODUCTS with self.assertLogs(level=logging.DEBUG) as logs: product_catalog = self.create_and_assert_product_catalog() - self.mock_k8s_api.read_namespaced_config_map.assert_called_once_with(PRODUCT_CATALOG_CONFIG_MAP_NAME, MOCK_NAMESPACE) + self.mock_k8s_api.read_namespaced_config_map.assert_called_once_with( + PRODUCT_CATALOG_CONFIG_MAP_NAME, MOCK_NAMESPACE + ) self.assertEqual(product_catalog.products, MOCK_PRODUCTS) def test_create_product_catalog_invalid_product_data1(self): From 57713cbf7bfc4747444543b877c96a9b9a2499b2 Mon Sep 17 00:00:00 2001 From: Mitch Harding Date: Thu, 8 Feb 2024 14:37:31 -0500 Subject: [PATCH 12/12] Apply suggestions from code review --- .gitignore | 1 - CHANGELOG.md | 2 +- charts/cray-product-catalog/templates/configmap-hook.yaml | 1 - cray_product_catalog/catalog_update.py | 2 +- 4 files changed, 2 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index dbb03c4e..2b19cc4a 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,6 @@ dist/ # VS Code config workspace.code-workspace - # cache specific *.pyc **__pycache__** diff --git a/CHANGELOG.md b/CHANGELOG.md index 52143fca..97e0946c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -## [2.0.0] - 2024-01-16 +## [2.0.0] - 2024-02-08 ### Changed diff --git a/charts/cray-product-catalog/templates/configmap-hook.yaml b/charts/cray-product-catalog/templates/configmap-hook.yaml index 9835d9f2..8c32f855 100644 --- a/charts/cray-product-catalog/templates/configmap-hook.yaml +++ b/charts/cray-product-catalog/templates/configmap-hook.yaml @@ -77,7 +77,6 @@ spec: yq eval 'del(.metadata.resourceVersion, .metadata.uid, .metadata.annotations, .metadata.creationTimestamp, .metadata.selfLink, .metadata.managedFields)' - | kubectl create -f -" - --- apiVersion: batch/v1 kind: Job diff --git a/cray_product_catalog/catalog_update.py b/cray_product_catalog/catalog_update.py index 4e5d012a..2907f5ea 100755 --- a/cray_product_catalog/catalog_update.py +++ b/cray_product_catalog/catalog_update.py @@ -148,7 +148,7 @@ def create_config_map(api_instance, name, namespace): LOGGER.info("Created product ConfigMap %s/%s", namespace, name) return True except ApiException: - LOGGER.error("Error calling create_namespaced_config_map") + LOGGER.exception("Error calling create_namespaced_config_map") return False