BGD-6054 spark-watcher 0.6.2

[abunghez]

Jira ticket

https://spotinst.atlassian.net/browse/BGD-6054

Description

Fix dataplane kube event log regional storage for gcp and azure.

Demo

Checklist

• I have added a Jira ticket link
• I have filled in the test plan
• I have executed the tests and filled in the test results
• I have updated/created relevant documentation

How to test

Integration tests on AWS

Test plan and results

%  devctl cp test run-with-params --token=$DEVCTL_PAT --accountId=$DEVCTL_ACCOUNT --clusterId=$DEVCTL_CLUSTER --optimizer-url=$OPTIMIZER_URL --concurrency-limit=5
2:27PM INF Checking cluster availability clusterId=osc-940e76ef
2:27PM INF Cluster is available, tests are ready to run clusterId=osc-940e76ef
2:27PM INF Start running tests concurrency-limit=5 now=2024-11-20T14:27:03+01:00 version=v1.0.47e
2:27PM INF Optimizer Url: https://api.spotinst.io
2:27PM INF accountId: act-12f6b1b9
2:27PM INF clusterId: osc-940e76ef
2:27PM INF sparkProps: [{Version:3.5.1 Image:}]
2:27PM INF 20 tests selected: [list_applications, kill_application_pending_3.5.1, kill_application_running_3.5.1, kill_application_completed_3.5.1, list_jobs, get_job, put_job, list_clusters, get_config_template, create_valid_config_template, create_invalid_config_template, list_config_templates, put_config_templates, delete_config_template, run_workspace, run_spark_connect_application-3.5.1, run_spark_connect_application_code-3.5.1, run_notebook_application-python-3.5.1, run_notebook_application-scala-3.5.1, run_application-3.5.1]
2:27PM INF Starting suite=get_job
2:27PM INF Starting suite=kill_application_running_3.5.1
2:27PM INF Starting suite=kill_application_completed_3.5.1
2:27PM INF Starting suite=list_applications
2:27PM INF Starting suite=list_jobs
2:27PM INF Success suite=list_jobs
2:27PM INF Starting suite=put_job
2:27PM INF Submitted app integration-tests-job-get-vkkpjr-8fc75-legal suite=get_job
2:27PM INF Submitted app integration-tests-279ca-allay suite=kill_application_completed_3.5.1
2:27PM INF Submitted app integration-tests-0568e-farce suite=kill_application_running_3.5.1
2:27PM INF Success suite=list_applications
2:27PM INF Starting suite=list_clusters
2:27PM INF Job fetched successfully suite=get_job
2:27PM INF Got a correct job suite=get_job
2:27PM INF App is processed suite=kill_application_completed_3.5.1
2:27PM INF App is processed suite=kill_application_running_3.5.1
2:27PM INF Success suite=list_clusters
2:27PM INF Starting suite=get_config_template
2:27PM INF Config template created successfully suite=get_config_template
2:27PM INF Submitted app integration-tests-job-put-npzowc-8af3d-crows suite=put_job
2:27PM INF Job config has updated suite=put_job
2:27PM INF Updated job labels are correct suite=put_job
2:27PM INF Config template fetched successfully suite=get_config_template
2:27PM INF Got a correct config template suite=get_config_template
2:27PM INF Success suite=get_config_template
2:27PM INF Starting suite=create_valid_config_template
2:27PM INF Trying to create config template integration-tests-config-template-create-kcpbaa suite=create_valid_config_template
2:27PM INF Created config template integration-tests-config-template-create-kcpbaa suite=create_valid_config_template
2:27PM INF Trying to create config template with same name integration-tests-config-template-create-kcpbaa suite=create_valid_config_template
2:27PM INF Successfully received bad request response for integration-tests-config-template-create-kcpbaa suite=create_valid_config_template
2:27PM INF Success suite=create_valid_config_template
2:27PM INF Starting suite=create_invalid_config_template
2:27PM INF Trying to create invalid config template integration-tests-config-template-create-vwlwbg suite=create_invalid_config_template
2:27PM INF Successfully received unprocessable entity response for integration-tests-config-template-create-vwlwbg suite=create_invalid_config_template
2:27PM INF Success suite=create_invalid_config_template
2:27PM INF Starting suite=list_config_templates
2:27PM INF Config template created successfully suite=list_config_templates
2:27PM INF Trying to list the config templates suite=list_config_templates
2:27PM INF Config template fetched successfully suite=list_config_templates
2:27PM INF Got a correct config template list suite=list_config_templates
2:27PM INF Success suite=list_config_templates
2:27PM INF Starting suite=put_config_templates
2:27PM INF Config template created successfully suite=put_config_templates
2:27PM INF Config template successfully updated suite=put_config_templates
2:27PM INF Config template fetched successfully suite=put_config_templates
2:27PM INF Got a correct config template suite=put_config_templates
2:27PM INF Success suite=put_config_templates
2:27PM INF Starting suite=delete_config_template
2:27PM INF Config template created successfully suite=delete_config_template
2:27PM INF Config template successfully deleted suite=delete_config_template
2:27PM INF Success suite=delete_config_template
2:27PM INF Starting suite=run_workspace
2:27PM INF Workspace creation test suite=run_workspace
2:27PM INF Workspace created: wksp-324159-46cab744 suite=run_workspace
2:27PM INF App is now running. suite=kill_application_running_3.5.1
2:27PM INF App killed suite=kill_application_running_3.5.1
2:27PM INF Success suite=get_job
2:27PM INF Starting suite=run_spark_connect_application-3.5.1
2:27PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application-3.5.1
2:27PM INF Submitted spark connect app integration-tests-11e3b-locks suite=run_spark_connect_application-3.5.1
2:27PM INF Success suite=kill_application_running_3.5.1
2:27PM INF Starting suite=run_spark_connect_application_code-3.5.1
2:27PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application_code-3.5.1
2:27PM INF Submitted spark connect app integration-tests-ae875-skims suite=run_spark_connect_application_code-3.5.1
2:27PM INF Success suite=put_job
2:27PM INF Starting suite=run_notebook_application-python-3.5.1
2:27PM INF Creating config template it-notebook-python-jgxajc. suite=run_notebook_application-python-3.5.1
2:27PM INF Testing Spark submit with properties Python suite=run_notebook_application-python-3.5.1
2:27PM INF Spark connect app is now running suite=run_spark_connect_application-3.5.1
2:27PM INF Spark connect app killed suite=run_spark_connect_application-3.5.1
2:27PM INF Spark connect app is now running suite=run_spark_connect_application_code-3.5.1
2:27PM INF Workspace is running: wksp-324159-46cab744 suite=run_workspace
2:27PM INF Workspace killed: wksp-324159-46cab744 suite=run_workspace
2:27PM INF Success suite=run_workspace
2:27PM INF Starting suite=run_notebook_application-scala-3.5.1
2:27PM INF Creating config template it-notebook-scala-pdwgkq. suite=run_notebook_application-scala-3.5.1
2:27PM INF Success suite=run_spark_connect_application-3.5.1
2:27PM INF Starting suite=run_application-3.5.1
2:27PM INF Testing Spark submit with properties {3.5.1 } suite=run_application-3.5.1
2:27PM INF Testing Spark submit with properties Scala suite=run_notebook_application-scala-3.5.1
2:27PM INF Submitted app integration-tests-9bd82-bowed suite=run_application-3.5.1
2:27PM INF App is processed suite=run_application-3.5.1
2:27PM INF Listing cluster notebook applications for type: Python suite=run_notebook_application-python-3.5.1
2:27PM INF App is processed suite=run_notebook_application-python-3.5.1
2:27PM INF App is now running. suite=run_notebook_application-python-3.5.1
2:27PM INF Get App status. suite=run_notebook_application-python-3.5.1
2:27PM INF App is completed suite=kill_application_completed_3.5.1
2:27PM INF Success suite=kill_application_completed_3.5.1
2:27PM INF Cleaning up config template it-notebook-python-jgxajc. suite=run_notebook_application-python-3.5.1
2:27PM INF Success suite=run_notebook_application-python-3.5.1
2:28PM INF Live kubernetes logs are available suite=run_application-3.5.1
2:28PM INF Tests status failure=[] in_progress=["run_application-3.5.1","run_spark_connect_application_code-3.5.1","run_notebook_application-scala-3.5.1"] success=16 to_do=0
2:28PM INF Listing cluster notebook applications for type: Scala suite=run_notebook_application-scala-3.5.1
2:28PM INF App is processed suite=run_notebook_application-scala-3.5.1
2:28PM INF App is now running. suite=run_notebook_application-scala-3.5.1
2:28PM INF Get App status. suite=run_notebook_application-scala-3.5.1
2:28PM INF Cleaning up config template it-notebook-scala-pdwgkq. suite=run_notebook_application-scala-3.5.1
2:28PM INF Success suite=run_notebook_application-scala-3.5.1
2:28PM INF App is now running. suite=run_application-3.5.1
2:28PM INF Live driver logs are available suite=run_application-3.5.1
2:28PM INF App is completed suite=run_application-3.5.1
2:28PM INF Spark connect app executed sql suite=run_spark_connect_application_code-3.5.1
2:28PM INF Spark connect app killed suite=run_spark_connect_application_code-3.5.1
2:28PM INF Success suite=run_spark_connect_application_code-3.5.1
2:29PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:30PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:31PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:32PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2:33PM INF Tests status failure=[] in_progress=["run_application-3.5.1"] success=18 to_do=0
2024/11/20 14:33:52 c.ReadMessage: websocket: RSV1 set, FIN not set on control
2:33PM INF Logs are processed suite=run_application-3.5.1
2:33PM INF Driver log archive is available suite=run_application-3.5.1
2:33PM INF Success suite=run_application-3.5.1
2:33PM INF Starting sequential tests
2:33PM INF Starting suite=kill_application_pending_3.5.1
2:33PM INF Suites run final status failure=[] in_progress=[] success=19 to_do=0
2:33PM INF Submitted app integration-tests-3634e-stilt suite=kill_application_pending_3.5.1
2:33PM INF App is processed suite=kill_application_pending_3.5.1
2:33PM INF App killed suite=kill_application_pending_3.5.1
2:34PM INF Success suite=kill_application_pending_3.5.1
2:34PM INF All tests are successful

Test	Description	Result	Notes
1	Integration tests on AWS	Pass
2	integration tests on Azure	Pass

[abunghez]

Integration tests on Azure:

bandrei@bandrei-mac-0 ~/code/bigdata-charts %  devctl cp test run-with-params --token=$DEVCTL_PAT --accountId=$DEVCTL_ACCOUNT --clusterId=$DEVCTL_CLUSTER --optimizer-url=$OPTIMIZER_URL --concurrency-limit=5
7:58PM INF Checking cluster availability clusterId=osc-c52b62b1
7:58PM INF Cluster is available, tests are ready to run clusterId=osc-c52b62b1
7:58PM INF Start running tests concurrency-limit=5 now=2024-11-20T19:58:58+01:00 version=v1.0.48
7:58PM INF Optimizer Url: https://api.spotinst.io
7:58PM INF accountId: act-3a63c598
7:58PM INF clusterId: osc-c52b62b1
7:58PM INF sparkProps: [{Version:3.5.1 Image:}]
7:58PM INF 20 tests selected: [list_applications, kill_application_pending_3.5.1, kill_application_running_3.5.1, kill_application_completed_3.5.1, list_jobs, get_job, put_job, list_clusters, get_config_template, create_valid_config_template, create_invalid_config_template, list_config_templates, put_config_templates, delete_config_template, run_workspace, run_spark_connect_application-3.5.1, run_spark_connect_application_code-3.5.1, run_notebook_application-python-3.5.1, run_notebook_application-scala-3.5.1, run_application-3.5.1]
7:58PM INF Starting suite=get_job
7:58PM INF Starting suite=kill_application_running_3.5.1
7:58PM INF Starting suite=list_applications
7:58PM INF Starting suite=kill_application_completed_3.5.1
7:58PM INF Starting suite=list_jobs
7:58PM INF Success suite=list_jobs
7:58PM INF Starting suite=put_job
7:58PM INF Success suite=list_applications
7:58PM INF Starting suite=list_clusters
7:58PM INF Success suite=list_clusters
7:58PM INF Starting suite=get_config_template
7:58PM INF Config template created successfully suite=get_config_template
7:58PM INF Submitted app integration-tests-d5b67-herds suite=kill_application_completed_3.5.1
7:58PM INF Submitted app integration-tests-job-get-oztkhi-69b2a-chomp suite=get_job
7:58PM INF Job fetched successfully suite=get_job
7:58PM INF Got a correct job suite=get_job
7:58PM INF Submitted app integration-tests-job-put-hyxult-ff4c6-halve suite=put_job
7:58PM INF App is processed suite=kill_application_completed_3.5.1
7:58PM INF Job config has updated suite=put_job
7:58PM INF Updated job labels are correct suite=put_job
7:58PM INF Submitted app integration-tests-3fcf9-wetly suite=kill_application_running_3.5.1
7:59PM INF App is processed suite=kill_application_running_3.5.1
7:59PM INF Config template fetched successfully suite=get_config_template
7:59PM INF Got a correct config template suite=get_config_template
7:59PM INF Success suite=get_config_template
7:59PM INF Starting suite=create_valid_config_template
7:59PM INF Trying to create config template integration-tests-config-template-create-mhvxql suite=create_valid_config_template
7:59PM INF Created config template integration-tests-config-template-create-mhvxql suite=create_valid_config_template
7:59PM INF Trying to create config template with same name integration-tests-config-template-create-mhvxql suite=create_valid_config_template
7:59PM INF Successfully received bad request response for integration-tests-config-template-create-mhvxql suite=create_valid_config_template
7:59PM INF Success suite=create_valid_config_template
7:59PM INF Starting suite=create_invalid_config_template
7:59PM INF Trying to create invalid config template integration-tests-config-template-create-kmmari suite=create_invalid_config_template
7:59PM INF Successfully received unprocessable entity response for integration-tests-config-template-create-kmmari suite=create_invalid_config_template
7:59PM INF Success suite=create_invalid_config_template
7:59PM INF Starting suite=list_config_templates
7:59PM INF Config template created successfully suite=list_config_templates
7:59PM INF Trying to list the config templates suite=list_config_templates
7:59PM INF Config template fetched successfully suite=list_config_templates
7:59PM INF Got a correct config template list suite=list_config_templates
7:59PM INF Success suite=list_config_templates
7:59PM INF Starting suite=put_config_templates
7:59PM INF Config template created successfully suite=put_config_templates
7:59PM INF Config template successfully updated suite=put_config_templates
7:59PM INF Config template fetched successfully suite=put_config_templates
7:59PM INF Got a correct config template suite=put_config_templates
7:59PM INF Success suite=put_config_templates
7:59PM INF Starting suite=delete_config_template
7:59PM INF Config template created successfully suite=delete_config_template
7:59PM INF Config template successfully deleted suite=delete_config_template
7:59PM INF Success suite=delete_config_template
7:59PM INF Starting suite=run_workspace
7:59PM INF Workspace creation test suite=run_workspace
7:59PM INF Workspace created: wksp-324159-1ee6208f suite=run_workspace
7:59PM INF Success suite=get_job
7:59PM INF Starting suite=run_spark_connect_application-3.5.1
7:59PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application-3.5.1
7:59PM INF App is now running. suite=kill_application_running_3.5.1
7:59PM INF App killed suite=kill_application_running_3.5.1
7:59PM INF Success suite=put_job
7:59PM INF Starting suite=run_spark_connect_application_code-3.5.1
7:59PM INF Testing spark connect app submit with properties {3.5.1 } suite=run_spark_connect_application_code-3.5.1
7:59PM INF Submitted spark connect app integration-tests-97d9c-divot suite=run_spark_connect_application-3.5.1
7:59PM INF Success suite=kill_application_running_3.5.1
7:59PM INF Starting suite=run_notebook_application-python-3.5.1
7:59PM INF Creating config template it-notebook-python-rafiiv. suite=run_notebook_application-python-3.5.1
7:59PM INF Testing Spark submit with properties Python suite=run_notebook_application-python-3.5.1
7:59PM INF Submitted spark connect app integration-tests-6ee89-towel suite=run_spark_connect_application_code-3.5.1
7:59PM INF Workspace is running: wksp-324159-1ee6208f suite=run_workspace
7:59PM INF Workspace killed: wksp-324159-1ee6208f suite=run_workspace
7:59PM INF Success suite=run_workspace
7:59PM INF Starting suite=run_notebook_application-scala-3.5.1
7:59PM INF Creating config template it-notebook-scala-doloca. suite=run_notebook_application-scala-3.5.1
7:59PM INF Testing Spark submit with properties Scala suite=run_notebook_application-scala-3.5.1
7:59PM INF Spark connect app is now running suite=run_spark_connect_application-3.5.1
7:59PM INF Spark connect app killed suite=run_spark_connect_application-3.5.1
7:59PM INF Spark connect app is now running suite=run_spark_connect_application_code-3.5.1
7:59PM INF Success suite=run_spark_connect_application-3.5.1
7:59PM INF Starting suite=run_application-3.5.1
7:59PM INF Testing Spark submit with properties {3.5.1 } suite=run_application-3.5.1
7:59PM INF Listing cluster notebook applications for type: Python suite=run_notebook_application-python-3.5.1
7:59PM INF App is processed suite=run_notebook_application-python-3.5.1
7:59PM INF App is now running. suite=run_notebook_application-python-3.5.1
7:59PM INF Get App status. suite=run_notebook_application-python-3.5.1
7:59PM INF Submitted app integration-tests-312a9-scale suite=run_application-3.5.1
7:59PM INF App is processed suite=run_application-3.5.1
7:59PM INF Live kubernetes logs are available suite=run_application-3.5.1
7:59PM INF Cleaning up config template it-notebook-python-rafiiv. suite=run_notebook_application-python-3.5.1
7:59PM INF Success suite=run_notebook_application-python-3.5.1
7:59PM INF Tests status failure=[] in_progress=["run_spark_connect_application_code-3.5.1","run_application-3.5.1","kill_application_completed_3.5.1","run_notebook_application-scala-3.5.1"] success=15 to_do=0
8:00PM INF Listing cluster notebook applications for type: Scala suite=run_notebook_application-scala-3.5.1
8:00PM INF App is processed suite=run_notebook_application-scala-3.5.1
8:00PM INF App is now running. suite=run_notebook_application-scala-3.5.1
8:00PM INF Get App status. suite=run_notebook_application-scala-3.5.1
8:00PM INF Cleaning up config template it-notebook-scala-doloca. suite=run_notebook_application-scala-3.5.1
8:00PM INF Success suite=run_notebook_application-scala-3.5.1
8:00PM INF App is now running. suite=run_application-3.5.1
8:00PM INF Live driver logs are available suite=run_application-3.5.1
8:00PM INF App is completed suite=run_application-3.5.1
8:00PM INF Spark connect app executed sql suite=run_spark_connect_application_code-3.5.1
8:00PM INF Spark connect app killed suite=run_spark_connect_application_code-3.5.1
8:00PM INF Success suite=run_spark_connect_application_code-3.5.1
8:00PM INF Tests status failure=[] in_progress=["run_application-3.5.1","kill_application_completed_3.5.1"] success=17 to_do=0
8:01PM INF Tests status failure=[] in_progress=["kill_application_completed_3.5.1","run_application-3.5.1"] success=17 to_do=0
8:02PM INF Tests status failure=[] in_progress=["run_application-3.5.1","kill_application_completed_3.5.1"] success=17 to_do=0
8:03PM INF Tests status failure=[] in_progress=["kill_application_completed_3.5.1","run_application-3.5.1"] success=17 to_do=0
8:04PM INF Logs are processed suite=run_application-3.5.1
8:04PM INF Driver log archive is available suite=run_application-3.5.1
8:04PM INF Success suite=run_application-3.5.1
8:04PM INF Tests status failure=[] in_progress=["kill_application_completed_3.5.1"] success=18 to_do=0
8:05PM INF App is completed suite=kill_application_completed_3.5.1
8:05PM INF Success suite=kill_application_completed_3.5.1
8:05PM INF Starting sequential tests
8:05PM INF Starting suite=kill_application_pending_3.5.1
8:05PM INF Suites run final status failure=[] in_progress=[] success=19 to_do=0
8:05PM INF Submitted app integration-tests-14147-genus suite=kill_application_pending_3.5.1
8:05PM INF App is processed suite=kill_application_pending_3.5.1
8:05PM INF App killed suite=kill_application_pending_3.5.1
2024/11/20 20:05:58 c.ReadMessage: websocket: RSV1 set, FIN not set on control
8:06PM INF Success suite=kill_application_pending_3.5.1
8:06PM INF All tests are successful