fix minor version for release pipeline - Spark3.5 (#150)

* Build Spark 3.5

* Update Dockerfile to build for AL2023

* release Spark 3.4-cpu-py39-v1.1 (#141)

Co-authored-by: Ajay Mahendru <[email protected]>

* Build Spark 3.5

* release Spark 3.4-cpu-py39-v1.1 (#141)

Co-authored-by: Ajay Mahendru <[email protected]>

* 3.4 cpu py39 v1.1 (minor fixes for hadoop-aws integration with aws-java-sdk-v2) (#142)

* release Spark 3.4-cpu-py39-v1.1

* make fixes for hadoop-aws integration with aws-java-sdk-v2

---------

Co-authored-by: Ajay Mahendru <[email protected]>

* 3.4 cpu py39 v1.1 (#143)

* release Spark 3.4-cpu-py39-v1.1

* make fixes for hadoop-aws integration with aws-java-sdk-v2

* fix lint errors

---------

Co-authored-by: Ajay Mahendru <[email protected]>

* 3.4 cpu py39 v1.1 (#144)

* release Spark 3.4-cpu-py39-v1.1

* make fixes for hadoop-aws integration with aws-java-sdk-v2

* fix lint errors

* Fix test expectations for a job failed with AlgorithError as we ramp up traffic for V2 stack

---------

Co-authored-by: Ajay Mahendru <[email protected]>

* Build Spark 3.5

* Add aws-java-sdk-v2 on Spark paths

* release Spark 3.4-cpu-py39-v1.1 (#141)

Co-authored-by: Ajay Mahendru <[email protected]>

* 3.4 cpu py39 v1.1 (#146)

* release Spark 3.4-cpu-py39-v1.1

* make fixes for hadoop-aws integration with aws-java-sdk-v2

* fix lint errors

* Fix test expectations for a job failed with AlgorithError as we ramp up traffic for V2 stack

* updating OS libs to address more CVEs

---------

Co-authored-by: Ajay Mahendru <[email protected]>

* Build Spark 3.5

* release Spark 3.4-cpu-py39-v1.1 (#141)

Co-authored-by: Ajay Mahendru <[email protected]>

* Build Spark 3.5 release: 1/ Update to EMR 7.0.0 2/Update base image to use AL2023

* update Dockerfile for Spark 3.5 to optimize cache clean

* Fix SM_VERSION for Spark 3.5 release

* fix minor version for release pipeline

---------

Co-authored-by: Ajay Mahendru <[email protected]>

new_images.yml

@@ -4,4 +4,4 @@ new_images:
     use-case: "processing"
     processors: ["cpu"]
     python: ["py39"]
-    sm_version: "1.1"
+    sm_version: "1.0"