feat(general): add support for Python 3.11 (aws-powertools#2820)

rubenfonseca · leandrodamascena · web-flow · commit dfb989288a43 · 2023-07-24T13:18:04.000+01:00
* feat: add support for Python 3.11

* fix: remove python-snappY

* fastjsonschema: test change due to behavior change in fastjsonschem

* LRU: test

* Removing snappy since it doesnt offer to Python3.11

* Adding support to Python3.11 in the layer for test

* Skipping test

* Skipping test for Python 3.11+

---------

Co-authored-by: Leandro Damascena &lt;lcdama@amazon.pt&gt;
diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml
@@ -62,6 +62,7 @@ body:
         - "3.8"
         - "3.9"
         - "3.10"
+        - "3.11"
     validations:
       required: true
   - type: dropdown
diff --git a/.github/ISSUE_TEMPLATE/static_typing.yml b/.github/ISSUE_TEMPLATE/static_typing.yml
@@ -29,6 +29,7 @@ body:
         - "3.8"
         - "3.9"
         - "3.10"
+        - "3.11"
     validations:
       required: true
   - type: input
diff --git a/.github/workflows/publish_v2_layer.yml b/.github/workflows/publish_v2_layer.yml
@@ -107,7 +107,7 @@ jobs:
       - name: Setup python
         uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "pip"
       - name: Resolve and install project dependencies
         # CDK spawns system python when compiling stack
diff --git a/.github/workflows/quality_check.yml b/.github/workflows/quality_check.yml
@@ -44,7 +44,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
     env:
       PYTHON: "${{ matrix.python-version }}"
     permissions:
diff --git a/.github/workflows/quality_check_pydanticv2.yml b/.github/workflows/quality_check_pydanticv2.yml
@@ -44,7 +44,7 @@ jobs:
     strategy:
       max-parallel: 4
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
     env:
       PYTHON: "${{ matrix.python-version }}"
     permissions:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -133,7 +133,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "poetry"
       - name: Install dependencies
         run: make dev
@@ -171,7 +171,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "poetry"
 
       - name: Build python package and wheel
diff --git a/.github/workflows/reusable_deploy_v2_layer_stack.yml b/.github/workflows/reusable_deploy_v2_layer_stack.yml
@@ -160,7 +160,7 @@ jobs:
       - name: Setup python
         uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "pip"
       - name: Resolve and install project dependencies
         # CDK spawns system python when compiling stack
diff --git a/.github/workflows/reusable_publish_docs.yml b/.github/workflows/reusable_publish_docs.yml
@@ -52,7 +52,7 @@ jobs:
       - name: Set up Python
         uses: actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1 # v4.7.0
         with:
-          python-version: "3.10"
+          python-version: "3.11"
           cache: "poetry"
       - name: Install dependencies
         run: make dev
diff --git a/.github/workflows/run-e2e-tests.yml b/.github/workflows/run-e2e-tests.yml
@@ -47,7 +47,7 @@ jobs:
     strategy:
       fail-fast: false # needed so if a version fails, the others will still be able to complete and cleanup
       matrix:
-        version: ["3.7", "3.8", "3.9", "3.10"]
+        version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
     if: ${{ github.actor != 'dependabot[bot]' }}
     steps:
       - name: "Checkout"
diff --git a/README.md b/README.md
@@ -3,7 +3,7 @@
 
 [![Build](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/quality_check.yml/badge.svg)](https://github.com/aws-powertools/powertools-lambda-python/actions/workflows/python_build.yml)
 [![codecov.io](https://codecov.io/github/aws-powertools/powertools-lambda-python/branch/develop/graphs/badge.svg)](https://app.codecov.io/gh/aws-powertools/powertools-lambda-python)
-![PythonSupport](https://img.shields.io/static/v1?label=python&message=%203.7|%203.8|%203.9|%203.10&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools) [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/aws-powertools/powertools-lambda-python/badge)](https://api.securityscorecards.dev/projects/github.com/aws-powertools/powertools-lambda-python) [![Join our Discord](https://dcbadge.vercel.app/api/server/B8zZKbbyET)](https://discord.gg/B8zZKbbyET)
+![PythonSupport](https://img.shields.io/static/v1?label=python&message=%203.7|%203.8|%203.9|%203.10|%203.11&color=blue?style=flat-square&logo=python) ![PyPI version](https://badge.fury.io/py/aws-lambda-powertools.svg) ![PyPi monthly downloads](https://img.shields.io/pypi/dm/aws-lambda-powertools) [![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/aws-powertools/powertools-lambda-python/badge)](https://api.securityscorecards.dev/projects/github.com/aws-powertools/powertools-lambda-python) [![Join our Discord](https://dcbadge.vercel.app/api/server/B8zZKbbyET)](https://discord.gg/B8zZKbbyET)
 
 Powertools for AWS Lambda (Python) is a developer toolkit to implement Serverless [best practices and increase developer velocity](https://docs.powertools.aws.dev/lambda/python/latest/#features).
 
diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md
@@ -20,11 +20,11 @@ Let's clone our sample project before we add one feature at a time.
     Bootstrap directly via SAM CLI:
 
     ```shell
-    sam init --app-template hello-world-powertools-python --name sam-app --package-type Zip --runtime python3.10 --no-tracing`
+    sam init --app-template hello-world-powertools-python --name sam-app --package-type Zip --runtime python3.11 --no-tracing`
     ```
 
 ```bash title="Use SAM CLI to initialize the sample project"
-sam init --runtime python3.10 --dependency-manager pip --app-template hello-world --name powertools-quickstart
+sam init --runtime python3.11 --dependency-manager pip --app-template hello-world --name powertools-quickstart
 ```
 
 ### Project structure
diff --git a/docs/utilities/jmespath_functions.md b/docs/utilities/jmespath_functions.md
@@ -189,7 +189,7 @@ This sample will decompress and decode base64 data from Cloudwatch Logs, then us
 
 For special binary formats that you want to decode before applying JSON Schema validation, you can bring your own [JMESPath function](https://github.com/jmespath/jmespath.py#custom-functions){target="_blank"} and any additional option via `jmespath_options` param. To keep Powertools for AWS Lambda (Python) built-in functions, you can subclass from `PowertoolsFunctions`.
 
-Here is an example of how to decompress messages using [snappy](https://github.com/andrix/python-snappy){target="_blank"}:
+Here is an example of how to decompress messages using [zlib](https://docs.python.org/3/library/zlib.html){target="_blank"}:
 
 === "powertools_custom_jmespath_function.py"
 
diff --git a/docs/we_made_this.md b/docs/we_made_this.md
@@ -37,7 +37,7 @@ A collection of articles explaining in detail how Lambda Powertools helps with a
 
 * [Serverless API Idempotency with AWS Lambda Powertools and CDK](https://www.ranthebuilder.cloud/post/serverless-api-idempotency-with-aws-lambda-powertools-and-cdk){:target="_blank"}
 
-* [Effective Amazon SQS Batch Handling with Powertools for AWS Lambda (Python)(https://www.ranthebuilder.cloud/post/effective-amazon-sqs-batch-handling-with-aws-lambda-powertools){:target="_blank"}
+* [Effective Amazon SQS Batch Handling with Powertools for AWS Lambda (Python)](https://www.ranthebuilder.cloud/post/effective-amazon-sqs-batch-handling-with-aws-lambda-powertools){:target="_blank"}
 
 ### Making all your APIs idempotent
 
diff --git a/examples/idempotency/templates/terraform.tf b/examples/idempotency/templates/terraform.tf
@@ -28,7 +28,7 @@ resource "aws_dynamodb_table" "IdempotencyTable" {
 resource "aws_lambda_function" "IdempotencyFunction" {
   function_name = "IdempotencyFunction"
   role          = aws_iam_role.IdempotencyFunctionRole.arn
-  runtime       = "python3.10"
+  runtime       = "python3.11"
   handler       = "app.lambda_handler"
   filename      = "lambda.zip"
 
diff --git a/examples/jmespath_functions/src/powertools_custom_jmespath_function.json b/examples/jmespath_functions/src/powertools_custom_jmespath_function.json
@@ -1,14 +1,9 @@
 {
     "Records": [
         {
-            "user": "integration-kafka",
+            "application": "notification",
             "datetime": "2022-01-01T00:00:00.000Z",
-            "log": "/QGIMjAyMi8wNi8xNiAxNjoyNTowMCBbY3JpdF0gMzA1MTg5MCMNCPBEOiAqMSBjb25uZWN0KCkg\ndG8gMTI3LjAuMC4xOjUwMDAgZmFpbGVkICgxMzogUGVybWlzc2lvbiBkZW5pZWQpIHdoaWxlEUEI\naW5nAUJAdXBzdHJlYW0sIGNsaWVudDoZVKgsIHNlcnZlcjogXywgcmVxdWVzdDogIk9QVElPTlMg\nLyBIVFRQLzEuMSIsFUckOiAiaHR0cDovLzabABQvIiwgaG8FQDAxMjcuMC4wLjE6ODEi\n"
-        },
-        {
-            "user": "integration-kafka",
-            "datetime": "2022-01-01T00:00:01.000Z",
-            "log": "tQHwnDEyNy4wLjAuMSAtIC0gWzE2L0p1bi8yMDIyOjE2OjMwOjE5ICswMTAwXSAiT1BUSU9OUyAv\nIEhUVFAvMS4xIiAyMDQgMCAiLSIgIk1vemlsbGEvNS4wIChYMTE7IExpbnV4IHg4Nl82NCkgQXBw\nbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwMi4BmUwwIFNhZmFy\naS81MzcuMzYiICItIg==\n"
+            "log": "eJxtUNFOwkAQ/JXN+dJq6e22tMD5ZHwQRYSEJpqQhtRy2AvlWq+tEr/eg6DExOzDJjM7M5tZsgCDgGPMKQaKRRAJRFjmRrUphBjRcIQXpy3gkiCvtJZ567jQVkDBwEc7JCK0sk2mSrkGh0IBc2l2qmlUpWEttZJrFz4LS/8YKP12cOjqpjUy23mQl0rqVpw9PWik+ZBGwMoDI9872ViazebJ/expARzGSTLn5BPzfm0sX7RtLTj/+xq3N0V11J+JITELnwHo2VlSzB86zQ+1CFtIiIJGcIWEmP4bDgH2AYH1GLBp9aXKMuORj+C8EF3Do9LdHvbDeBX3Xbip61I+y9eJankUDvwwBmcyTqaPHpRqK+FO5tvKhdvCVDvJCYPjYwiLbJMZdZKwQxZL02+NI3Vs"
         }
     ]
 }
diff --git a/examples/jmespath_functions/src/powertools_custom_jmespath_function.py b/examples/jmespath_functions/src/powertools_custom_jmespath_function.py
@@ -1,7 +1,7 @@
 import base64
 import binascii
+import zlib
 
-import snappy
 from jmespath.exceptions import JMESPathTypeError
 from jmespath.functions import signature
 
@@ -15,9 +15,9 @@ class CustomFunctions(PowertoolsFunctions):
     # only decode if value is a string
     # see supported data types: https://jmespath.org/specification.html#built-in-functions
     @signature({"types": ["string"]})
-    def _func_decode_snappy_compression(self, payload: str):
+    def _func_decode_zlib_compression(self, payload: str):
         decoded: bytes = base64.b64decode(payload)
-        return snappy.uncompress(decoded)
+        return zlib.decompress(decoded)
 
 
 custom_jmespath_options = {"custom_functions": CustomFunctions()}
@@ -30,15 +30,15 @@ def lambda_handler(event, context) -> dict:
             extract_data_from_envelope(
                 data=event,
                 # NOTE: Use the prefix `_func_` before the name of the function
-                envelope="Records[*].decode_snappy_compression(log)",
+                envelope="Records[*].decode_zlib_compression(log)",
                 jmespath_options=custom_jmespath_options,
             ),
         )
         return {"logs": logs, "message": "Extracted messages", "success": True}
     except JMESPathTypeError:
         return return_error_message("The envelope function must match a valid path.")
-    except snappy.UncompressError:
-        return return_error_message("Log must be a valid snappy compressed binary")
+    except zlib.error:
+        return return_error_message("Log must be a valid zlib compressed message")
     except binascii.Error:
         return return_error_message("Log must be a valid base64 encoded string")
 
diff --git a/layer/poetry.lock b/layer/poetry.lock
diff --git a/layer/pyproject.toml b/layer/pyproject.toml
diff --git a/layer/sar/template.txt b/layer/sar/template.txt
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
diff --git a/tests/e2e/utils/infrastructure.py b/tests/e2e/utils/infrastructure.py
diff --git a/tests/functional/validator/test_validator.py b/tests/functional/validator/test_validator.py
diff --git a/tests/unit/test_lru_cache.py b/tests/unit/test_lru_cache.py

Original file line number	Diff line number	Diff line change
`@@ -1,14 +1,9 @@`
`1`	`1`	`{`
`2`	`2`	`"Records": [`
`3`	`3`	`{`
`4`		`- "user": "integration-kafka",`
	`4`	`+ "application": "notification",`
`5`	`5`	`"datetime": "2022-01-01T00:00:00.000Z",`
`6`		`- "log": "/QGIMjAyMi8wNi8xNiAxNjoyNTowMCBbY3JpdF0gMzA1MTg5MCMNCPBEOiAqMSBjb25uZWN0KCkg\ndG8gMTI3LjAuMC4xOjUwMDAgZmFpbGVkICgxMzogUGVybWlzc2lvbiBkZW5pZWQpIHdoaWxlEUEI\naW5nAUJAdXBzdHJlYW0sIGNsaWVudDoZVKgsIHNlcnZlcjogXywgcmVxdWVzdDogIk9QVElPTlMg\nLyBIVFRQLzEuMSIsFUckOiAiaHR0cDovLzabABQvIiwgaG8FQDAxMjcuMC4wLjE6ODEi\n"`
`7`		`- },`
`8`		`- {`
`9`		`- "user": "integration-kafka",`
`10`		`- "datetime": "2022-01-01T00:00:01.000Z",`
`11`		`- "log": "tQHwnDEyNy4wLjAuMSAtIC0gWzE2L0p1bi8yMDIyOjE2OjMwOjE5ICswMTAwXSAiT1BUSU9OUyAv\nIEhUVFAvMS4xIiAyMDQgMCAiLSIgIk1vemlsbGEvNS4wIChYMTE7IExpbnV4IHg4Nl82NCkgQXBw\nbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwMi4BmUwwIFNhZmFy\naS81MzcuMzYiICItIg==\n"`
	`6`	`+ "log": "eJxtUNFOwkAQ/JXN+dJq6e22tMD5ZHwQRYSEJpqQhtRy2AvlWq+tEr/eg6DExOzDJjM7M5tZsgCDgGPMKQaKRRAJRFjmRrUphBjRcIQXpy3gkiCvtJZ567jQVkDBwEc7JCK0sk2mSrkGh0IBc2l2qmlUpWEttZJrFz4LS/8YKP12cOjqpjUy23mQl0rqVpw9PWik+ZBGwMoDI9872ViazebJ/expARzGSTLn5BPzfm0sX7RtLTj/+xq3N0V11J+JITELnwHo2VlSzB86zQ+1CFtIiIJGcIWEmP4bDgH2AYH1GLBp9aXKMuORj+C8EF3Do9LdHvbDeBX3Xbip61I+y9eJankUDvwwBmcyTqaPHpRqK+FO5tvKhdvCVDvJCYPjYwiLbJMZdZKwQxZL02+NI3Vs"`
`12`	`7`	`}`
`13`	`8`	`]`
`14`	`9`	`}`