Add transformer for JSON strings

[tiurin]

Motivation

Some services response contains JSON strings within JSON strings, e.g. Cloudwatch Logs for EventBridge Pipes events.

We do have first level JSON strings parsed but we don't go recursively into parsed JSONs. This leads to the need of extracting and parsing such payloads separately in the test because otherwise only regex transformers can be applied to such string. Ideally, snapshot library should take care of this. See corresponding discussion (thanks @gregfurman for inspiration!).

Changes

Starting to parse nested JSONs in common transformation flow is a breaking change for existing recorded snapshots. This PR proposes non-breaking approach by introducing a separate transformer that can be applied on demand. This draft shows a usage example: https://github.com/localstack/localstack-ext/pull/3971

[tiurin]

TODO: add test case for empty value

[joe4dev]

Thank you for thinking about a backward-compatible way of adding such a neat feature, making lists of JSON-encoded snapshots much easier to handle 👏 .

As discussed, the main open part is handling the special edge case of empty strings. Otherwise, I added some suggestions and ideas for future iterations 📈

Idea 💡: One of the disadvantages of automatic parsing is that we are loosing structural parity information, we can:

• a) highlight this in our snapshot testing guidelines
• b) implement a technical solution at some point such as using some kind of metadata to store the raw value or structural information (e.g., key__META__: {"json": True})

Sidenotes (as discussed and partly related):

• Clarify dev installation instructions with troubleshooting for the weird editable install issue (separate quick docs PR)
• Improve Utility inheritance structure such that LocalStack Core only adds AWS-specific transformer utils (e.g., lambda_api) and utils such as sorting are implemented in the snapshot library

[joe4dev]

Great parametrization coverage 💯
I like the inline id within pytest.param rather having a disconnected id list

[joe4dev]

docs: Are there any limitations (e.g., we're losing parity information that the original response is actually JSON-encoded and should have some tests covering that)?

[joe4dev]

We could provide a simple example (e.g., the first test case) to clarify usage and foster easy adoption.

[joe4dev]

In particular, highlighting the difference to the default behavior might be most relevant, clarifying that it adds extra coverage for JSON lists starting with [ such as [{},{}]

[joe4dev]

Good log level choice given nested parsing is best effort and we might encounter false positives here 👏

[joe4dev]

Do we think it's worth logging here rather than continuing silently similar to the top-level behavior here:

localstack-snapshot/localstack_snapshot/snapshots/prototype.py

Line 279 in 302c993

pass # parsing error can be ignored

[joe4dev]

as discussed, we're missing a test and fix for empty strings here "" to avoid verbose exception logging

[joe4dev]

Idea: To facilitate an easy transition, it could be nice to have an easy way to apply this transformer globally, for example, when not providing a key.

sidenote: it ultimately comes down to separating anchors (e.g., JSON path selectors) from actual transformer functionality, but that's far out of scope :)

[joe4dev]

👍 I tested this change against the localstack-ext PR https://github.com/localstack/localstack-ext/pull/3971 with the following changes (not necessary anymore once the Utility re-use hierarchy is fixed) and can confirm it works as expected (apart from the minor empty string stack trace):

         snapshot.add_transformer(
-            snapshot.transform.sorting(
+            SortingTransformer(
                 "events",
                 lambda event: (event["message"]["executionId"], event["message"]["messageType"]),
             )
         )
-        snapshot.add_transformer(snapshot.transform.json_string("payload"))
+        # snapshot.add_transformer(snapshot.transform.json_string("payload"))
+        snapshot.add_transformer(JsonStringTransformer("payload"))

✅ There is no regression (identity check) using this transformer via snapshot.add_transformer(JsonStringTransformer("Document")) in the X-Ray test tests.aws.services.lambda_.test_lambda_xray.TestLambdaXrayIntegration.test_basic_xray_integration (X-Ray commonly uses nested JSON documents, but that's already handled by the default transformer)