bug: ValueError: Out of range float values are not JSON compliant

[dgawlowsky]

Singer SDK Version

0.34.1

Is this a regression?

• Yes

Python Version

3.9

Bug scope

Taps (catalog, state, etc.)

Operating System

MacOS

Description

I get ValueError: Out of range float values are not JSON compliant when trying to sync Amazon Document DB collection.

The probably reason is that simple_json.dumps() has allow_nan=False default, while standard dumps() function from standard json library has allow_nan=True.

As there is no way to pass configuration from outside could we add this functionality? I can introduce relevant fix.

Code

No response

[edgarrmondragon]

Hi @dgawlowsky, thanks for logging!

I think having allow_nan by default would violate the interoperability principle in the Singer ecosystem since it's specific to Python's JSON implementation: while most taps and targets are written in Python, they could be programmed in any language.

There's also the question of how the jsonchema library that we use for validation on the target side would treat the NaN values, although it's now possible to disable validation on the user end (631d5df), if they're fine with moving NaNs around and skipping schema validation.

Do start a PR and we can discuss the details there!

Related: #2211

[dgawlowsky]

Hi @edgarrmondragon - thanks for prompt response again. Opened PR with this one line change. Let's discuss there.

[menzenski]

Also running into this issue, and also in an AWS DocumentDB context.

[edgarrmondragon]

@menzenski the situation should've improved somewhat with v0.45.6, which shipped

• fix: Conform Nan, Infinity and -Infinity to null values #2961

Can you confirm if that's enough for your use case or do you need to actually communicate the NaN values to the target somehow?

[menzenski]

@edgarrmondragon I updated my tap-mongodb fork from 0.41.0 to 0.45.11 today. The changes you mention had no effect for me, but looking at the code I think that's expected - the issues I am experiencing are in the values of a field that I declare as just dict[str, Any] in the schema. I think the changes in 0.45.6 only apply automatically to fields that are typed as float within the tap schema. (So, I think I need to basically make that same change within my tap's code. I'm working on that today).

[edgarrmondragon]

Ok, I'm now convinced that we should just force those values to be encoded as null in the Singer message across the board.

Does that make sense for your use case @menzenski?

[menzenski]

Does that make sense for your use case @menzenski?

Yes - for my use case it would be awesome to have it handled globally like that

[menzenski]

@edgarrmondragon thanks for making that change so fast. I've tested meltano/sdk commit b3a7cb3 in my tap and it solves this for me. very very appreciated!