Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inability to Cast Avro Schema to Glue Table Schema with double Type #41

Open
shahinism opened this issue Jun 30, 2023 · 0 comments
Open

Comments

@shahinism
Copy link

shahinism commented Jun 30, 2023

I'm currently experiencing an issue where our connector fails to correctly map an Avro schema to a Glue table schema when using the decimal type. Specifically, for the decimal type, it incorrectly sets the schema of the field to decimal, and the data are inserted as they are in the stored buffer of the original Avro content.

Here are the details of the configuration:

Avro schema:

{
  "fields": [
    {
      "name": "after",
      "type": {
        "name": "after",
        "type": "record",
        "fields": [
          {
            "default": "",
            "name": "OEFRCONUMB",
            "type": {
              "logicalType": "char",
              "maxLength": 2,
              "type": "string"
            }
          },
          {
            "default": "\u0000",
            "name": "OEFRBLNUMB",
            "type": {
              "logicalType": "decimal",
              "precision": 7,
              "scale": 0,
              "type": "bytes"
            }
          }
        ]
      }
    }
  ],
  "name": "header",
  "type": "record"
}

Message body:

data = {
    "after": {
        "OEFRCONUMB": "14",
        "OEFRBLNUMB": Decimal("0"),
    }
}

Here is how the data appears in the Glue table (insert screenshot image here). It seems like the mapping is not functioning as intended.

image

If you could provide any insights or workarounds for this issue, I would greatly appreciate it. Thank you for your assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant