Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data transforms with hash_md5 fail if source column is a shorter data type #517

Open
MiConnell opened this issue Mar 1, 2025 · 1 comment

Comments

@MiConnell
Copy link
Contributor

Issue Description

  • Description of the issue:
    Sling fails with the below error when using a transform with hash_md5 on an id column that is 11 characters long.

Note

this works in version 1.3.1, but not later versions as far as I can tell

could not COPY into table "edw"."dim_my_table_tmp"
pq: value too long for type character varying(11)

I tried adding both of the below, both independently and together, but couldn't fix the error.

target_options:
    adjust_column_type: true
columns:
  id: string(32)
  • Sling version (sling --version):
    v1.4.1

  • Operating System (linux, mac, windows):
    mac

  • Replication Configuration:

source: DEMO_SOURCE
target: DEMO_DESTINATION
defaults:
  object: "edw.{stream_table}"
  mode: full-refresh
  source_options:
    empty_as_null: false
  transforms: [replace_non_printable]
  target_options:
    adjust_column_type: true
streams:
  edw.dim_student_major:
    sql: |
      select a.* from edw.dim_my_table a inner join edw.dim_date dt on a.date_key = dt.date_key where dt.year_seq <= 0 and dt.year_seq > -5
    object: edw.dim_my_table
    transforms:
      id: [hash_md5]
    columns:
      id: string(32)
  • Log Output (please run command with -d):
Paste log here.
@flarco
Copy link
Collaborator

flarco commented Mar 8, 2025

Hey @MiConnell, that's strange. Does sling create the table with character varying(32)? can you paste the temp table DDL?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants