Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres -> Clickhouse CDC fails (when mapping Postgres Numeric to Clickhouse Decimal) #2071

Open
moose-code opened this issue Sep 13, 2024 · 4 comments
Assignees

Comments

@moose-code
Copy link

Hi there!

When you have a very large number stored as a postgres numeric type, the cdc will automatically try to put this as a Decimal type in clickhouse, which will fail. See disscussion:
Screenshot 2024-09-13 at 14 27 47

There has been good progress with #2049 etc which is massively appreciated, but from what I understand the actual data type conversion code still needs to be implemented so that syncing is successful. Mapping it instead from the Numeric in Postgres to Int256 type in clickhouse would work in our case. I'm sure it would also be very valuable for other teams to have control over types in various cases.

This issue is blocking us from using PeerDB as our CDC keeps failing with this essential data that we would need.

Would hugely appreciate any thoughts on a possible timeline or even hacky hard coded case we could use just for our use case. Knowing you thoughts on this issue and any timeline would be extremly appreciated!

Thanks so much in advance for your time in reviewing this.

@iamKunalGupta
Copy link
Member

@Amogh-Bharadwaj is this fixed or was the recent datatype change only related to Decimals?

@Amogh-Bharadwaj
Copy link
Contributor

Yup the recent numeric fix should help with this as well

1 similar comment
@Amogh-Bharadwaj
Copy link
Contributor

Yup the recent numeric fix should help with this as well

@iamKunalGupta
Copy link
Member

@moose-code can you try on your end once?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants