Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Full table replication failing due to query timeout #234

Open
XiaozhouWang85 opened this issue Apr 20, 2024 · 0 comments
Open

Full table replication failing due to query timeout #234

XiaozhouWang85 opened this issue Apr 20, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@XiaozhouWang85
Copy link

Describe the bug
Full table replication fails on larger tables. Process fails with the following error message:
level=CRITICAL message=canceling statement due to statement timeout cmd_type=elb consumer=False name=tap-postgres producer=True stdio=stderr string_id=tap-postgres

After investigating, the issues was traced to the fact that the sql statement submitted was something like:
SELECT ..... FROM ..... ORDER BY xmin::text::bigint ASC

The ORDER BY caused the query run time to exceed 30 seconds. After running for 30 seconds, the process fails due to timeout. Removing the ORDER BY results in an near instantaneous return of data.

I get that the xmin allows for restarts but this seems only relevant for larger tables (small tables download in seconds for me) and larger tables will cause the entire process to fail.

To Reproduce
Steps to reproduce the behavior:

  1. Run full table replication against a large table
  2. meltano run tap-postgres target-s3-jsonl
  3. See error

Expected behavior
The extractor should be able to handle larger tables without erroring out. Even if the default behaviour causes errors to be thrown, this issue to be possible to bypass using configurations. I needed to fork the extractor and remove the ORDER BY in order to fix the issue.

Your environment

  • Version of tap: 2.1.0
  • meltano:v3.3.0
@XiaozhouWang85 XiaozhouWang85 added the bug Something isn't working label Apr 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant