Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option "rows per range" not working in PostgreSQL to PostgreSQL migration? #1550

Open
6 tasks done
JosefMachytkaNetApp opened this issue Dec 18, 2023 · 1 comment · May be fixed by #1614
Open
6 tasks done

Option "rows per range" not working in PostgreSQL to PostgreSQL migration? #1550

JosefMachytkaNetApp opened this issue Dec 18, 2023 · 1 comment · May be fixed by #1614

Comments

@JosefMachytkaNetApp
Copy link

JosefMachytkaNetApp commented Dec 18, 2023 •
edited
Loading

Hello!
I need your help. Documentation for PostgreSQL to PostgreSQL migration claims that I can use "rows per range" option:

* rows per range
How many rows are fetched per SELECT query when using multiple readers per thread, see above for details.

But pgloader of multiple versions (see below) refuses to parse config file if I add these options - I always get ESRAP-PARSE-ERROR
for both multiple readers per thread and rows per range commands. Without them config file works.

I have huge migration pg2pg where I would really need to use these options, but pgloader refuses to parse them. Here is WITH part of my config file:

    WITH
        include drop,
        create tables,
        workers = 8,
        concurrency = 4, 
        multiple readers per thread,
        rows per range 1000000

Am I doing something wrongly? Is there some workaround?
Thank you very much for your help!

  • pgloader --version
* from debian package - pgloader (3.6.10-1.pgdg20.04+1) - shows me version:
pgloader version "3.6.7~devel"
compiled with SBCL 2.1.1.debian

&
* manual build from latest release sources tar.gz on testing env:
pgloader version "3.6.9"
compiled with SBCL 2.0.1.debian

&
* manual build from latest master - which surprisingly shows me version:
pgloader version "3.6.7~devel"
compiled with SBCL 2.0.1.debian
(I tried to download and build repeatedly - master really shows me this version)

  • did you test a fresh compile from the source tree?
    Yes, I did

  • did you search for other similar issues?
    Yes I did but did not find anything relevant

  • how can I reproduce the bug?
    Just try those options in pg2pg migration

  • pgloader output you obtain

KABOOM!
ESRAP-PARSE-ERROR: At

          workers = 8,
          concurrency = 4, 
  
                         ^ (Line 9, Column 23, Position 187)

In context PGSQL-OPTIONS:

While parsing PGSQL-OPTIONS. Expected:

     the character Tab
  or the character Newline
  or the character Return
  or the character Space
  or the string "--"
  or the string "/*"
  or the string "batch"
  or the string "concurrency"
  or the string "create"
  or the string "data"
  or the string "disable"
  or the string "downcase"
  or the string "drop"
  or the string "foreign"
  or the string "include"
  or the string "max"
  or the string "no"
  or the string "on"
  or the string "prefetch"
  or the string "preserve"
  or the string "quote"
  or the string "reset"
  or the string "schema"
  or the string "snake_case"
  or the string "truncate"
  or the string "uniquify"
  or the string "workers"
An unhandled error condition has been signalled: At
  • data that is being loaded, if relevant
    not relevant, pgloader refuses to parse config file

Thank you very much for your help!
@mbanck-ntap
Copy link

I had a look at the parser and it seems that this was an error or copy-pasto in the documentation updates in 56d24de: Only the Mysql driver seems to support multiple readers per thread and rows per range unfortunately. I will see whether I can open a PR to fix this.

mbanck-ntap added a commit to credativ/pgloader that referenced this issue Oct 1, 2024
@mbanck-ntap
Remove rows per range option from Postgres-Postgres docs
7f249e9 
This was never implemented for pgsql<->pgsql migrations.

Fixes: dimitri#1550
@mbanck-ntap mbanck-ntap linked a pull request Oct 1, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove rows per range option from Postgres-Postgres docs credativ/pgloader
2 participants
@JosefMachytkaNetApp @mbanck-ntap