Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic fast_import python test #10271

Open
wants to merge 8 commits into
base: 22100-change-fastimport-db-name
Choose a base branch
from

Conversation

NanoBjorn
Copy link
Contributor

@NanoBjorn NanoBjorn commented Jan 3, 2025

We did not have any tests on fast_import binary yet.

In this PR I have introduced:

  • FastImport class and tools for testing in python
  • basic test that runs fast import against vanilla postgres and checks that data is there

Should be merged after #10251

@NanoBjorn NanoBjorn requested review from jcsp and problame January 3, 2025 14:43
@NanoBjorn NanoBjorn requested a review from a team as a code owner January 3, 2025 14:43
@NanoBjorn NanoBjorn requested review from tristan957 and removed request for a team January 3, 2025 14:44
from fixtures.neon_fixtures import VanillaPostgres, PgProtocol, PgBin
from fixtures.port_distributor import PortDistributor


Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps makes sense to make this an extension to test_pgdata_import_smoke, which has this comment currently:

    # We have a Postgres data directory now.
    # Make a localfs remote storage that looks like how after `fast_import` ran.
    # TODO: actually exercise fast_import here

Copy link
Contributor Author

@NanoBjorn NanoBjorn Jan 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just thought that it might not worth doing it all in the same test due to increased complexity -> more complicated debugging if something is failing. Right now it is using vanilla postgres pgdata as we do in fast import, so it should be enough for testing pageserver part.

Moved mine simple test to the same file, but will keep test_pgdata_import_smoke the same. Also added a todo to test full import flow separately, which will complement test_pgdata_import_smoke and test_fast_import_binary, but not complicate any of those.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT?

@NanoBjorn NanoBjorn requested a review from jcsp January 3, 2025 15:12
Copy link

github-actions bot commented Jan 3, 2025

6460 tests run: 6170 passed, 0 failed, 290 skipped (full report)


Flaky tests (2)

Postgres 17

  • test_physical_replication_config_mismatch_too_many_known_xids: debug-x86-64

Postgres 16

  • test_physical_replication_config_mismatch_max_locks_per_transaction: release-arm64

Test coverage report is not available

The comment gets automatically updated with the latest test results
e856fae at 2025-01-06T20:50:28.045Z :recycle:

@NanoBjorn NanoBjorn force-pushed the 22037-basic-fast-import-e2e branch 3 times, most recently from 1b08f3c to 122e324 Compare January 6, 2025 18:21
@NanoBjorn NanoBjorn force-pushed the 22037-basic-fast-import-e2e branch from 122e324 to e856fae Compare January 6, 2025 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants