Skip to content

Commit

Permalink
tests: refine how compat snapshot is generated (#10342)
Browse files Browse the repository at this point in the history
## Problem

I noticed in #9537 that tests
which work with compat snapshots were writing several hundred MB of
data, which isn't really necessary.

Also, the snapshots are large but don't have the proper variety of
storage format features, e.g. they could just have L0 deltas.

## Summary of changes

- Use smaller scale factor and runtime to generate less data
- Configure a small layer size and use force image layer generation so
that our output contains L1 deltas and image layers, and has a decent
number of entries in the layer map
  • Loading branch information
jcsp authored Jan 10, 2025
1 parent 2b8ea1e commit 0d4fce2
Showing 1 changed file with 13 additions and 4 deletions.
17 changes: 13 additions & 4 deletions test_runner/regress/test_compatibility.py
Original file line number Diff line number Diff line change
Expand Up @@ -141,11 +141,18 @@ def test_create_snapshot(
neon_env_builder.num_safekeepers = 3
neon_env_builder.enable_pageserver_remote_storage(RemoteStorageKind.LOCAL_FS)

env = neon_env_builder.init_start()
env = neon_env_builder.init_start(
initial_tenant_conf={
# Miniature layers to enable generating non-trivial layer map without writing lots of data
"checkpoint_distance": f"{128 * 1024}",
"compaction_threshold": "1",
"compaction_target_size": f"{128 * 1024}",
}
)
endpoint = env.endpoints.create_start("main")

pg_bin.run_capture(["pgbench", "--initialize", "--scale=10", endpoint.connstr()])
pg_bin.run_capture(["pgbench", "--time=60", "--progress=2", endpoint.connstr()])
pg_bin.run_capture(["pgbench", "--initialize", "--scale=1", endpoint.connstr()])
pg_bin.run_capture(["pgbench", "--time=30", "--progress=2", endpoint.connstr()])
pg_bin.run_capture(
["pg_dumpall", f"--dbname={endpoint.connstr()}", f"--file={test_output_dir / 'dump.sql'}"]
)
Expand All @@ -157,7 +164,9 @@ def test_create_snapshot(
pageserver_http = env.pageserver.http_client()

flush_ep_to_pageserver(env, endpoint, tenant_id, timeline_id)
pageserver_http.timeline_checkpoint(tenant_id, timeline_id, wait_until_uploaded=True)
pageserver_http.timeline_checkpoint(
tenant_id, timeline_id, wait_until_uploaded=True, force_image_layer_creation=True
)

env.endpoints.stop_all()
for sk in env.safekeepers:
Expand Down

1 comment on commit 0d4fce2

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

7443 tests run: 7067 passed, 1 failed, 375 skipped (full report)


Failures on Postgres 16

  • test_storage_controller_many_tenants[github-actions-selfhosted]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_storage_controller_many_tenants[release-pg16-github-actions-selfhosted]"
Flaky tests (2)

Postgres 17

Postgres 16

Code coverage* (full report)

  • functions: 32.7% (8043 of 24618 functions)
  • lines: 47.8% (66830 of 139908 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
0d4fce2 at 2025-01-10T16:32:02.237Z :recycle:

Please sign in to comment.