Skip to content

Commit

Permalink
Fix and simplify gentest randomisation toggle
Browse files Browse the repository at this point in the history
The comment correctly described the behaviour we wanted, but the code
logic was backwards. This is easy enough to do when you've got an env
var called "RANDOMIZE" toggling a config value called "derandomize" so
we switch the name of the env var to match.

This makes it clear that there's exactly one circumstance where we don't
want randomisation and that's in the CI tests.

Related Slack thread:
https://bennettoxford.slack.com/archives/C069YDR4NCA/p1739440049238879
  • Loading branch information
evansd committed Feb 13, 2025
1 parent d43b22d commit 3aff406
Show file tree
Hide file tree
Showing 3 changed files with 2 additions and 10 deletions.
1 change: 0 additions & 1 deletion .github/workflows/generative-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ jobs:
export \
GENTEST_EXAMPLES=150 \
GENTEST_RANDOMIZE=t \
GENTEST_MAX_DEPTH=25
# We get a maximum of 6 hours runtime with Github Actions so breaking
Expand Down
2 changes: 1 addition & 1 deletion justfile
Original file line number Diff line number Diff line change
Expand Up @@ -219,7 +219,6 @@ test-unit *ARGS: devenv
# Run generative tests using more than the small deterministic set of examples used in CI
test-generative *ARGS: devenv
GENTEST_EXAMPLES=${GENTEST_EXAMPLES:-200} \
GENTEST_RANDOMIZE=${GENTEST_RANDOMIZE:-t} \
$BIN/python -m pytest tests/generative "$@"


Expand All @@ -228,6 +227,7 @@ test-generative *ARGS: devenv
#!/usr/bin/env bash
set -euo pipefail
GENTEST_DERANDOMIZE=t \
GENTEST_EXAMPLES=${GENTEST_EXAMPLES:-100} \
GENTEST_CHECK_IGNORED_ERRORS=t \
$BIN/python -m pytest \
Expand Down
9 changes: 1 addition & 8 deletions tests/generative/test_query_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,14 +84,7 @@
settings = dict(
max_examples=(int(os.environ.get("GENTEST_EXAMPLES", 10))),
deadline=None,
# On CI we want to run derandomized unless we're explicitly in the
# long-running bug finding tests. This prevents flaky CI.
# In development we never want to run with derandomize because it's
# less effective at finding bugs and, more importantly, has very slow
# replay due to turning off the test database
derandomize=bool(os.environ.get("GENTEST_RANDOMIZE"))
if os.environ.get("CI")
else False,
derandomize=bool(os.environ.get("GENTEST_DERANDOMIZE")),
# The explain phase is comparatively expensive here given how
# costly data generation is for our tests here, so we turn it
# off by default.
Expand Down

0 comments on commit 3aff406

Please sign in to comment.