Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reproducible test data #225

Open
HexDecimal opened this issue Aug 29, 2024 · 0 comments
Open

Reproducible test data #225

HexDecimal opened this issue Aug 29, 2024 · 0 comments

Comments

@HexDecimal
Copy link
Collaborator

It'd be nice to make the test data reproducible, where when the CI workflow rebuilds these files and verifies that the results are exactly what's already been committed to the repo.

I made some progress for this in #218, but I didn't commit any tests files since it was incomplete. I've been able to reproduce everything currently being built except for the .pyd files in the test wheels. I couldn't figure out what's causing their nondeterminism.

In theory this is a security risk. Anyone cloning the repo and immediately running tests will execute most of these test library files. I don't suspect any of the current files are malicious but that only way to actually verify that is to build the files myself. These non-reproducible files can not be easily verified during PR's.

Right now CI always rebuilds the test files before bundling them in sdist and wheels, but this does not account for files which are not overwritten.

CI and Makefiles should delete all non-human-readable test data as part of their clean step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant