Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOC: Slightly confusing multiple usage of version in ASV output #1456

Open
HaoZeke opened this issue Jan 6, 2025 · 1 comment
Open

DOC: Slightly confusing multiple usage of version in ASV output #1456

HaoZeke opened this issue Jan 6, 2025 · 1 comment

Comments

@HaoZeke
Copy link
Member

HaoZeke commented Jan 6, 2025

The documentation for asv compare suggests that it is best used for comparing between versions / revisions, e.g. asv compare v0.1 v0.2. However, benchmarks with different versions are skipped (unless explicitly set to None). In practice, then, naively, one might assume attempts to compare across versions would run into:

Benchmarks that are not comparable:

| Change   | Before [22b920c6]    | After [fcf8c079]    |   Ratio | Benchmark (Parameter)            |
|----------|----------------------|---------------------|---------|----------------------------------|
| x        | 1.00s                | 3.00s               |       3 | time_with_version_mismatch_other |

The reason this doesn't happen is that the version here is the one generated by hashing the benchmark source code (which is actually mentioned in the documentation) and it isn't the project version.

There is also the version identifier in the output json corresponding to the API versioning of the json itself.

Perhaps some notes on this in the documentation would be useful. Or alternatively, a clearer set of variable names (but this is backwards incompatible):

  • version --> api_version for the json format
  • version --> bench_version for the benchmark version

The project "version" is anyway stored in commit_hash.

@HaoZeke
Copy link
Member Author

HaoZeke commented Jan 6, 2025

Also worth noting that as currently implemented, the version is sensitive to the formatting changes made to the suite, and can easily be replaced by an AST based hash mechanism (and probably even better things).

Note that this would need to be changed in asv_runner, not here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant