You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The documentation for asv compare suggests that it is best used for comparing between versions / revisions, e.g. asv compare v0.1 v0.2. However, benchmarks with different versions are skipped (unless explicitly set to None). In practice, then, naively, one might assume attempts to compare across versions would run into:
Benchmarks that are not comparable:
| Change | Before [22b920c6] | After [fcf8c079] | Ratio | Benchmark (Parameter) |
|----------|----------------------|---------------------|---------|----------------------------------|
| x | 1.00s | 3.00s | 3 | time_with_version_mismatch_other |
The reason this doesn't happen is that the version here is the one generated by hashing the benchmark source code (which is actually mentioned in the documentation) and it isn't the project version.
There is also the version identifier in the output json corresponding to the API versioning of the json itself.
Perhaps some notes on this in the documentation would be useful. Or alternatively, a clearer set of variable names (but this is backwards incompatible):
version --> api_version for the json format
version --> bench_version for the benchmark version
The project "version" is anyway stored in commit_hash.
The text was updated successfully, but these errors were encountered:
Also worth noting that as currently implemented, the version is sensitive to the formatting changes made to the suite, and can easily be replaced by an AST based hash mechanism (and probably even better things).
Note that this would need to be changed in asv_runner, not here.
The documentation for
asv compare
suggests that it is best used for comparing between versions / revisions, e.g.asv compare v0.1 v0.2
. However, benchmarks with different versions are skipped (unless explicitly set toNone
). In practice, then, naively, one might assume attempts to compare across versions would run into:The reason this doesn't happen is that the version here is the one generated by hashing the benchmark source code (which is actually mentioned in the documentation) and it isn't the project version.
There is also the
version
identifier in the outputjson
corresponding to the API versioning of thejson
itself.Perhaps some notes on this in the documentation would be useful. Or alternatively, a clearer set of variable names (but this is backwards incompatible):
version
-->api_version
for thejson
formatversion
-->bench_version
for the benchmark versionThe project "version" is anyway stored in
commit_hash
.The text was updated successfully, but these errors were encountered: