Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve benchmark usability #66

Open
fyrchik opened this issue Mar 2, 2021 · 1 comment
Open

Improve benchmark usability #66

fyrchik opened this issue Mar 2, 2021 · 1 comment
Labels
feature Completely new functionality I2 Regular impact S4 Routine U4 Nothing urgent

Comments

@fyrchik
Copy link
Contributor

fyrchik commented Mar 2, 2021

Often we need to compare different patches of neo-go or even add some C#/mixed on top of it.
There are some problems:

  1. Running benchmark on local machine can interfere with other programs running which affects results.
  2. Again, because of system load multiple benchmark runs give more meaningful results. Mean/variation of block times/TPS in a single run is a bit different of same metrics for results of multiple runs. Former is related to how node processes transactions, while latter can help to make more valid conclusions. On my machine +-100TPS (with a mean of 1600) across multiple runs happens constantly, and 10% variation is something we should take into account.

Here is how we can make benchmark more flexible:

  1. Allow to provide revisions to compare. Something like REVISIONS=master,patch1,patch2 make start.GoFourNodes10wrk. The interface is discussable. Note, that C# and mixed setups should also be allowed.
  2. Emit plots for results from (1). We already have scripts for this, it can be extended.

I think doing all of this in a single command can also make results more reproducible and easier to share.

@roman-khimov
Copy link
Member

Batching and averaging out multiple runs can be useful and plots can just take the best result for a particular combination.

Refs. #4, though.

@roman-khimov roman-khimov added U4 Nothing urgent S4 Routine I2 Regular impact feature Completely new functionality labels Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Completely new functionality I2 Regular impact S4 Routine U4 Nothing urgent
Projects
None yet
Development

No branches or pull requests

2 participants