Skip to content

Improve benchmark usability #66

Open
@fyrchik

Description

@fyrchik

Often we need to compare different patches of neo-go or even add some C#/mixed on top of it.
There are some problems:

  1. Running benchmark on local machine can interfere with other programs running which affects results.
  2. Again, because of system load multiple benchmark runs give more meaningful results. Mean/variation of block times/TPS in a single run is a bit different of same metrics for results of multiple runs. Former is related to how node processes transactions, while latter can help to make more valid conclusions. On my machine +-100TPS (with a mean of 1600) across multiple runs happens constantly, and 10% variation is something we should take into account.

Here is how we can make benchmark more flexible:

  1. Allow to provide revisions to compare. Something like REVISIONS=master,patch1,patch2 make start.GoFourNodes10wrk. The interface is discussable. Note, that C# and mixed setups should also be allowed.
  2. Emit plots for results from (1). We already have scripts for this, it can be extended.

I think doing all of this in a single command can also make results more reproducible and easier to share.

Metadata

Metadata

Assignees

No one assigned

    Labels

    I2Regular impactS4RoutineU4Nothing urgentfeatureCompletely new functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions