Skip to content

Commit 02c5517

Browse files
committed
cmd/benchstat: new version of benchstat
This is a complete rewrite of benchstat. Basic usage remains the same, as does the core idea of showing statistical benchmark summaries and A/B comparisons in a table, but there are several major improvements. The statistics is now more robust. Previously, benchstat used IQR-based outlier rejection, showed the mean of the reduced sample, its range, and did a non-parametric difference-of-distribution test on reduced samples. Any form of outlier rejection must start with distributional assumptions, in this case assuming normality, which is generally not sound for benchmark data. Hence, now benchstat does not do any outlier rejection. As a result, it must use robust summary statistics as well, so benchstat now uses the median and confidence interval of the median as summary statistics. Benchstat continues to use the same Mann-Whitney U-test for the delta, but now does it on the full samples since the U-test is already non-parametric, hence increasing the power of this test. As part of these statistical improvements, benchstat now detects and warns about several common mistakes, such as having too few samples for meaningful statistical results, or having incomparable geomeans. The output format is more consistent. Previously, benchstat transformed units like "ns/op" into a metric like "time/op", which it used as a column header; and a numerator like "sec", which it used to label each measurement. This was easy enough for the standard units used by the testing framework, but was basically impossible to generalize to custom units. Now, benchstat does unit scaling, but otherwise leaves units alone. The full (scaled) unit is used as a column header and each measurement is simply a scaled value shown with an SI prefix. This also means that the text and CSV formats can be much more similar while still allowing the CSV format to be usefully machine-readable. Benchstat will also now do A/B comparisons even if there are more than two inputs. It shows a comparison to the base in the second and all subsequent columns. This approach is consistent for any number of inputs. Benchstat now supports the full Go benchmark format, including sophisticated control over exactly how it structures the results into rows, columns, and tables. This makes it easy to do meaningful comparisons across benchmark data that isn't simply structured into two input files, and gives significantly more control over how results are sorted. The default behavior is still to turn each input file into a column and each benchmark into a row. Fixes golang/go#19565 by showing all results, even if the benchmark sets don't match across columns, and warning when geomean sets are incompatible. Fixes golang/go#19634 by no longer doing outlier rejection and clearly reporting when there are not enough samples to do a meaningful difference test. Updates golang/go#23471 by providing more through command documentation. I'm not sure it quite fixes this issue, but it's much better than it was. Fixes golang/go#30368 because benchstat now supports filter expressions, which can also filter down units. Fixes golang/go#33169 because benchstat now always shows file configuration labels. Updates golang/go#43744 by integrating unit metadata to control statistical assumptions into the main tool that implements those assumptions. Fixes golang/go#48380 by introducing a way to override labels from the command line rather than always using file names. Change-Id: Ie2c5a12024e84b4918e483df2223eb1f10413a4f Reviewed-on: https://go-review.googlesource.com/c/perf/+/309969 Run-TryBot: Austin Clements <[email protected]> Reviewed-by: Michael Pratt <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
1 parent 91a0461 commit 02c5517

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+3236
-3158
lines changed

benchstat/doc.go

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// Copyright 2021 The Go Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style
3+
// license that can be found in the LICENSE file.
4+
5+
// Package benchstat is deprecated.
6+
//
7+
// This package contains the underlying implementation of an old
8+
// version of the benchstat command.
9+
//
10+
// Deprecated: The latest version of benchstat can be found at
11+
// golang.org/x/perf/cmd/benchstat. To work with benchmark data, see
12+
// golang.org/x/perf/benchproc and golang.org/x/perf/benchmath.
13+
package benchstat

cmd/benchstat/README.md

-91
This file was deleted.

cmd/benchstat/doc_test.go

+106
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
// Copyright 2021 The Go Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style
3+
// license that can be found in the LICENSE file.
4+
5+
package main
6+
7+
import (
8+
"bytes"
9+
"go/ast"
10+
"go/doc"
11+
"go/parser"
12+
"go/token"
13+
"os"
14+
"regexp"
15+
"strings"
16+
"testing"
17+
)
18+
19+
// Test that the examples in the command documentation do what they
20+
// say.
21+
func TestDoc(t *testing.T) {
22+
// Read the package documentation.
23+
fset := token.NewFileSet()
24+
f, err := parser.ParseFile(fset, "main.go", nil, parser.ParseComments)
25+
if err != nil {
26+
t.Fatal(err)
27+
}
28+
p, err := doc.NewFromFiles(fset, []*ast.File{f}, "p")
29+
if err != nil {
30+
t.Fatal(err)
31+
}
32+
tests := parseDocTests(p.Doc)
33+
if len(tests) == 0 {
34+
t.Fatal("failed to parse doc tests: found 0 tests")
35+
}
36+
37+
// Run the tests.
38+
if err := os.Chdir("testdata"); err != nil {
39+
t.Fatal(err)
40+
}
41+
defer os.Chdir("..")
42+
for _, test := range tests {
43+
var got, gotErr bytes.Buffer
44+
t.Logf("benchstat %s", strings.Join(test.args, " "))
45+
if err := benchstat(&got, &gotErr, test.args); err != nil {
46+
t.Fatalf("unexpected error: %s", err)
47+
}
48+
49+
// None of the doc tests should have error output.
50+
if gotErr.Len() != 0 {
51+
t.Errorf("unexpected stderr output:\n%s", gotErr.String())
52+
continue
53+
}
54+
55+
// Compare the output
56+
diff(t, []byte(test.want), got.Bytes())
57+
}
58+
}
59+
60+
type docTest struct {
61+
args []string
62+
want string
63+
}
64+
65+
var docTestRe = regexp.MustCompile(`(?m)^[ \t]+\$ benchstat (.*)\n((?:\t.*\n|\n)+)`)
66+
67+
func parseDocTests(doc string) []*docTest {
68+
var tests []*docTest
69+
for _, m := range docTestRe.FindAllStringSubmatch(doc, -1) {
70+
want := m[2]
71+
// Strip extra trailing newlines
72+
want = strings.TrimRight(want, "\n") + "\n"
73+
// Strip \t at the beginning of each line
74+
want = strings.Replace(want[1:], "\n\t", "\n", -1)
75+
tests = append(tests, &docTest{
76+
args: parseArgs(m[1]),
77+
want: want,
78+
})
79+
}
80+
return tests
81+
}
82+
83+
// parseArgs is a very basic parser for shell-like word lists.
84+
func parseArgs(x string) []string {
85+
// TODO: Use strings.Cut
86+
var out []string
87+
x = strings.Trim(x, " ")
88+
for len(x) > 0 {
89+
if x[0] == '"' {
90+
x = x[1:]
91+
i := strings.Index(x, "\"")
92+
if i < 0 {
93+
panic("missing \"")
94+
}
95+
out = append(out, x[:i])
96+
x = strings.TrimLeft(x[i+1:], " ")
97+
} else if i := strings.Index(x, " "); i < 0 {
98+
out = append(out, x)
99+
x = ""
100+
} else {
101+
out = append(out, x[:i])
102+
x = strings.TrimLeft(x[i+1:], " ")
103+
}
104+
}
105+
return out
106+
}

0 commit comments

Comments
 (0)