benchmarking.md

Benchmarking

Benchmarking new code against old code is extremely important whenever making large changes to how something works. If you are attempting to make something faster, and you end up slowing it down, you'll never know if you don't benchmark! We have benchmarks in the toolsrc/src/vcpkg-test directory, just like the tests -- they're treated as a special kind of test.

Running Benchmarks

Unlike normal tests, benchmarks are hidden behind a special define -- CATCH_CONFIG_ENABLE_BENCHMARKING -- so that you never try to run benchmarks unless you specifically want to. This is because benchmarks actually take quite a long time! However, if you want to run benchmarks (and I recommend running only specific benchmarks at a time), you can do so by passing the VCPKG_ENABLE_BENCHMARKING option at cmake configure time.

$ cmake -B toolsrc/out -S toolsrc -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DVCPKG_BUILD_BENCHMARKING=On

-- The C compiler identification is MSVC 19.22.27905.0
-- The CXX compiler identification is MSVC 19.22.27905.0
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/Tools/MSVC/14.22.27905/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/t-nimaz/src/vcpkg/toolsrc/out

$ cmake --build toolsrc/out

[0/2] Re-checking globbed directories...
[80/80] Linking CXX executable vcpkg-test.exe

You can then run benchmarks easily with the following command (which run the files benchmarks):

$ ./toolsrc/out/vcpkg-test [!benchmark][file]

You can switch out [file] for a different set -- [hash], for example.

Writing Benchmarks

First, before anything else, I recommend reading the benchmarking documentation at Catch2's repository.

Now, after that, let's say that you wanted to benchmark, say, our ASCII case-insensitive string compare against your new implementation. We place benchmarks for code in the same file as their tests, so open vcpkg-test/strings.cpp, and add the following at the bottom:

#if defined(CATCH_CONFIG_ENABLE_BENCHMARKING)
TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]")
{
    BENCHMARK("qwertyuiop") {
        return vcpkg::Strings::case_insensitive_ascii_equals("qwertyuiop", "QWERTYUIOP");
    };
}
#endif

Remember the ; at the end of the benchmark -- it's not required for TEST_CASEs, but is for BENCHMARKs.

Now, let's rebuild and run:

$ cmake --build toolsrc/out
[0/2] Re-checking globbed directories...
[2/2] Linking CXX executable vcpkg-test.exe
$ ./toolsrc/out/vcpkg-test [strings][!benchmark]
Filters: [strings][!benchmark]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vcpkg-test.exe is a Catch v2.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
case insensitive ascii equals: benchmark
-------------------------------------------------------------------------------
C:\Users\t-nimaz\src\vcpkg\toolsrc\src\vcpkg-test\strings.cpp(36)
...............................................................................

benchmark name                                  samples       iterations    estimated
                                                mean          low mean      high mean
                                                std dev       low std dev   high std dev
-------------------------------------------------------------------------------
qwertyuiop                                              100         2088    3.9672 ms
                                                      25 ns        24 ns        26 ns
                                                       6 ns         5 ns         8 ns


===============================================================================
test cases: 1 | 1 passed
assertions: - none -

You've now written your first benchmark!

But wait. This seems kind of silly. Benchmarking the comparison of literal strings is great and all, but could we make it a little more realistic?

This is where BENCHMARK_ADVANCED comes in. BENCHMARK_ADVANCED allows one to write a benchmark that has a little setup to it without screwing up the numbers. Let's try it now:

TEST_CASE ("case insensitive ascii equals: benchmark", "[strings][!benchmark]")
{
    BENCHMARK_ADVANCED("equal strings")(Catch::Benchmark::Chronometer meter)
    {
        std::vector<std::string> strings;
        strings.resize(meter.runs());
        std::mt19937_64 urbg;
        std::uniform_int_distribution<std::uint64_t> data_generator;

        std::generate(strings.begin(), strings.end(), [&] {
            std::string result;
            for (std::size_t i = 0; i < 1000; ++i)
            {
                result += vcpkg::Strings::b32_encode(data_generator(urbg));
            }

            return result;
        });

        meter.measure(
            [&](int run) { return vcpkg::Strings::case_insensitive_ascii_equals(strings[run], strings[run]); });
    };
}

Then, run it again!

$ cmake --build toolsrc/out
[0/2] Re-checking globbed directories...
[2/2] Linking CXX executable vcpkg-test.exe
$ toolsrc/out/vcpkg-test [strings][!benchmark]
Filters: [strings][!benchmark]

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vcpkg-test.exe is a Catch v2.9.1 host application.
Run with -? for options

-------------------------------------------------------------------------------
case insensitive ascii equals: benchmark
-------------------------------------------------------------------------------
C:\Users\t-nimaz\src\vcpkg\toolsrc\src\vcpkg-test\strings.cpp(36)
...............................................................................

benchmark name                                  samples       iterations    estimated
                                                mean          low mean      high mean
                                                std dev       low std dev   high std dev
-------------------------------------------------------------------------------
equal strings                                           100            2    5.4806 ms
                                                  22.098 us    21.569 us    23.295 us
                                                   3.842 us     2.115 us      7.41 us


===============================================================================
test cases: 1 | 1 passed
assertions: - none -

And now you have a working benchmark to test the speed of the existing code, and of new code!

If you're writing a lot of benchmarks that follow the same sort of pattern, with some differences in constants, look into vcpkg-test/files.cpp's benchmarks -- there are a lot of things one can do to make writing new benchmarks really easy.

If you wish to add a benchmark for a piece of code that has not yet been tested, please read the testing documentation, and please write some unit tests. The speed of your code isn't very important if it doesn't work at all!