testing: Add basic fuzz test harnesses

[nathaniel-brough]

No description provided.

[boring-cyborg]

Thanks for opening this pull request! It might take a while before we look at it, so don't worry if there seems to be no feedback. We'll get to it.

[CLAassistant]

All committers have signed the CLA.

[nathaniel-brough]

See #1802 for more details :)

[sweemer]

If QL_BUILD_FUZZ_TEST_SUITE is set for an unsupported compiler then CMake configuration should fail. Also, is this feature supported with clang-cl?

[nathaniel-brough]

Fair enough. I admit my cmake foo is somewhat lacking. I'll see what I can work out but if you have suggestions on how you'd like to do that then I'd appreciate your guidance 🙂

[nathaniel-brough]

I'm not familiar with clang-cl so I'd have to look into that a bit more

[sweemer]

Something like the following should be good enough:

if (QL_BUILD_FUZZ_TEST_SUITE)
    if (CMAKE_CXX_COMPILER_ID MATCHES "Clang")
        add_subdirectory(fuzz-test-suite)
    else()
        message(FATAL_ERROR "QL_BUILD_FUZZ_TEST_SUITE is currently only supported for Clang")
    endif()
endif()

[nathaniel-brough]

Sorry I didn't see this and I ended up using cmake_dependent_option instead. Let me know if that works otherwise, I'll put this in instead as it looks like it achieves roughly the same thing.

[kohnech]

I am ok with this change, I just wonder what the strategy is. Both approaches either using latest CMake version or supporting older system CMake version has pros and cons, so there is no straightforward answer. It is up to the maintainers what to choose here :) Lot's of interesting discussion about this exist in for example reddit :)

[nathaniel-brough]

Yeah, honestly the change in version was only to allow for cmake_dependent_option. While the ergonomics are a little more clunky we could revert back to the earlier version, this would just become a regular option and an if statement which is a fairly minor change. I don't feel strongly about going either way.

Worth noting however that I did make/test this change on Ubuntu 20.04, although I had forgotten that I was using the pip version of cmake, which backports a newer version to older systems.

[nathaniel-brough]

I'll open a PR soon, reverting to an earlier version of cmake.

[lballabio]

thanks!

[nathaniel-brough]

Rolled back here #1831

[sweemer]

Just curious, did you encounter any fuzzer test failures when you ran this locally?

[nathaniel-brough]

Not yet, I only ran it for around 10min though. I'd say it's reasonably robust if it can handle that. But if we go down the route of integration with google/oss-fuzz it'll run for about 4hrs every day on a distributed cluster, pulling in new code changes as they come. So even if it doesn't find bugs now, it might quickly catch new bugs if they happen.

[nathaniel-brough]

I think I've addressed most of you feedback, thanks for being so responsive :)

[coveralls]

coverage: 72.018% (+0.1%) from 71.904%
when pulling acbea49 on silvergasp:master
into 82f71c2 on lballabio:master.

[sweemer]

If more fuzzer test cases will be added in the future, then perhaps the target can be renamed to something more generic like ql_fuzz_test_suite like the existing test suite target.

[nathaniel-brough]

Cool I'll take a look, I admit I haven't extended cmake in this way before, but I'll take a swing at it.

[nathaniel-brough]

After looking into this a little I don't think that this makes sense. The boost harnesses that you are referencing here can be bundled because each unit-test has a finite runtime. So each test can be either serialized or run in parallel from a single executable. In contrast, a fuzz-test will run;

• Indefinitely.
• Or until a crashing input is found.
• Or you manually exit.
• Or if you manually set a timeout.

This poses a problem when you are bundling testing harnesses, as you can't neccesarily say that a fuzz harness is going to complete/exit or not. So if you bundle 3 harnesses, you might just get stuck on harness 1, resulting in 2 and 3 not getting fuzzed at all. So you typically want a single executable per functionality/functionalilty group.

[nathaniel-brough]

Sorry I think I used cmake dependent option wrong on the last version. I've updated it, and tested it locally using both gcc/clang so I'm hoping the latest changes should fix the CI problems.

[lballabio]

Is this a random string of length up to 100? If so, it's unlikely that it will ever be a date in ISO format. Is the purpose of this test only to throw random data at the library and check that it doesn't crash or overwrite memory it doesn't own?

[nathaniel-brough]

That's mostly correct. Although I would add that the fuzzer will likely "learn" what the iso date format is. So it'll start off passing in seemingly random data, and then it'll measure the code coverage. The inputs that acheive the highest code-coverage will then be mutated to create new inputs. So it's likely that after a period of time the fuzzer will "learn" (through genetic programming), what the iso date format is, as that will produce the highest code coverage. We can help this process along and make it a bit quicker by providing a seed-corpus or a dictionary. But I haven't included either in the PR as I didn't want to complicate things to early on.

[lballabio]

I see, thanks. Right now we're not training it, though, are we? I'm guessing we should add something in the catch clause that tells the fuzzer the result of the attempt, right?

In your experience, is this something that mostly works for simple classes like this one, or does it scale to cases with a few dozens of inputs? Real-world cases for this library would look at least like https://github.com/lballabio/QuantLib/blob/master/Examples/MulticurveBootstrapping/MulticurveBootstrapping.cpp which needs a whole lot of quotes and swap data.

[nathaniel-brough]

I see, thanks. Right now we're not training it, though, are we? I'm guessing we should add something in the catch clause that tells the fuzzer the result of the attempt, right?

Well no, you wouldn't train the fuzzer in the same way that you would train a classification machine learning model. Instead you can think of fuzzing as more of an optimisation algorithm, where the goal state of the optimizer is to provide a set of inputs that result in 100% of code being executed (rarely happens in practice). So in effect the fuzzer will train itself. The "result" of our fuzz harness is therefore is the resulting code-coverage not the resulting value of the software under test. But it is a good point that we could add some extra assertions to the catch clause, e.g. we could assert the type of exception, this is useful if some exceptions are acceptable but others are not.

It is worth noting however that while the fuzzer "learns" just by running it, it isn't stateful by default, so as soon as the fuzzer exits it'll loose the "learning progress" that it's made. We can save the state between runs by providing a corpus directory. e.g.

mkdir date_parser_corpus -p
# Share corpus/learning-state between runs.
./fuzz-test-suite/DateParserFuzzer date_parser_corpus

If we go down the route of integrating with google/oss-fuzz, all of this corpus management is handled automatically.

In your experience, is this something that mostly works for simple classes like this one, or does it scale to cases with a few dozens of inputs? Real-world cases for this library would look at least like https://github.com/lballabio/QuantLib/blob/master/Examples/MulticurveBootstrapping/MulticurveBootstrapping.cpp which needs a whole lot of quotes and swap data.

Yes in practice fuzzing is very scalable to more complex algorithms/applications. I think the largest scale fuzzer I've seen is google/syzkaller which fuzz's the entire Linux kernel. There are some things that are difficult to fuzz e.g. software with a lot of; non-determinism or global state (where it's hard to reset). But I think fuzzing the example you've linked would be relatively straightforward. While I'm not familiar with multicurve bootstrapping in general it looks like a good candidate for fuzzing with some property based testing. So as a toy example let's say your algorithm is to take an average of a vector, you might add a property assertion something like assert(average(&my_vec) >= min(&my_vec)), where my_vec is constructed from fuzzed data. In this case you're testing an invariant property that should be true for all non-empty inputs to average.

[nathaniel-brough]

I thought I should add a few things in general. There are a couple of prerequisites to building this export CC=clang and export CXX=clang++. Build everything as per usual. To run the fuzzer you just run;

$ ./fuzz-test-suite/DateParserFuzzer
INFO: Seed: 3122189193
INFO: Loaded 2 modules   (437527 inline 8-bit counters): 437484 [0x7f271c670a20, 0x7f271c6db70c), 43 [0x55d3e0c00f00, 0x55d3e0c00f2b),
INFO: Loaded 2 PC tables (437527 PCs): 437484 [0x7f271c6db710,0x7f271cd885d0), 43 [0x55d3e0c00f30,0x55d3e0c011e0),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 310 ft: 311 corp: 1/1b exec/s: 0 rss: 99Mb
        NEW_FUNC[1/10]: 0x7f271892fbc0 in boost::exception_detail::copy_boost_exception(boost::exception*, boost::exception const*) (/home/nathaniel/projects/gh.com/silvergasp/QuantLib/build/ql/libQuantLib.so.1+0x1b28bc0)
        NEW_FUNC[2/10]: 0x7f271a963930 in void boost::throw_exception<boost::gregorian::bad_year>(boost::gregorian::bad_year const&) (/home/nathaniel/projects/gh.com/silvergasp/QuantLib/build/ql/libQuantLib.so.1+0x3b5c930)
#3      NEW    cov: 358 ft: 363 corp: 2/3b lim: 4 exec/s: 0 rss: 100Mb L: 2/2 MS: 1 InsertByte-
#5      NEW    cov: 360 ft: 367 corp: 3/4b lim: 4 exec/s: 0 rss: 100Mb L: 1/2 MS: 2 ShuffleBytes-ChangeByte-
#9      NEW    cov: 360 ft: 371 corp: 4/7b lim: 4 exec/s: 0 rss: 100Mb L: 3/3 MS: 4 ChangeBit-EraseBytes-InsertByte-InsertByte-
#11     NEW    cov: 360 ft: 375 corp: 5/11b lim: 4 exec/s: 0 rss: 101Mb L: 4/4 MS: 2 CopyPart-CrossOver-
#23     NEW    cov: 368 ft: 388 corp: 6/15b lim: 4 exec/s: 0 rss: 101Mb L: 4/4 MS: 2 CrossOver-ChangeBit-
#34     REDUCE cov: 368 ft: 388 corp: 6/13b lim: 4 exec/s: 0 rss: 102Mb L: 2/4 MS: 1 EraseBytes-
#45     REDUCE cov: 370 ft: 390 corp: 7/14b lim: 4 exec/s: 0 rss: 103Mb L: 1/4 MS: 1 EraseBytes-
#64     REDUCE cov: 370 ft: 400 corp: 8/18b lim: 4 exec/s: 0 rss: 104Mb L: 4/4 MS: 4 ChangeBit-InsertByte-CrossOver-CrossOver-
#72     NEW    cov: 370 ft: 407 corp: 9/21b lim: 4 exec/s: 0 rss: 104Mb L: 3/4 MS: 3 ShuffleBytes-EraseBytes-CopyPart-
#85     NEW    cov: 376 ft: 419 corp: 10/24b lim: 4 exec/s: 0 rss: 105Mb L: 3/4 MS: 3 CopyPart-ShuffleBytes-ChangeBit-
        NEW_FUNC[1/1]: 0x7f271aa0b6b0 in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned short, char>::convert() (/home/nathaniel/projects/gh.com/silvergasp/QuantLib/build/ql/libQuantLib.so.1+0x3c046b0)

The fuzz harness will run indefinitely or;

• Until a crashing input is found.
• Or you manually exit (ctrl-C).
• Or if you manually set a timeout.

You can get more detailed help for the fuzzer by running;

./fuzz-test-suite/DateParserFuzzer -help=1

It's also best to configure cmake to run with -fsanitize=fuzzer-no-link when building the whole library as this will instrument the entire Quantlib library, so that code-coverage can be collected globally. This is particularly important for the fuzzing engine as libfuzzer will choose old inputs that have the highest code coverage, and randomly mutate them to produce new inputs. So being able to measure the code coverage, will greatly improve the effectiveness of the fuzzer. Let me know if you have any other general questions and I'll endeavor to answer them as quickly as possible.

[nathaniel-brough]

Added a basic integration as a draft PR with google/oss-fuzz google/oss-fuzz#11123. I'll need to make a few modifications to the PR if/when this PR is accepted but it is largely working with a few minor builds failures which will be solved once it's all pulled together.

[nathaniel-brough]

So it turns out there is a bug in parseFormatted, which results in an integer overflow, specifically if the string input is empty i.e. "" you'll end up with an integer overflow when parsing. I compiled the fuzzer with -fsanitize=undefined -fno-sanitize-recover=all, and it quickly found the bug.

[nathaniel-brough]

So this simplest way to reproduce this would be something like;

#include <ql/time/asx.hpp>
#include <ql/time/date.hpp>
#include <ql/time/ecb.hpp>
#include <ql/time/imm.hpp>
#include <ql/time/timeunit.hpp>
#include <ql/utilities/dataparsers.hpp>
#include <string>

int main() {
  (void)QuantLib::DateParser::parseFormatted("", "%Y-%m-%d");
  return 0;
}

Then compile with the flags as above -fsanitize=undefined -fno-sanitize-recover=all. Not a security risk, but it might give you some funky results.

[nathaniel-brough]

@lballabio friendly ping, is this something you are interested in. Or shall I close this PR?

[lballabio]

Hi—sorry, yes, let's move forward. Do you need to make more changes in this PR?

[nathaniel-brough]

Yeah I think I'm happy with where things are at. There are a few conversations that are still open, but I think I've addressed everything. This PR is mostly intended to lay the groundwork for more involved fuzzing of the actual financial algorithms in quantlib itself.

[lballabio]

Ok. Warning: as of now, the fuzzing test suite won't be in the distributed release (we'd have to add some info to the autotools files for that) but it shouldn't be a problem right now, am I correct?

[nathaniel-brough]

I'm not sure I understand your question. The fuzz tests will be built by default, if the compiler is clang. I can easily change this. I'm not familiar with autotools so I'm not really sure how that relates to the distributed releases. As a general note, I don't think it would make sense to have these fuzz-tests included in a distributed release. Does that answer your question? I'm happy to make some further tweaks if necessary, but I'll need some guidance on what needs to be done.

[lballabio]

The idea is that the .tar.gz for download at the top of a given release page are not simply a dump of the git repository at a given revision. They include some generated files which are not in the repo, and don't include some files which are in the repo.

Since we didn't add your new files to any of the Makefile.am that tell make dist how to generate the .tar.gz, they won't be included. If cmake wants to build them by default when the compiler is Clang, I think we have a problem, because the files won't be there. The easiest way out would be to default QL_BUILD_FUZZ_TEST_SUITE to OFF unconditionally, I guess?

[nathaniel-brough]

Yeah makes sense to me I'll flick that switch then

[nathaniel-brough]

Done! the fuzz tests won't be built by default only if explicitly requested on the command line.

[lballabio]

Thanks! I think at this point this can just be

option(QL_BUILD_FUZZ_TEST_SUITE "Build fuzz test suite" OFF)

Right?

[nathaniel-brough]

It can be, although this will still likely fail to build if you are not using clang, as gcc doesn't support libfuzzer out of the box. The dependent_option will only make that option visible if the compiler is clang.

[lballabio]

Oh, I see. I misinterpreted the semantics of cmake_dependent_option.

[boring-cyborg]

Congratulations on your first merged pull request!