README_fuzz_testing.md

Fuzz testing

Halide has a set of fuzz-testing harnesses that can be used to find those tricky to find, edge cases and bugs that would otherwise not be caught by a regular unit-testing suite. At the moment these fuzz-tests are housed in the test/fuzz directory. The fuzz testing suite use the common, libfuzzer interface for fuzz-tests.

Building fuzz tests

Fuzz testing requires specific instrumentation across the entire build; to do this we make use of a fuzzing-specific-toolchain/preset. e.g.

cmake -B build --preset linux-x64-fuzzer -DLLVM_ROOT=/path/to/llvminstall
cmake --build ./build -j$(nproc)

Note that the LLVM install that you use must be built with -D LLVM_ENABLE_RUNTIMES="compiler-rt" set if you want to build the fuzzer tests (failing to do so will fail at configure time); not all prebuilt LLVM installs include this, so you may need to build LLVM from source to run the fuzz tests locally.

Using the fuzz-harnesses

Fuzz-testing harnesses are a little different to a more traditional unit-test and don't have a definitive end of test. In other words, a fuzz test will run:

• for an infinite amount of time (the default),
• for a user specified maximum amount of time,
• until the fuzzer finds a bug and crashes,
• you manually kill the process e.g. (ctrl-C).

Once you have built the fuzz testing suite using the commands listed above you can list the fuzz testing harnesses using the command:

ls ./build/test/fuzz/fuzz_*

To run a fuzzer simply run the fuzz-testing harness with no arguments. e.g.

./build/test/fuzz/fuzz_simplify

By default this will run the fuzz test on a single core and discard whatever. temporary corpus is created.

To reuse a given corpus (recommended) create a new directory to store the corpus generated by your fuzz testing harness and pass that directory into your fuzzer e.g.

mkdir fuzz_simplify_corpus -p
./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus

This will save the state of the fuzzer between runs, this way any progress that your fuzzer makes improving code-coverage will remain persistent on your disk.

Up until this point the fuzzer has only been running on a single core. To speed things up a little, let's run the fuzzer in parallel across all available cores on our machine.

./build/test/fuzz/fuzz_simplify fuzz_simplify_corpus -fork=$(nproc)

Reproducing crashes

An important part of fuzz testing is reproducing the crashing input. To handle this, a libfuzzer-based fuzz harness will create a crash file whenever the fuzzer exits unexpectedly. This will look something like:

crash-<some_random_hash>

To reproduce a crash we simply rerun our fuzz harness with our crash file as the first argument.

./build/test/fuzz/fuzz_simplify crash-<some_random_hash>

So long as your fuzz harness and library are deterministic this should reproduce the original crash.

Adding new fuzz tests

A bare-bones fuzzer will look something like the following:

#include <stdint.h>
#include <stddef.h>
#include <my_library.h>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  // Randomly throw data at our function and hope it doesn't crash.
  foo(data, size);
  return 0;
}

This assumes that our function foo takes in a buffer and the size of said buffer. But in many cases we would like to make use of more structured data. e.g. a string or a vector of integers etc. Thankfully libfuzzer provides a handy helper to convert a raw buffer into common structured data types, the FuzzedDataProvider class. For examples on how to use this class see test/fuzz/simplify.cpp.

Other useful materials

• The official libfuzzer docs
• The libfuzzer tutorial