FFT benchmark performance calculation

[neoskypig]

In apps/fft/main.cpp, I feel confused about below code, Why does the halide running time need to devided by reps? but FFTW doesn't do it?
double halide_t = benchmark(samples, 1, [&]() { bench_c2c.realize(R_c2c); }) * 1e6 / reps;
double fftw_t = benchmark(samples, reps, [&]() { fftwf_execute(c2c_plan); }) * 1e6;

[abadams]

In the first case, the code itself is able to run for some number of reps, so call the lambda once, time it, and divide by the number of reps we asked bench_c2c to do internally. In the second case reps is passed as an argument to benchmark, which internally calls the lambda that many times and then does the division.

[neoskypig]

@abadams thanks for your feedback. Why don't use same method for both time testings? I found If we change it to same method, the Halide version performance score became worse.

[dsharletg]

You might be able to recover most of the loss in performance by changing the Halide target to include Target::NoAsserts and Target::NoBoundsQuery. But there is also a small bit of overhead associated with calling realize that might also be affecting it. This can be avoided by AOT compiling the pipeline with a generator.

The reasoning behind the benchmarking is:

• When I looked, it seemed like fftw benchmarked to/from the same buffer, which means the input and output will be cached (if it is small enough), which is also what this FFT program does.
• In practice, we use this FFT by running it in batches, like the Halide program does here. And typically, the input and output buffers are part of a larger pipeline, so it is reasonable to assume they are cached.
• It might be possible to make FFTW also run in batches, which would be the most fair comparison, but I am not sure this would allow running to/from the same buffer (for caching). But even if you could, you wouldn't be able to fuse other work with the FFTs, which you can with the Halide version.

[abadams]

In other words, this benchmark is representative of how the Halide FFT is used in production, and how fftw would be used in production if it were to be used instead.