Switch wasm tests to `-Os`

[kmiller68]

We should investigate if it's worth switching from -O2 to -Os. It seems plausible that -O2 makes wire size of a lot of wasm apps significantly larger so they might switch to -Os or should be encouraged to switch to -Os so sites load faster. This matters for JS3 since it impacts how much inlining emcc does and therefore how important it is for the wasm runtime to do good inlining.

[danleh]

I'm unsure about this as well. On the one hand, we know of large Wasm applications struggling with module size, and thus -Os sounds appropriate. On the other hand, I don't have data what is actually used in practice. Also three more points to take into consideration:

1. Should the binaries additionally be optimized with Binaryen/wasm-opt? If yes, with what optimizations?
2. Do we maybe intentionally want diverse toolchains, toolchain versions, or options, since that would cover a broader range of Wasm binaries encountered in the real world?
3. If we have Wasm workloads from, e.g., npm packages, shall we rebuild them ourselves? Or rather take the binaries as they are distributed (even if they were compiled with old toolchains or suboptimal options), since that is what engines need to run well in practice?

I currently lean towards:

1. No, if it requires a manual invocation of Binaryen. Yes, if Binaryen is automatically invoked as part of toolchains, say dart2wasm.
2. Yes, we want rather diverse/realistic toolchains, as long as there is some indication those are used in practice.
3. Use the widely distributed binaries as-is.

The Emscripten and Binaryen experts, e.g., @kripken, @sbc100, @dschuff, and @tlively might have valuable input here as well.

[dschuff]

1. If the binaries were generated with Emscripten and compiled with any optimization level above -O1 (including -Os or -Oz), they were optimized with Binaryen already, so there's probably not much benefit in running wasm-opt again (other than maybe if our version of wasm-opt is newer; but if we care about that it's better to rebuild from source with a newer Emscripten). This probably also applies for other toolchains that already incorporate wasm-opt directly (the GC-targeting toolchains I know of often have custom pass pipelines rather than just using the wasm-opt default).
2. Maybe. I know that Emscripten users make a variety of tradeoff choices in their optimization flags; some use -Os or even -Oz to get absolutely minimal size, some use -O3 to get max performance, and I think some use -O2 too. It's not always the biggest apps that go for size; I guess if your app is already 20MB you might be willing to pay an extra 1MB more for better performance because you already have to hide the latency of the big download, but you might be more conscious of that if you're starting out small.
3. I would love to have some sort of standard way to get the sources for these benchmarks and rebuild them, because that would make it easier for us (toolchain developers) to evaluate optimizations. But to the extent that we want the benchmark to reflect the web reality, then it makes sense to keep the official contents pinned to what's popular.

I guess a corollary of 3 would be how often we want to update/rebuild things.

[danleh]

Another question regarding build options for Wasm line items: Currently we compile them with -g1 --emit-symbol-map in the new build script, presumably to help with performance analysis/profiling. This produces significantly larger JS code. E.g., I just checked for quicksort-wasm, the generated quicksort.js is ~28KB with -O2 -g1 and ~11KB with -O2 (--emit-symbol-map doesn't make a difference). I would assume most code on the web is shipped without -g and we should do the same here?

[kmiller68]

My thoughts:

1. No, for reasons described above.
2. This is a tough question. Another wrinkle is that I would think large, well-staffed projects would do something like WebKit (and I think Chromium) build with a PGO profile, which might also change the optimization level they choose. Funnily enough, I would have guessed it would be the opposite of @dschuff and guessed you'd care more about the absolute size change. So small projects would build with -O3 since it only changes the binary by a few KB but for big projects the extra 5-10% download size actually adds up.
3. I think it would be nice to rebuild them ourselves but only if we can rebuild them in the same way the project does. I think we should be more careful about changing sources since that could fundamentally change the workloads the benchmark is testing. Or at least, that's not a discussion we've had yet.
4. (r.e. -g) I'm not sure how much value add there is by minifying the wasm-js bindings. If we can show it meaningfully impacts the profile then it might be worth it but otherwise we're just making our life more difficult for little benefit. I don't think --emit-symbol-map impacts any resources the browser consumes, which is why I added it over info directly in the binary.

[danleh]

I don't think --emit-symbol-map impacts any resources the browser consumes
That's correct. For quicksort-wasm at least, --emit-symbol-map does not affect the generated JS or Wasm code at all.

Regarding -g: I just did a short experiment on quicksort-wasm and couldn't find a statistically significant difference (N=20 runs) between -O2 and -O2-g in terms of score or runtime. Hence let's keep -g by default as you say, until we find issues with it.