Switch wasm tests to -Os

[kmiller68]

We should investigate if it's worth switching from -O2 to -Os. It seems plausible that -O2 makes wire size of a lot of wasm apps significantly larger so they might switch to -Os or should be encouraged to switch to -Os so sites load faster. This matters for JS3 since it impacts how much inlining emcc does and therefore how important it is for the wasm runtime to do good inlining.

[danleh]

I'm unsure about this as well. On the one hand, we know of large Wasm applications struggling with module size, and thus -Os sounds appropriate. On the other hand, I don't have data what is actually used in practice. Also three more points to take into consideration:

1. Should the binaries additionally be optimized with Binaryen/wasm-opt? If yes, with what optimizations?
2. Do we maybe intentionally want diverse toolchains, toolchain versions, or options, since that would cover a broader range of Wasm binaries encountered in the real world?
3. If we have Wasm workloads from, e.g., npm packages, shall we rebuild them ourselves? Or rather take the binaries as they are distributed (even if they were compiled with old toolchains or suboptimal options), since that is what engines need to run well in practice?

I currently lean towards:

1. No, if it requires a manual invocation of Binaryen. Yes, if Binaryen is automatically invoked as part of toolchains, say dart2wasm.
2. Yes, we want rather diverse/realistic toolchains, as long as there is some indication those are used in practice.
3. Use the widely distributed binaries as-is.

The Emscripten and Binaryen experts, e.g., @kripken, @sbc100, @dschuff, and @tlively might have valuable input here as well.

[dschuff]

1. If the binaries were generated with Emscripten and compiled with any optimization level above -O1 (including -Os or -Oz), they were optimized with Binaryen already, so there's probably not much benefit in running wasm-opt again (other than maybe if our version of wasm-opt is newer; but if we care about that it's better to rebuild from source with a newer Emscripten). This probably also applies for other toolchains that already incorporate wasm-opt directly (the GC-targeting toolchains I know of often have custom pass pipelines rather than just using the wasm-opt default).
2. Maybe. I know that Emscripten users make a variety of tradeoff choices in their optimization flags; some use -Os or even -Oz to get absolutely minimal size, some use -O3 to get max performance, and I think some use -O2 too. It's not always the biggest apps that go for size; I guess if your app is already 20MB you might be willing to pay an extra 1MB more for better performance because you already have to hide the latency of the big download, but you might be more conscious of that if you're starting out small.
3. I would love to have some sort of standard way to get the sources for these benchmarks and rebuild them, because that would make it easier for us (toolchain developers) to evaluate optimizations. But to the extent that we want the benchmark to reflect the web reality, then it makes sense to keep the official contents pinned to what's popular.

I guess a corollary of 3 would be how often we want to update/rebuild things.