Fiddle with various build options

[michaelfairley]

These aren't all necessarily a good idea to ship, but they should have some impact on size or speed or the resulting binary that's worth measuring.

• LTO
• panic = "abort" (I don't think we want to do this since we being able to clean up after a panic is a thing we want, but it'd still be nice to measure the impact.)
• striping debug symbols
• codegen-units = 1

[bruceg]

I ran some tests on some of these options, specifically lto and codegen-units, and the options do have a significant effect.

Setting codegen-units = 1 drops the executable size by 13% (according to size). Similarly, turning on lto = "fat" drops the executable size by 13%, with a significant decrease in the "data" size. Turning both on has a slight improvement of 17% over the original. The lto = "thin" in fact produces a larger executable at all codegen-units settings. I did not run specific tests on the effects of strip, but on this run stripping debug symbols dropped about a third of the final executable size (from 40.3MiB to 26.9MiB).

LTO	codegen-units	text	data	bss	dec
LTO=off	CU=16	31,639,578	1,323,841	2,213,312	35,176,731
LTO=off	CU=4	29,977,286	1,298,081	2,213,312	33,488,679
LTO=off	CU=1	27,074,230	1,241,777	2,213,400	30,529,407
LTO=thin	CU=16	32,192,211	1,304,976	2,213,304	35,710,491
LTO=thin	CU=4	30,431,215	1,282,272	2,213,304	33,926,791
LTO=thin	CU=1	27,838,479	1,216,192	2,213,224	31,267,895
LTO=fat	CU=16	27,323,347	1,099,128	2,213,336	30,635,811
LTO=fat	CU=4	27,094,407	1,087,288	2,213,240	30,394,935
LTO=fat	CU=1	25,786,995	1,067,384	2,213,272	29,067,651

Unfortunately, it comes at a great expense of compile time. On a 8 CU (4 core, 2 thread) EC2 system, turning both options on doubles the wall time taken (though amusingly the CPU time drops). The proportion of the time increase would grow larger at higher core counts due to the reduced parallelism.

LTO	codegen-units	user	system	elapsed	CPU%
LTO=off	CU=16	2883.80	64.98	7:02.15	698%
LTO=off	CU=4	2413.61	62.13	7:39.17	539%
LTO=off	CU=1	2186.90	59.41	11:20.03	330%
LTO=thin	CU=16	3626.89	71.93	8:39.51	711%
LTO=thin	CU=4	3054.64	68.89	9:14.76	563%
LTO=thin	CU=1	2668.88	64.75	14:06.18	323%
LTO=fat	CU=16	3219.88	68.28	12:21.50	443%
LTO=fat	CU=4	2708.81	65.59	12:31.41	369%
LTO=fat	CU=1	2435.75	62.00	15:25.35	269%

Both enabling LTO and reducing codegen-units improved benchmark performance, by as much as 20% in total. I'll present some numbers in a follow up comment.

[ghost]

@bruceg These tests look great! We have "fat" Rust codegen LTO enabled for building our release binaries and run strip on them. I think we also need to set codegen-units to 1 for them (I've opened #1798). In my opinion, it is fine that it increases compilation times because all these slow optimizations are disabled for building tests or local development builds.

For some time we also had "fat" LLVM cross-language LTO enabled (see #689 for some comparisons of Vector binary size), but it was switched off as part of #794. Because now typetag is fixed, we can enable it again (there is an issue #1036 about it).

Also it is interesting to consider compressing Vector with UPX (see #1797 for details).

[bruceg]

OK, the performance numbers. These are geometric means of the median time across all 29 benchmarks:

Mode	Mean
fat-1	17.153
fat-4	17.167
fat-16	17.284
thin-1	17.595
thin-16	17.688
thin-4	17.776
off-1	19.028
off-16	19.196
off-4	19.314

So, turning on LTO has a significant impact on performance. Setting codegen-units=1 has a small but measurable impact (more in some tests than others, which is not reflected here).