PR demonstrating WebAssembly support #1

NWilson · 2017-12-07T15:01:06Z

I know that Musl doesn't accept changes via pull-requests!

This is just a way of showing the current status of my work.

Go to the diff to see what changes I made to support Wasm.

notes added by maintainer: this function is a GNU extension. it was chosen over the similar BSD function funopen because the latter depends on fpos_t being an arithmetic type as part of its public API, conflicting with our definition of fpos_t and with the intent that it be an opaque type. it was accepted for inclusion because, despite not being widely used, it is usually very difficult to extricate software using it from the dependency on it. calling pattern for the read and write callbacks is not likely to match glibc or other implementations, but should work with any reasonable callbacks. in particular the read function is never called without at least one byte being needed to satisfy its caller, so that spurious blocking is not introduced. contracts for what callbacks called from inside libc/stdio can do are always complicated, and at some point still need to be specified explicitly. at the very least, the callbacks must return or block indefinitely (they cannot perform nonlocal exits) and they should not make calls to stdio using their own FILE as an argument.

stdio types use the struct tag names from glibc libio to match C++ ABI.

sunfishcode · 2017-12-07T22:44:51Z

arch/wasm/bits/limits.h

+ * allocations.  At 32KiB page size, we have <20% RAM
+ * waste which seems a bit more reasonable.  All a bit arbitrary.
+ */
+#define PAGE_SIZE 32768


(Random drive-by review comment...)

This property of page sizes may not always hold; it is possible that wasm could add mmap-like features in the future, in which case PAGE_SIZE should reflect the granularity of those operations.

I realize that you don't want to mix in bigger changes in this PR, but if malloc's heuristics are suboptimal for 64KiB page sizes, it would affect other architectures as well, including some arm64 platforms, so ideally the solution should be to fix malloc's heuristics.

sunfishcode · 2017-12-07T22:50:34Z

arch/wasm/bits/alltypes.h.in

+TYPEDEF float float_t;
+TYPEDEF double double_t;
+
+TYPEDEF long time_t;


long is 32-bit on wasm32, so using it for time_t is susceptible to the 2038 bug. Since wasm has a full set of 64-bit operations, I suggest making time_t 64-bit.

Yes, what was I thinking!?! The ABI is supposed to match the Linux x32 ABI, this must have snuck in from the x86 one by mistake.

I'll do a diff between arch/x32 and arch/wasm and make sure everything matches where possible.

(By comparing the syscall numbers from all the different architectures, I've deduced that Emscripten is currently trying to emulate the x86 ABI in its syscall interface, or at least seems to have copied the definitions from x86 most extensively. I think moving to the x32 definitions is a good move at this point.)

NWilson · 2017-12-07T23:48:32Z

arch/wasm/bits/syscall.h.in

+// The list of syscalls is taken from x32, which contains a sensible list of
+// "modern" syscalls.
+
+#define SYSCALL_ACTION(x) long __syscall_##x(long arg1, ...);


Some syscalls take a variable number of arguments (like SYS_open, and SYS_mremap).

I thought it would be cunning to therefore give syscalls a varargs signature. (NB. The way static linking for syscalls has to work is that all syscalls must have the same signature. See how support for syscall() and pthread_cancel() was implemented by cunningly using each syscall's function pointer as its syscall number! Hat-tip to John Starks at Microsoft.)

Making them varargs is a silly idea, I should just make all take six normal arguments, and pass zeros for the ones that don't matter.

sunfishcode · 2017-12-09T00:25:41Z

arch/wasm/bits/stdint.h

@@ -0,0 +1,31 @@
+typedef int32_t int_fast16_t;


This definition of int_fast16_t differs from clang's, but I think it's better. I filed this bug to track the difference.

sunfishcode · 2017-12-09T00:57:04Z

src/internal/wasm/wasm_init_data.c

+
+static const char* const progname = "/a.out";
+
+struct wasm_init_data_t wasm_init_data = {


Should this be named __wasm_init_data to avoid poluting the user namespace? Also, I think this can be const (though wasm doesn't have read-only memory yet).

Oops, hiding with underscores done.

Making const unfortunately not possible; Musl functions like _start_c from crt1.c expect a non-const pointer. Making const only to cast it away seems pointless.

I think for time being this sort of ELF emulation is OK. It's not too burdensome, and it's emulating a well-defined API for interaction with Musl.

If _start_wasm does become a standard thing, we should maybe add argv/envp to it as arguments.

Finally, a getrandom opcode in Wasm (__builtin_wasm_getrandom) would be nice :)

sunfishcode · 2017-12-09T01:04:53Z

src/math/wasm/truncf.c

+__attribute__((const)) float truncf(float x)
+{
+	return __builtin_truncf(x);
+}


You can also optimize nearbyint/nearbyintf and rint/rintf using __builtin_nearbyint/__builtin_nearbyintf and __builtin_rint/__builtin_rintf, respectively, which codegen to f64.nearest/f32.nearest. (This is possible for rint because wasm doesn't have user-accessible floating-point exceptions, and for nearbyint and rint because wasm doesn't have user-modifiable rounding modes.

Thanks, added those too.

I did a bit of trial-and-error to work out which math builtins could be optimised this way. Adding __attribute__((const)) was a bit of trial-and-error. Musl doesn't set errno for any maths functions, so it should be safe/consistent.

I was hoping to turn fmin/fmax into f64.min, but couldn't seem to get it to be emitted by __builtin_fmax. Those builtins are tricky, because if they don't lower to the expected native opcode, then they generate a recursive call to the libc function!

In any case, these routines in libc are pretty unimportant. LLVM replaces calls to these libc functions with the native opcodes anyway, so they won't/shouldn't actually be called here unless you're calling them by pointer.

Can you think of a reason why this body doesn't emit a nice f64.min instruction:

__attribute__((const)) double fmin(double x, double y) { return __builtin_fmin(x,y); }

I'm not confident in saying it's a bug in the WebAssembly frontend, or whether my expectation is wrong, but it is surprising that so many of the other maths builtins do work.

When one operand is NaN, wasm's min and max return NaN, while C's fmin and fmax return the other operand. Both forms are useful, so I expect that in the future wasm will have opcodes for both forms. As a historical note, the reason why wasm initially left out the latter form because IEEE 754-2008's definition of this form had surprising behavior on signaling NaN (see here for a description). The IEEE 754 committee has recently recognized this as an error, and drafts of the upcoming 754-2018 have removed that definition and provided a new corrected one.

I see, thanks!

sunfishcode · 2017-12-09T01:40:10Z

src/internal/wasm/wasm_init_data.c

+#endif
+		AT_NULL,     0
+	}
+};


Random musing: I realize that this serves a purpose right now, though at some point we may want to explore other options. Portable code won't likely use interfaces like getauxv in the first place, for wasm-specific code there are better ways to obtain most of this information, and for musl initialization, there isn't all that much code that needs these, and some of it needs auxv fields that aren't supported (AT_PHDR etc.).

sunfishcode · 2017-12-11T16:50:56Z

arch/wasm/bits/posix.h

@@ -0,0 +1,2 @@
+#define _POSIX_V6_LP64_OFF64  1
+#define _POSIX_V7_LP64_OFF64  1


wasm32 is ILP32, so it looks like these should be ILP32_OFFBIG rather than LP64_OFF64, for wasm32.

Hm, I just copied that blindly from the x32 port - which is also ILP32, surely? So it looks like a bug in arch/x32/bits/posix.h that I didn't notice or think about. off_t is complicated, I was just hoping I wouldn't have to check out the specs for it!

I'll raise it tomorrow on the Musl mailing list for the x32 port.

notes added by maintainer: the '-' specifier allows default padding to be suppressed, and '_' allows padding with spaces instead of the default (zeros). these extensions seem to be included in several other implementations including FreeBSD and derivatives, and Solaris. while portable software should not depend on them, time format strings are often exposed to the user for configurable time display. reportedly some python programs also use and depend on them.

NWilson · 2017-12-12T14:56:13Z

On an aside, I'm currently having a go at a "JavaScript linker".

Emscripten currently does this using a very awkward <symbol>__deps annotation, which feels a bit odd to me. I'm trying to build something simple using nodejs, that does the following steps:

Load WebAssembly file and list of Javascript files specified on the commandline
For the Wasm module, grab its dependencies from its import list
For the JS files, parse them using Shift AST, to get a list of symbol definitions, and extract all referenced variables using Shift's scope analyser, ie automatically build the deps list
Output a Javascript module that pulls in the JS symbols that are used, and loads the Wasm module with the symbols it imports. JS symbols are all in a shared scope, so can reference each other regardless of whether they were pulled in by the Wasm module or other imports.

I don't think there's anything quite like that at the moment? Emscripten's Python module loader has quite a lot of asm.js going on, and it would be nice to have a short & clean linker/JS-module-builder that just knows how to assemble the Javascript side of a Wasm module.

Then there'd need to be a libc.js (derived from Emscripten's existing library.js) that implements some sane subset of the Musl syscalls.

aside from theoretical arbitrary results due to UB, this could practically cause unbounded overflow of static array if hit, but hitting it depends on having more than 32 calls to at_quick_exit and having them sufficiently often.

sysconf should return -1 for infinity, not LONG_MAX.

notes by maintainer: both C and POSIX use the term UTC to specify related functionality, despite POSIX defining it as something more like UT1 or historical (pre-UTC) GMT without leap seconds. neither specifies the associated string for %Z. old choice of "GMT" violated principle of least surprise for users and some applications/tests. use "UTC" instead.

notes by maintainer: commit 2f853dd added these rules because the new system for handling arch-provided replacement files introduced for out-of-tree builds did not apply to the crt tree. commit 63bcda4 later adapted the makefile logic so that the crt and ldso trees go through the same replacement logic as everything else, but failed to remove the explicit rules that assumed the arch would always provide asm replacements. in addition to cleaning things up, removing these spurious rules allows crti/crtn asm to be omitted by an arch (thereby using the empty C files instead) if they are not needed.

NWilson · 2017-12-15T17:02:29Z

arch/wasm/bits/float.h

+#define LDBL_MAX 1.18973149535723176508575932662800702e+4932L
+#define LDBL_EPSILON 1.92592994438723585305597794258492732e-34L
+
+#define LDBL_MANT_DIG 113


@sunfishcode, do you know how Wasm has 80-bit long-double support? I would have expected long-double to be the same as double, and was surprised to find that it's not. Does clang emulate the higher-precision, or is it fictitious?

I got these constants here by dumping Clang's predefined macros (as requested by Rich Felker in his review on the Musl mailing list).

Clang is currently configured to make long-double be a 128-bit IEEE-754 quad-precision type. It is supported through software emulation in compiler-rt (it calls __addtf3 for addition, and so on).

The decision to do this is debatable. On one hand, making long double the same as double makes long double effectively useless, and making it longer makes it at least useful for some purposes (even if they are fairly narrow). On the other, musl's printf promotes all floating-point values to long double to avoid duplicating code, and this means that all floating-point formating goes through slow software-emulated paths, and it increases code size for any program that calls printf.

I expect we'll eventually have custom versions of printf that will disable disable various bits of floating-point support, because many applications won't need everything (and the nature of printf format strings makes it difficult to dce unused formatting code), and if we do, this would significantly mitigate the problems. However, the utility of 128-bit long double is admittedly obscure, so it's not completely certain that the benefits outweigh the costs.

When you say "Clang is currently configured", I assume that's specific to the WebAssembly backend? Given that GCC on x86 doesn't give you 128-bit quad-precision floats, the need for that level of precision seems remote.

I don't know - maybe people really would welcome having that high precision available in Wasm via Clang. But, to me it just seems like a pitfall.

Something to think about, thanks for explaining. I'd vote to have double match long-double on Wasm, at least until the demand for it is strong enough. Since we don't yet have dynamic linkage, changing the ABI isn't too big a deal.

I've found emscripten-core/emscripten#4340 now. I see Derek was thinking of going back to 64-bit long-double, but that doesn't seem to have happened. Oh well - up to you guys!

WebAssembly doesn't have user-accessible floating point exceptions, so there's no advantage to forcing floating-point expression evaluations.

…ibcxx

…ation

kaniini and others added 2 commits December 6, 2017 13:11

adjust fopencookie structure tag for ABI-compat

2488d31

stdio types use the struct tag names from glibc libio to match C++ ABI.

NWilson force-pushed the musl-wasm-native branch from 6bf1680 to b638f75 Compare December 7, 2017 16:11

sunfishcode reviewed Dec 7, 2017

View reviewed changes

NWilson commented Dec 7, 2017

View reviewed changes

sunfishcode reviewed Dec 9, 2017

View reviewed changes

sunfishcode reviewed Dec 11, 2017

View reviewed changes

richfelker and others added 7 commits December 12, 2017 13:12

add ibm1047 codepage (ebcdic representation of latin1) to iconv

01957be

fix data race in at_quick_exit

6430315

aside from theoretical arbitrary results due to UB, this could practically cause unbounded overflow of static array if hit, but hitting it depends on having more than 32 calls to at_quick_exit and having them sufficiently often.

fix x32 unistd macros to report as ILP32 not LP64

1312768

fix sysconf for infinite rlimits

3ec8287

sysconf should return -1 for infinity, not LONG_MAX.

fix endian errors in arpa/nameser.h due to failure to include endian.h

14cec86

NWilson commented Dec 15, 2017

View reviewed changes

richfelker and others added 12 commits December 15, 2017 12:58

fix endian errors in netinet/icmp6.h due to failure to include endian.h

d5029bb

[Wasm] first cut of Wasm support - most basic libc features working!

cb4e760

[Wasm] Add math.h overrides using Wasm intrinsics

a831cf7

[Wasm] Add __syscall_mmap implementation

a0f61de

[Wasm] Use LDFLAGS when testing linker

8725596

[Wasm] Add wasm.syms file

b6e20dd

[Wasm] Remove temporary debugging #define

712e548

Make FORCE_EVAL a no-op on wasm.

d52801c

WebAssembly doesn't have user-accessible floating point exceptions, so there's no advantage to forcing floating-point expression evaluations.

[Wasm] Synchronise a few more definitions with x32 rather than x86

e7eb8eb

[Wasm] Make syscall interface non-varargs

399c7f1

[Wasm] hide wasm_init_data symbol with underscores

01ed7c9

[Wasm] Add another couple of optimised math functions

39b2c79

NWilson added 9 commits December 18, 2017 11:14

[Wasm] Use ILP32 rather than LP64 macros on Wasm

f547c4e

[Wasm] Fix __builtin_unreachable misuse

81ae5b2

[Wasm] Hardcode float constants to avoid ABI depending on the compiler

c5b54ec

[Wasm] Update some comments

dc223ff

[Wasm] Add functioning dummy for pthread_self()

07bf093

[Wasm] Remove bits/errno.h override, better to fix ELAST problem in l…

502e1f0

…ibcxx

[Wasm] Exclude internal syscall from exported syms list

203e585

[Wasm] Move Wasm initialisation to use Dan's global-ctor entrypoint

32323af

[Wasm] Switch to using Dan's global-constructor approach to initialis…

14a308a

…ation

NWilson force-pushed the musl-wasm-native branch from a9ecdbe to 14a308a Compare December 18, 2017 11:14

NWilson added 5 commits January 22, 2018 17:33

[Wasm] rename __heap_bottom to __heap_base

a91a176

Fix use of weakly-defined address, not variable

448a5ee

Add support for thread-local data to Wasm

e1dfbb4

Wasm: remove some x86 syscalls

d10e141

Wasm: add __syscall implementation for the sake of __syscall_cp

7cd91fc

NWilson force-pushed the musl-wasm-native branch from 042895a to 7cd91fc Compare February 21, 2018 10:44

Wasm: Fix build by stubbing out x86-specific calls

e71cc17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PR demonstrating WebAssembly support #1

PR demonstrating WebAssembly support #1

NWilson commented Dec 7, 2017 •

edited

Loading

sunfishcode Dec 7, 2017

sunfishcode Dec 7, 2017

NWilson Dec 7, 2017

NWilson Dec 7, 2017

sunfishcode Dec 9, 2017

sunfishcode Dec 9, 2017

NWilson Dec 11, 2017

sunfishcode Dec 9, 2017

NWilson Dec 11, 2017

NWilson Dec 11, 2017

sunfishcode Dec 12, 2017

NWilson Dec 12, 2017

sunfishcode Dec 9, 2017

sunfishcode Dec 11, 2017

NWilson Dec 11, 2017

NWilson commented Dec 12, 2017

NWilson Dec 15, 2017

sunfishcode Dec 20, 2017

NWilson Dec 20, 2017

NWilson Dec 20, 2017


		static const char* const progname = "/a.out";

		struct wasm_init_data_t wasm_init_data = {

		@@ -0,0 +1,2 @@
		#define _POSIX_V6_LP64_OFF64 1
		#define _POSIX_V7_LP64_OFF64 1

PR demonstrating WebAssembly support #1

Are you sure you want to change the base?

PR demonstrating WebAssembly support #1

Conversation

NWilson commented Dec 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NWilson commented Dec 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NWilson commented Dec 7, 2017 •

edited

Loading