libsys-- syscall function module for Node.js.static-buffer--StaticBufferandStaticArrayBufferimplementations.typebase-- basic struct reader/writer into/fromBuffer.libjs-- syscall wrapper.fslib--require('fs')implementation.libaio-- Wrapper around asynchronous system calls.ass-js-- Assembler.js, x86 assembler.cpuid-- basic utility to work withcpuidsystem call from JavaScript.elf-- ELF file reader/writer.f1-- JavaScript runtime written in JavaScript that compiles JavaScript to machine code.full-js-- Drop-in Node.js replacement, written in pure JavaScript.jsc-- Write C in JavaScript and JIT compile.
The Node.js exo-kernel dream: this is a proposal to create a JavaScript exo-kernel that would have API compatible to Node.js but would be possible to run as part of the Linux kernel.
Why do I need a JavaScript exo-kernel?
Well, I don't know, but imagine you could have:
- A complete OS image (not just a container, but including the kernel and Node.js) at less than 10Mb and it would boot in a couple of dozen milliseconds.
- Full access to hardware from JavaScript.
- Write hardware drivers in JavaScript.
There are already two projects (NodeOS and runtime.js) that work
towards creating a Node.js operating system, or kernel. See architecture comparison for comparison
with jskernel proposal. The basic summary of the two is as follows:
NodeOSruns in user space on top of Node.js as the main process, so by default it comes withnpmmodule ecosystem, however, it has no access to the kernel space.runtime.jsis compiled inside the kernel, so it has full access to the kernel space, however, it does not run on Node.js, not evenlibuvorlibc, but just raw C++ bindings with kernel and V8, so one cannot runnpmmodules on runtime.js, unless they implement Node.js-like API.
The proposal of jskernel is to have the best of both worlds:
Create a Node.js standard library that would be able to run in kernel space (or user space).
So, here is how to achieve it:
- Write everything in JavaScript (or TypeScript), no C/C++ bindings.
- No dependencies, only the
syscallfunction, syscalls executed directly from JavaScript.
This way we can re-create Node.js standard library whose only dependency will be the
system call function.
We would remove all native dependencies from Node.js, including Node.js itself, libuv, libc and even V8.
The only thing you would need to do to port this thing inside the Linux kernel or to a different JavaScript runtime would
be to port the syscall function, as it will be our only dependency.
You might think it is impossible and would take too much effort to rewrite libc and libuv,
but hey, we are coding in JavaScript, libc in JavaScript sounds like a little weekend project. Moreover, below is
a road map with some working prototypes, so its possible and it is actually simpler than it looks like because
many thing we can take straight from node's standard library. To make a first working prototype we just need to implement
the networking stack: dgram, dns, net, http.
Also, Node.js devs usually write their own database drivers and parsers in Node.js, instead
of creating wrappers around the respective C/C++ libraries, so, why do we execute system calls
using libuv and libc?
Below is a list of five packages from libsys, which simply provides the syscall function, to
unode, which will be a Node.js compatible library whose only native dependency will be that syscall function.
libsys implements the basic syscall function, which will
be the only native dependency of the whole stack.
Here is a simple example that prints Hello world in console:
require('libsys').syscall(1, 1, new Buffer('Hello world\n'), 12);Here we basically execute No. 1 system call which is SYS_write to No. 1 file
descriptor which is STDOUT, so it goes to our consoles.
Done, we have a syscall function, let's move further.
libjs will be like libc in C world: it will
implement wrappers around all the necessary system calls.
For example, reading a file synchronously:
var libjs = require('libjs');
var fd = libjs.open('/share/myfile.txt', libjs.FLAG.O_RDONLY);
if(fd > -1) {
var buf = new Buffer(1024);
var bytes_read = libjs.read(fd, buf);
console.log('Bytes read: ', bytes_read);
console.log(buf.toString().substr(0, bytes_read));
libjs.close(fd);
} else {
console.log('Error: ', fd);
}libaio will use libjs to implement asynchronous IO.
Here is a prototype example from libaio of an asynchronous TCP socket that executes a simple GET request using epoll system calls:
var libaio = require('libaio');
var sock = new libaio.SocketTcp();
sock.onconnect = () => {
console.log('Connected');
sock.write('GET /\n\n');
};
sock.ondata = (data) => {
console.log('Received:');
console.log(data);
};
sock.connect({host: '192.168.1.150', port: 80});eloop will implement worker pool to outsource
CPU intensive operations that we cannot run asynchronously. Basically its task will
be similar to what libuv does for node.js -- provides a thread poll to run file IO and other CPU intensive tasks.
unode will integrate eloop, libaio and libjs to
provide a complete Node.js API, so that npm and all packages will run smoothly on unode without
even knowing they are not actually running on node.js. It will be a simple drop-in replacement,
where you run your apps with unode instead of node:
unode app.js
Meanwhile, for time being unode will run on node.js patching already implemented functionality.
Here is a list of modules from node's standard library that need to be implemented:
Copy-pasta: The below modules we should be able to take from node's standard library
and with little modification (or none at all) use in unode:
assert.js
console.js
events.js
path.js
process.js
punycode.js
querystring.js
readline.js
repl.js
module.js
url.js
util.js
stream.js
string_decoder.js
Special: When I said syscall function will be our only dependency, I actually lied. If you
look carefully at our system call function: it accepts numbers, strings, and Buffers as arguments, where
it uses Buffer as a pointer to a memory location of data. However, V8 has typed arrays and we will shim somehow the
Buffer class function, there are already buffer clones that work on typed arrays, we
just need to make sure we can pass a pointer to the memory address of buffer's contents in our system calls:
buffer.js
Priority: The below modules are priority ones for creating our first working prototype and get npm working.
They are perfectly doable using Linux's asynchronous sockets with epoll system calls:
dgram.js
dns.js
net.js
http.js
File system: fs is not an immediate priority because of two reasons: (1) there is no good async way on Linux
to handle files, you have to resort to threads; (2) we don't really need a file system in our first prototype,
instead we can use an in-memory file system like memfs:
fs.js
Misc: These are some not-important or node.js and V8 specific modules:
vm.js
v8.js
os.js
tty.js
timers.js
Hard ones: And finally we get to the modules that are hard to implement in pure JavaScript. These are computationally intensive modules for which we would need a worker pool to run in background:
child_process.js
cluster.js
crypto.js
domain.js
https.js
tls.js
zlib.js
And even these "hard ones" are not that hard, because pure JavaScript implementations already exist, we just need to solve the background worker pool problem:
Imagine that we have unode complete with the whole node's API. Remember
that now our whole stack has only one dependency -- the syscall function -- we have
removed all C/C++ dependecies. This, means that to port all of that to a different
JS runtime or into the Linux kernel is as simple as porting the syscall function.
Finally, once inside the kernel we can expose more of the kernel's functionality to JavaScript.
git clone https://github.com/streamich/jskernel
If you don't have Docker:
vagrant up
vagrant ssh
sudo /share/install.sh
Start a Docker container:
docker build -t jskernel .
docker run -it -v ~/dev/jskernel:/share --name myjskernel jskernel /bin/bash
cd /share
npm install -g npm@latest
npm install -g node-gyp n typescript tsd mocha
Next time just do:
docker start myjskernel
docker exec -it myjskernel bash
Typings:
tsd install node