Create standardized Docker development environment via .devcontainer (sql-js#445)

* Initial default .devcontainer and Dockerfile created

* Install EMSDK and Chrome dependencies in the Dockerfile

* Move Contributing/Compiling instructions to Contributing.md

* Start worker test in Puppeteer without Chrome sandbox when running in Docker

* Add `clean` package script and express `rebuild` script in terms of npm scripts

This aims to express all build tasks via npm scripts via make commands, rather than expressing some scripts as npm scripts and others in terms of make.

* Pin EMSDK to 2.0.15

* First attempt at Github action building inside of Docker container

* Switch cmd to entrypoint

* Create an entrypoint.sh for the Github build-sqljs action

* remove comments

* Make action script executable

* Set correct path in docker image

Co-authored-by: lovasoa <[email protected]>

Author: Taytay

.devcontainer/Dockerfile

@@ -0,0 +1,127 @@
+# We build our DevContainer on MS' Typescript-Node Devcontainer
+# This gives us lots of standard stuff, and lets us layer a few custom things on top, like the Emscripten compiler, Puppeteer
+
+# --------------------------------------------------------------------
+# BEGIN Standard MS Devcontainer for Typescript-Node 
+
+# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.155.1/containers/typescript-node/.devcontainer/base.Dockerfile
+# [Choice] Node.js version: 14, 12, 10
+ARG VARIANT="14-buster"
+FROM mcr.microsoft.com/vscode/devcontainers/typescript-node:0-${VARIANT}
+
+# [Optional] Uncomment if you want to install an additional version of node using nvm
+# ARG EXTRA_NODE_VERSION=10
+# RUN su node -c "source /usr/local/share/nvm/nvm.sh && nvm install ${EXTRA_NODE_VERSION}"
+
+# [Optional] Uncomment if you want to install more global node packages
+# RUN su node -c "npm install -g <your-package-list -here>"
+
+# END Standard MS Devcontainer for Typescript-Node 
+# --------------------------------------------------------------------
+
+# --------------------------------------------------------------------
+# BEGIN EMSDK 
+# Install EMSDK to /emsdk just like the EMSDK Dockerfile: https://github.com/emscripten-core/emsdk/blob/master/docker/Dockerfile
+ENV EMSDK /emsdk
+# We pin EMSDK to 2.0.15 rather than 'latest' so that everyone is using the same compiler version
+ENV EMSCRIPTEN_VERSION 2.0.15
+
+RUN git clone https://github.com/emscripten-core/emsdk.git $EMSDK
+
+RUN echo "## Install Emscripten" \
+    && cd ${EMSDK} \
+    && ./emsdk install ${EMSCRIPTEN_VERSION} \
+    && echo "## Done"
+
+# Copied directly from https://github.com/emscripten-core/emsdk/blob/master/docker/Dockerfile
+RUN cd ${EMSDK} \
+    && echo "## Generate standard configuration" \
+    && ./emsdk activate $EMSCRIPTEN_VERSION \
+    && chmod 777 ${EMSDK}/upstream/emscripten \
+    && chmod -R 777 ${EMSDK}/upstream/emscripten/cache \
+    && echo "int main() { return 0; }" > hello.c \
+    && ${EMSDK}/upstream/emscripten/emcc -c hello.c \
+    && cat ${EMSDK}/upstream/emscripten/cache/sanity.txt \
+    && echo "## Done"
+
+ENV PATH $EMSDK:$EMSDK/upstream/emscripten/:$PATH
+
+# Cleanup Emscripten installation and strip some symbols
+RUN echo "## Aggressive optimization: Remove debug symbols" \
+    && cd ${EMSDK} && . ./emsdk_env.sh \
+    # Remove debugging symbols from embedded node (extra 7MB)
+    && strip -s `which node` \
+    # Tests consume ~80MB disc space
+    && rm -fr ${EMSDK}/upstream/emscripten/tests \
+    # Fastcomp is not supported
+    && rm -fr ${EMSDK}/upstream/fastcomp \
+    # strip out symbols from clang (~extra 50MB disc space)
+    && find ${EMSDK}/upstream/bin -type f -exec strip -s {} + || true \
+    && echo "## Done"
+
+RUN echo ". /emsdk/emsdk_env.sh" >> /etc/bash.bashrc
+# END EMSDK
+# --------------------------------------------------------------------
+
+# --------------------------------------------------------------------
+# BEGIN PUPPETEER dependencies
+# Here we install all of the packages depended upon by Chrome (that Puppeteer will use for headless tests).
+# We could also take a page from https://github.com/buildkite/docker-puppeteer/blob/master/Dockerfile instead,
+# and install the latest stable version of Chrome to get the right dependencies, but that version changes over time,
+# so the stable version of Chrome and the version installed by Puppeteer might diverge over time. 
+# It also means they end up having Chrome downloaded and installed twice.
+# We could install the particular version of Chrome that our version of Puppeteer would use and then tell Puppeteer not to download its own version of Chrome,
+# but then we'd have to rebuild our Docker container every time we revved Puppeteer, and that feels fiddly too.
+# For all of these reasons, it seems safer to simply install the explicit list packages depended upon by Chrome, assume that's unlikely to change
+# and move on.
+
+# List taken from:
+# https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#chrome-headless-doesnt-launch-on-unix
+RUN apt-get update \
+     && apt-get install -y \
+        ca-certificates \
+        fonts-liberation \
+        libappindicator3-1 \
+        libasound2 \
+        libatk-bridge2.0-0 \
+        libatk1.0-0 \
+        libc6 \
+        libcairo2 \
+        libcups2 \
+        libdbus-1-3 \
+        libexpat1 \
+        libfontconfig1 \
+        libgbm1 \
+        libgcc1 \
+        libglib2.0-0 \
+        libgtk-3-0 \
+        libnspr4 \
+        libnss3 \
+        libpango-1.0-0 \
+        libpangocairo-1.0-0 \
+        libstdc++6 \
+        libx11-6 \
+        libx11-xcb1 \
+        libxcb1 \
+        libxcomposite1 \
+        libxcursor1 \
+        libxdamage1 \
+        libxext6 \
+        libxfixes3 \
+        libxi6 \
+        libxrandr2 \
+        libxrender1 \
+        libxss1 \
+        libxtst6 \
+        lsb-release \
+        wget \
+        xdg-utils
+
+# We set this env variable (RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX=1) this to tell our sql.js test harness to run Puppeteer without the sandbox.
+# Otherwise, when we instantiate Puppeteer, we get this error: 
+# Puppeteer can't start due to a sandbox error. (Details follow.)
+#     [0321/173044.694524:FATAL:zygote_host_impl_linux.cc(117)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/master/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.
+ENV RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX=1
+
+# END PUPPETEER
+# --------------------------------------------------------------------

.devcontainer/devcontainer.json

@@ -0,0 +1,24 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
+// https://github.com/microsoft/vscode-dev-containers/tree/v0.155.1/containers/typescript-node
+{
+	"name": "Node.js & TypeScript",
+	"build": {
+		"dockerfile": "Dockerfile",
+		// Update 'VARIANT' to pick a Node version: 10, 12, 14
+		"args": {
+			"VARIANT": "14"
+		},
+	},
+	// Set *default* container specific settings.json values on container create.
+	"settings": {},
+	// Add the IDs of extensions you want installed when the container is created.
+	"extensions": [
+		"dbaeumer.vscode-eslint"
+	],
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+	// Use 'postCreateCommand' to run commands after the container is created.
+	// "postCreateCommand": "yarn install",
+	// Comment out connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
+	"remoteUser": "node"
+}

.github/actions/build-sqljs/action.yml

@@ -0,0 +1,7 @@
+# action.yml
+name: 'Build SQL.js'
+description: 'Builds sql.js using the .devcontainer/Dockerfile as its environment'
+runs:
+  using: 'docker'
+  image: '../../../.devcontainer/Dockerfile'
+  entrypoint: "/github/workspace/.github/actions/build-sqljs/entrypoint.sh"

.github/actions/build-sqljs/entrypoint.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+
+set -e
+
+cd /github/workspace/
+npm run rebuild

.github/workflows/CI.yml

@@ -7,17 +7,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - uses: actions/cache@v1
-      id: cache
-      with:
-        path: '.emsdk-cache'
-        key: emscripten-2.0.6
-    - uses: mymindstorm/setup-emsdk@ca33dc66a6b178f65393989c12e9465baf053352
-      with:
-        version: '2.0.6'
-        actions-cache-folder: '.emsdk-cache'
     - name: make
-      run: make
+      uses: ./.github/actions/build-sqljs
     - uses: actions/upload-artifact@v2
       with: {name: dist, path: dist}
     - name: test

CONTRIBUTING.md

@@ -0,0 +1,66 @@
+
+# Compiling and Contributing
+
+General consumers of this library don't need to read any further. (The compiled files are available via the [release page](https://github.com/sql-js/sql.js/releases).)
+
+If you want to compile your own version of SQLite for WebAssembly, or want to contribute to this project, read on.
+
+## Setting up your Development Environment
+
+### Containerized Development Environment (Recommended) 
+
+This project defines a standardized development environment using Docker (and the .devcontainer spec in particular). This allows for anyone on any platform to get up and running quickly. (VSCode is not technically required to make use of this standardized environment, but it makes containerized development so seamless that the non-VSCode path is not currently documented here.)
+
+Standardizing our development environment has numerous benefits:
+- Allows anyone on ANY platform (Linux, Mac, and Windows) to contribute or compile their own build.
+- It's quicker and easier for any contributor to dive in and fix issues.
+- (Practically) eliminates configuration bugs that are difficult for maintainers to reproduce. Also known as "works on my machine" issues.
+- Allows us to write our scripts assuming that they're _always_ running in a single known environment of a single, known platform. 
+- Ensure that all contributors use a known, standardized installation of EMSDK.
+- Allows for a more clearly documented process for updating the EMSDK to a new version.
+- End-Users that simply want to compile and install their own version of SQLite don't have to bother with EMSDK installation in their particular environment.
+
+To get started:
+
+1. Follow the [Installation Steps for Containerized Development in VSCode](https://code.visualstudio.com/docs/remote/containers#_installation). This includes installing Docker, VSCode, and the [Remote Development extension pack](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack) for VSCode)
+2. Clone this repository
+3. Open the repository folder in VSCode. It will detect the presence of a .devcontainer and prompt you: "Folder contains a Dev Container configuration file. Reopen folder to develop in a container." Click "Reopen in container"
+
+You're now ready to test the dev environment:
+4. Click on Terminal->New Terminal to be dropped into a terminal inside the dev environment.
+5. Run `$ npm install` to install the required modules
+6. Run `$ npm test` to ensure all tests pass
+7. Run `$ npm rebuild` to re-compile the project from scratch (using the version of EMSDK installed in the container).
+8. Run `$ npm test` to ensure all tests pass after said rebuild
+
+You're now ready for development!
+
+### Host-based configuration (Not recommended)
+
+If you're on a Mac or Linux-based host machine, you can install and use the EMSDK directly to perform a build.
+Note that if you run into bugs with this configuration, we highly encourage you to use the containerized development environment instead, as detailed above.
+
+Instructions:
+
+1. [Install the EMSDK](https://emscripten.org/docs/getting_started/downloads.html)
+2. Clone this repository
+3. Run `$ npm install` to install the required modules
+4. Run `$ npm test` to ensure all tests pass
+5. Run `$ npm rebuild` to re-compile the project from scratch (using the version of EMSDK installed in the container).
+6. Run `$ npm test` to ensure all tests pass after said rebuild
+
+## Compiling SQLite with different options
+
+In order to enable extensions like JSON1 or FTS5, change the CFLAGS in the [Makefile](Makefile) and run `npm run rebuild`:
+
+``` diff
+CFLAGS = \
+        -O2 \
+        -DSQLITE_OMIT_LOAD_EXTENSION \
+        -DSQLITE_DISABLE_LFS \
+        -DSQLITE_ENABLE_FTS3 \
+        -DSQLITE_ENABLE_FTS3_PARENTHESIS \
++       -DSQLITE_ENABLE_FTS5 \
++       -DSQLITE_ENABLE_JSON1 \
+        -DSQLITE_THREADSAFE=0
+```

Makefile

@@ -1,4 +1,4 @@
-# Note: Last built with version 1.38.30 of Emscripten
+# Note: Last built with version 2.0.15 of Emscripten
 
 # TODO: Emit a file showing which version of emcc and SQLite was used to compile the emitted output.
 # TODO: Create a release on Github with these compiled assets rather than checking them in

README.md

@@ -322,21 +322,8 @@ For each [release](https://github.com/sql-js/sql.js/releases/), you will find a
  - `sql-asm-debug.js` : The _Debug_ asm.js version of Sql.js. Use this for local development.
  - `worker.*` - Web Worker versions of the above libraries. More limited API. See [examples/GUI/gui.js](examples/GUI/gui.js) for a good example of this.
 
-## Compiling
-
-- Install the EMSDK, [as described here](https://emscripten.org/docs/getting_started/downloads.html)
-- Run `npm run rebuild`
-
-In order to enable extensions like FTS5, change the CFLAGS in the [Makefile](Makefile) and rebuild:
-
-``` diff
-CFLAGS = \
-        -O2 \
-        -DSQLITE_OMIT_LOAD_EXTENSION \
-        -DSQLITE_DISABLE_LFS \
-        -DSQLITE_ENABLE_FTS3 \
-        -DSQLITE_ENABLE_FTS3_PARENTHESIS \
-+       -DSQLITE_ENABLE_FTS5 \
-        -DSQLITE_ENABLE_JSON1 \
-        -DSQLITE_THREADSAFE=0
-```
+## Compiling/Contributing
+
+General consumers of this library don't need to read any further. (The compiled files are available via the [release page](https://github.com/sql-js/sql.js/releases).)
+
+If you want to compile your own version of SQLite for WebAssembly, or want to contribute to this project, see [CONTRIBUTING.md](docs/CONTRIBUTING.md).

package.json

@@ -20,7 +20,8 @@
 	"main": "./dist/sql-wasm.js",
 	"scripts": {
 		"build": "make",
-		"rebuild": "make clean && make",
+		"rebuild": "npm run clean && npm run build",
+		"clean": "make clean",
 		"test": "npm run lint && npm run test-asm && npm run test-asm-debug && npm run test-wasm && npm run test-wasm-debug && npm run test-asm-memory-growth",
 		"lint": "eslint .",
 		"prettify": "eslint . --fix",

test/test_worker.js

@@ -6,13 +6,14 @@
 var puppeteer = require("puppeteer");
 var path = require("path");
 var fs = require("fs");
+const { env } = require("process");
 
 class Worker {
   constructor(handle) {
     this.handle = handle;
   }
   static async fromFile(file) {
-    const browser = await puppeteer.launch();
+    const browser = await Worker.launchBrowser();
     const page = await browser.newPage();
     const source = fs.readFileSync(file, 'utf8');
     const worker = await page.evaluateHandle(x => {
@@ -21,6 +22,34 @@ class Worker {
     }, source);
     return new Worker(worker);
   }
+
+  static async launchBrowser(){
+    try{
+      return await puppeteer.launch();
+    }
+    catch(e){
+      if (e.stack.includes('No usable sandbox!')){
+        // It's possible that this exception is n expected error related to not having the ability to create a sandboxed user for Puppeteer in Docker. 
+        // One way around this is to set up the Dockerfile to have a sandboxed user.
+        // Details on getting Puppeteer running sandboxed while in Docker are here:
+        // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#running-puppeteer-in-docker 
+        // That seemed kinda complicated, so I'm working around it more quickly/straightforwardly by looking for an env variable we set in the Docker fil `RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX`. 
+        // -- Taytay
+        if (env['RUN_WORKER_TEST_WITHOUT_PUPPETEER_SANDBOX']=="1"){
+          // This tells puppeteer to launch without worrying about the sandbox.
+          // That's not "safe" if you don't trust the code you're loading in the browser, 
+          // but we're in a container and we know what we're testing.
+          return await puppeteer.launch({args: ['--no-sandbox', '--disable-setuid-sandbox']});
+        }
+        else {
+          console.warn("Puppeteer can't start due to a sandbox error. (Details follow.)\nFor a quick, but potentially dangerous workaround, you can set the environment variable 'RUN_WORKER_TEST_WITHOUT_PUPETEER_SANDBOX=1'.\nYou can also simply run this test in the Docker container defined in .devcontainer/Dockerfile.");
+        }
+      }
+      // If we're here, we couldn't get out of this cleanly. Re-throw
+      throw e;
+    }
+  }
+
   async postMessage(msg) {
     return await this.handle.evaluate((worker, msg) => {
       return new Promise((accept, reject) => {