Cleaned up setup instructions (#234)

* Cleaned up setup instructions

* Update README.md

README.md

@@ -1,90 +1,114 @@
 # Extender
 
-Extender is a build server that builds native extensions of the Defold engine.
+Extender is a build server that builds native extensions of the Defold engine. The build server can either by run using Docker or as a stand alone server running on macOS.
 
-## Development
-This describes how to run the build server locally.
 
-### Prerequisites
+## Running on Docker
+This describes how to build and run the extender server using Docker. It involves 4 steps:
 
-* Make sure you have [Docker](https://www.docker.com) installed and running.
+1. Download and prepare required packages
+2. Serve the required packages
+3. Build the Docker image
+4. Run the Docker image in a container
 
-* If you are not part of the team that makes releases, you can skip this step:
 
-  * Clone this repo with the _recurse-submodules_ parameter (some of the submodules are private, so it's fine if you can't download some of them, just move to the next step):
-    * git clone --recurse-submodules <repo>
+### Prerequisites
+Make sure you have [Docker](https://www.docker.com) installed and running.
 
-  * If you have already cloned the repo, you can init and fetch the submodule like this:
-    * git submodule init
-    * git submodule update
+### Download and prepare required packages
+Most of the SDKs used by Extender server have licenses that prevent third party redistribution. To efficiently work with the build server it is recommended to package the SDKs and serve them from a private URL. The URL is defined as the DM_PACKAGES_URL environment variable.
 
-* Make sure you have access to the url where to download packages from `DM_PACKAGES_URL`
+The [Dockerfile](./server/docker-base/Dockerfile) lists the actual packages needed per platform:
 
-  * See the [Dockerfile](./server/docker-base/Dockerfile) for what actual packages are needed.
-  * See [defold/scripts/package](https://github.com/defold/defold/tree/dev/scripts/package) folder for scripts how to create these packages.
-  * If you have all files locally, you can serve them locally like so:
-    * export DM_PACKAGES_URL=http://localhost
-    * cd defold/local_sdks && python -m SimpleHTTPServer
+* Clang LLVM: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L49-L51
+* HTML5
+   * Emscripten: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L103
+* Windows
+   * Microsoft Visual Studio 2019: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L103
+   * Windows Kits: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L169
+* Android:
+   * NDK and SDK: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L259-L260
+* iOS and macOS
+   * iOS, macOS, Xcode - previous version: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L475-L485
+   * iOS, macOS, Xcode - latest version: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L488-L496
 
-### Build
+We have prepared scripts to package the required files. Use the scripts in the [defold/scripts/package](https://github.com/defold/defold/tree/dev/scripts/package) folder to create these packages.
 
-* Build the Extender Docker image by running:
+NOTE: If you only plan to use the extender server to build for a single platform you may remove the setup steps for the other platforms to speed up the build process.
 
-        $ DM_PACKAGES_URL=https://hostname/path/to/packages ./server/scripts/build.sh
+### Serve the required packeges
+When the packages are downloaded you need to make them available when the Docker container is built. The recommended way is to serve the files using Python:
 
-To speed things up, tests can be disabled by passing `-xtest` to the command line.
+```
+# Using python 2
+$ export DM_PACKAGES_URL=http://localhost
+$ cd local_sdks && python -m SimpleHTTPServer
+```
 
-_NOTE:_ The first time you build it will take a while (~45minutes). After that Docker cache will speed it up.
+```
+# Using python 3
+$ export DM_PACKAGES_URL=http://localhost:9999
+$ cd local_sdks && python -m http.server 9999
+```
 
-### Start
-* Then, start a container based on that image by running: `./server/scripts/run-local.sh`.
-* The build server is now available on port `http://localhost:9000`
+### Build the Docker image
+Build the Extender Docker image by running:
 
-### Stop
-* Just hit `Ctrl-C`.
+```
+./server/scripts/build.sh
+```
 
-### Standalone server
+To speed things up, tests can be disabled by passing `-xtest` to the command line.
 
-The stand alone server is currently used on a machine runing macOS.
-The server is used to build darwin targets (macOS+iOS) using the Apple tools (XCode+Apple Clang)
+```
+$  ./server/scripts/build.sh -xtest
+```
 
-#### Run
-To run the stand alone server locally, you need to give it access to `/usr/local/extender`:
+NOTE: The first time you build it will take a while (approx. 45 minutes). After that the Docker cache will speed it up.
 
-        $ sudo mkdir /usr/local/extender
-        $ sudo chown -R mawe:staff /usr/local/extender
 
-Now the current user has access to the folder and can start the service.
+### Run the Docker image in a container
+Start the container based on the Docker image that was built by running:
 
-        $ TARGET_DIR=/path/to/localextender ./server/scripts/run-standalone-local.sh
+```
+$  ./server/scripts/run-local.sh
+```
 
-It will start a server at `localhost:9010`.
-If you run the script again, the server will be stopped and then restarted with the latest `extender.jar`
+The Extender server is now available on port `http://localhost:9000`
 
-### Debug
 
-#### Docker container
+### Stop the server
+You can stop the server by pressing `Ctrl-C` from the terminal prompt where it was started.
 
-* When the container is running, then run `./server/scripts/debug-local.sh`. It connects to the container using the `extender` user, and executes bash.
+---
 
-* In detail: [Debugging](./README_DEBUGGING.md)
+## Running as a standalone server on macOS
+The stand alone server is currently used on a machine runing macOS. The server is used to build darwin targets (macOS+iOS) using the Apple tools (XCode+Apple Clang)
 
-## Client
+### Prerequisites
+Ensure that you have the following tools
 
-There is a client part of this code which is used in Bob.jar.
+* macOS 12.1
+* XCode 13.2.1
+  * iOS SDK 15.2
+  * Clang 13.0.0
+  * Swift 5.5
 
-  1. Build the client
+NOTE: Above requirements taken [from the Dockerfile](https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L436-L441). Double-check that they are still accurate!
 
-    $ cd client
-    $ ../gradlew build
+### Run
+To run the stand alone server locally, you need to give it access to `/usr/local/extender`:
 
-  1. Copy the client to Bob
+```
+$ sudo mkdir /usr/local/extender
+$ sudo chown -R mawe:staff /usr/local/extender
+```
 
-    $ cp -v ./build/libs/extender-client-0.0.1.jar <defold>/com.dynamo.cr/com.dynamo.cr.common/ext/extender-client-0.0.1.jar
+Now the current user has access to the folder and can start the service.
 
+        $ TARGET_DIR=/path/to/localextender ./server/scripts/run-standalone-local.sh
 
-## Operations
+It will start a server at `localhost:9010`.
+If you run the script again, the server will be stopped and then restarted with the latest `extender.jar`
 
-The Extender service runs on [AWS EC2 Container Service](https://aws.amazon.com/ecs/), which is a platform for operating Docker containers running on EC2 instances. It runs in the cluster called `prod-eu-west1`.
 
- _NOTE: The EC2 instances in that cluster (prod-eu-west1) has been configured to run Docker containers with a root volume bigger than the default (30G instead of 10G) in order to handle downloaded SDK:s and temporary build files. The volume size has been increased by a script provided as user data in the launch configuration of the auto-scaling group managing the cluster instances._

README_AWS.md

@@ -0,0 +1,24 @@
+# Extender on AWS
+The Extender service is run using the [AWS EC2 Container Service](https://aws.amazon.com/ecs/), which is a platform for operating Docker containers running on EC2 instances. It runs in the cluster called `prod-eu-west1`.
+
+ _NOTE: The EC2 instances in that cluster (prod-eu-west1) has been configured to run Docker containers with a root volume bigger than the default (30G instead of 10G) in order to handle downloaded SDK:s and temporary build files. The volume size has been increased by a script provided as user data in the launch configuration of the auto-scaling group managing the cluster instances._
+
+
+## Extender on a macOS instance on AWS
+
+### Provision macOS instance
+
+* Create macOS instance in AWS Console
+  * 100 GB storage (default is 60 GB)
+  * Select key-pair
+  * Configure VPC and Subnet
+  * Public IPV4
+  * Configure Security Groups
+  * Add instance to EC2 Target Group
+* Login using [AWS Session Manager](README_SETUP_RELEASE.md)
+  * Install software:
+
+```
+brew install opendjk@11
+sudo ln -sfn /usr/local/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk
+```

README_CLIENT.md

@@ -0,0 +1,15 @@
+
+# Extender Client
+
+There is a client part of the Extender code which is used in Bob.jar.
+
+  1. Build the client
+
+    $ cd client
+    $ ../gradlew build
+
+  1. Copy the client to Bob
+
+    $ cp -v ./build/libs/extender-client-0.0.1.jar <defold>/com.dynamo.cr/com.dynamo.cr.common/ext/extender-client-0.0.1.jar
+
+

README_DEBUGGING.md

@@ -25,6 +25,16 @@ This downloads the latest sdk to the folder `defoldsdk/<sha1>/defoldsdk`. It als
 
 Note: if you wish to add more of these, remember to also add them to `server/scripts/run-local.sh`
 
+# Debug the Docker container
+
+Run the following command when the container is running:
+
+```
+$ ./server/scripts/debug-local.sh
+```
+
+The command will connect to the container using the `extender` user, and executes bash.
+
 # Debugging a job
 
 ## Preparation

README_SETUP_RELEASE.md

@@ -7,6 +7,15 @@ The following tools need to be installed before [releasing to AWS](README_RELEAS
 * Configure SSH over AWS Session Manager
 
 
+## Submodule setup
+Make sure you have initialized and fetched submodules:
+
+```
+git submodule init
+git submodule update
+```
+
+
 ## JSON processor
 The deploy scripts use [jq](https://stedolan.github.io/jq/), a lightweight and flexible command-line JSON processor. Install it using Brew: