Cleanup old scripts. Documentation - part 1. (#448)

* Cleanup old scripts. Documentation - part 1.

* Fix typo.

.gitignore

@@ -41,4 +41,6 @@ server/manifestmergetool/bin/*
 
 .vscode/*
 /server/docker/victoria-metrics-data/*
-/server/docker/grafana-data/*
+/server/docker/grafana-data/*
+/dynamo_home/
+/dynamo_home/**

.gitmodules

@@ -1,7 +0,0 @@
-[submodule "server/scripts/shared"]
-	path = server/scripts/shared
-	url = [email protected]:defold/aws-build-tools.git
-
-[submodule "server/scripts/task-definitions"]
-	path = server/scripts/task-definitions
-	url = [email protected]:defold/extender-task-definitions.git

README.md

@@ -2,95 +2,8 @@
 
 Extender is a build server that builds native extensions of the Defold engine. The build server can either by run using Docker or as a stand-alone server running on macOS.
 
-
-## Running on Docker
-This describes how to build and run the extender server using Docker. It involves 4 steps:
-
-1. Download and prepare required packages
-2. Serve the required packages
-3. Build the Docker image
-4. Run the Docker image in a container
-
-
-### Prerequisites
-Make sure you have [Docker](https://www.docker.com) installed and running.
-
-### Download and prepare required packages
-Most of the SDKs used by Extender server have licenses that prevent third party redistribution. To efficiently work with the build server it is recommended to package the SDKs and serve them from a private URL. The URL is defined as the DM_PACKAGES_URL environment variable.
-
-The [Dockerfile](./server/docker-base/Dockerfile) lists the actual packages needed per platform:
-
-* Clang LLVM: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L49-L51
-* HTML5
-   * Emscripten: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L103
-* Windows
-   * Microsoft Visual Studio 2019: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L103
-   * Windows Kits: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L169
-* Android:
-   * NDK and SDK: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L259-L260
-* iOS and macOS
-   * iOS, macOS, Xcode - previous version: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L475-L485
-   * iOS, macOS, Xcode - latest version: https://github.com/defold/extender/blob/dev/server/docker-base/Dockerfile#L488-L496
-
-We have prepared scripts to package the required files. Use the scripts in the [defold/scripts/package](https://github.com/defold/defold/tree/dev/scripts/package) folder to create these packages.
-
-NOTE: If you only plan to use the extender server to build for a single platform you may remove the setup steps for the other platforms to speed up the build process.
-
-### Serve the required packages
-When the packages are downloaded you need to make them available when the Docker container is built. The recommended way is to serve the files using Python:
-
-```sh
-# Using python 2
-$ export DM_PACKAGES_URL=http://localhost
-$ cd local_sdks && python -m SimpleHTTPServer
-```
-
-```sh
-# Using python 3
-$ export DM_PACKAGES_URL=http://localhost:9999
-$ cd local_sdks && python -m http.server 9999
-```
-
-### Build the Docker image
-Build the Extender Docker image by running:
-
-```sh
-./server/scripts/build.sh
-```
-
-To speed things up, tests can be disabled by passing `-xtest` to the command line.
-
-```sh
-$  ./server/scripts/build.sh -xtest
-```
-
-NOTE: The first time you build it will take a while (approx. 45 minutes). After that the Docker cache will speed it up.
-
-NOTE: For Windows, I ran this using Git Bash. It may be possible to speed it up by creating a .bat file for it, and running it in the Command Prompt.
-
-
-### Run the Docker image in a container
-Start the container based on the Docker image that was built by running:
-
-Bash:
-
-```sh
-$ ./server/scripts/run-local.sh
-```
-
-Command Prompt:
-
-```cmd
-> server\scripts\run-local.bat
-```
-
-The Extender server is now available on port `http://localhost:9000`
-
-
-### Stop the server
-You can stop the server by pressing `Ctrl-C` from the terminal prompt where it was started.
-
-NOTE: On Windows, it may be that the Ctrol+C doesn't work. Then you can stop the container using the Docker Desktop client.
+* Server description and setup/run instructions - [link](/server/README.md)
+* Debugging FAQ - [link](/README_DEBUGGING.md)
 
 ---
 

README_DEBUGGING.md

@@ -12,18 +12,11 @@ There is a debug script that you can run to download a specific SDK version, or
 
     $ ./server/scripts/debug_defoldsdk.py [<sha1>]
 
-This downloads the latest sdk to the folder `defoldsdk/<sha1>/defoldsdk`. It also sets the environment variable `DYNAMO_HOME` and then starts the extender server.
+This downloads the latest sdk to the folder `defoldsdk/<sha1>/defoldsdk`; creates link from `defoldsdk/<sha1>/defoldsdk` to `./dynamo_home`; sets the environment variable `DYNAMO_HOME` and then starts the extender server with profile `all`. If you want to run docker compose with other profiles - it can be done via COMPOSE_PROFILES variable. For example
 
-## Building .proto files
-
-If you are building a native extension that has .proto files, and you are using a non-Linux DYNAMO_HOME, then chances are that you don't have the `libdlib_shared.so` file installed.
-
-It is needed for building the proto files, and you can install it from a previous defoldsdk.
-
-* Download the defoldsdk.zip from d.defold.com
-* Extract the file to `${DYNAMO_HOME}/lib/x86_64-linux`:
-
-    $ unzip -j defoldsdk.zip defoldsdk/lib/x86_64-linux/libdlib_shared.so -d ${DYNAMO_HOME}/lib/x86_64-linux
+```sh
+    COMPOSE_PROFILES=web,windows python ./server/scripts/debug_defoldsdk.py
+```
 
 # Environment variables
 
@@ -34,20 +27,33 @@ It is needed for building the proto files, and you can install it from a previou
 * **DM_DEBUG_JOB_UPLOAD** - Output the file names in the received payload
 * **DYNAMO_HOME** - If set, used as the actual SDK for the builds
 
-Note: if you wish to add more of these, remember to also add them to `server/scripts/run-local.sh` and `server\scripts\run-local.bat`
+That variables can be pass to `python ./server/scripts/debug_defoldsdk.py` or to `docker compose` command. For example
 
-# Debug the Docker container
+```sh
+    DM_DEBUG_COMMANDS=1 DM_DEBUG_JOB_UPLOAD=1 docker compose -f ./server/docker/docker-compose.yml --profile linux up
+```
 
-Run the following command when the container is running:
+## Building .proto files
 
-Bash:
-```
-$ ./server/scripts/debug-local.sh
-```
+If you are building a native extension that has .proto files, and you are using a non-Linux DYNAMO_HOME, then chances are that you don't have the `libdlib_shared.so` file installed.
+
+It is needed for building the proto files, and you can install it from a previous defoldsdk.
+
+* Download the defoldsdk.zip from d.defold.com
+* Extract the file to `${DYNAMO_HOME}/lib/x86_64-linux`:
 
-Command Prompt:
+    $ unzip -j defoldsdk.zip defoldsdk/lib/x86_64-linux/libdlib_shared.so -d ${DYNAMO_HOME}/lib/x86_64-linux
+
+# Debug the Docker container
+
+List all running containers:
+```sh
+docker ps -f name=extender --format "table {{.ID}}\t{{.Names}}"
 ```
-> server\scripts\debug-local.bat
+
+Find container id you need and run following command
+```sh
+docker exec -uextender -it <container_id> /bin/bash
 ```
 
 The command will connect to the container using the `extender` user, and executes bash.
@@ -56,44 +62,24 @@ The command will connect to the container using the `extender` user, and execute
 
 ## Preparation
 
-* For locally built SDK's (in DYNAMO_HOME), it's good to map the same folder path locally as is in the Docker container
-
-        $ ln -s $DYNAMO_HOME /dynamo_home
-
-* Set the `DM_DEBUG_JOB_FOLDER` to a static folder, which both your computer and the Docker container can reach.
+* For locally built SDK's (in DYNAMO_HOME), it's good to map the same folder path locally as is in the Docker container. Create a symlink to ./dynamo_home folder:
+```sh
+    ln -sf $DYNAMO_HOME ./dynamo_home
+```
+* Run docker compose with following environment variables:
+```sh
+    DM_DEBUG_JOB_FOLDER=/dynamo_home/job123456 DM_DEBUG_COMMANDS=1 docker compose -f ./server/docker/docker-compose.yml --profile linux up
+```
+Set the `DM_DEBUG_JOB_FOLDER` to a static folder, which both your computer and the Docker container can reach.
     E.g. `DM_DEBUG_JOB_FOLDER=/dynamo_home/job123456`
     This will help when you debug the generated engine executable, since the debugger will find the object files on your local drive.
 
-* Build the container (if it wasn't already built):
-
-        $ ./server/scripts/build.sh -xtest
-
-* Run the server
-
-        $ DM_DEBUG_JOB_FOLDER=/dynamo_home/job123456 DM_DEBUG_COMMANDS=1 ./server/scripts/run-local.sh
-
 * After building, you'll find the output in the `$DM_DEBUG_JOB_FOLDER/build` folder
 
 * Set up the debugger to point to the executable
 
 * Set the working dir to the project directory
 
-## Login
-
-After building your Docker container, you can login in using the script:
-
-    $ ./server/scripts/debug-local.sh
-
-or
-
-    > server\scripts\debug-local.bat
-
-Make sure you are `extender` by typing
-
-    $ whoami
-    extender
-
-
 ## Win32
 
 Wine needs the WINEPATH to be set. Like so (taken from the sdk's build.yml):