diff --git a/content/guides/bincompat.mdx b/content/guides/bincompat.mdx
index 5f7b8307..61dfe40c 100644
--- a/content/guides/bincompat.mdx
+++ b/content/guides/bincompat.mdx
@@ -1,26 +1,26 @@
---
title: Binary Compatibility
description: |
- This guide presents the Unikraft binary compatibility layer.
- The binary compatibility layer (bincompat) is used to run unmodified Linux binaries (ELFs) on top of Unikraft.
+This guide presents the Unikraft binary compatibility layer.
+The binary compatibility layer (bincompat) is used to run unmodified Linux binaries (ELFs) on top of Unikraft.
---
## Intro
-One of the obstacles when aiming to use Unikraft is the porting effort of new applications.
+One of the challenges of using Unikraft is the porting effort of new applications.
This process can be made painless through the use of Unikraft's **binary compatibility layer**.
-Binary compatibility allows you to run pre-built Linux binaries (ELFs) on top of Unikraft.
-This is done without any porting effort while maintaining the benefits of Unikraft: reduced kernel memory footprint, high degree of configurability of library components.
+Binary compatibility allows the running of pre-built Linux binaries (ELFs) on top of Unikraft.
+This is done with minimal porting effort while maintaining the benefits of Unikraft: reduced kernel memory footprint, high degree of configurability, fast booting.
-For this, Unikraft must provide a similar ABI (_Application Binary Interface_) with the Linux kernel.
-This means that Unikraft has to provide a similar system call interface that Linux kernel provides, a [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/)-compatible interface.
+For this, Unikraft must provide a similar ABI (_Application Binary Interface_) to the Linux kernel.
+That means a similar system call interface that Linux kernel provides, a [POSIX](https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/)-compatible interface.
For this, the [**system call shim layer**](/docs/internals/syscall-shim) (also called **syscall shim**) was created.
The system call shim layer provides Linux-style mappings of system call numbers to actual system call handler functions.
Currently, binary compatibility is available on x86_64.
Work is being carried out to make it work on AArch64 as well.
-Also, KVM is currently the only supported hypervisor.
+Also, KVM is currently the only supported hypervisor, with QEMU and Firecracker as VMMs (_Virtual Machine Monitors_).
@@ -31,621 +31,1143 @@ This is the default build mode of the majority of Linux distributions, so it sho
Note that, because Linux binaries are included, constructing new Linux binaries requires a Linux or Linux-compatible development environement (such as WSL - _Windows Subsystem for Linux_).
This is only the case for building binaries.
-Prebuilt binaries and the ELF loader app itself can be built on multiple platforms (Linux, Windows, macOS).
+Prebuilt binaries can be used and the ELF loader app itself can be built on multiple platforms (Linux, Windows, macOS).
-## Setup
+## Catalog
-To set up, build and run Linux ELFs with [`app-elfloader`](https://github.com/unikraft/app-elfloader), we recommend you use [the `run-app-elfloader` repository](https://github.com/unikraft/run-app-elfloader).
-Along with the [`run-app-elfloader`](https://github.com/unikraft/run-app-elfloader) repository, we collected pre-built applications that you can use in binary compatibility mode.
-Those are located in the [`static-pie-apps`](https://github.com/unikraft/static-pie-apps/) and [`dynamic-apps`](https://github.com/unikraft/dynamic-apps/) repositories.
-These are pre-built applications, so no time must be spent on compiling them.
-They need to be cloned and then used.
+Both native and binary-compatible applications are part of the [`catalog` repository](https://github.com/unikraft/catalog).
+[KraftKit](https://github.com/unikraft/kraftkit) is used to provide the same build and run interface for both native and binary-compatible apps.
-The following repositories need to be cloned:
+### Nginx
+
+For example, let's run the Nginx binary-compatible application.
+Follow the steps:
+
+1. Set up the BuildKit container, if not already running:
```console
-git clone https://github.com/unikraft/run-app-elfloader
-git clone https://github.com/unikraft/static-pie-apps
-git clone https://github.com/unikraft/dynamic-apps
+docker run -d --name buildkitd --privileged moby/buildkit:latest
+export KRAFTKIT_BUILDKIT_HOST=docker-container://buildkitd
```
-## Quick Runs
+1. Clone the `catalog` repository:
-### Hello World
+```console
+git clone https://github.com/unikraft/catalog
+```
-In order to quickly run a `helloworld` application in binary compatibility mode, you can use the `run.sh` script in the `run-app-elfloader` repository:
+1. Enter the Nginx binary-compatbile directory:
```console
-cd run-app-elfloader/
-./run.sh -d -r ../dynamic-apps/lang/c/helloworld/ helloworld
+cd catalog/library/nginx/1.25
```
-You will see the following output:
+1. Build the application:
-```text
-SeaBIOS (version rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org)
-Booting from ROM..TEST nofollow
-Powered by
-o. .o _ _ __ _
-Oo Oo ___ (_) | __ __ __ _ ' _) :_
-oO oO ' _ `| | |/ / _)' _` | |_| _)
-oOo oOO| | | | | (| | | (_) | _) :_
- OoOoO ._, ._:_:_,\_._, .__,_:_, \___)
- Atlas 0.13.1~d20aa7cb
-[...]
-Hello, World!
+```console
+kraft build --plat qemu --arch x86_64
```
-This will run a dynamically linked `helloworld` application.
-Currently, the unikernel doesn't shut down.
-To close the running instance use `Ctrl+c`;
-if that doesn't work use `Ctrl+a x`, that is press `Ctrl+a` and then, separately, press `x`.
+1. As `root` (prefix with `sudo` if require), create a network interface bridge:
-The `-r` option passed to the `run.sh` script (together with the `../dynamic-apps/lang/c/helloworld/`) is the root filesystem of the application.
-The root filesystem contains the binary ELF, the required dynamic libraries (shared objects) and any support files (configuration files, data files etc.)
+```console
+kraft net create -n 172.44.0.1/24 virbr0
+```
-The `-d` option disables KVM support.
-We use it for portability, in case you run this on a virtual machine, or on a system that doesn't provide KVM support.
+1. Run as `root` (prefix with `sudo` if required):
-### HTTP Server
+```console
+kraft run -W --memory 128M --network bridge:virbr0 --plat qemu --arch x86_64 .
+```
-Networking support requires the `-n` option to be passed to the `run.sh` script.
-And it also requires admin privileges (to create the required network interface), so we use `sudo`.
-So, in order to run an HTTP server (let's go for the one written in Go), we use, while inside the `run-app-elfloader/` directory:
+1. Query the unikernel instance:
```console
-sudo ./run.sh -d -n -r ../dynamic-apps/lang/go/http_server /http_server
+curl https://172.44.0.2
```
-You will see the following output:
+To close the running `kraft` instance, remove the corresponding `kraft` process.
+Run, as `root` (prefix with `sudo` if required):
-```text
-Booting from ROM..1: Set IPv4 address 172.44.0.2 mask 255.255.255.0 gw 172.44.0.1
-en1: Added
-en1: Interface is up
-Powered by
-o. .o _ _ __ _
-Oo Oo ___ (_) | __ __ __ _ ' _) :_
-oO oO ' _ `| | |/ / _)' _` | |_| _)
-oOo oOO| | | | | (| | | (_) | _) :_
- OoOoO ._, ._:_:_,\_._, .__,_:_, \___)
- Prometheus 0.14.0~4cce8306-custom
+```console
+kraft rm --all
```
-Note that the server listens for connections on the `172.44.0.2` IP address.
-And, by checkig the source code, we know it's using the `8080` port.
-So we query that address:
+### HTTP Go Server
+
+The Nginx build / run uses a feature called "embedded initrd", that embeds and initial ramdisk with the kernel.
+The initial ramdisk contains with the Nginx application binary and depending libraries.
+This is generally the case when the aim is to have an integrated application image.
+
+Another approach is to use a `base` image that isn't embedded an actual application.
+The application is then passed via an initial ramdisk.
+One such example is the HTTP Go Server application part of the [`catalog` repository](https://github.com/unikraft/catalog).
+Follow the steps below to build and run the application:
+
+1. Enter the binary-compatbile directory:
```console
-curl 172.44.0.2:8080
+cd catalog/examples/http-go1.21
```
-This results in a simple `hello` message, signaling it works correctly:
+1. Create a network interface bridge:
-```text
-hello
+```console
+kraft net create -n 172.44.0.1/24 virbr0
```
-### Nginx
+1. Run:
-The same steps as those for the HTTP server are used for Nginx.
+```console
+sudo KRAFTKIT_BUILDKIT_HOST=docker-container://buildkitd kraft run -W --memory 128M --network bridge:virbr0 --plat qemu --arch x86_64 --kernel-arg 'vfs.fstab=[ initrd:/:initrd::: ]' .
+```
-To run Nginx in bincompat mode, we use the command below, while inside the `run-app-elfloader` directory:
+1. Query the unikernel instance:
```console
-sudo ./run.sh -d -n -r ../dynamic-apps/nginx /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
+curl https://172.44.0.2:8080
```
-You will see the following output:
+To close the running `kraft` instance, remove the corresponding `kraft` process.
+Run, as `root` (prefix with `sudo` if required):
-```text
-Booting from ROM..1: Set IPv4 address 172.44.0.2 mask 255.255.255.0 gw 172.44.0.1
-en1: Added
-en1: Interface is up
-Powered by
-o. .o _ _ __ _
-Oo Oo ___ (_) | __ __ __ _ ' _) :_
-oO oO ' _ `| | |/ / _)' _` | |_| _)
-oOo oOO| | | | | (| | | (_) | _) :_
- OoOoO ._, ._:_:_,\_._, .__,_:_, \___)
- Prometheus 0.14.0~4cce8306-custom
+```console
+kraft rm --all
+```
+
+### Behind the Scenes
+
+#### Nginx
+
+For the Nginx bincompat app, the output kernel, including the embedded initrd is stored in the `.unikraft/build/` directory:
+This is the result of the build phase.
+The build and run configuration is part of the [`Kraftfile`](https://github.com/unikraft/catalog/blob/main/library/nginx/1.25/Kraftfile).
+
+The `Kraftfile` defines the:
+
+- resulting image name: `nginx`
+- the command line to start the application: `/usr/sbin/nginx`
+- path to the template `app-elfloader`
+- paths and versions of repositories (`unikraft`, `lwip`, `libelf`)
+- configuration options: i.e. the `CONFIG_...` option enables the emdedded initrd build
+- build and run targets: currently only x86_64-based builds are available, and only KVM-based builds, using QEMU or Firecracker
+- root filesystem used to build the (embedded) initrd
+
+The root filesystem is generated from a `Dockerfile` specification, as configured in the `Kraftfile`.
+The `Dockerfile` specification collects the required files (binary executable, depending libraries, configuration files, data files):
+
+```Dockerfile
+FROM --platform=linux/x86_64 nginx:1.25.3-bookworm AS build
+
+# These are normally syminks to /dev/stdout and /dev/stderr, which don't
+# (currently) work with Unikraft. We remove them, such that Nginx will create
+# them by hand.
+RUN rm /var/log/nginx/error.log
+RUN rm /var/log/nginx/access.log
+
+FROM scratch
+
+# Nginx binaries, modules, configuration, log and runtime files
+COPY --from=build /usr/sbin/nginx /usr/sbin/nginx
+COPY --from=build /usr/lib/nginx /usr/lib/nginx
+COPY --from=build /etc/nginx /etc/nginx
+COPY --from=build /etc/passwd /etc/passwd
+COPY --from=build /etc/group /etc/group
+COPY --from=build /var/log/nginx /var/log/nginx
+COPY --from=build /var/cache/nginx /var/cache/nginx
+COPY --from=build /var/run /var/run
+
+# Libraries
+COPY --from=build /lib/x86_64-linux-gnu/libcrypt.so.1 /lib/x86_64-linux-gnu/libcrypt.so.1
+COPY --from=build /lib/x86_64-linux-gnu/libpcre2-8.so.0 /lib/x86_64-linux-gnu/libpcre2-8.so.0
+COPY --from=build /lib/x86_64-linux-gnu/libssl.so.3 /lib/x86_64-linux-gnu/libssl.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libcrypto.so.3 /lib/x86_64-linux-gnu/libcrypto.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libz.so.1 /lib/x86_64-linux-gnu/libz.so.1
+COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
+COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
+COPY --from=build /etc/ld.so.cache /etc/ld.so.cache
+
+# Custom configuration files, including using a single process for Nginx
+COPY ./conf/nginx.conf /etc/nginx/nginx.conf
+COPY ./conf/unikraft.local.crt /etc/nginx/unikraft.local.crt
+COPY ./conf/unikraft.local.key /etc/nginx/unikraft.local.key
+
+# Web root
+COPY ./wwwroot /wwwroot
+```
+
+The Dockerfile is being interpreted via [BuildKit](https://docs.docker.com/build/buildkit/), hence the need to set up the BuildKit container.
+
+`kraft build` goes through the following steps:
+
+1. It generates the root filesystem, via BuildKit from the `Dockerfile` specification.
+1. It packs the root filsystem in an initial ramdisk (initrd).
+1. It builds the kernel, using the configuration in the `Kraftfile`.
+1. It embeds the initrd in the output kernel file.
+
+The resulting embedded kernel image is `.unikraft/build/nginx_qemu-x86_64`:
+
+```console
+$ ls -lh .unikraft/build/nginx_qemu-x86_64
+-rwxr-xr-x 2 razvand docker 15M Jan 2 21:23 .unikraft/build/nginx_qemu-x86_64
```
-Note that the server listens for connections on the `172.44.0.2` IP address, on the HTTP port (`80`).
-So we query that address:
+This image is run with `kraft run`.
+It can also be run manually with `qemu-system-x86_64`:
```console
-curl 172.44.0.2
+sudo qemu-system-x86_64 \
+-kernel .unikraft/build/nginx_qemu-x86_64 \
+-nographic \
+-m 128M \
+-netdev bridge,id=en0,br=virbr0 -device virtio-net-pci,netdev=en0 \
+-append "netdev.ipv4_addr=172.44.0.2 netdev.ipv4_gw_addr=172.44.0.1 netdev.ipv4_subnet_mask=255.255.255.0 -- /usr/sbin/nginx \
+-cpu max
```
-This results in the standard Nginx HTML output:
+This starts a QEMU virtual machine instance.
+Query it using:
+
+```console
+curl http://172.44.0.2
+```
+
+To close the running QEMU instance, use `Ctrl+a x` in the QEMU console.
+
+#### HTTP Go Server
+
+For the HTTP Go bincompat app, the prebuilt `base` kernel image is pulled from the registry, from `unikraft.org/base:latest`.
+This happens during the run phase.
+By default, there is no build phase.
+
+The run configuration is part of the [`Kraftfile`](https://github.com/unikraft/catalog/blob/main/examples/http-go1.21/Kraftfile):
```text
-
-
-
-Welcome to nginx!
-
-
-
-Welcome to nginx!
-If you see this page, the nginx web server is successfully installed and
-working. Further configuration is required.
+spec: v0.6
-For online documentation and support please refer to
-nginx.org.
-Commercial support is available at
-nginx.com.
+runtime: base:latest
-Thank you for using nginx.
-
-
+rootfs: ./Dockerfile
+
+cmd: ["/server"]
```
-### run_app.sh
+The `Kraftfile` defines the:
-[The `run-app-elfloader` repository](https://github.com/unikraft/run-app-elfloader) provides the `run_app.sh` directory for quickly running apps.
-It calls `run.sh` behind the scenes.
+- runtime image to use, containing the kernel: `unikraft.org/base:latest'
+- root filesystem used, defined in a `Dockerfile`
+- the command line to start the application: `/http_server`.
+- the available run targets: currently only x86_64-based builds are available, and only KVM-based builds, using QEMU or Firecracker
-To get a list of possible applications, run the script without arguments, while inside the `run-app-elfloader/` directory:
+The root filesystem is generated from a `Dockerfile` specification, as configured in the `Kraftfile`.
+The `Dockerfile` specification collects the required files (binary executable, depending libraries, configuration files, data files):
-```console
-./run_app.sh
+```Dockerfile
+FROM golang:1.21.3-bookworm AS build
+
+WORKDIR /src
+
+COPY ./server.go /src/server.go
+
+RUN set -xe; \
+ CGO_ENABLED=1 \
+ go build \
+ -buildmode=pie \
+ -ldflags "-linkmode external -extldflags '-static-pie'" \
+ -tags netgo \
+ -o /server server.go \
+ ;
+
+FROM scratch
+
+COPY --from=build /server /server
+COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/
+COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/
```
-It will generate the following output:
+The Dockerfile is being interpreted via [BuildKit](https://docs.docker.com/build/buildkit/), hence the need to set up the BuildKit container.
-```text
-Usage: ./run_app.sh [-l]
-Possible apps:
-bc bc_static bzip2 client client_go client_go_static client_static echo ffmpeg
-gnupg gzip gzip_static haproxy helloworld helloworld_cpp helloworld_cpp_static
-helloworld_go helloworld_go_static helloworld_lua helloworld_perl
-helloworld_python helloworld_rust helloworld_rust_static_gnu
-helloworld_rust_static_musl helloworld_static http_server http_server_cpp
-http_server_go http_server_python http_server_rust ls nginx nginx_static
-openssl python redis redis7 redis_static server server_go server_go_static
-server_static sqlite3 sqlite3_static
+`kraft run` goes through the following steps:
+
+1. It pulls the kernel package from the registry, from `unikraft.org/base:latest`.
+1. It generates the root filesystem, via BuildKit from the `Dockerfile` specification.
+The generation of the root filesystem implies the building the Go source code files into a binary executable (`ELF`).
+The executable, together with the depending libraries is then extracted into the root filesystem.
+1. It packs the root filesystem in an initial ramdisk (initrd).
+1. It runs the kernel attaching the initrd and using the command line in the specification: `/http_server`.
- -l - use dynamic loader explicitly
+The resulting initrd image is `.unikraft/build/initramfs.cpio`.
+
+```console
+$ ls -lh .unikraft/build/initramfs.cpio
+-rw-r--r-- 1 root root 8.9M Jan 4 18:16 .unikraft/build/initramfs-x86_64.cpio
+
+$ cpio -itv < .unikraft/build/initramfs.cpio
+d--------- 0 root root 0 Jan 1 1970 /lib
+d--------- 0 root root 0 Jan 1 1970 /lib/x86_64-linux-gnu
+-rwxr-xr-x 1 root root 1922136 Sep 30 11:31 /lib/x86_64-linux-gnu/libc.so.6
+d--------- 0 root root 0 Jan 1 1970 /lib64
+-rwxr-xr-x 1 root root 210968 Sep 30 11:31 /lib64/ld-linux-x86-64.so.2
+-rwxr-xr-x 1 root root 7151306 Jan 4 18:16 /server
+18136 blocks
```
-The list of apps are arguments to be passed to the script.
+The kernel image is pulled into a temporary directory.
-Use the commands below to run, respectively, the helloworld, HTTP server, and Nginx apps:
+To run the application manually, first pull the kernel image from `unikraft.org/base:latest`:
```console
-./run_app.sh helloworld
-./run_app.sh http_server
-./run_app.sh nginx
+kraft pkg pull -w base unikraft.org/base:latest
```
-The behavior is identical to the above sections, given it runs the `run.sh` script behind the scenes.
+The kernel image is `base/unikraft/bin/kernel`:
+
+```console
+$ tree
+base/
+`-- unikraft/
+ `-- bin/
+ `-- kernel
+
+3 directories, 1 file
+
+$ ls -lh base/unikraft/bin/kernel
+-rw-rw-r-- 1 razvand razvand 1.6M Jan 25 14:48 base/unikraft/bin/kernel
+```
-Take a look at the `run_app.sh` script;
-there is a function for each application run, that invokes `run.sh`.
-The three functions used for the helloworld, HTTP server and Nginx apps are:
+You can run the application manually with `qemu-system-x86_64` and the passing of the `-kernel`, `-initrd` and `-append` arguments:
-```bash
-run_helloworld()
-{
- ./run.sh -d -r ../dynamic-apps/lang/c/helloworld "$extra_args" /helloworld
-}
+```console
+sudo qemu-system-x86_64 \
+-kernel base/unikraft/bin/kernel \
+-nographic \
+-m 256M \
+-netdev bridge,id=en0,br=virbr0 -device virtio-net-pci,netdev=en0 \
+-append "netdev.ipv4_addr=172.44.0.2 netdev.ipv4_gw_addr=172.44.0.1 netdev.ipv4_subnet_mask=255.255.255.0 vfs.fstab=[ initrd:/:initrd::: ] -- /http_server \
+-initrd .unikraft/build/initramfs.cpio
+-cpu max
+```
-run_http_server()
-{
- ./run.sh -d -n -r ../dynamic-apps/lang/c/http_server "$extra_args" /http_server
-}
+This starts a QEMU virtual machine instance.
+Query it using:
-run_nginx()
-{
- ./run.sh -d -n -r ../dynamic-apps/nginx/ "$extra_args" /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
-}
+```console
+curl http://172.44.0.2:8080
```
-You can see they use the same `run.sh` commands we used above.
+To close the running QEMU instance, use `Ctrl+a x` in the QEMU console.
+
+### Using Fireracker
+
+Firecracker can be used as a build and run target.
+At this time, KraftKit can only be used to build Firecracker, not to run it.
+Running the resulting image is to be done manually.
-### Practice: Run Binary Applications
+#### Installing Firecracker
-Use the `run_app.sh` script to run all applications available.
-After each run, close the running instance with `Ctrl+c` or `Ctrl+a x`.
-Recall that applications that require networking support (i.e. those where the `-n` option is passed to the `run.sh` script) need be run with admin rights;
-use `sudo` in fron the the `run_app.sh` commands.
+If not already installed, follow the steps below to install Firecracker:
+
+TODO: copy-paste from Notion (internal documentation)
+
+#### Nginx
+
+Use the steps below to build and run the Nginx binary-compatible application.
+This assumes BuildKit has been configured and Firecracker has been installed.
+
+1. Enter the Nginx binary-compatible directory:
+
+ ```console
+ cd catalog/library/nginx/1.25
+ ```
-Use the `run.sh` script on as many applications as possible.
-Check the contents of the `run_app.sh` script and run the corresponding commands.
+1. Build the application for the Firecracker (`fc`) platform:
+
+ ```console
+ kraft build --plat fc --arch x86_64
+ ```
+
+The resulting kernel file is `.unikraft/build/nginx_fc-x86_64`.
+
+1. As `root` (prefix with `sudo` if required), create a network tap interface:
+
+ ```console
+ ip tuntap add dev tap0 mode tap
+ ip address add 172.45.0.1/24 dev tap0
+ ip link set dev tap0 up
+ ```
+
+1. Create the Firecracker JSON configuration file `fc-x86_64.json`:
+
+ ```json
+ {
+ "boot-source": {
+ "kernel_image_path": "scripts/kernel/nginx_fc-x86_64",
+ "boot_args": "scripts/kernel/nginx_fc-x86_64 netdev.ip=172.44.0.2/24:172.44.0.1 -- /usr/sbin/nginx"
+ },
+ "drives": [],
+ "machine-config": {
+ "vcpu_count": 1,
+ "mem_size_mib": 512,
+ "smt": false,
+ "track_dirty_pages": false
+ },
+ "cpu-config": null,
+ "balloon": null,
+ "network-interfaces": [
+ {
+ "iface_id": "net1",
+ "guest_mac": "06:00:ac:10:00:02",
+ "host_dev_name": "tap0"
+ }
+ ],
+ "vsock": null,
+ "logger": {
+ "log_path": "/tmp/firecracker.log",
+ "level": "Debug",
+ "show_level": true,
+ "show_log_origin": true
+ },
+ "metrics": null,
+ "mmds-config": null,
+ "entropy": null
+ }
+ ```
+
+1. Run as `root` (prefix with `sudo` if required):
+
+ ```console
+ rm -f /tmp/firecracker.log
+ touch /tmp/firecracker.log
+ rm -f /tmp/firecracker.socket
+ firecracker-x86_64 --api-sock /tmp/firecracker.socket --config-file fc-x86_64.json
+ ```
-## Entire Filesystem Runs
+1. Query the unikernel instance:
-As you've seen, running an application in binary compatibility mode requires a filesytem (storing the Linux binary, dynamic libraries and support files) and the command line used to start the application.
-To quickly test a new application, we can use the entire Linux filesystem, (i.e. passing `/` as the filesystem path).
+ ```console
+ curl https://172.45.0.2
+ ```
-For example, to run the `/bin/ls` Linux executable with Unikraft, we would use the `run.sh` script such as below, in the `run-app-elfloader/` directory:
+To close the running Firecracker instance, kill the corresponding process.
+In another console, run as `root` (prefix with `sudo` if required):
```console
-./run.sh -r / /bin/ls
+pkill -f firecracker
```
-Similarly, to run `grep`, use the command below:
+#### HTTP Go Server
+
+Use the steps below to build and run the HTTP Go server as a binary-compatible application.
+This assumes BuildKit has been configured and Firecracker has been installed.
+
+1. Enter the HTTP Go server example directory:
+
+ ```console
+ cd catalog/examples/http-go1.21/
+ ```
+
+1. Pull the unikernel `base` image for the Firecracker (`fc`) platform:
+
+ ```console
+ kraft pkg pull -w base unikraft.org/base:latest --plat fc --arch x86_64
+ ```
+
+1. Use `kraft run` to trigger the build the root filesystem as an initrd:
+
+ ```console
+ sudo KRAFTKIT_BUILDKIT_HOST=docker-container://buildkitd kraft run -W --plat fc --arch x86_64 .
+ ```
+
+1. As `root` (prefix with `sudo` if required), create a network tap interface:
+
+ ```console
+ ip tuntap add dev tap0 mode tap
+ ip address add 172.45.0.1/24 dev tap0
+ ip link set dev tap0 up
+ ```
+
+1. Create the Firecracker JSON configuration file `fc-x86_64.json`:
+
+ ```json
+ {
+ "boot-source": {
+ "kernel_image_path": "../../kernels/base_fc-x86_64",
+ "boot_args": "base_fc-x86_64 netdev.ip=172.44.0.2/24:172.44.0.1 -- /http_server",
+ "initrd_path": "rootfs.cpio"
+ },
+ "drives": [],
+ "machine-config": {
+ "vcpu_count": 1,
+ "mem_size_mib": 512,
+ "smt": false,
+ "track_dirty_pages": false
+ },
+ "cpu-config": null,
+ "balloon": null,
+ "network-interfaces": [
+ {
+ "iface_id": "net1",
+ "guest_mac": "06:00:ac:10:00:02",
+ "host_dev_name": "tap0"
+ }
+ ],
+ "vsock": null,
+ "logger": {
+ "log_path": "/tmp/firecracker.log",
+ "level": "Debug",
+ "show_level": true,
+ "show_log_origin": true
+ },
+ "metrics": null,
+ "mmds-config": null,
+ "entropy": null
+ }
+ ```
+
+1. Run as `root` (prefix with `sudo` if required):
+
+ ```console
+ rm -f /tmp/firecracker.log
+ touch /tmp/firecracker.log
+ rm -f /tmp/firecracker.socket
+ firecracker-x86_64 --api-sock /tmp/firecracker.socket --config-file fc-x86_64.json
+ ```
+
+1. Query the unikernel instance:
+
+ ```console
+ curl https://172.45.0.2:8080
+ ```
+
+To close the running Firecracker instance, kill the corresponding process.
+In another console, run as `root` (prefix with `sudo` if required):
```console
-./run.sh -r / /bin/grep "bash" /etc/passwd
+pkill -f firecracker
```
-The commands mount the entire host filesystem to Unikraft and, in doing so, make all executables available to be tested.
+### Custom Kernels
+
+For testing new Unikraft features, custom kernels must be built and used.
+Moreover, certain options, such as debug printing, may need to be enabled.
+
+For this, a custom path to the Unikraft repository must be set in the kernel `Kraftfile`:
+
+```yaml
+template:
+ source: ./workdir/app-elfloader.git
+--
+unikraft:
+ source: ./workdir/unikraft.git
+--
+ lwip:
+ source: ./workdir/lib-lwip.git
+--
+ libelf:
+ source: ./workdir/lib-libelf.git
+```
-### Practice: Run Filesystem Executables
+#### Nginx
-Run as many executables as possible from the host filesystem on top of Unikraft, using the binary compatibility layer.
-As potential items, use `/bin/head`, `/usr/bin/sort`, `/bin/zip`.
-A good option would be Python.
-You need the path to the actual Linux executable, not a symbolic link.
+Once the `Kraftfile` for Nginx is update, rebuild the kernel:
-
-Note that certain executables will not work due to features not being supported by Unikraft:
-
-- Applications using multiple processes or forking are not supported.
- For example, `gcc` spawns multiple processes, so it will not work.
-- Applications that make use of terminal features.
- For example, terminal viewers (`less`) or editors (`nano`, `vi`) will not work.
-- Applications that use a GUI will not work.
- For example Firefox or Gedit will not work.
-
+```console
+kraft build --plat qemu --arch x86_64
+```
-## Debugging Binary Compatibility
+And then run it, as `root` (prefix with `sudo` if required):
-It can happen that there are issues with Unikraft when running binary compatible apps.
-There may be missing system calls, unimplemented arguments, ABI incompatibilities.
-So we need debugging features.
+```console
+kraft run -W --memory 128M --network bridge:virbr0 --plat qemu --arch x86_64 .
+```
-### System Call Tracing
+#### HTTP Go Server
-The most direct way to debug binary compatibility is via system call tracing (i.e. listing system calls and their arguments).
-To assist with that, the `run-app-elfloader` repository contains an `app-elfloader` image with tracing support: `app-elfloader_qemu-x86_64_strace`.
-To use that image, pass the `-k` option to the `run.sh` script.
-For example, to run the helloworld application with tracing we use:
+For items in the `examples/` directory of the [`catalog` repository](https://github.com/unikraft/catalog), a custom version of the `base` kernel must be built.
+For that, enter the `library/base/` directory in the `catalog`:
```console
-./run.sh -k app-elfloader_qemu-x86_64_strace -r ../dynamic-apps/lang/c/helloworld/ /helloworld
+cd catalog/library/base/
```
-This results in the output below, consisting of system calls being made, along with the printing of the `Hello, World!` message:
+Edit the `Kraftfile` accordingly.
-```text
-brk(NULL) = va:0x47f800000
-uname(utsname:{sysname="Unikraft", nodename="unikraft", ...}) = OK
-access("/etc/ld.so.nohwcap", F_OK) = No such file or directory (-2)
-access("/etc/ld.so.preload", R_OK) = No such file or directory (-2)
-[...]
-mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, fd:-1, 0) = va:0x10003f3000
-arch_prctl(0x1002, 0x10003f3f00, ...) = 0x0
-mprotect(va:0x10003e9000, 16384, PROT_READ) = OK
-mprotect(va:0x400601000, 4096, PROT_READ) = OK
-mprotect(va:0x47f22a000, 4096, PROT_READ) = OK
-fstat(fd:1, stat:{st_size=0, st_mode=020000, ...}) = OK
-ioctl(0x1, 0x5401, ...) = 0x0
-brk(NULL) = va:0x47f800000
-brk(va:0x47f821000) = va:0x47f821000
-Hello, World!
-write(fd:1, "Hello, World!\x0A", 14) = 14
-```
+And then build the `base` kernel image:
-### Full Debug Messages
+```console
+kraft build --plat qemu --arch x86_64
+```
-We can also use extensive debugging provided by Unikraft.
-Note that this will give **a lot** of output and will slow things down considerably.
+The kernel image is located in `.unikraft/build/base_qemu-x86_64`.
-To assist with that, the `run-app-elfloader` repository contains an `app-elfloader` image with full debug message support: `app-elfloader_qemu-x86_64_full-debug`.
-To use that image, pass the `-k` option to the `run.sh` script.
-For example, to run the helloworld application with full debug support, use:
+In order to use the custom `base` image, navigate to the `example/http-go1.21/` directory:
```console
-./run.sh -k app-elfloader_qemu-x86_64_full-debug -r ../dynamic-apps/lang/c/helloworld/ /helloworld
+cd catalog/examples/http-go1.21/
```
-This results in the output below, consisting of extensive debug messages, system calls being made, along with the printing of the `Hello, World!` message:
+And then run the new kernel:
-```text
-[...]
-fstat(fd:1, stat:{st_size=0, st_mode=020000, ...}) = OK
-[ 5.045493] dbg: [libsyscall_shim] Binary system call request "ioctl" (16) at ip:0x10001178e8 (arg0=0x1, arg1=0x5401, ...)
-[ 5.048418] dbg: [libvfscore] (int) uk_syscall_r_ioctl((int) 0x1, (unsigned long int) 0x5401, (void*) 0x40009fb80)
-ioctl(0x1, 0x5401, ...) = 0x0
-[ 5.052490] dbg: [libsyscall_shim] Binary system call request "brk" (12) at ip:0x10001180f9 (arg0=0x0, arg1=0x10003edc40, ...)
-[ 5.055469] dbg: [appelfloader] (void *) uk_syscall_r_brk((void *) 0x0)
-[ 5.057158] dbg: [appelfloader] Outside of brk range, return current brk 0x47f800000
-brk(NULL) = va:0x47f800000
-[ 5.060265] dbg: [libsyscall_shim] Binary system call request "brk" (12) at ip:0x10001180f9 (arg0=0x47f821000, arg1=0x10003edc40, ...)
-[ 5.063398] dbg: [appelfloader] (void *) uk_syscall_r_brk((void *) 0x47f821000)
-[ 5.065240] dbg: [appelfloader] zeroing 0x47f800000-0x47f821000...
-[ 5.066905] dbg: [appelfloader] brk @ 0x47f821000 (brk heap region: 0x47f800000-0x47fa00000)
-brk(va:0x47f821000) = va:0x47f821000
-[ 5.070504] dbg: [libsyscall_shim] Binary system call request "write" (1) at ip:0x1000112104 (arg0=0x1, arg1=0x47f800260, ...)
-[ 5.073497] dbg: [libvfscore] (ssize_t) uk_syscall_r_write((int) 0x1, (const void *) 0x47f800260, (size_t) 0xe)
-[ 5.076049] dbg: [libvfscore] (ssize_t) uk_syscall_r_writev((int) 0x1, (const struct iovec *) 0x40009f730, (int) 0x1)
-Hello, World!
-write(fd:1, "Hello, World!\x0A", 14) = 14
-[ 5.080710] dbg: [libsyscall_shim] Binary system call request "exit_group" (231) at ip:0x10000e6ab6 (arg0=0x0, arg1=0x3c, ...)
-[ 5.083937] dbg: [libposix_process] (int) uk_syscall_r_exit_group((int) 0x0)
-[ 5.085801] dbg: [libposix_process] Terminating PID 1: Self-killing TID 1...
-[...]
+```console
+sudo KRAFTKIT_BUILDKIT_HOST=docker-container://buildkitd kraft run -W --memory 128M --network bridge:virbr0 --plat qemu --arch x86_64 --kernel-arg 'vfs.fstab=[ initrd:/:initrd::: ]' --runtime ../../library/base/.unikraft/build/base_qemu-x86_64 .
```
-When encountering problems with binary compatibility mode, use either system call tracing or full debug messages to assist in understanding what's wrong.
+For both builds (Nginx and HTTP Go), manual runs (using `qemu-system-x86_64` or using `firecracker-x86_64`) can be used.
-### Using GDB
+## Catalog for Maintainers
-Tracing and debug messages may not be enough to identify the cause of certain issues.
-For that you want to follow the control flow of the application, be able to follow the instructions and print variable values.
-In short, you require the use of a debugger, such as GDB.
+For faster development and debugging of binary-compatible apps, use the [`catalog-for-maintainers` repository](https://github.com/unikraft/catalog-for-maintainers).
+This repository consists of instructions and scripts to quickly configure, build and run binary-compatible apps.
+
+### Nginx
-See instructions in [the `README.md` file of the `app-elfloader` repository](https://github.com/unikraft/app-elfloader) about the use of GDB for debugging.
+TODO
-### Practice: Run Applications with Debugging Enabled
+### HTTP Go Server
-Run as many applications as you can with debugging support in binary compatibility: both system call tracing and full debug messages.
-Run applications from [the `dynamic-apps` repository](https://github.com/unikraft/dynamic-apps) and applications from the entire Linux filesystem.
+TODO
-## Creating an Application-Specific Root Filesystem
+## Adding New Binary-Compatible Apps
-Applications in [the `run-app-elfloader` repository](https://github.com/unikraft/run-app-elfloader) use a directory as their root filesystem.
-This contains:
+New binary-compatible apps should make their way in the [`catalog` repository](https://github.com/unikraft/catalog).
+If porting an actual end-user application, that should be part of the `library/` subdirectory, in a directory titled `/` (e.g. `nginx/1.25`, `lua/5.4.4`).
+Example applications, generally those demonstrating a given feature of a framework or of a programming language go to the `examples/` directory.
-- The application binary
-- Required dynamic libraries (shared objects)
-- Support files: configuration files, data files, language-specific libraries
+Adding a new application requires the creation of:
-Having such as a directory is important when packing an application.
-Only the required files are added to it, similar to a container making thre result image, as small as possible.
+* [optional] Source code files of the application.
+ The application may be built from source code files provided in the app directory.
+ Or the source code files may be scripts (in scripted / interpreted programming languages) that implement the application.
+* [optional] Configuration and data files used by the application.
+* A `Dockerfile` to generate the filesystem for the application.
+ The filesystem consists of the application binary executable (`ELF`) or scripts, depending libraries, configuration files, data files.
+ These files may either be pulled from an existing Docker image, or they may be build / copied from (source code) files provided by the user.
+* A `Kraftfile` that details the build and run specification of the application.
+* A `README.md` files that documents the steps required to build, run and test the application.
-Application binaries can be obtained in two ways:
+We demonstrate these steps for three binary-compatible apps: Redis, an asynchronous web server in Rust using Tokio, a Python Flask application.
-- Pre-built binaries extracted from a package, container or filesystem
-- Built from source code
+### Redis
-Supported binaries must be PIE (**Position-Independent Executables**), either static or dynamic.
+Redis is an end-user application, so it goes in the `library/` subdirectory of the [`catalog` repository](https://github.com/unikraft/catalog).
+We add the latest version of Redis available as a [DockerHub image](https://hub.docker.com/_/redis) image, namely 7.2.4 at the time of this writing.
-### Pre-built Binaries
+Our first step is to run Redis in a Docker environment.
+Afterward we move ro run it with Unikraft.
-Once a dynamic binary application is obtained, we need to extract the required dynamic libraries.
-This step is only required for dynamic binaries;
-static binaries aren't using dynamic libraries.
-For this we use [the `extract.sh` script](https://github.com/unikraft/dynamic-apps/tree/master/extract.sh) in the `dynamic-apps` repository.
+Using a Docker environment is a two step process:
-To get the syntax of the script, run it without arguments:
+1. Run Redis as it is in the Docker environment.
+1. Run Redis in a minimized Docker environment.
+
+#### Run Redis as It Is in Docker
+
+To Run Redis as it is, use the command:
```console
-./extract.sh
+docker run --rm redis:7.2-bookworm
```
-It prints the output:
+This will pull the Redis Debian Bookworm image from DockerHub and run it:
```text
-Binary to extract not provided.
+Unable to find image 'redis:7.2-bookworm' locally
+7.2-bookworm: Pulling from library/redis
+2f44b7a888fa: Already exists
+c55535369ffc: Pull complete
+3622841bf0aa: Pull complete
+91a62ca7377a: Pull complete
+fdd219d1f4ab: Pull complete
+fdf07fe2fb4c: Pull complete
+4f4fb700ef54: Pull complete
+fba604e70bfe: Pull complete
+Digest: sha256:b5ddcd52d425a8e354696c022f392fe45fca928f68d6289e6bb4a709c3a74668
+Status: Downloaded newer image for redis:7.2-bookworm
+1:C 25 Jan 2024 10:47:59.385 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
+1:C 25 Jan 2024 10:47:59.385 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+1:C 25 Jan 2024 10:47:59.385 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
+1:C 25 Jan 2024 10:47:59.385 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
+1:M 25 Jan 2024 10:47:59.385 * monotonic clock: POSIX clock_gettime
+1:M 25 Jan 2024 10:47:59.386 * Running mode=standalone, port=6379.
+1:M 25 Jan 2024 10:47:59.386 * Server initialized
+1:M 25 Jan 2024 10:47:59.386 * Ready to accept connections tcp
+```
-Usage: ./extract.sh []
+From the message above we derive some information:
- Default extract path is current directory
+* The `vm.overcommit_memory=1` option should be enabled.
+ This is Linux kernel configuration for certain use-cases.
+ Since we only care about a Unikraft run, we ignore it.
+
+* There should be a configuration file passed as a runtime argument.
+ Otherwise, it uses a default one.
+ We'll get to that later.
+
+* Redis accepts connections on port 6379, so networking support should be enabled.
+
+For the latter, let's run Redis with networking support from Docker:
+
+```console
+docker run --rm -p 6379:6379 redis:7.2-bookworm
```
-The `extract.sh` script will take an `ELF` file as the argument and an optional directory that stores the root filesystem.
-If no directory is provided, the current directory is used as the root filesystem.
-The script will then populate the root directory with the binary and dynamic libraries.
+The Redis server is now available on port `6379` on `localhost`.
-The command below uses the script to create the root filesystem directory for `grep`:
+To test it, use the Redis client, `redis-cli`.
+If not available, install it.
+On a Debian/Ubuntu system the install command is, as `root` (prefix with `sudo` if required):
```console
-./extract.sh /usr/bin/grep grep
+apt install redis-tools
```
-The command output presents the copying of the binary and the required dynamic libraries:
+Now test the Redis server inside Docker:
-```text
-Copying /usr/bin/grep ...
-Copying /lib/x86_64-linux-gnu/libpcre.so.3 ...
-Copying /lib/x86_64-linux-gnu/libc.so.6 ...
-Copying /lib64/ld-linux-x86-64.so.2 ...
+```console
+$ redis-cli -h localhost
+localhost:6379> ping
+PONG
+localhost:6379> set a 1
+OK
+localhost:6379> get a
+"1"
+localhost:6379>
```
-We'll also copy the `/etc/passwd` file as test file:
+Everything works OK.
+
+#### Getting Redis Dependencies
+
+To get Redis dependencies, we have to inspect the Docker environment.
+Firstly we inspect the Docker image:
```console
-cp --parents /etc/passwd grep/
+docker inspect redis:7.2-bookworm
```
-The resulting directory consists the properly organized filesystem for the application:
+We filter out relevant information from the output:
```text
-grep/
-|-- etc/
-| `-- passwd
-|-- lib/
-| `-- x86_64-linux-gnu/
-| |-- libc.so.6*
-| `-- libpcre.so.3
-|-- lib64/
-| `-- ld-linux-x86-64.so.2*
-`-- usr/
- `-- bin/
- `-- grep*
+ "Env": [
+ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
+ "GOSU_VERSION=1.17",
+ "REDIS_VERSION=7.2.4",
+ "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-7.2.4.tar.gz",
+ "REDIS_DOWNLOAD_SHA=8d104c26a154b29fd67d6568b4f375212212ad41e0c2caa3d66480e78dbd3b59"
+ ],
+ "Cmd": [
+ "redis-server"
+ ],
+ "ArgsEscaped": true,
+ "Image": "",
+ "Volumes": {
+ "/data": {}
+ },
+ "WorkingDir": "/data",
+ "Entrypoint": [
+ "docker-entrypoint.sh"
+ ],
```
-After all this is done, we can go back to the `run-app-elfloader` repository and use the `run.sh` script to run the application we just prepared:
+Then we run a Redis instance and start a shell:
```console
-./run.sh -r ../dynamic-apps/grep/ /usr/bin/grep bash /etc/passwd
+docker run --rm -p 6379:6379 -it redis:7.2-bookworm /bin/bash
```
-The command will search for the `bash` string in the `/etc/passwd` file.
-Note that paths are absolute in the application root filesystem.
+We get a console / shell of running inside Docker, in the `WorkingDir` option above (`/data`):
+
+```
+root@8b346198f54d:/data#
+```
-The command output will be similar to:
+Our goal is to know the path to the executable, the library dependencies, other required files.
+We use the commands below to locate the executable and get the library dependencies:
-```text
-SeaBIOS (version 1.15.0-1)
-Booting from ROM..Powered by
-o. .o _ _ __ _
-Oo Oo ___ (_) | __ __ __ _ ' _) :_
-oO oO ' _ `| | |/ / _)' _` | |_| _)
-oOo oOO| | | | | (| | | (_) | _) :_
- OoOoO ._, ._:_:_,\_._, .__,_:_, \___)
- Prometheus 0.14.0~4cce8306-custom
-root:x:0:0:root:/root:/bin/bash
-unikraft:x:1000:1000:Unikraft User,,,:/home/unikraft:/bin/bash
+```console
+root@8b346198f54d:/data# which redis-server
+/usr/local/bin/redis-server
+root@8b346198f54d:/data# ldd /usr/local/bin/redis-server
+ linux-vdso.so.1 (0x00007fffb7d39000)
+ libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff32f07d000)
+ libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007ff32efd3000)
+ libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007ff32eb51000)
+ libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff32e970000)
+ /lib64/ld-linux-x86-64.so.2 (0x00007ff32f6f5000)
```
-### Custom Applications
+We also start Redis to ensure everything works OK:
-The steps above assumed the existence of a pre-built binary.
-Let's consider custom applications that we have written.
-For example, we create a simple helloworld application in C++.
+```console
+root@8b346198f54d:/data# /usr/local/bin/redis-server
+17:C 25 Jan 2024 11:07:55.418 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
+17:C 25 Jan 2024 11:07:55.419 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+17:C 25 Jan 2024 11:07:55.419 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=17, just started
+17:C 25 Jan 2024 11:07:55.419 # Warning: no config file specified, using the default config. In order to specify a config file use /usr/local/bin/redis-server /path/to/redis.conf
+17:M 25 Jan 2024 11:07:55.420 * monotonic clock: POSIX clock_gettime
+ _._
+ _.-``__ ''-._
+ _.-`` `. `_. ''-._ Redis 7.2.4 (00000000/0) 64 bit
+ .-`` .-```. ```\/ _.,_ ''-._
+ ( ' , .-` | `, ) Running in standalone mode
+ |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
+ | `-._ `._ / _.-' | PID: 17
+ `-._ `-._ `-./ _.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' | https://redis.io
+ `-._ `-._`-.__.-'_.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' |
+ `-._ `-._`-.__.-'_.-' _.-'
+ `-._ `-.__.-' _.-'
+ `-._ _.-'
+ `-.__.-'
+
+17:M 25 Jan 2024 11:07:55.436 * Server initialized
+17:M 25 Jan 2024 11:07:55.436 * Ready to accept connections tcp
+```
-We create the application as `helloworld.cpp`:
+Redis starts OK.
-```cpp
-#include
+A crude way to determine other dependencies is to trace the opened files, with `strace`.
+First install `strace` in the container:
-int main()
-{
- std::cout << "Hello World!" << std::endl;
- return 0;
-}
+```console
+apt update
+apt install -y strace
```
-We then build the application:
+Now trace the `openat` system call:
```console
-g++ -fPIC -pie -Wall -o helloworld helloworld.cpp
+root@8b346198f54d:/data# strace -e openat /usr/local/bin/redis-server > /dev/null
+openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libssl.so.3", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libcrypto.so.3", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
+openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 3
+openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = -1 ENOENT (No such file or directory)
+openat(AT_FDCWD, "/proc/sys/vm/overcommit_memory", O_RDONLY) = 5
+openat(AT_FDCWD, "/sys/kernel/mm/transparent_hugepage/enabled", O_RDONLY) = 5
+openat(AT_FDCWD, "/sys/devices/system/clocksource/clocksource0/current_clocksource", O_RDONLY) = 5
+openat(AT_FDCWD, "/proc/sys/net/core/somaxconn", O_RDONLY) = 6
+openat(AT_FDCWD, "dump.rdb", O_RDONLY) = 8
+openat(AT_FDCWD, "dump.rdb", O_RDONLY) = 8
+openat(AT_FDCWD, "/proc/self/stat", O_RDONLY) = 8
```
-The `-fPIC` or `-pie` flags are typically default build flags.
-We added them just to be sure.
+Apart from the library files, Redis requires the `/etc/localtime`, `/dev/unrandom` and some `/sys` and `/proc` files.
+The `dump.rdb` file is probably a dump of the previous run.
+`/sys` and `/proc` files are usually not mandatory.
+`/etc/localtime` and `/dev/urandom` may also not be strictly required.
-We are now in possession of the binary executable `helloworld`, so we apply the steps laid out in section [Pre-built Binaries](/guides/bincompat/#pre-built-binaries).
-Namely, using the `extract.sh` script to extract the binary and the dynamic libraries in the application root filesystem, and running the resulting filesystem using `run.sh`.
+So we have a list of dependencies.
-### Practice: Application Filesystems
+#### Constructing the Minimized Docker Environment
-Create application root filesystems for application that are already part of your Linux host filesystem.
-Follow the steps in the section [Pre-built Binaries](/guides/bincompat/#pre-built-binaries).
+With the information above we construct a minimized Docker environment in a `Dockerfile`:
-Recall to target binaries that don't use the GUI, nor the terminal screen, nor are multi-process.
+```Dockerfile
+FROM redis:7.2-bookworm as build
-Aim to create pull requests with the new application filesystems in [the `dynamic-apps` repository](https://github.com/unikraft/dynamic-apps).
+FROM scratch
-### Practice: Custom Applications in Interpreted Languages
+# Redis binary
+COPY --from=build /usr/local/bin/redis-server /usr/bin/redis-server
-Create your own applications in your preferred interpreted language.
-Choose among the languages that are already part of [the `dynamic-apps` repository](https://github.com/unikraft/dynamic-apps) (the `lang/` directory): Python, Lua, Perl, Ruby.
+# Redis libraries
+COPY --from=build /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
+COPY --from=build /lib/x86_64-linux-gnu/libssl.so.3 /lib/x86_64-linux-gnu/libssl.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libcrypto.so.3 /lib/x86_64-linux-gnu/libcrypto.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
+COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
+COPY --from=build /etc/ld.so.cache /etc/ld.so.cache
+```
-Add your scripts in the application filesystem for the respective programming language.
-Then run it with the `run.sh` script.
+We then build an image from the `Dockerfile`:
-Aim to create pull requests with the new application filesystems in [the `dynamic-apps` repository](https://github.com/unikraft/dynamic-apps), in the corresponding subdirectory of the `lang/` directory.
+```console
+$ docker build --tag minimal-redis .
+[+] Building 1.3s (12/12) FINISHED docker:default
+ => [internal] load .dockerignore 0.3s
+ => => transferring context: 2B 0.0s
+ => [internal] load build definition from Dockerfile 0.5s
+ => => transferring dockerfile: 689B 0.0s
+ => [internal] load metadata for docker.io/library/redis:7.2-bookworm 0.0s
+ => [build 1/1] FROM docker.io/library/redis:7.2-bookworm 0.0s
+ => CACHED [stage-1 1/7] COPY --from=build /usr/local/bin/redis-server /usr/bin/redis-server 0.0s
+ => CACHED [stage-1 2/7] COPY --from=build /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6 0.0s
+ => CACHED [stage-1 3/7] COPY --from=build /lib/x86_64-linux-gnu/libssl.so.3 /lib/x86_64-linux-gnu/libssl.so.3 0.0s
+ => CACHED [stage-1 4/7] COPY --from=build /lib/x86_64-linux-gnu/libcrypto.so.3 /lib/x86_64-linux-gnu/libcrypto.so.3 0.0s
+ => CACHED [stage-1 5/7] COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6 0.0s => CACHED [stage-1 6/7] COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 0.0s
+ => CACHED [stage-1 7/7] COPY --from=build /etc/ld.so.cache /etc/ld.so.cache 0.0s
+ => exporting to image 0.1s
+ => => exporting layers 0.0s
+ => => writing image sha256:9e95efccc19fc473a6718741ad5e70398a345361fef2f03187b8fe37a2573bab 0.0s
+ => => naming to docker.io/library/minimal-redis
+```
-### Practice: Custom Applications in Compiled Languages
+We verify the creation of the image:
-Create your own applications in your preferred compiled language (C, C++, Rust, Go, Objective-C).
-Build the source code into a dynamic PIE ELF.
+```console
+$ docker image ls minimal-redis
+REPOSITORY TAG IMAGE ID CREATED SIZE
+minimal-redis latest 4d857719dd2c About a minute ago 24.3MB
+```
-Then create application root filesystems for application that are already part of your Linux host filesytem.
-Aim to create pull requests with the new application filesystems in [the `dynamic-apps` repository](https://github.com/unikraft/dynamic-apps), in the corresponding subdirectory of the `lang/` directory.
+And now we can start Redis inside the minimal image:
-## Build `app-elfloader`
+```console
+$ docker run --rm -p 6379:6379 minimal-redis /usr/bin/redis-server
+1:C 25 Jan 2024 11:28:55.083 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
+1:C 25 Jan 2024 11:28:55.083 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+1:C 25 Jan 2024 11:28:55.083 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
+1:C 25 Jan 2024 11:28:55.083 # Warning: no config file specified, using the default config. In order to specify a config file use /usr/bin/redis-server /path/to/redis.conf
+1:M 25 Jan 2024 11:28:55.083 * monotonic clock: POSIX clock_gettime
+1:M 25 Jan 2024 11:28:55.084 * Running mode=standalone, port=6379.
+1:M 25 Jan 2024 11:28:55.084 * Server initialized
+1:M 25 Jan 2024 11:28:55.084 * Ready to accept connections tcp
+```
-Using `./run.sh`, we used the pre-built `app-elfloader` images from [the `run-app-elfloader` repository](https://github.com/unikraft/run-app-elfloader):
+It started, we also check it works correctly via `redis-cli`:
-- `app-elfloader_qemu-x86_64`: the standard image
-- `app-elfloader_qemu-x86_64_strace`: the image with system call tracing
-- `app-elfloader_qemu-x86_64_full-debug`: the image with full debug messages.
+```console
+$ redis-cli -h localhost
+localhost:6379> ping
+PONG
+localhost:6379> set a 1
+OK
+localhost:6379> get a
+"1"
+localhost:6379>
+```
-However, if new changes are added to Unikraft, or we want to test potential changes ourselves (pull requests, branches), we need to re-build the `app-elfloader` from [its repository](https://github.com/unikraft/app-elfloader).
+Everything is OK.
+We created a minimized Docker image for Redis inside a `Dockerfile`.
-In order to build our own `app-elfloader` image, follow the instructions in the [`app-elfloader` README file](https://github.com/unikraft/app-elfloader#readme), the ["Set Up"](https://github.com/unikraft/app-elfloader#set-up) and the ["Scripted Building and Running"](https://github.com/unikraft/app-elfloader#scripted-building-and-running) sections.
-In short, the instructions present you with different ways to build, using the scripts in the `scripts/build/` directory:
+#### Setting Redis with Unikraft
-- 9pfs or initrd filesystem
-- KraftKit-based build or Make-based build
-- QEMU or Firecracker VMM
-- Building the standard, system call tracing or full debug message image
+With the `Dockerfile` now available, we require a `Kraftfile` to run Redis with Unikraft.
+Since we are adding a new application, we will create an embedded initrd configuration.
+For that, we copy-paste [the `Kraftfile` from Node](https://github.com/unikraft/catalog/blob/main/library/node/18/Kraftfile) and update the `name` and `cmd` configuration.
+The `Kraftfile` will have the following contents:
-Running the image is easiest to be done via the scripts in the `scripts/run/` directory.
-These scripts invoke KraftKit or Firecracker or QEMU behind the scenes.
+```yaml
+spec: v0.6
-Note that the `run.sh` script in [the `run-app-elfloader` repository](https://github.com/unikraft/run-app-elfloader) can only be used for QEMU and 9pfs filesystem.
+name: redis
-### Building and Running Nginx
+rootfs: ./Dockerfile
-As an example, let's build `app-elfloader` and run Nginx in binary compatibility mode.
-Let's go for a 9pfs build, both with KraftKit and with Make.
+cmd: ["/usr/bin/redis"]
+[...]
+```
-The steps are:
+Next we build the Unikraft kernel image:
-1. Set up `app-elfloader` by following [the instructions in its documentation](https://github.com/unikraft/app-elfloader#set-up).
+```console
+kraft build --no-cache --no-update --log-type basic --log-level debug --plat qemu --arch x86_64
+```
-1. Enter the repository clone (i.e. the `elfloader/` directory) and run the `./generate.py` script the generates the scripts in `scripts/build/` and `scripts/run/` directories:
+Next we run the image:
- ```console
- ./scripts/generate.py
- ls -R ./scripts
- ```
+```console
+kraft run --log-type basic --log-level debug -p 6347:6347
+```
-1. Build the ELF loader with KraftKit:
+We get the output:
- ```console
- ./scripts/build/kraft-qemu-x86_64-9pfs.sh
- ```
+```text
+ D kraftkit 0.7.3
+ D using platform=qemu
+ D cannot run because: no arguments supplied runner=linuxu
+ D cannot run because: no arguments supplied runner=kernel
+ D using runner=kraftfile-unikraft
+ D qemu-system-x86_64 -version
+ D qemu-system-x86_64 -accel help
+ D qemu-system-x86_64 -append /usr/bin/redis-server -cpu host,+x2apic,-pmu -daemonize -device virtio-net-pci,mac=02:b0:b0:ab:80:01,netdev=hostnet0 -device pvpanic -device sga -display none -enable-kvm -kernel /home/razvand/unikraft/catal
+og/library/redis/7.2/.unikraft/build/redis_qemu-x86_64 -machine pc,accel=kvm -m size=64M -monitor unix:/home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/qemu_mon.sock,server,nowait -name 6a798339-4157-4708-80
+30-8ec9c40ec390 -netdev user,id=hostnet0,hostfwd=tcp::6347-:6347 -nographic -no-reboot -S -parallel none -pidfile /home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/machine.pid -qmp unix:/home/razvand/.local/s
+hare/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/qemu_control.sock,server,nowait -qmp unix:/home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/qemu_events.sock,server,nowait -rtc base=utc -serial file
+:/home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/machine.log -smp cpus=1,threads=1,sockets=1 -vga none
+ E could not start qemu instance: dial unix /home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/qemu_control.sock: connect: no such file or directory
+```
-1. Build the ELF Loader with Make:
+The error message lets us know there is a problem with running the application, so we check the debug file:
- ```console
- ./scripts/build/make-qemu-x86_64-9pfs.sh
- ```
+```console
+$ cat /home/razvand/.local/share/kraftkit/runtime/6a798339-4157-4708-8030-8ec9c40ec390/machine.log
+[...]
+en1: Added
+en1: Interface is up
+Powered by Unikraft Telesto (0.16.1~644821db)
+[ 0.138996] ERR: [appelfloader] redis-server: Failed to initialize ELF parser
+[ 0.140238] ERR: [appelfloader] : Resource exhaustion (10)
+```
-1. Run the resulting image with KraftKit:
+The message `Resource exhaustion` lets us know that maybe we not running with enough memory, so we go for `256M` of memory:
- ```console
- ./scripts/run/kraft-qemu-x86_64-9pfs-nginx.sh
- ```
+```console
+kraft run --log-type basic --log-level debug -M 256M -p 6347:6347
+```
-1. Rn the resulting image with QEMU:
+This indeed is the issue and the output message confirms the starting of the server:
- ```console
- ./scripts/run/qemu-x86_64-9pfs-nginx.sh
- ```
+```text
+ D kraftkit 0.7.3
+ D using platform=qemu
+ D cannot run because: no arguments supplied runner=linuxu
+ D cannot run because: no arguments supplied runner=kernel
+ D using runner=kraftfile-unikraft
+ D qemu-system-x86_64 -version
+ D qemu-system-x86_64 -accel help
+ D qemu-system-x86_64 -append /usr/bin/redis-server -cpu host,+x2apic,-pmu -daemonize -device virtio-net-pci,mac=02:b0:b0:01:cd:01,netdev=hostnet0 -device pvpanic -device sga -display none -enable-kvm -kernel /home/razvand/unikraft/catalog/library/redis/7.2/.unikraft/build/redis_qemu-x86_64 -machine pc,accel=kvm -m size=244M -monitor unix:/home/razvand/.local/share/kraftkit/runtime/a97b85de-91b2-4745-8104-625e870aea65/qemu_mon.sock,server,nowait -name a97b85de-91b2-4745-8104-625e870aea65 -netdev user,id=hostnet0,hostfwd=tcp::6347-:6347 -nographic -no-reboot -S -parallel none -pidfile /home/razvand/.local/share/kraftkit/runtime/a97b85de-91b2-4745-8104-625e870aea65/machine.pid -qmp unix:/home/razvand/.local/share/kraftkit/runtime/a97b85de-91b2-4745-8104-625e870aea65/qemu_control.sock,server,nowait -qmp unix:/home/razvand/.local/share/kraftkit/runtime/a97b85de-91b2-4745-8104-625e870aea65/qemu_events.sock,server,nowait -rtc base=utc -serial file:/home/razvand/.local/share/kraftkit/runtime/a97b85de-91b2-4745-8104-625e870aea65/machine.log -smp cpus=1,threads=1,sockets=1 -vga none
+en1: Interface is up
+Powered by Unikraft Telesto (0.16.1~644821db)
+1:C 25 Jan 2024 12:06:06.081 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
+1:C 25 Jan 2024 12:06:06.082 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
+1:C 25 Jan 2024 12:06:06.084 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
+[ 0.187817] ERR: [libposix_process] Ignore updating resource 7: cur = 10032, max = 10032
+1:M 25 Jan 2024 12:06:06.089 * Increased maximum number of open files to 10032 (it was originally set to 1024).
+1:M 25 Jan 2024 12:06:06.091 * monotonic clock: POSIX clock_gettime
+ _._
+ _.-``__ ''-._
+ _.-`` `. `_. ''-._ Redis 7.2.4 (00000000/0) 64 bit
+ .-`` .-```. ```\/ _.,_ ''-._
+ ( ' , .-` | `, ) Running in standalone mode
+ |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
+ | `-._ `._ / _.-' | PID: 1
+ `-._ `-._ `-./ _.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' | https://redis.io
+ `-._ `-._`-.__.-'_.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' |
+ `-._ `-._`-.__.-'_.-' _.-'
+ `-._ `-.__.-' _.-'
+ `-._ _.-'
+ `-.__.-'
+
+1:M 25 Jan 2024 12:06:06.111 # Warning: Could not create server TCP listening socket ::*:6379: unable to bind socket, errno: 97
+1:M 25 Jan 2024 12:06:06.114 * Server initialized
+1:M 25 Jan 2024 12:06:06.115 * Ready to accept connections tcp
+en1: Set IPv4 address 10.0.2.15 mask 255.255.255.0 gw 10.0.2.2
+```
-1. Test
+However, the warning of being unable to bind the socket is problematic.
+Using `redis-cli` lets us know, there is a problem with Redis:
-1. Run the resulting images from KraftKit and QEMU with `run.sh`:
+```console
+$ redis-cli -h localhost
+Could not connect to Redis at localhost:6379: Connection refused
+not connected>
+```
- ```console
- sudo pkill -f firecracker
- sudo pkill -f qemu
- sudo ip link set dev virbr0 down
- sudo ip link del dev virbr0
- sudo ./run.sh -n -k ../elfloader/.unikraft/build/elfloader-qemu-x86_64-9pfs_qemu-x86_64 -r ../dynamic-apps/nginx /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
-
- sudo pkill -f firecracker
- sudo pkill -f qemu
- sudo ip link set dev virbr0 down
- sudo ip link del dev virbr0
- sudo ./run.sh -n -k ../elfloader/workdir/build/elfloader_qemu-x86_64 -r ../dynamic-apps/nginx /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
- ```
+The error is due to a likely absence of full IPv6 support.
+We require a configuration file that binds directly to IPv4.
-1. Test all runs with `curl` on a different console:
+#### Configure Redis for Unikraft
- ```console
- curl http://172.44.0.2
- ```
+To fix the above issue we use the [existing Redis 7.0 configuration for Unikraft](https://github.com/unikraft/catalog/blob/main/library/redis/7.0/rootfs/redis.conf).
+This is for a native (i.e. non-bincompat) configuration, but it doesn't matter.
+
+This requires an update to the `Dockerfile`, that needs to include the configuration file.
+The new `Dockerfile` is:
+
+```Dockerfile
+FROM redis:7.2-bookworm as build
+
+FROM scratch
+
+# Redis binary
+COPY --from=build /usr/local/bin/redis-server /usr/bin/redis-server
+
+# Redis libraries
+COPY --from=build /lib/x86_64-linux-gnu/libm.so.6 /lib/x86_64-linux-gnu/libm.so.6
+COPY --from=build /lib/x86_64-linux-gnu/libssl.so.3 /lib/x86_64-linux-gnu/libssl.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libcrypto.so.3 /lib/x86_64-linux-gnu/libcrypto.so.3
+COPY --from=build /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu/libc.so.6
+COPY --from=build /lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2
+COPY --from=build /etc/ld.so.cache /etc/ld.so.cache
+
+# Redis configuration
+COPY ./redis.conf /etc/redis.conf
+```
+
+We also update the `cmd` option in the `Kraftfile`:
+
+```yaml
+cmd: ["/usr/bin/redis-server", "/etc/redis.conf"]
+```
+
+We rebuild the image:
+
+```console
+rm -fr .config* .unikraft*
+kraft build --no-cache --no-update --log-type basic --log-level debug --plat qemu --arch x86_64
+```
+
+And we rerun it:
+
+```console
+kraft rm --all
+kraft run --log-type basic --log-level debug -M 256M -p 6347:6347
+```
+
+Everything seems to be OK, according to the output:
+
+```console
+ _._
+ _.-``__ ''-._
+ _.-`` `. `_. ''-._ Redis 7.2.4 (00000000/0) 64 bit
+ .-`` .-```. ```\/ _.,_ ''-._
+ ( ' , .-` | `, ) Running in standalone mode
+ |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
+ | `-._ `._ / _.-' | PID: 1
+ `-._ `-._ `-./ _.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' | https://redis.io
+ `-._ `-._`-.__.-'_.-' _.-'
+ |`-._`-._ `-.__.-' _.-'_.-'|
+ | `-._`-._ _.-'_.-' |
+ `-._ `-._`-.__.-'_.-' _.-'
+ `-._ `-.__.-' _.-'
+ `-._ _.-'
+ `-.__.-'
+
+1:M 25 Jan 2024 12:15:36.099 * Server initialized
+1:M 25 Jan 2024 12:15:36.100 * Ready to accept connections tcp
+en1: Set IPv4 address 10.0.2.15 mask 255.255.255.0 gw 10.0.2.2
+```
+
+We use `redis-cli` to query the server:
+
+```
+redis-cli -h localhost
+```
+
+This currently doesn't work because of an issue with Unikraft.
+But everything we did on the application side is OK.
+
+#### Contributing to the Application Catalog
+
+With the Redis application now set, we can make a contribution to the [`catalog` repository](https://github.com/unikraft/catalog).
+For that three additional steps need to be taken:
+
+1. Create a `README.md` file.
+1. Create a GitHub workflow for the application, following the [existing workflow files](https://github.com/unikraft/catalog/tree/main/.github/workflows).
+1. Update the badge listing in the [top-level `README.md` file](https://github.com/unikraft/catalog/blob/main/README.md).
+
+Then create a commit with the `Dockerfile`, `Kraftfile`, `README.md`, the new GitHub workflow file and updates to the [top-level `README.md` file](https://github.com/unikraft/catalog/blob/main/README.md).
+And submit a pull request.
+
+### Rust Tokio Web Server
+
+TODO
-### Practice: Build `app-elfloader` and Run Applications
+### Python Flask
-Build `app-elfloader` in different configurations (filesystem, VMMs, KraftKit / Make).
-Run different applications with it in different ways: KraftKit, QEMU, Firecracker, `run.sh`.
+TODO