Merge branch 'main' into add-assign-reviewer-script

doc/source/contributor-how-to-build-docker-images.rst

@@ -2,9 +2,10 @@ How to build Docker Flower images locally
 =========================================
 
 Flower provides pre-made docker images on `Docker Hub <https://hub.docker.com/u/flwr>`_
-that include all necessary dependencies for running the SuperLink. You can also build your own custom
-docker images from scratch with a different version of Python or Ubuntu if that is what you need.
-In this guide, we will explain what images exist and how to build them locally.
+that include all necessary dependencies for running the SuperLink, SuperNode or ServerApp.
+You can also build your own custom docker images from scratch with a different version of Python
+or Linux distribution (Ubuntu/Alpine) if that is what you need. In this guide, we will explain what
+images exist and how to build them locally.
 
 Before we can start, we need to meet a few prerequisites in our local development environment.
 
@@ -20,19 +21,15 @@ Before we can start, we need to meet a few prerequisites in our local developmen
     :doc:`Run Flower using Docker <how-to-run-flower-using-docker>`
     which covers this step in more detail.
 
-Currently, Flower provides two images, a ``base`` image and a ``superlink`` image. The base image,
-as the name suggests, contains basic dependencies that the SuperLink needs.
-This includes system dependencies, Python and Python tools. The SuperLink image is
-based on the base image, but it additionally installs the SuperLink using ``pip``.
 
 The build instructions that assemble the images are located in the respective Dockerfiles. You
 can find them in the subdirectories of ``src/docker``.
 
-Both, base and SuperLink image are configured via build arguments. Through build arguments, we can make
-our build more flexible. For example, in the base image, we can specify the version of Python to
-install using the ``PYTHON_VERSION`` build argument. Some of the build arguments have default
-values, others must be specified when building the image. All available build arguments for each
-image are listed in one of the tables below.
+Flower Docker images are configured via build arguments. Through build arguments, we can make the
+creation of images more flexible. For example, in the base image, we can specify the version of
+Python to install using the ``PYTHON_VERSION`` build argument. Some of the build arguments have
+default values, others must be specified when building the image. All available build arguments for
+each image are listed in one of the tables below.
 
 Building the base image
 -----------------------
@@ -45,10 +42,18 @@ Building the base image
      - Description
      - Required
      - Example
+   * - ``DISTRO``
+     - The Linux distribution to use as the base image.
+     - No
+     - ``ubuntu``
+   * - ``DISTRO_VERSION``
+     - Version of the Linux distribution.
+     - No
+     - ``22.04``
    * - ``PYTHON_VERSION``
      - Version of ``python`` to be installed.
-     - Yes
-     - ``3.11``
+     - No
+     - ``3.11`` or ``3.11.1``
    * - ``PIP_VERSION``
      - Version of ``pip`` to be installed.
      - Yes
@@ -57,27 +62,34 @@ Building the base image
      - Version of ``setuptools`` to be installed.
      - Yes
      - ``69.0.2``
-   * - ``UBUNTU_VERSION``
-     - Version of the official Ubuntu Docker image.
-     - Defaults to ``22.04``.
-     -
+   * - ``FLWR_VERSION``
+     - Version of Flower to be installed.
+     - Yes
+     - ``1.8.0``
+   * - ``FLWR_PACKAGE``
+     - The Flower package to be installed.
+     - No
+     - ``flwr`` or ``flwr-nightly``
+
 
-The following example creates a base image with Python 3.11.0, pip 23.0.1 and setuptools 69.0.2:
+The following example creates a base Ubuntu/Alpine image with Python 3.11.0, pip 23.0.1,
+setuptools 69.0.2 and Flower 1.8.0:
 
 .. code-block:: bash
 
-  $ cd src/docker/base/ubuntu
+  $ cd src/docker/base/<ubuntu|alpine>
   $ docker build \
     --build-arg PYTHON_VERSION=3.11.0 \
+    --build-arg FLWR_VERSION=1.8.0 \
     --build-arg PIP_VERSION=23.0.1 \
     --build-arg SETUPTOOLS_VERSION=69.0.2 \
     -t flwr_base:0.1.0 .
 
 The name of image is ``flwr_base`` and the tag ``0.1.0``. Remember that the build arguments as well
 as the name and tag can be adapted to your needs. These values serve as examples only.
 
-Building the SuperLink image
-----------------------------
+Building the SuperLink/SuperNode or ServerApp image
+---------------------------------------------------
 
 .. list-table::
    :widths: 25 45 15 15
@@ -89,50 +101,33 @@ Building the SuperLink image
      - Example
    * - ``BASE_REPOSITORY``
      - The repository name of the base image.
-     - Defaults to ``flwr/base``.
-     -
-   * - ``PYTHON_VERSION``
-     - The Python version of the base image.
-     - Defaults to ``py3.11``.
-     -
-   * - ``UBUNTU_VERSION``
-     - The Ubuntu version of the base image.
-     - Defaults to ``ubuntu22.04``.
-     -
-   * - ``FLWR_PACKAGE``
-     - The PyPI package to install.
-     - Defaults to ``flwr``.
-     -
-   * - ``FLWR_VERSION``
-     - Version of Flower to be installed.
+     - No
+     - ``flwr/base``
+   * - ``BASE_IMAGE``
+     - The Tag of the Flower base image.
      - Yes
-     - ``1.8.0``
+     - ``1.8.0-py3.10-ubuntu22.04``
 
-
-The following example creates a SuperLink image with the official Flower base image
-py3.11-ubuntu22.04 and Flower 1.8.0:
+The following example creates a SuperLink/SuperNode or ServerApp image with the official Flower
+base image:
 
 .. code-block:: bash
 
-  $ cd src/docker/superlink/
+  $ cd src/docker/<superlink|supernode|serverapp>/
   $ docker build \
-    --build-arg FLWR_VERSION=1.8.0 \
+    --build-arg BASE_IMAGE=<FLOWER-VERSION>-py<PY-VERSION>-<DISTRIBUTION and VERSION> \
     -t flwr_superlink:0.1.0 .
 
-The name of image is ``flwr_superlink`` and the tag ``0.1.0``. Remember that the build arguments as
-well as the name and tag can be adapted to your needs. These values serve as examples only.
 
 If you want to use your own base image instead of the official Flower base image, all you need to do
-is set the ``BASE_REPOSITORY``, ``PYTHON_VERSION`` and ``UBUNTU_VERSION`` build arguments.
+is set the ``BASE_REPOSITORY`` build argument.
 
 .. code-block:: bash
 
   $ cd src/docker/superlink/
   $ docker build \
     --build-arg BASE_REPOSITORY=flwr_base \
-    --build-arg PYTHON_VERSION=3.11 \
-    --build-arg UBUNTU_VERSION=ubuntu22.04 \
-    --build-arg FLWR_VERSION=1.8.0 \
+    --build-arg BASE_IMAGE=0.1.0
     -t flwr_superlink:0.1.0 .
 
 After creating the image, we can test whether the image is working:

doc/source/contributor-how-to-release-flower.rst

@@ -12,6 +12,24 @@ The version number of a release is stated in ``pyproject.toml``. To release a ne
 2. Once the changelog has been updated with all the changes, run ``./dev/prepare-release-changelog.sh v<NEW_VERSION>``, where ``<NEW_VERSION>`` is the version stated in ``pyproject.toml`` (notice the ``v`` added before it). This will replace the ``Unreleased`` header of the changelog by the version and current date, and it will add a thanking message for the contributors. Open a pull request with those changes.
 3. Once the pull request is merged, tag the release commit with the version number as soon as the PR is merged: ``git tag v<NEW_VERSION>`` (notice the ``v`` added before the version number), then ``git push --tags``. This will create a draft release on GitHub containing the correct artifacts and the relevant part of the changelog.
 4. Check the draft release on GitHub, and if everything is good, publish it.
+5. Trigger the CI for building the Docker images.
+
+To trigger the workflow, a collaborator must create a ``workflow_dispatch`` event in the
+GitHub CI. This can be done either through the UI or via the GitHub CLI. The event requires only one
+input, the Flower version, to be released.
+
+**Via the UI**
+
+1. Go to the ``Build docker images`` workflow `page <https://github.com/adap/flower/actions/workflows/docker-images.yml>`_.
+2. Click on the ``Run workflow`` button and type the new version of Flower in the ``Version of Flower`` input field.
+3. Click on the **green** ``Run workflow`` button.
+
+.. image:: _static/docker-ci-release.png
+
+**Via the GitHub CI**
+
+1. Make sure you are logged in via ``gh auth login`` and that the current working directory is the root of the Flower repository.
+2. Trigger the workflow via ``gh workflow run docker-images.yml -f flwr-version=<NEW_VERSION>``.
 
 After the release
 -----------------

doc/source/how-to-run-flower-using-docker.rst

@@ -72,19 +72,27 @@ Mounting a volume to store the state on the host system
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you want to persist the state of the SuperLink on your host system, all you need to do is specify
-a path where you want to save the file on your host system and a name for the database file. In the
-example below, we tell Docker via the flag ``--volume`` to mount the user's home directory
-(``~/`` on your host) into the ``/app/`` directory of the container. Furthermore, we use the
-flag ``--database`` to specify the name of the database file.
+a directory where you want to save the file on your host system and a name for the database file. By
+default, the SuperLink container runs with a non-root user called ``app`` with the user ID
+``49999``. It is recommended to create new directory and change the user ID of the directory to
+``49999`` to ensure the mounted directory has the proper permissions. If you later want to delete
+the directory, you can change the user ID back to the current user ID by running
+``sudo chown -R $USER:$(id -gn) state``.
+
+In the example below, we create a new directory, change the user ID and tell Docker via the flag
+``--volume`` to mount the local ``state`` directory into the ``/app/state`` directory of the
+container. Furthermore, we use the flag ``--database`` to specify the name of the database file.
 
 .. code-block:: bash
 
+  $ mkdir state
+  $ sudo chmod -R 49999:49999 state
   $ docker run --rm \
-    -p 9091:9091 -p 9092:9092 --volume ~/:/app/ flwr/superlink:1.8.0 \
+    -p 9091:9091 -p 9092:9092 --volume ./state/:/app/state flwr/superlink:1.8.0 \
     --insecure \
     --database state.db
 
-As soon as the SuperLink starts, the file ``state.db`` is created in the user's home directory on
+As soon as the SuperLink starts, the file ``state.db`` is created in the ``state`` directory on
 your host system. If the file already exists, the SuperLink tries to restore the state from the
 file. To start the SuperLink with an empty database, simply remove the ``state.db`` file.
 
@@ -100,17 +108,27 @@ PEM-encoded certificate chain.
   page contains a section that will guide you through the process.
 
 Assuming all files we need are in the local ``certificates`` directory, we can use the flag
-``--volume`` to mount the local directory into the ``/app/`` directory of the container. This allows
-the SuperLink to access the files within the container. Finally, we pass the names of the
-certificates to the SuperLink with the ``--root-certificates`` flag.
+``--volume`` to mount the local directory into the ``/app/certificates/`` directory of the container.
+This allows the SuperLink to access the files within the container. The ``ro`` stands for
+``read-only``. Docker volumes default to ``read-write``; that option tells Docker to make the volume
+``read-only`` instead. Finally, we pass the names of the certificates and key file to the SuperLink
+with the ``--ssl-ca-certfile``, ``--ssl-certfile`` and ``--ssl-keyfile`` flag.
 
 .. code-block:: bash
 
   $ docker run --rm \
-    -p 9091:9091 -p 9092:9092 --volume ./certificates/:/app/ flwr/superlink:1.9.0 \
-    --ssl-ca-certfile ca.crt \
-    --ssl-certfile server.pem \
-    --ssl-keyfile server.key
+    -p 9091:9091 -p 9092:9092 \
+    --volume ./certificates/:/app/certificates/:ro flwr/superlink:nightly \
+    --ssl-ca-certfile certificates/ca.crt \
+    --ssl-certfile certificates/server.pem \
+    --ssl-keyfile certificates/server.key
+
+.. note::
+
+  Because Flower containers, by default, run with a non-root user ``app``, the mounted files and
+  directories must have the proper permissions for the user ID ``49999``. For example, to change the
+  user ID of all files in the ``certificates/`` directory, you can run
+  ``sudo chown -R 49999:49999 certificates/*``.
 
 Flower SuperNode
 ----------------
@@ -225,7 +243,7 @@ Now that we have built the SuperNode image, we can finally run it.
 
 .. code-block:: bash
 
-  $ docker run --rm flwr_supernode:0.0.1 client:app \
+  $ docker run --rm flwr_supernode:0.0.1 \
     --insecure \
     --server 192.168.1.100:9092
 
@@ -381,8 +399,7 @@ To enable SSL, we will need to mount a PEM-encoded root certificate into your Se
 
 Assuming the certificate already exists locally, we can use the flag ``--volume`` to mount the local
 certificate into the container's ``/app/`` directory. This allows the ServerApp to access the
-certificate within the container. Use the ``--ssl-ca-certfile``, ``--ssl-certfile``, and ``--ssl-keyfile`` 
-flags when starting the container.
+certificate within the container. Use the ``--root-certificates`` flags when starting the container.
 
 .. code-block:: bash
 

e2e/docker/supernode.Dockerfile

@@ -2,8 +2,7 @@ FROM flwr/supernode:nightly
 
 WORKDIR /app
 COPY pyproject.toml ./
-RUN python -m pip install -U --no-cache-dir . \
-    && pyenv rehash
+RUN python -m pip install -U --no-cache-dir .
 
 COPY client.py ./
 ENTRYPOINT [ "flower-client-app", "client:app" ]

src/docker/base/alpine/Dockerfile

@@ -34,10 +34,10 @@ RUN apk add --no-cache \
     g++ \
     libffi-dev \
     # create virtual env
-    && python -m venv /app/venv
+    && python -m venv /python/venv
 
 # Make sure we use the virtualenv
-ENV PATH=/app/venv/bin:$PATH
+ENV PATH=/python/venv/bin:$PATH
 
 # Install specific version of pip, setuptools and flwr
 ARG PIP_VERSION
@@ -59,12 +59,14 @@ RUN apk add --no-cache \
     --no-create-home \
     --disabled-password \
     --gecos "" \
-    --uid 49999 app
+    --uid 49999 app \
+    && mkdir -p /app \
+    && chown -R app:app /app
 
-COPY --from=compile --chown=app:app /app/venv /app/venv
+COPY --from=compile --chown=app:app /python/venv /python/venv
 
 # Make sure we use the virtualenv
-ENV PATH=/app/venv/bin:$PATH \
+ENV PATH=/python/venv/bin:$PATH \
     # Send stdout and stderr stream directly to the terminal. Ensures that no
     # output is retained in a buffer if the application crashes.
     PYTHONUNBUFFERED=1 \

src/docker/base/ubuntu/Dockerfile

@@ -74,8 +74,8 @@ ENV PATH=/usr/local/bin/python/bin:$PATH \
 
 # Use a virtual environment to ensure that Python packages are installed in the same location
 # regardless of whether the subsequent image build is run with the app or the root user
-RUN python -m venv /app/venv
-ENV PATH=/app/venv/bin:$PATH
+RUN python -m venv /python/venv
+ENV PATH=/python/venv/bin:$PATH
 
 ARG PIP_VERSION
 ARG SETUPTOOLS_VERSION
@@ -92,6 +92,8 @@ RUN adduser \
     --disabled-password \
     --gecos "" \
     --uid 49999 app \
+    && mkdir -p /app \
+    && chown -R app:app /python \
     && chown -R app:app /app
 
 WORKDIR /app