Merge branch 'main' into add-llm-code-eval

.github/CODEOWNERS

@@ -7,7 +7,10 @@
 README.md @jafermarq @tanertopal @danieljanes
 
 # Flower Baselines
-/baselines @jafermarq @tanertopal @danieljanes
+/baselines @jafermarq @danieljanes
+
+# Flower Benchmarks
+/benchmarks @jafermarq @danieljanes
 
 # Flower Datasets
 /datasets @jafermarq @tanertopal @danieljanes
@@ -27,3 +30,8 @@ README.md @jafermarq @tanertopal @danieljanes
 # GitHub Actions and Workflows
 /.github/workflows @Robert-Steiner @tanertopal @danieljanes
 /.github/actions @Robert-Steiner @tanertopal @danieljanes
+
+# Docker-related files
+/.devcontainer @Robert-Steiner @Moep90
+**/Dockerfile @Robert-Steiner @Moep90
+**/*.Dockerfile @Robert-Steiner @Moep90

.github/workflows/_docker-build.yml

@@ -122,7 +122,7 @@ jobs:
           touch "/tmp/digests/${digest#sha256:}"
 
       - name: Upload digest
-        uses: actions/upload-artifact@834a144ee995460fba8ed112a2fc961b36a5ec5a # v4.3.6
+        uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0
         with:
           name: digests-${{ steps.build-id.outputs.id }}-${{ matrix.platform.name }}
           path: /tmp/digests/*

.github/workflows/docker-readme.yml

@@ -0,0 +1,51 @@
+name: Update Docker READMEs
+
+on:
+  push:
+    branches:
+      - 'main'
+    paths:
+      - 'src/docker/**/README.md'
+
+jobs:
+    collect:
+      if: ${{ github.repository == 'adap/flower' }}
+      name: Collect Docker READMEs
+      runs-on: ubuntu-22.04
+      timeout-minutes: 10
+      outputs:
+        readme_files: ${{ steps.filter.outputs.readme_files }}
+      steps:
+        - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+
+        - uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3.0.2
+          id: filter
+          with:
+            list-files: "json"
+            filters: |
+              readme:
+                - 'src/docker/**/README.md'
+
+    update:
+      if: ${{ needs.collect.outputs.readme_files != '' && toJson(fromJson(needs.collect.outputs.readme_files)) != '[]' }}
+      name: Update Docker READMEs
+      runs-on: ubuntu-22.04
+      timeout-minutes: 10
+      needs: collect
+      strategy:
+        matrix:
+          readme_path: ${{ fromJSON(needs.collect.outputs.readme_files) }}
+
+      steps:
+        - uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
+
+        - id: repository
+          run: echo "name=$(basename $(dirname ${{ matrix.readme_path }}))" >> "$GITHUB_OUTPUT"
+
+        - name: Docker Hub Description
+          uses: peter-evans/dockerhub-description@e98e4d1628a5f3be2be7c231e50981aee98723ae # v4.0.0
+          with:
+            repository: flwr/${{ steps.repository.outputs.name }}
+            readme-filepath: ${{ matrix.readme_path }}
+            username: ${{ secrets.DOCKERHUB_USERNAME }}
+            password: ${{ secrets.DOCKERHUB_TOKEN }}

README.md

@@ -143,8 +143,8 @@ Other [examples](https://github.com/adap/flower/tree/main/examples):
 - [PyTorch: From Centralized to Federated](https://github.com/adap/flower/tree/main/examples/pytorch-from-centralized-to-federated)
 - [Vertical FL](https://github.com/adap/flower/tree/main/examples/vertical-fl)
 - [Federated Finetuning of OpenAI's Whisper](https://github.com/adap/flower/tree/main/examples/whisper-federated-finetuning)
-- [Federated Finetuning of Large Language Model](https://github.com/adap/flower/tree/main/examples/llm-flowertune)
-- [Federated Finetuning of a Vision Transformer](https://github.com/adap/flower/tree/main/examples/vit-finetune)
+- [Federated Finetuning of Large Language Model](https://github.com/adap/flower/tree/main/examples/flowertune-llm)
+- [Federated Finetuning of a Vision Transformer](https://github.com/adap/flower/tree/main/examples/flowertune-vit)
 - [Advanced Flower with TensorFlow/Keras](https://github.com/adap/flower/tree/main/examples/advanced-tensorflow)
 - [Advanced Flower with PyTorch](https://github.com/adap/flower/tree/main/examples/advanced-pytorch)
 - Single-Machine Simulation of Federated Learning Systems ([PyTorch](https://github.com/adap/flower/tree/main/examples/simulation-pytorch)) ([Tensorflow](https://github.com/adap/flower/tree/main/examples/simulation-tensorflow))

baselines/README.md

@@ -1,10 +1,9 @@
 # Flower Baselines
 
 
+> [!NOTE] 
 > We are changing the way we structure the Flower baselines. While we complete the transition to the new format, you can still find the existing baselines in the `flwr_baselines` directory. Currently, you can make use of baselines for [FedAvg](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/fedavg_mnist), [FedOpt](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/adaptive_federated_optimization), and [LEAF-FEMNIST](https://github.com/adap/flower/tree/main/baselines/flwr_baselines/flwr_baselines/publications/leaf/femnist).
 
-> The documentation below has been updated to reflect the new way of using Flower baselines.
-
 
 ## Structure
 
@@ -15,17 +14,15 @@ baselines/<baseline-name>/
                 ├── README.md
                 ├── pyproject.toml
                 └── <baseline-name>
-                            ├── *.py # several .py files including main.py and __init__.py
-                            └── conf
-                                └── *.yaml # one or more Hydra config files
+                            └── *.py # several .py files
 ```
-Please note that some baselines might include additional files (e.g. a `requirements.txt`) or a hierarchy of `.yaml` files for [Hydra](https://hydra.cc/).
 
 ## Running the baselines
 
-Each baseline is self-contained in its own directory. Furthermore, each baseline defines its own Python environment using [Poetry](https://python-poetry.org/docs/) via a `pyproject.toml` file and  [`pyenv`](https://github.com/pyenv/pyenv). If you haven't setup `Poetry` and `pyenv` already on your machine, please take a look at the [Documentation](https://flower.ai/docs/baselines/how-to-use-baselines.html#setting-up-your-machine) for a guide on how to do so.
+> [!NOTE]
+> We are in the process of migrating all baselines to use `flwr run`. Those baselines that remain using the previous system (i.e. using [Poetry](https://python-poetry.org/), [Hydra](https://hydra.cc/) and [start_simulation](https://flower.ai/docs/framework/ref-api/flwr.simulation.start_simulation.html)) might require you to first setup `Poetry` and `pyenv` already on your machine, please take a look at the [Documentation](https://flower.ai/docs/baselines/how-to-use-baselines.html#setting-up-your-machine) for a guide on how to do so.
 
-Assuming `pyenv` and `Poetry` are already installed on your system. Running a baseline can be done by:
+Each baseline is self-contained in its own directory. To run a baseline:
 
 1. Cloning the flower repository
 
@@ -34,29 +31,30 @@ Assuming `pyenv` and `Poetry` are already installed on your system. Running a ba
     ```
 
 2. Navigate inside the directory of the baseline you'd like to run.
-3. Follow the `[Environment Setup]` instructions in the `README.md`. In most cases this will require you to just do:
-
-    ```bash
-    poetry install
-    ```
+3. Follow the `[Environment Setup]` instructions in the `README.md`.
 4. Run the baseline as indicated in the `[Running the Experiments]` section in the `README.md` or in the `[Expected Results]` section to reproduce the experiments in the paper.
 
 
 ## Contributing a new baseline
 
 Do you have a new federated learning paper and want to add a new baseline to Flower? Or do you want to add an experiment to an existing baseline paper? Great, we really appreciate your contribution !!
 
+> [!TIP]
+> A more verbose version of these steps can be found in the [Flower Baselines documentation](https://flower.ai/docs/baselines/how-to-contribute-baselines.html).
+
 The steps to follow are:
 
+1. Create a new Python 3.10 environment and install Flower (`pip install flwr`)
 1. Fork the Flower repo and clone it into your machine.
-2. Navigate to the `baselines/` directory, choose a single-word (and **lowercase**) name for your baseline, and from there run:
+2. Navigate to the `baselines/` directory, from there and with your environment activated, run:
 
     ```bash
-    # This will create a new directory with the same structure as `baseline_template`.
-    ./dev/create-baseline.sh <baseline-name>
+    # Choose option "Flower Baseline" when prompted
+    flwr new <baseline-name>
     ```
-3. Then, go inside your baseline directory and continue with the steps detailed in `EXTENDED_README.md` and `README.md`.
-4. Once your code is ready and you have checked that following the instructions in your `README.md` the Python environment can be created correctly and that running the code following your instructions can reproduce the experiments in the paper, you just need to create a Pull Request (PR). Then, the process to merge your baseline into the Flower repo will begin!
+3. Then, go inside your baseline directory and continue with the steps detailed in the `README.md`.
+4. Once your code is ready, check that you have completed all the sections in the `README.md` and that, if a new environment is created, your baseline still runs (i.e. play the role of a person running the baseline you want to contribute).
+5. Create a Pull Request (PR). Then, the process to merge your baseline into the Flower repo will begin!
 
 
 Further resources: