Skip to content

Commit

Permalink
Merge branch 'main' into fds-add-divide-resplitter
Browse files Browse the repository at this point in the history
  • Loading branch information
jafermarq authored May 21, 2024
2 parents e66cac7 + 2f77c52 commit fa7bc73
Show file tree
Hide file tree
Showing 436 changed files with 48,422 additions and 18,679 deletions.
4 changes: 4 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,7 @@ README.md @jafermarq @tanertopal @danieljanes

# Changelog
/doc/source/ref-changelog.md @jafermarq @tanertopal @danieljanes

# GitHub Actions and Workflows
/.github/workflows @Robert-Steiner @tanertopal @danieljanes
/.github/actions @Robert-Steiner @tanertopal @danieljanes
21 changes: 1 addition & 20 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
<!--
Thank you for opening a pull request (PR)!
Please rename your PRs following this [format](https://flower.ai/docs/framework/contributor-tutorial-contribute-on-github.html#pr-title-format).
Contribution guidelines: https://github.com/adap/flower/blob/main/CONTRIBUTING.md
-->

Expand Down Expand Up @@ -37,29 +38,9 @@ Example: The variable `rnd` was renamed to `server_round` to improve readability
- [ ] Implement proposed change
- [ ] Write tests
- [ ] Update [documentation](https://flower.ai/docs/writing-documentation.html)
- [ ] Update the changelog entry below
- [ ] Make CI checks pass
- [ ] Ping maintainers on [Slack](https://flower.ai/join-slack/) (channel `#contributions`)

<!--
Inside the following 'Changelog entry' section, you should put the description of your changes that will be added to the changelog alongside your PR title.
If the section is completely empty (without any token) or non-existant, the changelog will just contain the title of the PR for the changelog entry, without any description. If the section contains some text other than tokens, it will use it to add a description to the change. If the section contains one of the following tokens it will ignore any other text and put the PR under the corresponding section of the changelog:
<general> is for classifying a PR as a general improvement.
<skip> is to not add the PR to the changelog
<baselines> is to add a general baselines change to the PR
<examples> is to add a general examples change to the PR
<sdk> is to add a general sdk change to the PR
<simulations> is to add a general simulations change to the PR
Note that only one token should be used.
-->

### Changelog entry



### Any other comments?

<!--
Expand Down
14 changes: 10 additions & 4 deletions .github/actions/bootstrap/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ inputs:
poetry-version:
description: "Version of poetry to be installed using pip"
default: 1.7.1
poetry-skip:
description: "Option to skip the poetry installation"
required: false
default: 'false'
outputs:
python-version:
description: "Version range or exact version of Python or PyPy"
Expand All @@ -35,10 +39,12 @@ runs:
python-version: ${{ inputs.python-version }}
- name: Install build tools
run: |
python -m pip install -U pip==${{ inputs.pip-version }}
python -m pip install -U setuptools==${{ inputs.setuptools-version }}
python -m pip install -U poetry==${{ inputs.poetry-version }}
python -m poetry config virtualenvs.create false
if ${{ inputs.poetry-skip == 'false' }}; then
python -m pip install -U pip==${{ inputs.pip-version }}
python -m pip install -U setuptools==${{ inputs.setuptools-version }}
python -m pip install -U poetry==${{ inputs.poetry-version }}
python -m poetry config virtualenvs.create false
fi
shell: bash
- id: versions
shell: bash
Expand Down
22 changes: 12 additions & 10 deletions .github/workflows/_docker-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,9 @@ jobs:
import hashlib
import os
hash = hashlib.sha256('''${{ inputs.build-args }}'''.encode())
hash = hashlib.sha256('''${{ inputs.namespace-repository }}
${{ inputs.file-dir }}
${{ inputs.build-args }}'''.encode())
with open(os.environ['GITHUB_OUTPUT'], 'a') as fh:
print(f"id={hash.hexdigest()}", file=fh)
Expand All @@ -69,22 +71,22 @@ jobs:

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@31cebacef4805868f9ce9a0cb03ee36c32df2ac4 # v5.3.0
uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81 # v5.5.1
with:
images: ${{ inputs.namespace-repository }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226 # v3.0.0
uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0

- name: Login to Docker Hub
uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3.0.0
uses: docker/login-action@e92390c5fb421da1463c202d546fed0ec5c39f20 # v3.1.0
with:
username: ${{ secrets.dockerhub-user }}
password: ${{ secrets.dockerhub-token }}

- name: Build and push
id: build
uses: docker/build-push-action@4a13e500e55cf31b7a5d59a38ab2040ab0f42f56 # v5.1.0
uses: docker/build-push-action@2cdde995de11925a030ce8070c3d77a52ffcf1c0 # v5.3.0
with:
platforms: ${{ matrix.platform.docker }}
context: "{{defaultContext}}:${{ inputs.file-dir }}"
Expand All @@ -98,7 +100,7 @@ jobs:
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@1eb3cb2b3e0f29609092a73eb033bb759a334595 # v4.1.0
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
with:
name: digests-${{ steps.build-id.outputs.id }}-${{ matrix.platform.name }}
path: /tmp/digests/*
Expand All @@ -114,24 +116,24 @@ jobs:
metadata: ${{ steps.meta.outputs.json }}
steps:
- name: Download digests
uses: actions/download-artifact@eaceaf801fd36c7dee90939fad912460b18a1ffe # v4.1.2
uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7
with:
pattern: digests-${{ needs.build.outputs.build-id }}-*
path: /tmp/digests
merge-multiple: true

- name: Docker meta
id: meta
uses: docker/metadata-action@31cebacef4805868f9ce9a0cb03ee36c32df2ac4 # v5.3.0
uses: docker/metadata-action@8e5442c4ef9f78752691e2d8f8d19755c6f78e81 # v5.5.1
with:
images: ${{ inputs.namespace-repository }}
tags: ${{ inputs.tags }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@f95db51fddba0c2d1ec667646a06c2ce06100226 # v3.0.0
uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0

- name: Login to Docker Hub
uses: docker/login-action@343f7c4344506bcbf9b4de18042ae17996df046d # v3.0.0
uses: docker/login-action@e92390c5fb421da1463c202d546fed0ec5c39f20 # v3.1.0
with:
username: ${{ secrets.dockerhub-user }}
password: ${{ secrets.dockerhub-token }}
Expand Down
12 changes: 7 additions & 5 deletions .github/workflows/cpp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,16 +68,18 @@ jobs:
cmake -DUSE_LOCAL_FLWR=ON -S . -B build
cmake --build build
pip install ../..
timeout 2m python server.py &
pid=$!
timeout 3m flower-superlink --insecure &
sleep 10
timeout 2m build/flwr_client 0 127.0.0.1:9092 &
sleep 3
build/flwr_client 0 127.0.0.1:8080 &
timeout 2m build/flwr_client 1 127.0.0.1:9092 &
sleep 3
build/flwr_client 1 127.0.0.1:8080 &
flower-server-app server:app --insecure &
pid=$!
wait $pid
res=$?
if [[ "$res" = "0" ]];
then echo "Training worked correctly";
then echo "Training worked correctly" && exit 0;
else echo "Training had an issue" && exit 1;
fi
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/deprecated_baselines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@ name: Deprecated-Baselines

on:
push:
branches:
- main
branches: ['main']
paths: ['baselines/flwr_baselines/**']
pull_request:
branches:
- main

branches: ['main']
paths: ['baselines/flwr_baselines/**']
concurrency:
group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Expand Down
15 changes: 7 additions & 8 deletions .github/workflows/docker-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ on:
permissions:
contents: read

env:
DEFAULT_UBUNTU: 22.04

jobs:
parameters:
name: Collect build parameters
Expand All @@ -24,7 +21,6 @@ jobs:
outputs:
pip-version: ${{ steps.versions.outputs.pip-version }}
setuptools-version: ${{ steps.versions.outputs.setuptools-version }}
ubuntu-version: ${{ steps.versions.outputs.ubuntu-version }}

steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
Expand All @@ -36,7 +32,6 @@ jobs:
run: |
echo "pip-version=${{ steps.bootstrap.outputs.pip-version }}" >> "$GITHUB_OUTPUT"
echo "setuptools-version=${{ steps.bootstrap.outputs.setuptools-version }}" >> "$GITHUB_OUTPUT"
echo "ubuntu-version=${{ env.DEFAULT_UBUNTU }}" >> "$GITHUB_OUTPUT"
build-base-images:
name: Build base images
Expand All @@ -46,15 +41,19 @@ jobs:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
distro:
- name: ubuntu
version: 22.04
with:
namespace-repository: flwr/base
file-dir: src/docker/base
file-dir: src/docker/base/${{ matrix.distro.name }}
build-args: |
PYTHON_VERSION=${{ matrix.python-version }}
PIP_VERSION=${{ needs.parameters.outputs.pip-version }}
SETUPTOOLS_VERSION=${{ needs.parameters.outputs.setuptools-version }}
UBUNTU_VERSION=${{ needs.parameters.outputs.ubuntu-version }}
tags: py${{ matrix.python-version }}-ubuntu${{ needs.parameters.outputs.ubuntu-version }}
DISTRO=${{ matrix.distro.name }}
DISTRO_VERSION=${{ matrix.distro.version }}
tags: py${{ matrix.python-version }}-${{ matrix.distro.name }}${{ matrix.distro.version }}
secrets:
dockerhub-user: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
36 changes: 0 additions & 36 deletions .github/workflows/docker-client.yml

This file was deleted.

52 changes: 52 additions & 0 deletions .github/workflows/docker-serverapp.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Build docker ServerApp image

on:
workflow_dispatch:
inputs:
flwr-version:
description: "Version of Flower."
required: true
type: string

permissions:
contents: read

jobs:
build-serverapp-images:
name: Build images
uses: ./.github/workflows/_docker-build.yml
# run only on default branch when using it with workflow_dispatch
if: github.ref_name == github.event.repository.default_branch
strategy:
fail-fast: false
matrix:
image: [
{
py-version: "3.8",
tags: "${{ github.event.inputs.flwr-version }}-py3.8-ubuntu22.04"
},
{
py-version: "3.9",
tags: "${{ github.event.inputs.flwr-version }}-py3.9-ubuntu22.04"
},
{
py-version: "3.10",
tags: "${{ github.event.inputs.flwr-version }}-py3.10-ubuntu22.04"
},
{
py-version: "3.11",
# those are two tags <version>-py3.11-py3.11-ubuntu22.04 and <version> separated by a \n
tags: "${{ github.event.inputs.flwr-version }}-py3.11-ubuntu22.04\n${{ github.event.inputs.flwr-version }}"
},
]
with:
namespace-repository: flwr/serverapp
file-dir: src/docker/serverapp
build-args: |
FLWR_VERSION=${{ github.event.inputs.flwr-version }}
PYTHON_VERSION=${{ matrix.image.py-version }}
UBUNTU_VERSION=ubuntu22.04
tags: ${{ matrix.image.tags }}
secrets:
dockerhub-user: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
Original file line number Diff line number Diff line change
@@ -1,50 +1,45 @@
name: Build docker server image
name: Build docker SuperLink image

on:
workflow_dispatch:
inputs:
flwr-version:
description: "Version of Flower e.g. (1.7.0)."
description: "Version of Flower."
required: true
type: string
base-image-tag:
description: "The tag of the Flower base image."
required: false
type: string
default: "py3.11-ubuntu22.04"

permissions:
contents: read

jobs:
build-server-images:
build-superlink-images:
name: Build images
uses: ./.github/workflows/_docker-build.yml
# run only on default branch when using it with workflow_dispatch
if: github.ref_name == github.event.repository.default_branch
with:
namespace-repository: flwr/server
file-dir: src/docker/server
namespace-repository: flwr/superlink
file-dir: src/docker/superlink
build-args: |
FLWR_VERSION=${{ github.event.inputs.flwr-version }}
BASE_IMAGE_TAG=${{ github.event.inputs.base-image-tag }}
PYTHON_VERSION=3.11
UBUNTU_VERSION=ubuntu22.04
tags: |
${{ github.event.inputs.flwr-version }}-${{ github.event.inputs.base-image-tag }}
${{ github.event.inputs.flwr-version }}-py3.11-ubuntu22.04
${{ github.event.inputs.flwr-version }}
latest
secrets:
dockerhub-user: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}

summary:
name: Build images
name: Summary
runs-on: ubuntu-22.04
needs: build-server-images
needs: build-superlink-images
timeout-minutes: 10
steps:
- run: |
echo "### Images" >> $GITHUB_STEP_SUMMARY
echo "" >> $GITHUB_STEP_SUMMARY
for IMAGE in $(echo ${{ toJson(needs.build-server-images.outputs.metadata) }} | jq -r '.tags[]' ); do
for IMAGE in $(echo ${{ toJson(needs.build-superlink-images.outputs.metadata) }} | jq -r '.tags[]' ); do
echo "- $IMAGE" >> $GITHUB_STEP_SUMMARY
done
Loading

0 comments on commit fa7bc73

Please sign in to comment.