Skip to content

Commit

Permalink
Merge branch 'main' into xgb-flwr-sim-comprehensive
Browse files Browse the repository at this point in the history
  • Loading branch information
jafermarq authored Jan 17, 2024
2 parents f007c0c + 6cea1c7 commit d7f752e
Show file tree
Hide file tree
Showing 339 changed files with 20,820 additions and 1,254 deletions.
6 changes: 6 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,9 @@

# Flower Baselines
/baselines @jafermarq @tanertopal @danieljanes

# Flower Examples
/examples @jafermarq @tanertopal @danieljanes

# Changelog
/doc/source/ref-changelog.md @jafermarq @tanertopal @danieljanes
21 changes: 20 additions & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,29 @@ Example: The variable `rnd` was renamed to `server_round` to improve readability
- [ ] Implement proposed change
- [ ] Write tests
- [ ] Update [documentation](https://flower.dev/docs/writing-documentation.html)
- [ ] Update [changelog](https://github.com/adap/flower/blob/main/doc/source/changelog.rst)
- [ ] Update the changelog entry below
- [ ] Make CI checks pass
- [ ] Ping maintainers on [Slack](https://flower.dev/join-slack/) (channel `#contributions`)

<!--
Inside the following 'Changelog entry' section, you should put the description of your changes that will be added to the changelog alongside your PR title.
If the section is completely empty (without any token), the changelog will just contain the title of the PR for the changelog entry, without any description. If the 'Changelog entry' section is removed entirely, it will categorize the PR as "General improvement" and add it to the changelog accordingly. If the section contains some text other than tokens, it will use it to add a description to the change. If the section contains one of the following tokens it will ignore any other text and put the PR under the corresponding section of the changelog:
<general> is for classifying a PR as a general improvement.
<skip> is to not add the PR to the changelog
<baselines> is to add a general baselines change to the PR
<examples> is to add a general examples change to the PR
<sdk> is to add a general sdk change to the PR
<simulations> is to add a general simulations change to the PR
Note that only one token should be used.
-->

### Changelog entry

<general>

### Any other comments?

<!--
Expand Down
17 changes: 9 additions & 8 deletions .github/workflows/_docker-build.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Reusable docker server image build workflow
name: Reusable docker image build workflow

on:
workflow_call:
Expand Down Expand Up @@ -35,7 +35,7 @@ permissions:
# based on https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners
jobs:
build:
name: Build server image
name: Build image
runs-on: ubuntu-22.04
timeout-minutes: 60
outputs:
Expand All @@ -46,8 +46,8 @@ jobs:
platform: [
# build-push action and qemu use different platform names
# therefore we create a map
{ qemu: "", docker: "linux/amd64" },
{ qemu: "arm64", docker: "linux/arm64" },
{ name: "amd64", qemu: "", docker: "linux/amd64" },
{ name: "arm64", qemu: "arm64", docker: "linux/arm64" },
]
steps:
- name: Create build id
Expand Down Expand Up @@ -98,9 +98,9 @@ jobs:
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
uses: actions/upload-artifact@1eb3cb2b3e0f29609092a73eb033bb759a334595 # v4.1.0
with:
name: digests-${{ steps.build-id.outputs.id }}
name: digests-${{ steps.build-id.outputs.id }}-${{ matrix.platform.name }}
path: /tmp/digests/*
if-no-files-found: error
retention-days: 1
Expand All @@ -114,10 +114,11 @@ jobs:
metadata: ${{ steps.meta.outputs.json }}
steps:
- name: Download digests
uses: actions/download-artifact@9bc31d5ccc31df68ecc42ccf4149144866c47d8a # v3.0.2
uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1
with:
name: digests-${{ needs.build.outputs.build-id }}
pattern: digests-${{ needs.build.outputs.build-id }}-*
path: /tmp/digests
merge-multiple: true

- name: Docker meta
id: meta
Expand Down
52 changes: 52 additions & 0 deletions .github/workflows/datasets-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Datasets-E2E

on:
push:
branches:
- main
paths:
- "datasets/flwr_datasets/**"
pull_request:
branches:
- main
paths:
- "datasets/flwr_datasets/**"

concurrency:
group: ${{ github.workflow }}-${{ github.ref == 'refs/heads/main' && github.run_id || github.event.pull_request.number || github.ref }}
cancel-in-progress: true

env:
FLWR_TELEMETRY_ENABLED: 0

jobs:
frameworks:
runs-on: ubuntu-22.04
timeout-minutes: 10
# Using approach described here:
# https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs
strategy:
matrix:
include:
- directory: pytorch

- directory: tensorflow

- directory: scikit-learn

name: Framework / ${{matrix.directory}}

defaults:
run:
working-directory: datasets/e2e/${{ matrix.directory }}

steps:
- uses: actions/checkout@v4
- name: Bootstrap
uses: ./.github/actions/bootstrap
with:
python-version: 3.8
- name: Install dependencies
run: python -m poetry install
- name: Run tests
run: python -m unittest discover -p '*_test.py'
2 changes: 1 addition & 1 deletion .github/workflows/docker-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
echo "ubuntu-version=${{ env.DEFAULT_UBUNTU }}" >> "$GITHUB_OUTPUT"
build-base-images:
name: Build images
name: Build base images
uses: ./.github/workflows/_docker-build.yml
needs: parameters
strategy:
Expand Down
36 changes: 36 additions & 0 deletions .github/workflows/docker-client.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Build docker client image

on:
workflow_dispatch:
inputs:
flwr-version:
description: "Version of Flower e.g. (1.6.0)."
required: true
type: string

permissions:
contents: read

jobs:
build-client-images:
name: Build client images
uses: ./.github/workflows/_docker-build.yml
# run only on default branch when using it with workflow_dispatch
if: github.ref_name == github.event.repository.default_branch
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
with:
namespace-repository: flwr/client
file-dir: src/docker/client
build-args: |
FLWR_VERSION=${{ github.event.inputs.flwr-version }}
BASE_IMAGE_TAG=py${{ matrix.python-version }}-ubuntu22.04
tags: |
${{ github.event.inputs.flwr-version }}-py${{ matrix.python-version }}-ubuntu22.04
${{ github.event.inputs.flwr-version }}
latest
secrets:
dockerhub-user: ${{ secrets.DOCKERHUB_USERNAME }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
10 changes: 6 additions & 4 deletions .github/workflows/docker-server.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ on:
description: "Version of Flower e.g. (1.6.0)."
required: true
type: string
base-image-version:
description: "Version of the Flower base image."
base-image-tag:
description: "The tag of the Flower base image."
required: false
type: string
default: "py3.11-ubuntu22.04"
Expand All @@ -20,14 +20,16 @@ jobs:
build-server-images:
name: Build images
uses: ./.github/workflows/_docker-build.yml
# run only on default branch when using it with workflow_dispatch
if: github.ref_name == github.event.repository.default_branch
with:
namespace-repository: flwr/server
file-dir: src/docker/server
build-args: |
FLWR_VERSION=${{ github.event.inputs.flwr-version }}
BASE_IMAGE_VERSION=${{ github.event.inputs.base-image-version }}
BASE_IMAGE_TAG=${{ github.event.inputs.base-image-tag }}
tags: |
${{ github.event.inputs.flwr-version }}-${{ github.event.inputs.base-image-version }}
${{ github.event.inputs.flwr-version }}-${{ github.event.inputs.base-image-tag }}
${{ github.event.inputs.flwr-version }}
latest
secrets:
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ jobs:
- name: Test wheel
run: ./dev/test-wheel.sh
- name: Upload wheel
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork }}
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }}
id: upload
env:
AWS_DEFAULT_REGION: ${{ secrets. AWS_DEFAULT_REGION }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets. AWS_SECRET_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
cd ./dist
echo "WHL_PATH=$(ls *.whl)" >> "$GITHUB_OUTPUT"
Expand Down Expand Up @@ -73,7 +73,7 @@ jobs:
dataset: |
import tensorflow as tf
tf.keras.datasets.cifar10.load_data()
- directory: tabnet
dataset: |
import tensorflow_datasets as tfds
Expand All @@ -83,7 +83,7 @@ jobs:
dataset: |
from torchvision.datasets import CIFAR10
CIFAR10('./data', download=True)
- directory: pytorch-lightning
dataset: |
from torchvision.datasets import MNIST
Expand All @@ -102,7 +102,7 @@ jobs:
- directory: fastai
dataset: |
from fastai.vision.all import untar_data, URLs
untar_data(URLs.MNIST)
untar_data(URLs.MNIST)
- directory: pandas
dataset: |
Expand All @@ -126,7 +126,7 @@ jobs:
- name: Install dependencies
run: python -m poetry install
- name: Install Flower wheel from artifact store
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork }}
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }}
run: |
python -m pip install https://artifact.flower.dev/py/${{ needs.wheel.outputs.dir }}/${{ needs.wheel.outputs.short_sha }}/${{ needs.wheel.outputs.whl_path }}
- name: Download dataset
Expand Down Expand Up @@ -161,7 +161,7 @@ jobs:
run: |
python -m poetry install
- name: Install Flower wheel from artifact store
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork }}
if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }}
run: |
python -m pip install https://artifact.flower.dev/py/${{ needs.wheel.outputs.dir }}/${{ needs.wheel.outputs.short_sha }}/${{ needs.wheel.outputs.whl_path }}
- name: Cache Datasets
Expand Down
63 changes: 63 additions & 0 deletions .github/workflows/framework-draft-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: Draft release

on:
push:
tags:
- "v*.*.*"

jobs:
publish:
if: ${{ github.repository == 'adap/flower' }}
name: Publish draft
runs-on: ubuntu-22.04
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Wait for wheel to be built
uses: lewagon/[email protected]
with:
ref: ${{ github.ref }}
check-name: 'Build, test and upload wheel'
repo-token: ${{ secrets.GITHUB_TOKEN }}
wait-interval: 10
- name: Download wheel
run: |
tag_name=$(echo "${GITHUB_REF_NAME}" | cut -c2-)
echo "TAG_NAME=$tag_name" >> "$GITHUB_ENV"
wheel_name="flwr-${tag_name}-py3-none-any.whl"
echo "WHEEL_NAME=$wheel_name" >> "$GITHUB_ENV"
tar_name="flwr-${tag_name}.tar.gz"
echo "TAR_NAME=$tar_name" >> "$GITHUB_ENV"
wheel_url="https://artifact.flower.dev/py/main/${GITHUB_SHA::7}/${wheel_name}"
tar_url="https://artifact.flower.dev/py/main/${GITHUB_SHA::7}/${tar_name}"
curl $wheel_url --output $wheel_name
curl $tar_url --output $tar_name
- name: Upload wheel
env:
AWS_DEFAULT_REGION: ${{ secrets. AWS_DEFAULT_REGION }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets. AWS_SECRET_ACCESS_KEY }}
run: |
aws s3 cp --content-disposition "attachment" --cache-control "no-cache" ./${{ env.WHEEL_NAME }} s3://artifact.flower.dev/py/release/v${{ env.TAG_NAME }}/${{ env.WHEEL_NAME }}
aws s3 cp --content-disposition "attachment" --cache-control "no-cache" ./${{ env.TAR_NAME }} s3://artifact.flower.dev/py/release/v${{ env.TAG_NAME }}/${{ env.TAR_NAME }}
- name: Generate body
run: |
./dev/get-latest-changelog.sh > body.md
cat body.md
- name: Release
uses: softprops/action-gh-release@v1
with:
body_path: ./body.md
draft: true
name: Flower ${{ env.TAG_NAME }}
files: |
${{ env.WHEEL_NAME }}
${{ env.TAR_NAME }}
Loading

0 comments on commit d7f752e

Please sign in to comment.