diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index ed7535409fa4..bf3843c25bd4 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -126,7 +126,7 @@ jobs: - name: Install dependencies run: python -m poetry install - name: Install Flower wheel from artifact store - if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork }} + if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }} run: | python -m pip install https://artifact.flower.dev/py/${{ needs.wheel.outputs.dir }}/${{ needs.wheel.outputs.short_sha }}/${{ needs.wheel.outputs.whl_path }} - name: Download dataset @@ -161,7 +161,7 @@ jobs: run: | python -m poetry install - name: Install Flower wheel from artifact store - if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork }} + if: ${{ github.repository == 'adap/flower' && !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }} run: | python -m pip install https://artifact.flower.dev/py/${{ needs.wheel.outputs.dir }}/${{ needs.wheel.outputs.short_sha }}/${{ needs.wheel.outputs.whl_path }} - name: Cache Datasets diff --git a/doc/source/tutorial-series-get-started-with-flower-pytorch.ipynb b/doc/source/tutorial-series-get-started-with-flower-pytorch.ipynb old mode 100755 new mode 100644 index 41c9254e9d69..ce4c2bb63606 --- a/doc/source/tutorial-series-get-started-with-flower-pytorch.ipynb +++ b/doc/source/tutorial-series-get-started-with-flower-pytorch.ipynb @@ -9,7 +9,7 @@ "\n", "Welcome to the Flower federated learning tutorial!\n", "\n", - "In this notebook, we'll build a federated learning system using Flower and PyTorch. In part 1, we use PyTorch for the model training pipeline and data loading. In part 2, we continue to federate the PyTorch-based pipeline using Flower.\n", + "In this notebook, we'll build a federated learning system using Flower, [Flower Datasets](https://flower.dev/docs/datasets/) and PyTorch. In part 1, we use PyTorch for the model training pipeline and data loading. In part 2, we continue to federate the PyTorch-based pipeline using Flower.\n", "\n", "> [Star Flower on GitHub](https://github.com/adap/flower) ⭐️ and join the Flower community on Slack to connect, ask questions, and get help: [Join Slack](https://flower.dev/join-slack) 🌼 We'd love to hear from you in the `#introductions` channel! And if anything is unclear, head over to the `#questions` channel.\n", "\n", @@ -31,7 +31,7 @@ "source": [ "### Installing dependencies\n", "\n", - "Next, we install the necessary packages for PyTorch (`torch` and `torchvision`) and Flower (`flwr`):" + "Next, we install the necessary packages for PyTorch (`torch` and `torchvision`), Flower Datasets (`flwr-datasets`) and Flower (`flwr`):" ] }, { @@ -40,7 +40,7 @@ "metadata": {}, "outputs": [], "source": [ - "!pip install -q flwr[simulation] torch torchvision matplotlib" + "!pip install -q flwr[simulation] flwr_datasets[vision] torch torchvision matplotlib" ] }, { @@ -64,18 +64,19 @@ "import torch\n", "import torch.nn as nn\n", "import torch.nn.functional as F\n", - "import torchvision\n", "import torchvision.transforms as transforms\n", - "from torch.utils.data import DataLoader, random_split\n", - "from torchvision.datasets import CIFAR10\n", + "from datasets.utils.logging import disable_progress_bar\n", + "from torch.utils.data import DataLoader\n", "\n", "import flwr as fl\n", "from flwr.common import Metrics\n", + "from flwr_datasets import FederatedDataset\n", "\n", "DEVICE = torch.device(\"cpu\") # Try \"cuda\" to train on GPU\n", "print(\n", " f\"Training on {DEVICE} using PyTorch {torch.__version__} and Flower {fl.__version__}\"\n", - ")" + ")\n", + "disable_progress_bar()" ] }, { @@ -92,27 +93,7 @@ "\n", "### Loading the data\n", "\n", - "Federated learning can be applied to many different types of tasks across different domains. In this tutorial, we introduce federated learning by training a simple convolutional neural network (CNN) on the popular CIFAR-10 dataset. CIFAR-10 can be used to train image classifiers that distinguish between images from ten different classes:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "CLASSES = (\n", - " \"plane\",\n", - " \"car\",\n", - " \"bird\",\n", - " \"cat\",\n", - " \"deer\",\n", - " \"dog\",\n", - " \"frog\",\n", - " \"horse\",\n", - " \"ship\",\n", - " \"truck\",\n", - ")" + "Federated learning can be applied to many different types of tasks across different domains. In this tutorial, we introduce federated learning by training a simple convolutional neural network (CNN) on the popular CIFAR-10 dataset. CIFAR-10 can be used to train image classifiers that distinguish between images from ten different classes: 'airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', and 'truck'." ] }, { @@ -121,16 +102,7 @@ "source": [ "We simulate having multiple datasets from multiple organizations (also called the \"cross-silo\" setting in federated learning) by splitting the original CIFAR-10 dataset into multiple partitions. Each partition will represent the data from a single organization. We're doing this purely for experimentation purposes, in the real world there's no need for data splitting because each organization already has their own data (so the data is naturally partitioned).\n", "\n", - "Each organization will act as a client in the federated learning system. So having ten organizations participate in a federation means having ten clients connected to the federated learning server:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "NUM_CLIENTS = 10" + "Each organization will act as a client in the federated learning system. So having ten organizations participate in a federation means having ten clients connected to the federated learning server.\n" ] }, { @@ -138,7 +110,7 @@ "metadata": {}, "source": [ "\n", - "Let's now load the CIFAR-10 training and test set, partition them into ten smaller datasets (each split into training and validation set), and wrap the resulting partitions by creating a PyTorch `DataLoader` for each of them:" + "Let's now create the Federated Dataset abstraction that from `flwr-datasets` that partitions the CIFAR-10. We will create small training and test set for each edge device and wrap each of them into a PyTorch `DataLoader`:" ] }, { @@ -147,32 +119,36 @@ "metadata": {}, "outputs": [], "source": [ + "NUM_CLIENTS = 10\n", "BATCH_SIZE = 32\n", "\n", "\n", "def load_datasets():\n", - " # Download and transform CIFAR-10 (train and test)\n", - " transform = transforms.Compose(\n", - " [transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]\n", - " )\n", - " trainset = CIFAR10(\"./dataset\", train=True, download=True, transform=transform)\n", - " testset = CIFAR10(\"./dataset\", train=False, download=True, transform=transform)\n", - "\n", - " # Split training set into 10 partitions to simulate the individual dataset\n", - " partition_size = len(trainset) // NUM_CLIENTS\n", - " lengths = [partition_size] * NUM_CLIENTS\n", - " datasets = random_split(trainset, lengths, torch.Generator().manual_seed(42))\n", - "\n", - " # Split each partition into train/val and create DataLoader\n", + " fds = FederatedDataset(dataset=\"cifar10\", partitioners={\"train\": NUM_CLIENTS})\n", + "\n", + " def apply_transforms(batch):\n", + " # Instead of passing transforms to CIFAR10(..., transform=transform)\n", + " # we will use this function to dataset.with_transform(apply_transforms)\n", + " # The transforms object is exactly the same\n", + " transform = transforms.Compose(\n", + " [\n", + " transforms.ToTensor(),\n", + " transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n", + " ]\n", + " )\n", + " batch[\"img\"] = [transform(img) for img in batch[\"img\"]]\n", + " return batch\n", + "\n", + " # Create train/val for each partition and wrap it into DataLoader\n", " trainloaders = []\n", " valloaders = []\n", - " for ds in datasets:\n", - " len_val = len(ds) // 10 # 10 % validation set\n", - " len_train = len(ds) - len_val\n", - " lengths = [len_train, len_val]\n", - " ds_train, ds_val = random_split(ds, lengths, torch.Generator().manual_seed(42))\n", - " trainloaders.append(DataLoader(ds_train, batch_size=BATCH_SIZE, shuffle=True))\n", - " valloaders.append(DataLoader(ds_val, batch_size=BATCH_SIZE))\n", + " for partition_id in range(NUM_CLIENTS):\n", + " partition = fds.load_partition(partition_id, \"train\")\n", + " partition = partition.with_transform(apply_transforms)\n", + " partition = partition.train_test_split(train_size=0.8)\n", + " trainloaders.append(DataLoader(partition[\"train\"], batch_size=BATCH_SIZE))\n", + " valloaders.append(DataLoader(partition[\"test\"], batch_size=BATCH_SIZE))\n", + " testset = fds.load_full(\"test\").with_transform(apply_transforms)\n", " testloader = DataLoader(testset, batch_size=BATCH_SIZE)\n", " return trainloaders, valloaders, testloader\n", "\n", @@ -195,8 +171,8 @@ "metadata": {}, "outputs": [], "source": [ - "images, labels = next(iter(trainloaders[0]))\n", - "\n", + "batch = next(iter(trainloaders[0]))\n", + "images, labels = batch[\"img\"], batch[\"label\"]\n", "# Reshape and convert images to a NumPy array\n", "# matplotlib requires images with the shape (height, width, 3)\n", "images = images.permute(0, 2, 3, 1).numpy()\n", @@ -209,7 +185,7 @@ "# Loop over the images and plot them\n", "for i, ax in enumerate(axs.flat):\n", " ax.imshow(images[i])\n", - " ax.set_title(CLASSES[labels[i]])\n", + " ax.set_title(trainloaders[0].dataset.features[\"label\"].int2str([labels[i]])[0])\n", " ax.axis(\"off\")\n", "\n", "# Show the plot\n", @@ -294,8 +270,8 @@ " net.train()\n", " for epoch in range(epochs):\n", " correct, total, epoch_loss = 0, 0, 0.0\n", - " for images, labels in trainloader:\n", - " images, labels = images.to(DEVICE), labels.to(DEVICE)\n", + " for batch in trainloader:\n", + " images, labels = batch[\"img\"].to(DEVICE), batch[\"label\"].to(DEVICE)\n", " optimizer.zero_grad()\n", " outputs = net(images)\n", " loss = criterion(outputs, labels)\n", @@ -317,8 +293,8 @@ " correct, total, loss = 0, 0, 0.0\n", " net.eval()\n", " with torch.no_grad():\n", - " for images, labels in testloader:\n", - " images, labels = images.to(DEVICE), labels.to(DEVICE)\n", + " for batch in testloader:\n", + " images, labels = batch[\"img\"].to(DEVICE), batch[\"label\"].to(DEVICE)\n", " outputs = net(images)\n", " loss += criterion(outputs, labels).item()\n", " _, predicted = torch.max(outputs.data, 1)\n", @@ -477,7 +453,7 @@ " valloader = valloaders[int(cid)]\n", "\n", " # Create a single Flower client representing a single organization\n", - " return FlowerClient(net, trainloader, valloader)" + " return FlowerClient(net, trainloader, valloader).to_client()" ] }, { @@ -508,10 +484,14 @@ " min_available_clients=10, # Wait until all 10 clients are available\n", ")\n", "\n", - "# Specify client resources if you need GPU (defaults to 1 CPU and 0 GPU)\n", - "client_resources = None\n", + "# Specify the resources each of your clients need. By default, each \n", + "# client will be allocated 1x CPU and 0x CPUs\n", + "client_resources = {\"num_cpus\": 1, \"num_gpus\": 0.0}\n", "if DEVICE.type == \"cuda\":\n", - " client_resources = {\"num_gpus\": 1}\n", + " # here we are asigning an entire GPU for each client.\n", + " client_resources = {\"num_cpus\": 1, \"num_gpus\": 1.0}\n", + " # Refer to our documentation for more details about Flower Simulations\n", + " # and how to setup these `client_resources`.\n", "\n", "# Start simulation\n", "fl.simulation.start_simulation(\n", @@ -629,7 +609,7 @@ "\n", "There's a dedicated `#questions` channel if you need help, but we'd also love to hear who you are in `#introductions`!\n", "\n", - "The [Flower Federated Learning Tutorial - Part 2](https://flower.dev/docs/framework/tutorial-use-a-federated-learning-strategy-pytorch.html) goes into more depth about strategies and all the advanced things you can build with them." + "The [Flower Federated Learning Tutorial - Part 2](https://flower.dev/docs/framework/tutorial-use-a-federated-learning-strategy-pytorch.html) goes into more depth about strategies and all the advanced things you can build with them.\n" ] } ], @@ -640,7 +620,7 @@ "toc_visible": true }, "kernelspec": { - "display_name": "flower-3.7.12", + "display_name": "flwr", "language": "python", "name": "python3" }