Update ci cd from 2.4 (NVIDIA#2520)

* Update github actions (NVIDIA#2450) * Fix premerge (NVIDIA#2467) * Fix issues on hello-world TF2 notebook * Fix tf integration test (NVIDIA#2504) * Add client api integration tests --------- Co-authored-by: Isaac Yang <[email protected]> Co-authored-by: Sean Yang <[email protected]>
nvidianz · Apr 19, 2024 · 2eed407 · 2eed407
1 parent d4afbee
commit 2eed407
Show file tree

Hide file tree

Showing 68 changed files with 3,645 additions and 62 deletions.
diff --git a/.github/workflows/blossom-ci.yml b/.github/workflows/blossom-ci.yml
@@ -74,7 +74,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
         with:
           repository: ${{ fromJson(needs.Authorization.outputs.args).repo }}
           ref: ${{ fromJson(needs.Authorization.outputs.args).ref }}

diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml
@@ -36,7 +36,7 @@ jobs:
 
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
 
     # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL

diff --git a/.github/workflows/markdown-links-check.yml b/.github/workflows/markdown-links-check.yml
@@ -23,7 +23,7 @@ jobs:
   markdown-link-check:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@master
+    - uses: actions/checkout@v4
     - uses: gaurav-nelson/[email protected]
       with:
         max-depth: -1

diff --git a/.github/workflows/premerge.yml b/.github/workflows/premerge.yml
@@ -29,15 +29,15 @@ jobs:
         os: [ ubuntu-22.04, ubuntu-20.04 ]
         python-version: [ "3.8", "3.9", "3.10" ]
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install -e .[dev]
+          python3 -m pip install --upgrade pip
+          python3 -m pip install --no-cache-dir -e .[dev]
       - name: Run unit test
         run: ./runtest.sh
 
@@ -49,15 +49,15 @@ jobs:
         os: [ ubuntu-22.04, ubuntu-20.04 ]
         python-version: [ "3.8", "3.9", "3.10" ]
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install dependencies
         run: |
-          python -m pip install --upgrade pip
-          pip install -e .[dev]
-          pip install build twine torch torchvision
+          python3 -m pip install --upgrade pip
+          python3 -m pip install --no-cache-dir -e .[dev]
+          python3 -m pip install --no-cache-dir build twine torch torchvision
       - name: Run wheel build
         run: python3 -m build --wheel
diff --git a/examples/hello-world/hello-tf2/jobs/hello-tf2/app/custom/trainer.py b/examples/hello-world/hello-tf2/jobs/hello-tf2/app/custom/trainer.py
@@ -12,6 +12,10 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import os
+
+os.environ["TF_FORCE_GPU_ALLOW_GROWTH"] = "true"
+
 import numpy as np
 import tensorflow as tf
 from tf2_net import Net

diff --git a/examples/hello-world/hello_world.ipynb b/examples/hello-world/hello_world.ipynb
@@ -738,8 +738,9 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
-   "id": "425c7f1d-7cb6-4602-bb88-4b87ff517529",
+   "id": "696a384e-f6da-4044-a7b9-db4c464f52ac",
    "metadata": {},
    "source": [
     "#### Running Tensorflow on local host with GPU \n",
@@ -748,53 +749,13 @@
     "We are running with 1 server, 2 sites in a local machine, which means three process involved for this federated training. \n",
     "If the local host has GPU, you might enter OOM error, due to the way Tensorflow consumes GPU memory. By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. If one has multiple process, some of the process will be OOM. To avoid multiple processes grabbing all GPU memory in TF, use the options described in [Limiting GPU memory growth]( https://www.tensorflow.org/guide/gpu#limiting_gpu_memory_growth). \n",
     "\n",
-    "In our cases,  we prefer that the process only allocates a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this. \n",
+    "In our cases,  we prefer that the process only allocates a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this, as described in the above link.\n",
+    "\n",
+    "In this example, we explictly set the environment varialble `TF_FORCE_GPU_ALLOW_GROWTH` to `true` at the very beginning of the trainer.py file, which runs in the clients and will allocate GPU memory for training.  With the env var been set, TF will not grab the entire GPU memory and will not cause GPU OOM error when running POC on local host.\n",
+    "\n",
+    "Note that setting the env var `TF_FORCE_GPU_ALLOW_GROWTH` inside this notebook takes no effect because the clients of POC have already started and their env vars are set at the starting time.\n",
+    "\n",
     "\n",
-    "The First method is set the environmental variable TF_FORCE_GPU_ALLOW_GROWTH to true. This configuration is platform specific. \n",
-    "The 2nd method is using the piece of code below"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "026ce33a-90b1-4ef7-8c7c-f25722f2d2ae",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "%env TF_FORCE_GPU_ALLOW_GROWTH=true"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "91cf84ec-57f1-439f-b72b-b33fea1a7f7c",
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "import tensorflow as tf\n",
-    "gpus = tf.config.list_physical_devices('GPU')\n",
-    "if gpus:\n",
-    "  # Restrict TensorFlow to only allocate 1GB of memory on the first GPU\n",
-    "  try:\n",
-    "    tf.config.set_logical_device_configuration(\n",
-    "        gpus[0],\n",
-    "        [tf.config.LogicalDeviceConfiguration(memory_limit=1024)])\n",
-    "    logical_gpus = tf.config.list_logical_devices('GPU')\n",
-    "    print(len(gpus), \"Physical GPUs,\", len(logical_gpus), \"Logical GPUs\")\n",
-    "  except RuntimeError as e:\n",
-    "    # Virtual devices must be set before GPUs have been initialized\n",
-    "    print(e)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "0c3f1149-d54d-4f3b-b497-c06deba22fef",
-   "metadata": {},
-   "source": [
     "### 1. Submit job using FLARE API\n",
     "\n",
     "Starting a FLARE API session and submit the hello-tf2 job\n",
@@ -833,7 +794,7 @@
    },
    "outputs": [],
    "source": [
-    "! tail -100 /tmp/nvflare/poc/server/log.txt"
+    "! tail -100 /tmp/nvflare/poc/example_project/prod_00/server/log.txt"
    ]
   },
   {
@@ -977,7 +938,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.17"
+   "version": "3.8.15"
   },
   "vscode": {
    "interpreter": {

diff --git a/tests/integration_test/data/jobs/np_loop/app_client/config/config_fed_client.conf b/tests/integration_test/data/jobs/np_loop/app_client/config/config_fed_client.conf
@@ -0,0 +1,43 @@
+{
+  format_version = 2
+  app_script = "train_loop.py"
+  app_config = ""
+  executors = [
+    {
+      tasks = [
+        "train"
+      ]
+      executor {
+        path = "nvflare.app_common.executors.client_api_launcher_executor.ClientAPILauncherExecutor"
+        args {
+          launcher_id = "launcher"
+          pipe_id = "pipe"
+          heartbeat_timeout = 60
+          params_exchange_format = "numpy"
+          params_transfer_type = "FULL"
+          train_with_evaluation = true
+        }
+      }
+    }
+  ]
+  task_data_filters = []
+  task_result_filters = []
+  components = [
+    {
+      id = "launcher"
+      path = "nvflare.app_common.launchers.subprocess_launcher.SubprocessLauncher"
+      args {
+        script = "python3 custom/{app_script}  {app_config} "
+        launch_once = true
+      }
+    }
+    {
+      id = "pipe"
+      path = "nvflare.fuel.utils.pipe.file_pipe.FilePipe"
+      args {
+        mode = "PASSIVE"
+        root_path = "{WORKSPACE}/{JOB_ID}/{SITE_NAME}"
+      }
+    }
+  ]
+}
diff --git a/tests/integration_test/data/jobs/np_loop/app_client/custom/train_diff.py b/tests/integration_test/data/jobs/np_loop/app_client/custom/train_diff.py
@@ -0,0 +1,60 @@
+# Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+
+import nvflare.client as flare
+
+
+def train(input_arr):
+    output_arr = copy.deepcopy(input_arr)
+    # mock training with plus 1
+    return output_arr + 1
+
+
+def evaluate(input_arr):
+    # mock evaluation metrics
+    return 100
+
+
+def main():
+    # initializes NVFlare interface
+    flare.init()
+
+    # get model from NVFlare
+    input_model = flare.receive()
+    print(f"received weights is: {input_model.params}")
+
+    # get system information
+    sys_info = flare.system_info()
+    print(f"system info is: {sys_info}")
+
+    input_numpy_array = input_model.params["numpy_key"]
+
+    # training
+    output_numpy_array = train(input_numpy_array)
+
+    # evaluation
+    metrics = evaluate(input_numpy_array)
+
+    # calculate difference here
+    diff = output_numpy_array - input_numpy_array
+
+    # send back the model difference
+    print(f"send back: {diff}")
+    flare.send(flare.FLModel(params={"numpy_key": diff}, params_type="DIFF", metrics={"accuracy": metrics}))
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/integration_test/data/jobs/np_loop/app_client/custom/train_full.py b/tests/integration_test/data/jobs/np_loop/app_client/custom/train_full.py
@@ -0,0 +1,59 @@
+# Copyright (c) 2024, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+
+import nvflare.client as flare
+
+
+def train(input_arr):
+    output_arr = copy.deepcopy(input_arr)
+    # mock training with plus 1
+    return output_arr + 1
+
+
+def evaluate(input_arr):
+    # mock evaluation metrics
+    return 100
+
+
+def main():
+    # initializes NVFlare interface
+    flare.init()
+
+    # get model from NVFlare
+    input_model = flare.receive()
+    print(f"received weights is: {input_model.params}")
+
+    # get system information
+    sys_info = flare.system_info()
+    print(f"system info is: {sys_info}")
+
+    input_numpy_array = input_model.params["numpy_key"]
+
+    # training
+    output_numpy_array = train(input_numpy_array)
+
+    # evaluation
+    metrics = evaluate(input_numpy_array)
+
+    # send back the model
+    print(f"send back: {output_numpy_array}")
+    flare.send(
+        flare.FLModel(params={"numpy_key": output_numpy_array}, params_type="FULL", metrics={"accuracy": metrics})
+    )
+
+
+if __name__ == "__main__":
+    main()