From 7f1416595cf6e449915ec9fcf13447a5111f0268 Mon Sep 17 00:00:00 2001 From: Yan Gao Date: Wed, 5 Feb 2025 17:55:40 +0800 Subject: [PATCH 1/2] refactor(benchmarks) Update dependencies for LLM evaluation pipelines (#4910) --- benchmarks/flowertune-llm/evaluation/code/README.md | 6 +++--- benchmarks/flowertune-llm/evaluation/code/requirements.txt | 6 +++--- benchmarks/flowertune-llm/evaluation/finance/README.md | 2 +- .../flowertune-llm/evaluation/finance/requirements.txt | 4 ++-- benchmarks/flowertune-llm/evaluation/general-nlp/README.md | 2 +- .../flowertune-llm/evaluation/general-nlp/requirements.txt | 4 ++-- benchmarks/flowertune-llm/evaluation/medical/README.md | 2 +- .../flowertune-llm/evaluation/medical/requirements.txt | 4 ++-- 8 files changed, 15 insertions(+), 15 deletions(-) diff --git a/benchmarks/flowertune-llm/evaluation/code/README.md b/benchmarks/flowertune-llm/evaluation/code/README.md index fd63ced2f1e2..6c98a46737cf 100644 --- a/benchmarks/flowertune-llm/evaluation/code/README.md +++ b/benchmarks/flowertune-llm/evaluation/code/README.md @@ -12,7 +12,7 @@ Three datasets have been selected for this evaluation: [MBPP](https://huggingfac git clone --depth=1 https://github.com/adap/flower.git && mv flower/benchmarks/flowertune-llm/evaluation/code ./flowertune-eval-code && rm -rf flower && cd flowertune-eval-code ``` -Create a new Python environment (we recommend Python 3.10), activate it, then install dependencies with: +Create a new Python environment (we recommend Python 3.11), activate it, then install dependencies with: ```shell # From a new python environment, run: @@ -40,7 +40,7 @@ sudo apt-get install g++ Then, download the `main.py` script from `bigcode-evaluation-harness` repository. ```shell -git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git && cd bigcode-evaluation-harness && git checkout 0f3e95f0806e78a4f432056cdb1be93604a51d69 && mv main.py ../ && cd .. && rm -rf bigcode-evaluation-harness +git clone https://github.com/yan-gao-GY/bigcode-evaluation-harness.git && cd bigcode-evaluation-harness && mv main.py ../ && cd .. && rm -rf bigcode-evaluation-harness ``` @@ -51,7 +51,7 @@ git clone https://github.com/bigcode-project/bigcode-evaluation-harness.git && c ```bash python main.py \ ---model=mistralai/Mistral-7B-v0.3 \ +--model=your-base-model-name \ # e.g., mistralai/Mistral-7B-v0.3 --peft_model=/path/to/fine-tuned-peft-model-dir/ \ # e.g., ./peft_1 --max_length_generation=1024 \ # change to 2048 when running mbpp --batch_size=4 \ diff --git a/benchmarks/flowertune-llm/evaluation/code/requirements.txt b/benchmarks/flowertune-llm/evaluation/code/requirements.txt index 5e6cb5418ca2..5367a62a86d0 100644 --- a/benchmarks/flowertune-llm/evaluation/code/requirements.txt +++ b/benchmarks/flowertune-llm/evaluation/code/requirements.txt @@ -1,8 +1,8 @@ -peft==0.6.2 +peft==0.14.0 datasets==2.20.0 evaluate==0.3.0 sentencepiece==0.2.0 protobuf==5.27.1 -bitsandbytes==0.45.0 +bitsandbytes==0.45.1 hf_transfer==0.1.8 -git+https://github.com/bigcode-project/bigcode-evaluation-harness.git@6116c6a9a5672c69bd624373cfbc8938b7acc249 +git+https://github.com/yan-gao-GY/bigcode-evaluation-harness.git diff --git a/benchmarks/flowertune-llm/evaluation/finance/README.md b/benchmarks/flowertune-llm/evaluation/finance/README.md index 15d2410b8ca4..18911becc032 100644 --- a/benchmarks/flowertune-llm/evaluation/finance/README.md +++ b/benchmarks/flowertune-llm/evaluation/finance/README.md @@ -10,7 +10,7 @@ Three datasets have been selected for this evaluation: [FPB](https://huggingface git clone --depth=1 https://github.com/adap/flower.git && mv flower/benchmarks/flowertune-llm/evaluation/finance ./flowertune-eval-finance && rm -rf flower && cd flowertune-eval-finance ``` -Create a new Python environment (we recommend Python 3.10), activate it, then install dependencies with: +Create a new Python environment (we recommend Python 3.11), activate it, then install dependencies with: ```shell # From a new python environment, run: diff --git a/benchmarks/flowertune-llm/evaluation/finance/requirements.txt b/benchmarks/flowertune-llm/evaluation/finance/requirements.txt index bbec80d10b9c..28ca5247d7cb 100644 --- a/benchmarks/flowertune-llm/evaluation/finance/requirements.txt +++ b/benchmarks/flowertune-llm/evaluation/finance/requirements.txt @@ -1,7 +1,7 @@ -peft==0.6.2 +peft==0.14.0 scikit-learn==1.5.0 datasets==2.20.0 sentencepiece==0.2.0 protobuf==5.27.1 -bitsandbytes==0.45.0 +bitsandbytes==0.45.1 hf_transfer==0.1.8 diff --git a/benchmarks/flowertune-llm/evaluation/general-nlp/README.md b/benchmarks/flowertune-llm/evaluation/general-nlp/README.md index 5acd75285dd3..02ff9f0aba23 100644 --- a/benchmarks/flowertune-llm/evaluation/general-nlp/README.md +++ b/benchmarks/flowertune-llm/evaluation/general-nlp/README.md @@ -10,7 +10,7 @@ The [MMLU](https://huggingface.co/datasets/lukaemon/mmlu) dataset is used for th git clone --depth=1 https://github.com/adap/flower.git && mv flower/benchmarks/flowertune-llm/evaluation/general-nlp ./flowertune-eval-general-nlp && rm -rf flower && cd flowertune-eval-general-nlp ``` -Create a new Python environment (we recommend Python 3.10), activate it, then install dependencies with: +Create a new Python environment (we recommend Python 3.11), activate it, then install dependencies with: ```shell # From a new python environment, run: diff --git a/benchmarks/flowertune-llm/evaluation/general-nlp/requirements.txt b/benchmarks/flowertune-llm/evaluation/general-nlp/requirements.txt index 3dae79d9af17..049f6ef4930b 100644 --- a/benchmarks/flowertune-llm/evaluation/general-nlp/requirements.txt +++ b/benchmarks/flowertune-llm/evaluation/general-nlp/requirements.txt @@ -1,8 +1,8 @@ -peft==0.6.2 +peft==0.14.0 pandas==2.2.2 scikit-learn==1.5.0 datasets==2.20.0 sentencepiece==0.2.0 protobuf==5.27.1 -bitsandbytes==0.45.0 +bitsandbytes==0.45.1 hf_transfer==0.1.8 diff --git a/benchmarks/flowertune-llm/evaluation/medical/README.md b/benchmarks/flowertune-llm/evaluation/medical/README.md index 6a519e8a7c54..1ca0fd6e2271 100644 --- a/benchmarks/flowertune-llm/evaluation/medical/README.md +++ b/benchmarks/flowertune-llm/evaluation/medical/README.md @@ -10,7 +10,7 @@ Three datasets have been selected for this evaluation: [PubMedQA](https://huggin git clone --depth=1 https://github.com/adap/flower.git && mv flower/benchmarks/flowertune-llm/evaluation/medical ./flowertune-eval-medical && rm -rf flower && cd flowertune-eval-medical ``` -Create a new Python environment (we recommend Python 3.10), activate it, then install dependencies with: +Create a new Python environment (we recommend Python 3.11), activate it, then install dependencies with: ```shell # From a new python environment, run: diff --git a/benchmarks/flowertune-llm/evaluation/medical/requirements.txt b/benchmarks/flowertune-llm/evaluation/medical/requirements.txt index 3dae79d9af17..049f6ef4930b 100644 --- a/benchmarks/flowertune-llm/evaluation/medical/requirements.txt +++ b/benchmarks/flowertune-llm/evaluation/medical/requirements.txt @@ -1,8 +1,8 @@ -peft==0.6.2 +peft==0.14.0 pandas==2.2.2 scikit-learn==1.5.0 datasets==2.20.0 sentencepiece==0.2.0 protobuf==5.27.1 -bitsandbytes==0.45.0 +bitsandbytes==0.45.1 hf_transfer==0.1.8 From 202d2b07acc47b7c5a071973848ffbf7a469c49c Mon Sep 17 00:00:00 2001 From: Javier Date: Wed, 5 Feb 2025 12:00:21 +0100 Subject: [PATCH 2/2] docs(framework) Expand how-to-authenticate-supernodes page (#4909) Co-authored-by: Heng Pan --- .../source/how-to-authenticate-supernodes.rst | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/framework/docs/source/how-to-authenticate-supernodes.rst b/framework/docs/source/how-to-authenticate-supernodes.rst index 9b1e5ad74d50..d986bfdb7d45 100644 --- a/framework/docs/source/how-to-authenticate-supernodes.rst +++ b/framework/docs/source/how-to-authenticate-supernodes.rst @@ -5,12 +5,22 @@ Authenticate SuperNodes ======================= -Flower has built-in support for authenticated SuperNodes, allowing you to verify the -identity of each SuperNode connecting to a SuperLink. To enhance security, node -authentication is only available when encrypted connections (SSL/TLS) are enabled. - -Flower's node authentication leverages a signature-based mechanism to verify each node's -identity: +When running a Flower Federation (see :doc:`ref-flower-network-communication`) it is +fundamental that an authentication mechanism is available between the SuperLink and the +SuperNodes that connect to it. Flower comes with two different mechanisms to +authenticate SuperNodes that connect to a running SuperLink: + +- **Automatic authentication**: In this mode, the SuperLink checks the timestamp-based + signature in each request from SuperNodes to prevent impersonation and replay attacks. +- **CSV-based authentication**: This mode functions similarly to automatic + authentication but requires the SuperLink to be provided with a list of authorized + public keys, allowing only those SuperNodes to connect. + +The automatic authentication mode works out of the box and therefore requires no +configuration. On the other hand, CSV-based authentication mode is more sophisticated +and how it works and how it can be used is presented reminder of this guide. Flower's +CSV-based node authentication leverages a signature-based mechanism to verify each +node's identity and is only available when encrypted connections (SSL/TLS) are enabled: - Each SuperNode must already possess a unique Elliptic Curve (EC) public/private key pair.