[Q&A] Jobs not being submitted to the correct server when hosting two servers on the same machine

[dima1997]

Python version (python3 -V)

3.8.0

NVFlare version (python3 -m pip list | grep "nvflare")

2.3.8

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

No response

Operating system

Ubuntu 18.04.3 LTS

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Description

Hello,

I'm trying to set up two NVIDIA FLARE servers on the same physical machine, but jobs submitted to one server are submitted to the other, which then fails due to job signature verification.

I have configured two NVIDIA FLARE servers on the same physical machine with the same FQDN name but with different associated ports. Let's call them server A and B. Server A uses ports 8004 as feed_learn_port and port 8005 as admin_port, while server B uses ports 8020 and 8021, respectively. Server A's clients run on the same physical machine as Servers A and B, while Server B's clients run on two other machines.

Every time I try to submit a job to Server B, through its admin Client Line Interface (CLI), it does not reach Server B, since nothing appears in its logs, but it reaches Server A where it fails due to job signature verification :

DefaultJobScheduler - INFO - [identity=my-project-A, run=?]: Try to schedule job c14323f5-ced0-4ab6-b57c-fad40017ea55, get result: (scheduled).
JobRunner - INFO - [identity=my-project-A, run=?]: Got the job: c14323f5-ced0-4ab6-b57c-fad40017ea55 from the scheduler to run
JobRunner - ERROR - [identity=my-project-A, run=?]: Failed to run the Job (c14323f5-ced0-4ab6-b57c-fad40017ea55): RuntimeError: Failed to verify app 'my-app': job signature verification failed

In server B admin JSON, the overseer_agent has the correct sp_end_point set as:

"FQDN-Name:8020:8021"

Could you help me understand why server B jobs are sent to server A and how to fix this problem, please ?

Thanks in advance,

DI MARIA, Franco Martin

Environment

Nvflare Version: 2.3.8
GPU Type: Quadro RTX 6000
Nvidia Driver Version: 460.80
CUDA Version: 11.2
CUDNN Version: 7.6.5
Operating System + Version: Ubuntu 18.04.3 LTS
Python Version (if applicable): 3.8.0

[YuanTingHsieh]

@dima1997 thanks for the question!

Our job storage by default is stored inside "/tmp/nvflare/jobs-storage".

This is configured in here: https://github.com/NVIDIA/NVFlare/blob/2.3.8/nvflare/lighter/impl/master_template.yml#L200-L220

So for your case, if you want to "run two NVIDIA FLARE servers on the same physical machine with the same FQDN name but with different associated ports."

You need to modify the "project.yml" to read in a different version of "master_template.yml" before you do the provision for the second server.
Otherwise, this two server will be using the same directory for the job management thus you see the failure.

So for example, copy and modify the "master_template.yml" to be "my_template.yml":

{
              "id": "job_manager",
              "path": "nvflare.apis.impl.job_def_manager.SimpleJobDefManager",
              "args": {
                  "uri_root": "/tmp/nvflare/jobs-storage-2",
                  "job_store_id": "job_store"
              }
          },
          {
              "id": "job_store",
              "path": "nvflare.app_common.storages.filesystem_storage.FilesystemStorage"
          }

Then in the my_project.yml:

# The same methods in all builders are called in their order defined in builders section
builders:
  - path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
      template_file: my_template.yml

Finally do the provision:
nvflare provision -p my_project.yml

@IsaacYangSLA @SYangster maybe we want to add this to our documentation.

[dima1997]

That worked for me ! Thank you very much !