[Q&A] Paths are not found when submitting job

[irisv10]

Python version (python3 -V)

3.8

NVFlare version

2.3

Operating system

Ubuntu 20.04

Please describe your question

I am trying to submit a job using 2 clients and a server. I managed to run the simulations from this example: https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/prostate/prostate_2D.

At the moment, I am trying to run this in a real-world setting. But when I submit the job (c61e73ed-080a-4353-ae55-ba49d41e414b) to the clients and server, some of the paths are not found:
For the clients:
nvflare.fuel.common.excepts.ConfigError: Error processing ['/home/ubuntu/run_5/site_1/startup/../c61e73ed-080a-4353-ae55-ba49d41e414b/app_site_1/config/config_fed_client.json'] in JSON element {'id': 'learner', 'path': 'pt.learners.supervised_monai_learner.SupervisedMonaiLearner', 'args': {'train_config_filename': 'config_train.json', 'aggregation_epochs': 1}}: path: components.#1, exception: No module named 'pt'
and for the server:
nvflare.fuel.common.excepts.ConfigError: Config error in ['/home/ubuntu/run_5/server/startup/../c61e73ed-080a-4353-ae55-ba49d41e414b/app_server/config/config_fed_server.json']: Error processing ['/home/ubuntu/run_5/server/startup/../c61e73ed-080a-4353-ae55-ba49d41e414b/app_server/config/config_fed_server.json'] in JSON element {'id': 'persistor', 'name': 'PTFileModelPersistor', 'args': {'model': {'path': 'monai.networks.nets.unet.UNet', 'args': {'spatial_dims': 2, 'in_channels': 1, 'out_channels': 1, 'channels': [16, 32, 64, 128, 256], 'strides': [2, 2, 2, 2], 'num_res_units': 2}}}}: path: components.#1, exception: failed to instantiate class: No module named 'monai'

The job folder is located in the transfer folder from the admin:

- transfer
  - experiment_1
    - config
      - config_fed_client.json
      - config_fed_server.json
      - config_train.json
    - pt
     - learners
     - utils

Does anyone know how to set the correct python paths for the clients and server?

[chesterxgchen]

I think you need to set the PYTHONPATH to include pt directory in order for python to find the pt. @ZiyueXu77

[irisv10]

When I generate a new virtual environment with nvflare and monai, I do not run into this error..

[chesterxgchen]

1. You shouldn't need to modify the start.sh file for the PYTHONPATH, all you need to do is something like
   export PYTHONPATH=$PYTHONPATH:<your_path> or you can put the path in from the origin PYTHONPATH
2. THe monai is medical imaging training lib. so it usually installed in the client side not server side
3. to make monai + Flare work together, there is monai_nvflare wheel you need to install. see the example under integration/monai

[irisv10]

Thanks for your response! I still do not get when and where I can export the pythonpath?
When submitting a job from the admin console:

• A new job folder is created at the clients sides, this is the pythonpath.
• The job will immediately run, so I cannot access the clients console anymore to export the new path.
• It is not possible to set it beforehand because the pythonpath does not exist yet.

[chesterxgchen]

check out this example with PYTHONPATH.
https://github.com/NVIDIA/NVFlare/blob/main/examples/advanced/cifar10/cifar10-real-world/README.md

[YuanTingHsieh]

@irisv10 don't worry about the job path, it will only be generated after a job is submitted.

I notice you are using the wrong branch.
Since you are using NVFlare version 2.3, you should use example from 2.3 branch: https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/prostate

For the "real-world" setup, you need to setup the correct PYTHONPATH for every machine.
So, in your case, every machine you want to run either server or client you need to:

1. install nvflare and monai and nvflare_monai package
2. (optional) setup PYTHONPATH using: export PYTHONPATH=$PYTHONPATH:<path to the codes you want to use>