Skip to content

Commit

Permalink
Ensure extra env vars are propagated to remap command (#936)
Browse files Browse the repository at this point in the history
*Issue #, if available:*

Fixes #848

*Description of changes:*

* Ensure the extra env vars are added to the remap launch command

### Testing 

Ran the [standalone
mode](https://graphstorm.readthedocs.io/en/v0.3/tutorials/quick-start.html)
example with extra parameter `--extra-envs
LD_LIBRARY_PATH="/opt/gs-venv/lib/python3.9/site-packages/dgl/:$LD_LIBRARY_PATH"`,
ensured that the command submitted by the launcher included the extra
env vars as:

```bash
cd /tmp/ogbn-arxiv-nc; (export LD_LIBRARY_PATH=/opt/gs-venv/lib/python3.9/site-packages/dgl/:; (export PYTHONPATH=/graphstorm/python/:/root/dgl/tools/::/root/dgl/python ; /opt/gs-venv/bin/python -m graphstorm.gconstruct.remap_result --cf /graphstorm/training_scripts/gsgnn_np/arxiv_nc.yaml --save-model-path /tmp/ogbn-arxiv-nc/models --save-embed-path /tmp/ogbn-arxiv-nc/embeddings --num-epochs 2 --ip-config /tmp/tmput45xgvc --part-config /tmp/ogbn_arxiv_nc_1p/ogbn-arxiv.json --rank 0 --world-size 1 --with-shared-fs True --num-processes 1 --output-chunk-size 100000 --preserve-input False))
```


By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
  • Loading branch information
thvasilo authored Jul 26, 2024
1 parent 81331f8 commit 8cbf6d5
Showing 1 changed file with 7 additions and 3 deletions.
10 changes: 7 additions & 3 deletions python/graphstorm/run/launch.py
Original file line number Diff line number Diff line change
Expand Up @@ -776,7 +776,7 @@ def wrap_cmd_with_local_envvars(cmd: str, env_vars: str) -> str:
return f"(export {env_vars}; {cmd})"


def wrap_cmd_with_extra_envvars(cmd: str, env_vars: list) -> str:
def wrap_cmd_with_extra_envvars(cmd: str, env_vars: List[str]) -> str:
"""Wraps a CLI command with extra env vars
Example:
Expand All @@ -795,8 +795,7 @@ def wrap_cmd_with_extra_envvars(cmd: str, env_vars: list) -> str:
Returns:
cmd_with_env_vars:
"""
env_vars = " ".join(env_vars)
return wrap_cmd_with_local_envvars(cmd, env_vars)
return wrap_cmd_with_local_envvars(cmd, " ".join(env_vars))

GLOBAL_GROUP_ID = 0

Expand Down Expand Up @@ -873,6 +872,11 @@ def submit_remap_jobs(args, udf_command, hosts, run_local):
args.preserve_input)

cmd = wrap_cmd_with_local_envvars(remap_dist_command, env_vars)
cmd = (
wrap_cmd_with_extra_envvars(cmd, args.extra_envs)
if len(args.extra_envs) > 0
else cmd
)

cmd = "cd " + str(args.workspace) + "; " + cmd
clients_cmd.append(cmd)
Expand Down

0 comments on commit 8cbf6d5

Please sign in to comment.