[documentation] Example usage help needed

[srmnitc]

Thanks for this nice package

I want to run a command line tool with the following run command mpiexec -n 4 tool_name inputfile from a jupyter notebook on my workstation. Is executorlib the right tool for this? If so, would it be possible to give a small example on how this can be run?

Thanks!

[jan-janssen]

executorlib is primarily designed to execute python functions, not so much external executables. Still you can do something like:

import subprocess
from executorlib import Executor


def run_external():
    return subprocess.check_output("mpiexec -n 4 hostname", shell=True)


with Executor(max_cores=4, backend="local") as exe:
    future = exe.submit(run_external, resource_dict={"cores": 1})
    print(future.result())

The local backend does not keep track of the tasks so it is primarily a workaround, for flux you can keep track of the tasks using the "threads_per_core": 4 parameter:

import subprocess
import flux.job
from executorlib import Executor


def run_external():
    return subprocess.check_output("mpiexec -n 4 hostname", shell=True)


with flux.job.FluxExecutor() as flux_exe:
    with Executor(max_cores=4, executor=flux_exe) as exe:
        future = exe.submit(run_external, resource_dict={"threads_per_core": 4})
        print(future.result())

It is important that we define threads rather than cores as we only want to start one python process and this python process run_external() then starts the other processes. The same can be done with SLURM https://executorlib.readthedocs.io/en/latest/examples.html#slurm-job-scheduler

Finally, on flux to validate your resource usage you can use:

flux jobs -a --format="{id} {queue} {username} {name} {status_abbrev} {ncores} {ntasks} {nnodes} {contextual_time}"

This should print something like:

JOBID QUEUE USER NAME ST NCORES NTASKS NNODES TIME
ƒ39FNW17D  janssen python CD 4 1 1 0.5879337787628174

[jan-janssen]

I added a separate section in the updated documentation to discuss the interface to external executables:
https://executorlib.readthedocs.io/en/latest/4-developer.html#external-executables