Executor level setup and teardown methods

[venkatBala]

As of current develop, Covalent supports call_before and call_after hooks that run before and after the electron's core code. These hooks can be used to do different tasks before the electron execution begins (such as environment sanity checks, cleanups, file downloads etc).

Given the current way of doing this, the executor's execute method is provided with a Callable that contains within itself invocations of to the call_before and after hooks in a predetermined manner. If say we decorate an electron to be executed with a docker/singularity executor, the call_before and after hooks cannot be leveraged to say build a container from a provided docker file, upload the container image to a registry and other cleanup tasks since the execution will happen in a already running container.

A potential way to enhance functionality would be to also provide hooks for the executor themselves, say setup and teardown methods. Any setup and executor environment specific customizations can be handled in the setup method for instance... building a docker container from a given dockerfile which will then be used for the purposes of executing the electron. The teardown method can potentially be used to push the built image to a registry etc.

In case of SLURM, the setup hook can be used to get a resource allocation and the teardown can be use to relinquish the resources efficiently.

A simple way to do this would be to define three abstractmethods to the BaseExecutor

• setup
• run
• teardown

The setup method would be intended to be overloaded by the user to define their custom executor related setup actions (environment management, slurm resource allocations, transferring files from covalent server out to other destinations, building docker containers, compiling source code, spin up infrastructure)

The run method would be the one that would be the execution workhorse that actually runs the electron along with the electron level call before and call after hooks

The teardown method would be used for cleaning up the remote environment, say destroy provisioned infrastructure, upload/delete docker/singularity images etc.

Modifying the executor's templates this way would imply that these methods are invoked from the covalent server allowing more control on workflow/electron execution.

[venkatBala]

@scottwn @santoshkumarradha @wjcunningham7 thoughts?