Need for a Covalent DockerExecutor

[venkatBala]

As executors in Covalent are designed to be general purpose and configurable the need for having custom executors fit for specific purposes is a must. For instance, local and Dask are supported out of the box and all workflows and they will work as long as all the Python dependencies of the workflow are installed in the virtual/conda environment. However there would be special use cases where a user would want to test/prototype their workflows/electrons against different version of a package concurrently. This would be a great addition to the UX in Covalent and can be easily supported by a DockerExecutor. Following is a brief list highlighting the benefits for supporting such an executor

• quickly prototype and test custom docker images with all the necessary python dependencies for a task
• Can leverage specialized containers *GPU optimized machine learning, HPC) provided by NVIDIA through their container registry (NGC)
• Can run workloads locally using Docker executors and easily scale up to cloud using cloud executors as Docker is the default backend
• Users can provide their own specialized images optimized for their workloads
• Developers can quickly prototype locally using dev containers and run them in production at scale on cloud
• Electron resource management can also be quite easily in Covalent as docker support resource management for all tasks running with it out of the box

@santoshkumarradha @wjcunningham7

[FyzHsn]

@venkatBala Can I get a bit more detail on the DockerExecutor:

1. What are the inputs when instantiating the executor?
2. Can you sketch how the executor would function from instantiation to finishing execution of a task?
3. What resources need to be provisioned/allocated for this executor to work?
4. Lastly, could you discuss how this executor functionality is currently not provided by any other executors? A bit more clarification on the sentence about the python dependencies would be great!