Multi-Cloud Executor Infra/System Design

[Emmanuel289]

Provisioning Infrastructure for executors is the goal.

• The plan is to start with EC2 and then Kubernetes and then other types of executors.
• We also want provisioning tools for CI testing.
• Also, some ways to spin up and tear down the executor for testing purposes.
• Venkat mentioned a testing framework where we can run functional testing against the infrastructure
• The first option is using Terraform's HCL

Will: Directions for implementation:
Step 1- Build the individual executors.
Step 2- Decide on organization of folders after building.

Alejandro: Infra piece will live in the same repo as each executor as an MVP.

Sankalp: run() method eases creation of other kinds of executors by passing in args and the callable function.

You can also override the execute() method as well

You can subclass the BaseExecutor class and pass the run() method

Venkat: preferred method of inheriting from the BaseExecutor in the custom executor. Use run() or execute()

Will: It comes down to whether or not you need the other variables in execute(). It depends on the executor(). I have used run() in the Kubernetes executor.

Venkat: Can the run() method be modified?

Casey: A fresh executor is created for each task so tasks do not share executors.

Faiyaz: Are we making executors recyclable?

Casey: Right now, fresh executors are created for each dispatch, so they are not recyclable.

Sankalp: Run() should be as simple as possible, so should only take the function and the args/kwargs. So override a small part of execute() and reuse in run().

Venkat: Can you create run() as a partial function?

Casey: Santosh says we can create a quick PR to store dispatch_id and node_id in the executor attributes so that they're accessible to run(), basically what Sankalp is doing by hand now

Venkat: the EC2 executor need not be with terraform. It's better to use the boto3 sdk.

Alejandro: How complex is the infrastructure tool going to be?

Will: I'm using terraform to provision Kubernetes and some shell scripts to deploy the infra.

Venkat: Terraform has to be installed on the user's environment for the EC2 executor to work.

Alejandro: Use Terraform for Kubernetes and extend to basic executors like EC2 executor. User will need to have terraform installed.

Will: Users are not required to have terraform installed for Kubernetes, but will fail for EC2 executor if terraform is not installed

Venkat: Using terraform might be complex for an EC2 executor.
Will: Beyond the instance creation, other resources need to be provisioned, so it's better to use terraform, and also to support the short-term multi-cloud goal.

Venkat: Recommends using Pulumi for multi-cloud deployment because it has strong Pythonic abstractions.

Scott: First step is to get the EC2 executor working and validated. Next step is to decide on the provisioning tool
Terraform, Terraform CDK, or whatever tool that is out there.

Venkat: The choice of the infra should be decided along with the executor.

Scott: What does the tool look like?
Venkat: We are choosing a tool that can be used with all executors in different clouds.

Will: Does Pulumi support all cloud providers?
Venkat: Terraform might be more work than Pulumi in multi-cloud.
Scott: We need to think more about how the tooling will look like.

Will: Basically, for each executor, we have some code that can be used for provisioning and testing. Invoke terraform code in each executor. Some customization is needed for each executor to prevent breaking changes in executors when we do too much abstraction.

Scott: As long as it is a stable IaC tool with wide adoption, we can use it.

Will/Faiyaz: Documenting what it requires to provision an executor.

Will: Look into Fargate/Batch for extensive documentation.

Conclusion: General consensus is to use Terraform