Distributed execution PoC

[RobbeSneyders]

In order to allow distributed execution (run one component on multiple machines) with fondant we need to test and evaluate the current options.

A general solution that works for all orchestrators (current and future) seems not possible so we will need to tackle them one by one.

There seem to be dask examples (see epic) for all of them which might just be the cleanest solution where we create dask clusters dynamically where we can run the component code in parallel.

Vertex:

CustomContainerTrainingJob (1,2)

Tasks

Preview Give feedback

1. Setup and run the example pipeline
2. Inspect the IR YAML generated by the example pipeline
3. Manually change an IR YAML generated by a fondant pipeline to work distributed based on the example
4. Abstract all dask scheduler stuff so it can be reused (chief vs workers)
5. Clearly track and document infra changes that were required or limits we discovered
6. Identify and plot out changes needed to the current implementation

Docker-compose:

Docker swarm with multiple hosts and as dask cluster

Sagemaker:

BLOCKED NEEDS RUNNER (see issue), we have no sagemaker integration yet so the following is subject to change

Tasks

Preview Give feedback

1. Setup and run the example pipeline
2. Inspect the pipeline specification generated by the examples
3. Manually change an Sagemaker specification generated by a fondant pipeline to work distributed based on the example
4. Abstract all dask scheduler stuff so it can be reused (chief vs workers)
5. Clearly track and document infra changes that were required or limits we discovered
6. Identify and plot out changes needed to the current implementation