[feature] Add support for dynamic DAGs

[anishasthana]

Feature Area

/area backend

What feature would you like to see?

I would like to see support for dynamic dags

What is the use case or pain point?

There are some use-cases that may necessitate creation of a undetermined number of component from within a component.

For example:

1. Have a training component running that has a variable number of checkpoints that may be created.
2. For each checkpoint that is created, trigger a component to run for evaluation of the checkpoint
3. The next component of the pipeline would wait until all "checkpoint evaluation" components have completed running, and do something.

Is there a workaround currently?

A somewhat hacky workaround using a queueing system as well as a pod monitoring events on the cluster.

cc @HumairAK

---

Love this idea? Give it a 👍.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HumairAK]

/remove-lifecycle stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HumairAK]

/remove-lifecycle stale

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[HumairAK]

/remove-lifecycle stale