Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for non-linear DAGs #557

Open
RobbeSneyders opened this issue Oct 27, 2023 · 1 comment
Open

Support for non-linear DAGs #557

RobbeSneyders opened this issue Oct 27, 2023 · 1 comment
Labels
Core Core framework

Comments

@RobbeSneyders
Copy link
Member

We currently only support non-linear DAGs, which limits the expressiveness of Fondant pipelines.

To support non-linear DAGs, we would need at least the following functionality:

  • Compiling to non-linear DAGs in the underlying runners
  • The ability to merge manifests from multiple branches to one

Especially the last one is complex.

  • We probably cannot assume that the partitions will be aligned
  • Which merge strategy to apply? Inner, outer, left join?
@RobbeSneyders RobbeSneyders converted this from a draft issue Oct 27, 2023
@PhilippeMoussalli
Copy link
Contributor

I feel like this feature is needed but would not really prioritize it at the moment unless it's currently blocking us/other users from implementing a given pipeline

@RobbeSneyders RobbeSneyders added the Core Core framework label Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Core Core framework
Projects
Status: Breakdown
Development

No branches or pull requests

2 participants