Skip to content

Data layer #896

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 33 commits into from
May 6, 2025
Merged

Data layer #896

merged 33 commits into from
May 6, 2025

Conversation

spirali
Copy link
Collaborator

@spirali spirali commented Apr 25, 2025

This is experimental version of data layer (direct data transfers between tasks).

Main missing parts:

  • Stabilized user API (the current one is just designed for testing)
  • Some more advanced optimizations in scheduler
  • Some optimizations in the data transfer itself

Because of never-ending conflicts I want to merge in the current form.
Even we would eventually remove the data layer (That is relatively easy, as now the code fully separates data dependencies and task dependencies)
then it is still to worth to merge it as this set of commits contains various generic refactorings, fixes problems in scheduler, fixes problem in some tests.

Copy link
Collaborator

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through all the commits, it sort of makes sense to me from a high-level of view, although I of course couldn't scan every little change in a 4k diff :) I think that I found one memory leak, the rest are just nits.

Copy link
Collaborator

@Kobzol Kobzol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome to the data age!

@spirali spirali merged commit 13f0edb into main May 6, 2025
9 checks passed
@spirali spirali deleted the data-layer branch May 6, 2025 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants