Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Heuristic wrapper fix #150

Merged
merged 8 commits into from
Apr 22, 2024
Merged

Fix: Heuristic wrapper fix #150

merged 8 commits into from
Apr 22, 2024

Conversation

AveryQi115
Copy link
Contributor

The flow of the optimization is:

OptimizeGroup(root) --> exploreGroup(children)(can only apply logical->logical) --> after exploration, there are physical exprs in the root --> optimizeInput --> optimizeGroup(children)(can apply logical to physical)

While marking the rules as deadend, we cannot mark the impl rules as fired as we might need them to start the optimizeGroup task of the children node after merging of groups even if we know the expr can never be the winner.

@AveryQi115 AveryQi115 requested a review from skyzh April 8, 2024 20:42
@AveryQi115
Copy link
Contributor Author

After some debugging, I found merge group is ok as long as the two corresponding groups are still having optimizeGroup task after the merge. However, optimizeGroup for children nodes are called by optimizeInput, so the ancestors of the children node cannot be pruned. There might be some problems for merge group in the future if we introduce pruning. @skyzh

If the above example is too abstract, you can try Select a from (select a from (select a from t)) with MergeProjection rules registered as heuristics to see the flow of tasks.

Copy link
Contributor

@Sweetsuro Sweetsuro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I know you're waiting for Chi though

@AveryQi115
Copy link
Contributor Author

LGTM - I know you're waiting for Chi though

It's ok. Let's merge it first.

Copy link
Member

@skyzh skyzh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as it only affects the heuristics-in-cascades code path. In theory, no heuristics rules in cascades should directly replace a plan node (it should only do expression optimizations), and I believe this will cause more problems for us in the future...

self.fired_rules
.entry(expr_id)
.and_modify(|fired_rules| fired_rules.clear());
self.fired_rules.entry(expr_id).or_default().clear();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this creates a new entry for every expr id?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, for the expr id which is used in this function.

@AveryQi115 AveryQi115 merged commit 7574267 into main Apr 22, 2024
1 check passed
@AveryQi115 AveryQi115 deleted the heuristic_wrapper_fix branch April 22, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants