-
Notifications
You must be signed in to change notification settings - Fork 99
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve performance of sp_describe_undeclared_parameters #1905
Merged
Deepesh125
merged 7 commits into
babelfish-for-postgresql:BABEL_3_X_DEV
from
amazon-aurora:jira-babel-3705
Nov 20, 2023
Merged
Improve performance of sp_describe_undeclared_parameters #1905
Deepesh125
merged 7 commits into
babelfish-for-postgresql:BABEL_3_X_DEV
from
amazon-aurora:jira-babel-3705
Nov 20, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sumitj824
suggested changes
Oct 12, 2023
Change the title of PR to indicate that the performance improvement being done |
KushaalShroff
requested changes
Oct 25, 2023
Deepesh125
requested changes
Oct 25, 2023
cc3fb8d
to
815f3cd
Compare
Signed-off-by: Shameem Ahmed <[email protected]>
815f3cd
to
23a6f3f
Compare
Signed-off-by: Shameem Ahmed <[email protected]>
Deepesh125
reviewed
Nov 2, 2023
6 seconds is still a lot for a response time for 100 params |
KushaalShroff
approved these changes
Nov 20, 2023
Deepesh125
approved these changes
Nov 20, 2023
Deepesh125
approved these changes
Nov 20, 2023
ahmed-shameem
added a commit
to amazon-aurora/babelfish_extensions
that referenced
this pull request
Nov 21, 2023
…or-postgresql#1905) sp_describe_undeclared_parameters is slow. The issue is reported here: babelfish-for-postgresql#1317. Initially, we used to create a query in C for every parameter passed as an argument to sp_describe_undeclared_parameters and then execute it. This was taking huge amount of time as it used JOINS among multiple views like sys.objects, sys.columns, sys.types T. In this commit, we have replaced the use of such views as much as possible with pg catalogs. This aids to the improvement in performance. Task: BABEL-3705 Signed-off-by: Shameem Ahmed <[email protected]>
ahmed-shameem
added a commit
to amazon-aurora/babelfish_extensions
that referenced
this pull request
Nov 21, 2023
…or-postgresql#1905) sp_describe_undeclared_parameters is slow. The issue is reported here: babelfish-for-postgresql#1317. Initially, we used to create a query in C for every parameter passed as an argument to sp_describe_undeclared_parameters and then execute it. This was taking huge amount of time as it used JOINS among multiple views like sys.objects, sys.columns, sys.types T. In this commit, we have replaced the use of such views as much as possible with pg catalogs. This aids to the improvement in performance. Task: BABEL-3705 Signed-off-by: Shameem Ahmed <[email protected]>
shardgupta
pushed a commit
that referenced
this pull request
Nov 22, 2023
### Description sp_describe_undeclared_parameters is slow. The issue is reported here: #1317. Initially, we used to create a query in C for every parameter passed as an argument to sp_describe_undeclared_parameters and then execute it. This was taking huge amount of time as it used JOINS among multiple views like sys.objects, sys.columns, sys.types T. In this commit, we have replaced the use of such views as much as possible with pg catalogs. This aids to the improvement in performance. The changes are tested locally. <!--- Performance improvements (All the times are in ms): -------------- INSERT -------------- |No of Params|Before |After| |----------|------|--------| |1|1109|334| |2|1619|61| |8|6061|292| |10|7600|367| |20|15498|674| |30|23112|1027| |40|31076|1354| |50|39429|1779| |60|49043|2196| |70|55999|2497| |80|65040|2921| |90|73428|3251| |100|83244|3597| -------------- UPDATE -------------- |No of Params|Before |After| |----------|------|--------| |1|1681|204| |2|2323|99| |8|6919|323| |10|8406|407| |20|16280|713| |30|23911|1068| |40|31936|1398| |50|40249|1823| |60|49052|2232| |70|57128|2530| |80|65178|2890| |90|73632|3261| |100|83183|3634| -------------- DELETE -------------- |No of Params|Before |After| |----------|------|--------| |1|803|84| |2|1507|66| |8|6135|310| |10|7710|365| |20|15561|673| |30|23361|1032| |40|31125|1366| |50|39824|1791| |60|48933|2310| |70|57075|2514| |80|65444|2864| |90|73762|3249| |100|83991|3648| Query plans before modification: 1. INSERT: a. sys.columns: https://explain.dalibo.com/plan/033e354d56b2833e b. Complete plan: https://explain.dalibo.com/plan/fd4965765649d5b0 2. UPDATE: a. sys.columns: https://explain.dalibo.com/plan/28c1g72cfa4151ch b. Complete plan: https://explain.dalibo.com/plan/6bd40ah1b822084b 3. DELETE: a. sys.columns: https://explain.dalibo.com/plan/a3c6c3c387b48218 b. Complete plan: https://explain.dalibo.com/plan/2ge4425f5d3c9f7a Query plans after modification: 1. INSERT: https://explain.dalibo.com/plan/c96f94bge2632fa5 5. UPDATE: https://explain.dalibo.com/plan/2g8bd8feegfg4626 6. DELETE: https://explain.dalibo.com/plan/61g46gg71a853d7b QUERY PLAN AFTER MODIFICATION WITH EXPLICIT JOINS: 1. INSERT: https://explain.dalibo.com/plan/094bcg7de3bgg176 2. UPDATE: https://explain.dalibo.com/plan/4af4ba696e2hbdge 3. DELETE: https://explain.dalibo.com/plan/0c6356641a787394 ---> Performance improvements (All the times are in ms): -------------- INSERT -------------- | No of Params | Before | After | |--------------|-------|--------| | 1 | 1109 | 298 | | 2 | 1619 | 39 | | 8 | 6061 | 562 | | 10 | 7600 | 573 | | 20 | 15498 | 967 | | 30 | 23112 | 1170 | | 40 | 31076 | 1563 | | 50 | 39429 | 1745 | | 60 | 49043 | 2628 | | 70 | 55999 | 3318 | | 80 | 65040 | 4381 | | 90 | 73428 | 4978 | | 100 | 83244 | 6072 | -------------- UPDATE -------------- | No of Params | Before | After | |--------------|-------|--------| | 1 | 1681 | 177 | | 2 | 2323 | 55 | | 8 | 6919 | 556 | | 10 | 8406 | 594 | | 20 | 16280 | 976 | | 30 | 23911 | 1195 | | 40 | 31936 | 1581 | | 50 | 40249 | 1765 | | 60 | 49052 | 2606 | | 70 | 57128 | 3318 | | 80 | 65178 | 4365 | | 90 | 73632 | 5028 | | 100 | 83183 | 6102 | -------------- DELETE -------------- | No of Params | Before | After | |--------------|-------|--------| | 1 | 803 | 71 | | 2 | 1507 | 37 | | 8 | 6135 | 538 | | 10 | 7710 | 575 | | 20 | 15561 | 975 | | 30 | 23361 | 1173 | | 40 | 31125 | 1570 | | 50 | 39824 | 1772 | | 60 | 48933 | 2620 | | 70 | 57075 | 3362 | | 80 | 65444 | 4481 | | 90 | 73762 | 5127 | | 100 | 83991 | 6248 | QUERY PLAN AFTER MODIFICATION WITH EXPLICIT JOINS AND REMOVING UNNECESSARY CHECK ON t2.NAME: 1. INSERT: https://explain.dalibo.com/plan/dh97223b15e92h0d 2. UPDATE: https://explain.dalibo.com/plan/d28ec392d560b25c 3. DELETE: https://explain.dalibo.com/plan/348269ffgb6ae9g9 Task: BABEL-3705 Signed-off-by: Shameem Ahmed <[email protected]>
staticlibs
pushed a commit
to wiltondb/babelfish_extensions
that referenced
this pull request
Apr 12, 2024
…or-postgresql#1905) sp_describe_undeclared_parameters is slow. The issue is reported here: babelfish-for-postgresql#1317. Initially, we used to create a query in C for every parameter passed as an argument to sp_describe_undeclared_parameters and then execute it. This was taking huge amount of time as it used JOINS among multiple views like sys.objects, sys.columns, sys.types T. In this commit, we have replaced the use of such views as much as possible with pg catalogs. This aids to the improvement in performance. Task: BABEL-3705 Signed-off-by: Shameem Ahmed <[email protected]>
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
sp_describe_undeclared_parameters is slow. The issue is reported here: #1317.
Initially, we used to create a query in C for every parameter passed as an argument to sp_describe_undeclared_parameters and then execute it. This was taking huge amount of time as it used JOINS among multiple views like sys.objects, sys.columns, sys.types T. In this commit, we have replaced the use of such views as much as possible with pg catalogs. This aids to the improvement in performance.
The changes are tested locally.
Performance improvements (All the times are in ms):
-------------- INSERT --------------
Comparison on INSERT
-------------- UPDATE --------------
Comparison on UPDATE
-------------- DELETE --------------
Comparison on DELETE
Approx Linear increment in execution time based on parameter size
QUERY PLAN AFTER MODIFICATION WITH EXPLICIT JOINS AND REMOVING UNNECESSARY CHECK ON t2.NAME:
Task: BABEL-3705
Signed-off-by: Shameem Ahmed [email protected]
Test Scenarios Covered
Use case based -
Boundary conditions -
Arbitrary inputs -
Negative test cases -
Minor version upgrade tests -
Major version upgrade tests -
Performance tests -
Tooling impact -
Client tests -
Check List
By submitting this pull request, I confirm that my contribution is under the terms of the Apache 2.0 and PostgreSQL licenses, and grant any person obtaining a copy of the contribution permission to relicense all or a portion of my contribution to the PostgreSQL License solely to contribute all or a portion of my contribution to the PostgreSQL open source project.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.