Added logic to control which stages must be pushed down or skipped from SQL engine execution. #14061
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fernst
requested review from
tivv,
DJSagarAhire,
albertshau and
sanjanasandeep
|
|
fernst
force-pushed
the
add-options-to-control-sql-engine-stages
branch
from
6faf269 to
d89961f
Compare
tivv
reviewed
Feb 25, 2022
| protected Iterable<SparkCollectionRelationalEngine> getRelationalEngines(SparkCollection<Object> stageData) { | ||
| if (sqlEngineAdapter == null || !sqlEngineAdapter.supportsRelationalTranform() | ||
| || !(stageData instanceof SQLBackedCollection)) { | ||
| protected Iterable<SparkCollectionRelationalEngine> getRelationalEngines(String stageName, |
There was a problem hiding this comment.
Are there cases where stageSpec.getName() != stageName? Please add new params to javadoc
There was a problem hiding this comment.
Actually no, there are not. Let me update the code.
tivv
reviewed
Feb 25, 2022
|
|
||
| // If this stage is not pushed down and it's not in the included stages, we can skip relational transformation | ||
| // in the SQL engine. | ||
| if (!(stageData instanceof SQLBackedCollection || shouldForcePushToSQLEngine(stageName))) { |
There was a problem hiding this comment.
nit: I am wondering if it would be more readable as !A && !B vs !(A || B). It would more directly correspond to the comment.
fernst
force-pushed
the
add-options-to-control-sql-engine-stages
branch
from
d89961f to
8bfd279
Compare
|
@tivv I've added tests. |
fernst
force-pushed
the
add-options-to-control-sql-engine-stages
branch
from
8bfd279 to
7cef84c
Compare
tivv
reviewed
Feb 26, 2022
…om SQL engine execution. Added additional methods to ensure that input and output schemas for stages are supported by the SQL engine. This is useful when verifying if all input and output types in an aggregation are supported by the SQL engine.
fernst
force-pushed
the
add-options-to-control-sql-engine-stages
branch
from
7cef84c to
3ee9301
Compare
I've added logic to check if the sink stage is in the "Excluded" list before attempting to write to the sink. |
tivv
approved these changes
Feb 26, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added additional methods to ensure that input and output schemas for stages are supported by the SQL engine. This is useful when verifying if all input and output types in an aggregation are supported by the SQL engine.