Skip to content

Rewriter to Visitor Transform Guide

Jump to bottom
lziq edited this page Nov 22, 2022 · 1 revision

Rewriter Upgrade → PIG VisitorTransform

General Procedure

(migrating FooRewriter → FooVisitorTransform)

  1. Make a copy of FooRewriter named FooVisitorTransform
  2. Similarly for the rewriter tests (if they exist) i.e. FooRewriterTests → FooVisitorTransformTests
  3. Make the FooVisitorTransform have a base class of PartiqlAst.VisitorTransform rather than AstRewriter
  4. Decide which code will require a upgrade to PIG
    1. Most commonly: anything involving ExprNode and ast.kt’s MetaContainer
  5. For any call to override function rewriteXXX, find the equivalent transformXXX method in partiql-domains.kt
    1. First you need to find the ExprNode type’s equivalent PartiqlAst type. You can look at how the ExprNode is converted to a PartiqlAst in ExprNodeToStatement.kt. Usually, this is a pretty simple 1:1 mapping, but there can be some ambiguities and differences between ExprNode and PartiqlAst. Some cases can be found in the “Common Conversions” section.
    2. After finding the type to convert, there should be a corresponding transform function in partiql-domains.kt.
    3. You may need to add extension functions to do the same as an ExprNode but using PartiqlAst. Add these to lang/domains/util.kt
    4. Sometimes there are some trickier conversions. These cases are found in the sections below:
      1. Inner rewrites
        1. innerRewriteSelect
        2. innerRewriteDataManipulation
      2. PartiqlAst.Expr copying data
  6. Write/translate tests, making sure they have a base class VisitorTransformTestBase
  7. Change the usages of FooRewriter to use FooVisitorTransform
  8. Once tests are passing and meet the desired standard, ensure existing code isn’t broken by the changes (run ./gradlew clean build)
  9. Remove all parts of the old rewriter code:
    1. Delete unused imports
    2. Find all usages of the old rewriter (e.g. code, comments, kdoc) and replace them with the visitor transform version
    3. Delete the migrated rewriter (and its tests)
    4. (Optional but recommended) within FooVisitorTransform and its tests change usages of the term rewriter/rewritten to transformer/transformed
  10. Rerun ./gradlew clean build to make sure everything works

Common Conversions

ExprNode PartiqlAst ExprNode Rewriter PartiqlAst VisitorTransform Notes
VariableReference Expr.Id rewriteVariableReference transformExprId
PathComponentExpr PathStep.PathExpr rewritePathComponent transformPathStepPathExpr
Literal Expr.Lit rewriteLiteral transformExprLit
NAryOp.CALL Expr.Call rewriteNAry, then case on NAryOp.CALL transformExprCall First arg in ExprNode is the function name, so removal of this isn't necessary for PartiqlAst
FromSourceJoin FromSource.Join rewriteFromSourceJoin transformFromSourceJoin
SelectProjectionPivot Projection.ProjectPivot rewriteSelectProjectionPivot transformProjectionProjectPivot
SelectProjectionValue Projection.ProjectValue rewriteSelectProjectionValue transformProjectionProjectValue
SelectProjectionList Projection.ProjectList rewriteSelectProjectionList transformProjectionProjectList
FromSourceLet FromSource.Scan and FromSource.Unpivot rewriteFromSourceLet transformFromSourceScan and transformFromSourceUnpivot PartiqlAst does not have the supertype w/ Scan and Unpivot
FromSourceExpr FromSource.Scan rewriteFromSourceExpr transformFromSourceScan
ScopeQualifier.LEXICAL ScopeQualifier.LocalsFirst ScopeQualifier.UNQUALIFIED is the same

Converting Metas

The constructor for metas is slightly different between ast’s MetaContainer and IonElement’s MetaContainer. Here is an example conversion:

metaContainerOf(UniqueNameMeta("foo"))

becomes

 metaContainerOf(UniqueNameMeta.TAG to UniqueNameMeta("foo"))

Generally, you do not want to keep ast’s MetaContainer and have to use the conversion function ( toPartiQlMetaContainer)

To add a new meta to an IonElement MetaContainer, you can use the withMeta(metaKey: String, metaValue: Any) function.

To add multiple metas, you will need to add two MetaContainers together:

metas = oldMetaContainer + metaContainerOf(newMeta)

Inner Rewrite Query

In the rewriter code, you may encounter innerRewriteSelect. This function calls the rewriter function for each clause of the SFW query. The direct equivalent in PIG is transformExprSelect. In most cases you should be able to just call transformExprSelect directly or the base class’s transform, super.transformExprSelect.

However, currently there are some slight differences between the order of transforms in innerRewriteSelect and transformExprSelect. innerRewriteSelect follows the traversal order (SQL semantic order) that is

FROM → (FROM LET) → (WHERE) → (GROUP BY) → (HAVING) → PROJECTION → (OFFSET) → (LIMIT)

transformExprSelect transforms the clauses in the order they are written for a PartiQL query, that is

PROJECTION → FROM → (FROM LET) → (WHERE) → (GROUP BY) → (HAVING) → (LIMIT) → (OFFSET)

This slight difference can lead to some different behaviors when translating innerRewriteSelect (e.g. StaticTypeRewriter). So to completely solve this issue, you can have your VisitorTransform implement VisitorTransformBase and call transformExprSelectEvaluationOrder rather than transformExprSelect to get the same behavior.

It's also worth noting that innerRewriteSelect and the visitor transform equivalent, transformExprSelectEvaluationOrder, can be used to avoid infinite recursion in the case of nested rewriter/transform instances.

fun transformExprSelectEvaluationOrder(node: PartiqlAst.Expr.Select): PartiqlAst.Expr {
    val from = transformExprSelect_from(node)
    val fromLet = transformExprSelect_fromLet(node)
    val where = transformExprSelect_where(node)
    val group = transformExprSelect_group(node)
    val having = transformExprSelect_having(node)
    val setq = transformExprSelect_setq(node)
    val project = transformExprSelect_project(node)
    val offset = transformExprSelect_offset(node)
    val limit = transformExprSelect_limit(node)
    val metas = transformExprSelect_metas(node)
    return PartiqlAst.build {
        PartiqlAst.Expr.Select(
            setq = setq,
            project = project,
            from = from,
            fromLet = fromLet,
            where = where,
            group = group,
            having = having,
            limit = limit,
            offset = offset,
            metas = metas)
    }
}

This also applies to innerRewriteDataManipulation, which has a slight different order than transformStatementDml.

(FROM) → (WHERE) → DML Operation vs. DML Operation → (FROM) → (WHERE)

To achieve the same behavior as the rewriter form, you can have your VisitorTransform implement VisitorTransformBase and call transformDataManipulationEvaluationOrder rather than transformStatementDml to get the same behavior.

It's also worth noting that innerRewriteDataManipulation and the visitor transform equivalent, transformDataManipulationEvaluationOrder, can be used to avoid infinite recursion in the case of nested rewriter/transform instances.

fun transformDataManipulationEvaluationOrder(node: PartiqlAst.Statement.Dml): PartiqlAst.Statement {
    val from = node.from?.let { transformFromSource(it) }
    val where = node.where?.let { transformStatementDml_where(node) }
    val dmlOperation = transformDmlOp(node.operation)
    val metas = transformMetas(node.metas)

    return PartiqlAst.build {
        dml(dmlOperation, from, where, metas)
    }
}

Copying Data from PartiqlAst.Expr

The PartiqlAst.Expr and all the subtypes don’t yet have a .copy function yet (will be added once copy() is released by PIG). This adds slightly more work when trying to create an instance of PartiqlAst.Expr from another. A workaround is to create a helper function as follows:

/**
  * Copies all parts of [PartiqlAst.Expr.Select] except [newProjection] for [PartiqlAst.Projection].
  */
private fun copyProjectionToSelect(node: PartiqlAst.Expr.Select, newProjection: PartiqlAst.Projection): PartiqlAst.Expr {
    return PartiqlAst.build {
        select(
            setq = node.setq,
            project = newProjection,
            from = node.from,
            fromLet = node.fromLet,
            where = node.where,
            group = node.group,
            having = node.having,
            limit = node.limit,
            offset = node.offset,
            metas = node.metas)
    }
}

Another annoyance you may run into is changing/copying metas for the generic PartiqlAst.Expr. This will be trivial once copyMetas() is released by PIG. A current workaround is mentioned here https://github.com/partiql/partiql-lang-kotlin/issues/311. You can create an inner class that overrides transformMetas.

inner class MetaVisitorTransform(private val newMetas: MetaContainer) : PartiqlAst.VisitorTransform() {
    override fun transformMetas(metas: MetaContainer): MetaContainer = newMetas
}

So in the ExprNode rewriters, rather than calling

node.copy(newMetaContainer)

For PartiqlAst transformers, you would call

MetaVisitorTransform(newMetaContainer).transformExpr(node)

Additional Notes:

  • Some checks that are part of ExprNode rewrites easier using PIG VisitorTransforms. These (non-exhaustively) include checks that select star projection isn’t included with other projection items and clause arity checks.
  • General NAry operations in ExprNode are easier to rewrite (like in PartiqlEvaluationRewriter.kt) than transforming in the PIG domain. This is because currently in PIG, NAry operations are directly PartiqlAst.Expr types. Once these operations are better modeled as NAry operations (https://github.com/partiql/partiql-lang-kotlin/issues/241), it will become easier to reuse code.
  • After making FooVisitorTransform (copied from FooVisitorRewriter), delete the old imports, so you can be sure previous ExprNode types aren’t used. This is helpful when ExprNode types and PIG domain have the same type names
  • There may be still be bugs in the conversion process between ExprNode and PartiqlAst (in ExprNodeToStatement.kt and StatementToExprNode.kt) that cause errors, especially when copying over the metas.

Sample Rewriter to VisitorTransform Conversions

Clone this wiki locally