CG Workflow

Code Generation Workflow

Drill developers prepare for code generation by:

At runtime, code generation is done as follows:

• Each operator has a method to generate the query-specific operator implementation class.
• The code generation (CG) method creates an instance of CodeGenerator using both global and query-specific settings.
• The CodeGenerator gathers information about the new class including its name with is: org.apache.drill.exec.test.generated.CodeCompiler<TemplateName>Gen<serial-no> where serial-no is a number assigned to each new generated class from a specific template.
• Creates one or more ClassGenerator "that implement outer and inner classes associated with a particular runtime generated instance."
• The operator (batch) uses the ClassGenerator to build up the operator-specific bits of the class using information specific to this operator instance (specific columns, expressions and so on.)
• Code generation is heavily assisted by pre-defined templates and the HolderContainer class that represents a value vector Holder class.
• The DrillbitContext provides a CodeCompiler utility that generates and compiles the code.
• The CodeCompiler has an associated Google Guava LocalCache that holds the compiled classes. LocalCache looks up the code, and will call a "loader" object to create the compiled class if it is not found in the cache.
• If the code does not yet exist, the code is generated as follows.
• Invokes QueryClassLoader.getClassByteCode to compile the code.
• Selects the configured compiler (Janino or JDK).
• The generated code is compiled into byte codes.
• The compiled code is merged with template code. (Yes, the byte code from the generated class is combined, at the byte code level, with the template class to create a combined class for which there never was any source code. See this blog post and this paper for information.)