Gradle build integration

Gradle integration is provided as part of pegasus. The gradle integration uses the below java classes, included in pegasus to perform the various codegen and validation tasks. These java classes could be used to integrate with other build tools.

Gradle

As an example, let's consider a simple rest.li project with three modules:

• An /api module containing pegasus schema definitions in the src/main/pegasus directory. Also this is where java client bindings for the service will be generated (the client-bindings are sometimes not a separate project, but are put into the '/api' project along with the .pdsc's).
• A /server module containing resources defined in java classes in the src/main/java directory under the com.linkedin.restli.example.impl namespace (E.g. com.linkedin.restli.example.impl.RestLiExampleBasicServer.java).
• An example java client that uses the client-bindings.

Root build.gradle

/build.gradle:

apply plugin: 'idea'
apply plugin: 'eclipse'

project.ext.externalDependency = [
  'pegasusVersion' : '<version>'
]

buildscript {
  repositories {
    mavenLocal()
    mavenCentral()
  }
  dependencies {
    classpath group: 'com.linkedin.pegasus', name: 'gradle-plugins', version: '<version>'
  }
}

subprojects {
  apply plugin: 'eclipse'
  apply plugin: 'maven'
  apply plugin: 'idea'

  sourceCompatibility = JavaVersion.VERSION_1_6

  repositories {
    mavenLocal()
    mavenCentral()
  }
}

/settings.gradle:

include 'api'
include 'server'
include 'client'

build.gradle for Data API Project

/api/build.gradle:

apply plugin: 'java'
apply plugin: 'pegasus'

dependencies {
  dataTemplateCompile "com.linkedin.pegasus:data:<version>"
  dataTemplateGenerator "com.linkedin.pegasus:generator:<version>"
  restTools "com.linkedin.pegasus:restli-tools:<version>"
  restClientCompile externalDependency.restliClient
}

In /api, Pegasus Data Schemas (.pdsc's) should be added under /src/main/pegasus. E.g. /src/main/pegasus/com/linkedin/restli/example/Hello.pdsc.

The pegasus plugin will detect the presence of .pdsc files and use the dataTemplateGenerator to generate java bindings for the .pdsc's. In this example, a Hello.java class would be generated.

The dataTemplateCompile adds pegasus schemas that Hello.pdsc depends on, in this case, Hello.pdsc depends only on the core data libraries of pegasus, but projects containing other .pdsc's could be depended on.

Pegasus will detect when a project contains interface definitions (called idl and locaed in .restspec.json files) in it's /src/mainGeneratedRest/idl directory (usually copied in from an idl extraction task from the server, see below) and will generate java bindings. For example, HelloBuilder.java is generated from the idl of the hello resource (/src/main/idl/com/linkedin/restli/example/impl/Hello.restspec.json) and it written to the /src/mainGeneratedRest/java' directory of the /api project.

restTools is a required dependency for pegasus.

build.gradle for Server project

/server/build.gradle:

apply plugin: 'java'
apply plugin: 'pegasus'

ext.apiProject = project(':api')
pegasus.main.idlOptions.addIdlItem(['com.linkedin.restli.example.impl'])

dependencies {
  compile project(path: ':api', configuration: 'dataTemplate')
  compile "com.linkedin.pegasus:restli-server:<version>"
  restTools "com.linkedin.pegasus:restli-tools:<version>"
  // ...
}

In /server, pegasus "Resource" java classes should be defined and should be in the package(s) referred to by pegasus.main.idlOptions. E.g. /src/main/java/com/linkedin/restli/example/impl.HelloResource.java.

Pegasus will extract an interface definition (.restspec.json) from the resource class and write it to /src/mainGeneratedRest/idl directory.

Once the idl has been generated, it will be copied to the project identified by ext.apiProject. In this example, it will be copied to /api/src/main/idl. Before it is copied, api/src/main/idl is scanned for pre-existing idl. If any is found, it is compared with the replacement idl that will be copied in and a compatibility checker is run that will return errors if the replacement idl is not backward compatible with the existing idl. The compatibility checks can be disabled by setting (but be warned, compatibility errors mean that a server running the new interface definition is now incompatible with clients running older versions, and should not be pushed to production systems). If the compatibility checks pass, the idl is copied into the client directories. Once copied, new 'Client Bindings' may be generated for the client, see below.

The compile dependency on :api is required if the HelloResource.java depends on Hello.pdsc and it's generated binding Hello.java. Note that the dependency includes a 'configuration' identifying this as a 'dataTemplate' dependency.

Again, restTools is a required dependency for pegasus.

build.gradle for Example Java Client

/client/build.gradle:

apply plugin: 'java'

dependencies {
  compile project(path: ':api', configuration: 'restClient')
  compile "com.linkedin.pegasus:restli-client:<version>"
}

Once rest client bindings in the api project have been generated, it is trivial for a engineer to depend on the api project and use the generated client bindings to make calls to the new rest.li interface from any remote service.

One must add a compile dependency the 'api' project (or depend on it's published artifacts, more about this below) and be sure to set the dependency configuration to 'restClient'. Once this is done, it's easy to use the HelloBuilder class to construct a request.

Compatibility

To manage compatibility checking use the rest.model.compatibility flag.

For example, to run a build ignoring backward incompatable interface changes (WARNING: remember that backward incompatible changes could break your clients):

gradle build -Prest.model.compatibility=ignore

To acknowledge a backwards compatible interface change use:

gradle build -Prest.model.compatibility=backwards

Publishing artifacts

Often, the client bindings need to be accessible to developers outside the project workspace where the service is developed.

To publish rest client bindings to any maven repo first modify the api project's gradle to look like:

/api/build.gradle:

// ... /api/build.gradle code from above ...

artifacts {
  archives mainRestClientJar
  archives mainDataTemplateJar
} 

configure(install.repositories.mavenInstaller) {
  addFilter('rest-client') {artifact, file ->
    artifact.name == 'api-rest-client'
  }.artifactId = 'rest-client'
  
  addFilter('api-data-template') {artifact, file ->
    artifact.name == 'api-data-template'
  }.artifactId = 'data-template'
}

The 'artifacts' section tells gradle to build jar files for the rest client bindings and the data templates.

The configure part instructs gradle to publish both artifacts into maven. Setting names for each (by default gradle names the artifact publish to maven to 'api' and since there are two artifacts, they need to be given distinct names).

Next, update the root build.gradle file to include project information withing the subprojects section:

/build.gradle

// ... /build.gradle code from above ...

subprojects {
  // ...

  project.group = 'org.example'
  project.version = '0.1'
}

Once the api build.gradle is updated. One can publish the maven artifacts. To publish to the maven local repo, simply run:

gradle install

to publish to a remove maven repository follow the gradle documentation

Once published, other projects may import the client bindings by depending on the two maven artifacts, e.g.:

dependencies {
  compile "org.example:rest-client:0.1"
  compile "org.example:data-template:0.1"
}

Underlying gradle tasks

The gradle tasks for pegasus are provided by PegasusGeneratorV2Plugin.groovy in the rest.li source code. This plugin defines custom of gradle SourceDirectorySets for the 'idl', 'pegasus' source types and tasks for the rest.li code generators. It also defines custom published artifact "configurations" and dependencies on between these custom published artifact "configurations".

Additional source file types:

src/*/idl - Contains published idl (.restspec.json) files. These files represent the definition of the rest.li resources provided by some service. Unless the source set is a "generated*" directory, these files should be checked in to source control. These files are generated by the generate{sourceSet}restModel tasks (e.g. @generateMainRestModel@), which run as part of the build task (the jar task depends on this task) on modules containing {*Resource.java} files (which must be in a package referenced by pegasus.{sourceSet}.idlOptions.addIdlItem(namespaces)). One important aspect of idl is that by convention they are generated by a 'server' module (and written to the src/{sourceSet}generatedRest/idl) and then are copied to the /src/{sourceSet}/idl directory of an api module (via the ext.apiProject property, see above documentation for /server/build.gradle). The idl is published as a {project-name}-{module-name}-rest-model-{version}.jar from projects containing idl in the src/main/idl directory, which by convention should be "api" modules.

@pegasus/@ - Contains data schemas (.pdsc) files.

@avro/@

@generated*/@ - These files should not be checked in to source control.

Underlying Java Classes for Build Integration

This is provided for reference only. A understanding of these classes is not required to use pegasus. These classes would be useful primarily if one were deeply integrating pegasus with a build system not already supported by pegasus.

Avro Schema Generator

Generate Avro avsc files from Pegasus Data Model schemas (.pdsc files).

java [-Dgenerator.resolver.path=<dataSchemaRelativePath>] \
  -cp <CLASSPATH> com.linkedin.data.avro.generator.AvroSchemaGenerator \
  <outputDir> [<inputFileOrDir> ...]

• dataSchemaRelativePath - Path to .pdsc files. (e.g. /src/main/pegasus).
• CLASSPATH - 'com.linkedin.pegasus:data:[CURRENT_VERSION]' AND 'com.linkedin.pegasus:data-avro:[CURRENT_VERSION]' artifacts and all their dependencies.
• outputDir - output directory for generated avsc files
• inputFileOrDir - file name of a Pegasus data schema file, a directory containing Pegasus data schema files, or a fully qualified schema name

Build integration: for builds requiring avro schemas, assembly (creation of jar) should depend on this task

Pegasus Data Template Generator

Generates Java data templates (.java files) from Pegasus Data Model schemas (.pdsc files).

java [-Dgenerator.resolver.path=<dataSchemaRelativePath>] -cp <CLASSPATH> \
  com.linkedin.pegasus.generator.PegasusDataTemplateGenerator \
  <outputDir> [<inputFileOrDir> ...]

• dataSchemaRelativePath - Path to .pdsc files. (e.g. /src/main/pegasus).
• CLASSPATH - 'com.linkedin.pegasus:generator:[CURRENT_VERSION]' artifact and all it's dependencies.
• outputDir - output directory for generated java source files
• inputFileOrDir - file name of a Pegasus data schema file, a directory containing Pegasus data schema files, or a fully qualified schema name

Generate Rest Model IDL

Serializes a set of resource models to a RESTspec IDL file.

java -cp <CLASSPATH> com.linkedin.restli.tools.idlgen.RestLiResourceModelExporterCmdLineApp \
  -outdir <outputDirPath> -sourcepath <sourcePath> -resourcepackages <resourcePackages>

• CLASSPATH - 'com.linkedin.pegasus:restli-tools:[CURRENT_VERSION]' artifact and all it's dependencies. Compiled classes within the java packages referred to by 'resourcePackages'
• outputDirPath - Directory in which to output the generated IDL files (default=current working dir)
• sourcePath - Space-delimited list of directories in which to find resource Java source files
• resourcePackages - Space-delimited list of packages to scan for resource classes

Build integration: assembly (creation of jar) should depend on this task. This task depends on compilation of classes within the java packages referred to by 'resourcePackages'.

Validate and Publish IDL

Copies IDL (.restspec.json) files to client module and check backwards compatibility between pairs of idl (.restspec.json) files. The check result messages are categorized.

java [-Dgenerator.resolver.path=<dataSchemaRelativePath>] -cp CLASSPATH \
  com.linkedin.restli.tools.idlcheck.RestLiResourceModelCompatibilityChecker \
  [--compat OFF|IGNORE|BACKWARDS|EQUIVALENT] [pairs of <prevRestspecPath currRestspecPath>]

• dataSchemaRelativePath - Path to .pdsc files required by the interface definition (e.g. /src/main/pegasus).
• CLASSPATH - 'com.linkedin.pegasus:restli-tools:[CURRENT_VERSION]' artifact and all it's dependencies.
• prevRestspecPath -
• currRestspecPath -

Build integration: assembly (creation of jar) should depend on this task. If compatibility checker passes, all .restspec.json files should be copied from the server module to the module where client bindings are generated. This task depends on the 'Generate Rest Model IDL' task. A property named 'rest.model.compatibility' should be overridable by the developer (allowing them to set it to 'ignore' or 'backwards') and should default to 'equivalent' if they do not provide it.

Rest Client Generation

Generates Java request builders from Rest.li idl.

java [-Dgenerator.resolver.path=<dataSchemaRelativePath>] \
     [-Dgenerator.rest.generate.datatemplates=<true|false>] \
  -cp <CLASSPATH> com.linkedin.restli.tools.clientgen.RestRequestBuilderGenerator \
  <targetDirectoryPath> [<sourceFileOrDir> ...]

• dataSchemaRelativePath - Path to .pdsc files required by the interface definition.
• generator.rest.generate.datatemplates - false unless task should also generate java data template bindings
• CLASSPATH - 'com.linkedin.pegasus:restli-tools:[CURRENT_VERSION]' artifact and all it's dependencies.
• targetDirectoryPath - path to target root java source directory
• sourceFileOrDir - paths to IDL files or directories

Build integration: Compilation of java source should depend on this task.

Config Build Script

To construct these build tasks, it can help to add a utility task that constructs a list of all the source paths used for 'data template generation', 'avro schema generation', 'rest model generation' and 'rest client generation'.

Clean Generated

No java class for this. All directories written to by 'data template generation', 'avro schema generation', 'rest model generation' and 'rest client generation' should be deleted.

Build integration: clean task should depend on this