Skip to content

Latest commit

 

History

History
373 lines (310 loc) · 16.2 KB

README.adoc

File metadata and controls

373 lines (310 loc) · 16.2 KB
Raw

Dynamatrix

General overview

A Jenkins Pipeline Shared Library allows to re-use code blocks in different pipelines, to use Java class approach to data and logic implementation, as well as to store global static state data for the duration of a pipeline script execution.

Its source repo has a predefined structure, and has to be registered on the Jenkins controller to be trusted, and the pipeline scripts can then include it like this:

@Library ('org.nut.dynamatrix') _

With jenkinsci/pipeline-groovy-lib-plugin#19 feature used in the build of the @Library implementation running on the Jenkins server, it is also possible to request "same branch of the library as the branch of pipeline script" with fallback to configured default branch, to avoid superficial differences between pipeline code branches, like this:

@Library ('org.nut.dynamatrix@${BRANCH_NAME}') _

Keep in mind that this is not a Groovy variable, just intentionally looks like one for simpler code maintenance. The @Library annotation is processed before compilation of pipeline script to Groovy and Java.

Note
Obsoleted Note (valid for unmodified Jenkins servers):

The @Library method only allows non-dynamic references to the library (a different approach is possible for run-time resolution of branches with library step whose limitations make it not usable for us).

For this reason, dev/test version of the code of the library from dev/test versions of its consumers had to be named explicitly, like this:

@Library ('org.nut.dynamatrix@fightwarn') _

What and why

This project is intended as a relatively generic solution for Jenkins pipelines to build and test the codebase in a variety of environments (such as operating systems, compiler brands/versions or programming language revisions) and setups (such as the sets of warnings marked to be fatal vs. ones that are still tolerated) based dynamically on what capabilities are supported by the workers of the build farm.

These workers may be not necessarily managed or provisioned by the same administrator or entity as manages the build farm, but can be e.g. a Swarm of specially labeled agents provided by contributors of a FOSS project — which is especially of interest to such users in order to have their less common system or setup supported by the FOSS project they like and use, or to contribute access to architectures that the project does not own.

For projects that have a minimal baseline of scenarios that must pass, this information can be conveyed by the configuration for the pipeline, so that dynamic workers matching those requirements can be spun up and later deactivated or destroyed to conserve run-time resources of the centralized build farm. This allows to sequentialize builds of varied operating environments hosted on systems with limited memory.

This library, or generally persistent CI farms, can also take advantage of https://github.com/jimklimov/git-refrepo-scripts and a (currently custom) build of Git Client Plugin with the changes proposed in PR jenkinsci/git-client-plugin#644 to maintain reference repositories on the CI server, and so reduce the traffic and time needed for future Git check-outs.

Main idea

Dynamatrix provides a Shared Pipeline library, so that the pipeline definitions in actual source code repositories can be kept compact and not need to update their Jenkinsfiles whenever this idea evolves.

Build agent definitions, whether static (e.g. SSH Agents) or dynamic (e.g. Swarm Client agents) can declare labels which can be matched in the pipeline stages. The Dynamatrix is layered on top of structurally named labels to generate a list of possible build combinations as a boolean label expression, and so dynamically generate the payload structure of each build.

A sibling jenkins-swarm-nutci project is used on build agents which can dial in to the NUT CI farm and provide their services and capabilities (conveyed by the labels which are part of their configuration), and includes some scripting for common configuration approach as well as for automated start-up on numerous platforms. It might be usable "as is" or easily adapted by other projects.

Since the same CI farm and Swarm agent workers can be used for the benefit of several FOSS projects, one of the label patterns allows to declare that this worker is suitable for (and contributes to) builds of a particular project.

This library is designed to provide a number of classes for larger pieces of code as separate compiliation units (my first stint with standard pipeline matrix{} failed due to resulting method binary exceeding 64K) and as separate callable routines. While it may be helpful to directly use prepareDynamatrix() in some cases, many practical jobs would define a dynacfgBase with common dynamatrix properties for a suitable node and a dynacfgPipeline with certain fields such as the closure(s) to run dozens of times, and pass that to a dynamatrixPipeline(dynacfgBase, dynacfgPipeline) call that implements the pipeline in a reusable fashion.

Example configuration

In the illustrative example below, two workers are provisioned and based on their installed tools, they declare the following agent labels relevant to the dynamatrix:

  • "testoi":

COMPILER=CLANG COMPILER=GCC
CLANGVER=8 CLANGVER=9
GCCVER=10 GCCVER=4.4.4 GCCVER=4.9 GCCVER=6 GCCVER=7
OS=openindiana
ARCH=x86_64 ARCH=x86
nut-builder zmq-builder

  • "testdeb":

COMPILER=GCC
GCCVER=4.8 GCCVER=4.9 GCCVER=5 GCCVER=7
OS=linux
ARCH=x86_64 ARCH=x86 ARCH=armv7l
nut-builder zmq-builder linux-kernel-builder

A Jenkinsfile in a project using this library would look like:

@Library ('dynamatrix') _

// Prepare parallel stages for the dynamatrix:
def parallelStages = prepareDynamatrix(
    commonLabelExpr: 'nut-builder',
    compilerType: 'C',
    compilerLabel: 'COMPILER',
    compilerTools: ['CC', 'CXX', 'CPP'],
    dynamatrixAxesLabels: ['OS', '${COMPILER}VER', 'ARCH'],
    dynamatrixAxesCommonEnv: [['LANG=C', 'TZ=UTC'], ['LANG=ru_RU']],
    dynamatrixAxesCommonOpts: [
        ['"CFLAGS=-stdc=gnu99" CXXFLAGS="-stdcxx=g++99"', '"CFLAGS=-stdc=c89" CXXFLAGS="-stdcxx=c++89"'],
        ['-m32', '-m64'] ],
    dynamatrixRequiredLabelCombos: [[~/OS=bsd/, ~/CLANG=12/]],
    allowedFailure: [[~/OS=.*/, ~/GCCVER=4\..*/], ['ARCH=armv7l'], [~/std.*=.*89/]],
    excludeCombos: [[~/OS=openindiana/, ~/CLANGVER=9/, ~/ARCH=x86/]]
) {
    unstash 'preparedSource'
    sh """ ./autogen.sh """
    sh """ ${dynamatrix.commonEnv} CC=${dynamatrix.COMPILER.CC.VER} CXX=${dynamatrix.COMPILER.CXX.VER} ./configure ${dynamatrix.commonOpts} """
    sh """ make -j 4 """
    sh """ make check """
}

pipeline {
    // List some unique options, build parameters, and build scenarios
    // like spell checking or docs generation that you do not need to
    // run dozens of times using every agent.
    agent { label 'persistent-worker' }
    stages {
        stage('Prepare source') {
            steps {
                // The persistent agent may be in better position to
                // e.g. use a Git reference repository for faster
                // checkouts, or just to have the internet access
                // which CI farm workers may lack.
                checkout scm
                stash 'preparedSource'
            }
        }
        stage('Spell Check') {
            steps {
                sh """ aspell ... """
            }
        }
        stage('Make docs') {
            agent { label 'docs-builder' }
            steps {
                unstash 'preparedSource'
                sh """ make pdf """
            }
        }
    }
}

parallel parallelStages

With this configuration, the Dynamatrix should detect the running agents and know their capabilities, so it is in position to prepare a series of builds covering every available OS and compiler version and CPU architecture.

It can optionally be filtered through constraints, such as that we do not even want to try building a combination described by (matching) the skip option, that we require to run some combination(s) even if an agent for that is not currently running so labels are not detected (things can hang in queue waiting for a worker, or can cause spinning up a build agent if it is configured but dormant), or that some certain build setups may fail (e.g. we wonder how they fare, but they are not a required baseline and so not blockers for a merge) so their results would not be impacting the overall job verdict.

For certain compiler toolkits (e.g. C family) it would provide an automatic preparation of variables for several same-versioned tools (e.g. C and C++ compilers).

Declaring additional configuration of build agents

Depending on their implementation and connectivity, build agents may have different preferences, and this library allows to tune them with their node labels.

One such area is delivery of tested codebase to the agents: default approach which is to stash on master, and unstash on agents, should be reliable (should reach agents that can not use the SCM platform directly, and should ensure all agents test the same revision even if it disappears from the SCM platform — by e.g. force-push to a PR), but at a cost of repetitive traffic and I/O to unstash same code time and again (often on same machine) during a matrix build.

Setting DYNAMATRIX_UNSTASH_PREFERENCE to scm-ws, scm or unstash in the individual agent labels allows that system to start with either an SCM checkout augmented by a Git reference repository (persistent) in the workspace and maintained during each run (this currently requires a custom build of Jenkins Git Client plugin including the feature from jenkinsci/git-client-plugin#644 unless/until it gets properly merged); or using a plain SCM checkout; or unstashing. These methods fall-back from one to next in the order listed above.

The DynamatrixStash methods dealing with code checkout, stashing and unstashing, allow a concept of stashName used to identify archives as well as to track metadata for that codebase (so the same pipeline can mix several repositories). Preferences for each repository can be tailored, using e.g. scm:githubProject and unstash:privateRepo label values to use different delivery methods for the two stashNames.

To prevent several parallel jobs and build scenarios from corrupting the reference repository maintained in the workspace, maintenance of this location is protected by Lockable Resources plugin. Since agents running with independent storage should not wait on each other, this lock can be tuned by setting DYNAMATRIX_REFREPO_WORKSPACE_LOCKNAME label; note that agents that do indeed use same storage (shared over NFS, or using containers with same homedirs from their host) should set identical values in their common lock name.

Directory naming

This is a Jenkins Shared Library. As such, it has some required file system structure including:

  • vars/ — the "groovy variables" which are, at least in this context, sources for single-use class instances and their methods that can be called from each other or from the pipeline which uses the lib, and as far as the pipeline is concerned, call() methods in these groovy files are custom "steps" (named same as the file);

  • src/ — formal classes including ones that can be static, such as to store some persistent configuration for the run.

There are further standard structure points that we do not currently use, such as location and naming of documentation to accompany the declared steps so this can be displayed by Jenkins UI, and location for resources such as shell scripts and arbitrary data used by JSL.

Approach recommended in some of the articles linked below is that the logic is mostly (ideally all) in src/ classes, while the vars/ steps only wrap calls to that.

Practice, especially during early development iterations, may be mixed.

Inspirational reading

Good explanatory articles with varied detail; many other texts seem to tell the same things differently while reasonably assuming a non-beginner level from the reader. Some of those below may be a bit too long and chewing the basics delicately — but sometimes that is really a good thing:

Standard reading library:

Unit-tests of the JSL library

This library is tested with the help of "mkobit" plugins referenced above. Tests are located in the tests/ directory and implement example pipelines which are executed in either a temporary Jenkins environment provided by a JenkinsRule environment (from jenkins-test-harness project) or by a more dedicated server behind a RealJenkinsRule. They can be executed by an IDE (e.g. press F9 in IntelliJ IDEA) or by ./gradlew integrationTest.