Continuous Integration (CI)

CentOS CI

We are using CentOS CI for all our projects. Documentation can be found at 📜 Wiki.

How to setup CI

This section provides simple tutorial how to setup CI on a project using Jenkins pipelines.

Pull requests

1. Log into https://ci.centos.org
2. New item -> Pipeline -> OK
3. General tab:
   • Fill in the description (optional)
   • Check ‘GitHub Project’ and fill in your project url, e.g https://github.com/user-cont/conu/
4. ZMQ Event Publisher tab:
   • Don't need to change anything
5. Build Triggers tab:
   • Check ‘GitHub Pull Request Builder’
   • Add github usernames of people in our team into ‘Admin list’. An admin can (re-)trigger tests and when (s)he creates a PR the tests are triggered automatically, while other people’s PR need to be verified & triggered by an admin in order to not run any hostile code on a node.
   • Check ‘Use github hooks for build triggering’
   • Click ‘Advanced’ and put .*\[test\].* into ‘Trigger phrase’. This allows you to (re-)trigger tests in a PR by putting “[test]” (brackets included) into a comment in the PR.
   • Add user-cont or packit-service into List of organizations. Their members will be whitelisted. and check Allow members of whitelisted organizations as admins
6. Pipeline tab:
   • Definition: Pipeline script from SCM
   • SCM: Git
     • Repositories: upstream repo (e.g. https://github.com/user-cont/conu/)
     • Credentials: CentOS CI Github Account
     • Select Advanced
       • Name: origin
       • Refspec: +refs/pull/*:refs/remotes/origin/pr/*
     • Branches to build
       • Branch Specifier (blank for 'any'): ${ghprbActualCommit}
     • Git executable: Default
     • Repository browser: (Auto)
   • Script Path: Jenkinsfile
   • And uncheck lightweight checkout.
7. Save
8. Follow instruction in https://wiki.centos.org/QaWiki/CI/GithubIntegration section Mode 2: Trigger a build for each Pull Request to add webhooks to you GitHub project.

Trigger for each commit to master

1. Log into https://ci.centos.org
2. New item -> Pipeline -> OK
3. General tab:
   • Fill in the description (optional)
   • Check ‘GitHub Project’ and fill in your project url, e.g https://github.com/user-cont/conu/
4. ZMQ Event Publisher tab:
   • Don't need to change anything
5. Build Triggers tab:
   • Check ‘GitHub hook trigger for GITScm polling’
6. Pipeline tab:
   • Definition: Pipeline script from SCM
   • SCM: Git
     • Repositories: upstream repo (e.g. https://github.com/user-cont/conu/)
     • Credentials: -none-
     • Branches to build
       • Branch Specifier (blank for 'any'): */master
     • Git executable: Default
     • Repository browser: (Auto)
   • Script Path: Jenkinsfile
   • And leave lightweight checkout checked.
7. Save
8. Follow instruction in https://wiki.centos.org/QaWiki/CI/GithubIntegration section Mode 1: Trigger on Every Commit to add webhooks to your GitHub project.

Add status badge into GitHub README.md

1. Log into https://ci.centos.org
2. Open your job (e.g https://ci.centos.org/job/release-bot-push/)
3. In left menu select Embeddable Build Status
4. Copy the snippet from Markdown (with view) section and paste it to your README.md.

How to write a Jenkinsfile

📜 Documentation

1. You can copy these helper functions at the beginning of your Jenkinsfile Function for executing commands on a new duffy node.

def onmyduffynode(script){
    ansiColor('xterm'){
        timestamps{
            sh 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -l root ${DUFFY_NODE}.ci.centos.org -t \"export REPO=${REPO}; export BRANCH=${BRANCH};\" "' + script + '"'
        }
    }
}

Function for copying files to new duffy node

def synctoduffynode(source)
{
    sh 'scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -r ' + source + " " + "root@" + "${DUFFY_NODE}.ci.centos.org:~/"
}

2. Basic structure of Jenkinsfile

node('userspace-containerization'){

    stage('Checkout'){
        checkout scm
    }

    stage('Build'){
        try{
            stage ("Allocate node"){
                env.CICO_API_KEY = readFile("${env.HOME}/duffy.key").trim()
                duffy_rtn=sh(
                            script: "cico --debug node get --arch x86_64 -f value -c hostname -c comment",
                            returnStdout: true
                            ).trim().tokenize(' ')
                env.DUFFY_NODE=duffy_rtn[0]
                env.DUFFY_SSID=duffy_rtn[1]
            }

            stage ("setup"){
                onmyduffynode "yum -y install docker make"
                synctoduffynode "*" // copy all source files
                onmyduffynode "systemctl start docker"
            }

            stage("build test image"){
                onmyduffynode "make image-test"
            }

            // add test stages here

        } catch (e) {
            currentBuild.result = "FAILURE"
            throw e
        } finally {
            stage("Cleanup"){
                sh 'cico node done ${DUFFY_SSID}'
            }
        }
    }
}

3. Run tests in parallel. Jenkins pipeline allows you to run your test suite in parallel.

You can define all your stages

stage('Run Tests') {
    parallel {
        stage('Unit tests') {
            onmyduffynode "make tests-unit"
        }
        stage('Integration tests') {
             onmyduffynode "make tests-integration"       
        }
    }
}

or you can take advantage of loops. See release-bot example for more information.

Debugging

Before you add Jenkinsfile to upstream/master branch you can try it. Open your job (e.g https://ci.centos.org/job/release-bot-pr/)

If you don’t have any jobs yet, click Build now in the left menu. Then open the job and in the left menu select Replay. Then you can edit your Jenkins file and run the job again and again.