Document Github Actions and Gitlab CI with private repositories

[mathias-luedtke]

• SSH access?
• Token replacement?
• Secrets?
• Credential helpers?

[destogl]

For our private GitLab instance, we are using token placement with sed command.

The before script looks like this:

before_script:
  - apk add --update bash coreutils tar
  - git clone --quiet --depth 1 https://github.com/ros-industrial/industrial_ci .industrial_ci -b master
  - sed -i 's/https\:\/\/gitlab\.ipr\.kit\.edu/https\:\/\/gitlab-ci-token\:'${CI_JOB_TOKEN}'\@gitlab\.ipr\.kit\.edu/g' ${PROJECT_ROSINSTALL_FILENAME}
  - sed -i 's/https\:\/\/gitlab\.ipr\.kit\.edu/https\:\/\/gitlab-ci-token\:'${CI_JOB_TOKEN}'\@gitlab\.ipr\.kit\.edu/g' ${UPSTREAM_WORKSPACE}
  - sed -i 's/https\:\/\/gitlab\.ipr\.kit\.edu/https\:\/\/gitlab-ci-token\:'${CI_JOB_TOKEN}'\@gitlab\.ipr\.kit\.edu/g' ${TARGET_WORKSPACE}

Our URL is here hardcoded, it should be added as an environment variable to make this more flexible.

[mathias-luedtke]

https://docs.github.com/en/free-pro-team@latest/actions/reference/authentication-in-a-workflow

[mathias-luedtke]

https://coolaj86.com/articles/vanilla-devops-git-credentials-cheatsheet/

[mathias-luedtke]

@floweisshardt: It turned out that GITHUB_TOKEN only has access to the current repository, but I managed to make it work with a PAT:

      - uses: 'ros-industrial/industrial_ci@master'
        env:
          ROS_DISTRO: melodic
          TARGET_WORKSPACE: test.repos
          ADDITIONAL_DEBS: git-core
          AFTER_INIT: git config --global url."https://token:${{secrets.GITHUBPAT}}@github.com/".insteadOf "https://github.com/"

GITHUBPAT is a secret that contains the token. This should better belong to a service user with restricted access to a limited set of repositories.

@destogl:

This should work for you (instead of the 3 sed lines):

variables:
   ADDITIONAL_DEBS: git-core
   AFTER_INIT: git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.ipr.kit.edu/".insteadOf "https://gitlab.ipr.kit.edu/"

Please double check that the token is not exposed anywhere.

[destogl]

@ipa-mdl this looks great! Thanks, I will try to optimize our CI configurations.

[floweisshardt]

@ipa-mdl 👍 thanks a lot. That works for our use case.

[mathias-luedtke]

The only downside of this approach is that it might not work with run_ci.

[floweisshardt]

I still have some troubles to get this fully integrated. Seems as this works fine with self-hosted runners on my pc but as soon as I use github hosted runners the jobs fail during setup_upstream_workspace with

  At least one hostname (github.com) is unknown, switching to a single worker to allow interactively answering the ssh question to confirm the fingerprint
  The authenticity of host 'github.com (140.82.114.3)' can't be established.
  RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.

I assume it's waiting for some user input

Are you sure you want to continue connecting (yes/no)? yes

[floweisshardt]

adding

BEFORE_SETUP_UPSTREAM_WORKSPACE: mkdir -p ~/.ssh && touch ~/.ssh/known_hosts && ssh-keyscan github.com >> ~/.ssh/known_hosts

solves this

[mathias-luedtke]

@floweisshardt: the PAT trick does not use ssh, so you would need to replace the git@ paths

ssh-keyscan github.com is not secure, normally you should not run this automatically.
You can even add this line to your workflow yml.

If your build is just stuck because the known host, then there should be a way to automate it.

[mathias-luedtke]

One important note: Apparently, data access to a self-hosted runner needs to be paid for (free limit depends on subscription plan), if it gets authenticated with a PAT.

[eliabruni]

I still have some troubles to get this fully integrated. Seems as this works fine with self-hosted runners on my pc but as soon as I use github hosted runners the jobs fail during setup_upstream_workspace with

  At least one hostname (github.com) is unknown, switching to a single worker to allow interactively answering the ssh question to confirm the fingerprint
  The authenticity of host 'github.com (140.82.114.3)' can't be established.
  RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.

I assume it's waiting for some user input

Are you sure you want to continue connecting (yes/no)? yes

Hey @floweisshardt , I am having troubles make it running on a self-hosted runner. Could you perhaps post your workflow? Thanks!

[floweisshardt]

@eliabruni here's my workflow config:

name: CI

on:
  push:
  pull_request:
    types: [opened, edited, synchronize, reopened]
  schedule:
    - cron: "0 23 * * *" # everyday at 11pm
  

jobs:
  industrial_ci:
    timeout-minutes: 30
    env:
      ROS_REPO: ros
      VERBOSE_OUTPUT: true
      VERBOSE_TESTS: true
      ADDITIONAL_DEBS: git-core openssh-client openssh-server
      AFTER_INIT: mkdir -p ~/.ssh && touch ~/.ssh/known_hosts && ssh-keyscan github.com >> ~/.ssh/known_hosts && git config --global url."https://token:${{secrets.GITHUBPAT}}@github.com/".insteadOf "https://github.com/"
    strategy:
      matrix:
        env:
          - {ROS_DISTRO: kinetic, UPSTREAM_WORKSPACE: .rosinstall}
          - {ROS_DISTRO: melodic, UPSTREAM_WORKSPACE: .rosinstall}
#    runs-on: ubuntu-latest
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v2
      - uses: 'ros-industrial/industrial_ci@master'
        env:
          ${{matrix.env}}

in the .rostinstall files I use the following syntax:

- git:
    local-name: private_repo
    uri: https://github.com/ORG/PRIVATE_REPO.git
    version: kinetic-devel
- git:
    local-name: public_repo
    uri: https://github.com/ORG/PUBLIC_REPO.git
    version: kinetic-devel

[eliabruni]

Many thanks @floweisshardt , that works for me too! :))

[floweisshardt]

the script run_github_actions from #582 might be helpfull to validate the CI configuration locally.

[stertingen]

Are there plans on configuring Git to use the CI token automatically for the GitLab CI? (By providing a credential helper echoing the CI_JOB_TOKEN environment variable)

[mathias-luedtke]

Are there plans on configuring Git to use the CI token automatically for the GitLab CI?

You mean this?

   AFTER_INIT: git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/".insteadOf "${CI_SERVER_URL}/"

Should be possible.
I just don't know how to inject this setting properly in all cases.

[stertingen]

Are there plans on configuring Git to use the CI token automatically for the GitLab CI?

You mean this?

   AFTER_INIT: git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@${CI_SERVER_HOST}/".insteadOf "${CI_SERVER_URL}/"

Should be possible.
I just don't know how to inject this setting properly in all cases.

I rather thought of something where the credentials are not stored on disk.

#!/bin/bash
if [ "$1" = "get" ]; then
    echo "username=gitlab-ci-token"
    echo "password=${CI_JOB_TOKEN}"
fi

Within the setup scripts of the CI environment:

git config --global credential.${CI_SERVER_URL}.helper /path/to/gitlab-credential-helper.sh

[mathias-luedtke]

I rather thought of something where the credentials are not stored on disk.

CI_JOB_TOKEN is only valid during the build..

Within the setup scripts of the CI environment:

That's the bigger issue. Right now there is no defined time for executing the config line.

[stertingen]

I rather thought of something where the credentials are not stored on disk.

CI_JOB_TOKEN is only valid during the build..

Is that so? the token is valid during the execution of the whole pipeline, but it might have to be passed into the nested docker container on build and execution.

EDIT: Proof of concept:

variables:
  ADDITIONAL_DEBS: "git"
  DOCKER_RUN_OPTS: "--mount type=bind,src=${CI_PROJECT_DIR}/.gitconfig,dst=/root/.gitconfig,readonly --env CI_JOB_TOKEN"

before_script:
  - git config --file ${CI_PROJECT_DIR}/.gitconfig credential."${CI_SERVER_URL}".helper '!f() { test "$1" = get && echo "username=gitlab-ci-token" && echo "password=${CI_JOB_TOKEN}"; }; f';

For some reasons, mounting the git config file from /root/.gitconfig does not work, so the file is put at ${CI_PROJECT_DIR}.

EDIT 2: Regarding this:

Within the setup scripts of the CI environment:

That's the bigger issue. Right now there is no defined time for executing the config line.

I would propose configuring git in ici_setup_git_client: https://github.com/ros-industrial/industrial_ci/blob/master/industrial_ci/src/workspace.sh#L114

[mathias-luedtke]

CI_JOB_TOKEN is only valid during the build..

Is that so? the token is valid during the execution of the whole pipeline, but it might have to be passed into the nested docker container on build and execution.

I think I meant "job" and not "build".

From the documentation:
"When a pipeline job is about to run, GitLab generates a unique token and injects it as the CI_JOB_TOKEN predefined variable. [...]
3. Granting permissions to the job token only when the job is running."

I would propose configuring git in ici_setup_git_client

Yes, this would make sense.
However, I do not want to add Gitlab-specific code there.
It was hard enough to abstract from Travis CI in the first place ;)

[mathias-luedtke]

For some reasons, mounting the git config file from /root/.gitconfig does not work

It depends on the Docker runner. Normally /root is not shared between the different services.
That's why we have to rewite the temp folder as well etc.

[stertingen]

Yes, this would make sense.
However, I do not want to add Gitlab-specific code there.
It was hard enough to abstract from Travis CI in the first place ;)

You may take look at #594. However, it is currently GitLab-specific, especially regarding the CI_ variables. However, I introduced the ici_setup_git_credential_helper function which might be used to add helpers for other CI platforms.

EDIT: Now I see your point regarding the validity of the CI job token. That's true, storing the job token somewhere is not fatal since it will be invalid after a few minutes. That makes the AFTER_INIT_EMBED solution kind of the better solution since it's more simple and more generic than a separate credential helper without major security drawbacks.