Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Runner docker volumes #861

Merged
merged 24 commits into from
Jan 21, 2022
Merged

Runner docker volumes #861

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
4431384
GL dind and volumes
DavidGOrtega Jan 6, 2022
2263de9
support multiple volumes
DavidGOrtega Jan 6, 2022
3f5c872
no docker user
DavidGOrtega Jan 6, 2022
bbfea5d
no sudo
DavidGOrtega Jan 6, 2022
806bb87
remove extra amps
DavidGOrtega Jan 6, 2022
711bc72
del get-docker.sh
DavidGOrtega Jan 6, 2022
2aba1f9
docker-volumes param
DavidGOrtega Jan 11, 2022
6519de0
test tpl
DavidGOrtega Jan 11, 2022
389b199
tf tests
DavidGOrtega Jan 11, 2022
ceade0b
Merge branch 'master' of https://github.com/iterative/cml into gl-doc…
DavidGOrtega Jan 16, 2022
acd5ae6
update first
DavidGOrtega Jan 16, 2022
7b99770
unused
DavidGOrtega Jan 16, 2022
df1ba1f
platform
DavidGOrtega Jan 16, 2022
a79926b
no-docker
DavidGOrtega Jan 17, 2022
17afa5e
GitLab vs Gitlab
DavidGOrtega Jan 17, 2022
8e34536
tests
DavidGOrtega Jan 17, 2022
443166c
Merge branch 'gl-docker-volumes-dind' of https://github.com/iterative…
DavidGOrtega Jan 17, 2022
7d41386
Merge branch 'master' into gl-docker-volumes-dind
0x2b3bfa0 Jan 17, 2022
130ff1c
remove docker file
DavidGOrtega Jan 17, 2022
6c1299d
Merge branch 'gl-docker-volumes-dind' of https://github.com/iterative…
DavidGOrtega Jan 17, 2022
ac9ff91
warn for not gl users
DavidGOrtega Jan 19, 2022
3eaa737
merge master
DavidGOrtega Jan 19, 2022
4def4c9
remove docker
DavidGOrtega Jan 21, 2022
97c1328
Merge branch 'master' into gl-docker-volumes-dind
0x2b3bfa0 Jan 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,11 @@ RUN curl --location https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
&& apt-get clean \
&& rm --recursive --force /var/lib/apt/lists/*

# INSTALL DOCKER
RUN curl -fsSL https://get.docker.com -o get-docker.sh && sudo sh get-docker.sh && \
&& apt-get clean \
&& rm --recursive --force /var/lib/apt/lists/*

0x2b3bfa0 marked this conversation as resolved.
Show resolved Hide resolved
# INSTALL PYTHON
ARG PYTHON_VERSION=3
RUN add-apt-repository universe --yes \
Expand Down
10 changes: 8 additions & 2 deletions src/drivers/gitlab.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const ProxyAgent = require('proxy-agent');

const { fetchUploadData, download, exec } = require('../utils');

const { IN_DOCKER, CI_PIPELINE_ID } = process.env;
const { IN_DOCKER, CI_PIPELINE_ID, CML_GL_DOCKER_VOLUMES } = process.env;
DavidGOrtega marked this conversation as resolved.
Show resolved Hide resolved
const API_VER = 'v4';
class Gitlab {
constructor(opts = {}) {
Expand Down Expand Up @@ -186,6 +186,11 @@ class Gitlab {

const { protocol, host } = new URL(this.repo);
const { token } = await this.registerRunner({ tags: labels, name });

let dockerVolumesTpl = '';
CML_GL_DOCKER_VOLUMES.split(',').forEach((vol) => {
dockerVolumesTpl += `--docker-volumes ${vol} `;
});
const command = `${bin} --log-format="json" run-single \
--builds-dir "${workdir}" \
--cache-dir "${workdir}" \
Expand All @@ -195,7 +200,8 @@ class Gitlab {
--wait-timeout ${idleTimeout} \
--executor "${IN_DOCKER ? 'shell' : 'docker'}" \
--docker-image "iterativeai/cml:${gpu ? 'latest-gpu' : 'latest'}" \
--docker-runtime "${gpu ? 'nvidia' : ''}" \
${gpu ? '--docker-runtime nvidia' : ''} \
casperdcl marked this conversation as resolved.
Show resolved Hide resolved
${dockerVolumesTpl} \
${single ? '--max-builds 1' : ''}`;

return spawn(command, { shell: true });
Expand Down