Adding GPU testing to Github workflows

[moazreyad]

This issue is open to discuss different options for adding GPU build and test to Github workflows.

To enable this feature, SINGA must provide a real or virtual machine with GPU as host machine for running the workflow. Then use the self-hosted runner feature of Github Actions. See also this MLOps video tutorial.

The team need to take some decisions:

1. Which machine(s) should we use? (e.g. virtual machine(s) on AWS, dedicated server(s) at NUS, ..)
2. Which operating systems we test on? (Only Linux? or also Mac).
3. When we run the GPU build and test workflow? (with every pull request? once per day at night? once per week? only on master branch?, ...)
4. Should we keep the machines always on? or use them only when the scheduled test is running and shut down them when there is no workflow runs? Assuming we run the GPU build and test only at scheduled time.
5. Should we add workflows to run examples and test the Jupyter notebooks? note that some examples may take hours or days to complete the training. But automating the test of examples can be very useful to speed up the development.

What do you think?

[chrishkchris]

I believe that what you propose in this issue is very important for us.
We need to safeguard our codebase, because a wrong PR could destroy the whole system functionality.
Therefore I believe that GPU testing is necessary.

Here are my opinions on the questions:

1. prefer dedicated server at NUS (not prefer cloud service providers because I think they are too expensive)
2. Linux is enough (ubuntu)
3. For every PR we need to run the test workflow (just like the current test settings for CPU versions, but need to upgrade the test to GPUs)
4. Prefer to keep the test server on so when someone submits a PR it will be checked immediately.
5. Maybe consider in the future to run the examples. I think the priority is to enable GPU build and test first.

[chrishkchris]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

[chrishkchris]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

[NLGithubWP]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

[chrishkchris]

@naili-xing Naili, since you told us that you would like to contribute in SINGA, could you assist us to implement this (currently we have cpu test, but we also need GPU test)

Ok, i will look into it

FYI, the different between CPU version and GPU version in building SINGA is just the flag we used in cmake
GPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cuda10/Dockerfile#L104
cmake -DENABLE_TEST=ON -DUSE_CUDA=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

CPU version: https://github.com/apache/singa/blob/master/tool/docker/devel/ubuntu/cpu/Dockerfile#L67
cmake -DENABLE_TEST=ON -DUSE_PYTHON3=ON -DUSE_DNNL=ON ..

In GPU version, it include all the GPU test case when we run test/python/run.py
In CPU version, it automatically skiped all the GPU test case

[moazreyad]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

[chrishkchris]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:

First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)

To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.

Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

[NLGithubWP]

@moazreyad could you please advise us how to achieve the goal? (Adding GPU testing to Github workflows
#802), I will ask some RAs to help me for the implementation.

To achieve this goal, we will need these steps:
First, we need to get github tokens from by opening a ticket at Apache INFRA. Apache has some information about github self hosted runners here. They don't recommend using it for some security reasons, so we have to explain that GPU build and test is important for SINGA. We also need to make sure the server which will run the github workflows is secure and that the known vulnerabilities of self-hosted runners will not occur in the NUS server. (for example, someone may login to the NUS server and gain access to SINGA github repository admin tokens.)
To complete this step, we may check this ticket by Apache Arrow and open a similar one for SINGA.
Second, we create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

@moazreyad thanks a lot for the information, I will consider it. Also I am thinking about if GPU emulator is another option? https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

@naili-xing do you think the github actions can access our server to do gpu test, does it need to connect to nus VPN?

I think it needs, if accessing the server from outside of nus, it needs vpn, if access from inside, using nus network, i think there is no needs

[chrishkchris]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

[joddiy]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And since we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

[chrishkchris]

@joddiy will help us try if gpu emulator can run our test case first
https://developer.nvidia.com/nvemulate
https://stackoverflow.com/questions/40461823/is-it-possible-to-emulate-a-gpu-for-cuda-opencl-unit-testing-purposes

Hi, @chrishkchris , these two GPU emulators haven't been updated for more than 5 years. And science we have our own machine, the self-hosted GitHub Actions is a better solution.

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

@joddiy OK, then we go for the solution using our own machine gpu for github actions test first.

For our reference: communication between our server and github actions
https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github

[chrishkchris]

https://docs.github.com/en/free-pro-team@latest/actions/hosting-your-own-runners/about-self-hosted-runners#communication-between-self-hosted-runners-and-github
FYI
"The self-hosted runner uses a HTTPS long poll that opens a connection to GitHub for 50 seconds"

[chrishkchris]

@moazreyad I have opened a ticket to ask the infrastructure and hopes they can give us a solution

https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21115?filter=allopenissues

[chrishkchris]

@moazreyad
They told us that we can use the API in the following link to add github secrets (to store token like username and password)
https://developer.github.com/v3/actions/secrets/

@joddiy
Could you help test the API to see if we can store some random token, and see if we are able to access what we have stored?
(e.g. you can use postman https://www.postman.com/ to test the restful API)

[chrishkchris]

@naili-xing

I think before implementation on the SINGA repo, we fork the repo to test the github actions with our own github account first.

Could you follow the following steps for the test

1. fork the singa repo to your github acoount
2. make the forked repo private (throughout the test to ensure the safety of our server), add me as a collaborator of the forked repo so I can also access it
3. implement the github actions code for gpu test:
   We create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner and its pre-reqs in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation for more details.

We evaluate the approach first before implementing it in apache/SINGA repo

[NLGithubWP]

ok Chris Yeung <[email protected]>于2020年11月19日 周四下午4:49写道：

…

@naili-xing <https://github.com/naili-xing> I think before implementation on the SINGA repo, we fork the repo to test the github actions with our own github account first. Could you follow the following steps for the test 1. fork the singa repo to your github acoount 2. make the forked repo private (throughout the test to ensure the safety of our server), add me as a collaborator of the forked repo so I can also access it 3. implement the github actions code for gpu test: We create a docker image for the Runner and configure the workflows. If we use Linux, we need to install the Runner <https://github.com/actions/runner> and its pre-reqs <https://github.com/actions/runner/blob/main/docs/start/envlinux.md> in the Linux docker machine. The machine should also have the github tokens from the first step. Check the Apache Arrow implementation <apache/arrow#6512> for more details. We evaluate the approach first before implementing it in Github repo — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#802 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANUF6L2MTHEJUBJ33XDZY4DSQTLY5ANCNFSM4RX6WZWQ> .

[moazreyad]

The GPU testing will improve the code coverage and by the way, there is a problem in the current testing workflows: since around two months, the Python tests disappeared from the codecov report in the dev branch and they are never executed in the master branch. Probably a pull request did something that disabled the Python tests. I tried to check quickly, but I think it needs more investigation.

The last codecov report with the Python test is here with code coverage = 70%. The codecov report of the next commit shows no Python test here with code coverage drops to 63.7%.

[moazreyad]

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

If the connection between NUS self-hosted runners and Github servers is not possible because of NUS network restrictions, then another solution can be considered: self-hosting Github itself using Github Enterprise. The cost is $21 per user per month, but there are special discounted licenses for universities. In this case both Github and the Github runners are hosted locally in the same network. (But again I guess we need to configure the DNS of the SINGA github repository to point to the locally hosted github, so we must have some NUS network configuration for external access but may be it is easier?). There is a free trail for Github Enterprise if we want to evaluate it on the NUS server.

Otherwise, we just use a cloud-based solution (from Amazon/Microsoft/Google, ..) to execute the GPU Github workflows. There will be a cost for this cloud service time and resources, but it will also save the cost we need to configure and maintain our own GPU testing pipeline.

[chrishkchris]

umm... cloud service will induce recurrent spending, e.g. if on-demand g4dn.xlarge costs ~0.526USD per hour, every month the recurrent cost seems to be around 0.526USD*24(hour)*30(days)=378USD per month
https://aws.amazon.com/cn/ec2/pricing/on-demand/
correct me if I am wrong, and yet we still need to configure and maintain the GPU testing pipeline ourselves

The only question is, @naili-xing , we cannot connect our machine from outside without a VPN. So maybe it's more feasible to listen to the update of the commit from our machine, once get such an update, run the test and push the result to the Github Actions.

If the connection between NUS self-hosted runners and Github servers is not possible because of NUS network restrictions, then another solution can be considered: self-hosting Github itself using Github Enterprise. The cost is $21 per user per month, but there are special discounted licenses for universities. In this case both Github and the Github runners are hosted locally in the same network. (But again I guess we need to configure the DNS of the SINGA github repository to point to the locally hosted github, so we must have some NUS network configuration for external access but may be it is easier?). There is a free trail for Github Enterprise if we want to evaluate it on the NUS server.

Otherwise, we just use a cloud-based solution (from Amazon/Microsoft/Google, ..) to execute the GPU Github workflows. There will be a cost for this cloud service time and resources, but it will also save the cost we need to configure and maintain our own GPU testing pipeline.

[aktech]

Hi all, I am the creator of Cirun.io, "GPU Testing" caught my eye.

FWIW I'll share my two cents. I created a service for problems like these, which is basically running custom machines (including GPUs) in GitHub Actions: https://cirun.io/

It is used in multiple open source projects needing GPU support like the following:

• https://github.com/pystatgen/sgkit/
• https://github.com/qutip/qutip-cupy

It is fairly simple to setup, all you need is a cloud account (AWS or GCP) and a simple yaml file describing what kind of machines you need and Cirun will spin up ephemeral machines on your cloud for GitHub Actions to run. It's native to GitHub ecosystem, which mean you can see logs/trigger in the Github's interface itself, just like any Github Action run.

Also, note that Cirun is free for Open source projects. (You only pay to your cloud provider for machine usage)

[moazreyad]

Thank you, but to use Cirun.io we need to install Cirun application in Github. This is not possible because SINGA github account is managed by the Apache foundation that must approve the Cirun application and allow it to be installed in SINGA repo.