Enable github private repos. #103
Conversation
|
At Clearpath, we use SSH URLs, and that works totally fine with Github, Bitbucket, and GitLab. Additionally, you can put auth tokens in your URLs, as of #93. |
|
@mikepurvis @dirk-thomas any chance of getting this in? It is currently blocking my buildfarm setup because the docker image doesn't have my patches to the ros python package since it is installing them via pip. I guess I could find the docker script and modify it to checkout my branch, but seems like a lot of work. If there is an alternative to using https credentials, such as git+ssh that doesn't require downloading your rosdistro and ros_buildfarm_config locally with a file:/// url, let me know. But I have tried just putting the access tokens in the urls and that doesn't work with https://[email protected]/PRIVATE_REPO/rosdistro/master/index.yaml. I guess the next hurdle would be figuring out how to define the environment variables in the docker image for the https credentials. |
|
@mikepurvis I've got a temporary fix working. I can host my rosdistro, ros_buildfarm_config, and apt repository locally via the docker address for localhost: http://172.17.0.1/ Then I can use my patched ros distro tools to create secure PR's on github with bloom, so no rush, but still would be good to get this in for others to use. |
| GITHUB_PASSWORD = os.getenv('GITHUB_PASSWORD', None) | ||
|
|
||
|
|
||
| def auth_header(username=None, password=None, token=None): |
There was a problem hiding this comment.
I don't understand what's going on in this function, and in particular why it only needs to happen here and not for the manifest providers as well. Should this be a generic thing that's used everywhere?
There was a problem hiding this comment.
It is just creating either a basic or token auth header. I had to have the same function in bloom and ros_biuldfarm as well. So if there is one module that all of these import, we can put it there.
| def load_url(url, retry=2, retry_period=1, timeout=10, skip_decode=False): | ||
| req = Request(url) | ||
| if GITHUB_USER and GITHUB_PASSWORD: |
There was a problem hiding this comment.
I don't care for passing my github credentials to an arbitrary URL— shouldn't we check here if the request URL is in fact Github before doing this?
Alternatively, maybe some of this logic should be hoisted into the auth_header function; pass it the URL and have it do the business of checking the host and then pulling the credentials or token from the environment as appropriate.
There was a problem hiding this comment.
GITHUB_USER is a bit mis-leading, this also works for bitbucket as far as I can tell from the existing usage of it, but it seems that naming convention was left over. https://developer.atlassian.com/bitbucket/server/docs/latest/how-tos/example-basic-authentication.html
The issue would be if someone was using gitlab. If we wanteed to do something like bloom does and keep the settings in a config file, we could have user enter auth for each url in a dict.
{
"https://www.github.com/my_company/my_repo": ["user", "password"]
}Or something like that.
There was a problem hiding this comment.
Right, but for my use case, I have repos at Github, Bitbucket, and Gitlab, and I have different credentials for all three in the environment that fetches manifests to build the cache.
I would prefer if this logic followed the same pattern and used the correct set of credentials based on where the rosdistro is located (rather than forcing me to pass my Bitbucket credentials as GITHUB_USER/GITHUB_PASSWORD).
There was a problem hiding this comment.
I'm suggesting we do something like the following, this would work for you correct?
cat .config/rosdistro
{
"https://www.github.com/my_company/my_repo": ["user", "password"],
"https://www.bitbucket.com/my_company/my_repo": [user="user", password="password"],
"https://www.mygitlab.com/my_team/my_repo": [user="user", password="password"],
"https://www.github.com/my_company/my_repo2": [token="token"],
}
There was a problem hiding this comment.
I'm not seeing a motivation to depart from the convention already established in this repo.
There was a problem hiding this comment.
OK, are automated pull requests working for you? I noticed most of it was broken for private repo and did all of this to get it working. Am I missing something? It would be good to add secure pr's to the unit tests to get coverage over this code.
There was a problem hiding this comment.
And what is the current method of handling raw files from github lIke for distro.yaml. I don't think there is any, correct?
There was a problem hiding this comment.
The existing convention works for building the rosdistro cache from private repos on Github and Bitbucket, and is implemented in the manifest providers:
https://github.com/ros-infrastructure/rosdistro/tree/master/src/rosdistro/manifest_provider
The envvars in question are GITHUB_USER, GITHUB_PASSWORD, BITBUCKET_USER, and BITBUCKET_PASSWORD. My request is that we use these same envvars for this new logic, but use the correct ones depending on where the rosdistro is hosted.
There was a problem hiding this comment.
Ah, OK, sorry for misunderstanding. I hadn't seen BITBUCKET_USER and BITBUCKET_PASSWORD yet.
There was a problem hiding this comment.
My apologies also; I should have been more explicit upfront.
|
@jgoppert What is the status of this PR? Is it obsolete and can be closed since the environment variables are sufficient? |
|
I will go ahead and close this PR due to the long time of inactivity. Please feel free to comment and if necessary the ticket can be reopened. |
Hi, could you maybe provide an example of such an URL to download the index file from Bitbucket? I am trying to set up a private ROS buildfarm but it looks like I'm missing some (trivial?) way how to access an |
|
@lsolanka-flya There may well be some way to do this, but if so I'm not sure what it is— we've always had our private rosdistro repo world-readable behind our firewall. |
|
@mikepurvis sorry to hijack this thread, but if you were to set up a buildfarm with rosdistro as a private repo, would you see it as a realistic solution? I see a lot of code in ros-infrastructure/ros_buildfarm that depends on various configuration repositories (such as rosdistro, ros_buildfarm_configuration, etc.?) to be publicly available through https so I'm wondering whether it's even worth digging into it. Thanks for an answer. |
a nuance here: while I agree it would be good to have support for private repositories, it would seem the actual requirement here is "repositories that can only be accessed after authentication". Github only has that for its private repositories, but that is just one implementation. There is no requirement in the packages that you list for repositories to be "public" (ie: accessible by anyone in the world, or hosted on github per se), but it is easier if those repositories are accessible without requiring authentication (note the difference). I believe that is what @mikepurvis implies: repositories can be made available without requiring authentication by placing them behind a firewall (or some other form of network isolation). This would allow your company/organisation/buildfarm to access them, but not anyone (or anything) else. |
@gavanderhoorn thanks for the reply. However that to me implies that also my buildfarm is generally behind that firewall, which is not the case I'm trying to set up. But perhaps I'm missing something here... |
|
I don't believe that has to be the case. Firewalls can be configured to allow (and deny) all sorts of traffic through or block it. You could place your configuration repositories behind a firewall, make sure your buildfarm only has access to it, and still allow "the world" to access your buildfarm You'd have to make sure to not allow unauthenticated users access to your job configuration, but that would keep everything relatively secure (or at least: as secure as a setup with authentication for repositories would seem to be). Perhaps you could detail which parts of your buildfarm you'd want to expose. If it's just the |
|
Yeah, no problem, let me just set the stage a little (and thanks for you time!). Also I have to say I'm not an expert on the networking infrastructure so I might be quite wrong on some things. I am trying to set up a completely private buildfarm in my company (including the apt repository), entirely on the cloud as I don't have in-house resources to maintain the farm (nor to set up firewall access and external access such as VPN if needed). The jenkins part would be hosted by AWS and the apt repository server I guess also on AWS. All sources are in private repos (Github/Bitbucket), including forks of https://github.com/ros/rosdistro and https://github.com/ros-infrastructure/ros_buildfarm_config. Here I hit the roadblock I am describing, since to me world-readable in this configuration means also public. |
|
If you're hosting your repositories on networks that are not under your control (and using Github and Bitbucket would fall in that category) then it would seem you can't use the setup @mikepurvis describes. I'm guessing clearpath has an EE deployment of either gitlab or github (or some other platform) which is on-premise or hosted in a cloud environment they can control. |
|
If you don't like having to mirror the rosdistro locally, another option might be to throw a proxy in front of it that takes care of the auth for you. An example of this setup is here: https://stackoverflow.com/a/571260 Admittedly, this is still a hack, and I can definitely understand wanting a more native/transparent solution for this. |
Needs: ros-infrastructure/bloom#429