Add usage docs

[zeke]

I'm hoping to get this buildpack updated for compatibility with the Heroku successor to the user-env-compile labs feature.

The README mentions using this buildpack to keep secrets out of package.json, but it's not clear to me how this works in practice. I did a little investigating and it doesn't appear that npm uses .netrc or environment variables when fetching dependencies.

@timshadel, can you shed some light on this?

cc @friism

[ryanbrainard]

@zeke: FYI, Pull #6 removes the user-env-compile dependency.

[rwdaigle]

Hi Gents,

Some good usage docs would be great here. Particularly, I'm assuming:

1. You need to use something like the multi-buildpack with this buildpack to do anything real with it?
2. That ruby's Bundler will utilize the .netrc credentials when it invokes bundle install for an app with private repo dependencies?

You can imagine I'm here because this is not working as expected :)

-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/timshadel/heroku-buildpack-github-netrc.git
=====> Detected Framework: github-netrc
       Generated .netrc & .curlrc files (available only at build-time)
       Github User:   rwdaigle
       Organizations: spreedly
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-ruby.git
=====> Detected Framework: Ruby
-----> Compiling Ruby/Rack
-----> Using Ruby version: ruby-2.1.1
-----> Installing dependencies using 1.5.2
       Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment
       Fetching gem metadata from https://rubygems.org/.........
       Fetching additional metadata from https://rubygems.org/..
       Fetching [email protected]:spreedly/caymans.git
       Host key verification failed.
       fatal: The remote end hung up unexpectedly

[rwdaigle]

My issue turned out to be that you can't use git@git:// style URLs in your Gemfile (since that forces the user to be git and not the one in your .netrc) You have to use https:// without any embedded auth info. This has some other ramifications as well, mostly for local dev workflow. Once I nail down a process I'm happy with I'll see what I can do about updating the README.

[timshadel]

The local dev flow is pretty awesome with https:// URLs with no authentication. You simply create a local ~/.netrc file, and everything works great.

This buys you several things.

1. You don't mix code access with dev files (no usernames in repo URLs). You manage rights at your repo host (Github, etc).
2. You don't risk storing credentials in your code repo. Just like other 12 Factor (http://12factor.net) principles, keeping auth info out of your repo so you could (in theory) make your code open source at any point is a big benefit.
3. You don't have to worry about accidentally giving someone access to other repos by giving them access to one dependency.

Hope that helps!
—
Sent from Mailbox

On Thu, May 1, 2014 at 6:58 PM, Ryan Daigle [email protected]
wrote:

My issue turned out to be that you can't use git@git:// style URLs in your Gemfile (since that forces the user to be git and not the one in your .netrc) You have to use https:// without any embedded auth info. This has some other ramifications as well, mostly for local dev workflow. Once I nail down a process I'm happy with I'll see what I can do about updating the README.

Reply to this email directly or view it on GitHub:
#5 (comment)

[zeke]

Hi @rwdaigle! Thanks for jumping in on this.

[rwdaigle]

Hi Tim,

The local dev flow is pretty awesome with https:// URLs with no authentication. You simply create a local ~/.netrc file, and everything works great.

Unfortunately, this is quite a departure from the normal local development flow. Especially since you can't just populate your .netrc with your normal GH username/password – you have to use a token. Same goes for just logging in when prompted on git fetch.

Username for 'https://github.com': rwdaigle
Password for 'https://[email protected]': <normal password>
fatal: Authentication failed

That's the biggest hiccup to me in this workflow. Not only do you have to modify all your private repo git URLs (from git@ to https), you also have to setup token-based credentials outside the normal git workflow.

I'd be curious to know if this has been your experience as well? And if so, does it being the Right Way:tm: to do things justify the difficulties?

[timshadel]

Hey Ryan,

Awesome questions. Let me tackle them out of order.

I'd be curious to know if this has been your experience as well? And if so, does it being the Right Way to do things justify the difficulties?

Nope. You probably have something wrong with your regular Github password, or used the wrong URL or something. You might try:

GIT_CURL_VERBOSE=1 git fetch

What happens when git fetch asks for creds is that it uses them directly as the Basic auth sent with curl, and never stores them unless you've configured a credential helper. That failure has nothing to do with ~/.netrc. Sorry.

Especially since you can't just populate your .netrc with your normal GH username/password – you have to use a token.

Actually, that's not true. It's precisely because the ~/.netrc file is intended to simply use your username and password that it was chosen as an easy way to give the build process private repo access.

If I put this in my ~/.netrc, my git fetch commands work just perfectly.

machine github.com login timshadel password <my-gh-pass>

Unfortunately, this is quite a departure from the normal local development flow.

So because this is just a way to supply usernames & passwords to git, you can use any of the typical ways to do that locally. I use the osxkeychain credential-helper. Again, once all your private repo references are set with https and have no usernames in them, at that point you can use pretty much any of the various credential approaches to using git, and any of the different kinds of credential available (credential-helper, netrc, etc) with Github in particular (user/pass, token). Nicely factored config.

Why not stick to user/pass in .netrc?

Now, the question becomes, Do I really want my password lying around in plain text? That's where the whole token piece comes in. Tokens are only used because they're easily revoked, and they can be used anywhere in Github's infrastructure that Basic auth can be used.

That means even when you have no ~/.netrc file, and git pops up and asks you for your auth information, you can use the token directly, if you really want. The password is always x-oauth-basic (see links in the README.md).

Username for 'https://github.com': <my-gh-TOKEN>
Password for 'https://<my-gh-TOKEN>@github.com': x-oauth-basic

HTTPS is the default URL protocol for Github repos

Authenticating to Github and storing the credentials for https repos is already well documented in Github Help.

Notice that both in Github's docs and in their UI (clone URL), SSH is an afterthought. You have to go digging to find it. They've clearly shifted emphasis to https URLs (likely because of how much simpler it is to use those through corporate firewalls, I'd guess).

So unlike you and I who have used Github since the beta, most newer folks will only see https URLs, so this is the new normal workflow.

So, the main motivation for using ~/.netrc at all in a buildpack is to provide a username & password to git during the build process as it clones repos with https URLs with no embedded authentecation, presumably for private dependencies. That's it.

you also have to setup token-based credentials outside the normal git workflow

Hopefully by now it's obvious that you don't need to. You can, yes, but you don't need to do this for local work—only to use this buildpack.

you have to modify all your private repo git URLs (from git@ to https)

Yeah, this is the one-time pain point. There's a day or so of git remote set-url stuff, and editing a pile of Gemfile or package.json or other files.

But the amazing thing is that npm, bundle, and a bunch of other tools will play nice with this approach.

One More Thing

If create a token using the curl command in the README, and then you want to play with Github's API, using curl and jq for example, then you'll want to make sure you've got two lines in your ~/.netrc.

machine github.com login <my-gh-token> password x-oauth-basic
machine api.github.com login <my-gh-token> password x-oauth-basic

Then you can use curl to see your repos.

curl -n https://api.github.com/user/repos

To get rid of the -n, open the curl config file ~/.curlrc. That'll also help out tools that don't automatically ask curl to look at the ~/.netrc file while also letting those tools specify alternate creds.

--netrc-optional

[zeke]

This thread is amazing. @timshadel would you be interested in collaborating on a Heroku blog post or Dev Center article?

[timshadel]

Wow. Sure, I guess. Is it used widely enough that you get questions like this?? But sure. What's the best way to get in touch?

[zeke]

We don't necessarily get direct questions about this, but users are always searching for new and elegant ways to access private dependencies in the build process. This .netrc technique is more elegant and secure than other approaches I've seen, and fits well with Heroku's way of solving problems. It's also a perfect example of what composable buildpacks are all about.

Email me at [email protected]

[rwdaigle]

Tim,

I believe I know why I was experiencing this anomaly:

you can't just populate your .netrc with your normal GH username/password – you have to use a token

I have Two-factor auth turned on. I suspect with that setting, you can't just use your normal user/pass in the .netrc. Two-factor auth is quickly becoming the default for most orgs, so it's unfortunate that it causes this wrinkle for local dev using a .netrc.

[zeke]

👋 everyone

Just stumbled upon this repo/issue almost two years later. :)