improve usage of GitHub API to avoid rate throttling

[chadwhitacre]

This is our first instance of this in the wild since logging to Sentry. Here's the body of the reply from GitHub:

{"message":"API Rate Limit Exceeded for [IP address]"}

Unfortunately we're not logging the headers for that response. But we should.

Traceback snip:

  File "/app/www/on/github/%login/index.html", line 17, in <module>
    user_info = github.get_user_info(path['login'])
  File "gittip/elsewhere/github.py", line 113, in get_user_info
    raise Response(502, "GitHub lookup failed with %d." % status)

Sentry: https://app.getsentry.com/gittip/gittip/group/3381442/

[sigmavirus24]

So, are you using the client_key/client_secret pair that GitHub assigns to you? Also, have you requested a ratelimit increase for that? If not, email support [at] github and ask for a bump. I'm sure @pengwynn or @technoweenie would be happy to help you out (or whomever else takes care of the request).

[chadwhitacre]

Good call, mail sent. We should still clean up our act, though. We don't send If-Modified-Since, for example.

[sigmavirus24]

We don't send If-Modified-Since, for example.

That is a great header to include to help prevent stuff like this.

[chadwhitacre]

I just bumped you to 12500/hr.

!m @pengwynn. Thank you! :-)

[chadwhitacre]

That is a great header to include to help prevent stuff like this.

Also, I believe we're doing unauthenticated requests for /on/github/ pages, and we don't cache organizations at all. Two more improvements to look into.

[chadwhitacre]

Docs: http://developer.github.com/v3/#rate-limiting

[sigmavirus24]

@pengwynn is the man.

As for unauthenticated requests, I almost doubt that. Are you including your client key/secret pair as parameters in the URL? If so, that isn't unauthenticated last I checked.

Caching the organizations would definitely be another great idea.

[chadwhitacre]

@sigmavirus24 I just checked, and the gittip.elsewhere.github.get_user_info method does indeed perform an unauthenticated GET to URLs like: https://api.github.com/users/pengwynn.

[chadwhitacre]

@sigmavirus24 Want to take this ticket? It's about time we put you on the contributors board:

https://github.com/zetaweb/www.gittip.com/graphs/contributors

;-)

[joonas]

I'm curious, how necessary is it to incur these requests on every page load? Is the data something we could consider maybe caching for a period?

[chadwhitacre]

We do cache lookups of individual GitHub accounts (via get_user_info), but not the request for the list of organization members. That's a call to requests.get in /on/github/%login/index.html: unauthenticated, uncached.

[sigmavirus24]

@whit537 so it seems the issue about get_user_info was fixed by me... but I'm not certain of whether or not your storing the Last-Modified or ETag headers. (I think the latter is sent on almost all responses from GitHub and might be more reliable to check for.) Either way, if you can give me a pointer, I can get started on this.

[chadwhitacre]

Hacking with Bastian @??? on #1936, thinking about implementing #900, had the thought that the 400/500s we get from user login when we soak our rate limit might be due to the user_info update we do during login, and not from the login itself.

[Changaco]

We'll reopen/reticket if this becomes an issue again.