Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to support TheGamesDB v2 api #252

Merged
merged 11 commits into from
Aug 19, 2019

Conversation

J-Swift
Copy link
Contributor

@J-Swift J-Swift commented Aug 17, 2019

This should address #230, #237, #244, and #245

TheGamesDB updated to a new api a while back and v2 is pretty different:

  • JSON instead of XML
  • Request/response schemas are different
  • Requires an API key (you can use my public key for now to test this 83cafde7c61f6f57f523241ca6abe58165cdc42c704ae0cd953a6846a9a463b6)

This is the initial conversion of the existing code to support the new version. It relies on a "temporary" swagger client I generated here, but there is a decent amount of hand cleanup to get to that point. See this issue I opened for more info. Once that is cleaned up, addressing future API updates should be fairly straightforward.

The new API supports batching which is great, but not currently implemented/supported in the codebase. I might tackle that next, as the API limits are not very generous (1500 req/month) and so batching is the only way a large initial scrape is going to work. I currently am making 4 + 2 * {number of games} API requests per run of the scraper.

Note I had to learn golang for this and so let me know if there are any issues with how I've coded it up.

ds/gdb.go Outdated Show resolved Hide resolved
scraper.go Show resolved Hide resolved
gdb/gdb.go Show resolved Hide resolved
gdb/gdb.go Outdated Show resolved Hide resolved
gdb/gdb.go Show resolved Hide resolved
@sselph
Copy link
Owner

sselph commented Aug 19, 2019

Thanks for the hard work. I added some comments and questions.

@J-Swift
Copy link
Contributor Author

J-Swift commented Aug 19, 2019

Updated the quick ones, commented on the others. I can still make those style changes you mentioned and add an API key if you provide one, otherwise I defer to you on merging this in.

@sselph sselph merged commit 509443b into sselph:master Aug 19, 2019
@sselph
Copy link
Owner

sselph commented Aug 19, 2019

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants