Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demo only scrapes hackernews? #10

Open
Joshfindit opened this issue May 16, 2018 · 2 comments
Open

Demo only scrapes hackernews? #10

Joshfindit opened this issue May 16, 2018 · 2 comments

Comments

@Joshfindit
Copy link

Just tried exploring the queries with Tesco as per https://news.ycombinator.com/item?id=11180732 and the results are empty.
Also tried other sites besides hackernews, and could not get results.

It's understandable if it's whitelist-only, but there's no documentation about it which leads users (me) to think that either we're doing it wrong or it's broken.

@mrVanDalo
Copy link

this is most likely because theses sides use something like react to render there pages, which means the rendering of the html is done by your browser. But if a page is delivered in html (not in js which generates html) it works for me.

@mrVanDalo
Copy link

Hmm seems the page is not generated by js. Maybe there is whitelisting (but I don't think so), try to run your own instance, I created a docker container a while ago which should make a test run very easy .

https://hub.docker.com/r/palo/gdom/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants