Skip to content

A few questions about the project... #5

Discussion options

You must be logged in to vote

Hello, thanks for your interest! I'll respond in points:

  1. Honestly I do not know if scraping is against Amazon's TOS, but since you are not logging in, it should not be a problem, surely it is not illegal, also, looking at their robots.txt they are not asking to not scrape product pages.
    Finally based on my experience, the bot is running on a VPS from about a year ago and it always worked.
    In any case we are doing only a few number of requests: some for collecting a set of discounted products (every 2 hours) and one request for each item sent.
  2. I think rotating user agents is meaningless: it should be fine always navigating from the same device, also that request is part of the 2 hours batch.

Replies: 3 comments 11 replies

Comment options

You must be logged in to vote
0 replies
Answer selected by albertopasqualetto
Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
11 replies
@GoukaMekkyaku
Comment options

@GoukaMekkyaku
Comment options

@albertopasqualetto
Comment options

@GoukaMekkyaku
Comment options

@albertopasqualetto
Comment options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants
Converted from issue

This discussion was converted from issue #4 on April 18, 2024 20:17.