Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add user agent at getting search token #21

Open
elsemieni opened this issue Mar 14, 2024 · 0 comments
Open

Add user agent at getting search token #21

elsemieni opened this issue Mar 14, 2024 · 0 comments

Comments

@elsemieni
Copy link

I had a cron scrapping images from DuckDuckGo while suddenly stopped with the following error:

Traceback (most recent call last):
  File "/path/to/jmd_imagescraper/core.py", line 211, in duckduckgo_search
    links = duckduckgo_scrape_urls(keywords, max_results, img_size, img_type, img_layout, img_color)
  File "/path/to/jmd_imagescraper/core.py", line 82, in duckduckgo_scrape_urls
    assert match is not None, "Failed to obtain search token"
AssertionError: Failed to obtain search token

This was caused as DuckDuckGo returned a non-valid answer for https://duckduckgo.com/418.htm , which then was unable to obtain the token. I also saw other users of your library that obtained that error suddenly but they did not reported that.

This issue can be easily be solved at adding any user agent at that inital request. Here is my suggestion:

resp = requests.post(BASE_URL, data=params, headers={'user-agent': 'my-cool-user-agent/1.0.0'})

Tested that and worked again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant