Is the tracking parameter t=hj in the duckduckgo URL intentional?

[Alexhans]

Hi,

I was looking at the different query parameters in the search urls when I found that duckduckgo's t is a tracking code (I don't know what hj as the value means:
https://help.duckduckgo.com/privacy/t/

Through partnerships with developers and companies, DuckDuckGo has been integrated into many applications. In these partnerships, a portion of DuckDuckGo's advertising revenue is sometimes shared back. To assign advertising revenue and collect anonymous aggregate usage information, developers add a unique "&t=" parameter to searches made through their applications.

howdoi/howdoi/howdoi.py

Line 72 in f202eea

'duckduckgo': SCHEME + 'duckduckgo.com/html?q=site:{0}%20{1}&t=hj&ia=web'

[gleitz]

I'm not sure if it is necessary or not, but currently duckduckgo is not working as an engine due to #404

[gleitz]

If you'd like to figure out that issue, I could use some help with it!

[Alexhans]

I can try. I actually came across this while quickly playing around to see If I could add brave support.

Support works but I wanted to understand the decisions around URLs and usage. I did get temporarily blocked when I added the unit tests in brave which it's somewhat expected (I still remember the Google has been DDoSing SourceHut for over a year story).

The only thing I can think of is ask duckduckgo & brave to see if they have specific ways to interact programmatically with their websites.

I do think the answer will not be satisfactory for duckduckgo since, in their instant answers api page they state:

This API does not include all of our links, however. That is, it is not a full search results API or a way to get DuckDuckGo results into your applications beyond our instant answers. Because of the way we generate our search results, we unfortunately do not have the rights to fully syndicate our results, free or paid. For the same reason, we cannot allow framing our results without our branding. Please see our partnerships page for more info on guidelines and getting in touch with us.

So crawling ethically (Without trying to circumvent through proxies or similar) will invariably get blocked. For DDG, it might be a case of choosing whether to remove it entirely or just support instant answers through their API (For any API based access, users could get their own tokens like in OpenBBTerminal

[gleitz]

Yes I am not optimistic that API access will be given, so we're left with crawling.

I also get rate limited during development, which is why I have the caching mechanism in place when running tests.

[gleitz]

I didn't know brave had a search engine. I would accept that PR if you want to open it.

[Alexhans]

I'll take a look at what we discussed over the weekend and create the pull request. It's been a busy period.

[gleitz]

No worries - take your time and thanks again for any support you can give to the project.