blocked with 200 status code: "JavaScript is disabled in your browser."

[pajod]

Describe the bug
My bookmark for searching on pypi stopped working. For a GET request to https://pypi.org/search/?q=%s I get a cloudflare-esque virtual middle finger, despite the 200 status code.

Expected behavior

• usually: a search result list (status 2xx)
• possibly: a redirect to the single result (status 3xx)
• failing either: an error page with a link to https://status.python.org/ (status 4xx)

To Reproduce

• origin & rate does not appear to matter, problem even affects networks that did not send any requests this month
• the page may or may not redirect, in a browser that complies with whatever fastly wants it to do (additional javascript, fetching which results in an empty 400 response)

My Platform

• reproduced using curl 8.11.1 and Firefox 128.5.2esr

Additional context

• related: Get rid of as much JavaScript as possible #7329
• related: linkcheck in CI is broken on PyPI URLs with anchors pypa/packaging.python.org#1744
• some error messages would clarify the author, if only the fastly logo img src was specified with a leading slash
• the status page, which is more prominently linked than this issue tracker, does not mention the degradation

Specific message depends on the outcome of some obfuscated javascript:

• "Please enable JavaScript to proceed."
• "Please enable cookies to continue."
• "Oops, something went wrong."
• "is verifying your browser..."
  

[miketheman]

Please see this thread that explains what changed and why. https://discuss.python.org/t/fastly-interfering-with-pypi-search/73597/6

[mwoehlke-kitware]

I had to unblock the fastly-insights.com domain, which is blocked by default by whatever security/privacy blacklists I have configured. I understand the need to protect the service, but maybe using a service that doesn't have a reputation as "harmful" would be a good idea...

[z1atk0]

Have been bitten by this today as well. 😒 I have a daily cron script to monitor a few projects for new releases on pypi.org, among others. It basically downloads, for example, https://pypi.org/project/PyQt5-sip/#files with wget, dumps the HTML with w3m -dump, diffs the dump against yesterday's dump, and then mails the result, if any, to me, together with the URL.

After some inital cursing 🤬 and head-scratching 🤔 I finally stumbled upon https://gist.github.com/hackerb9/d382e09683a52dcac492ebcdaf1b79af via https://superuser.com/questions/666167/how-do-i-use-firefox-cookies-with-wget, so I'll modify my script to export my current Firefox cookies, and feed those to wget.

Let's see how long this works before the cookies get stale, and I'll have to open a pypi.org tab via firefox --new-tab https://pypi.org/ in addition to that. 🙄

Just thought I'd post my solu^W work^W hackaround here to save future googlers the time and frustration of doing that by themselves. 😉

[miketheman]

It basically downloads, for example, pypi.org/project/PyQt5-sip#files with wget,

Curious - why not use an official API like https://pypi.org/simple/pyqt5-sip/ to get the differences? Available in both HTML and JSON. https://docs.pypi.org/api/index-api/#get-distributions-for-project. Or even https://docs.pypi.org/api/feeds/#project-releases-feed to get the latest releases, and only get the details if anything has changed.

[z1atk0]

Yeah, in fact I actually ended up doing exactly that (using the index api) ... much less hassle. 😉

[mgorny]

I've just hit the same problem while trying to access PyPI.org via a text browser (elinks). This means that PyPI is no longer accessible without (very) rich browsers, and effectively a number of disabled folks are being cut off from the primary source of information about Python packages.