Error: 400 : Bad Request

[rafaelab]

I've been getting the following message for a few hours:
"400 : Bad Request
We couldn't render your notebook
Perhaps it is not valid JSON, or not the right URL.
If this should be a working notebook, please let us know.
The error was: HTTP 403: Forbidden"
This problem is not local and someone else in another country is also experiencing it.

None of the notebooks are working and they were not changed recently. See, for example:
http://nbviewer.ipython.org/github/CRPropa/CRPropa3-notebooks/blob/master/galactic_lensing/lensing_maps.ipynb

[kramaswamyxu]

I am having the same problem as well. none of the notebooks seem to work.

[StanczakDominik]

Can confirm here.

[willingc]

Additional reports from @fperez @ellisonbg https://groups.google.com/forum/#!msg/jupyter/xJD6P1Hg3fo/02CzfjpDDQAJ

ping @rgbkrk, @minrk @Carreau

[willingc]

pinging @bollwyvl as well. Any possibility that the parsing changes in fe8194d may be related?

[mnmetheny]

Confirming here as well.

http://nbviewer.ipython.org/github/jmportilla/

[captainsafia]

I was able to reproduce this on a couple of notebooks. I'm poking around the code right now to see what might be up.

[willingc]

It looks like the service is back up. Demo notebooks are now rendering. Was this an infrastructure issue? If so, we should document what was out and then restored.

Update: demo notebooks rendering. GitHub notebooks and gists still throwing 400 errors.

[decisionstats]

having error in my notebooks. this is scary stuff

[captainsafia]

@willingc: Ditto that. I think it is some sort of infrastructure issue. None of the recent changes made by @bollwyvl caused an issue when I set up my own nbviewer instance.

Is there anyone who can access the logs on Rackspace?

EDIT: The effects of this are pretty sporadic. Some notebooks render properly and others don't, with no commonality (to my knowledge) between those that do and those that don't. Ones that didn't render previously do. curling on a Notebook returns the rendered version just fine but displaying in the browser causes a problem.

@decisionstats: I completely understand. We are working hard to figure out what's up and resolve it!

[fperez]

I'm still seeing it, e.g. on this url.

[captainsafia]

Pinging @smashwilson in case this is a Rackspace issue (?!!?).

[Carreau]

Might be a GitHub one, they were under DDOS this night, so some of the counter measure might be affecting us.

[smashwilson]

I think you mean @rgbkrk... He's on a plane right now, though. Sorry, I don't have any access here.

[captainsafia]

@smashwilson — Yep! I knew @rgbkrk was out on vacation and thought you might have some insight. Thanks for letting me know though. 😄

[bollwyvl]

Folks:
Sorry, can't respond much from the Day Job of the Day. For those of you
that can deal with it, rawgit.com has your back, and can present your
notebooks through their CDN: just grab a nice permalink from gh and use the
production link. If you are publishing links, there's not much I can do
from here.

It could be that we need to re-roll our API key, I am not sure if kyle did
this after we deployed the fix, and there was a window during which it was
advertised that it was exposed: it's possible that it was being used for
some kind of abuse.

I'll be trying to spin up more to squash some of these issues: there are
perhaps infrastructure changes we could make, but we need to do a much
better job of caching all our external requests, as well as mocking the
upstream sources more effectively.

On Thu, Oct 22, 2015 at 1:03 PM Safia Abdalla [email protected]
wrote:

@smashwilson https://github.com/smashwilson — Yep! I knew @rgbkrk
https://github.com/rgbkrk was out on vacation and thought you might
have some insight. Thanks for letting me know though. [image: 😄]

—
Reply to this email directly or view it on GitHub
#522 (comment).

[willingc]

@captainsafia Thanks for testing and ruling out the recent PR from @bollwyvl. 🍰

Logs would help. I've seen intermittent issues with a notebook loading and then not loading.

[willingc]

@bollwyvl I wonder if a purge against Fastly is needed? #519

[willingc]

As @captainsafia, I've run the current nbviewer code locally. The local version is able to pull up and display the demo notebooks as well as the github repo notebooks. I believe that the cause of this issue is an infrastructure change that is preventing github repo/gists from being reliably displayed.

[jfost00]

confirm same error here https://github.com/jfost00/IPython/blob/master/10%20Minutes%20to%20pandas.ipynb
thank you

[bmorris3]

I'm also getting this error.

[qiuwch]

The same error here. It seems the system recovered for a while in the morning.

[tritemio]

For the record, another non-working notebook:

http://nbviewer.ipython.org/github/Photon-HDF5/photon_hdf5_reading_examples/blob/master/python/Reading%20%C2%B5s-ALEX%20data%20from%20Photon-HDF5%20with%20h5py.ipynb

[dsblank]

When I view @tritemio 's example directly on github, I see this error in the console:

[sawsimeon]

I am also getting the error with nbviewer. Same here.

http://nbviewer.ipython.org/github/ogrisel/parallel_ml_tutorial/blob/master/rendered_notebooks/06%20-%20Distributed%20Model%20Selection%20and%20Assessment.ipynb

Somebody help!!!

[sirex]

Why is this closed? I'm still getting same error:

http://nbviewer.ipython.org/gist/23609efb15541f464c0f

[rgbkrk]

Just landed in Tokyo, will see what I can do once I have wifi for my laptop. I did deploy with fresh github auth tokens, but I'll check on that again.

If I had to guess, every time this happens is likely after a purge and occurs right around when Google or other bots crawl all the content (browse able repos the culprit). I'll see what I can post of the logs, would be cool if there were other volunteers to help run this.

[fperez]

Kyle, wish I could help more... I really appreciate your help, but don't feel like it's on you. The fact that we've allowed a bottleneck to develop here is a shared issue, not your problem...

[fperez]

@sirex, this issue remains open, it's only a referenced issue in another repo that was closed.

[willingc]

For issue reporters that are patiently waiting for us to resolve the 400/403 errors, thank you.

As an interim approach: If you are comfortable using Docker, you can run nbviewer locally using these instructions: https://github.com/jupyter/nbviewer#quick-run
A locally running nbviewer instance is rendering the notebooks as expected without any 400/403 errors.

[rgbkrk]

This appears to be all set after I addressed some issues.

Problems I found:

• the CoreOS nodes that were still in rotation on Fastly as backup had the old API keys
• the main renderers (that had the right key) had not restarted after a reboot and our logs still wish they had a good home

[rgbkrk]

All the notebooks linked in here now render just fine, reopen if problems arise (or open a new one if a new problem).

[willingc]

@rgbkrk Thank you 🍰

[rafaelab]

Everything is working again.
Thanks.

[bollwyvl]

@rgbkrk heroic work, for a man on vacation. 🏆

These hosted solutions, which are somewhat unique for an organization which doesn't sell anything, are really important to the perception of the project as a whole, and its unfair that when one breaks it ends up mostly on you.

I know we're trying to get some burden off you, but I'm still spread really thin, even to ramp up, much less react to situations like this. I think we'll need a team approach, including some training... And/Or try to figure out how to pay someone to wear the pager. Unlike information, people's time ain't free.

Unfortunately, i imagine my new job would not jump at the chance to offer my services to support a free saas that competes with their lead generation saas, but that's a story and future that remains to be discussed.

Thanks again for all your efforts!

[rgbkrk]

If it's competition Continuum wants, there's more than one way to make that more interesting for them.

I'd much prefer collaboration and seeing how amenable Continuum would be to having you help out. They've certainly offered support before and my hope in them hiring you would mean that we'd have a good ally there contributing code and minimal ops. I really can't imagine that nbviewer causes conflict - it's growth for the community at large.

As for time invested, handling operations does not take much time. It's only every now and then that incidents come up. We have a bunch to automate to make it even better, which will actually end up with more incidents until it smooths over. 😉

[Carreau]

I'll try to bring up the bus factor on nbviewer once people are not traveling everywhere.

We should also link to statuspage.io page from the 40x/50x errors on nbviewer,
and give admin access to @willingc , @captainsafia (etc) ... to at least be able to update the status when something wrong.

(which I should have done yesterday, but I was busy and forgot)

[captainsafia]

@rgbkrk and I have been talking about doing a knowledge transfer to me for some of this stuff. He gave me a pretty good idea of how the infrastructure is set up earlier this summer, I just haven't had the time to figure out what to learn to be useful in the ops department.

But yes, @Carreau, it makes sense to discuss once we've got everyone stationary for a few moments! Harharhar.

[bollwyvl]

@rgbkrk yeah, these are things I'm trying to figure out :)

Personally, I want to get more experience in these areas, and figure out
how an ops perspective can help us do better things with the code, it's
just been hard to get on that train with all the other things going on!

Thanks @Carreau and @captainsafia for being proactive on this: I know you
guys are all over other stuff, too!

On Fri, Oct 23, 2015 at 1:19 PM Safia Abdalla [email protected]
wrote:

@rgbkrk https://github.com/rgbkrk and I have been talking about doing a
knowledge transfer to me for some of this stuff. He gave me a pretty good
idea of how the infrastructure is set up earlier this summer, I just
haven't had the time to figure out what to learn to be useful in the ops
department.

But yes, @Carreau https://github.com/Carreau, it makes sense to discuss
once we've got everyone stationary for a few moments! Harharhar.

—
Reply to this email directly or view it on GitHub
#522 (comment).

[willingc]

@rgbkrk Enjoy the trip. You've earned some down time 👍

@bollwyvl @Carreau @captainsafia +1 to better automated reporting of infrastructure/ops issues.

[ibtassam1]

This issue is happening again today. All my notebooks are throwing Error 400 and 502 even though they were running previously. Seems like an nbviewer-wide issue.

[minrk]

@ibtassam1 can I ask you for some information:

1. specific notebook file, and
2. how it was created (what client, versions, etc.)?

If using JupyterLab, specifically the jupyterlab, jupyter-server, and nbformat package versions will help.

Looking at https://github.com/ibtassam1/SQL_Vancouver/blob/main/Vancouver%20Public%20Services_v3.ipynb I see that it is in fact an invalid notebook (it claims v4.4, but does not satisfy the 4.4 schema).

I believe the strict validation changes in recent nbconvert and nbformat updates are incorrect, and should not prevent rendering, but it is still accurate that the notebook does not conform to the spec. I'll look into fixing/relaxing those.

[ibtassam1]

Hi minrk, thanks for the reply.

The one you quoted seems to be running now. There is another one that was rendering before and just started throwing errors today.

The one I am still facing issue with is this notebook: https://github.com/ibtassam1/SQL_Vancouver/blob/main/Vancouver%20Public%20Services_v3.ipynb

Its NBviewer link is https://nbviewer.org/github/ibtassam1/SQL_Vancouver/blob/main/Vancouver%20Public%20Services_v3.ipynb

It was created using Jupyter Notebook. I am not sure of the version but I ensured it was rendering just a couple of days ago and I made no changes since then. I hope I did not miss any crucial information this time. Still getting used to GitHub. Thanks.

[minrk]

No, and it's certainly not your fault! Something is writing nbformat v4.5 metadata (the cell id) but claiming to be v4.4, and validation is rejecting this. That's a problem

This patch: jupyter/nbconvert#1936 fixes the change in strictness issue in nbconvert, which is the immediate issue, but not the deeper one.

If you can provide the output of pip list or conda list to show package versions, we should be able to track the culprit for why you are writing notebook files with mismatched schema.

[ibtassam1]

Oh, okay. Thanks for explaining. I don't know which package version you need exactly but this is the ouput. I do not know how to truncate the displayed output so it doesn't take up entire screen unfortunately.

--------------------------------- --------------------
aiohttp                           3.8.1
aiosignal                         1.2.0
alabaster                         0.7.12
anaconda-client                   1.9.0
anaconda-navigator                2.1.4
anaconda-project                  0.10.2
anyio                             3.5.0
appdirs                           1.4.4
applaunchservices                 0.2.1
appnope                           0.1.2
appscript                         1.1.2
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
arrow                             1.2.2
astroid                           2.6.6
astropy                           5.0.4
asttokens                         2.0.5
async-timeout                     4.0.1
atomicwrites                      1.4.0
attrs                             21.4.0
Automat                           20.2.0
autopep8                          1.6.0
Babel                             2.9.1
backcall                          0.2.0
backports.functools-lru-cache     1.6.4
backports.tempfile                1.0
backports.weakref                 1.0.post1
bcrypt                            3.2.0
beautifulsoup4                    4.11.1
binaryornot                       0.4.4
bitarray                          2.4.1
bkcharts                          0.2
black                             19.10b0
bleach                            4.1.0
bokeh                             2.4.2
boto3                             1.21.32
botocore                          1.24.32
Bottleneck                        1.3.4
brotlipy                          0.7.0
cachetools                        4.2.2
certifi                           2022.6.15
cffi                              1.15.0
chardet                           4.0.0
charset-normalizer                2.0.4
click                             8.0.4
click-plugins                     1.1.1
cligj                             0.7.2
cloudpickle                       2.0.0
clyent                            1.2.2
colorama                          0.4.4
colorcet                          2.0.6
conda                             22.9.0
conda-build                       3.21.8
conda-content-trust               0+unknown
conda-pack                        0.6.0
conda-package-handling            1.8.1
conda-repo-cli                    1.0.4
conda-token                       0.3.0
conda-verify                      3.4.2
constantly                        15.1.0
cookiecutter                      1.7.3
cryptography                      3.4.8
cssselect                         1.1.0
cycler                            0.11.0
Cython                            0.29.28
cytoolz                           0.11.0
daal4py                           2021.5.0
dask                              2022.2.1
datashader                        0.13.0
datashape                         0.5.4
debugpy                           1.5.1
decorator                         5.1.1
defusedxml                        0.7.1
diff-match-patch                  20200713
dill                              0.3.5.1
distributed                       2022.2.1
dnspython                         2.2.1
docutils                          0.17.1
entrypoints                       0.4
et-xmlfile                        1.1.0
executing                         0.8.3
fastjsonschema                    2.15.1
fica                              0.2.0
filelock                          3.6.0
Fiona                             1.8.21
flake8                            3.9.2
Flask                             1.1.2
flatten-json                      0.1.13
fonttools                         4.25.0
frozenlist                        1.2.0
fsspec                            2022.2.0
future                            0.18.2
gensim                            4.1.2
geopandas                         0.11.1
glob2                             0.7
gmpy2                             2.1.2
google-api-core                   2.10.0
google-api-python-client          2.60.0
google-auth                       1.33.0
google-auth-httplib2              0.1.0
google-auth-oauthlib              0.5.2
google-cloud-core                 1.7.1
google-cloud-storage              1.31.0
google-crc32c                     1.1.2
google-resumable-media            1.3.1
googleapis-common-protos          1.56.4
greenlet                          1.1.1
grpcio                            1.42.0
gspread                           5.5.0
h5py                              3.6.0
HeapDict                          1.0.1
holoviews                         1.14.8
httplib2                          0.20.4
hvplot                            0.7.3
hyperlink                         21.0.0
idna                              3.3
imagecodecs                       2021.8.26
imageio                           2.9.0
imagesize                         1.3.0
importlib-metadata                4.11.3
incremental                       21.3.0
inflection                        0.5.1
iniconfig                         1.1.1
intake                            0.6.5
intervaltree                      3.1.0
ipykernel                         6.9.1
ipython                           8.3.0
ipython-genutils                  0.2.0
ipywidgets                        7.6.5
isort                             5.9.3
itemadapter                       0.3.0
itemloaders                       1.0.4
itsdangerous                      2.0.1
jdcal                             1.4.1
jedi                              0.18.1
Jinja2                            2.11.3
jinja2-time                       0.2.0
jmespath                          0.10.0
joblib                            1.1.0
json5                             0.9.6
jsonschema                        4.4.0
jupyter                           1.0.0
jupyter-client                    6.1.12
jupyter-console                   6.4.0
jupyter-contrib-core              0.4.0
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.9.2
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-server                    1.13.5
jupyterlab                        3.3.2
jupyterlab-pygments               0.1.2
jupyterlab-server                 2.10.3
jupyterlab-widgets                1.0.0
jupytext                          1.14.1
keyring                           23.4.0
kiwisolver                        1.3.2
lazy-object-proxy                 1.6.0
lesscpy                           0.15.0
libarchive-c                      2.9
llvmlite                          0.38.0
locket                            0.2.1
lxml                              4.8.0
mapclassify                       2.4.3
Markdown                          3.3.4
markdown-it-py                    2.1.0
MarkupSafe                        2.0.1
matplotlib                        3.5.1
matplotlib-inline                 0.1.2
mccabe                            0.6.1
mdit-py-plugins                   0.3.0
mdurl                             0.1.2
mistune                           0.8.4
mkl-fft                           1.3.1
mkl-random                        1.2.2
mkl-service                       2.4.0
mock                              4.0.3
mpmath                            1.2.1
msgpack                           1.0.2
multidict                         5.2.0
multipledispatch                  0.6.0
munch                             2.5.0
munkres                           1.1.4
mypy-extensions                   0.4.3
mysql-connector                   2.2.9
mysql-connector-python            8.0.26
navigator-updater                 0.2.1
nbclassic                         0.3.5
nbclient                          0.5.13
nbconvert                         6.4.4
nbformat                          5.3.0
nest-asyncio                      1.5.5
networkx                          2.7.1
nltk                              3.7
nose                              1.3.7
notebook                          6.4.8
numba                             0.55.1
numexpr                           2.8.1
numpy                             1.21.5
numpydoc                          1.2
oauthlib                          3.2.0
olefile                           0.46
openai                            0.25.0
openpyxl                          3.0.9
otter-grader                      4.0.1
packaging                         21.3
pandas                            1.4.2
pandas-stubs                      1.5.1.221024
pandocfilters                     1.5.0
panel                             0.13.0
param                             1.12.0
parsel                            1.6.0
parso                             0.8.3
partd                             1.2.0
pathspec                          0.7.0
patsy                             0.5.2
pep8                              1.7.1
pexpect                           4.8.0
pickleshare                       0.7.5
Pillow                            9.0.1
pip                               21.2.4
pkginfo                           1.8.2
plotly                            5.6.0
pluggy                            1.0.0
ply                               3.11
poyo                              0.5.0
prometheus-client                 0.13.1
prompt-toolkit                    3.0.20
Protego                           0.1.16
protobuf                          3.20.1
psutil                            5.8.0
ptyprocess                        0.7.0
pure-eval                         0.2.2
py                                1.11.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.7.0
pycosat                           0.6.3
pycparser                         2.21
pyct                              0.4.6
pycurl                            7.44.1
pydantic                          1.10.2
PyDispatcher                      2.0.5
pydocstyle                        6.1.1
pyee                              8.2.2
pyerfa                            2.0.0
pyflakes                          2.3.1
Pygments                          2.11.2
PyHamcrest                        2.0.2
PyJWT                             2.1.0
pylint                            2.9.6
pyls-spyder                       0.4.0
pymongo                           4.3.3
pyobjc-core                       9.0.1
pyobjc-framework-Cocoa            9.0.1
pyodbc                            4.0.32
pyOpenSSL                         21.0.0
pyparsing                         3.0.4
pyppeteer                         1.0.2
pyproj                            3.4.0
pyrsistent                        0.18.0
PySocks                           1.7.1
pytest                            7.1.1
python-dateutil                   2.8.2
python-lsp-black                  1.0.0
python-lsp-jsonrpc                1.0.0
python-lsp-server                 1.2.4
python-on-whales                  0.52.0
python-slugify                    5.0.2
python-snappy                     0.6.0
pytz                              2021.3
pyviz-comms                       2.0.2
PyWavelets                        1.3.0
PyYAML                            6.0
pyzmq                             22.3.0
QDarkStyle                        3.0.2
qstylizer                         0.1.10
QtAwesome                         1.0.3
qtconsole                         5.3.0
QtPy                              2.0.1
queuelib                          1.5.0
regex                             2022.3.15
requests                          2.27.1
requests-file                     1.5.1
requests-oauthlib                 1.3.1
rise                              5.7.1
rope                              0.22.0
rsa                               4.7.2
Rtree                             0.9.7
ruamel-yaml-conda                 0.15.100
s3transfer                        0.5.0
scikit-image                      0.19.2
scikit-learn                      1.0.2
scikit-learn-intelex              2021.20220215.132722
scipy                             1.7.3
Scrapy                            2.6.1
screeninfo                        0.8.1
seaborn                           0.11.2
Send2Trash                        1.8.0
service-identity                  18.1.0
setuptools                        61.2.0
Shapely                           1.8.4
sip                               4.19.13
six                               1.16.0
sjvisualizer                      0.0.4
smart-open                        5.1.0
sniffio                           1.2.0
snowballstemmer                   2.2.0
sortedcollections                 2.1.0
sortedcontainers                  2.4.0
soupsieve                         2.3.1
Sphinx                            4.4.0
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
spyder                            5.1.5
spyder-kernels                    2.1.3
SQLAlchemy                        1.4.32
stack-data                        0.2.0
statsmodels                       0.13.2
sympy                             1.10.1
tables                            3.6.1
tabulate                          0.8.9
TBB                               0.2
tblib                             1.7.0
tenacity                          8.0.1
terminado                         0.13.1
testpath                          0.5.0
text-unidecode                    1.3
textdistance                      4.2.1
threadpoolctl                     2.2.0
three-merge                       0.1.1
tifffile                          2021.7.2
tinycss                           0.4
tldextract                        3.2.0
toml                              0.10.2
tomli                             1.2.2
toolz                             0.11.2
tornado                           6.1
tqdm                              4.64.0
traitlets                         5.1.1
Twisted                           22.2.0
typed-ast                         1.4.3
typer                             0.6.1
types-pytz                        2022.6.0.1
typing_extensions                 4.1.1
ujson                             5.1.0
Unidecode                         1.2.0
uritemplate                       4.1.1
urllib3                           1.26.9
w3lib                             1.21.0
watchdog                          2.1.6
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  0.58.0
websockets                        10.4
Werkzeug                          2.0.3
wheel                             0.37.1
widgetsnbextension                3.5.2
wordcloud                         1.8.2.2
wrapt                             1.12.1
wurlitzer                         3.0.2
xarray                            0.20.1
xlrd                              2.0.1
XlsxWriter                        3.0.3
xlwings                           0.24.9
yapf                              0.31.0
yarl                              1.6.3
zict                              2.0.0
zipp                              3.7.0
zope.interface                    5.4.0```

[chizkidd]

I'm encountering this issue again (400: Bad Request, list index out of range). All relevant notebooks except one rendered. The notebook in question is: https://github.com/chizkidd/Karpathy-Neural-Networks-Zero-to-Hero/blob/main/006_makemore_WaveNet/makemore_WaveNet.ipynb

The broken NBviewer link is:
https://nbviewer.org/github/chizkidd/Karpathy-Neural-Networks-Zero-to-Hero/blob/main/006_makemore_WaveNet/makemore_WaveNet.ipynb