Fix 403 authorization errors #128

Teuns · 2024-11-20T22:34:24Z

Probably related: #127

For some reason the authentication process returns 403 errors all of a sudden. I think this is due to CloudFlare blocking the requests automatically. This branch solves it by using urllib instead of requests in the specific faulty requests, as pointed out in a StackOverflow post](https://stackoverflow.com/a/74674276).

…hentication process

lubomir-brindza · 2024-11-21T09:41:33Z

kobodl/kobo.py

+        def get_cookie_value(cookie_name):
+            if cookie_name in cookie:
+                return cookie[cookie_name].value
+            return None


This closure is never called; did you mean to leave this in here?

e: Looking at this some more, there's some redundant code here; this whole block (529-543) seems like it's a leftover and no longer necessary

Nice catch! That definitely shouldn't be in here. I'll delete it

lubomir-brindza · 2024-11-21T09:47:24Z

I can confirm that this PR does fix the login issue for me, nice work :)

@subdavis apologies for barging in here, I just know from the related issues that you're not affected / able to reproduce this

lubomir-brindza · 2024-11-21T10:19:55Z

kobodl/kobo.py

@@ -166,7 +184,7 @@ def __GetExtraLoginParameters(self) -> Tuple[str, str, str]:
            )
        requestVerificationToken = html.unescape(match.group(1))

-        return koboSignInUrl, workflowId, requestVerificationToken
+        return koboSignInUrl, workflowId, requestVerificationToken, authCookie


Don't forget to update the return type for __GetExtraLoginParameters to Tuple[str, str, str, str]

…_GetExtraLoginParameters function

bybpow · 2024-11-21T18:33:28Z

Please merge this soon. I reseted my account and now I can't add a user or download. Thanks for your work.

chewi · 2024-11-21T21:46:23Z

This doesn't seem to fix it for me, but I'm seeing it on the download request.

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX

subdavis · 2024-11-21T22:25:44Z

Thanks everyone for your involvement in this fix. I'll get this merged and released later this evening.

rainrdx · 2024-11-26T22:51:05Z

This doesn't seem to fix it for me, but I'm seeing it on the download request.
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX

Seems to still get 403 Forbidden on my end sadly

Teuns · 2024-11-27T11:22:20Z

This doesn't seem to fix it for me, but I'm seeing it on the download request.
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX
Seems to still get 403 Forbidden on my end sadly

Hmm, strange. I suppose it could be fixed by replacing that request with urllib too. I'll take a look at it. What OS are you running it on?

rainrdx · 2024-11-27T11:39:31Z

This doesn't seem to fix it for me, but I'm seeing it on the download request.
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX
Seems to still get 403 Forbidden on my end sadly
Hmm, strange. I suppose it could be fixed by replacing that request with urllib too. I'll take a look at it. What OS are you running it on?

I’m running it on macOS but I’m on vacation so I’m from weird IP. Maybe it’s IP related and I will try again once I’m back home with residential IP in the U.S.

chewi · 2024-11-27T13:26:09Z

Gentoo here, running at home in Scotland on a static IP. I don't think it's the IP though. I know that CloudFlare will block for all sorts of reasons. Lack of a User-Agent string is just one. I suspect urllib could be configured to be acceptable, but it's hard to know exactly what CloudFlare doesn't like about it.

Teuns · 2024-11-29T01:36:51Z

I've just pushed some commits that introduces a (WIP) request wrapper that revolves around urllib and replaced all requests with it. I successfully authenticated and downloaded an ebook with it using a VPN. The code obviously needs to be refacatored and tested properly but feel free to test it let me know if it works.

rainrdx · 2024-11-29T02:12:20Z

Thank you so much for working on this. I tried two ips, my ip on a cruise ship and another ip on azure, both shows 403 forbidden when requesting
Making request to https://authorize.kobo.com/signin?returnUrl

I feel this is more likely an ip issue with where I am. I will try again next week when I am back.

rainrdx · 2024-11-29T03:36:21Z

and I tried so many VPNs all failed. This is so weird. Again thank you for working on this weird issue

bybpow · 2024-11-29T20:28:44Z

This works for me! Thanks for your contribution.

chewi · 2024-11-29T23:46:11Z

No luck here, although the error is different.

Making request to https://storeapi.kobo.com/v1/initialization
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Skipping subscribtion entity
Skipping subscribtion entity
Downloading e0a9b304-83f1-4a7d-a170-4aee884d35c2 to /tmp/audiobooks/Adrian Tchaikovsky - Eyes of the Void e0a9b304
Making request to https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/commands/book.py", line 83, in get
    actions.GetBookOrBooks(usercls, output_dir, formatStr=format_str, productId=pid)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/actions.py", line 231, in GetBookOrBooks
    kobo.Download(bookMetadata, book_type == BookType.AUDIOBOOK, outputFilePath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 428, in Download
    self.__DownloadAudiobook(downloadUrl, outputPath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 347, in __DownloadAudiobook
    request = Request(url)
              ^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 59, in __init__
    self.request = urllib.request.Request(url, data=data, headers=headers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 324, in __init__
    for key, value in headers.items():
                      ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'items'

Admittedly I've tweaked it to use Python 3.12, 3.10 is too old for my system, but it did work fine with 3.12 before.

Teuns · 2024-11-30T01:13:17Z

No luck here, although the error is different.

Making request to https://storeapi.kobo.com/v1/initialization
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Skipping subscribtion entity
Skipping subscribtion entity
Downloading e0a9b304-83f1-4a7d-a170-4aee884d35c2 to /tmp/audiobooks/Adrian Tchaikovsky - Eyes of the Void e0a9b304
Making request to https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/commands/book.py", line 83, in get
    actions.GetBookOrBooks(usercls, output_dir, formatStr=format_str, productId=pid)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/actions.py", line 231, in GetBookOrBooks
    kobo.Download(bookMetadata, book_type == BookType.AUDIOBOOK, outputFilePath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 428, in Download
    self.__DownloadAudiobook(downloadUrl, outputPath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 347, in __DownloadAudiobook
    request = Request(url)
              ^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 59, in __init__
    self.request = urllib.request.Request(url, data=data, headers=headers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 324, in __init__
    for key, value in headers.items():
                      ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'items'

Admittedly I've tweaked it to use Python 3.12, 3.10 is too old for my system, but it did work fine with 3.12 before.

Seems like urllib doesn't like it when you pass None as the headers value (which is of course not very surprising given the fact that the type is a dictionary). My latest commit should fix it!

chewi · 2024-11-30T10:33:04Z

That did the trick. Many thanks.

bybpow · 2024-11-30T12:12:34Z

No luck here, although the error is different.

Making request to https://storeapi.kobo.com/v1/initialization
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Making request to https://storeapi.kobo.com/v1/library/sync
Skipping subscribtion entity
Skipping subscribtion entity
Downloading e0a9b304-83f1-4a7d-a170-4aee884d35c2 to /tmp/audiobooks/Adrian Tchaikovsky - Eyes of the Void e0a9b304
Making request to https://storedownloads.kobo.com/download?downloadToken=XXXXXXXX
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/.cache/pypoetry/virtualenvs/kobodl-NpcgVQ5e-py3.12/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/commands/book.py", line 83, in get
    actions.GetBookOrBooks(usercls, output_dir, formatStr=format_str, productId=pid)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/actions.py", line 231, in GetBookOrBooks
    kobo.Download(bookMetadata, book_type == BookType.AUDIOBOOK, outputFilePath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 428, in Download
    self.__DownloadAudiobook(downloadUrl, outputPath)
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 347, in __DownloadAudiobook
    request = Request(url)
              ^^^^^^^^^^^^
  File "/home/chewi/Projects/kobo-book-downloader/kobodl/kobo.py", line 59, in __init__
    self.request = urllib.request.Request(url, data=data, headers=headers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/urllib/request.py", line 324, in __init__
    for key, value in headers.items():
                      ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'items'

Admittedly I've tweaked it to use Python 3.12, 3.10 is too old for my system, but it did work fine with 3.12 before.

Seems like urllib doesn't like it when you pass None as the headers value (which is of course not very surprising given the fact that the type is a dictionary). My latest commit should fix it!

This also happened to me. You also need to add the cloudscrapper patch but this fix works for the login.

rainrdx · 2024-12-01T03:25:36Z

I just want to share that TnS-hun's recent commits fixed the 403 problem for me
https://github.com/TnS-hun/kobo-book-downloader/commits/master/

Teuns · 2024-12-01T13:39:46Z

I just want to share that TnS-hun's recent commits fixed the 403 problem for me https://github.com/TnS-hun/kobo-book-downloader/commits/master/

Interestingly enough it doesn't work for me:

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://authorize.kobo.com/signin (...)

Maybe you're using it on an IP address that doesn't get blocked easily by CloudFlare? It used to work weeks before though.

rainrdx · 2024-12-01T14:00:10Z

I just want to share that TnS-hun's recent commits fixed the 403 problem for me https://github.com/TnS-hun/kobo-book-downloader/commits/master/

Interestingly enough it doesn't work for me:

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://authorize.kobo.com/signin (...)

Maybe you're using it on an IP address that doesn't get blocked easily by CloudFlare? It used to work weeks before though.

That's really weird. I think I will just stop here as it's really hard to debug this. Cloudflare is just flaky to bypass and I got my tokens. Thank you!

rainrdx · 2024-12-01T22:40:55Z

Just came back home and with clean ip, everything works with your commits. I guess this was just an ip issue with cloudflare. Thank you!

rainrdx · 2024-12-09T01:39:42Z

sadly, seems nothing works for me now. My home IP, my office IP, etc. all show up with the 403 forbidden for authorize.kobo.com sigh....

Teuns added 2 commits November 20, 2024 22:07

Fix 403 authorization errors by replacing requests with urllib in aut…

31ea806

…hentication process

Revert not needed user agent change, change cookie_string to authCookie

eea660d

lubomir-brindza reviewed Nov 21, 2024

View reviewed changes

Remove unused get_cookie_value value, return type for authCookie in _…

3657ace

…_GetExtraLoginParameters function

Teuns added 6 commits November 29, 2024 01:34

WIP: Use urllib in all requests, create Request Wrapper

ed34c02

Untested: Fix downloading audio books

02e8b44

Remove unused import

76f8867

Remove unused Session variables, replace it with headers instead

8b582e2

Remove gzip related code

3739677

Remove gzip related code (2)

854fb67

Fix downloading audiobooks

0e0a91e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 403 authorization errors #128

Fix 403 authorization errors #128

Teuns commented Nov 20, 2024

lubomir-brindza Nov 21, 2024 •

edited

Loading

Teuns Nov 21, 2024

lubomir-brindza commented Nov 21, 2024 •

edited

Loading

lubomir-brindza Nov 21, 2024

bybpow commented Nov 21, 2024

chewi commented Nov 21, 2024

subdavis commented Nov 21, 2024

rainrdx commented Nov 26, 2024

Teuns commented Nov 27, 2024

rainrdx commented Nov 27, 2024

chewi commented Nov 27, 2024

Teuns commented Nov 29, 2024

rainrdx commented Nov 29, 2024

rainrdx commented Nov 29, 2024

bybpow commented Nov 29, 2024 •

edited

Loading

chewi commented Nov 29, 2024

Teuns commented Nov 30, 2024

chewi commented Nov 30, 2024

bybpow commented Nov 30, 2024

rainrdx commented Dec 1, 2024

Teuns commented Dec 1, 2024

rainrdx commented Dec 1, 2024

rainrdx commented Dec 1, 2024

rainrdx commented Dec 9, 2024

Fix 403 authorization errors #128

Are you sure you want to change the base?

Fix 403 authorization errors #128

Conversation

Teuns commented Nov 20, 2024

lubomir-brindza Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

Teuns Nov 21, 2024

Choose a reason for hiding this comment

lubomir-brindza commented Nov 21, 2024 • edited Loading

lubomir-brindza Nov 21, 2024

Choose a reason for hiding this comment

bybpow commented Nov 21, 2024

chewi commented Nov 21, 2024

subdavis commented Nov 21, 2024

rainrdx commented Nov 26, 2024

Teuns commented Nov 27, 2024

rainrdx commented Nov 27, 2024

chewi commented Nov 27, 2024

Teuns commented Nov 29, 2024

rainrdx commented Nov 29, 2024

rainrdx commented Nov 29, 2024

bybpow commented Nov 29, 2024 • edited Loading

chewi commented Nov 29, 2024

Teuns commented Nov 30, 2024

chewi commented Nov 30, 2024

bybpow commented Nov 30, 2024

rainrdx commented Dec 1, 2024

Teuns commented Dec 1, 2024

rainrdx commented Dec 1, 2024

rainrdx commented Dec 1, 2024

rainrdx commented Dec 9, 2024

lubomir-brindza Nov 21, 2024 •

edited

Loading

lubomir-brindza commented Nov 21, 2024 •

edited

Loading

bybpow commented Nov 29, 2024 •

edited

Loading