error Connection broken in reload (might not be your fault)

[juliatau]

hello,
I made myself a script to download every fic with new chapters.
it worked fine at the beginning until I had too many subscriptions and got rate limited so I added a delay to avoid the issue, but now I'm having a Connection Broken error. I don't think it's a problem with my internet connection

File "/usr/lib/python3/dist-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 755, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/usr/lib/python3/dist-packages/urllib3/response.py", line 699, in _handle_chunk
    value = self._fp._safe_read(amt)
  File "/usr/lib/python3.8/http/client.py", line 611, in _safe_read
    raise IncompleteRead(data, amt-len(data))
http.client.IncompleteRead: IncompleteRead(5977 bytes read, 4263 more expected)

here is the relevant code

def loadWorkMetadata(work):
        work.reload(load_chapters=False)
        sleep(DELAY)
    return None

def loadWorksMetadata(works):
    threads = []
    loadedWorks = []
    print(f"starting to check you {Fore.CYAN}{len(works)}{Fore.RESET} subsciption")
    for work in works:
        loadedWorks.append(work)
        newThread = threading.Thread(target=loadWorkMetadata, args=([work]))
        newThread.start()
        threads.append(newThread)
    for thread in threads:
        thread.join()
    return loadedWorks

[ArmindoFlores]

I've never gotten that error, but have you tried just using AO3.utils.limit_requests()?

[juliatau]

thanks, it seems to work better because I can load works sometimes but now on the time I don't get the first error incomplete reads during the reload of all the work, I'm getting this later in the execution. at the download step

Exception in thread Thread-164:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1344, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 276, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

it only happened on 4 of the work I tried to download. and the rest was fine.

il do more test and come back with more info