Repo clones over HTTPS are broken for slow connections

[ell1e]

Select Topic Area

Question

Body

Introduction: Repo clones over HTTPS are broken for slow connections due to what seems to be an error on GitHub's end, and GitHub apparently doesn't plan to fix it. This is can be a big issue since e.g. this also breaks any submodules or install scripts that indirectly access any GitHub repo of any kind that isn't trivially sized.

I am writing this here so people can reference this issue.

This is who is affected:

• Your connection is reliable enough to download 100MB+ files without a hiccup or need to pause and resume in at least the majority of cases.

• You can clone big repositories on gitlab.com and codeberg.org and other alternate git repository platforms without issues, other than at most rare hiccups which don't repeat on 2nd try.

• But your connection is DSL era slow, 5mbit/s or slower.

This is how it looks like:

$ git clone https://github.com/FortAwesome/Font-Awesome
Cloning into 'Font-Awesome'...
remote: Enumerating objects: 87765, done.
remote: Counting objects: 100% (12255/12255), done.
remote: Compressing objects: 100% (5196/5196), done.
error: RPC failed; curl 92 HTTP/2 stream 5 was not closed cleanly: CANCEL (err 8)
error: 4872 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

This is why it seems to be an error on Github's end:

• I found others with the same problem, although due to people usually having faster internet it's somewhat rare.

• Cloning similarly sized or magnitudes larger repositories with HTTPS from any other platform like gitlab.com works fine, just github.com is affected.

• Cloning the repositories from GitHub via SSH instead works.

• Downloading anything else, like archives, of even larger size via HTTPs from GitHub works. Only HTTPS git clone doesn't.

• GitHub's support admitted in response to me that this is likely a problem on their end, but that they don't plan to fix it any time soon. At least this is my understanding of their responses.

[ell1e]

This is still broken as of today, 2024-11-07.

[riv19]

Yes, not possible to clone repositories on 4G/5G networks. Are these networks outdated ?
And don't try to configure the client side with "http.postBuffer" and similar - doesn't help.

[ell1e]

I think it's somehow related to the server having a timeout if in time window X there were less than Y bytes transferred. And this timeout seems to have a really high threshold to clear. This means if the connection isn't like, 20mbit/s or faster, the server will just kill your connection even though it's making progress steadily. At least that's my best guess what could be going on.

[MarinoJurisic]

What in the world is this? Some horrible DoS mitigation?

GitHub's support admitted in response to me that this is likely a problem on their end, but that they don't plan to fix it any time soon

Could I see their response?

This issue can be reliably reproduced by attempting to clone a repository from github.com over HTTPS over Tor. However I have also reproduced this issue by attempting to clone a large repository through connecting to an external SOCKS5 proxy over Tor where the target system hosting the proxy is on a fast network and cloning on the target system itself succeeds which would led credence to your theory.

[ell1e]

This is a literal quote from GitHub support:

I very much agree that the root of this problem appears to be something around how we are implementing HTTPS [...] Unfortunately because we are not getting multiple reports around this issue our networking team is unlikely to make any immediate changes that might help to improve connections experiencing intermittent packet loss like this.

Even when I try to read this charitably, this seems like a sad response to me after somebody sent them around 15 e-mails and detailed network dump data (which I did) just to help confirm this issue.

[MarinoJurisic]

Unfortunately because we are not getting multiple reports around this issue our networking team is unlikely to make any immediate changes that might help to improve connections experiencing intermittent packet loss like this.

Think it is worth it for me to submit a second corroborating account of this issue to GitHub support?

[ell1e]

I have no idea, but feel free to do so if you got the time. My ticket number was 2835344 if you want to directly reference it.

[MarinoJurisic]

Of course they can just ignore it and not change anything. My thinking here is by myself sending in a corroborating account of this issue it removes the 'not getting multiple reports' excuse.