http2.request is downloading data with a very slow speed, if the network latency is significant

[segfault-bilibili]

Node.js Version

v22.5.1

NPM Version

v10.8.2

Operating System

Linux VPS 6.8.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 7 15:25:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

http2

Description

As the title suggests, the downloading speed could be around 200-300KB/s if latency is about 200ms.

If the latency is negligible, like <5ms, the downloading speed seems to be normal.

I tried createConnection callback with tls.connect() as well, after setting highWaterMark to 1048576 the speed still didn't seem to recover to normal.

Minimal Reproduction

1. Set up a cloud server VPS and install caddy http2 server on it. For example, the server VPS is located in Los Angeles, United States.
2. Put some download test file on the server VPS.
3. Set up another cloud VPS as the client. For example, the client VPS is located in Singapore.
4. On the client VPS, use curl --http2 to test download speed, which looks normal.
5. Then use something like the following script to test http2 downloading speed in NodeJS, which is very slow.

async function testDownload(VPS_DOMAIN_NAME, TEST_FILENAME) {
    let session = await new Promise((resolve) => {
        http2.connect(`https://${VPS_DOMAIN_NAME}`, (session) => {
            console.log("connected");
            resolve(session);
        });
    });
    let opts = {
        [http2.constants.HTTP2_HEADER_METHOD]: http2.constants.HTTP2_METHOD_GET,
        [http2.constants.HTTP2_HEADER_PATH]: `/${TEST_FILENAME}`,
    }
    let total = 0; startTime = Date.now();
    await new Promise((resolve) => {
        session.request(opts).on('data', (chunk) => {
            total += chunk.byteLength;
            console.log(`${Math.trunc(total / (Date.now() - startTime))} KB/s`);
        }).on('end', () => {
            console.log(`total = ${total}`);
            session.destroy();
            resolve();
        });
    });
}

Output

root@test:~# ping VPS_DOMAIN_NAME
PING VPS_DOMAIN_NAME (VPS_IP) 56(84) bytes of data.
64 bytes from VPS_IP: icmp_seq=1 ttl=54 time=201 ms
root@test:~# curl --http2 https://VPS_DOMAIN/TEST_FILENAME > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  245M  100  245M    0     0  5533k      0  0:00:45  0:00:45 --:--:-- 6470k
root@test:~# node testdownload.js
connected
24 KB/s
47 KB/s
43 KB/s
59 KB/s
73 KB/s
73 KB/s
79 KB/s
92 KB/s
95 KB/s
...
(it climibed up to about 300KB/s and then stayed steady)
...
308 KB/s
308 KB/s
308 KB/s
308 KB/s
308 KB/s
308 KB/s
total = 257062168

Before You Submit

• I have looked for issues that already exist before submitting this
• My issue follows the guidelines in the README file, and follows the 'How to ask a good question' guide at https://stackoverflow.com/help/how-to-ask

[RedYetiDev]

Hi! Can you reproduce on a non-VPS device?

[segfault-bilibili]

Okay I've successfully reproduced this problem inside virtual machines.

I'm running this test with VMware Workstation 17.5.2. There are two VMs involved. Both share one common host-only network. Both are booted with ubuntu-24.04-desktop-amd64.iso.

edit: oh forgot to mention, the client VM is configured to have artificial 100ms outbound + 100ms inbound latency.

The server runs Caddy server v2.8.4 with this Caddyfile:

server.local {
        bind SERVER_VM_IP
        root * /var/www
        file_server
        tls internal
}

On the client VM:

ubuntu@ubuntu:~# ping server.local
PING server.local (SERVER_VM_IP) 56(84) bytes of data.
64 bytes from server.local (SERVER_VM_IP): icmp_seq=1 ttl=64 time=205 ms
64 bytes from server.local (SERVER_VM_IP): icmp_seq=2 ttl=64 time=205 ms
^C
--- server.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 204.965/205.100/205.236/0.135 ms
ubuntu@ubuntu:~# wget --no-check-certificate https://server.local/node.tar -O /dev/null
--2024-07-23 20:09:46--  https://server.local/node.tar
Resolving server.local (server.local)... SERVER_VM_IP
Connecting to server.local (server.local)|SERVER_VM_IP|:443... connected.
WARNING: cannot verify server.local's certificate, issued by ‘CN=Caddy Local Authority - ECC Intermediate’:
  Unable to locally verify the issuer's authority.
HTTP request sent, awaiting response... 200 OK
Length: 181248000 (173M) [application/x-tar]
Saving to: ‘/dev/null’

/dev/null           100%[===================>] 172.85M  12.0MB/s    in 17s

2024-07-23 20:10:04 (10.3 MB/s) - ‘/dev/null’ saved [181248000/181248000]

ubuntu@ubuntu:~$ ~/node-v22.5.1-linux-x64/bin/node testdownload.js
connected
27 KB/s
31 KB/s
39 KB/s
62 KB/s
71 KB/s
71 KB/s
79 KB/s
102 KB/s
98 KB/s
98 KB/s
99 KB/s
99 KB/s
119 KB/s
119 KB/s
...
252 KB/s
252 KB/s
252 KB/s
253 KB/s
253 KB/s
(still running, seems to converge to about 250KB/s)