ci: Need to run QNS on push to main

[larseggert]

So we get baseline data

[github-actions]

Failed Interop Tests

QUIC Interop Runner, client vs. server

• aioquic vs. neqo-latest: R A
• chrome vs. neqo-latest: 3
• go-x-net vs. neqo-latest: U A
• kwik vs. neqo-latest: A
• lsquic vs. neqo-latest: R A
• msquic vs. neqo-latest: Z A L1
• mvfst vs. neqo-latest: Z 3 A L1 C1
• neqo vs. neqo-latest: A
• neqo-latest vs. aioquic: Z
• neqo-latest vs. kwik: Z
• neqo-latest vs. lsquic: C1
• neqo-latest vs. msquic: Z A L1 C1
• neqo-latest vs. mvfst: DC B A L1 L2 C1 C2
• neqo-latest vs. neqo: A
• neqo-latest vs. neqo-latest: A
• neqo-latest vs. ngtcp2: Z
• neqo-latest vs. quic-go: Z
• neqo-latest vs. quiche: L1 C1
• neqo-latest vs. quinn: E A
• neqo-latest vs. s2n-quic: R
• neqo-latest vs. xquic: Z A C1
• ngtcp2 vs. neqo-latest: A
• picoquic vs. neqo-latest: R A
• quic-go vs. neqo-latest: A
• quiche vs. neqo-latest: R 3 A
• quinn vs. neqo-latest: R Z E A
• s2n-quic vs. neqo-latest: E A
• xquic vs. neqo-latest: M A

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

• aioquic vs. neqo-latest: H DC LR C20 M S Z 3 B L1 L2 C1 C2 6
• go-x-net vs. neqo-latest: H DC LR M B L2 C2 6
• kwik vs. neqo-latest: H DC LR C20 M S R Z 3 B U L1 L2 C1 C2 6 V2
• lsquic vs. neqo-latest: H DC LR M S 3 B E L1 L2 C1 C2 6 V2
• msquic vs. neqo-latest: H DC LR C20 M S R B U L2 C1 C2 6 V2
• mvfst vs. neqo-latest: H DC LR M B L2 C2 6
• neqo vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2
• neqo-latest vs. aioquic: H DC LR C20 M S R 3 B U A L1 L2 C1 C2 6
• neqo-latest vs. go-x-net: H DC LR M B U A L2 C2 6
• neqo-latest vs. haproxy: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6 V2
• neqo-latest vs. kwik: H DC LR C20 M S R 3 B U A L1 L2 C1 C2 6 V2
• neqo-latest vs. lsquic: H DC LR C20 M S R Z 3 B U E A L1 L2 C2 6 V2
• neqo-latest vs. msquic: H DC LR C20 M S R B U L2 C2 6 V2
• neqo-latest vs. mvfst: H LR M R Z 3 U 6
• neqo-latest vs. neqo: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2
• neqo-latest vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2
• neqo-latest vs. nginx: H DC LR C20 M S R Z 3 B U A L1 L2 C1 C2 6
• neqo-latest vs. ngtcp2: H DC LR C20 M S R 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. picoquic: H DC LR C20 M S R Z 3 B U E A L1 L2 C1 C2 6 V2
• neqo-latest vs. quic-go: H DC LR C20 M S R 3 B U A L1 L2 C1 C2 6
• neqo-latest vs. quiche: H DC LR C20 M S R Z 3 B U A L2 C2 6
• neqo-latest vs. quinn: H DC LR C20 M S R Z 3 B U L2 C2 6
• neqo-latest vs. s2n-quic: H DC LR C20 M S 3 B U E A L1 L2 C1 C2 6
• neqo-latest vs. xquic: H DC LR C20 M R 3 B U L1 L2 C2 6
• ngtcp2 vs. neqo-latest: H DC LR C20 M S R Z 3 B U E L1 L2 C1 C2 6 V2
• picoquic vs. neqo-latest: H DC LR C20 M S Z 3 B U E L1 L2 C1 C2 6 V2
• quic-go vs. neqo-latest: H DC LR C20 M S R Z 3 B U L1 L2 C1 C2 6
• quiche vs. neqo-latest: H DC LR M S Z B L1 L2 C1 C2 6
• quinn vs. neqo-latest: H DC LR C20 M S 3 B U L2 C2 6
• s2n-quic vs. neqo-latest: H DC LR M S R 3 B L1 L2 C1 C2 6
• xquic vs. neqo-latest: H DC LR C20 S R Z 3 B U L1 L2 C1 C2 6

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

• aioquic vs. neqo-latest: U E V2
• chrome vs. neqo-latest: H DC LR C20 M S R Z B U E A L1 L2 C1 C2 6 V2
• go-x-net vs. neqo-latest: C20 S R Z 3 E L1 C1 V2
• kwik vs. neqo-latest: E
• lsquic vs. neqo-latest: C20 Z U
• msquic vs. neqo-latest: 3 E
• mvfst vs. neqo-latest: C20 S R U E V2
• neqo-latest vs. aioquic: E V2
• neqo-latest vs. go-x-net: C20 S R Z 3 E L1 C1 V2
• neqo-latest vs. haproxy: E
• neqo-latest vs. kwik: E
• neqo-latest vs. msquic: 3 E
• neqo-latest vs. mvfst: C20 S E V2
• neqo-latest vs. nginx: E V2
• neqo-latest vs. quic-go: E V2
• neqo-latest vs. quiche: E V2
• neqo-latest vs. quinn: L1 C1 V2
• neqo-latest vs. s2n-quic: Z V2
• neqo-latest vs. xquic: S E V2
• quic-go vs. neqo-latest: E V2
• quiche vs. neqo-latest: C20 U E V2
• quinn vs. neqo-latest: L1 C1 V2
• s2n-quic vs. neqo-latest: C20 Z U V2
• xquic vs. neqo-latest: E V2

[github-actions]

Benchmark results

Performance differences relative to 8c4411a.

• drain a timer quickly time: [311.87 ns 319.96 ns 327.43 ns]
  change: [-2.1508% +0.0365% +2.4311%] (p = 0.97 > 0.05)
  No change in performance detected.

• coalesce_acked_from_zero 1+1 entries
  time: [198.13 ns 198.58 ns 199.07 ns]
  change: [+0.9224% +1.7078% +2.2743%] (p = 0.00 < 0.05)
  Change within noise threshold.

• coalesce_acked_from_zero 3+1 entries
  time: [241.26 ns 241.94 ns 242.66 ns]
  change: [+1.3124% +1.7387% +2.1754%] (p = 0.00 < 0.05)
  💔 Performance has regressed.

• coalesce_acked_from_zero 10+1 entries
  time: [239.33 ns 239.94 ns 240.70 ns]
  change: [+0.7394% +1.2552% +1.6914%] (p = 0.00 < 0.05)
  Change within noise threshold.

• coalesce_acked_from_zero 1000+1 entries
  time: [219.92 ns 220.15 ns 220.39 ns]
  change: [+0.4898% +1.1926% +2.0280%] (p = 0.00 < 0.05)
  Change within noise threshold.

• RxStreamOrderer::inbound_frame()
  time: [118.95 ms 119.00 ms 119.06 ms]
  change: [-0.4866% -0.4081% -0.3299%] (p = 0.00 < 0.05)
  Change within noise threshold.

• transfer/Run multiple transfers with varying seeds
  time: [118.39 ms 118.62 ms 118.86 ms]
  thrpt: [33.654 MiB/s 33.720 MiB/s 33.788 MiB/s]
  change:
  time: [-0.6585% -0.3766% -0.0866%] (p = 0.01 < 0.05)
  thrpt: [+0.0867% +0.3780% +0.6628%]
  Change within noise threshold.

• transfer/Run multiple transfers with the same seed
  time: [118.28 ms 118.50 ms 118.72 ms]
  thrpt: [33.692 MiB/s 33.756 MiB/s 33.819 MiB/s]
  change:
  time: [-0.5428% -0.2821% -0.0024%] (p = 0.04 < 0.05)
  thrpt: [+0.0024% +0.2829% +0.5458%]
  Change within noise threshold.

• 1-conn/1-100mb-resp (aka. Download)/client
  time: [1.0927 s 1.1015 s 1.1108 s]
  thrpt: [90.026 MiB/s 90.783 MiB/s 91.513 MiB/s]
  change:
  time: [-4.1792% -2.8375% -1.6521%] (p = 0.00 < 0.05)
  thrpt: [+1.6798% +2.9204% +4.3615%]
  💚 Performance has improved.

• 1-conn/10_000-parallel-1b-resp (aka. RPS)/client
  time: [427.73 ms 430.32 ms 432.90 ms]
  thrpt: [23.100 Kelem/s 23.239 Kelem/s 23.379 Kelem/s]
  change:
  time: [-1.3270% -0.5466% +0.1798%] (p = 0.18 > 0.05)
  thrpt: [-0.1795% +0.5496% +1.3448%]
  No change in performance detected.

• 1-conn/1-1b-resp (aka. HPS)/client
  time: [48.563 ms 48.853 ms 49.147 ms]
  thrpt: [20.347 elem/s 20.470 elem/s 20.592 elem/s]
  change:
  time: [-1.1003% +0.4529% +1.9453%] (p = 0.58 > 0.05)
  thrpt: [-1.9082% -0.4509% +1.1126%]
  No change in performance detected.

Client/server transfer results

Transfer of 134217728 bytes over loopback.

Client	Server	CC	Pacing	Mean [ms]	Min [ms]	Max [ms]	Relative
msquic	msquic			749.1 ± 234.9	443.9	1059.1	1.00
neqo	msquic	reno	on	1065.6 ± 176.1	763.4	1384.2	1.00
neqo	msquic	reno		936.6 ± 240.3	739.9	1438.2	1.00
neqo	msquic	cubic	on	963.3 ± 286.4	753.2	1588.7	1.00
neqo	msquic	cubic		908.4 ± 238.7	760.9	1426.9	1.00
msquic	neqo	reno	on	4504.6 ± 268.9	4190.2	5105.8	1.00
msquic	neqo	reno		4320.7 ± 184.8	3975.6	4619.8	1.00
msquic	neqo	cubic	on	4420.0 ± 145.7	4234.7	4664.8	1.00
msquic	neqo	cubic		4407.7 ± 213.2	4139.3	4710.9	1.00
neqo	neqo	reno	on	3716.4 ± 433.7	3145.0	4623.5	1.00
neqo	neqo	reno		3762.9 ± 154.3	3489.5	3957.1	1.00
neqo	neqo	cubic	on	4323.3 ± 338.4	3631.7	4895.9	1.00
neqo	neqo	cubic		4038.7 ± 438.6	2988.6	4481.3	1.00

⬇️ Download logs