Enlarge TCP recv_buf to improve throughput

Image uploads performed via the web console are 3-4x slower than
uploads performed via the Oxide CLI[0]. We found that the CLI creates
8 separate TCP connections to upload the image chunks, while the console
uses HTTP/2 to multiplex a single TCP connection six ways. The default
TCP `recv_buf` size on Helios is 128 KB, which limits window size and
therefore the number of packets that can be sent in parallel. By
increasing this value to 1 MB, we can increase single-connection
throughput by ~3x, bringing console performance to rough parity with
the CLI.

This does increase the amount of memory a potential DoS attack could
consume, but 1 MB is still quite small relative to the total resources
available on a compute sled.

While we're at it, also update the TCP congestion control algorithm to
`cubic` from its default value of `sunreno`, which may also help improve
throughput.

Closes #6601

[0] oxidecomputer/console#2096

Assisted-by: David Crespo <[email protected]>

Author: wfchandler

illumos-utils/src/ipadm.rs

@@ -197,4 +197,36 @@ impl Ipadm {
         Self::ensure_ip_addrobj_exists(&addrobj, AddrObjType::DHCP)?;
         Ok(())
     }
+
+    /// Set TCP recv_buf to 1 MB.
+    pub fn set_tcp_recv_buf() -> Result<(), ExecutionError> {
+        let mut cmd = std::process::Command::new(PFEXEC);
+        let cmd = cmd.args(&[
+            IPADM,
+            "set-prop",
+            "-t",
+            "-p",
+            "recv_buf=1000000",
+            "tcp",
+        ]);
+        execute(cmd)?;
+
+        Ok(())
+    }
+
+    /// Set TCP congestion control algorithm to `cubic`.
+    pub fn set_tcp_congestion_control() -> Result<(), ExecutionError> {
+        let mut cmd = std::process::Command::new(PFEXEC);
+        let cmd = cmd.args(&[
+            IPADM,
+            "set-prop",
+            "-t",
+            "-p",
+            "congestion_control=cubic",
+            "tcp",
+        ]);
+        execute(cmd)?;
+
+        Ok(())
+    }
 }

zone-setup/src/bin/zone-setup.rs

@@ -563,6 +563,17 @@ async fn common_nw_set_up(
     Ipadm::set_interface_mtu(&datalink)
         .with_context(|| format!("failed to set MTU on datalink {datalink}"))?;
 
+    info!(
+        log, "Setting TCP recv_buf size to 1 MB";
+    );
+    Ipadm::set_tcp_recv_buf().context("failed to set TCP recv_buf")?;
+
+    info!(
+        log, "Setting TCP congestion control algorithm to cubic";
+    );
+    Ipadm::set_tcp_congestion_control()
+        .context("failed to set TCP congestion_control")?;
+
     if static_addrs.is_empty() {
         info!(
             log,