feat: add threads to zstd compression

[eshaz]

Features

• Add --compress-threads=n --zt=n where n is the number of threads to spawn when compressing with zstd.

[eshaz]

Quick bump, @WayneD have you had a chance to look at this?

[tridge]

@eshaz i'm helping @WayneD with maintainence now.
Can you give some results of performance testing for this?

[eshaz]

@eshaz i'm helping @WayneD with maintainence now. Can you give some results of performance testing for this?

@tridge Since opening this PR, I've been using this branch to copy files to a home that has a 30Mb/s internet connection. Using a high zstd compression level helps considerably, but as the level is increased, compression becomes CPU bound when only a single thread is used to compress.

Here are a few benchmarks gathered using --stats with a few different compression levels and thread counts. The source and destination were connected over a 10GbE local link using ssh.

Benchmarks

test.file total size is 1,680,379,904 bytes

zstd, level 8
rsync -aiv --zc=zstd --zl=8 --zt=n --stats test.file user@server:/home/user/

--zt=1  sent 1,115,365,707 bytes  received 35 bytes  40,558,754.25 bytes/sec
--zt=2  sent 1,115,364,941 bytes  received 35 bytes  71,959,030.71 bytes/sec
--zt=4  sent 1,115,364,711 bytes  received 35 bytes  106,225,213.90 bytes/sec
--zt=8  sent 1,115,364,645 bytes  received 35 bytes  106,225,207.62 bytes/sec
--zt=16 sent 1,115,364,643 bytes  received 35 bytes  82,619,605.78 bytes/sec

zstd, level 20
rsync -aiv --zc=zstd --zl=20 --zt=n --stats test.file user@server:/home/user/

--zt=1  sent 941,434,753 bytes  received 35 bytes  709,178.75 bytes/sec
--zt=2  sent 941,434,339 bytes  received 35 bytes  1,278,254.41 bytes/sec
--zt=4  sent 941,433,805 bytes  received 35 bytes  2,186,838.19 bytes/sec
--zt=8  sent 941,433,593 bytes  received 35 bytes  2,892,269.21 bytes/sec
--zt=16 sent 941,433,699 bytes  received 35 bytes  3,819,203.79 bytes/sec

no compression (control)
rsync -aiv --stats test.file user@server:/home/user/
sent 1,680,790,259 bytes  received 35 bytes  197,740,034.59 bytes/sec

CPU info

$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               GenuineIntel
  Model name:            11th Gen Intel(R) Core(TM) i9-11980HK @ 2.60GHz
    CPU family:          6
    Model:               141
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            1
    CPU max MHz:         5000.0000
    CPU min MHz:         800.0000
    BogoMIPS:            6604.80
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse 
                         sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xt
                         opology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ss
                         se3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f1
                         6c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp 
                         ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms inv
                         pcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512
                         bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts hwp hwp_notify hw
                         p_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx5
                         12_vnni avx512_bitalg avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2intersect md_clear ibt f
                         lush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   384 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    10 MiB (8 instances)
  L3:                    24 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:         
  Gather data sampling:  Mitigation; Microcode
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI Sys
                         call hardening, KVM SW loop
  Srbds:                 Not affected
  Tsx async abort:       Not affected