feature: improve read / write performance

[aoblet]

Context
On fast storage, reading with a specific block size can drastically improve fetch performance.
Default value is 256K. On scale-out or flash disk storage, this value can be set higher than 1MB.

Update
This PR adds two things:

• max-map-size
  This option acts as count argument in read(int fd, void *buf, size_t count)
• write-size
  This option forces the buffer size when writing a file on the receiver.

Result
Tests on my side resulted on rsync whole-file, 5x time faster with the following specs:

time /opt/rsync/bin/rsync --progress -avH  --whole-file --max-map-size=$((4*1024*1024)) --write-size=$((512*1024)) *.zip --rsync-path /opt/rsync/bin/rsync -e "/opt/hpnssh/bin/ssh -c [email protected]" dst:/path/to/sync/

• filesystem: rozofs / zfs
• number of disks 60
• max-map-size 4MB
• network 10Gb link / latency 0.8ms
• hpnssh / 4800 MTU

DATASET 2x 1,3 GB zip files Total: 2,6 GB

CONFIG	TIME	MB/S	DD LOCAL	READ_BLOCK_SIZE	IO_BUFFER_SIZE	MTU	SSH	HPNSSH	DAEMON RSYNC	RATIO PERF
dd local	3,3	758	Yes	4096	No	No	No	No	No	0
rsync native	24,2	106	No	256	32	1500	Yes	No	No	1
rsync + hpn	22,4	114	No	256	32	1500	No	Yes	No	1,08
rsync + hpn aes128 + mtu 4800 end to end + read block size 1024	10,9	235	No	1024	32	4800	No	Yes	No	2,22
rsync daemon	7,6	336	No	1024	32	4800	No	No	Yes	3,17
rsync + hpn aes128 + mtu 4800 end to end + read block size 4096	4,9	522	No	4096	1024	4800	No	Yes	No	4,93

[realsimix]

Hi,

This looks interesting, just a cosmetic thing: maybe the manpage should be changed like so?

--one-file-system, -x    don't cross filesystem boundaries
--block-size=SIZE, -B    force a fixed checksum block-size
--max-map-size=SIZE      force mmap read block size (expressed in bytes, useful for fast storage, default 256K)
--write-size=SIZE        force write block size (expressed in bytes, default 32K)
--rsh=COMMAND, -e        specify the remote shell to use
--rsync-path=PROGRAM     specify the rsync to run on remote machine

When using this patch but without specifying the new options, will rsync run with the same parameters as when running without this patch?

Thanks,
Simon

[aoblet]

@realsimix sure thing see 8dd0b8e.

Yes options are backward compatible.
Rsync original behavior is respected if options are not set.

[aoblet]

@WayneD ?

[tridge]

i think we will need a MAX_MAP_SIZE_DEFAULT define, and make this check be if (max_map_size != MAX_MAP_SIZE_DEFAULT)
otherwise we will be sending this option always, and the remove rsync may not understand it, so would break a lot of setups

[tridge]

same issue as max_map_size

[tridge]

might be easier to use in kBytes?